Saggy skin, on both the face and body, is often associated with the loss of fat. The deterioration or reduction of collagen and elastin in the dermis are another cause of saggy skin. While anyone can get saggy skin, it’s more likely to occur in people as they age.