Perception of emotion in the face is a key component of human social cognition and is considered vital for many domains of life; however, little is known about how this ability differs across the lifespan for men and women. We addressed this question with a large community sample (N = 100,257) of persons ranging from younger than 15 to older than 60 years of age. Participants were viewers of the television show "Tout le Monde Joue", and the task was presented on television, with participants responding via their mobile devices. Applying latent variable modeling, and establishing measurement invariance between males and females and across age, we found that, for both males and females, emotion perception abilities peak between the ages of 15 and 30, with poorer performance by younger adults and declining performance after the age of 30. In addition, we show a consistent advantage by females across the lifespan, which decreases in magnitude with increasing age. This large scale study with a wide range of people and testing environments suggests these effects are largely robust. Implications are discussed.