Paternity testing using genetic markers has shown that extra-pair paternity (EPP) is common in many pair-bonded species [1, 2]. Evolutionary theory and empirical data show that extra-pair copulations can increase the fitness of males as well as females [3, 4]. This can carry a significant fitness cost for the social father, who then invests in rearing offspring that biologically are not his own [5]. In human populations, the incidence and correlates of extra-pair paternity remain highly contentious [2, 6, 7]. Here, we use a population-level genetic genealogy approach [6, 8] to reconstruct spatiotemporal patterns in human EPP rates. Using patrilineal genealogies from the Low Countries spanning a period of over 500 years and Y chromosome genotyping of living descendants, our analysis reveals that historical EPP rates, while low overall, were strongly impacted by socioeconomic and demographic factors. Specifically, we observe that estimated EPP rates among married couples varied bymore than an order of magnitude, from 0.4% to 5.9%, and peaked among families with a low socioeconomic background living in densely populated cities of the late 19th century. Our results support theoretical predictions that social context can strongly affect the outcomes of sexual conflict in human populations by modulating the incentives and opportunities for engaging in extra-pair relationships [9-11]. These findings show how contemporary genetic data combined with in-depth genealogies open up a new window on the sexual behavior of our ancestors.