The genetic causes of cancer include both somatic mutations and inherited germline variants. Large-scale tumor sequencing has revolutionized the identification of somatic driver alterations but has had limited impact on the identification of cancer predisposition genes (CPGs). Here we present a statistical method, ALFRED, that tests Knudson's two-hit hypothesis to systematically identify CPGs from cancer genome data. Applied to similar to 10,000 tumor exomes the approach identifies known and putative CPGs - including the chromatin modifier NSD1 - that contribute to cancer through a combination of rare germline variants and somatic loss-of-heterozygosity (LOH). Rare germline variants in these genes contribute substantially to cancer risk, including to similar to 14% of ovarian carcinomas, similar to 7% of breast tumors, similar to 4% of uterine corpus endometrial carcinomas, and to a median of 2% of tumors across 17 cancer types.