The largest animals are marine filter feeders, but the underlying mechanism of their large size remains unexplained. We measured feeding performance and prey quality to demonstrate how whale gigantism is driven by the interplay of prey abundance and harvesting mechanisms that increase prey capture rates and energy intake. The foraging efficiency of toothed whales that feed on single prey is constrained by the abundance of large prey, whereas filter-feeding baleen whales seasonally exploit vast swarms of small prey at high efficiencies. Given temporally and spatially aggregated prey, filter feeding provides an evolutionary pathway to extremes in body size that are not available to lineages that must feed on one prey at a time. Maximum size in filter feeders is likely constrained by prey availability across space and time.