Climate extremes, such as droughts or heat waves, can lead to harvest failures and threaten the livelihoods of agricultural producers and the food security of communities worldwide. Improving our understanding of their impacts on crop yields is crucial to enhance the resilience of the global food system. This study analyses, to our knowledge for the first time, the impacts of climate extremes on yield anomalies of maize, soybeans, rice and spring wheat at the global scale using sub-national yield data and applying a machine-learning algorithm. We find that growing season climate factors-including mean climate as well as climate extremes-explain 20%-49% of the variance of yield anomalies (the range describes the differences between crop types), with 18%-43% of the explained variance attributable to climate extremes, depending on crop type. Temperature-related extremes show a stronger association with yield anomalies than precipitation-related factors, while irrigation partly mitigates negative effects of high temperature extremes. We developed a composite indicator to identify hotspot regions that are critical for global production and particularly susceptible to the effects of climate extremes. These regions include North America for maize, spring wheat and soy production, Asia in the case of maize and rice production as well as Europe for spring wheat production. Our study highlights the importance of considering climate extremes for agricultural predictions and adaptation planning and provides an overview of critical regions that are most susceptible to variations in growing season climate and climate extremes.