Climatic impact on agricultural production is a serious concern, as it is directly linked to food security and poverty. Whereas there are empirical studies that examine this issue with parametric approaches focusing on the “mean” level of variables, few studies have addressed climatic impacts in general settings. Given this paucity, we characterize the impacts on crop yield distributions with a non-parametric approach. We examine the case of rice yield in Andhra Pradesh, India, an important state producing rice as a main crop but reported to be vulnerable to climate change. Employing 34 years of data, we apply quantile regressions to untangle the climatic impacts across the quantiles of rice yield, finding three main results. First, substantial heterogeneity in the impacts of climatic variables can be found across the yield distribution. Second, the direction of the climatic impacts on rice yield highly depends on agro-climatic zones. Third, seasonal climatic impacts on rice yield are significant. More specifically, a monsoon-dependent crop is more sensitive to temperature and precipitation, whereas a winter crop remains largely resilient to changes in the levels of climate variables. These findings clarify the idiosyncratic climatic impacts on agriculture in India, and call for location- and season-specific adaptation policies.