Large-scale survey studies of educational achievement are becoming increasingly frequent, and they are visibly present in both educational policy debates and within the educational research community. One main aim of these studies is to provide descriptions of inputs, processes and outcomes, and another aim is to provide explanations of how different factors interrelate to produce educational outcomes. These aims are difficult to reach, which in combination with the fact that the comparative studies are typically more policy driven than theory driven, are reasons why these studies are contested on quality grounds. In this article, a set of fundamental methodological challenges related to the validity of the measurement instruments and to the possibility of making inferences about causality are identified and discussed in relation to examples of different studies. Strengths and weaknesses of different research approaches are discussed, and it is proposed that the dichotomy between qualitative and quantitative approaches should be replaced with distinctions between low-and high-level inference approaches with respect to data, generalization and explanation. It is concluded that while the international studies easily invite misuse and misinterpretation, they also offer possibilities for improving the quality of educational research, because the high-quality data generated by these studies can be taken advantage of in research on causal effects of factors in and out of educational systems.