In this article we discuss how apparently indicators that may appear straightforward, such as gender differences, need to be interpreted with extreme care. In particular, we consider how the assessment framework, and the methodology of international surveys, may have a potential impact on the results and on the indicators. Through analysis of Programme for International Student Assessment (PISA) data we show how increases or decreases in the achievement of some groups of students (either of whole countries or population subgroups like males and females) can, at least partially, result from variations in the framework or the methodology of the respective assessments. The analyses provide evidence that the gender gap is larger for open-ended questions, for continuous texts and for more cognitively demanding reading tasks.