The OECD “Programme for International Student Assessment” or (PISA) is one of the largest-scale international efforts that have been launched to assess students’ scientific literacy. Such an international assessment would likely exert a profound impact on the science education policies of the participating countries/regions, including Hong Kong. This paper sets out to examine critically how scientific literacy has been assessed by PISA through analyzing its assessment frameworks and released sample items. It was found that the PISA 2000 and 2003 assessments of science have used a narrower definition of scientific literacy, as compared to that of PISA 2006 and what scientific literacy was construed for science education. However, even PISA 2006 appears to be more valid in its assessment framework, its validity was also called into question when the sample items for the trial study were examined. Knowledge about science was found largely about the processes of science, rather than the nature of science as described in the assessment framework. Besides, it intertwined with knowledge of science in a hidden manner. The application of knowledge of science in novel, real-life situations was also jeopardized because of the issue of curricular relevance. Besides these major problems, the article has discussed the problems with the concept of scientifically investigable questions and identifying research question of an investigation. Overall, the findings raised concern over what the PISA’s measure of scientific literacy actually means.