Southern and Eastern Africa Consortium for Monitoring Educational Quality 2000

Sampling
Sample designs in the field of education are usually prepared amid a network of competing constraints. These designs need to adhere to established survey sampling theory and, at the same time, give due recognition to the financial, administrative, and socio-political settings in which they are to be applied. The "best" sample design for a particular project is one that provides levels of sampling accuracy that are acceptable in terms of the main aims of the project, while simultaneously limiting cost, logistic, and procedural demands to manageable levels. The major constraints that were established prior to the preparation of the sample designs for the SACMEQ II Project have been listed below.

Target Population: The target population definitions should focus on Grade 6 pupils attending registered mainstream government or non-government schools. In addition, the defined target population should be constructed by excluding no more than 5 percent of pupils from the desired target population.

Bias Control: The sampling should conform to the accepted rules of scientific probability sampling. That is, the members of the defined target population should have a known and non-zero probability of selection into the sample so that any potential for bias in sample estimates due to variations from "epsem sampling" (equal probability of selection method) may be addressed through the use of appropriate sampling weights.

Sampling Errors: The sample estimates for the main criterion variables should conform to the sampling accuracy requirements set down by the International Association for the Evaluation of Educational Achievement. That is, the standard error of sampling for the pupil tests should be of a magnitude that is equal to, or smaller than, what would be achieved by employing a simple random sample of 400 pupils.

Response Rates: Each SACMEQ country should aim to achieve an overall response rate for pupils of 80 percent. This figure was based on the wish to achieve or exceed a response rate of 90 percent for schools and a response rate of 90 percent for pupils within schools.

Administrative and Financial Costs: The number of schools selected in each country should recognize limitations in the administrative and financial resources available for data collection.

Other Constraints: The number of pupils selected to participate in the data collection in each selected school should be set at a level that will maximize validity of the within-school data collection for the pupil reading and mathematics tests.

The Specification of the Target Population
The target population for both the SACMEQ I and SACMEQ II Projects was focussed on the Grade 6 level for three main reasons.

First, Grade 6 identified a point near the end of primary schooling where school participation rates were reasonably high for most of the seven countries that participated in the SACMEQ I data collection during 1995-1997, and also reasonably high for most of the fourteen countries that participated in the SACMEQ II collection during 2000-2002. For this reason, Grade 6 represented a point that was suitable for making an assessment of the contribution of primary schooling towards the literacy and numeracy levels of a broad cross-section of society.

Second, the NRCs considered that testing pupils at grade levels lower than Grade 6 was problematic - because in some SACMEQ countries the lower grades were too close to the transition point between the use of local and national languages by teachers in the classroom. This transition point generally occurred at around Grade 3 level - but in some rural areas of some countries it was thought to be as high as Grade 4 level.

Third, the NRCs were of the opinion that the collection of home background information from pupils at grade levels lower than Grade 6 was likely to lack validity for certain key "explanatory" variables. For example, the NRCs felt that children at lower grade levels did not know how many years of education that their parents had received, and they also had difficulty in accurately describing the socioeconomic environment of their own homes (for example, the number of books at home).

The Stratification Procedures
The stratification procedures adopted for the study employed explicit and implicit strata. The explicit stratification variable, "Region", was applied by separating each sampling frame into separate regional lists of schools prior to undertaking the sampling. The implicit stratification variable was "School Size" - as measured by the number of Grade 6 pupils.

The main reason for choosing Region as the explicit stratification variable was that the SACMEQ Ministries of Education wanted to have education administration regions as "domains" for the study. That is, the Ministries wanted to have reasonably accurate sample estimates of population characteristics for each region.

There were two other reasons for selecting Region as the main stratification variable. First, this was expected to provide an increment in sampling precision due to known between-region differences in the educational achievement of pupils - especially between predominantly urban and predominantly rural regions. Second, this approach provided a broad geographical coverage for the sample - which was necessary in order to spread the fieldwork across each country in a manner that prevented the occurrence of excessive administrative demands in particular regions.

The use of School Size as an implicit stratification variable within regions also offered increased sampling precision because it provided a way of sorting the schools from "mostly rural" (small schools) to "mostly urban" (large schools). It was known that this kind of sorting was linked to the main criterion variables for the study - with urban schools likely to have higher resource levels and better pupil achievement scores than rural schools.

Sample Design Framework
The SACMEQ II sample designs were prepared by using a specialized software system (SAMDEM) that enabled the high-speed generation of a range of sampling options which satisfied the statistical accuracy constraints set down for the project, and at the same time also addressed the logistical and financial realities of each country.

Note: Details of sampling design procedures are presented in the "Mauritius Working Report".