Conversion of forestry to agricultural land use is a major change associated with loss of vegetation in the Lake Victoria basin. This study aimed at determining the trend and extent of land use changes, their drivers and the local people’s perception of the effects of land use change on Ssese islands. Change analysis using vegetation cover map 1960, Landsat TM 2001 and Aster 2006 satellite images demonstrated the drastic change in land use associated with the introduction of oil palm plantation occurred between 2001 and 2006. In the period 1960 – 2001, agriculture increased by 473 hectares at the expense of grasslands and forest while a drastic increase by 5223 hectares in agriculture occurred in the period 2001 – 2006. Much as agricultural expansion is known as major driver of deforestation, it is observed that government policy to alleviate poverty through the modernization of agriculture, proved to be the mega driver of land use change in Ssese Islands. Contrary to government policy to alleviate poverty, the local people had a negative view about the oil palm establishment in Ssese islands in preference to the ecological and socio-economic values they attach to the forest.