We investigated how local scale variation in human impacts influenced forest structure and tree species richness within Mt Elgon National Park, Uganda. We assessed basal area (BA), stem density, diameter at breast height (dbh) and indicators of human activity in 343 plots in four study sites, on transects running inwards from the boundary of the park. Mt Elgon hosts the only remaining natural forest in a densely populated region (150–1000 p/km2). All study sites suffered past encroachment for agriculture and were in various stages of recovery or renewed-clearing at the time of the study. Areas recovering from encroachment had lower mean BA (BA = 3–11 m2/ha), dbh and often also lower stem densities than forest that had never been cleared (BA = 21–43 m2/ha), even 35 years after abandonment and with restoration planting. Human impacts were found beyond 2 km into the park. Although most activities decreased with distance inside the boundary, their prevalence varied among sites. High coefficients of variation in BA (Cv = 0.8–1.1) and stem density (Cv = 1.0–2.2) within sites, together with the evidence of sustained human activities, suggest that forest use histories strongly influenced local forest structure. Mean BA increased with distance inside the boundary in all sites, but stem densities reflected more complex patterns. Large trees (dbh P 20 cm) were most affected by former clearing for agriculture. The collection of stems used as crop-supports reduced regeneration and the density of smaller stems at one site. In another site, charcoal making was associated with the smallest mean BA and marked variability in forest structure. Grazed forest consisted of large trees with very little regeneration. On forest margins in two sites grazing, generally together with fire and tree-cutting, had eroded the forest edge and prevented regeneration. Human impacts as well as natural gradients had major impacts on species richness patterns. Several areas in intermediate states of disturbance showed higher tree species richness than either old-growth forest or more severely degraded areas. This study illustrates the fine scale variation due to local impacts within one forest.