Primate socioecological studies have attempted to derive general frameworks using the average behavioural traits of species or genera to place them into categories. However, with the accumulation of primate studies, it is timely to place more emphasis on understanding within-species variation in social structure. In this review we have four objectives. First, we examine within-species variation in the potential determinants of social structure, including diet, demography, predation and infanticide, and document considerable variation. Second, we present case studies of within-species variation in social structure to illustrate the potential magnitude of this variation. For example, there are cases within a single interbreeding population where multi-male, uni-male, fission–fusion and monogamous groups are found. Third, by examining widespread primate lineages that occur in a variety of habitats, we note that there are differences in the magnitude of variation in social structures across different lineages and as a result we consider phylogenetic constraints on phenotypic variation in social structure. Finally, we reflect on the implications of extensive variation in social structure. We suggest that primate social structure will represent a combination of adaptation to present-day environment and phylogenetic inertia. To advance our understanding of the relative contribution of phylogeny versus ecology we propose two approaches. One approach is to compare groups in the same interbreeding population that inhabit different ecological conditions. Any differences that are found can be attributed to ecological differences, since phylogeny should not play a role within a single population. The second approach is to study distantly related species that have similar social structures to illustrate how similar ecological pressures might be operating to select for parallel social structures.