Recent investigations suggest the Hypothalamic-Pituitary-Adrenal Axis (HPA) plays an important role in neuropsychiatric and neurocognitive disorders in older adults. Effects of HPA activity on cognition and depression are well documented. It is unclear, however, if this relationship is mediated by psychosocial stress. Here we report on the current status of the literature in this field and also present our findings from a large, longitudinal investigation of the interaction of psychosocial stress and HPA activity on neurocognitive and neuropsychiatric function. In a sample of 162 community-dwelling older adults, aged 60–102 years, we investigated the relationship of waking cortisol, slope of diurnal cortisol and psychosocial stress to cognition, mood and brain function. Older adults with increased waking cortisol exhibited poorer memory performance (p < 0.01). This effect was strongest in individuals with the Apolipoprotein ε 4 allele, who are at increased risk for the development of cognitive decline and dementia. Additionally, older adults with higher waking cortisol had reduced hippocampal volumes (p < 0.01). There was no effect of slope of diurnal cortisol on any of the cognitive or brain measures, in either ε 4 or non-ε 4 carriers. Although we observed no impact of cortisol on mood, individuals with the short allele of the 5HTT gene promoter polymorphism region (5HTTLPR), which is associated with lower levels of serotonin uptake, had poorer cognitive function and higher levels of cortisol. Strikingly, none of these observed effects were impacted by psychosocial stress. Implications of our findings for the role of neuroendocrine function in age-related neurocognitive and neuropsychiatric disorders will be discussed.