Studies have shown that aerobic exercise has the potential to improve cognition and reduce brain atrophy in older adults. However, the literature is equivocal with regards to the specificity or generality of these effects. To this end, we report results on cognitive function and brain structure from a 6-month training intervention with 60 sedentary adults (64–78 years) randomized to either aerobic training or stretching and toning control training. Cognitive functions were assessed with a neuropsychological test battery in which cognitive constructs were measured using several different tests. Freesurfer was used to estimate cortical thickness in frontal regions and hippocampus volume. Results showed that aerobic exercisers, compared to controls, exhibited a broad, rather than specific, improvement in cognition as indexed by a higher “Cognitive score,” a composite including episodic memory, processing speed, updating, and executive function tasks (p = 0.01). There were no group differences in cortical thickness, but additional analyses revealed that aerobic fitness at baseline was specifically related to larger thickness in dorsolateral prefrontal cortex (dlPFC), and hippocampus volume was positively associated with increased aerobic fitness over time. Moreover, “Cognitive score” was related to dlPFC thickness at baseline, but changes in “Cognitive score” and dlPFC thickness were associated over time in the aerobic group only. However, aerobic fitness did not predict dlPFC change, despite the improvement in “Cognitive score” in aerobic exercisers. Our interpretation of these observations is that potential exercise-induced changes in thickness are slow, and may be undetectable within 6-months, in contrast to change in hippocampus volume which in fact was predicted by the change in aerobic fitness. To conclude, our results add to a growing literature suggesting that aerobic exercise has a broad influence on cognitive functioning, which may aid in explaining why studies focusing on a narrower range of functions have sometimes reported mixed results.