The cognitive decline associated with normal aging was long believed to be due primarily to decreased synaptic density and neuron loss. Recent studies in both humans and non-human primates have challenged this idea, pointing instead to disturbances in white matter (WM) including myelin damage. Here, we review both cross-sectional and longitudinal studies in humans and non-human primates that collectively support the hypothesis that WM disturbances increase with age starting at middle age in humans, that these disturbances contribute to age-related cognitive decline, and that age-related WM changes may occur as a result of free radical damage, degenerative changes in cells in the oligodendrocyte lineage, and changes in microenvironments within WM.