Objective This study aimed to examine the association of loneliness and social isolation on cognition over a 3-year follow-up period in middle- and older-aged adults. Methods Data from a Spanish nationally representative sample were analyzed (n = 1691; aged 50 years or older). Loneliness, social isolation, and cognition (immediate recall, delayed recall, verbal fluency, forward digit span, backward digit span, and a composite cognitive score) were assessed both at baseline and at follow-up. Adjusted generalized estimating equations models were performed. Results Loneliness was significantly associated with lower scores in the composite cognitive score, immediate and delayed recall, verbal fluency, and backward digit span (B = -0.14 to B = -3.16; P < .05) and with a more rapid decline from baseline to follow-up in two out of six cognitive tests. Higher social isolation was associated with lower scores in the composite cognitive score, verbal fluency, and forward digit span (B = -0.06 to B = -0.85; P < .05). The effect of loneliness and social isolation on cognition remained significant after the exclusion of individuals with depression. Conclusions Both loneliness and social isolation are associated with decreased cognitive function over a 3-year follow-up period. The development of interventions that include the enhancement of social participation and the maintenance of emotionally supportive relationships might contribute to cognitive decline prevention and risk reduction.