This paper explores the usefulness of OpenAIRE, BASE, Recolecta, and Google Scholar (GS) for evaluating open access (OA) policies that demand a deposit in a repository. A case study was designed focusing on 762 financed articles with a project of FIS-2012 of the Instituto de Salud Carlos III, the Spanish national health service's main management body for health research. Its finance is therefore subject to the Spanish Government OA mandate. A search was carried out for full-text OA copies of the 762 articles using the four tools being evaluated and with identification of the repository housing these items. Of the 762 articles concerned, 510 OA copies were found of 353 unique articles (46.3%) in 68 repositories. OA copies were found of 81.9% of the articles in PubMed Central and copies of 49.5% of the articles in an institutional repository (IR). BASE and GS identified 93.5% of the articles and OpenAIRE 86.7%. Recolecta identified just 62.2% of the articles deposited in a Spanish IR. BASE achieved the greatest success, by locating copies deposited in IR, while GS found those deposited in disciplinary repositories. None of the tools identified copies of all the articles, so they need to be used in a complementary way when evaluating OA policies.