Unmanned aerial vehicles (UAVs) are emerging as enablers for supporting many applications and services, such as precision agriculture, search and rescue, temporary network deployment or coverage extension, and security. UAVs are being considered for integration into emerging 5G networks as aerial users or network support nodes. We propose to leverage UAVs in 5G to assist in the prevention, detection, and recovery of attacks on 5G networks. Specifically, we consider jamming, spoofing, eavesdropping, and the corresponding mitigation mechanisms that are enabled by the versatility of UAVs. We introduce the hot zone, safe zone, and UAV-based secondary authorization entity, among others, to increase the resilience and confidentiality of 5G radio access networks and services. We present simulation results and discuss open issues and research directions, including the need for experimental evaluation and a research platform for prototyping and testing the proposed technologies.