UAVs are an unfortunate inevitability. Their inability to truly "see and avoid" is what is keeping them in restricted airspace while CONUS. Of course nextgen and ADSB are supposed to fix the "see and avoid" issue along with companies developing small on board radar. The requirement for visual observers was originally a 3rd class medical but when the time frames changed for how long the medicals were good for they had to go to a 2nd class medical. The medical is merely for the vision test. The FAA was not kosher with a person going 5 years between tests. Along with the restricted airspace UAVs must operate under an FAA COA which governs their conditions of operations.
Privately operated UAVs that are not operating under the FAAs oversight or approval I believe are a serious safety hazard. Fortunately at this point they are relatively small.
The problem I foresee is the military attempting to use semi-autonomous UAVs and have operators controlling them in the NAS without training them to at least private pilot standards. Bringing chaos to a structured system. Nextgen and ADSB can't help someone who doesn't know how to help themselves. That's just my $.02.