UAV's came onto the scene rapidly to fill a military requirement quickly and at a reasonable (by DoD standards) cost. They were successful at that.
Despite all the lost platforms, UAS' in general have been a resounding military success story...this from the perspective of a senior SOF guy/ISR consumer.
But none of that translates to suitability for routine ops in civilian airspace, or civilian use. Different requirements, different standards, different cost/safety equations.
The various technical issues of military platforms is not good reason why UAVs should be banned from civil airspace. It does however, clearly highlight that current systems/tech is not suitable for civil use. By civil use I mean unrestricted ops without chase planes, restricted areas, or impairment of existing manned operations...ie, you cannot require each manned platform to install a new $200K ADS-B version to support UAS see-and-avoid. You also cannot ban manned aircraft from half the NAS.
The UAS lobby would LOVE nothing more than to place the burden of UAS integration on the manned community, ie the manned community would eat the cost of regulations and hardware to accommodate UAS. This is obviously unreasonable and will be fought tooth-and-nail by the alphabet soup.
Current military technology simply won't work in civil airspace. You almost need a clean-slate development program, which will require a lot of money and the collaboration of government, airframers, and customers...and all three of those entities will have to rely on and trust each other. Airframers won't do the R&D if they can't sell the product, customers won't buy them if regulations do not permit operations, and the government needs to understand the technology in detail in order to develop the regs. Three-way chicken vs. egg scenario...it's gonna take a while. And it's going to cost a lot more than military systems which operate in restricted airspace.