In order for an alleged violation of 14 CFR to be investigated and elevated to an Enforcement case, a case has to be built by "preponderance of the evidence" which is the standard required by Administrative Law. See FAA Order 2150.3b
No, in order to stand up to appeal an enforcement action must have some basis in preponderance, but the FAA has a long history of taking action against pilots without being able to back it up. Under Administrative law, one is guilty until proven innocent; one's first chance for defense comes after one has already been "violated."
The FAA has often been more than willing to fire the shot knowing that often pilots don't appeal it, or that the damage is done anyway.