I wouldn't worry to much. I've seen several FOs in your situation or even worse, where they had caused a deviation. The feds almost always go for the Captain. It will also help you if you have a commercial instead of a ATP, as the FAA usually has more lenience.
I would say worse case scenario you will get FAA counseling over the phone and they will consider it a warning.
Leave the company if you have something else lined up. Get out while you can, these things will repeat themselves especially if you keep flying with this guy.