There are two schools of thought here. The first says, "If you don't think that you can or want to do it, then do yourself and everybody else a favor and get out of flying". The second says, "Don't let one bad/marginal experience make you rush to a bad decision". I'd lean toward the second.
Those of us that are old and crotchity have all had experiences such as yours. I've personally stopped a couple of check/training rides to discuss the instructors attitude or approach to his job and whether or not I was going to continue with him/her that day. I once walked out of a debriefing because one of the instructors ticked me of so badly. I had no problem doing these things because I was confident that I was right. And I sure would not have let those experiences lead me to a poor decision. Like smacking the guy or quitting.
Give it time, Grasshoppa.