I've noticed this too. It doesn't bother me. I've seen guys wear flights suits when flying a C-152. I have a military old style brown WWII flying jacket and I've never been in the military. I don't pretend to be. Besides the jacket was a gift. When I'm cold, I wear a jacket. When the sun is in my eyes, I wear sunglasses. Yes, they are Raybans. I do have a non-aviation jacket and non Rayban sunglass that I also wear at times. I don't care if people who wear this stuff are pilots or not.
Many pilots and student pilots go through a phase of "acting and/or looking like a pilot". It's harmless most of the time.