As a student pilot who's logged only 20 hours, I can tell you I get nervous, too. You're definitely not alone. See, the interesting thing about me is that, even though i have almost zero flying hours logged, i have hundreds of hours of experience flying with my father since i was 3. He isn't a CFI so I couldn't log the hours as dual. So, just like you, i know my way around a plane. He's an incredibly skilled aircraft mechanic (now a mx manager for an airline), too, so i've seen the inner workings of everything from cessnas to 747s.
Honestly i find it wierd that i was cool, confident, and relaxed on my first solo, but terrified on my 3rd when i left the traffic pattern for the first time. I knew what to do, i'd been doing it and watching it get done my entire life. But the "What if's" always got to me. I couldnt bring myself to do stalls because i kept worrying about messing up.
On my final landing of my 3rd solo, i was too focused on slowing down to remember to keep the nose up. As a result of too much weight on the nose n too much speed, the front nosegear vibrated something awful when i tried to brake. Luckily i remained calm, pulled back on the wheel, then lightly tapped the brakes. I got off the runway in time, but used up wayyy too much runway for my own good.
What i think we should do is, like everyone says, just keep soloing. Also, make sure you're flying the plane, and don't let the plane fly you. I'm looking forward to restarting flight school in a few weeks so I can start practicing more. Good luck! You'll do just fine.