Also consider that aerodynamic controls were insufficient for normal attitude control at altitude. Spacecraft attitude thrusters were used for at least yaw, you can see them on the nose during the preflight. Using those to control an F-104 has got to be a special kind of special.