I agree with USMCFLYR and your own thoughts. You are missing out on all of the thing that happen when you are relying on yourself without the safety net in the right seat.
It may very well be a safety thing - if you recall the JFK Jr crash, one of the interesting things was that he had a very high ration of dual received to total time. There is a school of thought that this left him ill-equipt to handle things on his onw and that was a part of what led to the accident.
Nothing to feel "guilty" about and there's no reason to avoid using dual cross countries to help meet the instrument requirements , but you may well be cheating yourself of valuable experience (and a lot of fun!) by combining all of your cross countries with instruction.