Is it selfish? Maybe, maybe not. It depends on how you approach it.
First off, if you want to be a military pilot and you have your degree, at least in the Navy, you apply for that, and if you don't get a pilot slot, you don't have to join. It's not like ROTC or the academy where you might end up driving a boat against your wishes. I'm not sure if they give the color blindness test at MEPS, but they do have alternatives besides the book with the numbers. I'd advise doing some more research on the color blindness tests approved and administered by the different services. That might make this question a moot point.
Secondly, if both you and your wife weigh the pros and cons and conclude that making you a military pilot/grunt/boat driver/desk jockey is the best course of action for your family... then it is most certainly not selfish. If you join without her knowing exactly what will be required of her and your family, then it is selfish.
The military can be a very good deal. It could also be a tremendous and insurmountable obstacle for your family/marriage. It depends on what you guys want out of life.