If you fly into big military or international airports, you likely will never need a military passport. And while you can get by with ID card & NATO orders in NATO countries, it can be a hassle at "real" airports.

If you operate into something less than big airports, in less than developed countries, you may want to have the mil passport available - often they'll want to take them from you either to process, copy, or to hold until you leave - in this case I'd rather give up my brown one than my blue one. Often, having different colored passports amongst the crew can be more of a challenge than all the same color (regardless of brown or blue).