I would avoid flying with just one CFI for all of your ratings. For starters, most are looking to build time and turnover at your average flight school is pretty high as a result of it only being a stepping stone. Second, you'll learn much more from a variety of instructors who have had a variety of experiences and a number of ways to do things.
Having said that, yes, it does matter. You're spending a ton of money and you should have an instructor you're going to mesh with. I've had a couple that I coudln't stand and one that I ended a flight early because he had me so angry.