It's all fine and dandy to pull it back for fuel savings, etc.

Till you're going somewhere busy and center asks why you're going so slow. At this point it's either 1) Speed up because you're screwing up flow into the airport or 2) Vectors off course/Ait change because you're screwing up flow into the airport.