YES you can deviate from it. What if we have a similar situation (frequency congested), but you need to turn to remain clear of clouds. Do you deviate from the heading? Heck yeah! Because not deviating from the clearance becomes a FAR violation. Remember who is SOLELY responsible for your flight.

If I'm not 100% sure, I am deviating from the instruction. Especially if I'm outside the Class B, turning to stay outside of it if you're not sure is reasonable, in my opinion. That's like holding your position if you're not sure you can cross a runway... it's just common sense.

Plus, I think the FAA would be hard pressed to violate you for disregarding an ATC instruction when you're OUTSIDE the class B.