I agree your viewpoint is conservative, but the way the AIM reads(4-4-12), it says you comply with the altitudes/speeds after you get a “descend via” clearance. I’m with you though; I’ve always started complying at the first point on the arrival.