I wouldn't hesitate for an instant to delay pickup to whatever time ensures that I, and more importantly, the rest of the crew, is sufficiently rested. I wouldn't care what the duty pilot, scheduling or anyone else said.

That said I tend to agree with your last point.