If you served as an AGR to get the retirement, then it you pay it back, because of the retirement. If someone, just does an AGR slot until something better comes around, that person shouldn't be forced to pay back the VSP.
I love how the Air Force still finds way of screwing you, even when you're not in the service anymore.
They state on the website
"Q: If I receive the VSP and later return to active duty do I have to pay the VSP back?
A: Yes, Per law, an officer who accepts VSP and later returns to active duty (regardless of Service), including a full time active duty position in the Reserve or Guard, will have an amount deducted from their pay (schedule to be determined) until the total deducted from basic pay equals the total amount received. Exceptions: members involuntarily recalled to active duty or full-time National Guard duty IAW certain subsections of US Code 10 may not be subject to recoupment. This is not a change to the program, but many officers may not be aware of this provision."