Turn down the indicated IAS to minimum clean and use the VS you need. It will got to idle and you can control the rate of descend perfectly. As soon as you reached the constraint, you can fly your advised speed again.
In my experience the A320 is very pitch sensitive in open descend, the pitch attitude changes all the time. For passenger comfort VS is in my opinion the better choice.