You can hand fly the baby bus just fine. It's more like CWS, make minor corrections in the direction of the runway/FD's and let go of the stick for awhile. Then make new adjustment and repeat the process. I was finger drumming the glareshield while 'hand flying' last trip. It just keeps going in the direction, and pitch attitude, that you had when you stopping making inputs.
Turning off the auto thrust, IMO, works better when the winds aren't stable. Otherwise the thrust goes crazy, more, less, chasing speed changes.