imo: Definitely don't bring the baby. It's a fancy dinner with lots of quiet periods where someone is speaking to the room.
As a side note, I really enjoyed it and it was a special bonus to have the wife fly in too. After all the years I spent trying to get on at Delta, that dinner really was the icing on the top of training week.
Find, beg, borrow, or steal a babysitter and go! It only happens once!