In Chapter 5 of "Soldier's Heart", Charley witnessed from his first battle that the Congressmen and their families were on the hill. They had drove from Washington to picnic and watch the battle from a safe distance. They most likely returned to Washingston after the battle.