Vex means that he is making someone feel annoyed when something important is happening, I checked the definition in the dictionary and it states that Vex: make (someone) feel annoyed, frustrated, or worried, especially with trivial matters.
"the memory of the conversation still vexed him"