I think that both of them are suffering because of what they did, and he believes that they committed a crime by not respecting or doing what was right. Hester was a married woman, yet she cheated on her husband with Dimmesdale, with whom she got a child. This was their crime - they lived in a Puritan society which condemned their actions as blasphemous in the eyes of God.
