The pastor is one of the most isolated characters even though he is still welcomed by the community. Due to his heavy heart from his affair with Hester he isolated himself from society to punish himself for the hidden sin. The minister was also frustrated with the behavior of his “flock” this isolating himself further because of his frustration with them.
If you’re going for obvious then Hester would be the most isolated character and her daughter.
A) her affair with an “unknown” male resulted in an unsanctioned marriage. They thought she had fornicated with the devil perhaps since she did not want to come clean about what human male was the father of her child. Branding her a possible witch.
B) she wore a scarlet letter on her breast to show her cardinal sin and warn others away. This placing an even further barrier between herself and others.
C) her child was a near devil incarnate which frightened everyone, her mother included.
Hope this helps:)