In uncle tom's cabin, Harriet Beecher Stowe took on the pro-slavery position by contending that good intentions of some owners cannot make up for an evil institution.
Even though there were some good slave owners who didn't want to completely oppress their slaves, they were ultimately slave owners - they owned people, made them work for them, and thought of them as tools. Slavery shouldn't have existed in the first place, and whether or not slave owners are good doesn't justify its existence.