h^2=x^2+y^2 and xy=168 so y=168/x
625=x^2+28224/x^2
625=(x^4+28224)/x^2
x^4-625x^2+28224=0
I'll use the quadratic because this is a big number :P
x^2=(625±√(277729))/2
x^2=(625±527)/2
x^2=49
x=7
168/7=24
So the dimensions of the painting is 7 inches by 24 inches.