Note that √((1 - x)/x) is defined only as long as 0 < x ≤ 1.
Consider a right triangle with reference angle θ such that
[tex]\cot(\theta) = \sqrt{\dfrac{1-x}x}[/tex]
In other words, on an appropriate domain,
[tex]\theta = \cot^{-1}\left(\sqrt{\dfrac{1-x}x}\right)[/tex]
In such a triangle, you would find that
[tex]\sin(\theta) = \sqrt x[/tex]
so f(x) reduces a bit to
[tex]f(x) = \cos\left(2 \tan^{-1}(\sqrt x)\right)[/tex]
Now consider another triangle with reference angle ɸ such that
[tex]\tan(\phi) = \sqrt x \implies \phi = \tan^{-1}(\sqrt x)[/tex]
In this triangle, you would find
[tex]\cos(\phi) = \dfrac1{\sqrt{1+x}}[/tex]
Recalling the double angle identity for cosine, it follows that
[tex]f(x) = \cos(2\phi) = 2 \cos^2(\phi) - 1 = \dfrac{1-x}{1+x}[/tex]
Differentiating with respect to x yields
[tex]f'(x) = -\dfrac2{(1+x)^2}[/tex]
while
[tex]f(x)^2 = \dfrac{(1-x)^2}{(1+x)^2}[/tex]
It follows that
[tex]f'(x)(x-1)^2 - 2f(x)^2 = 0[/tex]
so B is the correct choice.