First, note that for any differentiable functions [tex]g(x),G(x)[/tex], we have [tex]G(g(x))[/tex] attaining its relative extrema at the same points as [tex]g(x)[/tex]. This follows from the fact that
[tex]\bigg(G(g(x))\bigg)'=G'(g(x))g'(x)=0[/tex]
and so [tex]G(g(x))[/tex] has critical points at the same values of [tex]x[/tex] as does [tex]g(x)[/tex].
Now recall that the distance between any point in [tex]\mathbb R^3[/tex] and [tex](10,11,0)[/tex] is
[tex]d(x,y,z)=\sqrt{(x-10)^2+(y-11)^2+z^2}[/tex]
By the fact mentioned above, we know that the minimum distance between any point on the surface [tex]z^2=xy+1[/tex] will occur at the same point [tex](x,y,z)[/tex] that minimizes the squared distance; that is,
[tex]\hat d(x,y,z)=(x-10)^2+(y-11)^2+z^2[/tex]
We can then use the following function as the Lagrangian:
[tex]L(x,y,z,\lambda)=(x-10)^2+(y-11)^2+z^2+\lambda(z^2-xy-1)[/tex]
which has partial derivatives
[tex]\begin{cases}L_x=2(x-10)-\lambda y\\L_y=2(y-11)-\lambda x\\L_z=2z+2\lambda z\\L_\lambda=z^2-xy-1\end{cases}[/tex]
Set each partial derivative equal to 0. The third equation yields
[tex]2z+2\lambda z=2z(1+\lambda)=0\implies z=0\text{ or }\lambda=-1[/tex]
In the case of [tex]z=0[/tex], we have [tex]xy+1=0[/tex], or [tex]y=-\dfrac1x[/tex]. Note that it must be the case that [tex]x\neq0[/tex]. However, substituting this into the first two equations gives
[tex]\begin{cases}2(x-10)+\frac\lambda x=0\\2\left(-\frac1x-11)-\lambda x=0\end{cases}\implies\begin{cases}2x^2-20x+\lambda=0\\-\lambda x^2-22x-2=0\end{cases}[/tex]
[tex]\implies (2-\lambda)x^2-42x+\lambda-2=0[/tex]
from which it follows necessarily that [tex]x=0[/tex] and [tex]\lambda=2[/tex]. So we've arrived at a contradiction.
This means we must have [tex]\lambda=-1[/tex] From this, our first two equations become
[tex]\begin{cases}2(x-10)+y=0\\2(y-11)+x=0\end{cases}\implies\begin{cases}2x+y=20\\x+2y=22\end{cases}[/tex]
[tex]\implies x=6,y=8[/tex]
which in turn yield
[tex]z^2=48+1=49\implies z=\pm\sqrt{49}=\pm7[/tex]
Therefore we have two points on [tex]z^2=xy+1[/tex] which are closest to (10, 11, 0), namely [tex](6,8,\pm7)[/tex].
