The Lagrangian for this function and the given constraints is
[tex]L(x,y,z,\lambda_1,\lambda_2)=x^2+y^2+z^2+\lambda_1(x+2y+z-4)+\lambda_2(x-y-8)[/tex]
which has partial derivatives (set equal to 0) satisfying
[tex]\begin{cases}L_x=2x+\lambda_1+\lambda_2=0\\L_y=2y+2\lambda_1-\lambda_2=0\\L_z=2z+\lambda_1=0\\L_{\lambda_1}=x+2y+z-4=0\\L_{\lambda_2}=x-y-8=0\end{cases}[/tex]
This is a fairly standard linear system. Solving yields Lagrange multipliers of [tex]\lambda_1=-\dfrac{32}{11}[/tex] and [tex]\lambda_2=-\dfrac{104}{11}[/tex], and at the same time we find only one critical point at [tex](x,y,z)=\left(\dfrac{68}{11},-\dfrac{20}{11},\dfrac{16}{11}\right)[/tex].
Check the Hessian for [tex]f(x,y,z)[/tex], given by
[tex]\mathbf H(x,y,z)=\begin{bmatrix}f_{xx}&f_{xy}&f_{xz}\\f_{yx}&f_{yy}&f_{yz}\\f_{zx}&f_{zy}&f_{zz}\end{bmatrix}=\begin{bmatrix}=\begin{bmatrix}2&0&0\\0&2&0\\0&0&2\end{bmatrix}[/tex]
[tex]\mathbf H[/tex] is positive definite, since [tex]\mathbf v^\top\mathbf{Hv}>0[/tex] for any vector [tex]\mathbf v=\begin{bmatrix}x&y&z\end{bmatrix}^\top[/tex], which means [tex]f(x,y,z)=x^2+y^2+z^2[/tex] attains a minimum value of [tex]\dfrac{480}{11}[/tex] at [tex]\left(\dfrac{68}{11},-\dfrac{20}{11},\dfrac{16}{11}\right)[/tex]. There is no maximum over the given constraints.
