After removing the square corners, you're left with a cross shape that folds into a cuboid with base dimensions [tex]12-x[/tex] by [tex]12-x[/tex], and height [tex]x[/tex]. The volume of this box is [tex]x(12-x)^2[/tex].
Writing this as a function, we can take the derivative to find the critical points and determine the value of [tex]x[/tex] that maximizes the volume:
[tex]V(x)=x(12-x)^2\implies V'(x)=(12-x)^2+2x(12-x)(-1)=
3 x^2 - 48 x + 144=3(12-x)(4-x)[/tex]
[tex]V'(x)=0[/tex] when [tex]x=12[/tex] or [tex]x=4[/tex]. But [tex]x=12[/tex], the cut we try to make isn't a cut; the piece of cardboard stays flat and unchanged. So the volume is maximized when [tex]x=4[/tex] (you can verify that the derivative behaves accordingly; that is, [tex]V'(x)>0[/tex] for [tex]x<4[/tex] and [tex]V'(x)<0[/tex] for [tex]x>4[/tex]).
The largest volume of the box is then [tex]V(4)=256[/tex] cubic inches.