[tex]\mathbf F(x,y,z)=z\tan^{-1}(y^2)\,\mathbf i+z^3\ln(x^2+3)\,\mathbf j+z\,\mathbf k[/tex]
[tex]\implies\mathrm{div}\mathbf F(x,y,z)=0+0+1=1[/tex]
By the divergence theorem, the flux of [tex]\mathbf F[/tex] across the *closed* surface [tex]\mathcal S[/tex] combined with the plane [tex]z=2[/tex] is given by a volume integral over the closed region:
[tex]\displaystyle\iint_{\mathcal S}\mathbf F\cdot\mathrm d\mathbf S=\iiint_{\mathcal R}\nabla\cdot\mathbf F\,\mathrm dV[/tex]
So in fact, to find the flux over [tex]\mathcal S[/tex] alone, we'll need to subtract the flux of [tex]\mathbf F[/tex] over the planar portion, oriented outward. First, compute the volume integral by converting to cylindrical coordinates:
[tex]x^2+y^2+z=18[/tex]
[tex]z=2\implies x^2+y^2=16\implies r^2=16\implies r=4[/tex]
[tex]\displaystyle\iiint_{\mathcal R}\mathrm dV=\int_{\theta=0}^{\theta=2\pi}\int_{r=0}^{r=4}\int_{z=2}^{z=18-r^2}r\,\mathrm dz\,\mathrm dr\,\mathrm d\theta=128\pi[/tex]
If the surface does actually contain [tex]z=2[/tex], then you can stop here; otherwise, continue.
Now, parameterize the part of the *closed* surface in [tex]z=2[/tex] by
[tex]\mathbf s(r,\theta)=r\cos\theta\,\mathbf i+r\sin\theta\,\mathbf j+2\,\mathbf k[/tex]
where [tex]0\le r\le4[/tex] and [tex]0\le\theta\le2\pi[/tex]. We get a surface element
[tex]\mathrm d\mathbf S=(\mathbf s_r\times\mathbf s_\theta)\,\mathrm dr\,\mathrm d\theta=(r\,\mathbf k)\,\mathrm dr\,\mathrm d\theta[/tex]
We don't need to worry about the first two components of
and so the surface integral over this region is
[tex]\displaystyle\iint_{z=2\,\land\,x^2+y^2\le16}\mathbf F\cdot\mathrm d\mathbf S=\int_{\theta=0}^{\theta=2\pi}\int_{r=0}^{r=4}2r\,\mathrm dr\,\mathrm d\theta=32\pi[/tex]
Then the total flux over [tex]\mathcal S[/tex] alone is [tex](128-32)\pi=96\pi[/tex].
