let's say, the amount that each side was increased was "a"
so, it was 5 wide and 12 long, now it's 5+a wide and 12+a long
we know the area is 120, so whatever "a" is, we know that
(5+a)(12+a) = 120
so [tex]\bf (5+a)(12+a) = 120\implies 60+17a+a^2=120
\\\\\\
a^2+\underline{17} a\underline{-60}=0\qquad \qquad
\begin{cases}
20\cdot -3\implies \underline{-60}\\\\
20-3\implies \underline{17}
\end{cases}[/tex]
use those factors, solve for "a"