[tex](1+ax)(1+bx)^4=(1+ax)\displaystyle\sum_{k=0}^4\binom4k(bx)^k[/tex]
When [tex]k=0[/tex], the sum contributes [tex]\binom40(bx)^0=1[/tex];
when [tex]k=1[/tex], it contributes [tex]\binom41(bx)^1=4bx[/tex];
when [tex]k=2[/tex], it contributes [tex]\binom42(bx)^2=6b^2x^2[/tex].
In the complete expansion, the [tex]x[/tex] term is then [tex]ax\cdot1+1\cdot4bx=(a+4b)x=0x[/tex], and the [tex]x^2[/tex] term is [tex]ax\cdot4bx+1\cdot6b^2x^2=(4ab+6b^2)x^2=-40x[/tex].
So we have
[tex]\begin{cases}a+4b=0\\4ab+6b^2=-40\end{cases}\implies a=\pm8,b=\mp2[/tex]
