[tex]5ab^{3} +7-3a^{2} b ^{2} +a^{3}b+10[/tex] can be simplified to by adding the 7 and 10 to get [tex]a^{3}b-3a^{2} b ^{2}+ 5ab^{3} + 17[/tex].
[tex]5ab^{3}+3a^{2}b^{2}+a^{3}b -10[/tex] cannot be simplified any more by combining like terms.
By distributing the 2b into the parentheses, you can simplify the expression:
[tex]5ab^{3}+2b(3ab^{2})+a^{3}b-10\\
5ab^{3}+6ab^{3}+a^{3}b-10\\
11ab^{3}+a^{3}b-10[/tex]
Here you can just add:
[tex]5a^{3}b^{3}+3a^{2}b^{2}+a^{3}b^{3}-10ab\\
6a^{3}b^{3}+3a^{2}b^{2}-10ab[/tex]
Thus, the only expression that cannot simplify any more using adding like terms is the second, [tex]5ab^{3}+3a^{2}b^{2}+a^{3}b -10[/tex].
