Expected value is the sum, for all possible outcomes, the value of each outcome times its probability. In other words,
E[X]=sum x*P(X=x)
For this given situation there are 4 outcomes (out of 36) that rolls a sum of 5 [1,4; 2,3; 3,2; 4,1], so p(5)=4/36=1/9 and its value is $6.
There are 3 possible outcomes for a sum of 10 (again out of 36), so P(10)=3/36=1/12, with a value of $9.
All the other outcomes carry zero value, so the expected payoff is therefore
E[X]=sum x*P(x)
=$6*(1/9)+$9*(1/12)+$0(29/36)
=$(2/3+3/4)
=$17/12
=$1.42 (approx.)
So the expected payout is $1.42, not forgetting that it costs a certain amount to play the game.