The word most similar to dominion in this context is: influence.
The narrator is here expressing how anxiety and horror have taken possession over him, and though he would like to find rest in sleep, he cannot fall asleep, and as he tries to rationalize the situation in which he finds himself, he cannot find peace, and so nervousness has a hold of him, it has dominion over him, or in other words, nervousness´ influence over him is too strong to be able to let go of it.
