The legal theory under which states were able to reject federal laws deemed unconstitutional was known as nullification.
Nullification, in United States constitutional history, refers to a legal theory according to which the right to nullify, or invalidate, any federal law which that state has deemed unconstitutional under the United States Constitution (in opposition to the state's own constitution). The theory of nullification has never been legally upheld by federal courts.
