It is termed as "Sect".
Though in the beginning the term “sect” was actually used to classify a religious groups which was separate but now it can be used for any group of people or organization which separates themselves form a larger orthodox religious belief, for example in Islam these days besides the generally established groups a contemporary sect has come into light called “Ahmadiyya Muslim Community” which seeks to change the perspective of Islam, this is called a “sect”.