The reason why the Hermit Tribes were able to become powerful to the point where even the top sects wouldn't want to have an all-out war against them was because of the Secret Enclaves connected to their tribe.
Other than being able to help the cultivators awaken their bloodlines and special skills, these Secret Enclaves have many natural protective barriers stopping invaders from entering them.
In the initial stages, they were just normal cultivation families; however, since they were able to silently develop themselves and didn't reveal themselves to the world, they were eventually termed as Hermit Tribes.
With such Secret Enclaves and the rules of their ancestors about living quietly and developing steadily, these Hermit Tribes were able to peacefully develop themselves and gain a steady foothold in the province.
"It really worked… I was able to enter the Primal Beast Enclave on my own."
