Since they are Demon Cultivators, their reputation is very poor, making them easy targets for attacks.
In comparison, even though Ning Zhuo lacks reputation, at least he isn't as completely negative as Ban Ji.
So, in the first Xingyun Trial of Master Qinghuangzi, he took advantage of this weakness and successfully united a large group of cultivators.
Ban Ji's own arrogant attack posture also helped Ning Zhuo from another perspective.
After all, most people knew of Ban Ji and had a sense of dread towards him.
As for whether it was unfair to attack as a group?
Of course not—what fairness is there in dealing with Demon Cultivators? Everyone just stands together!
In their previous conversation, Ning Zhuo once again solidified this fact to the crowd—Ban Ji is a Demon Cultivator, while I, Ning Zhuo, belong to the Righteous Path. We are all good people, and collectively opposing Ban Ji is a righteous act!