Ronin may roam the world, but their activities are actually limited to nearby areas, often just wandering around the vicinity, so it's not uncommon for them to know each other.
Knowing each other leads to banding together, and the weak, once they band together, mistakenly believe they've become strong, leading to greed that doesn't match their actual capabilities.
Anyone with a bit of insight knows clearly that all the assets of everyone in the longhouse combined might not even be as valuable as the sword on Lann's back.
Ronin fight and kill for money; they are not warriors who can be loyal to others, speaking of loyalty and trust.
Thus, their group's resentment of being neglected by the main family, coupled with their greed for the sword and wealth... the viciousness in the eyes of the ronin gathered by Pingbala Lang was clear even to Kotaro.