Despite the obvious hint, the women didn't back down and, on the contrary, were convinced that Andrew was just putting on a show, as he didn't want his wives to find out what had happened.
Although in this world it's not unusual for powerful men to have multiple wives, it's true that infidelity isn't allowed. It's important to understand that a harem and being faithful are two things that strangely coexist in the common sense of this world.
In short, relationships with multiple women are normal for a man, as long as they are acknowledged. That is, for Andrew, having multiple wives is accepted. However, having romantic or sexual affairs with other women would be considered infidelity.
