That smirk practically said, "So what if you don't agree? Do I need your permission? With your mother on my side, can you do anything she doesn't want you to do? No—you can only chicken out even if you're unhappy. That's the perk of having a strong wife, little boy."
Kallus, seeing that look and understanding exactly what his father's smirk was saying—albeit indirectly—couldn't help but feel frustrated, with a flicker of anger mixed in. But he was helpless in this situation. With his mother already in that mood, if he dared to respond to his father's provocation—a provocation his mother knew nothing about—it would only end with him being beaten by Amelia until his face turned red, black, and blue.
Otherwise, he would have shot back, "Why are you being so smug, old man? What perk? Just say it—she dominates you, and you can't do anything but be Mother's slave. It's not just me who chickens out! What about you? Don't you remember how many times you've embarrassed yourself?" Dammit…