Banishing a being to the spirit realm didn't just mean that they would be thrown inside and forgotten.
It meant that they would be made a part of the spirit realm itself!
King Allen forced them all to sign a life and death contract with the very 'laws' of the 'spirit realm' itself!
As long as those 'laws' of the 'spirit realm' bind them, they can't physically get out of the spirit realm; at most, they can only be summoned in the form of spirit beasts like other spirit beasts.
This was also one of the main reasons why the shadow dragon king was waiting for King Allen to come back and free him and his kin from this cursed contract of eternal suffering.
Without having that cursed contract broken, the 'laws' of the 'spirit realm' were never going to let them escape out of it.
The same rule applies to Elisia Mistilteinn's soul fragments; they belong to the spirit realm and can't be taken out with normal means.
