question on raise spectre gem

Hi,

First time playing a spectre build, and I had trouble summoning my frostbearer, turtle, forest warrior between the 2 gems setup, the spell echo, the corpse to buy, etc.

I am now about to buy 21 level gem, and wondering how it will react when I replace the level 20: do I need to be in town and just switch them, or do I need to be in hideout and go to the troublesome summoning again? anything else to know so I don't lose the corpse I bought

Thx