First area of an astrolabe is very laggy

While playing with harvest astrolabes, I noticed that the first map I open after applying the astrolabe has huge lag when interacting with the monsters in it. You can see here below: the ms when not interacting with monsters is the first part of the graph; the second part was as soon as I activated the harvest here. It has never caused a disconnect, but it has caused some very extreme desync.


When in the mirage of that same map, it's regular stability for me. The ms there is always 6 to 8, as you can see below.


I'm really confused why this happens. All my settings between these two areas are identical, and are at almost the lowest settings the game can run. I use Vulkan and double buffering, my fps remains consistent for the duration. There is nobody else using the network in a strange way at that time, as that would also influence the mirage area I assume. I've run dozens of maps without lag but every first map of the fruiting astrolabe has this. I've not tested it with other astrolabes.

Anyone has an idea what could be going on here? How can I fix it?