Are you me? I came to the forum to make the exact same thread asking the exact same question.
I governor-ed two low-priority core worlds in the same turn, after having built orbital Monsatium and orbital Thulium extractors on each respectively. And yeah, it sucks to realise your adjacency bonuses AND mining yields are now hampered by a not-great, un-upgradable tile improvement.
Also, I loaded back to a previous save, just to test whether I could go and delete the orbital improvement BEFORE turning the planet into a core world, just to see if that might be a workaround to the problem - but it isn't, it still deletes the underlying resource when you do that