The "fire governor" mechanic is turning out to be a major misstep as currently devised. The 10% empire-wide penalty has to go.
IDEA: when firing the governor, there is a 5-10 turn cooldown before the governor can be reassigned. No other penalties other than to the leader's own loyalty.