However, the planets still have high IPs. that doesn't make sense to me.
The IP generated by an alien planet is completely inependent of anything you do. It is based on factors like what they have built there, how high the population is, even how nice the planet is to live on.
What matters for culture flipping is getting the ratio of influence over 4. Once you pass that point, there is a chance the world will rebel and join you.
How do I go about influenncing that area to my color?
1) Increase your base influence ability with special projects, research, and mining influence resources.
2) Build influence starbases around the planets you want to influence (but this will annoy the AIs, so be careful).
3) Invade the highest influence planets in that area.