There are many systems at work and the easiest thing to miss is the adjacency and moral bonuses. It's not very clear by the improvement type in the tool tip is tells you which bonuses will level the improvement. There are also growth improvements with other adjacency types (Colonization Center levels with All Construction). I'm my current game I'm playing the Korath and turn 31 I have 0.07 (1 pop every ~15 turns) pop even with a -15% growth penalty.
One of the first buildings you build is the colonization center. It's 0.01 natural growth plus 0.01 growth per "all construction" adjacency level. In my example, I have a level 3 but you can easily stack it to 9 or 10 and get 0.10 growth per turn so you'll need those cities. You will also get up to 50% growth bonus from high moral. Then in the tech tree there are two primary routes although you can take both. Colonization has a hospital line (percentile growth) and Governance has the Immigration line has flat growth options but moral penalty.
If you play around with it you'll find you can stack it quite well and then "shuttle" people off with a Colony ship to colonies that have worse growth.
