I think you'll get the diplomatic penalty regardless of this, the only thing that seems to matter for the AI is the proximity of them to him. Although I cannot rule out other factors weighing in, such as the diplomatic trust. I remember getting the hit (and the AI informing me to remove my troop ships) even when these were located in my own ZOC, actually in orbit of my own planets to be more precise.
It's kinda better to not cross AI territory with troopships, and/or try to expand only by conquering your next neighbours (as such your own space is also more easily to defend)
