@ GCFL: Erm, dunno?
I just made up some numbers. I think, particularly in the early game, the 75/50/25 splits would be too fine for the coarse grain of the damage vs hitpoints calculations anyway. Might be simplest just to have:
1. Fight to the death
2. Fight to half-damage
3. Retreat at first damage
With 1 being default for any ship or fleet with 1 point or more of attack strength, and an invisible 4th option ('Run like the clappers!') being the default for entirely unarmed units.
Half-damage on a fleet would simpyly be the lesser of half the total hitpoints or half the number of ships, I suppose.
Though lately, I've been fond of BattleStar Groups - one heavily armoured biggie and a support of as many sharp-toothed fighters as my Logistics will allow. Fighters get parallel built on junk planets and are cheaply replacable, but provide an awesome first-turn throw-weight, whereas biggies get lovingly crafted at my Manufacturing Capital and hope to survive every fight. In the fleet orders system, I might find them retreating to soon on Retreat at half-damage - I sometimes expect the entire Viper wing to get wiped out, only for the Battlestar to finish off rhe enemy survivors. I guess that Fight to the death would still be my option of choice, there.
Perhaps the retreat chance should also take account of the enemy Fleet order - if they're both on Retreat at first damage, then the chance of retreat is higher than if the enemy is on Fight to the death. That would be invisible, though.
As to applying it, I think it would be over-complicated to split a fleet's 'roll' across its individual ships. Either it makes the roll, and the remaining ships escape, or it fails, and they all die. Either way, however, making the roll forfeits all remaining attacks, so the enemy takes no more damage.
A potential nasty is what happens if the enemy fleet is the attacker and still has movement? They can then just attack again. Should a fleet default to order 4 during any enemy move in which they have already taken order 2 or 3? Would we then find ourselves slamming into AI constructors multiple times in order to kill them?
Taking account of the remaining moves helps here - a ship which has already moved its full allowance would have (next to?) zero chance to retreat. But GCII ships move at turn END - that could throw a spanner in the works.
The idea needs a bit more input from better brains than mine...