Ok, today, I discovered I am still an http://en.wikipedia.org/wiki/Idiot.
Yep, I tried to add 2 public networks to one of my CloudSigma servers and one of them didn't work.
I thought everything was to blame but my configuration (as always). Well, I managed to discover what the problem was and how to correct it.
The problem is that since there is only one default route, packets going through eth1 didn't know how to go back to where they came from. This is solved by adding a rule and telling the kernel where to look for info on those packets:
How
# first my NIC configuration
# /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=dhcp
DEFROUTE=yes
## cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=dhcp
DEFROUTE=no
# my routing table
## ip route
111.111.111.0/24 dev eth0 proto kernel scope link src 111.111.111.111
222.222.222.0/23 dev eth1 proto kernel scope link src 222.222.222.222
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
default via 111.111.111.1 dev eth0
# look for info on packets comming from network 222.222.222.0/23 on table 1
ip rule add from 222.222.222.0/23 tab 1 priority 500
# append to default gateway telling it to look for info on table 1
ip route add default via 222.222.222.1 dev eth1 tab 1
# flush cache
ip route flush cache
Rationalization
So, eth0 (111.111.111.111) is the default route. It is declared in ifcfg-eth0. If I do not declare DEFROUTE=no on eth1, then, the last NIC to become available becomes the default route. So, I specify which is the default so I can add rules later.
Then, there is eth1 (222.222.222.222) which is a completely different network. We add the rules needed for the info of it to be found on it's own table and we add it to the default.
This works ipso facto. I don't know if it will survive a reboot, but, hey, I know my readers will tell me if it does or not.