We just had this same exact thing happen to us 2 weeks ago and finished working thru it with IBM. It took them quite a while to diagnose the problem. At first, the hardware CE started changing IO boards and service processor boards thinking one was fried.
But, everytime we would try booting up with our 10/100 ethernet card installed, we would get a '0777' LED and the system would stop. We knew the card was good.
Here is what the problem was. We had 4 entries in our ODM definitions that had null value for the 'NAME=' field. We used the odmupdate command to delete this specific lines from the ODM. It turns out that the system was hanging when it was trying to configure the EN0 device while it was running cfgmgr on bootup.
I'll be that just prior to this problem starting, you either migrated up to 5.1 or installed a maintenance level or something similar to this didn't you.
IBM now has a knowledgebase case on this problem on the software support side. The specialist who helped us was 'Ron' and he was a TCP/IP specialist with IBM in Austin.