The mksysb is really your best line of defense against a serious problem. no matter what happens with your upgrade, you'd always have that image to put you right back into a stable state.. providing you have a method to install it back into your system, such as mentioned above.
i've done a half dozen or so migrations to 5.3 and so far i've seen no issues, everything ran real smooth. a few things i would check for tho, and these usually happen after any aix upgrade:
1 - your sendmail.cf will get overwritten, note your customizations beforehand and alter the new file to fit your needs.
2 - some services in inetd.conf will be turned back on. i think ntalk is one of them. so if you're in the habit of shutting off unwanted services, check this file after you upgrade. think this might also happen after a ML upgrade too.
3 - /etc/motd is copied to /tmp/bos/etc and replaced with a generic 'welcome to aix 5.*' motd. so, you may want to copy back your old one if you use the motd.
4 - if you use any of the toolbox rpm's that have been ported over from linux, check to see if there are any versions of them specific to aix 5.3 and upgrade those as well after you get to 5.3
5 - verify that any tunables (network & i/o) that you set for 5.2 got carried over to 5.3. i'm not sure if these get wiped out, but it wouldn't hurt to check.
hmm, i guess those are the 'major' things i would look out for...