what platform (Solaris, AIX, HP-UX)?
I have seen similar errors installing 32-bit 8.1.7 on Linux which were caused by glibc and/or gcc version incompatibilities but I'm assuming you're commercial if you're running 64.
my suggestion would be to take the Oracle Install tech manual for your particular platform and go through line by line to verify packages, patch levels, etc.
finally, much as I'd rather have my appendix out again than deal w/OSS, you may need to open a TAR for this...