Are you sending the mksysb to tape or a filesystem? If you are sending it to a filesystem that is a plain JFS filesystem (that is not large file enabled), it sounds as if you are running into the mksysb file being more than 2 GB. Either rebuild your filesystem as a JFS2 filesystem or rebuild it as a JFS filesystem that is large file enabled.
You can tell if the filesystem is large file enabled by doing an lsfs -q /filesystem. If you see bf: false, you have a plain JFS filesystem that is not large file enabled.