I believe that IPSec in AIX is IP security, which filters traffic based on rules you define.Think it can be used to set up AIX as a firewall.-> smitty tcpip -> configure IP security.Just have a look around,perhaps it will offer you the solution you're looking for.
rgds,
R.