Actually if you are not using NIS, then the line limit starting with AIX 4.3.3 and with AIX 5.1 is 2048 characters if you vi the file, but if you don't vi the file, then you can have more users per group on a line.
The easier way to solve the problem is to split the /etc/group file so that the lines can be edited. You can use split to do this. Then you can add a new group name, but use the same group ID. For example to add more people to the 'staff' group use a line
like
staff1:!:1:member5000,member5001,mebmer5002
If you list the user member5000, then it will show 'staff' as the group becaus staff is earlier in the file with GID of 1.
lsuser member5001
lsgroup staff or lsgroup staff1
will show all the users in the staff group from both lines. You can have multiple
lines and they do not need to be contiguous.