Sounds like an application (which one?) issue rather than OS, although some kernel tunables may need adjusting if you need to run that many processes under normal operation. I had a glance at a nearby Solaris 10 box and v.v_proc (maximum number of processes) is set by default to 3946, which sounds quite similar to your number, maybe that's the one that needs to be adjusted?
Annihilannic.