I would run PerfMon on that server and look at the .NET CLR Memory Gen'x' counters to see how your well your garbage collection is working.
I suspect you aren't disposing of some of your unmanaged resources (database handles, file handles, etc), so when a user connects the system is unable to provide a connection for them. If there's handles available, the site works; if there aren't, they get the "service unavailable" message.
Chip H.