If you modem is external it may be a cable problem. I saw this a few years ago on modems attached to Alloy network boards. One of the signal pins in the modem cable was causing it to happen, although I don't remember which it was. Some users fixed it by just breaking off the offending pin in the male end of the cable where it attached to the modem. Sorry I can't remember more about it. Best bet is to make sure you are using the correct serial cable.
If your modem is internal, I don't have a clue! Sorry! However, it sounds like the ring indicator signal is triggering the restart event.
Do you have FAX software that might be monitoring the modem and is trying to answer the phone? If that is the case, it is possible that corruption in that software is causing the system to crash/reboot when it activates.