If you are looking for a purely technical solution, then you can purchase a "telephony firewall" like SecureLogix's TeleWall, or Sentry Telecom's PhoneWall. I personally feel that these are rather expensive solutions, but they are highly effective.
Also, I believe that ISS's Internet Scanner has a module for checking to see if your Windows PCs have a modem installed. Wouldn't get you to whether or not the modem is in use, but it would certainly narrow your search. You could potentially isolate this test, and write a script to check just for that.
If you can read a remote machine's registry, then it should be possible to detect. Unless they are using a Java app to communicate with the modem. It is possible to communicate with hardware controller-based modems in Java without having OS drivers installed.
But I have to agree with Bill, you are probably far better off using a written policy and spot checking. Set your call duration alarm (if your PBX supports them) relativley low, and check out any calls that alert (data calls are generally significantly longer than voice calls). You will have to make an example of someone, like forcing leave without pay to the first violator.
pansophic