I agree about the memory leak being a good possibility, but since you described the primary perfomance problem being with accessing a file on the server and not with the performance of the application on the box, I also suspect Disk and network performance. You have gone down the correct path with a defrag, but it is good to check the disk IO performance with perfmon, (you will need to enable diskperf if it is not already done. Find instructions by typing "diskperf" at the command line.) you will need to look at a counter under physical (or logical) disk called current disk queue length and average disk queue length. If this counter is higher that 2 or 3 you have a bottleneck with you disk IO subsystem. If the server has an array controller you may be able to tune the caching to add more read cache to help, or you may need to change hardware. As a note if your server is also running SQL be careful with array cache settings as write cache being enabled (the default) can corrupt your database.
If your disk IO looks OK, check the network port you are connected to. If it is a switch port (hopefully) you are best swerved if you can hard set to the highest possible common setting (hopefully 100 meg full duplex) (careful when making a change here as it may temporarily disconnect your server. If your nic and port are at autosense (the default) you can check the switch port for error rates and identify the problem before you start making changes.
As a final note, I assumed that before you started to dig into the server as a problem you tried to access the file from multiple other locations in the network to make sure it is not an issue with the workstation.
I hope this helps,
Jay Mosser
jaym@optymgroup.com