So, if I understand correctly, this is a straight call coming in through PRI on a system, that is being routed via the E2T to the IP set directly off of that system. You didn't indicate, so I'm assuming this is a standalone single controller system.
If that is the case, then it sounds like it's simply a network issue. Regardless of the type of call (i.e. ACD), the audio path is the same once it is established. Sure, if you are doing RAC or silent monitoring, then the audio can get re-routed through a DSP, but the path from PRI end to the IP set is just passing-through the TDM bus.
What you need to do is enable the IP Phone Analyzer and see what it says when a call goes bad. If it complains about the network, then you need to look at it. If there are no complaints, then it's a more serious issue on the controller itself. Perhaps a bad echo canceller chip for example. You first need to narrow down where in the system the problem is occurring.