Have you tried downloading and installing Internet Explorer 8 outside of Windows Update to see if the installation will complete?  Below is a link to the download.

- Windows Internet Explorer 8 for Windows XP

http://www.microsoft.com/downloadS/details.aspx?familyid=341C2AD5-8C3D-4347-8C03-08CDECD8852B&displaylang=en
 

Joey
CCNA, MCSA 2003, MCP, A+, Network+, Wireless#

Have you tried the basic troubleshooting step of temporarily disabling antivirus software and other realtime scanning software that you are running?

You could even use MsConfig to stop all third party programs and services from starting.

310560 - How to Troubleshoot By Using the Msconfig Utility in Windows XP
http://support.microsoft.com/kb/310560

Are you giving it enough time to complete the installation process, it takes quite a while if I remember, as it checks for updates and downloads any available, as well as malicious software scanning?

This has happened everytime I do any windows updates,all updates go ok till it gets to the malicious software update and it starts ok and then just hangs a few min.later.with or without virus software running.If I run the malicious software from a download it runs ok? and no malicious software found on the scan,seems like it's only through windows update it does this.shouldn't have to wait more that 30min.to install explorer 8,when it hangs on checking computer for malicious software? Weird.   

What happens if you delete mrt.exe from the System32 folder (there is no uninstall), and when offered downloads of it by Windows Update, you refuse to accept them and tell them not to offer them any more and hide the download notification?  Alternatively after deletion you could download mrt.exe again and see if it behaves better.

Microsoft® Windows® Malicious Software Removal Tool (KB890830)
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=ad724ae0-e72d-4f54-9ab3-75b8eb148356

Have you looked at the mrt logs?

The Malicious Software Removal Tool writes details about the result of its execution in the %windir%\debug\mrt.log log file.

Read on from the paragraph "How to parse the log file" in this article.

Deployment of the Microsoft Windows Malicious Software Removal Tool in an enterprise environment
http://support.microsoft.com/?kbid=891716

Have you tried installing IE8 and not checking the box that request to check for updates?  I am still of two minds whether 30 minutes is long enough to get this installed, perhaps you should wait an hour to see if it goes in.  Yes, you should NOT have to wait so long for an install and update, but from a troubleshooting point of view it is something that should be checked, and eliminated as a cause.

As you probably had several goes at getting IE8 in it may pay you to clean out your temp files manually and also run Disk Cleanup and ChkDsk.  

Try running ChkDsk to check your drive for errors. Right-click your Drive icon/ Properties/ Tools/ Error Checking.  Select both boxes.

Run the System File Checker program from the Run Box by typing.....Sfc /Scannow in it and have your XP CD handy.

Besides the mrt.log, have a look at the Windows Update log too and see if you can pickup any clues from that.

How to read the Windowsupdate.log file
http://support.microsoft.com/kb/902093#appliesto