I ran into this a while back when attempting to upgrade the GCC 3.4 port. Apparently my FreeBSD server had crashed or dumped core during the previous install, and attempting to upgrade (using portupgrade) threw a "directory not empty" error on a directory that contained no files (/usr/ports/lang/gcc34/work/gcc-3.4-20041112/libstdc++-v3/config/cpu).
I rebooted single-user and ran 'fsck -p', which showed softupdate and other inconsistencies in the file system, but did not correct them. The problem here was that the /usr/lost+found directory was full.
I continued to have this problem for a few months without finding a soultion, so I was unable to do the portupgrade.
I deleted all files and subdirectories in /usr/lost+found, rebooted single user, and ran 'fsck -p'. The filesystem errors were corrected.
I rebooted multi-user and I was able to remove the directory I was having issues with. I was then able to upgrade the GCC 3.4 port (with portupgrade) and have been running happily since.