Questions for you: What is the real name of this file? Does ls -bl show a different name? I can delete ~! from within sh as well as ksh.
For troublesome files, I create a shell script from ls -b. In this case, knowing that the filename is only two characters long, ls -b ?? > rm.sh. I then edit rm.sh by insertinf rm -i in front of each file I want to remove and removing all others. Then, sh ./rm.sh.