If you didn't install using RPM, then yes you need to remove it by hand. First off, if it is being started at bootup, then you need to remove the startup script. Usually this is something like /etc/rc.d/init.d/postfix. This script however, is called from another script depending on the runlevel you boot into. For example, if you boot into server mode (runlevel 3) then you miy find a script something like /etc/rc.d/rc3.d/s80postfix. The "s" in front of it means that it is a start script and the "80" is the priority or order that it will start. Some scripts that requir to be shutdown a certain way will also have a kill script and they will begin with the letter "k". Once you've remove the start script(s), you can type "whereis postfix" at the command line and it will tell you where most of its files are.