Don't know about Fedora 4 but on the older versions, it used to ask you whether you wanted a graphical login. Just say no and you will get a non graphical login, in which case X does not startup.
Lots of ways of starting X. I normally use startx. The script is somewhere in /etc. I'm working on a very old (1995) version - they may have moved it somewhere else.