think about it this way, when u deny or accept any kind of traffic using an ordinary filtering firewall, u will specify which IP/Network to deny or allow.. same principal, which is that someone, not everyone, should be allowed access.. everyone else is just a cracker, and shouldn't be allowed..
however, to answer your question, u can deny telnet access on the VTY terminals without using an access list:
conf t
line vty 0 4
transport input none
this basically disbales all protocols from being accepted on the VTY terminal, including Telnet.. so, not even u can get in...
good luck..