Its possible to apply an ACL to the tftp-server command to specify the which host's can access the server on the router:
tftp-server flash: 1350
This is from a router running 12.2(13)T IOS, it specifies using the extended ACL 1350. You can also use a normal statndard ACL (1-99) as well. I dont know when this command was introduced, use the router help to see if its available in your IOS version.