When you execute a traceroute command (ie traceroute www.yahoo.com), your machine sends out 3 UDP packets with a TTL (Time-to-Live) of 1. When those packets reach the next hop router, it will decrease the TTL to 0 and thus reject the packet. It will send an ICMP Time-to-Live Exceeded (Type 11), TTL equal 0 during transit (Code 0) back to your machine - with a source address of itself, therefore you now know the address of the first router in the path.
Next your machine will send 3 UDP packets with a TTL of 2, thus the first router that you already know passes the packets on to the next router after reducing the TTL by 1 to 1. The next router decreases the TTL to 0, thus rejecting the packet and sending the same ICMP Time-to-Live Exceeded with its address as the source back to your machine. Thus you now know the first 2 routers in the path.
This keeps going until you reach the destination. Since you are sending UDP packets with the destination address of the host you are concerned with, once it gets to the destination the UDP packet is wanting to connect to the port that you have sent as the destination port, since it is an uncommon port, it will most like be rejected with an ICMP Destination Unreachable (Type 3), Port Unreachable (Code 3). This ICMP message is sent back to your machine, which will understand this as being the last hop, therefore traceroute will exit, giving you the hops between you and the destination.
The UDP packet is sent on a high port, destined to another high port. On a Linux box, these ports were not the same, although usually in the 33000. The source port stayed the same throughout the session, however the destination port was increase by one for each packet sent out.
One note, traceroute actually sends 1 UDP packet of TTL, waits for the return ICMP message, sends the second UDP packet, waits, sends the third, waits, etc, etc, etc.
If during the session, you receive * * *, this could mean that that router in the path does not return ICMP messages, it returns messages with a TTL too small to reach your machine or a router with buggy software. After a * * * within the path, traceroute will still increment the TTL by 1, thus still continuing on in the path determination.
I really have no idea any localized place to get information about traceroute (although if you are on *nix, try the man page - it didn't look too bad in the brief look I had at it...).