I'll assume your set up is: Wall(phone line) to DSL modem to Router to each computer.
Make sure that the Router is doling out internal IPs (and that the Linux box isn't trying to as well (read: turn off the DHCP service if the linux box is running one)) or if your using static internal IPs make sure the Linux Boxen isn't using a conflicting IP. Then tell linux how to get to the outside world....
Network settings for eth0 (or what ever ethernet card is hooked to the router) should be something like:
Active on start
(if your the only one using this computer) Allow all users to enable disable this device.
IF USING Dynamic intrenal IPs:
Automatically determine IP adress settings with (most
likely will be DHCP or bootp.
Automatically obtain DNS.
ELSE
Type in the static internal IP for the machine (making
sure it doesn't conflict with any other syetems
Then activate the device eth0. If it doesn't become active or if system hangs, play with it a bit (changing the method for getting IP). When in doubt RTFM.
If the router is already sending data to the windows machines, then it's fine... So keep playing on the Network Confinguration tool provided with your distro, or go play with the config files (if you don't have a tool (which is unlikely), you like doing things the hardway or you want to learn about the system a bit more). I'm babling... Hope that helps.