HELP! DNS issues causing domain wide headaches!

[aziel11]

I am relativly new to the IT world, about two years ago I was thrust into the position of running a computer network of about 200 computers, 4 servers and 500 users. The network was grown up over time and everything was a mess when I took over. Everything seemed to run fine untill about four months ago when something happened that caused many of the computers to loose their connection to the domain controller. We were running all windows server 2000 with one gateway server handling DNS, DHCP, WINS. One server running exchange and the primary DC. One server doing file sharing and running some server/client software and a backup DC.
I added another server as a DC with a clean install of 2003. Since then I have taken the gateway server offline and replaced it with a clean install of 2003. The current setwup is this:
One server (2003) running DNS, DCHP, WINS, the internet gateway, VPN, and its a DC. One 2003 server just doing backup DC. One server (2000) doing file sharing and software. One server (2000) doing Exchange 2000 and DC as well as secondary DNS. Right now I can't get a VPN connection to the gateway server, it says the username password is invaild on the domain. If I terminal server into that server and restart Routeing and Remote Access service then the VPN will connect, but I have to restart that service everytime I want to VPN in. Also occasionally I can't terminal server into that server, I get a RPC server is unavailible error. If I physically log in at the console and restart the RPC locator service then I can log in remotely again. The exchange server is off site and there is a VPN link through the routeing and remote access controll panel, that link is allways functional and seems unaffected by the above mentioned problems.
Also all the windows NT machines are unable to loginto the domain and must either use local accounts or cached accounts to log in. Every computer on the local network can ping every other computer includeing the DC's its just that some of them don't find the DC on login.

Does anyone have any idea of where I should begin fixing this?
Also I have talked to our finance manager about getting some money to buy three new servers to replace the four two-year-old servers we have now. Has anyone tried replacing all of the servers on a domain? If so what is the easiest way to do it? Please help me?

----Update-----
here is the output from dcdiag on our gateway server (BICH):

Domain Controller Diagnosis

Performing initial setup:
Done gathering initial info.

Doing initial required tests

Testing server: Default-First-Site-Name\SHIVA
Starting test: Connectivity
......................... SHIVA passed test Connectivity

Testing server: Default-First-Site-Name\DOC
Starting test: Connectivity
......................... DOC passed test Connectivity

Testing server: Default-First-Site-Name\BICH
Starting test: Connectivity
......................... BICH passed test Connectivity

Doing primary tests

Testing server: Default-First-Site-Name\SHIVA
Starting test: Replications
[Replications Check,SHIVA] A recent replication attempt failed:
From DOC to SHIVA
Naming Context: CN=Schema,CN=Configuration,DC=carehawaii,DC=com
The replication generated an error (8524):
The DSA operation is unable to proceed because of a DNS lookup failure.
The failure occurred at 2005-04-13 11:56:01.
The last success occurred at 2005-03-25 10:56:40.
77 failures have occurred since the last success.
The guid-based DNS name 954f8a22-7e03-4782-ae49-28748fbadc43._msdcs.carehawaii.com
is not registered on one or more DNS servers.
[Replications Check,SHIVA] A recent replication attempt failed:
From DOC to SHIVA
Naming Context: CN=Configuration,DC=carehawaii,DC=com
The replication generated an error (8524):
The DSA operation is unable to proceed because of a DNS lookup failure.
The failure occurred at 2005-04-13 12:43:57.
The last success occurred at 2005-03-25 10:56:40.
675 failures have occurred since the last success.
The guid-based DNS name 954f8a22-7e03-4782-ae49-28748fbadc43._msdcs.carehawaii.com
is not registered on one or more DNS servers.
[Replications Check,SHIVA] A recent replication attempt failed:
From DOC to SHIVA
Naming Context: DC=carehawaii,DC=com
The replication generated an error (8524):
The DSA operation is unable to proceed because of a DNS lookup failure.
The failure occurred at 2005-04-13 12:47:15.
The last success occurred at 2005-03-25 11:07:39.
1620 failures have occurred since the last success.
The guid-based DNS name 954f8a22-7e03-4782-ae49-28748fbadc43._msdcs.carehawaii.com
is not registered on one or more DNS servers.
......................... SHIVA passed test Replications
Starting test: NCSecDesc
......................... SHIVA passed test NCSecDesc
Starting test: NetLogons
......................... SHIVA passed test NetLogons
Starting test: Advertising
Fatal ErrorsGetDcName (SHIVA) call failed, error 1355
The Locator could not find the server.
......................... SHIVA failed test Advertising
Starting test: KnowsOfRoleHolders
......................... SHIVA passed test KnowsOfRoleHolders
Starting test: RidManager
......................... SHIVA passed test RidManager
Starting test: MachineAccount
......................... SHIVA passed test MachineAccount
Starting test: Services
......................... SHIVA passed test Services
Starting test: ObjectsReplicated
......................... SHIVA passed test ObjectsReplicated
Starting test: frssysvol
......................... SHIVA passed test frssysvol
Starting test: frsevent
There are warning or error events within the last 24 hours after the

SYSVOL has been shared. Failing SYSVOL replication problems may cause

Group Policy problems.
......................... SHIVA failed test frsevent
Starting test: kccevent
An Warning Event occured. EventID: 0x800004F1
Time Generated: 04/13/2005 12:40:47
Event String: An attempt by the Knowledge Consistency Checker

An Warning Event occured. EventID: 0x800004F1
Time Generated: 04/13/2005 12:41:04
Event String: An attempt by the Knowledge Consistency Checker

An Warning Event occured. EventID: 0x800004F1
Time Generated: 04/13/2005 12:41:21
Event String: An attempt by the Knowledge Consistency Checker

......................... SHIVA failed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0xC0001F60
Time Generated: 04/13/2005 12:37:45
Event String: The browser service has failed to retrieve the

......................... SHIVA failed test systemlog
Starting test: VerifyReferences
......................... SHIVA passed test VerifyReferences

Testing server: Default-First-Site-Name\DOC
Starting test: Replications
......................... DOC passed test Replications
Starting test: NCSecDesc
......................... DOC passed test NCSecDesc
Starting test: NetLogons
......................... DOC passed test NetLogons
Starting test: Advertising
......................... DOC passed test Advertising
Starting test: KnowsOfRoleHolders
......................... DOC passed test KnowsOfRoleHolders
Starting test: RidManager
......................... DOC passed test RidManager
Starting test: MachineAccount
......................... DOC passed test MachineAccount
Starting test: Services
......................... DOC passed test Services
Starting test: ObjectsReplicated
Authoritative attribute options on BICH (writeable)
usnLocalChange = 45347
LastOriginatingDsa = DOC
usnOriginatingChange = 3725167
timeLastOriginatingChange = 2005-03-28 14:03:50
VersionLastOriginatingChange = 3
Out-of-date attribute options on SHIVA (writeable)
usnLocalChange = 8515
LastOriginatingDsa = DOC
usnOriginatingChange = 1363
timeLastOriginatingChange = 2003-01-10 17:17:19
VersionLastOriginatingChange = 1
......................... DOC failed test ObjectsReplicated
Starting test: frssysvol
......................... DOC passed test frssysvol
Starting test: frsevent
......................... DOC passed test frsevent
Starting test: kccevent
......................... DOC passed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0x40011006
Time Generated: 04/13/2005 12:15:15
Event String: The connection was aborted by the remote WINS.

An Error Event occured. EventID: 0x40011006
Time Generated: 04/13/2005 12:45:15
Event String: The connection was aborted by the remote WINS.

......................... DOC failed test systemlog
Starting test: VerifyReferences
......................... DOC passed test VerifyReferences

Testing server: Default-First-Site-Name\BICH
Starting test: Replications
......................... BICH passed test Replications
Starting test: NCSecDesc
......................... BICH passed test NCSecDesc
Starting test: NetLogons
......................... BICH passed test NetLogons
Starting test: Advertising
Warning: DsGetDcName returned information for \\doc.carehawaii.com, when we were trying to reach BICH.
Server is not responding or is not considered suitable.
......................... BICH failed test Advertising
Starting test: KnowsOfRoleHolders
......................... BICH passed test KnowsOfRoleHolders
Starting test: RidManager
......................... BICH passed test RidManager
Starting test: MachineAccount
......................... BICH passed test MachineAccount
Starting test: Services
......................... BICH passed test Services
Starting test: ObjectsReplicated
Authoritative attribute servicePrincipalName on DOC (writeable)
usnLocalChange = 3712410
LastOriginatingDsa = BICH
usnOriginatingChange = 37006
timeLastOriginatingChange = 2005-03-26 14:04:14
VersionLastOriginatingChange = 8
Out-of-date attribute servicePrincipalName on SHIVA (writeable)
usnLocalChange = 79943
LastOriginatingDsa = BICH
usnOriginatingChange = 24848
timeLastOriginatingChange = 2005-03-24 14:56:47
VersionLastOriginatingChange = 5
Authoritative attribute msDS-HasInstantiatedNCs on DOC (writeable)
usnLocalChange = 3706595
LastOriginatingDsa = BICH
usnOriginatingChange = 32821
timeLastOriginatingChange = 2005-03-25 16:12:27
VersionLastOriginatingChange = 5
Out-of-date attribute msDS-HasInstantiatedNCs on SHIVA (writeable)
usnLocalChange = 79748
LastOriginatingDsa = BICH
usnOriginatingChange = 24607
timeLastOriginatingChange = 2005-03-24 11:59:27
VersionLastOriginatingChange = 3
Authoritative attribute msDS-hasMasterNCs on DOC (writeable)
usnLocalChange = 3706595
LastOriginatingDsa = BICH
usnOriginatingChange = 32821
timeLastOriginatingChange = 2005-03-25 16:12:27
VersionLastOriginatingChange = 7
Out-of-date attribute msDS-hasMasterNCs on SHIVA (writeable)
usnLocalChange = 79748
LastOriginatingDsa = BICH
usnOriginatingChange = 22370
timeLastOriginatingChange = 2005-03-24 11:42:27
VersionLastOriginatingChange = 5
......................... BICH failed test ObjectsReplicated
Starting test: frssysvol
......................... BICH passed test frssysvol
Starting test: frsevent
There are warning or error events within the last 24 hours after the

SYSVOL has been shared. Failing SYSVOL replication problems may cause

Group Policy problems.
......................... BICH failed test frsevent
Starting test: kccevent
......................... BICH passed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0x40011006
Time Generated: 04/13/2005 11:58:55
Event String: The connection was aborted by the remote WINS.

An Error Event occured. EventID: 0x40011006
Time Generated: 04/13/2005 12:28:55
Event String: The connection was aborted by the remote WINS.

......................... BICH failed test systemlog
Starting test: VerifyReferences
......................... BICH passed test VerifyReferences

Running partition tests on : Schema
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom

Running partition tests on : Configuration
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom

Running partition tests on : carehawaii
Starting test: CrossRefValidation
......................... carehawaii passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... carehawaii passed test CheckSDRefDom

Running partition tests on : DomainDnsZones
Starting test: CrossRefValidation
......................... DomainDnsZones passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom

Running partition tests on : ForestDnsZones
Starting test: CrossRefValidation
......................... ForestDnsZones passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom

Running enterprise tests on : carehawaii.com
Starting test: Intersite
......................... carehawaii.com passed test Intersite
Starting test: FsmoCheck
......................... carehawaii.com passed test FsmoCheck


Thanks if you can help,
M

 

[GlenJohnson]

Ok, first off, system overload. Way to much info. When you go into the event logs, what do you find?

Glen A. Johnson
Johnson Computer Consulting
If you're from Northern Il, Southern Wi, feel free to join Tek-Tips in Chicago IL

 

[SilviaS]

Are the DC's in correct time sync?

 

[aziel11]

I'm pretty sure that the DC's are all time synched to less than one minute. I'm not sure of the exact events I'll have to post those tomorrow, but there are NETLOGON errors when the server returns the RPC Server is unavailible error.

 

[aziel11]

As relating to the RPC server error on remote login, here is the event associated with the error:

Event ID:1219
Source: Winlogon
Description: Logon rejected for DOMAIN\Administrator. Unable to obtain Terminal Server User Configuration. Error: The RPC server is unavailable.

Hope that helps.

 

[SilviaS]

Do you have enought licenses left?

 

[aziel11]

yes we have licences. But I'm confused, how would the server know how many licenses we have and whether there were more clients than licenses? As I understand it the CAL's are someting that we as a business have to keep track of not the actual hardware. Please correct me if I am wrong but I never entered any license info into the server.

 

[SilviaS]

terminal server records this info, we had a problem like this a while back turned out it was XP, do any of your 2000 clients have a problem

 

[DG659]

sounds like youve got the DC`s out of sync (if thats posible)id look at this one first since you said it was offsite "One server (2000) doing Exchange 2000 and DC as well as secondary DNS"

i dont know much about multi DC setups so i could be talking rubbish.

 

[aziel11]

Silvia,
We don't have any windows 2000 clients. But I'm still confused, how does terminal server keep track, for example we just purchased 100 more CAL's a few months back. How does terminal server know that we now have 100 more licenses? Also I really don't think that is the issue, there are problems with communication between DC's and between clients and DC's. None of the windows NT machines can authenticate logins, but they can ping all the DC's and the DC's can ping eachother and the clients. Thanks,
M

 

[5jgibbs]

it looks like DNS replication and AD replication if failing and thats causing your problem.. make sure that your domain controlers have permition to access each other...

make sure you have a foward lookup zone for each DC on the network so it knows were to replicate to.. .. keep posting more info.. i will try to eval it further..

 

[SilviaS]

Do you have any security policies in place regarding authentication protocol?
ie. lanman v2 kerberos