replication issues

[58sniper]

4 hours before I go on vacation, the boss dumps a new client on me. "They can't connect to some other location..."

2003 SBS box in HQ
2003 SP1 box in remote location
box-to-box VPN that appears to be working fine.

Servers are both DCs, and both GCs.
They are configured as two separate sites in ADSS.

From HQ, I can browse onto the remote server; I can view remote DNS in MMC

From remote, I cannot browse onto HQ box. If I browse the 'hood, and select the HQ server, I get "\\smg-sh is not accessible. You might not have permission to use this network resource. Contact the administrator of this server to find out ifyou have access permissions. Logon Failure: The target account name is incorrect."

Keep in mind I'm logged into the remote server as the domain admin - the same account that lets me browse from the HQ server to the remote.

A ton of issues have been resolved. A bunch of services on the HQ side were disabled including w32time, intersite-messenging. I got that taken care of and a crapload of KDC errors on the HQ server stopped. I'm still seeing them on the remote server every 15 minutes. They amount to 3611, 1865, and 1566 errors - 4 of each every 15 minutes. Time sync was established with an outside source on the HQ server, and both servers are within a minute of each other.

I've verifed that the sites are configured and part of the defaultipsitelink. DNS shows counters that are off by THOUSANDS, so the problem has been going on for a while. Event logs for Directory Service only go back a couple of days, and the KDC errors go all the way back.

From both sides, I can ping the IP and name of the other server. DNS on both side, while out of sync with the other, does have the DC and GC records for both servers correct.

DCDIAG shows the following from the remote server (\\jrserver):

Domain Controller Diagnosis

Performing initial setup:
Done gathering initial info.

Doing initial required tests

Testing server: Detroit\jrserver
Starting test: Connectivity
......................... jrserver passed test Connectivity

Doing primary tests

Testing server: Detroit\jrserver
Starting test: Replications
[Replications Check,jrserver] A recent replication attempt failed:
From SMG-SH to jrserver
Naming Context: DC=ForestDnsZones,DC=smg,DC=local
The replication generated an error (1256):
The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
The failure occurred at 2006-03-24 23:17:16.
The last success occurred at 2006-02-15 19:09:49.
3558 failures have occurred since the last success.
[Replications Check,jrserver] A recent replication attempt failed:
From SMG-SH to jrserver
Naming Context: DC=DomainDnsZones,DC=smg,DC=local
The replication generated an error (1256):
The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
The failure occurred at 2006-03-24 23:17:16.
The last success occurred at 2006-02-15 19:09:49.
3561 failures have occurred since the last success.
[Replications Check,jrserver] A recent replication attempt failed:
From SMG-SH to jrserver
Naming Context: CN=Schema,CN=Configuration,DC=smg,DC=local
The replication generated an error (-2146893022):
The target principal name is incorrect.
The failure occurred at 2006-03-24 23:17:16.
The last success occurred at 2006-02-15 19:09:49.
3567 failures have occurred since the last success.
[Replications Check,jrserver] A recent replication attempt failed:
From SMG-SH to jrserver
Naming Context: CN=Configuration,DC=smg,DC=local
The replication generated an error (-2146893022):
The target principal name is incorrect.
The failure occurred at 2006-03-24 23:17:16.
The last success occurred at 2006-02-15 19:09:49.
3569 failures have occurred since the last success.
[Replications Check,jrserver] A recent replication attempt failed:
From SMG-SH to jrserver
Naming Context: DC=smg,DC=local
The replication generated an error (-2146893022):
The target principal name is incorrect.
The failure occurred at 2006-03-24 23:17:16.
The last success occurred at 2006-02-15 19:09:49.
1502 failures have occurred since the last success.
REPLICATION-RECEIVED LATENCY WARNING
jrserver: Current time is 2006-03-24 23:18:02.
DC=ForestDnsZones,DC=smg,DC=local
Last replication recieved from SMG-SH at 2006-02-15 19:20:54.
DC=DomainDnsZones,DC=smg,DC=local
Last replication recieved from SMG-SH at 2006-02-15 19:20:54.
CN=Schema,CN=Configuration,DC=smg,DC=local
Last replication recieved from SMG-SH at 2006-02-15 19:20:54.
CN=Configuration,DC=smg,DC=local
Last replication recieved from SMG-SH at 2006-02-15 19:20:54.
DC=smg,DC=local
Last replication recieved from SMG-SH at 2006-02-15 19:20:54.
REPLICATION-RECEIVED LATENCY WARNING
Source site:
CN=NTDS Site Settings,CN=Default-First-Site-Name,CN=Sites,CN=Configurat
ion,DC=smg,DC=local
Current time: 2006-03-24 23:18:02
Last update time: 2006-02-15 19:05:26
Check if source site has an elected ISTG running.
Check replication from source site to this server.
......................... jrserver passed test Replications
Starting test: NCSecDesc
......................... jrserver passed test NCSecDesc
Starting test: NetLogons
......................... jrserver passed test NetLogons
Starting test: Advertising
......................... jrserver passed test Advertising
Starting test: KnowsOfRoleHolders
[SMG-SH] DsBindWithSpnEx() failed with error -2146893022,
The target principal name is incorrect..
Warning: SMG-SH is the Schema Owner, but is not responding to DS RPC Bi
nd.
[SMG-SH] LDAP bind failed with error 8341,
A directory service error has occurred..
Warning: SMG-SH is the Schema Owner, but is not responding to LDAP Bind
.
Warning: SMG-SH is the Domain Owner, but is not responding to DS RPC Bi
nd.
Warning: SMG-SH is the Domain Owner, but is not responding to LDAP Bind
.
Warning: SMG-SH is the PDC Owner, but is not responding to DS RPC Bind.

Warning: SMG-SH is the PDC Owner, but is not responding to LDAP Bind.
Warning: SMG-SH is the Rid Owner, but is not responding to DS RPC Bind.

Warning: SMG-SH is the Rid Owner, but is not responding to LDAP Bind.
Warning: SMG-SH is the Infrastructure Update Owner, but is not respondi
ng to DS RPC Bind.
Warning: SMG-SH is the Infrastructure Update Owner, but is not respondi
ng to LDAP Bind.
......................... jrserver failed test KnowsOfRoleHolders
Starting test: RidManager
......................... jrserver failed test RidManager
Starting test: MachineAccount
......................... jrserver passed test MachineAccount
Starting test: Services
......................... jrserver passed test Services
Starting test: ObjectsReplicated
......................... jrserver passed test ObjectsReplicated
Starting test: frssysvol
......................... jrserver passed test frssysvol
Starting test: frsevent
There are warning or error events within the last 24 hours after the
SYSVOL has been shared. Failing SYSVOL replication problems may cause
Group Policy problems.
......................... jrserver failed test frsevent
Starting test: kccevent
An Warning Event occured. EventID: 0x8000061E
Time Generated: 03/24/2006 23:07:16
Event String: All domain controllers in the following site that
An Error Event occured. EventID: 0xC000051F
Time Generated: 03/24/2006 23:07:16
Event String: The Knowledge Consistency Checker (KCC) has
An Warning Event occured. EventID: 0x80000749
Time Generated: 03/24/2006 23:07:16
Event String: The Knowledge Consistency Checker (KCC) was
An Warning Event occured. EventID: 0x8000061E
Time Generated: 03/24/2006 23:07:16
Event String: All domain controllers in the following site that
An Error Event occured. EventID: 0xC000051F
Time Generated: 03/24/2006 23:07:16
Event String: The Knowledge Consistency Checker (KCC) has
An Warning Event occured. EventID: 0x80000749
Time Generated: 03/24/2006 23:07:16
Event String: The Knowledge Consistency Checker (KCC) was
An Warning Event occured. EventID: 0x8000061E
Time Generated: 03/24/2006 23:07:16
Event String: All domain controllers in the following site that
An Error Event occured. EventID: 0xC000051F
Time Generated: 03/24/2006 23:07:16
Event String: The Knowledge Consistency Checker (KCC) has
An Warning Event occured. EventID: 0x80000749
Time Generated: 03/24/2006 23:07:16
Event String: The Knowledge Consistency Checker (KCC) was
An Warning Event occured. EventID: 0x8000061E
Time Generated: 03/24/2006 23:07:16
Event String: All domain controllers in the following site that
An Error Event occured. EventID: 0xC000051F
Time Generated: 03/24/2006 23:07:16
Event String: The Knowledge Consistency Checker (KCC) has
An Warning Event occured. EventID: 0x80000749
Time Generated: 03/24/2006 23:07:16
Event String: The Knowledge Consistency Checker (KCC) was
......................... jrserver failed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0x825A0011
Time Generated: 03/24/2006 22:47:38
(Event String could not be retrieved)
An Error Event occured. EventID: 0x40000004
Time Generated: 03/24/2006 22:48:14
Event String: The kerberos client received a
An Error Event occured. EventID: 0x40000004
Time Generated: 03/24/2006 22:48:14
Event String: The kerberos client received a
An Error Event occured. EventID: 0x40000004
Time Generated: 03/24/2006 22:54:11
Event String: The kerberos client received a
An Error Event occured. EventID: 0x40000004
Time Generated: 03/24/2006 22:59:32
Event String: The kerberos client received a
An Error Event occured. EventID: 0x40011006
Time Generated: 03/24/2006 23:17:26
Event String: The connection was aborted by the remote WINS.
An Error Event occured. EventID: 0x40000004
Time Generated: 03/24/2006 23:18:02
Event String: The kerberos client received a
An Error Event occured. EventID: 0x40000004
Time Generated: 03/24/2006 23:18:02
Event String: The kerberos client received a
......................... jrserver failed test systemlog
Starting test: VerifyReferences
......................... jrserver passed test VerifyReferences

Running partition tests on : ForestDnsZones
Starting test: CrossRefValidation
......................... ForestDnsZones passed test CrossRefValidation

Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom

Running partition tests on : DomainDnsZones
Starting test: CrossRefValidation
......................... DomainDnsZones passed test CrossRefValidation

Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom

Running partition tests on : Schema
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom

Running partition tests on : Configuration
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom

Running partition tests on : smg
Starting test: CrossRefValidation
......................... smg passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... smg passed test CheckSDRefDom

Running enterprise tests on : smg.local
Starting test: Intersite
......................... smg.local passed test Intersite
Starting test: FsmoCheck
......................... smg.local passed test FsmoCheck

and from the HQ server (\\smg-sh)

Domain Controller Diagnosis

Performing initial setup:
Done gathering initial info.

Doing initial required tests

Testing server: Default-First-Site-Name\SMG-SH
Starting test: Connectivity
......................... SMG-SH passed test Connectivity

Doing primary tests

Testing server: Default-First-Site-Name\SMG-SH
Starting test: Replications
REPLICATION LATENCY WARNING
SMG-SH: This replication path was preempted by higher priority work.
from jrserver to SMG-SH
Reason: The operation completed successfully.
The last success occurred at (never).
Replication of new changes along this path will be delayed.
Progress is occurring normally on this path.
REPLICATION LATENCY WARNING
SMG-SH: This replication path was preempted by higher priority work.
from jrserver to SMG-SH
Reason: The operation completed successfully.
The last success occurred at (never).
Replication of new changes along this path will be delayed.
Progress is occurring normally on this path.
......................... SMG-SH passed test Replications
Starting test: NCSecDesc
......................... SMG-SH passed test NCSecDesc
Starting test: NetLogons
......................... SMG-SH passed test NetLogons
Starting test: Advertising
......................... SMG-SH passed test Advertising
Starting test: KnowsOfRoleHolders
......................... SMG-SH passed test KnowsOfRoleHolders
Starting test: RidManager
......................... SMG-SH passed test RidManager
Starting test: MachineAccount
......................... SMG-SH passed test MachineAccount
Starting test: Services
......................... SMG-SH passed test Services
Starting test: ObjectsReplicated
......................... SMG-SH passed test ObjectsReplicated
Starting test: frssysvol
......................... SMG-SH passed test frssysvol
Starting test: frsevent
There are warning or error events within the last 24 hours after the
SYSVOL has been shared. Failing SYSVOL replication problems may cause
Group Policy problems.
......................... SMG-SH failed test frsevent
Starting test: kccevent
......................... SMG-SH passed test kccevent
Starting test: systemlog
......................... SMG-SH passed test systemlog
Starting test: VerifyReferences
......................... SMG-SH passed test VerifyReferences

Running partition tests on : ForestDnsZones
Starting test: CrossRefValidation
......................... ForestDnsZones passed test CrossRefValidation

Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom

Running partition tests on : DomainDnsZones
Starting test: CrossRefValidation
......................... DomainDnsZones passed test CrossRefValidation

Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom

Running partition tests on : Schema
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom

Running partition tests on : Configuration
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom

Running partition tests on : smg
Starting test: CrossRefValidation
......................... smg passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... smg passed test CheckSDRefDom

Running enterprise tests on : smg.local
Starting test: Intersite
......................... smg.local passed test Intersite
Starting test: FsmoCheck
......................... smg.local passed test FsmoCheck

I've gone through every resource I can think of short of calling PSS.

Anyone got some ideas?

Pat Richard, MCSE(2) MCSA:Messaging, CNA(2)

 

[TechyMcSe2k]

http://support.microsoft.com/kb/898060/en-us

We had an issue with our remote DC's not replicating when they had SP1 installed. This is one place to look for a solution to your problem.

 

[58sniper]

Actually, it wasn't that. Kerberos was totally hosed. I pulled the remote server out of the domain and added it back in and the problem went away immediately. What a freakin' pain.

Pat Richard, MCSE(2) MCSA:Messaging, CNA(2)