Huge network traffic from datastore

[adevreede]

I'm working at a site, about 80 servers in 27 zones. FR3/SP3. They all use a direct connection to the datastore. The problem is as follows: Many of the servers are executing the stored procedure (in the datastore database) JoinReadEntryByName (349??), requesting large amounts of data from the datastore. Some of the servers in a zone are fine, they run the procedure about 600 times in 6 hours, which is not too bad, but the other servers run the procedure more like 30000(!!) times in 6 hours. The applications are the same on the servers, amount of users are the same, all the server have been configured exactly alike. There is no problem early in the morning when there are no users on the servers, but as user start logging in, traffic starts building up. The amount of data the datastore is sending in response to these requests is overloading the wan links. The traffic can get up to 1 gigs in 24 hours, PER SERVER! Does anyone know what the problem could be? Or even what the stored procedure JoinReadEntryByName is used for?

 

[mitchero]

What are their requirements for zoning? Twenty seven zones in a single FARM? Are all 27 zones in seperate locations? Managed by location specific admin? I would say that the problem they are experiencing could/should be resolved with an architectural overhaul. Probably not what you want to hear.

 

[adevreede]

27 different locations. We're working on a redesign, with a lot more farms and fewer zones. But this is going to take a while, and the traffic is killing the WAN right now... So yes, I had heard that suggestion befeore ;-)

 

[mitchero]

Ouch. I figured that you had. Must be an admin nightmare. Is there a political/technical obstacle that would prevent you from creating one farm with one zone?..or maybe two zones with a replicated data store if you want redundancy. Er, I haven't found any helpful info on the datastore question. I'll pose the question to Citrix.

 

[CitrixEngineer]

27 Zones is what is causing the huge amount of network traffic, IMO!

My understanding of how it works is that (e.g.) every time a user logs on, a session start event is recorded in the local data store, then IMA transfers this from the local Zone master to ALL the others. This is easily monitored using Network Manager.

With so many zones, it's possible that updates are sent by one Zone master before the updates are completed from another.

I know this doesn't help the immediate issue - but I hope it helps your design!

 

[adevreede]

Redesign does seem to be the solution. But even though I believe the design is causing a lot of the traffic, the amount of traffic is simply too ridiculous for event the worst of designs . I've now set up some of the larger zones to all connect to the datastore through the datacollector only. On one site, with 11 machines, this has reduced the amount of JoinReadEntryByName 349 queries by 60 %. but on another site, with 6 machines, the amount of JoinReadEntryByName queries has actually increased by 150%! Luckily, the site with the reduced traffic is a WAN location, where the site with increased traffic is in the same location as the datastore, so the overall situation has improved.
Still no clue what the &%%$&$&$ query does though....

 

[CitrixEngineer]

As an idea, you could set up a zone consisting of one server (Data Collector) that serves small locations at a Main site. The smaller sites would then update that Data Collector, which, as Zone Master, would update the Zone Master of the Main site in record time.

I'll look into the JoinReadEntryByName query...

HTH

 

[adevreede]

Problem has been solved. The datastore was being queried for updated printer information continually. A private hotfix from Citrix stopped the updates from occuring, and reduced traffic by more than 99%.

 

[bran2235]

Where is that hotfix??
What exactly did it do???


Tanks,
Brandon