UDP packets flooding my DMZ

[qhulk]

Hi everyone,
this is my first post here.

Here's the situation. My company has a SonicWALL pro 200 firewall appliance with three ports, one for LAN one for WAN and a DMZ.

We have 3 Domain controllers running Windows 2000 Advanced Server. One is located behind the LAN and services our internal users. We shallcall it the WKCL.com domain and users get to the internet out the WAN

port on the Sonicwall. On the DMZ port we have two other domain

controllers, each separate domains. One is exchange.ICL.com and it is

our mail server. The other is our webserver and it is called

web.webikl.com. So in essence we have three completely separate

domains, two of which share the DMZ port. All IPs are static and are

on the same subnet.

Here's the problem: several times a day our webserver puts out 28000

UDP packets per second, more than a million bytes of stuff, which

pounds the DMZ on our SonicWALL, opening up 30000 ports and shutting

down all communications for everyone, including the LAN. The only fix is to reboot the webserver and wait for it to happen again a few hours later.

Steps I've taken: We are running Norton AV 2003 with the latest definitions on all our DC's and scans happen daily. No viruses have been found.

I have run separate trojan scans on all DC's, using TauScan and no trojans have been found.

We thought we might have had a routing loop so we nuke and paved all hree DC's and made them separate domains that don't need to replicate to each other (previously they did). I have also attempted to put the webserver outside the DMZ but the problem reappeared and pounded our router. Putting the webserver outside without protection isn't really an option either, as it is critical to our business.

I have put on Network Monitor and Performance Monitor. Performance

Monitor shows me that it's definitely the webserver doing this and it is UDP packets, not TCP packets going out.

Network Monitor shows me that the destination IP's seem to be

completely random as is the destination ports. The only common port is the source port 1434 which is Microsoft SQL monitor. I can't shut out this port as management says webserver needs it.

On the surface it looks like an attack but as the servers are brand new, with all SP's and hotfixes applied and running behind the firewall, it doesn't seem likely.

What else should I be looking at? Any ideas would be greatly appreciated.

 

[wbg34]

Although it does not at first glance seem possible your servers reaction sounds alot like slammer. Read the description here:

http://www.extremenetworks.com/denialNotice.asp

It is simple enough to check for using symantecs removal tool found here.

http://securityresponse.symantec.com/avcenter/venc/data/w32.sqlexp.worm.removal.tool.html

 

[GiaBetiu]

What do you have installed on that IIS server? Some customized ISAPI filters or extension dlls?
I would name this freindly fire. When a component from IIS is flooding some ports..

Of course can be a virus too,. but you said that you tested this.

Gia Betiu
giabetiu@chello.nl
Computer Eng. CNE 4, CNE 5, MCSE Win2K

 

[qhulk]

thanks for your quick reply, WGB34. I downloaded that check from Symantec, ran it, nothing showed up. I agree that this looks like a virus/trojan/worm, but nothing shows.

A little more information. This storming problem used to occur on the Exchange server before we nuke and paved and reconfigured Exchange to be it's own separate domain. Exchange and the LAN DC replicated through a second NIC off of Exchange and through our internal switch. This looked like a routing loop (out the Exchange NIC, in through the internal switch, out the LAN port and back in through the DMZ) so we decided to make Exchange it's own domain and DC, thereby eliminating any loops. Since we made that change the problem disappeared from Exchange. We had reason to reboot our webserver (which hadn't been rebooted for ten months with no problems) and whammo! all of a sudden the storming happens on the webserver. Exchange is now smooth as silk and webserver barfs up UDP packets several times a day.

I will check up on what you suggest, GiaBetui.

 

[GiaBetiu]

Well,

First, I think that you should redisign that naming structure. Shall I understand that all those servers have their own AD forest??
As about replication, there is a tool in Windows, "AD sites and Services". There you can define/refine replication.


Gia Betiu
giabetiu@chello.nl
Computer Eng. CNE 4, CNE 5, MCSE Win2K

 

[qhulk]

All these servers have their own AD forest, that is true. They are completely separate domains. Users authenticate to the main DC on the Lan. Users get their email by authenticating through their Outlook clients.

AD Sites and Services is the first place I went when trying to replicate, but I kept getting RPC unavailable errors. Frustrations were mounting so high when I was first hired that nuke /n paving seemed like the best idea. Creating a separate Domain/AD for Exchange seemed like the simplest solution, since it would never have to replicate to anyone. Our organization is small enough that we won't need any other email servers for quite some time and managing users accounts is a snap since there are less than 20 at one time.

BTW, those domain names I gave above are only examples, not our real domain names.

 

[GiaBetiu]

Ok,... now it seems that has nothing to do with an ISAPi or something like this. Is just a configuration problem
Question is where?
Firewall is in NAT mode?
Sniff those UDP packets. What is the UDP source port? Destination?
What about DNS?
Are they using same DNS server? Or every domain with their own???

Gia Betiu
giabetiu@chello.nl
Computer Eng. CNE 4, CNE 5, MCSE Win2K

 

[qhulk]

Firewall has NAT enabled only for LAN. DMZ is Standard Mode/NAT disabled. All DC's are their own DNS resolvers, with forwarders pointing to our ISP.

Here's what the UDP tale of the tape tells me:
UDP: Src Port: unknown, (1634); Dst Port: Unknown (1434); Length-384 (0x180)

Underneath that we have:
UDP: Source port = 0x0662
UDP: Destination Port = 0x059A
UDP: Total length = 384 (0x180) bytes
UDP: UDP checksum - 0xA867
UDP: Data: number of data bytes remaining = 376 (0x0178)

 

[qhulk]

One other piece of info. I ran task manager to see what was running when the storm happened and sqlsrvr.exe jumped up to 80-90% cpu usage. So SQL is doing this but the operative question is why?

 

[ricpinto]

R u running Veritas back up program? How big is your RAM?
I'm not really good about this but I encountered same problem sqlsrvr.exe is 90% cpu usage. i reboot the server but it will happen again after several hours with the same problem. This sqlsrvr program is from my veritas ver 9. I stopped Veritas services and other unnecesarry services and put it to manual start up.. I only bcak up during night time and afterwards I stopped the srvcs again. I also upgrade to W2Ksp3. I put back the Veritas services automatic for good when I upgraded my RAM.

Hope it will help.

 

[qhulk]

As it happens, Ricpinto, we are running Veritas Backup for Windows 9.0. We have 1.5 gigs of ram on our webserver and 512 on Exchange.

In my first post I mentioned that we had reason to reboot the server, and that reason was I installed Veritas 9.0. Interesting...

I think tomorrow I will shutdown all Veritas services during the day and see what that does. You installed more ram, you say? Did your problem go away with more ram? If this turns out to be Veritas then we will probably chuck it out the window and find something else. Lemme know what you did and if your problems went away.

 

[ricpinto]

Don't tell me you are running HP Netserver as well. Actually it doesn't matter how big is your RAM, my concern is how many percent is your mem usage compared to the physical RAM. On my other server which is IBM x series 50% or less then I have no problem with veritas 9.

Actually I really never solve the problem because I've used my old reliable lower version of Veritas, upgrade my RAM and update it to win2ksp3. I really don't trust ver 9 on my Netserver so I've discontinued using it and beside I can't afford to test because my users were cursing me and even my boss was not happy anymore. They need the server to run already...

What I know is one of the 3 was the possible solution. 1. upgrade ram, 2. Ver 9 got bugs (lower ver is OK), 4. update win2ksp3.

Hope it can help again..

 

[qhulk]

This is uncanny! We are running HP Netservers! We have a couple of TC4100 and an LC2000. All servers are running Win2k with SP3 and all the hotfixes that Windows Update can give me. So you were never able to get it working? That's kind of a downer since Veritas cost upwards of $1400 (Can) and we are not likely drop it without getting our money's worth.

Anywho, this morning I completely uninstalled Veritas 9.0 and all instances of sqlsrvr.exe disappeared from Task Manager. Will keep you updated as to whether problem disappears. You never mentioned this, did you notice a flood of UDP packets coming from the problem server as well?

 

[GiaBetiu]

Having a lok in a table of wellknown ports here is the result:
ms-sql-m 1434/udp Microsoft-SQL-Monitor
loaprobe 1634/udp Log On America Probe

So,... can be a link with what you have on that server?

Gia Betiu
giabetiu@chello.nl
Computer Eng. CNE 4, CNE 5, MCSE Win2K

 

[qhulk]

The thing about those ports is that the source port always changes...but the destination port remains always 1434. The IP addresses it is sending to are always changing as well, and every 30 I can expect a multicast address. Right now my money is on Veritas, but I won't know for sure until the end of the day if we survive without having to reboot. I put in a call to my nearest support and hopefully he gets back to me with some concrete answers.

 

[GiaBetiu]

Interesting,... why your application is using that SQL Monitor? Are the developers guys complaining that their application is not workig properly, or that there are errors? Is looking like:
- the application is trying to have a connection via that SQLmon port
- application doesn't have acknowledgement on that port
- ok, is trying the other port (maybe a range)
etc., etc
Think about this idea. Start with this flow, and see how the developers are accessing those ports. See how they are finding the destination? Name resolution problems? IP routing problems?

Gia Betiu
giabetiu@chello.nl
Computer Eng. CNE 4, CNE 5, MCSE Win2K

 

[qhulk]

Actually we don't have any developers, we are resellers to government. And SQL was never installed, the webserver just hosts our website that local users and customers access. No name resolution problems to report anywhere either.

 

[ricpinto]

My gosh.. my netserver is LC2000. I have a problem of UDP flooding as well and its also my webserver. Maybe now you understand why my users are cursing me because they can't surf the net.. just like that I swap my ver 9 with one my other IBM server ( veritas 8.6). So 4 me it's not a waste totally.

About the sqlsrvr thing, it's the database engine of veritas 9. It's not really the MS SQL server!!

I can bet 100%... you won't reboot today.

 

[qhulk]

heh heh, this is cool! It's 12:29 and the webserver has been running error free since 8:15am. So this Veritas database engine...am I to assume that this is a thing that Veritas needs to install and run or Veritas will not work? If so then I don't see any other way we can use it unless I install it, change all services to startup type manual and only turn them on when I wanna do a backup.

 

[SgtB]

Here's an idea. Your webserver only needs to talk to the outside via port 80 (or 443) right? Well then turn on TCP/IP filtering on the webserver (2000 webserver right?) to only allow port 80/443/whatever you need. That way the UDP packets won't even get as far as the NIC in the webserver.
That way you can keep your Bexec 9.0!
Just a thought!

Cool upcoming game! Check it out!

http://www.planetside-universe.com

!