Phones are 'sluggish'.

[PBXTech]

Hello all.
Wondering if I can get some help with this one.
As of yesterday, folks are having issues with dialing, hanging up, and receiving calls.
At times, you can attempt on-hook dialing but the phone doesn't go off hook until 10-15 seconds after you have dialed an extension or an outside number.
Also a phone won't immediately hang up when the handset is placed back onto the switch-hook.

The only thing that has changed is that our CLEC put 'unknown/unknown' for our caller id settings on our ISDN-PRIs due to folks in our organization concerned about the HIPAA Privacy Rule.

System is an S8300/G700 with 6+ S8300LSP/G700s all running 3.1.4 (642.1)

 

[LoopbackJack]

What kind of phones are they? Are all phones/locations being affected? Maybe you are running out of DSP resources?

 

[PBXTech]

4620/4610/4602 sets.
phones across our network (MPLS) are being affected.
i just had a call on my first line appearance (4620) that i couldn't pick up. the caller called back and said it rang but went dead.

 

[LoopbackJack]

Just the remote locations are affected? Is Voice QoS set up on your MPLS? Has voice quality been affected or is it just the control traffic being delayed? The remote sites are getting their call control from the Primary S8300, if packets are being delayed or sent out of order it would result in delayed dial tone, lighting of lights, pushing of buttons etc. If your DSP's are cool, I would lean towards an issue on your WAN. Just because you have your DSCP and QoS settings set up in the PBX, your MPLS vendor would need to honor those tags across their network.
Other thoughts... Firmware current? List trace have any more clues?

 

[PBXTech]

it's even happening here at our main site where i'm at. QoS has been set up and working fine for the past 3+ years on the MPLS. it's just the control traffic that is being delayed. the remote sites all have their own ISDN-PRIs for inbound and outbound calling.

 

[LoopbackJack]

Something is delaying the control traffic. Have there been any changes to your default route settings or have the IT folks been playing with the routers? How's your processor occupancy?
Does anything interesting show up on a trace, or in the event or alarm logs?
If the problem just came up and there have been no changes in your network, considered rebooting the system after hours.
Also, check the firmware on your gateways and IP sets and make sure they are current for your release. I've seen firmware problems come up for no reason, and updating fixed the issues.



Rock on, be strong... SG
APDS UC, APDS CC, ACE IP Telephony

 

[PBXTech]

here's a trace showing packet loss.


time data
15:06:29 G729A ssff ps:20 rn:1/2 169.123.210.4:2106 169.123.165.4:2094
15:06:29 xoip: fax:Relay modemff tty:US 169.123.210.4:2106 (igc)
15:06:29 xoip: fax:Relay modemff tty:US 169.123.165.4:2094 (igc)
15:06:29 G729 ssff ps:20 rn:1/2 169.123.210.4:2106 169.123.165.4:2094
15:06:29 G711MU ssff ps:20 rn:2/2 169.123.165.116:2884 169.123.165.4:2110
15:06:29 xoip: faxT modemff tty:US 169.123.165.4:2110 uid:0x946
VOIP data from: 169.123.165.4:2110
15:06:31 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:06:31 Pkloss:* * 0 0 0 0 0 0 0 0: Oofo:0 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:06:41 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:06:41 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:06:51 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:06:51 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110

time data
15:07:01 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:07:01 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:07:11 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:07:11 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:07:21 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:07:21 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:07:41 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:07:41 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:07:51 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:07:51 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:08:01 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0

time data
15:08:01 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:08:11 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:08:11 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:08:21 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:08:21 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:08:31 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:08:31 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:08:41 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:08:41 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:08:51 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:08:51 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
time data
VOIP data from: 169.123.165.4:2110
15:09:01 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:09:01 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:09:11 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:09:11 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:09:21 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:09:21 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
VOIP data from: 169.123.165.4:2110
15:09:31 Jitter:0 0 0 0 0 0 0 0 0 0: Buff:8 WC:0 Avg:0
15:09:31 Pkloss:* * * * * * * * * *: Oofo:255 WC:0 Avg:0
15:09:35 active attendant 3109 cid 0xd9
15:09:37 dial 3137
15:09:37 ring station 3137 cid 0xd9
15:09:37 G711MU ssff ps:20 rn:1/1 169.123.210.52:2808 169.123.210.9:2078

 

[LoopbackJack]

That packet loss is not good... I would start by checking the switch port the S8300 is plugged into. Make sure both sides are locked down to 100/full. You have a networking issue somewhere bro. Those ***'s are 100% packet loss..not good at all. Maybe a router went haywire, check with your IT folks and have them do some testing.
Can you plug a phone into the same network switch as the S8300 and see if the problem persists? The problem is either there or downstream from it.

Rock on, be strong... SG
APDS UC, APDS CC, ACE IP Telephony

 

[PBXTech]

it happened at the Main site between two phones. it didn't even go over the MPLS network. so those guys latched onto that when i said that part.
phone -> patch cord -> jack -> wire run -> patch panel -> patch cord -> Extreme 240 -> patch cord -> Extreme 450 -> 3xG700 stack with S8300.
everything has been verified as 100/Full.
i did do a shutdown of the S8300 and then removed power from the Main G700 stack Friday evening. still have a few end users in other locations with G700/S8300LSPs w/Extreme 240s having issues.

 

[LoopbackJack]

It sounds like the problem could be in the Extremes and or G700's then. Do those switches have the latest code in them? How about gateway firmware? If they are under maintenence you could open up a ticket with Avaya.

Rock on, be strong... SG
APDS UC, APDS CC, ACE IP Telephony