Caller can hear but Called Can't 1

[mikej97204]

I had a strange issue happen today that I wanted to see if anyone has an idea if it's possible that our switch could have a problem.

I have a G3r ver 9.5 with an EPN. We are a call center where our customers call into us and we call out to obtain a interpreter and sometimes a 3rd party to conference together with the first caller.

This morning our customers were calling into us just fine. When the call agent called out to get an interpreter, the agent could hear the interpreter but the interpreter could not hear the agent. This started with a few agents and then spread to about 15 agents with some not being affected at all. This went on for about 15 minutes and then cleared up.

The confusing part is that our default outbound is over QWest ISDN PRI T1's. But because we can't dial toll free numbers, our toll frees go out over Electric Lightwave/Integra ISDN PRI T1's. We have a number of interpreters in a call center that we call over the toll frees and some of the calls acted the same way when those interpreters were called... the calling agent could hear the distant end but they could not hear the agent.

Thhis happened this morning and again this afternoon for about the same period of time. I opened tickets with both carriers through our carrier point of contact but he is saying that it wasn't a carrier problem it was our switch. My response is that I don't know of any way that a TN 464F T1 card can go into half duplex like that and only work one way on a call. Not only that, but one carrier is in the R cabinet and the other carrier is in the EPN cabinet, so they aren't even on the same backplane or even in the same cabinet.

While I know this can happen with VOIP, we don't run VOIP. All of our T1's are ISDN PRI. Our 3rd phone carrier is MCI that comes in over a DS3 and as far as I know, none of the calls that came in our out over the DS3 had any problems. I plan on changing our default outbound for non-toll frees to go out the MCI path tomorrow to see if they are affected should it happen again.

My question has to do with the R switch. I just moved us over to it from a G3si and my first thought was maybe it was something in the switch, but there are no alarms or errors and it ran fine the rest of the day when our traffic through it was much higher and all 4 of the QWest T1's filled up and rolled over to the MCI T1's. So it doesn't seem to be a traffic volume issue.

Does anyone know if it is possible for something in the R hardware or software to allow a caller to hear the other end but not be heard? And for this to only happen for about 15 minutes and then clear up?

Thanks for any insight into this.

Mike

 

[inerguard]

You are experiencing one-way-transmission and the problem is with your telco.

if you do a "list measurements ds1 log 01a04" where 01a04 is the position of the DS1 board you will see errors on the circuit. This is your clue that the problem is at the Telco providers exchange, not your switch.


[Started on Version 3 software 15 years a go]

 

[mikej97204]

Thanks a lot for that info. I did the command on the T1's in question and one of the DS1 boards showed CSS=255 and LOFC=255 during one of the time periods of the log. It wasn't at the time of the first incidence though, but about an hour later. It gives me something to keep an eye on if it happens again. Thanks again.

Mike

 

[mikej97204]

Ok, I have an update to this problem. What I suspect is that there isn't a problem with the 2nd carrier on my TK GP 2. This would mean that the real problem is only on TK GP 9, which is 4 ISDN PRI T1's that we get from Qwest. All 4 of these T1's are on TN464 cards in the EPN cabinet of the R swtich.

What I am wondering is if the fiber link that connects the EPN cabinet to the main cabinet might have a problem in the TX side of the fiber link or in one of the TN750C cards? Although I don't know why it would be sporadic and not all the time. I'm not sure the length of the fiber link but the two cabinets are about 8 feet apart, so it's probably a 15 foot long fiber link. I checked and it's not kinked, pinched or twisted and the connections are snug.

Is this another piece of equipment that either works or it doesn't or could there be a sporadic problem with it?

Thanks

Mike

 

[mikej97204]

Another update. After identifing the problem looked to be on the EPN cabinet, the people we bought the R switch from dug deeper into it and realized that, although we have a TN775 card in the EPN we didn't have a TN771 card. And the TN771 is required in the EPN when there are ISDN PRI circuits to handle, amoung several things, bad packets and packet loss. So they put one in on Friday and I'll see how it goes during our traffic period on Monday. If there are no more problems, I can move all 4 original T1's back to the EPN cabinet, plus add some more.

Mike

 

[mikej97204]

Well, putting in the TN771 card did not slove the problem. It went all day, including during our peak time when everything was almost maxed out without any problems at all. It was when traffic started slowing down that we had another case of the called party could not hear. As soon as I changed my outbound route path to go out a different path, it cleared up. I've moved 3 of the 4 T1's off of the EPN and back to the main carrier so I don't run out of outbound capacity, but I can't leave them there as I'm afraid I will run out of time slots.

I'm back with the vendor now and he's doing some more thinking. I've suggested that since the EPN on our G3si is the same, we could try swapping the whole thing and see if the problem goes away. Or swap the interface cards and fiber link first or something to try and find out what is the problem... the cabinet or the cards. I sure wish we could get this figured out though.

Mike

 

[mikej97204]

Since I always hate reading about problems that never get closed out with the final resolution, I'm adding the last part that the problem appears to be fixed and what was found.

Yesterday, Qwest admitted that they had a Master Ticket open on our problem... which I was told meant that they had other customers complaining about one way audio and it wasn't just us. They found something bad in a 5ess switch somewhere and fixed it last night. I've not been told where or what was fixed, but just having it fixed is good enough. We didn't have a single problem today (and we usually had 2 or 3 each day at different times) so it does look like it's fixed.

Mike

 

[inerguard]

Can I gloat and say told you so, lol

It's always the Telco thjat's at fault, never the Definity!

Only joking about the gloating bit, but glad that it's fixed, I know how frustrating a fault is when you know it's the Telco and they blame everything else in the world (even the weather) except their own systems.


[Started on Version 3 software 15 years a go]

 

[mikej97204]

You can gloat. We lost several days of pushing Qwest to work on this problem because several people wouldn't believe me when I said the problem was not in the switch. Once, I found that our ticket with Qwest had been closed out saying it was an internal issue and another time I found that updates and more call record examples that had been sent in had never been entered into the ticket for Qwest to trace and track down which T1 path the problems were happening on. I'm glad it's fixed, but it was a frustrating week and half. This is one time I'm not going to mind at all handing my boss the bill for the hours I had to spend with a very high level tech while we chased ghosts and moved trunks around and tried different things to isolate it within the switch itself. The silver lining in this though, he did find some configuration settings in the trunk groups that, while they weren't causing this problem, could have caused us problems in the past or in the future. And he was excellent at explaining what he was looking at and mostly walked me through doing the work so I'd know what was changed and why. So I did get some good practice creating and changing signaling groups, trunks, changing T1's between trunks, running tests and interpreting test data.

I was trained on the Roam Redwood back in the mid 80's and have been a PBX administrator on several other phone systems over the years (mostly I've been computers, network and systems) and have been administrating the Definity G3si for the last 5 years and now the G3r. I'm not an expert, but at least I'm past knowing enough to be dangerous for the most part.

Mike

 

[Coineandude]

We are/were having the same issue with QWest here.

I got the same buzz from the QWest tech. He told me on the phone that there was some wider spread issue and other customers had complained of similar issues and they were working to resolve it.

Then I saw my ticket had been closed and the issue was described as customer equipment error.

eh?????

Anyway I hope you are right and it is fixed.

--Shane

 

[victor5908]

QWest seems to be highly problematic-I have experienced similiar issues with them in the past.

 

[mikej97204]

Hi Shane, we had it happen to us again at 8:09 Pacific time. So I sent in the call examples again that I got from this morning. As soon as I changed my route pattern over to MCI, it cleared up. We have 3 more MCI T1's that we have requested be activated on our MCI DS3 that we are going to use to replace the Qwest lines until they really get it fixed.

I'm in Portland. What part of the country are you in?

Mike

 

[Coineandude]

I am in the Seattle Metro area and yes, we are still seeing the behavior here.

bleh. Makes me wish I still had those AT&T spans floating around...

 

[mikej97204]

Here is what we found out this afternoon and since you are in Seattle, it probably applies to you as well:


"The TMC Tech Anthony, has heard from the level 3 techs at Qwest and they have admitted that the problem has not been resolved at the "Tequilla" switch site here in the NorthWest. (what an appropriate name, it has a worm in it) All four of your circuits home off of that switch before ever going out onto the Qwest network, hence the problem."

That was what the guy that interfaces between us and our telco carriers found out this afternoon. We are hoping that it will be fixed before Monday. We've not had Qwest all that long (less than 6 months I believe) and so far, we have had more problems with them than all the other carriers we have and have had, combined. Everyone has problems from time to time, that's a given, it's how fast they respond and fix it that is the key part of it.

Mike