CMS loses SPI Session / PBX Software Interchage Request!

[Stinney]

In 2 days time, the CMS has lost it's SPI Session and as a result all agents company wide show up as in the state of OTHER until they receive a call or log out and back in again.

At the same exact time the PBX is showing a Software Request in the Initialization Causes:

Cause Action Escalated Mode
Software Request 1 no Active

Our networking group doesn't see any issues, so we can't point to any network issues for this problem.

Any thoughts?

- Stinney

Favorite all too common vendor responses: "We've never seen this issue before." AND "No one's ever wanted to use it like that before.

 

[Stinney]

This has happened 2 more times, the last without a corrisponding interchange event in the PBX, but it effected both of our ACDs where the first 2 only effected ACD 1.

We've checked the CLAN to make sure that it's dedicated to the CMS only and has no alarms or errors in the PBX. The port on the switch is 100/Full and shows no errors. The network is not showing any errors.

One piece of information I left out was that we are using CMS rrv12cc.c

ANY help would be greately appreciated.

Thanks.

- Stinney

Favorite all too common vendor responses: "We've never seen this issue before." AND "No one's ever wanted to use it like that before.

 

[BIS]

Stinney,
not much help, but see if there is anything in the logs.

Run the command

./getlog

in the /cms/toolsbin directory. Especially in spi_log, spi_err and spi_lnk

 

[Stinney]

Wow, this is an awesome command. However, I've never used it before.

Unfortunately, I don't know what I'm looking at when I read some of the reports. This is the output from the spi_err_1 there is no spi_err only spi_err_1 to spi_err_8:

Fri Mar 14 23:26:43 [57] PBX DATAX 23:27:00 03/14/08 00028 calls ======
Fri Mar 14 23:27:43 [58] PBX DATAX 23:28:00 03/14/08 00029 calls ======
Fri Mar 14 23:28:43 [59] PBX DATAX 23:29:00 03/14/08 00030 calls ======
Fri Mar 14 23:29:43 [58] PBX DATAX 23:30:00 03/14/08 00031 calls ======
Fri Mar 14 23:30:43 [58] PBX DATAX 23:31:00 03/14/08 00002 calls ======

It doesn't report any errors on the days we had an issue.

The spi_lnk and spi_log files go back to 2004, nothting current.



- Stinney

Favorite all too common vendor responses: "We've never seen this issue before." AND "No one's ever wanted to use it like that before.

 

[Reynolds723]

Hi Stinney,

I had a very similar issue to this a few months back, it turned out that the IPSI in the PN the CLAN card used for CMS was in was interchanging, we have S8720's, G650's in an IP connect configuration and CM 3.1.4. The cisco switches used for core networks A & B had dropped their config and were auto negotiating. Every time there were contention issues the IPSI's interchanged causing the SPI session to drop.

However if you're not seeing something similar to the below in the spi error logs then maybe not your issue.

Fri Nov 10 12:15:05 [05] SPI GOOD_BYE 15
Fri Nov 10 12:15:05 [05] PBX 362 calls ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX call ignored
Fri Nov 10 12:15:05 [05] PBX QUIESCENT
Fri Nov 10 12:18:44 [00] SPI STARTED
Fri Nov 10 12:18:47 [00] TCP CLIENT ATTACHED
Fri Nov 10 12:18:47 [00] SESS pmachine starting state QUIESCENT
Fri Nov 10 12:18:47 [00] TCP OPERATIONAL
Fri Nov 10 12:18:47 [00] PBX WSA
Fri Nov 10 12:18:47 [00] SESS DATAX
Fri Nov 10 12:18:47 [00] PBX RTCS
Fri Nov 10 12:18:47 [00] PBX RTCS
Fri Nov 10 12:18:52 [47] PBX requesting full translations
Fri Nov 10 12:18:52 [47] PBX PUMP-UP
Fri Nov 10 12:18:52 [47] PBX PUMP-UP 12:18:47 11/10/06 00000 calls ======
Fri Nov 10 12:18:52 [52] PBX PUMP-UP 12:18:52 11/10/06 00000 calls ======
Fri Nov 10 12:18:52 [52] PBX PUMP-UP 12:18:52 11/10/06 00000 calls ======
Fri Nov 10 12:18:54 [52] PBX PUMP-UP 12:18:52 11/10/06 00000 calls ======
Fri Nov 10 12:19:02 [52] PBX PUMP-UP 12:19:02 11/10/06 00000 calls ======
Fri Nov 10 12:19:04 [02] PBX PUMP-UP 12:19:02 11/10/06 00000 calls ======
Fri Nov 10 12:19:12 [02] PBX PUMP-UP 12:19:12 11/10/06 00000 calls ======
Fri Nov 10 12:19:13 [12] PBX PUMP-UP 12:19:12 11/10/06 00000 calls ======
Fri Nov 10 12:19:15 [13] PBX PUMP-UP 12:19:13 11/10/06 00000 calls ======
Fri Nov 10 12:19:16 [14] PBX PUMP-UP 12:19:14 11/10/06 00000 calls ======
Fri Nov 10 12:19:17 [15] PBX PUMP-UP 12:19:15 11/10/06 00000 calls ======
Fri Nov 10 12:19:18 [16] PBX PUMP-UP 12:19:16 11/10/06 00000 calls ======
Fri Nov 10 12:19:19 [17] PBX PUMP-UP 12:19:17 11/10/06 00000 calls ======
Fri Nov 10 12:19:21 [18] PBX PUMP-UP 12:19:18 11/10/06 00000 calls ======
Fri Nov 10 12:19:29 [28] PBX PUMP-UP 12:19:28 11/10/06 00000 calls ======
Fri Nov 10 12:19:31 [28] PBX PUMP-UP 12:19:28 11/10/06 00000 calls ======
Fri Nov 10 12:19:40 [38] PBX PUMP-UP 12:19:38 11/10/06 00000 calls ======
Fri Nov 10 12:19:41 [39] PBX PUMP-UP 12:19:39 11/10/06 00000 calls ======
Fri Nov 10 12:19:42 [40] PBX PUMP-UP 12:19:40 11/10/06 00000 calls ======
Fri Nov 10 12:19:43 [41] PBX PUMP-UP 12:19:41 11/10/06 00000 calls ======
Fri Nov 10 12:19:40 [38] PBX PUMP-UP 12:19:38 11/10/06 00000 calls ======
Fri Nov 10 12:19:41 [39] PBX PUMP-UP 12:19:39 11/10/06 00000 calls ======
Fri Nov 10 12:19:42 [40] PBX PUMP-UP 12:19:40 11/10/06 00000 calls ======
Fri Nov 10 12:19:43 [41] PBX PUMP-UP 12:19:41 11/10/06 00000 calls ======
Fri Nov 10 12:19:43 [42] PBX DATAX
Fri Nov 10 12:19:43 [42] PBX link was down 00 secs

You could also see the ipsi interchange at the same time in the history in ASA.

Hope that helps mate.

Regards

Craig

 

[telecomturner]

What firmware version on the IPSI's. We are at 40 and that seems to be very stable. (I know we are a few versions behind.)

 

[Stinney]

We're on 40 as well.

Avaya support has looked at "everything" and says the issue is on our network. And of course, we see no issues.

- Stinney

Favorite all too common vendor responses: "We've never seen this issue before." AND "No one's ever wanted to use it like that before.

 

[Reynolds723]

It wasn't the firmware version of the IPSI's that was the issue, we were on the latest firmware at the time too, it was the Cisco switches in the Avaya core network that caused the issue, all our IPSI's accross the entire enterprise were up and down like yo-yo's because someone had forgot to configure the Cisco's. Once we set them up and hard coded the ports to 100 full n there were no further occurrences.

Cheers

Craig

 

[Stinney]

Software interchange request issue is a bug in 3.0.x

We upgraded to 3.1.4 and it hasn't happened since.

- Stinney

Favorite all too common vendor responses: "We've never seen this issue before." AND "No one's ever wanted to use it like that before.