Use Linux CLI Main and ESS/LSP log files /var/log/ecs
MG registration issues
egrep -R "MG=#" /var/log/ecs/2016-072*
Use IP address for media-gateway or Survivable processor
tech@cm-srv1> egrep -i 172.19.40.200 2013-0819*
2013-0819-052630.log:20130819:072058477:153667264:capro(4095):MED:[ DENYEVT ERR event=2047 d1=172.19.40.200 d2=d78c0943 ]
2013-0819-052630.log:20130819:072259092:153667363:capro(4095):MED:[mgRegister:IPV4addr 172.19.40.200]
2013-0819-052630.log:20130819:072712277:153668009:capro(4095):MED:[ConnClosed: MG=#143 disconnected: socket closure, moved to link-bounce state/: near_ipaddr = 172.18.0.50, far_ipaddr = 172.19.40.200 actIntf 0]
tech@-srv1> egrep -i mgreg 2013-0819*
2013-0819-052630.log:20130819:072059093:153667268:capro(4095):MED:[mgRegister MG=#143 First attempt to return to main from LSP]
2013-0819-052630.log:20130819:072259091:153667362:capro(4095):MED:[mgRegister:success MG#=143, MG_G450, Ipif=0x1, SerialNum=10N551705532, HW Vintage=1, FW ver=30.12.1]
2013-0819-052630.log:20130819:072259092:153667363:capro(4095):MED:[mgRegister:IPV4addr 172.19.40.200]
2013-0819-052630.log:20130819:072259092:153667364:capro(4095):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-052630.log:20130819:072259092:153667365:capro(4095):MED:[mgRegister:success MG Restart type=MG_RETURNS_FRM_LSP, action=LINK_UP_CNPR]
2013-0819-094306.log:20130819:101616529:153709867:capro(4095):MED:[mgRegister MG=#131 First attempt to return to main from LSP]
2013-0819-101718.log:20130819:101816533:153713138:capro(4095):MED:[mgRegister:success MG#=131, MG_G430, Ipif=0x1, SerialNum=10IS27382377, HW Vintage=1, FW ver=31.22.0]
2013-0819-101718.log:20130819:101816533:153713139:capro(4095):MED:[mgRegister:IPV4addr 172.19.105.200]
2013-0819-101718.log:20130819:101816533:153713140:capro(4095):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-101718.log:20130819:101816533:153713141:capro(4095):MED:[mgRegister:success MG Restart type=MG_RETURNS_FRM_LSP, action=LINK_UP_CNPR]
2013-0819-113312.log:20130819:115548830:153753707:capro(24515):MED:[mgRegister:success MG#=131, MG_G430, Ipif=0x1, SerialNum=10IS27382377, HW Vintage=1, FW ver=31.22.0]
2013-0819-113312.log:20130819:115548830:153753708:capro(24515):MED:[mgRegister:IPV4addr 172.19.105.200]
2013-0819-113312.log:20130819:115548830:153753709:capro(24515):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-113312.log:20130819:115548830:153753710:capro(24515):MED:[mgRegister:success MG Restart type=MG_DISC_WARM_BOOT, action=LINK_UP_CNPR]
2013-0819-113312.log:20130819:115548837:153753711:capro(24515):MED:[mgRegister:success MG#=132, MG_G430, Ipif=0x1, SerialNum=10IS22391933, HW Vintage=1, FW ver=30.12.1]
2013-0819-113312.log:20130819:115548837:153753712:capro(24515):MED:[mgRegister:IPV4addr 172.19.110.200]
2013-0819-113312.log:20130819:115548837:153753713:capro(24515):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-113312.log:20130819:115548837:153753714:capro(24515):MED:[mgRegister:success MG Restart type=MG_DISC_WARM_BOOT, action=LINK_UP_CNPR]
2013-0819-113312.log:20130819:115549176:153753735:capro(24515):MED:[mgRegister:success MG#=1, MG_G450, Ipif=0x1, SerialNum=10IS08362153, HW Vintage=1, FW ver=30.12.1]
2013-0819-113312.log:20130819:115549176:153753736:capro(24515):MED:[mgRegister:IPV4addr 172.18.0.60]
2013-0819-113312.log:20130819:115549176:153753737:capro(24515):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-113312.log:20130819:115549176:153753738:capro(24515):MED:[mgRegister:success MG Restart type=MG_DISC_WARM_BOOT, action=LINK_UP_CNPR]
2013-0819-113312.log:20130819:115549177:153753739:capro(24515):MED:[mgRegister:success MG#=2, MG_G450, Ipif=0x1, SerialNum=10IS07359357, HW Vintage=1, FW ver=30.12.1]
2013-0819-113312.log:20130819:115549177:153753740:capro(24515):MED:[mgRegister:IPV4addr 172.18.0.61]
2013-0819-113312.log:20130819:115549177:153753741:capro(24515):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-113312.log:20130819:115549177:153753742:capro(24515):MED:[mgRegister:success MG Restart type=MG_DISC_WARM_BOOT, action=LINK_UP_CNPR]
2013-0819-113312.log:20130819:115549178:153753743:capro(24515):MED:[mgRegister:success MG#=101, MG_G450, Ipif=0x1, SerialNum=09IS46285015, HW Vintage=1, FW ver=30.12.1]
2013-0819-113312.log:20130819:115549178:153753744:capro(24515):MED:[mgRegister:IPV4addr 172.18.0.62]
2013-0819-113312.log:20130819:115549178:153753745:capro(24515):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-113312.log:20130819:115549178:153753746:capro(24515):MED:[mgRegister:success MG Restart type=MG_DISC_WARM_BOOT, action=LINK_UP_CNPR]
2013-0819-113312.log:20130819:115549258:153753753:capro(24515):MED:[mgRegister:success MG#=130, MG_G430, Ipif=0x1, SerialNum=10IS07359657, HW Vintage=1, FW ver=31.22.0]
2013-0819-113312.log:20130819:115549258:153753754:capro(24515):MED:[mgRegister:IPV4addr 172.19.28.200]
2013-0819-113312.log:20130819:115549258:153753755:capro(24515):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-113312.log:20130819:115549258:153753756:capro(24515):MED:[mgRegister:success MG Restart type=MG_DISC_WARM_BOOT, action=LINK_UP_CNPR]
2013-0819-113312.log:20130819:115549349:153753760:capro(24515):MED:[mgRegister:success MG#=120, MG_G450, Ipif=0x1, SerialNum=09IS46288850, HW Vintage=1, FW ver=30.12.1]
2013-0819-113312.log:20130819:115549349:153753761:capro(24515):MED:[mgRegister:IPV4addr 172.19.2.200]
2013-0819-113312.log:20130819:115549349:153753762:capro(24515):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-113312.log:20130819:115549349:153753763:capro(24515):MED:[mgRegister:success MG Restart type=MG_DISC_WARM_BOOT, action=LINK_UP_CNPR]
2013-0819-113312.log:20130819:115549588:153753771:capro(24515):MED:[mgRegister:success MG#=121, MG_G450, Ipif=0x1, SerialNum=10IS09366406, HW Vintage=1, FW ver=31.22.0]
2013-0819-113312.log:20130819:115549588:153753772:capro(24515):MED:[mgRegister:IPV4addr 172.19.37.200]
2013-0819-113312.log:20130819:115549588:153753773:capro(24515):MED:[mgRegister:IPV6addr 0.0.0.0]
2013-0819-113312.log:20130819:115549588:153753774:capro(24515):MED:[mgRegister:success MG Restart type=MG_DISC_WARM_BOOT, action=LINK_UP_CNPR]
2013-0819-113312.log:20130819:115549698:153753776:capro(24515):MED:[mgRegister:success MG#=139, MG_G450, Ipif=0x1, SerialNum=10N552706484, HW Vintage=1, FW ver=30.12.1]
2013-0819-113312.log:20130819:115549698:153753777:capro(24515):MED:[mgRegister:IPV4addr 172.19.49.200]
2013-0819-113312.log:20130819:115549698:153753778:capro(24515):MED:[mgRegister:IPV6addr 0.0.0.0]
IPSI network issues
egrep -i pcd.*sanity /var/log/ecs/2016-072* | wc
egrep -i pcd.*sanity /var/log/ecs/2016-072* | egrep too.*many | wc
egrep -i pcd.*sanity /var/log/ecs/2016-072* | egrep too.*many
tech@8800B> egrep -i pcd.*sanity /var/log/ecs/2016-072[78]*
/var/log/ecs/2016-0727-003023.log:20160727:055034651:1469899:pcd(8004):MED:[[7:0] checkSlot: sanity failure (1)]
/var/log/ecs/2016-0727-003023.log:20160727:055035651:1469903:pcd(8004):MED:[[7:0] checkSlot: sanity failure (2)]
/var/log/ecs/2016-0727-003023.log:20160727:055036651:1469904:pcd(8004):MED:[[7:0] checkSlot: sanity failure (3)]
/var/log/ecs/2016-0727-003023.log:20160727:055037651:1469905:pcd(8004):MED:[[7:0] checkSlot: sanity failure (4)]
/var/log/ecs/2016-0727-003023.log:20160727:055038647:1469906:pcd(8004):MED:[[7:0] checkSlot: sanity failure (5)]
/var/log/ecs/2016-0727-003023.log:20160727:055039647:1469909:pcd(8004):MED:[[7:0] checkSlot: sanity failure (6)]
/var/log/ecs/2016-0727-003023.log:20160727:055040647:1469910:pcd(8004):MED:[[7:0] checkSlot: sanity failure (7)]
/var/log/ecs/2016-0727-003023.log:20160727:055041647:1469913:pcd(8004):MED:[[7:0] checkSlot: sanity failure (8)]
/var/log/ecs/2016-0727-003023.log:20160727:055042647:1469914:pcd(8004):MED:[[7:0] checkSlot: sanity failure (9)]
/var/log/ecs/2016-0727-003023.log:20160727:055043647:1469915:pcd(8004):MED:[[7:0] checkSlot: sanity failure (10)]
/var/log/ecs/2016-0727-003023.log:20160727:055044647:1469917:pcd(8004):MED:[[7:0] checkSlot: sanity failure (11)]
/var/log/ecs/2016-0727-003023.log:20160727:055045651:1469918:pcd(8004):MED:[[7:0] checkSlot: sanity failure (12)]
/var/log/ecs/2016-0727-003023.log:20160727:055046651:1469929:pcd(8004):MED:[[7:0] checkSlot: sanity failure (13)]
/var/log/ecs/2016-0727-003023.log:20160727:055047651:1469931:pcd(8004):MED:[[7:0] checkSlot: sanity failure (14)]
/var/log/ecs/2016-0727-003023.log:20160727:055048651:1469932:pcd(8004):MED:[[7:0] checkSlot: too many sanity failures (15)]
/var/log/ecs/2016-0727-061120.log:20160727:080720653:1472116:pcd(8004):MED:[[7:0] checkSlot: sanity failure (1)]
/var/log/ecs/2016-0727-061120.log:20160727:080721653:1472119:pcd(8004):MED:[[7:0] checkSlot: sanity failure (2)]
/var/log/ecs/2016-0727-061120.log:20160727:080722653:1472120:pcd(8004):MED:[[7:0] checkSlot: sanity failure (3)]
/var/log/ecs/2016-0727-061120.log:20160727:080723653:1472121:pcd(8004):MED:[[7:0] checkSlot: sanity failure (4)]
/var/log/ecs/2016-0727-061120.log:20160727:080724653:1472122:pcd(8004):MED:[[7:0] checkSlot: sanity failure (5)]
/var/log/ecs/2016-0727-061120.log:20160727:080725649:1472123:pcd(8004):MED:[[7:0] checkSlot: sanity failure (6)]
/var/log/ecs/2016-0727-061120.log:20160727:080726649:1472124:pcd(8004):MED:[[7:0] checkSlot: sanity failure (7)]
/var/log/ecs/2016-0727-061120.log:20160727:080727649:1472125:pcd(8004):MED:[[7:0] checkSlot: sanity failure (8)]
/var/log/ecs/2016-0727-061120.log:20160727:080728649:1472126:pcd(8004):MED:[[7:0] checkSlot: sanity failure (9)]
/var/log/ecs/2016-0727-061120.log:20160727:080729648:1472127:pcd(8004):MED:[[7:0] checkSlot: sanity failure (10)]
/var/log/ecs/2016-0727-061120.log:20160727:080730648:1472128:pcd(8004):MED:[[7:0] checkSlot: sanity failure (11)]
/var/log/ecs/2016-0727-061120.log:20160727:080731648:1472129:pcd(8004):MED:[[7:0] checkSlot: sanity failure (12)]
/var/log/ecs/2016-0727-061120.log:20160727:080732648:1472130:pcd(8004):MED:[[7:0] checkSlot: sanity failure (13)]
/var/log/ecs/2016-0727-061120.log:20160727:080733648:1472131:pcd(8004):MED:[[7:0] checkSlot: sanity failure (14)]
/var/log/ecs/2016-0727-061120.log:20160727:080734648:1472132:pcd(8004):MED:[[7:0] checkSlot: too many sanity failures (15)]
/var/log/ecs/2016-0727-145147.log:20160727:201931650:1484129:pcd(8004):MED:[[7:0] checkSlot: sanity failure (1)]
tech@ESS> almdisplay -res
ID MO Source On Bd Lvl Ack Date
1 CLR CLR CLR Y Wed Jul 27 10:19:14 CDT 2016
2 CLR CLR CLR Y Wed Jul 27 08:07:34 CDT 2016
3 ESS CL 000 n MIN Y Wed Jul 27 08:27:54 CDT 2016
4 CLR CLR CLR Y Tue Jul 26 02:31:32 CDT 2016
5 ESS CL 000 n MAJ Y Wed Jul 27 05:55:36 CDT 2016
6 LIC-ERR n MAJ Y Wed Jul 27 05:55:36 CDT 2016
7 CLR CLR CLR Y Sun Jul 24 19:23:21 CDT 2016
8 ESS CL 000 n MIN Y Tue Jul 26 02:16:52 CDT 2016
9 ESS CL 000 n MIN Y Sun Jul 24 18:13:31 CDT 2016
10 A GAM 2 WRN N Sun Jul 24 08:55:06 CDT 2016
11 CLR CLR CLR N Sun Jul 24 17:58:54 CDT 2016
12 CLR CLR CLR Y Sun Jul 24 03:12:30 CDT 2016
13 ESS CL 000 n MAJ Y Sun Jul 24 07:52:11 CDT 2016
14 LIC-ERR n MAJ Y Sun Jul 24 07:52:11 CDT 2016
15 CLR CLR CLR Y Fri Jul 22 02:38:03 CDT 2016
16 ESS CL 000 n MIN Y Sun Jul 24 02:17:40 CDT 2016
17 ESS CL 000 n MIN Y Fri Jul 22 02:18:33 CDT 2016
18 CLR CLR CLR Y Wed Jul 20 22:07:46 CDT 2016
19 SP-REG-S n MIN N Thu Jul 21 06:11:21 CDT 2016
20 CLR CLR CLR Y Mon Jul 18 03:59:31 CDT 2016
21 ESS CL 000 n MAJ Y Wed Jul 20 21:09:08 CDT 2016
22 LIC-ERR n MAJ Y Wed Jul 20 21:09:08 CDT 2016
23 SP-REG-S n MIN Y Wed Jul 20 22:01:53 CDT 2016
24 CLR CLR CLR Y Sat Jul 16 03:34:52 CDT 2016
25 ESS CL 000 n MIN Y Mon Jul 18 02:16:00 CDT 2016
26 CLR CLR CLR Y Thu Jul 14 17:04:29 CDT 2016
27 ESS CL 000 n MIN Y Sat Jul 16 02:16:02 CDT 2016
28 CLR CLR CLR Y Thu Jul 14 03:04:20 CDT 2016
29 ESS CL 000 n MAJ Y Thu Jul 14 11:59:38 CDT 2016
30 LIC-ERR n MAJ Y Thu Jul 14 11:59:38 CDT 2016
31 ESS CL 000 n MIN Y Thu Jul 14 02:16:00 CDT 2016
tech@LSP> almdisplay -res
ID MO Source On Bd Lvl Ack Date
1 CLR CLR CLR Y Wed Nov 04 06:47:46 MST 2015
2 LIC-ERR n MAJ Y Wed Nov 04 05:33:21 MST 2015
3 SP-REG-S n MIN Y Wed Nov 04 05:29:59 MST 2015
tech@LSP2> /opt/ecs/bin/almdisplay -res
ID MO Source On Bd Lvl Ack Date
1 CLR CLR CLR Y Wed Oct 22 01:42:57 EDT 2014
2 SP-REG-S n MIN N Wed Oct 22 01:51:58 EDT 2014
3 CLR CLR CLR Y Wed Oct 22 01:06:05 EDT 2014
4 LIC-ERR n MAJ Y Wed Oct 22 01:19:16 EDT 2014
5 SP-REG-S n MIN Y Wed Oct 22 01:31:02 EDT 2014
6 SP-REG-S n MIN Y Wed Oct 22 01:19:01 EDT 2014
7 LIC-ERR n MAJ Y Wed Oct 22 00:42:03 EDT 2014
8 SP-REG-S n MIN Y Wed Oct 22 00:41:43 EDT 2014