Smart questions
Smart answers
Smart people
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Member Login




Remember Me
Forgot Password?
Join Us!

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips now!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

Join Tek-Tips
*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.
Jobs from Indeed

Link To This Forum!

Partner Button
Add Stickiness To Your Site By Linking To This Professionally Managed Technical Forum.
Just copy and paste the
code below into your site.

mfoc (MIS) (OP)
22 Sep 04 10:15
So, I've gone round and round with Qwest now.  They say the line looks great, but yet for the past few days, ny Internet T1 has been dropping.  There are a ton of input errors and CRC's incrementing (this may have been happening before the line dropped for the first time), and when the line drops, obviously, I have an alarm on my WIC.  I still have the CD as green, but the alarm comes on.  I've swapped the WIC out and verified the settings with QWEST to make sure it's configured properly.  Nothing has changed recently - just started happening Tuesday.  When the line drops, it states "line protocol is down."

Here're a few items to look at:

sh int serial 0/0
Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: connected to QwestVPN
  Internet address is 65.117.41.222/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 11/255, rxload 44/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:14:05
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/12/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
  5 minute input rate 269000 bits/sec, 43 packets/sec
  5 minute output rate 69000 bits/sec, 38 packets/sec
     39652 packets input, 29786817 bytes, 0 no buffer
     Received 84 broadcasts, 0 runts, 0 giants, 0 throttles
     22558 input errors, 1213 CRC, 21297 frame, 0 overrun, 0 ignored, 48 abort
     35543 packets output, 9079608 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

sh service-module serial 0/0
Module type is T1/fractional
    Hardware revision is 0.88, Software revision is 0.2,
    Image checksum is 0x73D70058, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last user loopback performed:
    dte loopback
    duration 00:02:55
Last module self-test (done at startup): Passed
Last clearing of alarm counters 00:15:22
    loss of signal        :    0,
    loss of frame         :    0,
    AIS alarm             :    0,
    Remote alarm          :    0,
    Module access errors  :    0,
Total Data (last 67 15 minute intervals):
    1 Line Code Violations, 13795 Path Code Violations
    1948 Slip Secs, 4818 Fr Loss Secs, 1 Line Err Secs, 640 Degraded Mins
    24792 Errored Secs, 24674 Bursty Err Secs, 4744 Severely Err Secs, 74 Unavai
l Secs
Data in current interval (772 seconds elapsed):
    0 Line Code Violations, 255 Path Code Violations
    8 Slip Secs, 7 Fr Loss Secs, 0 Line Err Secs, 5 Degraded Mins
    110 Errored Secs, 110 Bursty Err Secs, 7 Severely Err Secs, 0 Unavail Secs

Qwest has pretty much told me that my router is bad and almost refuses to troubleshoot with me any more until I replace it.  I don't buy it.  Tha last three times I've had problems with this line, it's been because of their edge router.  The last time it went down, they told me that I had too much internet traffic and it was causing their router to crash....?????  They are supposed to be sending a local telco tech out this morning to see if they can run some head-to-head.
robertjo24 (MIS)
22 Sep 04 11:16
Issues with stubborn telco’s are the worst. I would insist on a head to head test from their smart jack/LIU at your location back to the provider. I would not rely upon tests performed remotely by them looping the smart jack/LIU, because there are some components in their equipment past the point of loop in the card. We have had issues with defective card slots in the NIU, where the telco has looped the smart jack and tested clean, yet we were still incrementing errors.
Just as a point of interest, have you checked the cabling between the router interface and their jack? It could cause some problems if the cable got pinched or attacked by rodents.
You never know…
mfoc (MIS) (OP)
22 Sep 04 11:25
Just a little more info.  This T1 comes off a DS3 in our server room.  Fiber to the DS3 cabinet - coax from the FLM 150 to the mux - 6' patch from the mux breakout to the WIC. There's some confusion as to who owns the DS3 equipment.  Our service manager tells us one thing, but the records indicate otherwise.  The only equipment that's ours should be the 6' patch and in.  I checked the cable and it was OK.  It'll be interesting to see what they see off of that line.
NTPeave (MIS)
23 Sep 04 3:38
Have you heard of the "Output Interpreter" from cisco.com? It's quite handy for troubleshooting some problems. It's reachable from the home page after you have logged in if you have an appropriate login. Ask your local Cisco folks for one - it shouldn't be particularly sensitive/hard to get access to. It works by just pasting in output from most of the 'show' commands, and then it gives recommendations of problems/fixes. As an example, here's what it says about the two show commands you included in your post:

SHOW INTERFACE SERIAL NOTIFICATIONS (if any)

Interface Serial0/0 (up/up)
  INFO: Please click the link below to have the IP SUBNET CALCULATOR automatically
  calculate the supported range of IP addresses for the configured network and subnet
  mask.
  'Ip Subnet Calculator for 65.117.41.222/30'
  
  WARNING: This interface has received a high number of packets with incorrect
  CRCs (corrupted data) (> 0.1%).
  Problems that may cause this symptom include:
  a. Noisy serial line
  b. Serial cable is too long or cable from the CSU/DSU to the router is not
     shielded
  c. SCTE mode is not enabled on the DSU
  d. The CSU line clock is incorrectly configured
  e. A Ones density problem on the link (incorrect framing or coding
     specification), exists
  f. Verify the queuing strategies are the same on both ends of the link.
  TRY THIS:
  1. Ensure that the line is clean enough for transmission requirements. Shield
     the cable if necessary.
  2. Make sure the cable is within the recommended length (no more than 50 feet
     [15.24 meters], or 25 feet [7.62 meters] for the link).
  3. Ensure that all devices are properly configured for a common line clock.
     Set serial clock transmit external (SCTE) on the local and remote DSU. If
     you are attempting serial connections at speeds greater than 64 kbps with
     a CSU/DSU that does not support (SCTE), you might have to invert the
     transmit clock on the router. Inverting the transmit clock compensates
     for phase-shifts between the data and clock signals.
  4. Make certain that the local and remote CSU/DSU are configured for the
     same framing and coding scheme as that used by the leased-line or other
     carrier service (for example, ESF/B8ZS).
  5. Contact your leased-line or other carrier service and have them perform
     integrity tests on the line.
  
  WARNING: This interface has received a high number of packets (more than 1% total
  input interface traffic) with framing errors.
  A framing error occurs when a packet does not end on an 8-bit byte boundary.
  Problems that may cause this symptom include:
  a. Noisy serial line
  b. Improperly designed cable; serial cable is too long; the cable from the
     CSU or DSU to the router is not shielded
  c. SCTE (serial clock transmit external) mode is not enabled on the DSU; the
     CSU line clock is incorrectly configured; one of the clocks is configured
     for local clocking
  d. There is a Ones density problem on the link (incorrect framing or
     coding specification)
  TRY THIS:
  1. Ensure that the line is clean enough for transmission requirements. Shield
     the cable if necessary.
  2. Make sure the cable is within the recommended length (no more than 50 feet
  
     [15.24 meters], or 25 feet [7.62 meters] for the link).
  3. Ensure all devices are properly configured for a common line clock. Set
     serial clock transmit external (SCTE) on the local and remote DSU. If you
     are attempting serial connections at speeds greater than 64 kbps with a
     CSU/DSU that does not support (SCTE), you might have to invert the
     transmit clock on the router. Inverting the transmit clock compensates for
     phase-shifts between the data and clock signals.
  4. Make certain that the local and remote CSU/DSU are configured for the same
     framing and coding scheme as that used by the leased-line or other carrier
     service (for example, ESF/B8ZS).
  5. Contact your leased-line or other carrier service and have them perform
     integrity tests on the line.
  
REFERENCE: For more information on Serial Lines, see:
  Troubleshooting Serial Line Problems
  Configuring Serial Interfaces
  Troubleshooting Serial Lines
  Loopback Tests for T1/56K Lines

REFERENCE: Command Reference - 'show interface serial'

SHOW SERVICE-MODULE NOTIFICATIONS (if any)

For the 'T1/fractional' interface:
  INFO: With Extended Super Frame, the remote alarm condition is signaled out
  of band, in the Facility Data Link. Thus with ESF, it is recommended to enable
  remote alarms with the 'service-module t1 framing esf' command.
  
NOTE: Although the output from the 'show service-module serial [number]' command
can be used to troubleshoot some problems, it cannot assist with all problems. It
should be used in conjunction with other commands such as 'show interface serial',
'show isdn status' and 'show isdn service' (if ISDN is applicable). Submit the
output from these commands to Output Interpreter for an expert analysis.

REFERENCE: For more information, see: Configuring Serial Interfaces for CSU/DSU
Service Modules.

NTPeave (MIS)
23 Sep 04 3:49
Have you heard of the "Output Interpreter" from www.cisco.com? It's quite handy for troubleshooting some problems. It's reachable from the home page after you have logged in if you have an appropriate login. Ask your local Cisco folks for one - it shouldn't be particularly sensitive/hard to get access to. It works in one of several cool ways by just pasting in output from most of the 'show' commands, and then it gives recommendations of problems/fixes. As an example, here's what it says about the two show commands you included in your post:

SHOW INTERFACE SERIAL NOTIFICATIONS (if any)

Interface Serial0/0 (up/up)
  INFO: Please click the link below to have the IP SUBNET CALCULATOR automatically
  calculate the supported range of IP addresses for the configured network and subnet
  mask.
  'Ip Subnet Calculator for 65.117.41.222/30'
  
  WARNING: This interface has received a high number of packets with incorrect
  CRCs (corrupted data) (> 0.1%).
  Problems that may cause this symptom include:
  a. Noisy serial line
  b. Serial cable is too long or cable from the CSU/DSU to the router is not
     shielded
  c. SCTE mode is not enabled on the DSU
  d. The CSU line clock is incorrectly configured
  e. A Ones density problem on the link (incorrect framing or coding
     specification), exists
  f. Verify the queuing strategies are the same on both ends of the link.
  TRY THIS:
  1. Ensure that the line is clean enough for transmission requirements. Shield
     the cable if necessary.
  2. Make sure the cable is within the recommended length (no more than 50 feet
     [15.24 meters], or 25 feet [7.62 meters] for the link).
  3. Ensure that all devices are properly configured for a common line clock.
     Set serial clock transmit external (SCTE) on the local and remote DSU. If
     you are attempting serial connections at speeds greater than 64 kbps with
     a CSU/DSU that does not support (SCTE), you might have to invert the
     transmit clock on the router. Inverting the transmit clock compensates
     for phase-shifts between the data and clock signals.
  4. Make certain that the local and remote CSU/DSU are configured for the
     same framing and coding scheme as that used by the leased-line or other
     carrier service (for example, ESF/B8ZS).
  5. Contact your leased-line or other carrier service and have them perform
     integrity tests on the line.
  
  WARNING: This interface has received a high number of packets (more than 1% total
  input interface traffic) with framing errors.
  A framing error occurs when a packet does not end on an 8-bit byte boundary.
  Problems that may cause this symptom include:
  a. Noisy serial line
  b. Improperly designed cable; serial cable is too long; the cable from the
     CSU or DSU to the router is not shielded
  c. SCTE (serial clock transmit external) mode is not enabled on the DSU; the
     CSU line clock is incorrectly configured; one of the clocks is configured
     for local clocking
  d. There is a Ones density problem on the link (incorrect framing or
     coding specification)
  TRY THIS:
  1. Ensure that the line is clean enough for transmission requirements. Shield
     the cable if necessary.
  2. Make sure the cable is within the recommended length (no more than 50 feet
  
     [15.24 meters], or 25 feet [7.62 meters] for the link).
  3. Ensure all devices are properly configured for a common line clock. Set
     serial clock transmit external (SCTE) on the local and remote DSU. If you
     are attempting serial connections at speeds greater than 64 kbps with a
     CSU/DSU that does not support (SCTE), you might have to invert the
     transmit clock on the router. Inverting the transmit clock compensates for
     phase-shifts between the data and clock signals.
  4. Make certain that the local and remote CSU/DSU are configured for the same
     framing and coding scheme as that used by the leased-line or other carrier
     service (for example, ESF/B8ZS).
  5. Contact your leased-line or other carrier service and have them perform
     integrity tests on the line.
  
REFERENCE: For more information on Serial Lines, see:
  Troubleshooting Serial Line Problems
  Configuring Serial Interfaces
  Troubleshooting Serial Lines
  Loopback Tests for T1/56K Lines

REFERENCE: Command Reference - 'show interface serial'

SHOW SERVICE-MODULE NOTIFICATIONS (if any)

For the 'T1/fractional' interface:
  INFO: With Extended Super Frame, the remote alarm condition is signaled out
  of band, in the Facility Data Link. Thus with ESF, it is recommended to enable
  remote alarms with the 'service-module t1 framing esf' command.
  
NOTE: Although the output from the 'show service-module serial [number]' command
can be used to troubleshoot some problems, it cannot assist with all problems. It
should be used in conjunction with other commands such as 'show interface serial',
'show isdn status' and 'show isdn service' (if ISDN is applicable). Submit the
output from these commands to Output Interpreter for an expert analysis.

REFERENCE: For more information, see: Configuring Serial Interfaces for CSU/DSU
Service Modules.

NTPeave (MIS)
23 Sep 04 3:59
Sorry! Trigger happy on the 'submit post' button...
As penance, I offer a better description of what cisco.com claims its output interpreter will do for you. You'll still need a login though...

"Did you know that Output Interpreter supports the following functionality?

Show command outputs from your Router, Switch or PIX Firewall. Reference the list of supported show commands for details.

Error Messages generated by your Router, Switch or PIX Firewall. Simply copy and paste your Error or Log Messages from your Router, Switch or PIX Firewall into the Output Interpreter.

Decodes and analyzes your Router or Switch's stack trace for any possible bugs. Copy and paste the "show version" command output followed by Traceback or Stack Trace and Alignment data.

Is able to convert the 'apply', 'conduit' and 'outbound' statements of your PIX Firewall to equivalent 'access-list' statements. Copy and paste "show tech-support" or "write terminal" command output of your PIX Firewall.

Decodes and analyzes Configuration Register. Copy and paste the "show version" or "show tech-support" command output into the Output Interpreter "
mfoc (MIS) (OP)
23 Sep 04 8:51
NTPeave - That's an interesting tool.  I have a login at cisco.com, so I mght have to play with that a little more.

Well, I think we fixed the problem.  The local tester came out and took a look at the line.  Of course, the physical layer looked great (as usual), but I was still incrementing errors.  We determined that the problem indicated some sort of clock source issue.  My router is set to line, and their edge router was set to internal.  This is the way they claim it should be in order to operate normally.  Well, they switched their router line also and it cleared all the errors right up.  I haven't had a single error since they changed it yesterday at about 4PM.

So, here's the funny thing... They are now confused as to how each router is getting it's clocking signal.  Apparently, there's no equipment in between that could actually provide this.  The tech that was on-site said my Widebank 28 mux had the capability, but it was set to internal, which he said actually meant line according to the manual on the unit....?????  Are these guys professionals?  I really wonder sometimes.  Before the tech came out, it had reached the point of them not troubleshooting the problem any more until we REPLACED OUR ROUTER!!!

You know, the past three times now that I've had a problem on this line, I waste hours and hours of time having to prove to Qwest that my equipment is good.  Each time, it turns out to be something on their side.  It's getting old.  Time to look for a redundant solution.

Thanks
marsd (IS/IT--Management)
23 Sep 04 12:07
It's not unusual for this sh!t to happen.
I worked for three years with an smds circuit that
was very 'weird' and learned to diagnose DCE issues.
I had one bad cable and two bad config errors. The bad
cable isue was mine and the bad config issues were theirs.
If you look at the US IT marketb you will see why this
sh!t happens. The senior guys are fired and new guys are
brought in at a certain salary level. Set your clockrate
by that.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Back To Forum

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close