Physical Layer Troubleshooting

Troubleshooting the physical layer starts with checking for cabling issues. The vast majority of these issues can be overcome if you follow the directions in the documentation of your equipment. Usually, cabling problems are evident at the initial implementation of the service and include issues such as the wrong crossover cable or faulty cables. Later in the chapter, some cable pinpoint examples are discussed. Physical problems are probably the number-one suspect on a circuit that has already been installed and turned up and after time has developed an outage. In most cases, if no one has changed the configurations, it is usually a punchdown, bridge clip, line card, or cable break issue.

Line and Clocking Problems

Clocking problems in serial connections can lead to either chronic loss of the connection or to degraded performance. The most common indicators for physical layer problems are shown in Example 17-3.

Example 17-3. If This Service Is Provisioned for 56-kbps, Check the Status of Serial 0

 1604-frame#show ip interface brief <output omitted> Serial0   unassigned   YES NVRAM  down down <output omitted> 

Under 1602-frame#show interface serial 0, the last line reports the status of the physical layer:

 <output omitted> DCD=down  DSR=down  DTR=down  RTS=down  CTS=down <output omitted> 

These signals are usually referred to as the modem and originally come from RS-232, with the following description:

• DCD (Data Carrier Detect) Provided by DCE

• DSR (Data Set Ready) Provided by DCE

• DTR (Data Terminal Ready) Provided by DTE

• RTS (Request To Send) Provided by DTE

• CTS (Clear To Send) Provided by DCE

Although numerous combinations of up and down signals exist, at this stage it is easy to determine what part of the device is failing. DTE is responsible for DTR and RTS, and DCE is responsible for the rest of the signals. When the clockrate command is missing from the DCE side, DCD, DSR, and CTS are shown as inactive or down.

If the line is functional but is reporting an excessive number of errors, you should be concerned about the quality of the physical layer. Use the command in Example 17-4 to check the physical layer.

Example 17-4. This Part of the Output Is Your Main Focus

 1604-frame#show interfaces serial 0 <output omitted> 237979 packets input, 21491750 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 5 giants, 0 throttles ! The following line is the part of the output you should review carefully: 198487 input errors, 44907 CRC, 20259 frame, 0 overrun, 0 ignored, 133321 abort 29689 packets output, 3054656 bytes, 0 underruns 0 output errors, 0 collisions, 329 interface res <output omitted> 

You should also be concerned about a high number of aborts; as you can see in this case, 133321 aborts represent the vast majority of all input errors (about 70 percent). Start with the configuration on both ends. Check the remote user's end first, for interface availability, as shown in Example 17-5. The output in Example 17-5 can be seen from the show version or show hardware commands, which shows that this router has one on-board 56-kbps serial interface.

Example 17-5. Output from the show hardware Command

 1602-frame#show  hardware <output omitted> 1 Serial network interface(s) On-board Switched 56K Line Interface. <output omitted> 

The clocking issue is addressed in the remote user's router configuration. In the case of a 56-kbps access rate, two configuration commands define the clocking and signaling, as shown in Example 17-6.

Example 17-6. A Fragment from the 1602 Configuration, Defining the 56-kbps Access Rate

 1602-frame#show running-config <output omitted> ! interface Serial0  no ip address  encapsulation frame-relay IETF  no fair-queue  service-module 56k clock source line  service-module 56k network-type dds  frame-relay lmi-type ansi ! <output omitted> 

The service module built into Cisco routers, including the 16xx, 252x, 26xx, 3600, 4xxx, and 7xxx series, supports two modes of 56-kbps settings for network-type: 56-kbps DSU/CSU and 56-kbps switched modes. The second of the following two commands identifies which side provides the clocking:

 1602-frame(config-if)#service-module 56k network-type [dds | switched] 1602-frame(config-if)#service-module 56k clock source [line | internal] 

NOTE

Dataphone Digital Service (DDS) provides digital transmission service between the CPE and the provider's switch at data rates between 2.4 kbps and 56 kbps. The service is available over two twisted pair lines.^[1]

The service module report is shown in Example 17-7. The lines are self-explanatory.

Example 17-7. Verifying the Functionality of the Built-In Service Module

 1602-frame#show service-module serial 0 Module type is 4-wire Switched 56K in DDS mode, Receiver has no alarms. Current line rate is 56 Kbits/sec and role is DSU side, Last clearing of alarm counters 3d03h     oos/oof                :    0,     loss of signal         :    2, last occurred 2d02h     loss of sealing current:    0,     CSU/DSU loopback       :    0,     loopback from remote   :    0,     DTE loopback           :    0,     line loopback          :    0, 1602-frame# 

In the case of a fractional T1 (or full T1) configuration, the expected output from the configuration should be as displayed in Example 17-8. The output in Example 17-8 can be seen from the show hardware or show version commands and shows that this router has one on-board 56 Kbps interface and one WIC T1-DSU card plugged-in.

Example 17-8. Output from the show version Command

 1602-frame#show version <output omitted>  2 Serial network interface(s) On-board Switched 56K Line Interface. WIC T1-DSU <output omitted> 

The configuration settings in the remote user's router are shown in Example 17-9.

Example 17-9. This Fragment of the Configuration Defines a 128-kbps Access Rate (2 Slots x 64 kbps Each)

 1602-frame#show running-config interface Serial1 <output omitted> encapsulation frame-relay service-module t1 timeslots 1-2 <output omitted> 

NOTE

These settings are important and the remote user needs to be aware of which circuit is ordered and provisioned. A 56-k circuit can be a true 56-k circuit (4 wire or 2 wire), but it can sometimes be a T1 circuit (4 wire, T1 port speed) with 56-k PVC. The WIC-1DSU-T1 cannot be used for 56-k leased line/DDS service, and of course the 56-k WIC cannot be used for T1 service.

This configuration defines two slots at 64 kbps each, which results in a 128-kbps access rate. In case the line is provisioned for 128 kbps, 384 kbps, or any other fraction of a T1, the built-in service module (serving a 56-kbps port) slows down because the router does not use this clocking (see Example 17-10).

Example 17-10. The Receiver Has Loss of Signal (LOS) because Serial 0 Is Administratively Down. Meanwhile, the WIC Card Is Functioning Properly

 1602-frame#show service-module Module type is 4-wire Switched 56K in DDS mode, ! You can expect the receiver to lose the signal, because the ! WIC card and DCE will provide the clocking, but not the DDS. Receiver has loss of signal, Current line rate is 56 Kbits/sec and role is DSU side, Last clearing of alarm counters 6w1d     oos/oof                :    0,     loss of signal         :    1, current duration 6w1d     loss of sealing current:    0,     CSU/DSU loopback       :    0,     loopback from remote   :    0,     DTE loopback           :    0,     line loopback          :    0, Module type is T1/fractional     Hardware revision is 0.88, Software revision is 0.2,     Image checksum is 0xED22BEC5, Protocol revision is 0.1 ! This line usually indicates that the WIC card is functioning properly. Receiver has no alarms. Framing is ESF, Line Code is B8ZS, Current clock source is line, Fraction has 2 timeslots (64 Kbits/sec each), Net bandwidth is 128 Kbits/sec. Last module self-test (done at startup): Passed Last clearing of alarm counters 6w1d     loss of signal        :    0,     loss of frame         :    0,     AIS alarm             :    0,     Remote alarm          :    0,     Module access errors  :    0, Total Data (last 96 15 minute intervals):     0 Line Code Violations, 0 Path Code Violations     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 13 Degraded Mins     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs Data in current interval (781 seconds elapsed):     0 Line Code Violations, 0 Path Code Violations     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs 1602-frame# 

NOTE

For troubleshooting FT1 and T1s, see Chapter 6, "Dial Design and Configuration Solutions," or RFC 1232.

The clocking and line issues are irrelevant to the situation, when the serial interface reports the output in Example 17-11.

Example 17-11. A Typical Output When the Interface Is Administratively Shut Down

 1602-frame#show interfaces serial 0 <output omitted>   Serial0 is Administratively Down, Line Protocol is Down   Hardware is QUICC Serial (with onboard CSU/DSU) <output omitted> 

The first command is used for an interface, whereas the second command is used for subinterface. In this case, you need to un-shut the interface with either of the following two commands:

 1602-frame(config-if)#no shutdown 1602-frame(config-subif)#no shutdown 

Serial Interface 0 and Line Protocol Is Down

When the interface serial 0 is not administratively down, the clocking problems have to be your primary focus, especially when the router reports the output in Example 17-12.

Example 17-12. The Line Problems Are Evident from This Output

 1602-frame#show interfaces serial 0 Serial0 is Down, Line Protocol is Down Hardware is QUICC Serial (with onboard CSU/DSU) <output omitted> 

In general, this problem occurs because clocking is not available. The following cases describe situations when the serial interface is not administratively down, but the router is still reporting both layers down. This case is typical for new service installations. The root cause of these problems is due to either or both of the following two cases:

• Case One Clocking problems can be because of incorrect cabling, faulty connectors, faulty devices in the D-Mark, and faulty passive or active repeaters. A common problem with the physical wiring is a reversed pair, which shows a line down/protocol down condition. Problems also occur if the CSU/DSU cable is longer than 50 feet [15.24 meters] or is unshielded.

• Case Two Another category of problems result from a faulty DSU/CSU, faulty line cards, an incorrect DSU/CSU configuration, or D-Mark equipment (commonly referred to as smart-jack and is usually on the MPOE, but in case of extended D-marks, can be in another location).

One of the most widespread issues is incorrect cabling. In Table 17-1, the correct cabling is specified for the two most common cases: 56-kbps and fractional T1 WIC cards.

NOTE

Remember that matching pinouts does not mean that the signaling is correct.

Table 17-1. 56-kbps DSU/CSU and WIC T1 Pinpoints

56-kbps CSU/DSU Pinouts (8-Pin RJ-48S)
Pin Number	Description
1	Transmit
2	Transmit
7	Receive
8	Receive
WIC T1 Pinouts (8-Pin RJ-48S)
Pin Number	Description
1	Transmit
2	Transmit
4	Receive
5	Receive

NOTE

It's always recommended that you refer to the available technical documentation of the customer premises equipment (CPE).

The CSU/DSU derives the data clock from the data that passes through it. Therefore, to recover the clock, the CSU/DSU hardware must receive at least one 1-bit value for every 8 bits of data that pass through it; this is known as ones density. Maintaining ones density allows the hardware to recover the data clock reliably.

Serial communications use a mechanism called serial clock transmit external (SCTE) terminal timing. SCTE is the clock that is echoed back from the data terminal equipment (DTE) device (for example, a router), to the data communications equipment (DCE) device. When the DCE device uses SCTE instead of its internal clock to sample data from the DTE, it is better able to sample the data without error, even if there is a phase shift in the cable between the CSU/DSU and the router. SCTE is an important element in serial transmissions and it uses access rates that are fractions of a T1.

To isolate clocking problems, besides the ones listed earlier, Cisco routers provide a set of tools to troubleshoot the CSU/DSU by using the show, debug, and loopback commands:

 1602-frame#debug service-module 1602-frame# show service-module 1602-frame(config-if)# loopback dte 1602-frame(config-if)# loopback line 1602-frame(config-if)# loopback remote 

Refer to Figure 17-1 for the show service-module command. Note that, in addition to the listed commands, under the T1 controller of the core router, you can use the following useful command: 7206-frame(config-controller)#loopback diag.

The first command reports the functionality of the module, as shown in Example 17-13.

Example 17-13. Debugging the Service Module

 1602-frame#debug service-module Service module debugging is on 1602-frame# 6w5d: SERVICE_MODULE(0): lxt441 interrupt 1 status A7 loop 0 6w5d: SERVICE_MODULE(0): lxt441 interrupt 1 status 87 loop 0 6w5d: SERVICE_MODULE(0): lxt441 interrupt 1 status A7 loop 0 6w5d: SERVICE_MODULE(0): lxt441 interrupt 1 status 87 loop 0 

The output reports that the service module is working properly.

Case One Problem Resolution

It is necessary to perform a series of loopback tests, both local and remote, to determine whether the DSU or CSU is causing the problem. Using the following commands, you can loopback the DTE, line, or the remote site:

 1602-frame(config-if)# loopback dte 1602-frame(config-if)# loopback line 1602-frame(config-if)# loopback remote 

It is always good to keep a record of all the different tests performed. In general, when looping and running these tests, always check the number of errors. Different ping tests (samples and sizes) are also recommended during this testing. You must determine whether there is a trend in the accumulation of errors and on what side of the User-Network Interface (UNI) they occur. If errors are accumulating on both ends of the connection (hub site and spoke site), it is most likely because of a DSU problem. If only the DSU end is accumulating errors, there is a problem with that particular end. An excessive number of aborts is a symptom of end-to-end line problems and it is recommended that you work with the local exchange carrier (LEC), who provides the local loop, to isolate the issue. Some Cisco line cards (T1/FT1 WIC) have a loopback button that, when engaged, allows the LEC to loop the spoke side for intrusive testing and clocking.

Case Two Problem Resolution

The problems for this case are related to faulty DSU/CSU, faulty line cards, and incorrect DSU/CSU configurations.

An incorrect CSU configuration is usually related to the clocking source. The following questions should be considered:

• What is the source of clocking (line or internal)?

• Do the impedance parameters match the manufacturer's documentation?

An incorrect DSU configuration is related to several important checkpoints. The following questions should be considered:

• Is SCTE enabled, especially for fractional T1s?

• Is the ones density maintained properly and do the framing and coding schemes match on both sides (Extended Superframe [ESF], bipolar 8-zero substitution [B8ZS], alternate mark inversion [AMI])?

• Although it is not common, is AMI coding configured? If it is, you might need to invert the transmit clock to match the LEC's requirements. For this purpose, Cisco provides an additional command that is set for 4000 series and 7000 series routers, and jumpers in the older versions of the equipment.

NOTE

Always see the Cisco IOS configuration guides and command references for the version of IOS that you are running on your router. You can also refer to available user guides for your CPE.

Performance Issues Related to the Physical Layer

You can experience line problems even if the router reports that the line is not down, as shown in the output in Example 17-14.

Example 17-14. The Line Shows Up and the Protocol Is Up, But the Performance Issues Exist

 1602-frame#show interfaces serial 1 <output omitted> Serial1 is up, line protocol is up <output omitted> 

However, the connection experiences slow performance, intermittent connectivity, and high latency. Some of the most common reasons for poor performance follow:

• Noisy (dirty) serial lines

• Cables longer than 50 feet [15.24 meters] or cables are unshielded

• Noisy or poor patch panel connections

• External EMI sources, such as unshielded cables running too close to EMI sources

A useful tool in this situation is #show interface serial 0, which allows you to not only check the status of the line at any moment (snapshot), but to monitor the dynamics of the changes. This command reports the output in Example 17-15 (see Table 17-2 for an explanation).

Example 17-15. Snapshot of the Serial 0

 1602-frame#show interfaces serial 0 Serial0 is up, line protocol is up   Hardware is QUICC Serial (with onboard CSU/DSU)   MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,      reliability 255/255, txload 1/255, rxload 1/255   Encapsulation FRAME-RELAY, loopback not set   Keepalive set (10 sec)   LMI enq sent  53714, LMI stat recvd 53710, LMI upd recvd 0, DTE LMI up   LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0   LMI DLCI 0  LMI type is ANSI Annex D  frame relay DTE   FR SVC disabled, LAPF state down   Broadcast queue 0/64, broadcasts sent/dropped 8958/0, interface broadcasts 3   Last input 00:00:00, output 00:00:00, output hang never   Last clearing of "show interface" counters 1w0d   Queuing strategy: fifo   Output queue 0/40, 0 drops; input queue 1/75, 0 drops   5 minute input rate 0 bits/sec, 2 packets/sec   5 minute output rate 0 bits/sec, 0 packets/sec      230129 packets input, 44936503 bytes, 0 no buffer      Received 0 broadcasts, 0 runts, 1 giants, 0 throttles      7241 input errors, 3594 CRC, 2701 frame, 0 overrun, 0 ignored, 944 abort      229128 packets output, 49424950 bytes, 0 underruns      0 output errors, 0 collisions, 2509 interface resets      0 output buffer failures, 0 output buffers swapped out      6 carrier transitions      DCD=up  DSR=up  DTR=up  RTS=up  CTS=up 

Table 17-2. The Output From # show interface serial 0 and Its Description

Output	Description
Serial0 is [up | down] administratively down	Indicates whether the clocking is presently available or not (matches Data Carrier Detect [DCD] signal in the last line of the output); and whether the line was taken down by the administrator.
Line protocol is [up | down]	Indicates a successful exchange of the configured LMI and keepalives (if they are configured).
Hardware is QUICC Serial (with onboard CSU/DSU)	Hardware information, showing the router has a built-in CSU/DSU-interface.
MTU 1500 bytes BW 1544 Kbit DLY 20000 usec	Maximum Transmit Unit (MTU) is 1500 bytes. Bandwidth is 1554 kbps. The routing protocols, such as Interior Gateway Routing Protocol (IGRP) and Enhanced Interior Gateway Routing Protocol (EIGRP), use this parameter to compute the routing metrics. If the bandwith differs from the default value (1544 or 1536), the command 7206-frame(config-if) #bandwidth value specifies the correct access rate of the line. Delay in the interface in milliseconds.
reliability 255/255, txload 1/255, rxload 1/255	Reliability, transmit load (txload) and receive load (rxload) are a fraction of 255. Thus, 255/255 is the highest and 1/255 is the lowest.
Encapsulation FRAME-RELAY, loopback not set	The encapsulation of the interface is Frame Relay and the loopback is not set.
Keepalive set (10 sec)	Keepalive messages are set to 10-second intervals. In some cases, an interval of 30 seconds is acceptable, in other cases, the engineers set the DSU interval to 8 seconds, which is less than the switch interval, and it is set up this way to prevent the line from flapping. One of the techniques used for troubleshooting is to negate the keepalives with 7206-frame(config-if)#no keepalive to prevent the line from flapping reports and to rely only on LMIs.
LMI enq sent 53714 LMI stat recvd 53710 LMI upd recvd 0 DTE LMI up	The number of LMI enquiry messages sent by the DTE. The number of LMI status messages received from the switch. The number of LMI updates received. An indication of DTE LMI [up | down]. (Matches some of the parameters of #show frame-relay status)
LMI enq recvd 0 LMI stat sent 0 LMI upd sent 0	The number of LMI status enquiries received by the switch. The number of LMI status messages sent to the switch. The number of LMI updates sent to the switch. (Matches some of the parameters of #show frame-relay status)
LMI DLCI 0 LMI type is ANSI AnnexD Frame Relay DTE	The type of LMI signaling (see Chapter 14 or the following chapter). In this case, DLCI 0 indicates ANSI AnnexD LMI. The other options are cisco and q.933.
FR SVC disabled Link Access Procedures for Frame-Mode Bearer Services (LAPF) state down	This report is for PVC design (frame-relay), but not Frame-Switching. This is why the SVC is down. LAPF down means LAPF is not performing the signaling in this circuit. Only the core functions of LAPF are involved.
Broadcast queue 0/64, broadcasts sent/dropped 8958/0, interface broadcasts 3	The line reports the status of the broadcast queue. In Frame Relay, this queue is separate from others.
Last input 00:00:00, output 00:00:00	The number of hours, minutes, and seconds since the last packet was received (input) or sent (output) by this interface.
Output hang never	Indicates hours:minutes:seconds or never since the last reset of this interface. If the time is more than 24 hours, you can see 2d3h (2 days and 3 hours), but if this format is not large enough, you can see *****, which indicates more characters than can be expressed in this format.
Last clearing of "show interface" counters 1w0d	Often when troubleshooting, you need to start with fresh data. This value indicates the last time in hours:minutes:seconds (or never) that the interface was cleared by using any of the following commands: 1602-frame#clear counters serial 0 7206-frame#clear counters serial 3/0:0 7206-frame#clear interface serial 3/0:0
Queuing strategy: fifo	The FIFO queuing strategy stands for First-In-First-out. For other queuing strategies, see www.cisco.com.
Output queue 0/40, 0 drops Input queue 1/75, 0 drops	An input queue with self-explanatory parameters and the number of dropped packets because of a lack of queue slots. If the number of dropped packets is high, you might want to check the interface buffers using the following command: 1602-frame#sh buffers
Conversations 0/1/16 (active/max active/max total) Reserved conversations 0/0 (allocated/max allocated)	Multiplexing many logical (virtual) interfaces into one physical interface is called data conversations. This information exists only for the D channel. BRI channels do not have signaling functions.
5 minute input rate 0 bits/sec, 2 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec	The average rate of data exchange in and out in 5-minute intervals, measured in bit/sec and packets/sec. It indicates exponentially weighted values, which are cumulative values and not snapshot values. You only use this parameter as an approximation of the traffic, to check traffic trends, or to compare with other interfaces.
230,129 packets input, 44,936,503 bytes, 0 no buffer Received 0 broadcasts 0 runts 1 giants 0 throttles	The number of successfully received packets and bytes. The number of instances of a lack of input buffer slots. Total number of received broadcasts on this interface. The number of discarded packets because their size was smaller than the minimum packet size. The number of discarded packets because their size was greater than the maximum packet size. The number of times the receiver on the port was disabled, possibly because of buffer or processor overload.
7241 input errors 3594 CRC 2701 frame 0 overrun 0 ignored 944 abort	The total number of input errors since the last #clear counters command, which are categorized as follows: Cyclic redundancy check (CRC) errors that usually indicate noise or other transmission problems. Incorrectly received packets, or packets with a non-integer number of octets, which usually indicates noise or other transmission problems. The number of times when the input rate exceeded the receiver's ability to buffer and handle the data. The number of times when the hardware ran low in internal buffers. Broadcast storms and bursts of noise can be the possible cause of this problem. The number of aborts. It shows an illegal number of 1s in the interface, which is an indication of clocking problems on the interface.
229,128 packets output, 49,424,950 bytes 0 underruns 0 output errors 0 collisions 2509 interface resets 0 output buffer failures, 0 output buffers swapped out 6 carrier transitions	The number of output packets and bytes in the interface that were passed without errors. The number of times when the transmitter runs faster than the router can handle. The number of all output errors. The number of collisions on the interface, which might come from the contention failures. A number of collisions more than 45 percent should be investigated. The number of interface resets indicates "no signal" in the interface. On the serial line, this is caused by a malfunctioning modem DSU/CSU or a cable problem. If the router sees the DTR up but protocol down, it resets the interface in an effort to trigger the LMI or keepalive exchange. The number of buffer failures and swapped output buffers. The number of carrier transitions, where an excessive number indicates either intrusive testing performed by the LEC, or switch card/line problems that must be investigated with the LEC.
DCD = up DSR = up DTR = up RTS = up CTS = up	DCD clock is up. Data Set Ready (DSR), an input signal, is up. Data Terminal Ready, an output signal, is up. Request To Send (RTS), an output signal, is up. Clear To Send (CTS), an input signal, is up. All five ups indicate that the serial interface is up.

If at any step of the troubleshooting process, the number of errors exceeds an approximate range of 0.5 percent to 2.0 percent of traffic on the interface, clocking problems are likely to exist somewhere in the WAN.