Tag: Cisco

Cisco 45xx Supervisor 6LE Defect

As I tend to work with some Cisco 4506 and similiar switches - equipped with WSX45SUP6LE / WS-X45-SUP6L-E Supervisor Engines, I had the unfortunate delight to seem those rebooting at random.

Cisco did put out an entry in their bug tool ("Sup6LE reloads silently (Reset State: 00000201) / watchdog CISR0: 0x80") with the Bug Tracker ID CSCtf85481 - however, of the multiple accounts I had, I only found it on one of those. Other accounts did refuse my access to this file. As we confronted our Cisco Tech Support prior to finding the bug ourself, there was "no known error" which represented itself in rebooting the switch at random - so they said.

How do you find out it your board is one of those faulty ones?
a) If the serial number is JAE1422xxxx or higher (e.g., JAE1445xxxx, JAE1729xxx, etc) they board should not be affected. If it is in the range provided - check b)
b) If the "Hardware Revision" is 1.3 or higher, it is not affected. [sh idprom supervisor]
c) If either a) or b) are not true. Then the board may be affected by this bug. Contact TAC.

Another sign of the error is the "Last reload reason: Unknown reason" output of show version

It seems like the first batch of Supervisor 6L-E Engines were faulty on a HW level, so only a replacement will help. The engines will start rebooting once a while after a certain amount of time - with the time between each rebooting getting shorter.

Cisco RAM Problem (Phone/Linecard)

As a matter of fact, I've been working for more than 8 years with Cisco equipment and continue to do so. I really like Ciscos products, especially in the router / switch sector and had the pleasure to work with products in the range of Switching, Routing, Communications / Phones, Wifi, Datacenter Connectivity and Security. However, I had 3 unpleasent events with Ciscos products and I want to take the time to talk about two of those, as they occured because of the same reason.

If you don't know about Ciscos RAM problem, I want to give you a quick heads up: Fact is that Cisco installed defective SDRAM in almost all their products ranging from 2005-2012. The products with this defective RAM would work as normally, however, after being in use for more than 2 years AND an reboot, the products would fail - and stay that way. Cisco got to know about that problem in 2010, as they state themself, however, they informed users in 2012 for the first time. You can find out more about the topic on http://www.cisco.com/go/memory - this website was in 2014... As you can see, quite a lot of different products, including Routers like the 18xx/28xx series, Phones like the 79xx, the ASA55xx firewalls, Firewall Service Modules and more.

1.) Phones
As we had switched over to Cisco Phones a long time ago, we had multiple thousands of Cisco 79xx phones standing around and starting to die in 2014. We just got more an more messages from different customers that the phones just "went blank" and did not come up again. Only the speaker button was lit and thats it. As more and more phones died and we already opened up our own little graveyard, we went to Cisco with our problem - however, we never received an answer - until I figured out the problem myself: By disassembling some 7945, 7965 and 7975 - inspecting them and working around them with an self-made Serial Cable to the phones. It seemed like they would not start to unpack their image... As I figured the CPU should be fine an flash too, I came up with the theory that the SD-RAM was broken and found Ciscos website. However, I still insisted on proving my theory in the only way possible: Resurrecting one of our 7975 corpses from the graveyard.

I found the really good teardown on globalspec.com which stated that the SDRAM in this phone was a Samsung K4H561638H-UCB3 [SDRAM - DDR, 256Mb (16M x 16), 166 MHz, 2.5V, TSSOP 66]. After that I just removed the Motherboard from the Phone, removed the RAM with help from a friend (he got some really nice SMD reballing workstation :)) - and soldered in the new RAM. Without reflashing any Firmware or reset, it just worked after putting it back together! This proved my point.

(Picture was taken from http://electronics360.globalspec.com/article/3227/cisco-7975g-ip-phone-teardown)

2.) Linecards
Just some months ago, we had another accident with a linecard: One of our core switches rebooted due to power failure and after that, our 10 Gig Linecard, which connected one of our two main storage systems to the core, failed.

Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      xxxxxxxxxxx
  5    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       xxxxxxxxxxx

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  xxxxxxxxxxxxxx to xxxxxxxxxxxxxx   3.2   Unknown      Unknown      Other
  5  xxxxxxxxxxxxxx to xxxxxxxxxxxxxx   4.7   8.5(4)       12.2(33)SXH8 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status 
---- --------------------------- ------------------ ----------- ------- -------
  1  Centralized Forwarding Card WS-F6700-CFC       xxxxxxxxxxx  4.1    Other
  5  Policy Feature Card 3       WS-F6K-PFC3B       xxxxxxxxxxx  2.7    Ok
  5  MSFC3 Daughterboard         WS-SUP720          xxxxxxxxxxx  2.12   Ok

Mod  Online Diag Status 
---- -------------------
  1  Unknown
  5  Pass
Router# show power
system power redundancy mode = redundant
system power redundancy operationally = non-redundant
system power total =     2771.16 Watts (65.98 Amps @ 42V)
system power used =       859.74 Watts (20.47 Amps @ 42V)
system power available = 1911.42 Watts (45.51 Amps @ 42V)
                        Power-Capacity PS-Fan Output Oper
PS   Type               Watts   A @42V Status Status State
---- ------------------ ------- ------ ------ ------ -----
1    WS-CAC-3000W       2771.16 65.98  OK     OK     on 
2    WS-CAC-3000W       2771.16 65.98  -      -      off
                        Pwr-Requested  Pwr-Allocated  Admin Oper
Slot Card-Type          Watts   A @42V Watts   A @42V State State
---- ------------------ ------- ------ ------- ------ ----- -----
1    WS-X6704-10GE       295.26  7.03   295.26  7.03  on    on
5    WS-SUP720-3B        282.24  6.72   282.24  6.72  on    on
6    (Redundant Sup)       -     -      282.24  6.72  -     -
Router#show platform hardware pfc mode
PFC operating mode : PFC3B

However, after replacing the Memory with new one, everything worked out - the Linecard was usable again!
I found information about the problem on Cisco again - after I resolved the problem: http://www.cisco.com/c/en/us/support/docs/field-notices/637/fn63743.html

The diagnostic test could be started with diagnostic start system test all

So, these are two problems I personally came across with Cisco Systems which failed, due to faulty memory and I decided to describe here - maybe some people stumble across these keywords and find the solution for their failing devices.

Cisco Prime Infrastructure 3.1 Cheat Sheet

A little cheat sheet for myself. All commands can be used via SSH or Shell:

Show Config: show run
Show Inventory: show inventory (Does show i.e. how many CPUs and RAM is installed. This does match normally to the Version of Cisco Prime. i.E. 8 vCPUs and 16 GB RAM -> Express Plus Type)

Show status of prime: ncs status

Backup OS and Application: backup PI311 repository defaultRepo
Backup Application only: backup PI311appOnly repository defaultRepo application NCS

Activate OS Shell: shell

Locations of different files
defaultRepo: /localdisk/defaultRepo
Config: /storedconfig
License Files: /opt/CSCOlumos/licenses

To transfer the backup files to a safe place, just use scp :).

[1841] Reset Password / Config, ROMMON Upgrade and Software Upgrade on Cisco 1841

1.) Reset Password / Config

- Connect to the router via Serial Cable

- Power on the router, but send a break during the first 60 seconds of boot time to get to the rommon CLI

- enter confreg 0x2142 followed by a reset

- Router is booting up, as soon as it does enter the system configuration dialog, answer the question with no

- enable

- conf t

- config-register 0x2102

- exit

- wr mem (because I just want to overwrite the old config :)!)

2.) ROMMON Upgrade

- Get the latest / needed ROMMON upgrade from Cisco

- Copy it i.e. to your CF card via Card Reader, TFTP or other means

- Boot up the router and enter privleged / enable mode

- Enter the upgrade command: upgrade rom-monitor file flash:<Filename>

- Answer yes to get the process starteted

- After the upgrade the router will reload

3.) Software Upgrade

- Get the latest/needed image from Cisco

- Just copy it to the CF card via Card Reader, TFTP or other means

- Reload

Cisco PIX 506e Software Upgrade

Following the Hardware Upgrade, the Software needs to be renewed, too!
The latest PIX OS running on the 506e would be 6.3.5.125.
You can upgrade to that and use the pdm to configure the Device.
Or... you can go to ASA!
As described here ( http://www.rownet.co.uk/installing-v7-software-on-a-cisco-pix-506e/ ) you can do that quite easily.

But in short:
- Boot into Pix
- Start downloading an pdm file with copy tftp://Your_TFTP_Server_IP_Address/Your_pdmfile_name flash:pdm and disconnect the ethernet cable!
- After disconnecting, the PIX will erase the pdm from your pix, clearing the flash space for pdm
- clear flashfs
- reload your PIX into Monitor Mode / RMON
- activate your interface (interface 0 is E0, interface 1 E1)
- enter ip addr of tftp server ( server ), filename ( tftp ), ip addr of your pix ( address ) and start downloading to pix
- It will boot from tftp into ASA
- Copy ASA Image: copy tftp://Your_TFTP_Server_IP_Address/pix712.bin flash:
- Reload

Done, you got ASA.
No ASDM / PDM anymore, because not enough space, but ASA ^^

Cisco PIX 506e Hardware Upgrade

The Cisco PIX Series has been "the" well known and powerful firewall appliance of the last decade - and for some people, this decade ain't over yet.
Cisco has released the successor "ASA" some years ago, but many people still got a PIX running.
Reasons are simple: The PIX Series has an Appliance for every problem and is rock solid.
My personal experience with that Firewall Series started with a "burned out" PIX 520 (that one is getting its own entry soon ^^).
After that, I got more into Security by doing my CCNA Security studies.
While doing the CCNAS I also bought the smallest PIX, the PIX 501 from Ebay for 70€ or so.
Sometime later, I discovered an dead PIX 506e in my Office - and I just couldn't help myself and started taking it apart.

Shortly after disassembling I stumbeled upon this Blog: http://hackaday.com/2008/09/28/upgrading-the-cisco-pix-506e/
And that looked very promissing.

Cisco started upgrading some PIX Appliances with new Software Versions.
6.3.5 was the last "PIX" Software. After that, Version 7 and Version 8 were "ASA".
( While 7 was more some kind of bridge version, at least it feels like it... )
Neither my 501,520 or the 506e do run 7.x or 8.x - at least thats what Cisco does say.
Problem is the amount of memory onboard: 501 and 506e only got 8 MB of Flash - not upgradable.
The 520 got 2 or 16 MB Flash - but ain't supported. Another problem is the amount of RAM and CPU.

So.. I did start on these problems.

First thing after opening the 506e was exchanging the CMOS Battery.
Seriously, I felt as that this PIX was always crashing because of some dead battery.
At least, it give me a better feeling having that thing loaded up again.

After that, I was looking for the RAM: SDRAM, 100 MHz FSB.
I looked through the inventory and found 2 bars, each 256 MB, 133 MHz.
Maybe not the best idea, the 506e is only going on 100 MHz, but worth a try.
And - it did work. So, we got the 32 MB replaced by 512 MB.
I think that should be... enough.

RAM is done, Battery replaced, but what about the CPU?
Good Question!
The 506e is running on an 300 MHz Pentium 2 Celeron (SL36A, Mendocino Core, 128 KByte L2, 2V) - not really much.
So I was looking around again and found an awesome 1 GHz Pentium 3 (SL4C8, Coppermine, 256 KByte L2, 1.7V).
After pluging in and testing I found out that this thing was really working. Problem were the 133 MHz FSB - so the CPU did only run with 750 MHz - more than enough!
I was seriously happy, but a problem was coming up: Heat.

The PIX 506e enclosing is really badly build: The CPU Cooler is just sitting some milimeters under the hood, not ventilationholes anywhere except at the end of the case. You can even see some dust burned into the case inlay above the cpu cooler... "nice". So - the new CPU would be really too much for this case. And my idea was correct: Some minutes after closing the case and running the firewall - the CPU got shutdown because of thermal problems. Ok! What to do now? Well - solution was easy: Just cut out the steel enclosing above the CPU Cooler, get some special cloth above it - so nobody would touch in - and close the case. Problem solved.

The next question came up: Well, what does that "little tweaking" really did to the firewall?
Solution to that: Benchmark!
I fired up iperf / jperf with following command:
bin/iperf.exe -c SERVERIP -P 4 -i 20 -p 5001 -w 512.0k -l 512.0k -f m -t 3600
1 Hours, special Packet Size, 4 Parallel Threads. That should "burn-in"....
...and it did: After 20 Minutes with really superior performance ( CPU on 10% by delivering 92 MBit/s! ) the CPU died.
And the powersupply? Well - felt like on fire, too. Damn.

So, the CPU seemed to be too much for the little firewall...
But I didn't wanted to stop there.
The next burn in with the old CPU took place.
Everything was fine, nothing was hurt except the dead CPU.
The firewall was running hour on hour smooth with 100% Network Stress.
( With the 300 MHz CPU, the PIX was already working above 40%... well,... not as good )
Ok.

After stumbeling around in my cases, I finally found the best CPU I came up with at the moment:
SL3XY, Coppermine, 256 KByte L2, 1,65V - an Pentium 3 with 733 MHz.
And I didn't even knew wheter that thing was still working.
I really thought I fried it already some years earlier...
Well, it seemed like... not!

In the end, the Firewall did work at 550 MHz ( 133 MHz FSB aswell... ) for over 4 hours,
26% CPU Load - nice! I think thats ok (The power supply also stayed reasonably cold).

The last thing I did was installing an passiv Heatsink on the AGP Chipset of the PIX.
It was getting hot for no reason... So.... some better cooling than the naked Chip itself is always nice...

So - that was the Hardware. But what about the PIX OS?
Mh...

Next Post 😉

[79XX] Cisco IP Phone Factory Reset & Reboot

To Factory Reset a Cisco IP Phone do the following:
1. Press and hold #, disconnect and reconnect Power
2. The Phone will check through the Line LEDs, on an 7911 or similar it will just light up the both Menu Keys and the Reciever \ Handset LED - after that, let go of #
3. Enter following: 1 2 3 4 5 6 7 8 9 * 0 #
4. The Phone should start up and grab the latest config from your Callmanager

To just Reboot a Cisco IP Phone do following:
1. Go into any Menu
2. Enter #
3. Phone does reload
( 7940, 7960: Hold *,6,Settings )

To open up the Configuration on a Phone:
1. Go into Menu Setting
2. Enter **##

To Factory Reset and erase the firmware on a Cisco IP Phone do the following:
1. Press and hold #, disconnect and reconnect Power
2. The Phone will check through the Line LEDs, on an 7911 or similar it will just light up the both Menu Keys and the Reciever \ Handset LED - after that, let go of #
3. Enter following: 3 4 9 1 6 7 2 8 5 0 * #
4. The Phone should start up and grab the latest Firmware from your Callmanager

[CCNA] Cisco Networking Academy Material Error

If you open CCNA Material in the Internet Explorer, it does always Popup the Error that it doesn't want to execute these Scripts as they might be dangerous. You can click that away and work with the Course Materials - but it just tends to go on to ones' nerves. So to disable that: Internet Explorer, Extras, Internet Options, Advanced. Look for Security and enable "Execution of Interactive Content within Files on local Computer" and "Execution of Interactive Content within CDs on local Computer".

[iPAQ] Building Serial Console for iPAQ 3600 to Access Cisco Equipment

After I got the iPAQ 3660 working again, I wanted to use it as Serial Console to Work on the Cisco Equipment "on the road". To achieve that, we need two or three things:

1.) An special connector Cable to attach the Serial Console of the iPAQ with an RS232 Connector / the Cisco Rollover Cable
2.) An Terminal Software. I did choose vxHpcPlus for that ( http://www.cam.com/vxhpcplus.html )
3.) Something that makes it possible to use  Landscape Mode so that you can read the Console Output more easily:  NYDITOT Virtual Display, Version 5.02 ( http://www.nyditot.com/Products.asp )

1.) Building the Connector / Adaptor Cable
I did order an iPAQ 3600 to Serial Connector for Active Sync. Sadly it was defect, so I did choose to rebuild it to these Specs. First I did open up the iPAQ Connector and desolder it. Than I did insert it into the iPAQ and identified the needed Connector Ports. So if you look at the bottom of the iPAQ, placed face up on flat surface you will see these Connectors:

2 4 6 8 10 12
1 3 5 7  9 11

Wiring:
4 - GND
7 - TX
8 - RX
11 - 3,3V

Then I got an RS232 Male Connector and did wire these things up like that:

5 4 3 2 1
6 7 8 9

( Looking at the Solder Connectors! Not the "Connector Face"! )

2 - RX
3 - TX
5 - GND

I did left the 3,3V Cable "floating around" for maybe future projects.

2.) Terminal Software
Download the vxHpcPlus from the Website and install it with the Help of ActiveSync.

The Settings for the Serial Port are:

Direct Connect - Async:
Port: "Seriallkabel an COM1:"
Settings: 9600, Databits: 8, Parity: None, Stop: 1, Flow Control: None
Telnet: Send NUL after CR
Emulation: VT100,
Font size: 6,
German Keyboard
80 Colums,
Vertial Scroll,
Auto Wrap,
Scroll,
Hot Button

You can also change these to 115200 if you want.

3.) NYDITOT Virtual Display, Landscape
Download and install with ActiveSync. You can use it to use Landscape Mode which makes reading the Serial Console more easy. Its plain forward, so play with it.

Thanks a lot to Bev Howard for the excellent Printout of the Connector Port ( http://bevhoward.com/serial.htm#36xx ) this Site: http://www.handhelds.org/Compaq/iPAQH3600/iPAQ_H3600.html