Cisco 45xx Supervisor 6LE Defect

As I tend to work with some Cisco 4506 and similiar switches - equipped with WSX45SUP6LE / WS-X45-SUP6L-E Supervisor Engines, I had the unfortunate delight to seem those rebooting at random.

Cisco did put out an entry in their bug tool ("Sup6LE reloads silently (Reset State: 00000201) / watchdog CISR0: 0x80") with the Bug Tracker ID CSCtf85481 - however, of the multiple accounts I had, I only found it on one of those. Other accounts did refuse my access to this file. As we confronted our Cisco Tech Support prior to finding the bug ourself, there was "no known error" which represented itself in rebooting the switch at random - so they said.

How do you find out it your board is one of those faulty ones?
a) If the serial number is JAE1422xxxx or higher (e.g., JAE1445xxxx, JAE1729xxx, etc) they board should not be affected. If it is in the range provided - check b)
b) If the "Hardware Revision" is 1.3 or higher, it is not affected. [sh idprom supervisor]
c) If either a) or b) are not true. Then the board may be affected by this bug. Contact TAC.

Another sign of the error is the "Last reload reason: Unknown reason" output of show version

It seems like the first batch of Supervisor 6L-E Engines were faulty on a HW level, so only a replacement will help. The engines will start rebooting once a while after a certain amount of time - with the time between each rebooting getting shorter.

Cisco RAM Problem (Phone/Linecard)

As a matter of fact, I've been working for more than 8 years with Cisco equipment and continue to do so. I really like Ciscos products, especially in the router / switch sector and had the pleasure to work with products in the range of Switching, Routing, Communications / Phones, Wifi, Datacenter Connectivity and Security. However, I had 3 unpleasent events with Ciscos products and I want to take the time to talk about two of those, as they occured because of the same reason.

If you don't know about Ciscos RAM problem, I want to give you a quick heads up: Fact is that Cisco installed defective SDRAM in almost all their products ranging from 2005-2012. The products with this defective RAM would work as normally, however, after being in use for more than 2 years AND an reboot, the products would fail - and stay that way. Cisco got to know about that problem in 2010, as they state themself, however, they informed users in 2012 for the first time. You can find out more about the topic on http://www.cisco.com/go/memory - this website was in 2014... As you can see, quite a lot of different products, including Routers like the 18xx/28xx series, Phones like the 79xx, the ASA55xx firewalls, Firewall Service Modules and more.

1.) Phones
As we had switched over to Cisco Phones a long time ago, we had multiple thousands of Cisco 79xx phones standing around and starting to die in 2014. We just got more an more messages from different customers that the phones just "went blank" and did not come up again. Only the speaker button was lit and thats it. As more and more phones died and we already opened up our own little graveyard, we went to Cisco with our problem - however, we never received an answer - until I figured out the problem myself: By disassembling some 7945, 7965 and 7975 - inspecting them and working around them with an self-made Serial Cable to the phones. It seemed like they would not start to unpack their image... As I figured the CPU should be fine an flash too, I came up with the theory that the SD-RAM was broken and found Ciscos website. However, I still insisted on proving my theory in the only way possible: Resurrecting one of our 7975 corpses from the graveyard.

I found the really good teardown on globalspec.com which stated that the SDRAM in this phone was a Samsung K4H561638H-UCB3 [SDRAM - DDR, 256Mb (16M x 16), 166 MHz, 2.5V, TSSOP 66]. After that I just removed the Motherboard from the Phone, removed the RAM with help from a friend (he got some really nice SMD reballing workstation :)) - and soldered in the new RAM. Without reflashing any Firmware or reset, it just worked after putting it back together! This proved my point.

(Picture was taken from http://electronics360.globalspec.com/article/3227/cisco-7975g-ip-phone-teardown)

2.) Linecards
Just some months ago, we had another accident with a linecard: One of our core switches rebooted due to power failure and after that, our 10 Gig Linecard, which connected one of our two main storage systems to the core, failed.

Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      xxxxxxxxxxx
  5    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       xxxxxxxxxxx

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  xxxxxxxxxxxxxx to xxxxxxxxxxxxxx   3.2   Unknown      Unknown      Other
  5  xxxxxxxxxxxxxx to xxxxxxxxxxxxxx   4.7   8.5(4)       12.2(33)SXH8 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status 
---- --------------------------- ------------------ ----------- ------- -------
  1  Centralized Forwarding Card WS-F6700-CFC       xxxxxxxxxxx  4.1    Other
  5  Policy Feature Card 3       WS-F6K-PFC3B       xxxxxxxxxxx  2.7    Ok
  5  MSFC3 Daughterboard         WS-SUP720          xxxxxxxxxxx  2.12   Ok

Mod  Online Diag Status 
---- -------------------
  1  Unknown
  5  Pass
Router# show power
system power redundancy mode = redundant
system power redundancy operationally = non-redundant
system power total =     2771.16 Watts (65.98 Amps @ 42V)
system power used =       859.74 Watts (20.47 Amps @ 42V)
system power available = 1911.42 Watts (45.51 Amps @ 42V)
                        Power-Capacity PS-Fan Output Oper
PS   Type               Watts   A @42V Status Status State
---- ------------------ ------- ------ ------ ------ -----
1    WS-CAC-3000W       2771.16 65.98  OK     OK     on 
2    WS-CAC-3000W       2771.16 65.98  -      -      off
                        Pwr-Requested  Pwr-Allocated  Admin Oper
Slot Card-Type          Watts   A @42V Watts   A @42V State State
---- ------------------ ------- ------ ------- ------ ----- -----
1    WS-X6704-10GE       295.26  7.03   295.26  7.03  on    on
5    WS-SUP720-3B        282.24  6.72   282.24  6.72  on    on
6    (Redundant Sup)       -     -      282.24  6.72  -     -
Router#show platform hardware pfc mode
PFC operating mode : PFC3B

However, after replacing the Memory with new one, everything worked out - the Linecard was usable again!
I found information about the problem on Cisco again - after I resolved the problem: http://www.cisco.com/c/en/us/support/docs/field-notices/637/fn63743.html

The diagnostic test could be started with diagnostic start system test all

So, these are two problems I personally came across with Cisco Systems which failed, due to faulty memory and I decided to describe here - maybe some people stumble across these keywords and find the solution for their failing devices.

Cisco Prime Infrastructure 3.1 Cheat Sheet

A little cheat sheet for myself. All commands can be used via SSH or Shell:

Show Config: show run
Show Inventory: show inventory (Does show i.e. how many CPUs and RAM is installed. This does match normally to the Version of Cisco Prime. i.E. 8 vCPUs and 16 GB RAM -> Express Plus Type)

Show status of prime: ncs status

Backup OS and Application: backup PI311 repository defaultRepo
Backup Application only: backup PI311appOnly repository defaultRepo application NCS

Activate OS Shell: shell

Locations of different files
defaultRepo: /localdisk/defaultRepo
Config: /storedconfig
License Files: /opt/CSCOlumos/licenses

To transfer the backup files to a safe place, just use scp :).

[1841] Reset Password / Config, ROMMON Upgrade and Software Upgrade on Cisco 1841

1.) Reset Password / Config

- Connect to the router via Serial Cable

- Power on the router, but send a break during the first 60 seconds of boot time to get to the rommon CLI

- enter confreg 0x2142 followed by a reset

- Router is booting up, as soon as it does enter the system configuration dialog, answer the question with no

- enable

- conf t

- config-register 0x2102

- exit

- wr mem (because I just want to overwrite the old config :)!)

2.) ROMMON Upgrade

- Get the latest / needed ROMMON upgrade from Cisco

- Copy it i.e. to your CF card via Card Reader, TFTP or other means

- Boot up the router and enter privleged / enable mode

- Enter the upgrade command: upgrade rom-monitor file flash:<Filename>

- Answer yes to get the process starteted

- After the upgrade the router will reload

3.) Software Upgrade

- Get the latest/needed image from Cisco

- Just copy it to the CF card via Card Reader, TFTP or other means

- Reload

[VSS] Upgrading Software on c6509e-VSS / Sup720-10GE

1.) Find an suitable image on cisco.com [i.e. s72033-adventerprisek9_wan-mz.122-33.SXJ1.bin ]
2.) Copy the image to the CF of active Supervisor by TFTP or FTP
copy tftp://IP/images/s72033-adventerprisek9_wan-mz.122-33.SXJ1.bin disk0:
3.) Verfiy the image against the MD5 and internal checksum
verify /md5 disk0:s72033-adventerprisek9_wan-mz.122-33.SXJ1.bin
verify disk0:s72033-adventerprisek9_wan-mz.122-33.SXJ1.bin
4.) Delete old Software on active Supervisor CF
del sw1-slot5-disk0:s72033-ipservicesk9_wan-mz.122-33.SXI5.bin
5.) Unconfigure other Boot Images
conf t
no boot system
config-register 0x2102
boot system flash disk0:
end
copy running-config startup-config
6.) Reload active Supervisor
redundancy reload shelf 1
7.) Wait for the ex-Active to come up again on local Console, sh redundancy should tell you that the ex-Active is now in Cold Standby as there is an Image mismatch. Thats ok, Image does work, so now lets swap the CF cards
8.) Reload now active Supervisor
redundancy reload shelf 2
9.) Wait for the second System to come up again, the VSS should run again in Active / Standby Hot Mode
10.) Copy Bootimage from shelf2 (with CF of shelf1) to shelf1 CF
copy sw-2-slot5-disk0:s72033-adventerprisek9_wan-mz.122-33.SXJ1.bin sw-1-slot5-disk0:
11.) Delete other old images form shelf1 CF
12.) Done

Not the best / Cisco way, but it should hold your Active / Standby Config in right order.

[IPv6] Dualstack EIGRP Routing on c3560

IPv6.

Everyone is talking about it.

And not too soon, ppl will need it.

As my corporation needs to switch to v6 soon, I'll started to work into this hexadecimal notated nightmare ;-).

I'll got some basic stuff written down already - but I'm not feeling like starting off IPv6 in my blog with too much boring theory.

So lets jump into a full blown network! ^^

 

Tada!

Well - its not too "full blown" - but a working network at last: We got our Routers, Switches, PCs and EIGRP - and an Application thats driven by IPv4 and IPv6 called "ping" and "ping6" - thats something, ain't it?

But first: What is Dual Stack? Well. Thats an easy one: Running your network on Dual Stack does mean using IPv4 and IPv6 in co-existence. Nothing more and nothing less. Your normal Ethernet Switch does Switch IPv4 and IPv6 the same - as Layer 2 does not differ between both. Layer 2 is MAC, IPv4/6 do come in mind on Layer 3 - to the Routers mind, so to say.

Sadly, I don't had any Cisco 1841 or similar machines for this project - and I did wanted to test these settings in an rather small Lab before jumping to our "spare" Cisco VSS Team and hacking that thing to kingdom come. Therefore, I  used serval Catalyst 3560 PC 8, which are small Cisco Layer3+ Switches and could replace the routers.

To get the Switches to become IPv6 Routers, we need to install an new IOS with IP-Services Feature Set or higher. You can do that be erasing the Switch flash and download an new IOS by using the archive sw command from an tftp ( archive download-sw /allow-feature-upgrade /reload tftp://IP/c3560-ipservicesk9-tar.122-55.SE.tar ). As you see, I did use the 12.2(55)SE Firmware, as the new 58 got some really nasty changes in Terms of Smartport Features.

After our Switch(es) did reload, we need to configure an new SDM Template. The Switching Database Manager is not some piece of software, in truth its more some kind of ressource template. By changing this SDM Template, you change the way how the Ternary Content Addressable Memory (TCAM) is seperated. The TCAM is an very small but fast piece of RAM which does save Information like MAC Addresses, Routes, VLANs and so on. If you change the SDM Template, you change the amount of space which is avaiable for an Information. i.e. VLANs. If you choose the "Routing" Template - your Switch will put more memory onto Routes, if you choose the "VLAN" Template, your switch will support more VLANs but disable routing - and if you choose "Default", you get an mix of both. So, SDM Templates can be mission critical. And, they are fixed! You can't change the amount or divison of memory - only which template is used. As we want to use our Switch not only as Layer 2 Device, but DualStack Router, we need to change to an DualStack Routing SDM template - to really support IPv6 and Routing.

 

sdm prefer dual-ipv4-and-ipv6 routing

 

is the right command for this. You need to reload your switch after this change!
After the reload, we got our "router" ready. So, lets start by setting up our SW1 as EIGRPv4 Router. First we go to our Client Interface (f0/3):

 

conf t 

int vlan 3
ip address 192.168.2.1 255.255.255.0
no shut

int f0/3
switchport access vlan 3
spanning-tree portfast
no shut

end

 

So, thats our Gateway Link were we'll connect our Windows XP SP3 PC with IPv4 (192.168.2.2 / 24, GW is 192.168.2.1).
Now to the "Router Link" which does connect to the Core Router (g0/1):

 

conf t
int vlan 1
ip address 192.168.0.1 255.255.255.0
no shut
int g0/1
switchport mode access
switchport access vlan 1
switchport nonegotiate
no shut
end

 

This link is set to nonegotiate because we do not want the Switches to start an trunk - but route our traffic through EIGRPv4.
Now, we enable EIGRPv4:

 

conf t
router eigrp 1
network 192.168.0.0
network 192.168.2.0
end

 

Basically, thats very easy to understand: We start the EIGRPv4 proccess, issuing it the ID "1" (you can choose which you want and need to create EIGRP Borders) and inform it about the direct connected networks on this router (192.168.0.0 on g0/1 and 192.168.2.0 on f0/3). And thats it! If you got EIGRPv4 running on Core and SW2 as well (SW2 with same config as shown here, only different IP Networks, Core is going to have the same configuration as the Uplink Port but with IPv4 192.168.0.2 on the SW1 end and 192.168.4.2 on the SW2 end), you'll have an running EIGRP supported network!
Now that we got EIGRPv4 running, we'll jump up to EIGRPv6!
Actually, its not to different from the old Version but these:
- You do not configure networks in an central EIGRP process, but do configure EIGRPv6 directly on the interfaces
- You do need to "no shut" the central EIGRPv6 process once, otherwise it won't work.
- There are other differences, but not needed to mention here - check google if you're interested.
For our IPv6 Client Network, we'll start off like this (f0/2):

 

conf t
int vlan 2
ipv6 address 2000:2::1/64
ipv6 eigrp 1
no shut
int f0/2
switchport access vlan 2
spanning-tree portfast
no shut
end

 

As you see, there is not much difference: We do configure an IPv6 Global Unicast on the interface and enable the EIGRPv6 Proccess with ID "1" (does not interfere with EIGRPv4! They do not "see" each other as they are IPv4 or IPv6 only!) on that Interface. Connected - to an Windows XP SP3 PC with IPv6(2000:2::2/64, GW is 2000:2::1).
Now to the "Router Link" which does connect to the Core Router (g0/1), we need to add an IPv6 Address for EIGRPv6 to work:

 

conf t
int vlan 1
ipv6 address 2000:1::1/64
ipv6 eigrp 1
end

 

After that, we need to "no shut" the EIGRPv6 Proccess - enabling it with this command:

 

conf t
ipv6 router eigrp 1
no shut
end

 

Yeah, and thats it!
To just give the additional feedback for Win XP SP3:
To install the IPv6 Stack, you can enter ipv6 install in the Commandline.
To add an IPv6 Address to an interface, enter netsh interface ipv6 add address "INTERFACENAME" IPV6ADDRESS in the Commandline ( i.e. netsh interface ipv6 add address "Lan-Verbindung" 2000:2::2 for our IPv6_SW1 PC).
Finally you can check your connection via ping and ping6 for IPv6.
So, I hope you like it.

 

And for further studies, I attach the config files! ^^

 
 

As final job I did attach the interfaces f0/4, which do run Dual Stack as well - got an IPv4 and IPv6 Address. I configured the Windows XP SP3 PCs with both IPv4 and IPv6 Addresses, and now they are ping each other over both protocols at the same time. The Problem you'll come up soon is following: IPv4 and IPv6 are seperated stacks. No connection between both. So if you got an IPv6 PC, it won't be able to access IPv4 Ressources - and vice versa. So you'll need some kind of "Concentrator" or "Proxy" which will translate between both like an oversized NAT. Actually - thats the Name of that Translator: NAT-PT ("Protocol Translator") - its only one feature that can deliever IPv4-IPv6 Translation, but actually I read somewhere its considered "bad" already, so it might not be a good choice (and End2End Connection will suffer the same problems as your VPN on IPv4 with NAT.). I'm not putting an real solution for that problem here, as I'm diving into the Problem myself - but I wanted to just hint to this.
Thanks!

 

Cisco PIX 506e Software Upgrade

Following the Hardware Upgrade, the Software needs to be renewed, too!
The latest PIX OS running on the 506e would be 6.3.5.125.
You can upgrade to that and use the pdm to configure the Device.
Or... you can go to ASA!
As described here ( http://www.rownet.co.uk/installing-v7-software-on-a-cisco-pix-506e/ ) you can do that quite easily.

But in short:
- Boot into Pix
- Start downloading an pdm file with copy tftp://Your_TFTP_Server_IP_Address/Your_pdmfile_name flash:pdm and disconnect the ethernet cable!
- After disconnecting, the PIX will erase the pdm from your pix, clearing the flash space for pdm
- clear flashfs
- reload your PIX into Monitor Mode / RMON
- activate your interface (interface 0 is E0, interface 1 E1)
- enter ip addr of tftp server ( server ), filename ( tftp ), ip addr of your pix ( address ) and start downloading to pix
- It will boot from tftp into ASA
- Copy ASA Image: copy tftp://Your_TFTP_Server_IP_Address/pix712.bin flash:
- Reload

Done, you got ASA.
No ASDM / PDM anymore, because not enough space, but ASA ^^

Cisco PIX 506e Hardware Upgrade

The Cisco PIX Series has been "the" well known and powerful firewall appliance of the last decade - and for some people, this decade ain't over yet.
Cisco has released the successor "ASA" some years ago, but many people still got a PIX running.
Reasons are simple: The PIX Series has an Appliance for every problem and is rock solid.
My personal experience with that Firewall Series started with a "burned out" PIX 520 (that one is getting its own entry soon ^^).
After that, I got more into Security by doing my CCNA Security studies.
While doing the CCNAS I also bought the smallest PIX, the PIX 501 from Ebay for 70€ or so.
Sometime later, I discovered an dead PIX 506e in my Office - and I just couldn't help myself and started taking it apart.

Shortly after disassembling I stumbeled upon this Blog: http://hackaday.com/2008/09/28/upgrading-the-cisco-pix-506e/
And that looked very promissing.

Cisco started upgrading some PIX Appliances with new Software Versions.
6.3.5 was the last "PIX" Software. After that, Version 7 and Version 8 were "ASA".
( While 7 was more some kind of bridge version, at least it feels like it... )
Neither my 501,520 or the 506e do run 7.x or 8.x - at least thats what Cisco does say.
Problem is the amount of memory onboard: 501 and 506e only got 8 MB of Flash - not upgradable.
The 520 got 2 or 16 MB Flash - but ain't supported. Another problem is the amount of RAM and CPU.

So.. I did start on these problems.

First thing after opening the 506e was exchanging the CMOS Battery.
Seriously, I felt as that this PIX was always crashing because of some dead battery.
At least, it give me a better feeling having that thing loaded up again.

After that, I was looking for the RAM: SDRAM, 100 MHz FSB.
I looked through the inventory and found 2 bars, each 256 MB, 133 MHz.
Maybe not the best idea, the 506e is only going on 100 MHz, but worth a try.
And - it did work. So, we got the 32 MB replaced by 512 MB.
I think that should be... enough.

RAM is done, Battery replaced, but what about the CPU?
Good Question!
The 506e is running on an 300 MHz Pentium 2 Celeron (SL36A, Mendocino Core, 128 KByte L2, 2V) - not really much.
So I was looking around again and found an awesome 1 GHz Pentium 3 (SL4C8, Coppermine, 256 KByte L2, 1.7V).
After pluging in and testing I found out that this thing was really working. Problem were the 133 MHz FSB - so the CPU did only run with 750 MHz - more than enough!
I was seriously happy, but a problem was coming up: Heat.

The PIX 506e enclosing is really badly build: The CPU Cooler is just sitting some milimeters under the hood, not ventilationholes anywhere except at the end of the case. You can even see some dust burned into the case inlay above the cpu cooler... "nice". So - the new CPU would be really too much for this case. And my idea was correct: Some minutes after closing the case and running the firewall - the CPU got shutdown because of thermal problems. Ok! What to do now? Well - solution was easy: Just cut out the steel enclosing above the CPU Cooler, get some special cloth above it - so nobody would touch in - and close the case. Problem solved.

The next question came up: Well, what does that "little tweaking" really did to the firewall?
Solution to that: Benchmark!
I fired up iperf / jperf with following command:
bin/iperf.exe -c SERVERIP -P 4 -i 20 -p 5001 -w 512.0k -l 512.0k -f m -t 3600
1 Hours, special Packet Size, 4 Parallel Threads. That should "burn-in"....
...and it did: After 20 Minutes with really superior performance ( CPU on 10% by delivering 92 MBit/s! ) the CPU died.
And the powersupply? Well - felt like on fire, too. Damn.

So, the CPU seemed to be too much for the little firewall...
But I didn't wanted to stop there.
The next burn in with the old CPU took place.
Everything was fine, nothing was hurt except the dead CPU.
The firewall was running hour on hour smooth with 100% Network Stress.
( With the 300 MHz CPU, the PIX was already working above 40%... well,... not as good )
Ok.

After stumbeling around in my cases, I finally found the best CPU I came up with at the moment:
SL3XY, Coppermine, 256 KByte L2, 1,65V - an Pentium 3 with 733 MHz.
And I didn't even knew wheter that thing was still working.
I really thought I fried it already some years earlier...
Well, it seemed like... not!

In the end, the Firewall did work at 550 MHz ( 133 MHz FSB aswell... ) for over 4 hours,
26% CPU Load - nice! I think thats ok (The power supply also stayed reasonably cold).

The last thing I did was installing an passiv Heatsink on the AGP Chipset of the PIX.
It was getting hot for no reason... So.... some better cooling than the naked Chip itself is always nice...

So - that was the Hardware. But what about the PIX OS?
Mh...

Next Post 😉

[79XX] Cisco IP Phone Factory Reset & Reboot

To Factory Reset a Cisco IP Phone do the following:
1. Press and hold #, disconnect and reconnect Power
2. The Phone will check through the Line LEDs, on an 7911 or similar it will just light up the both Menu Keys and the Reciever \ Handset LED - after that, let go of #
3. Enter following: 1 2 3 4 5 6 7 8 9 * 0 #
4. The Phone should start up and grab the latest config from your Callmanager

To just Reboot a Cisco IP Phone do following:
1. Go into any Menu
2. Enter #
3. Phone does reload
( 7940, 7960: Hold *,6,Settings )

To open up the Configuration on a Phone:
1. Go into Menu Setting
2. Enter **##

To Factory Reset and erase the firmware on a Cisco IP Phone do the following:
1. Press and hold #, disconnect and reconnect Power
2. The Phone will check through the Line LEDs, on an 7911 or similar it will just light up the both Menu Keys and the Reciever \ Handset LED - after that, let go of #
3. Enter following: 3 4 9 1 6 7 2 8 5 0 * #
4. The Phone should start up and grab the latest Firmware from your Callmanager