Pages

Friday, September 8, 2017

Juniper Space Security Director Policy Hit Counts Not Updated Automatically


Issue Symptons:
  • Normally, each firewall rule on the SRX auto-updates a snmp counter for hit-count, regardless of whether 'count' is configured or not.  Juniper Space Security Director periodically polls these OIDs and updates the hit-count.   
  • In Junper Space 16.1 R1, the issue found is unable to view policy hit counts from Juniper Space Security Director, but SRX itself is keep updating. 

Actions Taken:
  • Verify Security Appliance Policy Hits from Command line
root@fw-mgmt-2> show security policies hit-count 
node1:
--------------------------------------------------------------------------

Logical system: root-logical-system
Index From zone To zone Name
Policy count
1 Vlan2 Vlan1 Baramondi_Monitor 0
2 Vlan2 Vlan1 10 4428
3 Vlan2 Vlan1 50 0
4 Vlan2 Vlan1 40 11136
5 Vlan2 Vlan1 default-logdrop 0
6 Vlan2 Vlan1 53 2007
7 Vlan2 Vlan1 54 0
8 Vlan2 Vlan1 55 0
9 Vlan2 MGMT 6 538
10 Vlan2 MGMT 23 0
11 Vlan2 MGMT 74 2
12 Vlan2 MGMT default-logdrop 81
13 Office Vlan1 default-logdrop 0
14 Office Vlan1 60 447
15 Office Vlan1 Office_Archive 0
16 Office Vlan1 58 0
17 Office Vlan1 Baramondi_Monitor-1 0
18 Office MGMT Office_Archive-1 0
19 Office MGMT default-logdrop 0
20 Vlan1 Vlan2 Baramondi_Rules 0
21 Vlan1 Vlan2 VA 0
22 Vlan1 Vlan2 A_Office_2_Vlan2 292
23 Vlan1 Vlan2 default-logdrop 1696
24 Vlan1 Office VA-1 0
25 Vlan1 Office Baramondi_Rules-1 0
26 Vlan1 Office Device-Zone-1 0
27 Vlan1 Office 4 1299
28 Vlan1 Office default-logdrop 0
........


It is clearly there is hit counts on SRX itself, but they are not being pulled/pushed into Space. Log collecter has beenconfigured and it is receiving logs from this SRX.
Solutions: 

  • Let Juniper Space manually probe latest policy hits
From Firewall Policies page, you can manually 


  • This is a known issue reported in 16.1R1 version and the workaround to resolve this issue is the following:

From Space CLI run the following command:

mysql -ujboss -pnetscreen sm_db -e "update RuntimePreferencesEntity set value='2' where name='POLICY_HIT_COUNT_MAX_SUBJOBS_PER_NODE';" 




Space release 16.1R2.7 (381623)

Last login: Fri Sep 8 15:27:35 2017 from 10.94.200.168

Welcome to the Junos Space network settings utility.

Initializing, please wait


Junos Space Settings Menu

1> Change Password
2> Change Network Settings
3> Change Time Options
4> Retrieve Logs
5> Security
6> Expand VM Drive Size
7> (Debug) run shell

A> Apply changes
Q> Quit
R> Redraw Menu

Choice [1-7,AQR]: 7

[sudo] password for admin:
[root@space-c6186f1b3edb ~]#
[root@space-c6186f1b3edb ~]#
[root@space-c6186f1b3edb ~]#
[root@space-c6186f1b3edb ~]#
mysql -ujboss -pnetscreen sm_db -e "update RuntimePreferencesEntity set value='2' where name='POLICY_HIT_COUNT_MAX_SUBJOBS_PER_NODE';"
Warning: Using a password on the command line interface can be insecure.
[root@space-c6186f1b3edb ~]#





    • RESTART JBOSS:

First, identify which node has the vip (eth0:0).  Check each node with the following:
# ifconfig eth0:0
Recommended restart process (only if JBOSS restart is needed):
1.     Stop processes on all JBOSS nodes
# service jmp-watchdog stop
#
servicejboss stop
#
service jboss-dc stop
Note: jboss-dc is only active on VIP node, but the command won't do any harm

2.     Confirm if JBOSS has been turned off
# service jboss status

3.     Start services on the vip node
# service jmp-watchdog start
Note: watchdog starts other processes

4.     After you can login, start services on all of the other nodes
# service jmp-watchdog start



[root@space-c6186f1b3edb ~]# service jmp-watchdog stop
[root@space-c6186f1b3edb ~]# service jboss stop
found and stop jboss
[root@space-c6186f1b3edb ~]# service jboss-dc stop
stop domain controller
[root@space-c6186f1b3edb ~]# service jboss status
jboss is stopped
[root@space-c6186f1b3edb ~]# service jmp-watchdog start
jmp-watchdog running
[root@space-c6186f1b3edb ~]# ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr C6:18:6F:1B:3E:DB
inet addr:10.94.200.22 Bcast:10.94.200.127 Mask:255.255.255.128
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:145




Notes: Just got confirmation. It is a bug and it wont be fixed until Juniper Space 17.2.



Wednesday, September 6, 2017

Cisco IOS Command Tips and Tricks - Part 2

Cisco IOS command list is getting longer , and it has been split into two posts:

21. Auto secure

Cisco also provides a One-step lockdown-like feature at the command line! This feature is called AutoSecure. It uses the command shown below:

auto secure [management | forwarding] [no-interact | full] [ntp | login | ssh | firewall | tcp-intercept]


22. Change Site-to-Site VPN Idle time out to 5 minutes

For IOS Router

R1(config)#crypto ipsec security-association idle-time 300


For ASA

ASA1(config)#group-policy GP_1.1.1.2 attributes
ASA1(config-group-policy)#vpn-idle-timeout 300

ASA1(config-group-policy)#vpn-session-timeout none

23. Find Cisco Devices Serial Number
Usually we could use 'show version' command to find out SN , but for Nexus you will have to use 'show inventory' to get Chassis SN

Nexus1# show inventory
NAME: "Chassis", DESCR: "Nexus5548 Chassis"
PID: N5K-C5548UP       , VID: V01 , SN: SSI163604J

NAME: "Module 1", DESCR: "O2 32X10GE/Modular Universal Platform Supervisor"
PID: N5K-C5548UP       , VID: V01 , SN: FOC1645D5F

NAME: "Fan 1", DESCR: "Chassis fan module"
PID: N5548P-FAN        , VID: N/A , SN: N/A

NAME: "Fan 2", DESCR: "Chassis fan module"
PID: N5548P-FAN        , VID: N/A , SN: N/A

NAME: "Power supply 1", DESCR: "AC power supply"
PID: N55-PAC-750W      , VID: V02 , SN: ART16310D6

NAME: "Power supply 2", DESCR: "AC power supply"
PID: N55-PAC-750W      , VID: V02 , SN: ART16310SQ

NAME: "Module 3", DESCR: "O2 Daughter Card with L3 ASIC"
PID: N55-D160L3-V2     , VID: V01 , SN: FOC1637NT8

NAME: "FEX 101 CHASSIS", DESCR: "N2K-C2248TP-1GE  CHASSIS"
PID: N2K-C2248TP-1GE   , VID: V03 , SN: SSI1637FHK

NAME: "FEX 101 Module 1", DESCR: "Fabric Extender Module: 48x1GE, 4x10GE Supervi
sor"
PID: N2K-C2248TP-1GE   , VID: V03 , SN: FOC1645WH9

NAME: "FEX 101 Fan 1", DESCR: "Fabric Extender Fan module"
PID: N2K-C2248-FAN     , VID: N/A , SN: N/A

NAME: "FEX 101 Power Supply 1", DESCR: "Fabric Extender AC power supply"
PID: N2200-PAC-400W    , VID: V04 , SN: LIT16390H9

NAME: "FEX 101 Power Supply 2", DESCR: "Fabric Extender AC power supply"
PID: N2200-PAC-400W    , VID: V04 , SN: LIT16390HH

NAME: "FEX 102 CHASSIS", DESCR: "N2K-C2248TP-1GE  CHASSIS"
PID: N2K-C2248TP-1GE   , VID: V03 , SN: SSI16390DJ

24. Show Commands Information

Flash: IOS  
RAM: All tables and configuration when router is running. Info will  be lost when power off.
ROM: POST, Bootstrap, Mini-IOS
NVRAM:Startup-config

25. SYSLOG Writing to Local Disk or Flash Card


Router(config)# logging persistent url disk0:/syslog size 134217728 filesize 16384        
%Warning: the ratio between logging persistent size and logging persistent filesize is 134217728/16384, suggested ratio is less than 196 in order to achieve good system performace

Router(config)# logging persistent url disk0:/syslog size 134217728 filesize 1342177  


Router#sh log 

Syslog logging: enabled (0 messages dropped, 9 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.




No Inactive Message Discriminator.



    Console logging: disabled

    Monitor logging: level debugging, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level debugging, 8665 messages logged, xml disabled,
                    filtering disabled
    Exception Logging: size (4096 bytes)
    Count and timestamp logging messages: disabled
    Persistent logging: enabled, url disk0:/syslog, disk space 134217728 bytes, file size 1342177 bytes, batch size 4096 bytes

No active filter modules.


    Trap logging: level debugging, 8669 message lines logged

        Logging to 10.2.2.3  (udp port 514, audit disabled,
              link up),
              8668 message lines logged, 
              0 message lines rate-limited, 
              0 message lines dropped-by-MD, 
              xml disabled, sequence number disabled
              filtering disabled
        Logging Source-Interface:       VRF Name:
        Loopback0                       

Log Buffer (8192 bytes):


12 EST: %C7600_PLATFORM-SP-3-LOW_BATT: Low Voltage detected for NVRAM Battery
........[Omitted]

Router# copy disk0:/syslog ftp://myuser:mypass@192.168.1.129/syslog




26. Check Routers Power Supply Status


Router#show environment 
SYSTEM POWER SUPPLY STATUS
==========================
 Internal Power Supply 1 Type: AC
 Internal Power Supply 1 12V Output Status: Normal

 Internal Power Supply 2 Type: AC

 Internal Power Supply 2 12V Output Status: Fail

SYSTEM FAN STATUS

=================
 Fan 1 OK, Low speed setting
 Fan 2 OK, Low speed setting
 Fan 3 OK, Low speed setting
 Fan 4 OK, Low speed setting
 Fan 5 OK, Low speed setting

SYSTEM TEMPERATURE STATUS

=========================
 Intake Left temperature: 25 Celsius, Normal
 Intake Right temperature: 22 Celsius, Normal
 Exhaust Right temperature: 29 Celsius, Normal
 Exhaust Left temperature: 31 Celsius, Normal
 CPU temperature: 24 Celsius, Normal
 Power Supply Unit 1 temperature: 24 Celsius, Normal
          
REAL TIME CLOCK BATTERY STATUS
==============================
 Battery OK (checked at power up)

CPU CORE TEMPERATURE STATUS

===========================
  CPU Core current reading, target = 24, reading = 62 : Normal
  CPU Core out of target history in past 1045868.0 mins:
     Consecutive out-of-range mins high water mark: 0.0
     Total minutes out-of-range: 0.0
     Last 5 minutes out-of-range: 0.0
     Last 5 minutes consecutive out-of-range: 0.0
     Last 5 minutes read failure count: 0


SYSTEM WATTAGE

===============
 Motherboard, EHWIC, PVDM, Power Supply and Fan Power consumption = 123.1 W
 Total System Power consumption is: 123.1 W

 Environmental information last updated 00:00:11 ago


27. Portchannel Adding vlan without downtime

For trunk links, it is best to take out portfast and bpduguard from interface configuration.

no spanning-tree portfast
no spanning-tree bpduguard enable


Always add or remove VLANs from the port-channel interface since the individual physical links will inherit the VLAN set automatically.  If added a VLAN to a phyiscal  layer2 interface instead of the logical layer2 port-channel, it will bring the etherchannel down.

For example, to add vlan 50 to etherchannel 3:
conf t
interface port-channel 3
switchport trunk allowed vlan add 50
end
show run interface gi0/3
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2-49
switchport mode trunk
channel-group 3 mode on


The last line "channel-group 1 mode on" indicates that this port is part of etherchannel 3.  If you try to add a VLAN 50 to the etherchannel by  adding the VLAN 50 to the physical port g0/3, this port will be removed from  the etherchannel 3 and this will cause spanning tree to throw a wobbly.


28. Cisco Switch Lights Meaning

SYSTEM(SYST) Light

Overall status of the switch.

  • Off: Switch is not powered on
  • Green: Switch is working fine
  • Amber: Switch is powered on but faulty


REDUNDANT POWER SUPPLY(RPS) Light

Provides backup power to the switch if the main supply goes off.

  • Off: No RPS available,
  • Green: RPS is working fine
  • Blinking Green: Providing backup to some other device
  • Amber: RPS is faulty
  • Flashing Amber: RPS is providing backup(primary power off)


DUPLEX

Duplex status of the switch ports.

  • Off: Switch port is half duplex
  • Green: Switch port is full duplex


UTIL

Utilization status of the switch ports.


SPEED

Speed status of the switch ports.

  • Off: Switch port is operating at 10Mbps
  • Green: Switch port is operating at 100Mbps
  • Flashing green: Switch port is operating at 1000Mbps


STAT

Status of the switch ports.

  • Off: No device connected/port is administratively down.
  • Green: Device is connected.
  • Blinking green: Port is sending/receiving data.
  • Alternating green amber: Fault in link/Frames experiencing error
  • Amber: Port is blocked by Spanning Tree Protocol




Tuesday, September 5, 2017

Troubleshooting Cisco IPSec Site to Site VPN - "QM Rejected"

There was a VPN issue to troubleshoot recently. It was between Juniper SRX and Cisco Router. It seems straightforward but it took quite a long time to troubleshoot because of communication. All steps listed here for my future reference.

Some other related posts:

Diagram

1. Enabled Debugging on Cisco IOS Router

vpn-R1#
debug crypto ipsec
Crypto IPSEC debugging is on

vpn-R1#
debug crypto isakmp
Crypto ISAKMP debugging is on

vpn-R1#
debug crypto engine
Crypto Engine debugging is on

vpn-R1#terminal monitor







2. Debugging results
It shows invalidated proposal and ISAKMP deleted node with reason "QM rejected"

003345: Sep 5 10:49:31.279 EDT: ISAKMP (1060): received packet from 20.20.34.50 dport 500 sport 500 Global (R) QM_IDLE
003346: Sep 5 10:49:31.279 EDT: ISAKMP: set new node 1450843075 to QM_IDLE
003347: Sep 5 10:49:31.279 EDT: ISAKMP:(1060): processing HASH payload. message ID = 1450843075
003348: Sep 5 10:49:31.279 EDT: ISAKMP:(1060): processing SA payload. message ID = 1450843075
003349: Sep 5 10:49:31.279 EDT: ISAKMP:(1060):Checking IPSec proposal 1
003350: Sep 5 10:49:31.279 EDT: ISAKMP: transform 0, ESP_3DES
003351: Sep 5 10:49:31.279 EDT: ISAKMP: attributes in transform:
003352: Sep 5 10:49:31.279 EDT: ISAKMP: authenticator is HMAC-SHA
003353: Sep 5 10:49:31.279 EDT: ISAKMP: group is 2
003354: Sep 5 10:49:31.279 EDT: ISAKMP: SA life type in seconds
003355: Sep 5 10:49:31.279 EDT: ISAKMP: SA life duration (VPI) of 0x0 0x0 0xE 0x10
003356: Sep 5 10:49:31.279 EDT: ISAKMP: encaps is 1 (Tunnel)
003357: Sep 5 10:49:31.279 EDT: ISAKMP:(1060):atts are acceptable.
003358: Sep 5 10:49:31.279 EDT: IPSEC(validate_proposal_request): proposal part #1
003359: Sep 5 10:49:31.279 EDT: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 200.153.121.58:0, remote= 20.20.34.50:0,
local_proxy= 10.9.106.18/255.255.255.255/256/0,
remote_proxy= 10.21.51.128/255.255.255.255/256/0,
protocol= ESP, transform= esp-3des esp-sha-hmac (Tunnel),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0


003361: Sep 5 10:49:31.279 EDT: Crypto mapdb : proxy_match
src addr : 10.9.106.18
dst addr : 10.21.51.128

protocol : 0
src port : 0
dst port : 0
.......
003369: Sep 5 10:49:31.283 EDT:
map_db_find_best did not find matching map
003370: Sep 5 10:49:31.283 EDT: IPSEC(ipsec_process_proposal):
proxy identities not supported
003371: Sep 5 10:49:31.283 EDT: ISAKMP:(1060):
IPSec policy invalidated proposal with error 32
003372: Sep 5 10:49:31.283 EDT: ISAKMP:(1060):
phase 2 SA policy not acceptable! (local 20.15.121.58 remote 20.20.34.50)
003373: Sep 5 10:49:31.283 EDT: ISAKMP: set new node -2087949406 to QM_IDLE
003374: Sep 5 10:49:31.283 EDT: ISAKMP:(1060):Sending NOTIFY PROPOSAL_NOT_CHOSEN protocol 3
spi 739556696, message ID = 2207017890
003375: Sep 5 10:49:31.283 EDT: ISAKMP:(1060): sending packet to 20.20.34.50 my_port 500 peer_port 500 (R) QM_IDLE
003376: Sep 5 10:49:31.283 EDT: ISAKMP:(1060):Sending an IKE IPv4 Packet.
003377: Sep 5 10:49:31.283 EDT: ISAKMP:(1060):purging node -2087949406
003378: Sep 5 10:49:31.283 EDT: ISAKMP:(1060):deleting node 1450843075 error TRUE reason
"QM rejected"
003379: Sep 5 10:49:31.283 EDT: ISAKMP:(1060):Node 1450843075, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
vpn-router#
003380: Sep 5 10:49:31.283 EDT: ISAKMP:(1060):Old State = IKE_QM_READY New State = IKE_QM_READY

003382: Sep  5 10:50:21.284 EDT: ISAKMP:(1060):purging node 1450843075




3. Checking configuration

Cisco Router's crypto map

vpn-router#
show crypto map
Interfaces using crypto map NiStTeSt1:

Crypto Map IPv4 "vpn" 20 ipsec-isakmp
Description: VPN to C
Peer = 20.20.34.50
Extended IP access list C-VPN-List
access-list C-VPN-List permit ip
host 10.9.106.18 host 10.1.254.19
Current peer: 20.20.34.50
Security association lifetime: 4608000 kilobytes/3600 seconds
Responder-Only (Y/N): N
PFS (Y/N): Y
DH group: group2
Mixed-mode : Disabled
Transform sets={
Conductor: { esp-3des esp-sha-hmac } ,
}






Peer Juniper SRX 550 configuraiton:

set interfaces st0 unit 22 description Cisco-R
set security ike policy ike-policy-Cisco-R mode main
set security ike policy ike-policy-Cisco-R proposals ike-proposal-Padrao
set security ike policy ike-policy-Cisco-R pre-shared-key ascii-text "$APfTz0O"
set security ike gateway ike-gate-Cisco-R ike-policy ike-policy-Cisco-R
set security ike gateway ike-gate-Cisco-R address 20.15.121.58
set security ike gateway ike-gate-Cisco-R remote-identity inet 20.15.121.58
set security ike gateway ike-gate-Cisco-R external-interface reth0.0
set security ipsec vpn ipsec-vpn-Cisco-R bind-interface st0.22
set security ipsec vpn ipsec-vpn-Cisco-R ike gateway ike-gate-Cisco-R
set security ipsec vpn ipsec-vpn-Cisco-R
ike proxy-identity local 10.21.51.128/32
set security ipsec vpn ipsec-vpn-Cisco-R ike proxy-identity remote 10.9.106.18/32
set security ipsec vpn ipsec-vpn-Cisco-R ike proxy-identity service any
set security ipsec vpn ipsec-vpn-Cisco-R ike ipsec-policy ipsec-policy-P
set security ipsec vpn ipsec-vpn-Cisco-R establish-tunnels immediately




set security ike proposal ike-proposal-P authentication-method pre-shared-keys
set security ike proposal ike-proposal-P dh-group group2
set security ike proposal ike-proposal-P authentication-algorithm sha1
set security ike proposal ike-proposal-P encryption-algorithm 3des-cbc
set security ike proposal ike-proposal-P lifetime-seconds 28800


set security ipsec policy ipsec-policy-P perfect-forward-secrecy keys group2
set security ipsec policy ipsec-policy-P proposals ipsec-proposal-Padrao
set security ipsec proposal ipsec-proposal-P protocol esp
set security ipsec proposal ipsec-proposal-P authentication-algorithm hmac-sha1-96
set security ipsec proposal ipsec-proposal-P encryption-algorithm 3des-cbc
set security ipsec proposal ipsec-proposal-P lifetime-seconds 3600




Cisco IOS Router 1900 Configuration:

crypto isakmp policy 2
encr 3des
authentication pre-share
group 2
lifetime 28800
!


crypto isakmp key 76tyYuty!2@ address 20.13.194.17
!crypto isakmp aggressive-mode disable

crypto ipsec transform-set C esp-3des esp-sha-hmac
mode tunnel


crypto map vpn 20 ipsec-isakmp
description VPN to C
set peer 20.20.34.50
set transform-set Conductor
set pfs group2
match address C-VPN-List


ip access-list extended C-VPN-List
permit ip host 10.9.106.18
host 10.1.254.19

interface GigabitEthernet0/0
ip address 20.15.121.58 255.255.255.248
duplex auto
speed auto
crypto map vpn




Solution:
It clearly shows access-list not matching on both end. Juniper SRX has a NAT from 10.1.254.19 to 10.21.51.128. Cisco router only knows their Internal IP 10.1.254.19. Once Cisco router changed access-list to match Juniper. Tunnel is up right away.

Wednesday, August 23, 2017

Cisco Wireless LAN Controller Redundancy Solutions: High Availability

There are two options for Cisco Wireless Controller redundancy solutions, either Backup Controllers or High Availability, depending on the firmware version of WLC's, failover time requirement, and budget.


Using Backup Controller method, a single controller at another location can act as a backup for access points when they lose connectivity with the primary controller in the local region. Centralized and regional controllers do not need to be in the same mobility group. You can specify a primary, secondary, and tertiary controller for specific access points in your network. Using the controller GUI or CLI, you can specify the IP addresses of the backup controllers, which allows the access points to fail over to controllers outside of the mobility group. You can set the Primary and Secondary controllers for the AP on the controller via the GUI, the CLI, or even SNMP. With Backup Controllers, in the case of a WLC failure, APs would begin to search for their Secondary Controller and re-establish their CAPWAP tunnel. The obvious downside is the outage that occurs from the client prospective while the AP drops it's tunnel and begins to build it again to the Secondary Controller.

The new High Availability (HA) feature (that is, AP SSO) set within the Cisco Unified Wireless Network software release version 7.3 and 7.4 allows the access point (AP) to establish a CAPWAP tunnel with the Active WLC and share a mirror copy of the AP database with the Standby WLC. The APs do not go into the Discovery state when the Active WLC fails and the Standby WLC takes over the network as the Active WLC. There is only one CAPWAP tunnel maintained at a time between the APs and the WLC that is in an Active state. The overall goal for the addition of AP SSO support to the Cisco Unified Wireless LAN is to reduce major downtime in wireless networks due to failure conditions that may occur due to box failover or network failover. Once you purchase a second WLC and license it specifically to serve as a standby,  it shares an IP address and session/Config/AP information with the main controller.

Relate Posts:



Now with all of that said, which is better? Based on some professional opinions I found from Internet:

"Backup Controllers is the cheaper way if your existing 5508's at different networks have enough available capacity to carry the load of either site, and your business can tolerate a few minutes of downtime in the event of a WLC failure. In that case, simply configure your secondary controllers on each Access Point, and off you go. Note that there is some more management overhead with using Backup Controllers, you will have to configure VLANs/Interfaces for all of your SSID's in each network, make your AP Groups on each controller if you use them, etc. Now, when the WLC goes down at their site, users will experience some downtime as the AP's migrate, but at least they're not down hard.

High Availability becomes a reasonable solution here if you don't have capacity on your existing 5508's and/or your business can't tolerate the failover time in the Backup Controllers method. In other words, if you don't have capacity on your existing controller to use them as a backup for each other, and you're going to have to spend some money anyways, I would recommend looking into the High Availability solution and pricing."

Topology: 


WLC-1 Active : (It is already in Production)
Management: 10.9.1.10/24
Redundancy-MGNT: 10.9.1.22/24
Service-Port: 10.9.20.30 /24
Virtual: 2.2.2.2
WLC-2 Secondary
Management: 10.9.1.21/24 (This is only used temporary to complete configuration)
Redundancy-MGNT: 10.9.1.23 /24
Service-Port: 10.9.20.31 /24
Virtual: 2.2.2.2 (needs to be the same as the Active Unit)

Some Terms: 

Redundancy Management Interface

The IP address on this interface should be configured in the same subnet as the management interface. This interface will check the health of the Active WLC via network infrastructure once the Active WLC does not respond to Keepalive messages on the Redundant Port. This provides an additional health check of the network and Active WLC, and confirms if switchover should or should not be executed. Also, the Standby WLC uses this interface in order to source ICMP ping packets to check gateway reachability. This interface is also used in order to send notifications from the Active WLC to the Standby WLC in the event of Box failure or Manual Reset. The Standby WLC will use this interface in order to communicate to Syslog, the NTP server, and the TFTP server for any configuration upload.

Redundancy Port

This interface has a very important role in the new HA architecture. Bulk configuration during boot up and incremental configuration are synced from the Active WLC to the Standby WLC using the Redundant Port. WLCs in a HA setup will use this port to perform HA role negotiation. The Redundancy Port is also used in order to check peer reachability sending UDP keep-alive messages every 100 msec (default timer) from the Standby WLC to the Active WLC. Also, in the event of a box failure, the Active WLC will send notification to the Standby WLC via the Redundant Port. If the NTP server is not configured, a manual time sync is performed from the Active WLC to the Standby WLC on the Redundant Port. This port in case of standalone controller and redundancy VLAN in case of WISM-2 will be assigned an auto generated IP Address where last 2 octets are picked from the last 2 octets of Redundancy Management Interface (the first 2 octets are always 169.254).

Note: After HA enabled, you only have console access to both WLCs using the console port. Only the primary HA you have access to GUI. When AP SSO is enabled, there is no SNMP access on the service port for both the WLCs in the HA setup. You either console into the WLCs or you https/ssh to the primary HA. The primary is the only one that can be reachable using http/https/telnet/ssh with AP SSO enabled.
Initial Configuration

System Name [Cisco_e3:7b:64] (31 characters max): 
AUTO-INSTALL: no interfaces registered.

AUTO-INSTALL: process terminated -- no configuration loaded

Enter Administrative User Name (24 characters max):
admin
Enter Administrative Password (3 to 24 characters):
**********
Re-enter Administrative Password : **********


Service Interface IP Address Configuration [static][DHCP]:
static
Service Interface IP Address:
10.9.20.31
Service Interface Netmask: 255.255.255.128

Enable Link Aggregation (LAG) [yes][NO]: no

Management Interface IP Address:
10.9.1.21
Management Interface Netmask: 255.255.255.0
Management Interface Default Router:
10.9.1.1
Cleaning up Provisioning SSID
Management Interface VLAN Identifier (0 = untagged):
Management Interface Port Num [1 to 8]: 1
Management Interface DHCP Server IP Address:
Invalid response

Management Interface DHCP Server IP Address:
10.9.1.2

Enable HA [yes][NO]: no

Virtual Gateway IP Address:
2.2.2.2

Mobility/RF Group Name:
Invalid response


Mobility/RF Group Name:
test

Network Name (SSID):
Invalid response


Network Name (SSID): gdsci-commercial

Configure DHCP Bridging Mode [yes][NO]: no

Allow Static IP Addresses [YES][no]: yes

Configure a RADIUS Server now? [YES][no]: no
Warning! The default WLAN security policy requires a RADIUS server.
Please see documentation for more details.

Enter Country Code list (enter 'help' for a list of countries) [US]:

Enable 802.11b Network [YES][no]: n
Invalid response


Enable 802.11b Network [YES][no]: no
Enable 802.11a Network [YES][no]: no
Enable Auto-RF [YES][no]: no

Configure a NTP server now? [YES][no]: no
Configure the system time now? [YES][no]: no

Warning! No AP will come up unless the time is set.
Please see documentation for more details.

Would you like to configure IPv6 parameters[YES][no]: no

Configuration correct? If yes, system will save it and reset. [yes][NO]: yes

Configuration saved!
Resetting system with new configuration...





Configuration HA Steps:

Physical Ports and Connections on both WLC 5508s:

1. Configure Interface IP addresses based on Plan

Active WLC-1: (It is already in production)

Secondary WLC-2: ( new Added)


2. Controller Redundancy Global Configuration

Active WLC-1: (It is already in production)



Secondary WLC-2: ( new Added)


3. Connect Redundancy Port Cable



4. Change SSO configuration on primary WLC


It will save your configuration and reboot it.
(Cisco Controller) 

Enter User Name (or 'Recover-Config' this one-time only to reset configuration to factory defaults)

User: Creating license client restartability thread

Exit Called
Switchdrvr exited!
Restarting system.


WLCNG Boot Loader Version 1.0.20 (Built on Jan 9 2014 at 19:02:44 by cisco)
Board Revision 1.3 (SN: FCW2017B091, Type: AIR-CT5508-K9) (G)

Verifying boot loader integrity... OK.

OCTEON CN5645-NSP pass 2.1, Core clock: 600 MHz, DDR clock: 330 MHz (660 Mhz data rate)
FPGA Revision 1.7
Env FW Revision 1.8
USB Console Revision 2.2
CPU Cores: 10
DRAM: 1024 MB
Flash: 32 MB
Clearing DRAM........ done
Network: octeth0', octeth1
' - Active interface
E - Environment MAC address override
CF Bus 0 (IDE): OK
IDE device 0:
- Model: SGEFD1GHB9P1D221 Firm: FW981 Ser#: STP194512FP
- Type: Hard Disk
- Capacity: 977.4 MB = 0.9 GB (2001888 x 512)


Press <ESC> now to access the Boot Menu...

Loading primary image (8.2.151.0)
100%

36899306 bytes read
Launching...
init started: BusyBox v1.6.0 (2010-05-13 17:50:10 EDT) multi-call binary
starting pid 847, tty '': '/etc/init.d/rcS'
Decompressing... OK
Validating...... OK
Set PLX switch MPS settings .............!!!!!!!
Detecting Hardware ...
set smp_affinity for irq 48
003f
DP from CGE5.0 ...
starting pid 1156, tty '/dev/ttyS0': '/usr/bin/gettyOrMwar'
Setting up ZVM
Exporting LD_LIBRARY_PATH

Cryptographic library self-test....
Testing SHA1 Short Message 1
Testing SHA256 Short Message 1
Testing SHA1 Short Message 1
SHA1 POST PASSED
passed!

XML config selected
Validating XML configuration
octeon_device_init: found 1 DPs
readCPUConfigData: cardid 0x6070001
Cisco is a trademark of Cisco Systems, Inc.
Software Copyright Cisco Systems, Inc. All rights reserved.

Cisco AireOS Version 8.2.151.0
Firmware Version FPGA 1.7, Env 1.8, USB console 2.2
Initializing OS Services: ok
Initializing Serial Services: ok
Initializing Network Services: ok
Initializing Licensing Services: ok

License daemon start initialization.....

License daemon running.....
Starting Statistics Service: ok
Starting Licensing Services: ok
Starting ARP Services: ok
Starting Trap Manager: ok
Starting Network Interface Management Services: ok
Starting System Services: ok
Starting FIPS Features: ok : Not enabled
Starting SNMP services: ok
Starting Fastpath Hardware Acceleration: ok
Starting Fastpath Console redirect : ok
Starting Fastpath DP Heartbeat : ok
Fastpath CPU0.00: Starting Fastpath Application. SDK-1.8.0, build 269. Flags-[DUTY CYCLE] : ok
Fastpath CPU0.00: Initializing last packet received queue. Num of cores(10)
Fastpath CPU0.00: Init MBUF size: 1856, Subsequent MBUF size: 2041
Fastpath CPU0.00: Core 0 Initialization and FIPS self-test: ok
Fastpath CPU0.00: 10 Cores are being initialized
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing Timer...done.
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing NBAR AGING Timer...done.
Fastpath CPU0.00: Received instruction to get link status
Fastpath CPU0.01: Core 1 Initialization and FIPS self-test: ok
Fastpath CPU0.02: Core 2 Initialization and FIPS self-test: ok
Fastpath CPU0.03: Core 3 Initialization and FIPS self-test: ok
Fastpath CPU0.04: Core 4 Initialization and FIPS self-test: ok
Fastpath CPU0.05: Core 5 Initialization and FIPS self-test: ok
Fastpath CPU0.06: Core 6 Initialization and FIPS self-test: ok
Fastpath CPU0.07: Core 7 Initialization and FIPS self-test: ok
Fastpath CPU0.08: Core 8 Initialization and FIPS self-test: ok
Fastpath CPU0.09: Core 9 Initialization and FIPS self-test: ok
Starting Switching Services: ok
Starting QoS Services: ok
Starting Policy Manager: ok
Starting Data Transport Link Layer: ok
Starting Access Control List Services: ok
Starting System Interfaces: ok
Starting Client Troubleshooting Service: ok
Starting Certificate Database: ok
Starting VPN Services: ok
Starting Management Frame Protection: ok
Starting DNS Services: ok
Starting Redundancy: Starting Peer Search Timer of 120 seconds
Initiate Role Negotiation Message to peer

Role Negotiation timeout. Didn't find Peer..

Peer WLC is not reachable
Negotiation Timeout. Primary controller will become Active-no-Peer
ok

Start rmgrPingTaskok
Starting LWAPP: ok
Starting CAPWAP: ok
Starting LOCP: ok
Starting Security Services: ok
Starting Policy Manager: ok
Starting Authentication Engine: ok
Starting Mobility Management: ok
Starting Ethernet-over-IP: ok
Starting Capwap Ping Component: ok
Starting AVC Services: ok
Starting AVC Flex Services: ok
Starting Virtual AP Services: ok
Starting AireWave Director: ok
Starting Network Time Services: ok
Starting Cisco Discovery Protocol: ok
Starting Broadcast Services: ok
Starting Logging Services: ok
Starting DHCP Server: ok
Starting IDS Signature Manager: ok
Starting RFID Tag Tracking: ok
Starting RF Profiles: ok
Starting Power Supply and Fan Status Monitoring Service: ok
Starting Mesh Services: ok
Starting TSM: ok
Starting CIDS Services: ok
Starting Ethernet-over-IP: ok
Starting DTLS server: enabled in CAPWAP
Starting CleanAir: ok
Starting WIPS: ok
Starting SSHPM LSC PROV LIST: ok
Starting RRC Services: ok
Starting SXP Services: ok
Starting Alarm Services: ok
Starting FMC HS: ok
Starting IPv6 Services: ok
Starting Config Sync Manager : ok
Starting Hotspot Services: ok
Starting PMIP Services: ok
Starting Tunnel Services New: ok
Starting Portal Server Services: ok
Starting mDNS Services: ok
Starting Management Services:
Web Server: CLI: Secure Web: ok

(Cisco Controller)

Enter User Name (or 'Recover-Config' this one-time only to reset configuration to factory defaults)

User:


The error is because of missing SSO configuration on Secondary WLC . After did SSO configuration on Secondary WLC. All goes well.


....
Starting Management Frame Protection: ok
Starting DNS Services: ok
Starting Redundancy: Starting Peer Search Timer of 120 seconds
Initiate Role Negotiation Message to peer
Found the Peer. Starting Role Determination...ok
Start rmgrPingTaskok
Starting LWAPP: ok
....
(Cisco Controller) >show redundancy summary 
Redundancy Mode = SSO ENABLED
Local State = ACTIVE
Peer State = STANDBY HOT
Unit = Primary
Unit ID = 04:62:73:7A:73:E0
Redundancy State = SSO
Mobility MAC = 04:62:73:7A:73:E0
BulkSync Status = Complete
Average Redundancy Peer Reachability Latency = 444 Micro Seconds
Average Management Gateway Reachability Latency = 3572 Micro Seconds

(Cisco Controller) >show redundancy detail 
Redundancy Management IP Address................. 10.9.1.22
Peer Redundancy Management IP Address............ 10.9.1.23
Redundancy Port IP Address....................... 169.254.1.22
Peer Redundancy Port IP Address.................. 169.254.1.23
Peer Service Port IP Address..................... 10.9.20.31


Redundancy Timeout Values.....:
----------------------------------------------------
Keep Alive Timeout : 100 msecs
Peer Search Timeout : 120 secs


Number of Routes................................. 0

Destination Network Netmask Gateway
------------------- ------------------- -------------------

(Cisco Controller) >





After SSO is enabled, WLC is rebooted, and the XML configuration is synced, WLC 1 will
transition its state to Active and WLC 2 will transition its state to Standby HOT. From this point
onwards, GUI/Telnet/SSH for WLC 2 on the management interface will not work, as all the
configurations and management should be done from the Active WLC. If required, the Standby
WLC (WLC 2, in this example) can only be managed via the Console or Service Port.
Also, once the Peer WLC transitions to the Standby Hot state, -Standby keyword is automatically
appended to the Standby WLCs prompt name.


Here is output from Secondary WLC after enabled SSO:
(Cisco Controller) 
User: Creating license client restartability thread

Updating license storage ... Done.

Exit Called
Switchdrvr exited!
Restarting system.


WLCNG Boot Loader Version 1.0.20 (Built on Jan 9 2014 at 19:02:44 by cisco)
Board Revision 1.3 (SN: FCW2114B5DU, Type: AIR-CT5508-K9) (G)

Verifying boot loader integrity... OK.

OCTEON CN5645-NSP pass 2.1, Core clock: 600 MHz, DDR clock: 330 MHz (660 Mhz data rate)
FPGA Revision 1.7
Env FW Revision 1.8
USB Console Revision 2.2
CPU Cores: 10
DRAM: 1024 MB
Flash: 32 MB
Clearing DRAM........ done
Network: octeth0', octeth1
' - Active interface
E - Environment MAC address override
CF Bus 0 (IDE): OK
IDE device 0:
- Model: VRFDFC41024ECEE-P1 Firm: 100511g Ser#: VIR20455730
- Type: Hard Disk
- Capacity: 1000.1 MB = 0.9 GB (2048256 x 512)


Press <ESC> now to access the Boot Menu...

Loading primary image (8.2.151.0)
100%

36899306 bytes read
Launching...
init started: BusyBox v1.6.0 (2010-05-13 17:50:10 EDT) multi-call binary
starting pid 847, tty '': '/etc/init.d/rcS'
Decompressing... OK
Validating...... OK
Set PLX switch MPS settings .............!!!!!!!
Detecting Hardware ...
set smp_affinity for irq 48
003f
DP from CGE5.0 ...
starting pid 1156, tty '/dev/ttyS0': '/usr/bin/gettyOrMwar'
Setting up ZVM
Exporting LD_LIBRARY_PATH

Cryptographic library self-test....
Testing SHA1 Short Message 1
Testing SHA256 Short Message 1
Testing SHA1 Short Message 1
SHA1 POST PASSED
passed!

XML config selected
Validating XML configuration
octeon_device_init: found 1 DPs
readCPUConfigData: cardid 0x6070001
Cisco is a trademark of Cisco Systems, Inc.
Software Copyright Cisco Systems, Inc. All rights reserved.

Cisco AireOS Version 8.2.151.0
Firmware Version FPGA 1.7, Env 1.8, USB console 2.2
Initializing OS Services: ok
Initializing Serial Services: ok
Initializing Network Services: ok
Initializing Licensing Services: ok

License daemon start initialization.....

License daemon running.....
Starting Statistics Service: ok
Starting Licensing Services: ok
Starting ARP Services: ok
Starting Trap Manager: ok
Starting Network Interface Management Services: ok
Starting System Services: ok
Starting FIPS Features: ok : Not enabled
Starting SNMP services: ok
Starting Fastpath Hardware Acceleration: ok
Starting Fastpath Console redirect : ok
Starting Fastpath DP Heartbeat : ok
Fastpath CPU0.00: Starting Fastpath Application. SDK-1.8.0, build 269. Flags-[DUTY CYCLE] : ok
Fastpath CPU0.00: Initializing last packet received queue. Num of cores(10)
Fastpath CPU0.00: Init MBUF size: 1856, Subsequent MBUF size: 2041
Fastpath CPU0.00: Core 0 Initialization and FIPS self-test: ok
Fastpath CPU0.00: 10 Cores are being initialized
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing Timer...done.
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing NBAR AGING Timer...done.
Fastpath CPU0.00: Received instruction to get link status
Fastpath CPU0.01: Core 1 Initialization and FIPS self-test: ok
Fastpath CPU0.02: Core 2 Initialization and FIPS self-test: ok
Fastpath CPU0.03: Core 3 Initialization and FIPS self-test: ok
Fastpath CPU0.04: Core 4 Initialization and FIPS self-test: ok
Fastpath CPU0.05: Core 5 Initialization and FIPS self-test: ok
Fastpath CPU0.06: Core 6 Initialization and FIPS self-test: ok
Fastpath CPU0.07: Core 7 Initialization and FIPS self-test: ok
Fastpath CPU0.08: Core 8 Initialization and FIPS self-test: ok
Fastpath CPU0.09: Core 9 Initialization and FIPS self-test: ok
Starting Switching Services: ok
Starting QoS Services: ok
Starting Policy Manager: ok
Starting Data Transport Link Layer: ok
Starting Access Control List Services: ok
Starting System Interfaces: ok
Starting Client Troubleshooting Service: ok
Starting Certificate Database: ok
Starting VPN Services: ok
Starting Management Frame Protection: ok
Starting DNS Services: ok
Starting Redundancy: ok
Start rmgrPingTaskok
Starting LWAPP: ok
Starting CAPWAP: ok
Starting LOCP: ok
Starting Security Services: ok
Starting Policy Manager: ok
Starting Authentication Engine: ok
Starting Mobility Management: ok
Starting Ethernet-over-IP: ok
Starting Capwap Ping Component: ok
Starting AVC Services: ok
Starting AVC Flex Services: ok
Starting Virtual AP Services: ok
Starting AireWave Director: ok
Starting Network Time Services: ok
Starting Cisco Discovery Protocol: ok
Starting Broadcast Services: ok
Starting Logging Services: ok
Starting DHCP Server: ok
Starting IDS Signature Manager: ok
Starting RFID Tag Tracking: ok
Starting RF Profiles: ok
Starting Power Supply and Fan Status Monitoring Service: ok
Starting Mesh Services: ok
Starting TSM: ok
Starting CIDS Services: ok
Starting Ethernet-over-IP: ok
Starting DTLS server: enabled in CAPWAP
Starting CleanAir: ok
Starting WIPS: ok
Starting SSHPM LSC PROV LIST: ok
Starting RRC Services: ok
Starting SXP Services: ok
Starting Alarm Services: ok
Starting FMC HS: ok
Starting IPv6 Services: ok
Starting Config Sync Manager : ok
Starting Hotspot Services: ok
Starting PMIP Services: ok
Starting Tunnel Services New: ok
Starting Portal Server Services: ok
Starting mDNS Services: ok
Starting Management Services:
Web Server: CLI: Secure Web: ok

(Cisco Controller)

Enter User Name (or 'Recover-Config' this one-time only to reset configuration to factory defaults)

User:
Configuration Saved!
System will now reboot!
Creating license client restartability thread

Updating license storage ... Done.

Exit Called
Switchdrvr exited!
Restarting system.


WLCNG Boot Loader Version 1.0.20 (Built on Jan 9 2014 at 19:02:44 by cisco)
Board Revision 1.3 (SN: FCW2114B5DU, Type: AIR-CT5508-K9) (G)

Verifying boot loader integrity... OK.

OCTEON CN5645-NSP pass 2.1, Core clock: 600 MHz, DDR clock: 330 MHz (660 Mhz data rate)
FPGA Revision 1.7
Env FW Revision 1.8
USB Console Revision 2.2
CPU Cores: 10
DRAM: 1024 MB
Flash: 32 MB
Clearing DRAM........ done
Network: octeth0', octeth1
' - Active interface
E - Environment MAC address override
CF Bus 0 (IDE): OK
IDE device 0:
- Model: VRFDFC41024ECEE-P1 Firm: 100511g Ser#: VIR20455730
- Type: Hard Disk
- Capacity: 1000.1 MB = 0.9 GB (2048256 x 512)


Press <ESC> now to access the Boot Menu...

Loading primary image (8.2.151.0)
100%

36899306 bytes read
Launching...
init started: BusyBox v1.6.0 (2010-05-13 17:50:10 EDT) multi-call binary
starting pid 847, tty '': '/etc/init.d/rcS'
Decompressing... OK
Validating...... OK
Set PLX switch MPS settings .............!!!!!!!
Detecting Hardware ...
set smp_affinity for irq 48
003f
DP from CGE5.0 ...
starting pid 1156, tty '/dev/ttyS0': '/usr/bin/gettyOrMwar'
Setting up ZVM
Exporting LD_LIBRARY_PATH

Cryptographic library self-test....
Testing SHA1 Short Message 1
Testing SHA256 Short Message 1
Testing SHA1 Short Message 1
SHA1 POST PASSED
passed!

XML config selected
XML config selected
Validating XML configuration
octeon_device_init: found 1 DPs
readCPUConfigData: cardid 0x6070001
Cisco is a trademark of Cisco Systems, Inc.
Software Copyright Cisco Systems, Inc. All rights reserved.

Cisco AireOS Version 8.2.151.0
Firmware Version FPGA 1.7, Env 1.8, USB console 2.2
Initializing OS Services: ok
Initializing Serial Services: ok
Initializing Network Services: ok
Initializing Licensing Services: ok

License daemon start initialization.....

License daemon running.....
Starting Statistics Service: ok
Starting Licensing Services: ok
Starting ARP Services: ok
Starting Trap Manager: ok
Starting Network Interface Management Services: ok
Starting System Services: ok
Starting FIPS Features: ok : Not enabled
Starting SNMP services: ok
Starting Fastpath Hardware Acceleration: ok
Starting Fastpath Console redirect : ok
Starting Fastpath DP Heartbeat : ok
Fastpath CPU0.00: Starting Fastpath Application. SDK-1.8.0, build 269. Flags-[DUTY CYCLE] : ok
Fastpath CPU0.00: Initializing last packet received queue. Num of cores(10)
Fastpath CPU0.00: Init MBUF size: 1856, Subsequent MBUF size: 2041
Fastpath CPU0.00: Core 0 Initialization and FIPS self-test: ok
Fastpath CPU0.00: 10 Cores are being initialized
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing Timer...done.
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing NBAR AGING Timer...done.
Fastpath CPU0.00: Received instruction to get link status
Fastpath CPU0.01: Core 1 Initialization and FIPS self-test: ok
Fastpath CPU0.02: Core 2 Initialization and FIPS self-test: ok
Fastpath CPU0.03: Core 3 Initialization and FIPS self-test: ok
Fastpath CPU0.04: Core 4 Initialization and FIPS self-test: ok
Fastpath CPU0.05: Core 5 Initialization and FIPS self-test: ok
Fastpath CPU0.06: Core 6 Initialization and FIPS self-test: ok
Fastpath CPU0.07: Core 7 Initialization and FIPS self-test: ok
Fastpath CPU0.08: Core 8 Initialization and FIPS self-test: ok
Fastpath CPU0.09: Core 9 Initialization and FIPS self-test: ok
Starting Switching Services: ok
Starting QoS Services: ok
Starting Policy Manager: ok
Starting Data Transport Link Layer: ok
Starting Access Control List Services: ok
Starting System Interfaces: ok
Starting Client Troubleshooting Service: ok
Starting Certificate Database: ok
Starting VPN Services: ok
Starting Management Frame Protection: ok
Starting DNS Services: ok
Starting Redundancy: Starting Peer Search Timer of 120 seconds
Initiate Role Negotiation Message to peer
Found the Peer. Starting Role Determination...ok

Start rmgrPingTaskok
Starting LWAPP: ok
Starting CAPWAP: ok
Starting LOCP: ok
Starting Security Services: ok
Starting Policy Manager: ok
Starting Authentication Engine: ok
Starting Mobility Management: ok
Starting Ethernet-over-IP: ok
Starting Capwap Ping Component: ok
Starting AVC Services: ok
Starting AVC Flex Services: ok
Starting Virtual AP Services: ok
Starting AireWave Director: ok
Starting Network Time Services: ok
Starting Cisco Discovery Protocol: ok
Starting Broadcast Services: ok
Starting Logging Services: ok
Starting DHCP Server: ok
Starting IDS Signature Manager: ok
Starting RFID Tag Tracking: ok
Starting RF Profiles: ok
Starting Power Supply and Fan Status Monitoring Service: ok
Starting Mesh Services: ok
Starting TSM: ok
Starting CIDS Services: ok
Starting Ethernet-over-IP: ok
Starting DTLS server: enabled in CAPWAP
Starting CleanAir: ok
Starting WIPS: ok
Starting SSHPM LSC PROV LIST: ok
Starting RRC Services: ok
Starting SXP Services: ok
Starting Alarm Services: ok
Starting FMC HS: ok
Starting IPv6 Services: ok
Starting Config Sync Manager : ok
Starting Hotspot Services: ok
Starting PMIP Services: ok
Starting Tunnel Services New: ok
Starting Portal Server Services: ok
Starting mDNS Services: ok
Starting Management Services:
Web Server: CLI: Secure Web: ok

(Cisco Controller-Standby)

Enter User Name (or 'Recover-Config' this one-time only to reset configuration to factory defaults)

User: admin
Password:**********
(Cisco Controller-Standby) >



SSO HA Status


5. Remove SSO HA Configuration:
5.1 Disabling SSO on HA Pair 1
On primary controller, disable SSO using the command:
config redundancy mode disable

The Active and Standby WLCs reboot once this command is executed. The standby controller, when it comes back after the reboot, has the same IP address on interfaces as the primary controller and all the ports disabled.

5.2. On the standby controller
re-enter the correct IP addresses corresponding to the management and dynamic interfaces and execute the following command:
config port adminmode all enable

5.3. Save the configuration on the controller.


6. Test Failover

To make the Secondary WLC2 become the primary WLC, we made a manual takeover. Connect via SSH to the management ip address (10.9.20.30 in our case) and type in the following command:

redundancy force-switchover

(Cisco Controller) >redundancy force-switchover 

Warning: Saving configuration change causes all the configurations to be saved on flash.
If this is not what you intend to do, do not type 'y' below.

The system has unsaved changes.
Would you like to save them now? (y/N) y


Configuration Saved!Creating license client restartability thread

Exit Called
Switchdrvr exited!
Restarting system.

....




(Cisco Controller-Standby) >
(Cisco Controller-Standby) >


HA completed successfully, WLC switch over detection time : ~2 msec and APs switch over time : ~2 msec



(Cisco Controller) >
(Cisco Controller) >
(Cisco Controller) >
Blocked: Configurations blocked as standby WLC is still booting up.
You will be notified once configurations are Unblocked

Unblocked: Configurations are allowed now...






7. Upgrade Firmware/Code on HA deployment

Upgrading firmware / code to new release is not service upgrading process. The communication will be down since both WLC will need to be rebooted almost at the same time. Please check my another post for detailed steps:
Cisco Wireless Controller 5508 Configuration - Tips and Tricks






Reference: