Advanced Checkpoint Gaia CLI Commands (Tips and Tricks)

With my most populous post "Basic Checkpoint Gaia CLI Commands (Tips and Tricks)", I would like to
collect some more advanced troubleshooting commands used in my daily work into this post. Actually, some of commands are not only for Checkpoint Gaia, it will be for SPLAT or IPSO platform as well. This post will keep updating as soon as I have something new.

1. fw ctl chain

Check Checkpoint Security Gateway packet inspection order/chain. For more details, check the post "How Firewalls (Security Gateways) Handle the Packets?"

in chain (18):
        0: -7f800000 (f28854f0) (ffffffff) IP Options Strip (in) (ipopt_strip)
        1: -7d000000 (f1796f10) (00000003) vpn multik forward in
        2: - 2000000 (f177cb70) (00000003) vpn decrypt (vpn)
        3: - 1fffff8 (f1787c00) (00000001) l2tp inbound (l2tp)
        4: - 1fffff6 (f2886ca0) (00000001) Stateless verifications (in) (asm)
        5: - 1fffff5 (f28bce30) (00000001) fw multik misc proto forwarding
        6: - 1fffff2 (f17a4df0) (00000003) vpn tagging inbound (tagging)
        7: - 1fffff0 (f177a150) (00000003) vpn decrypt verify (vpn_ver)
        8: - 1000000 (f29049c0) (00000003) SecureXL conn sync (secxl_sync)
        9:         0 (f282f810) (00000001) fw VM inbound  (fw)
        10:         1 (f28a6b30) (00000002) wire VM inbound  (wire_vm)
        11:   2000000 (f177b5e0) (00000003) vpn policy inbound (vpn_pol)
        12:  10000000 (f2902cb0) (00000003) SecureXL inbound (secxl)
        13:  7f600000 (f287ab70) (00000001) fw SCV inbound (scv)
        14:  7f730000 (f2a13500) (00000001) passive streaming (in) (pass_str)
        15:  7f750000 (f2c0bef0) (00000001) TCP streaming (in) (cpas)
        16:  7f800000 (f2885890) (ffffffff) IP Options Restore (in) (ipopt_res)
        17:  7fb00000 (f2fac050) (00000001) HA Forwarding (ha_for)
out chain (15):
        0: -7f800000 (f28854f0) (ffffffff) IP Options Strip (out) (ipopt_strip)
        1: -78000000 (f1796ef0) (00000003) vpn multik forward out
        2: - 1ffffff (f1779a10) (00000003) vpn nat outbound (vpn_nat)
        3: - 1fffff0 (f2c0bd70) (00000001) TCP streaming (out) (cpas)
        4: - 1ffff50 (f2a13500) (00000001) passive streaming (out) (pass_str)
        5: - 1ff0000 (f17a4df0) (00000003) vpn tagging outbound (tagging)
        6: - 1f00000 (f2886ca0) (00000001) Stateless verifications (out) (asm)
        7:         0 (f282f810) (00000001) fw VM outbound (fw)
        8:         1 (f28a6b30) (00000002) wire VM outbound  (wire_vm)
        9:   2000000 (f1779c30) (00000003) vpn policy outbound (vpn_pol)
        10:  10000000 (f2902cb0) (00000003) SecureXL outbound (secxl)
        11:  1ffffff0 (f17887b0) (00000001) l2tp outbound (l2tp)
        12:  20000000 (f177d5b0) (00000003) vpn encrypt (vpn)
        13:  7f700000 (f2c0e340) (00000001) TCP streaming post VM (cpas)
        14:  7f800000 (f2885890) (ffffffff) IP Options Restore (out) (ipopt_res)

2. Proxy Arp

a. Use the Gaia portal.
Network Management -> Arp -> Proxy ARP

b. Use the command line (in Gaia):
add arp proxy ipv4-address 172.16.0.8 interface eth0 real-ipv4-address 172.16.0.22

Actually the GAIA command above convert it automatically to a file called local.arp

c. Use the command line (in expert mode):
Then insert the information directly to /opt/CPsuite-R76/fw1/conf/local.arp
echo "172.16.0.8 00:0c:29:f1:b7:74 172.16.0.22" >> $FWDIR/conf/local.arp


Verify the changes after a policy push with command "fw ctl arp":

[Expert@CP1:0]# fw ctl arp  (10.9.3.21) at 00-1c-7f-32-cc-15  (10.9.3.53) at 00-1c-7f-32-cc-15  (10.9.3.35) at 00-1c-7f-32-cc-15  (10.9.3.26) at 00-1c-7f-32-cc-15  (10.9.3.29) at 00-1c-7f-32-cc-15  (10.9.3.80) at 00-1c-7f-32-cc-15  (191.24.11.13) at 00-1c-7f-33-07-ae interface 191.24.11.116  (10.9.3.25) at 00-1c-7f-32-cc-15  (10.9.3.61) at 00-1c-7f-32-cc-15  (10.9.3.28) at 00-1c-7f-32-cc-15  (10.9.3.24) at 00-1c-7f-32-cc-15  (10.9.3.27) at 00-1c-7f-32-cc-15 FW-GAIA> show arp proxy all IP Address              MAC Address / Interface         Real IP Address 200.0.0.102           eth0

Reference: Checkpoint SPLAT Manual Proxy ARP Configuration Example

3. fw ctl zdebug drop

lists all dropped packets in real time gives an explanation why the packet is dropped

4. TCPDUMP

• tcpdump port 257   , <– on the firewall, this will allow you to see if the logs are passing from the firewall to the manager, and what address they are heading to.
• tcpdump -i WAN.15  <- to capture everything on this interface
• tcpdump -i eth1.16 icmp  <– to capture just PINGs on this interface
• tcpdump -i  Mgmt -vvv -s0 -w tcpdumpfile.log   <– this captures the FULL packets to a file usefull for wireshark the -s0 stops the files being shortened
• tcpdump -i INT port 67   <– view dhcp requests
• tcpdump -eP -nni any host 10.9.4.30 <-disable both name and service port resolution while performing a capture, by using the -nn option; -e Print the link-level header on each dump line. This can be used, for example, to print MAC layer addresses for protocols such as Ethernet and IEEE 802.11. -p--no-promiscuous-mode.
• tcpdump -i any  <- any can be used to tell tcpdump to listen on all interfaces
• tcpdump -n  <- disable to lookup and translate hostnames and ports.

Reference: Understanding TCPDUMP Output

5. FW Monitor

• fw monitor -e 'accept host(192.168.1.12);'   <-- Show packets with IP 192.168.1.12 as SRC or DST
• fw monitor -e 'accept src=192.168.1.12 and dst=192.168.3.3;' <--Show all packets from 192.168.1.12 to 192.168.3.3
• fw monitor -pi ipopt_strip -e 'accept udpport(53);' <--Show UDP port 53 (DNS) packets, pre-in position is before 'ippot_strip'
• fw monitor -m O -e 'accept udp and (sport>1023 or dport>1023);' <-- Show UPD traffic from or to unprivileged ports, only show post-out
• fw monitor -e 'accept net(192.168.1.0,24) and tracert;' <--Show Windows traceroute (ICMP, TTL<30) from and to network 192.168.1.0/24
• fw monitor -v 23 -e 'accept tcpport(80);' <--Show Capture web traffic for VSX virtual system ID 23
• fw monitor -e 'accept ip_p=50 and ifid=0;' <--Show all ESP (IP protocol 50) packets on the interface with the ID 0. (List interfaces and corresponding IDs with fw ctl iflist)
• srfw monitor -o output_file.cap <--Show traffic on a SecuRemote/SecureClient client into a file. srfw.exe is in $SRDIR/bin (C:\Program Files\CheckPoint\SecuRemote\bin)

6. VPN tu

vpn tu  or  vpn tunnelutil

********** Select Option **********

(1)  List all IKE SAs
(2)  List all IPsec SAs
(3)  List all IKE SAs for a given peer (GW) or user (Client)
(4)  List all IPsec SAs for a given peer (GW) or user (Client)
(5)  Delete all IPsec SAs for a given peer (GW)
(6)  Delete all IPsec SAs for a given User (Client)
(7)  Delete all IPsec+IKE SAs for a given peer (GW)
(8)  Delete all IPsec+IKE SAs for a given User (Client)
(9)  Delete all IPsec SAs for ALL peers and users
(0)  Delete all IPsec+IKE SAs for ALL peers and users

(Q)  Quit 



7. Disk/File/Folder Commands

Checkpoint SK60080 displays some solutions to resolve excessive disk consumption on SPLAT/Gaia/IPSO/Lunix OS system. Here are some helpful commands:
a. df -h  (view the partition table and its associated utilization)
b. du -h --max-depth=1 /opt | sort -n -r   (examine disk space utilization at directory-level)
c. ls -1 $FWDIR/conf/db_versions/repository/ | wc -l   (check the number of database revisions on a Security Management server)
d. ls -l $RTDIR/distrib/* | wc -l  (counts the number of records)
e. evstop & evstart (Stop / start the Eventia / SmartEvent)
f. rm -r $RTDIR/distrib/* (Purge this directory of stale records)

g. ls -lR /var/log/dump/usermode/   (Find and delete old core dump files)

h. ls -lR /var/crash/  (Find and delete old core dump files)

i. rm $FWDIR/log/2009*.log*  (removes all old log files for year 2009)

8. Connections

CP-1> fw tab -t connections -s
HOST                  NAME                               ID #VALS #PEAK #SLINKS
localhost             connections                      8158    77   948     179

Note: 
The NAME Id is the actual table number. 
The VALS colum is the current number of connections that are in the connections table at the time the command was run. 
The PEAK number is the max number of connections that have been recorded since the last reboot. 
The SLINKS table is a table of symbolic link that point to the real connection entry. There are usually 4 symbolic links per connection. This way no matter which direction the packet comes, there will be an entry for it. There is more to it than that, but that is the general idea.

CP-1> fw ctl pstat

System Capacity Summary:
  Memory used: 8% (62 MB out of 696 MB) - below watermark
  Concurrent Connections: 0% (79 out of 24900) - below watermark
  Aggressive Aging is in detect mode

Hash kernel memory (hmem) statistics:
  Total memory allocated: 71303168 bytes in 17408 (4096 bytes) blocks using 1 pool
  Total memory bytes  used:  9703728   unused: 61599440 (86.39%)   peak: 18891512
  Total memory blocks used:     2665   unused:    14743 (84%)   peak:     4705
  Allocations: 198489371 alloc, 0 failed alloc, 198382561 free

System kernel memory (smem) statistics:
  Total memory  bytes  used: 117769900   peak: 120093268
  Total memory bytes wasted:   996590
    Blocking  memory  bytes   used:  2530356   peak:  2557584
    Non-Blocking memory bytes used: 115239544   peak: 117535684
  Allocations: 433810 alloc, 28 failed alloc, 432937 free, 0 failed free
  vmalloc bytes  used: 114086588 expensive: no

Kernel memory (kmem) statistics:
  Total memory  bytes  used: 56103032   peak: 66020104
  Allocations: 198922588 alloc, 28 failed alloc
               198815489 free, 0 failed free
  External Allocations: 0 for packets, 0 for SXL

Cookies:
        90753187 total, 0 alloc, 0 free,
        7839 dup, 2107678 get, 160176 put,
        91154457 len, 0 cached len, 0 chain alloc,
        0 chain free

Connections:
        231169 total, 7807 TCP, 4665 UDP, 182351 ICMP,
        36346 other, 0 anticipated, 3 recovered, 79 concurrent,
        948 peak concurrent

Fragments:
        0 fragments, 0 packets, 0 expired, 0 short,
        0 large, 0 duplicates, 0 failures

NAT:
        80509/0 forw, 5266/0 bckw, 85750 tcpudp,
        16 icmp, 10440-949656 alloc

Sync:
        Version: new
        Status: Able to Send/Receive sync packets
        Sync packets sent:
         total : 864451,  retransmitted : 0, retrans reqs : 15,  acks : 1826
        Sync packets received:
         total : 3614413,  were queued : 30, dropped by net : 15
         retrans reqs : 0, received 11745 acks
         retrans reqs for illegal seq : 0
         dropped updates as a result of sync overload: 0
        Callback statistics: handled 11588 cb, average delay : 1,  max delay : 5

9. Check Point SecureXL

To enable SecureXL, run the command:
CP[admin]# fwaccel on

To disable SecureXL, run the command:
CP[admin]# fwaccel off

Note: The fwaccel off command is not persistent and SecureXL will be enabled again after a reboot of the system. SecureXL can be permanently disabled through the CPconfig utility.

To check the number of accelerated connection and other SecureXL statistics: CP[admin]# netstat -f
To check the number of accelerated SA (VPN traffic): CP[admin]# netstat -s
To check overall SecureXL statistics:  CP[admin]# fwaccel stat

10.View Checkpoint Log from CLI

expert mode 

fw log -n | morefw log -n -f | https

or 

normal mode without pipe

11. Revision Control Versions Location on Management Server

[Expert@CP-Management]# cd /opt/CPsuite-R75.20/fw1/conf/db_versions/repository/

[Expert@CP-Management]# ls1  11  12  13  14  15  16  17  18  2  3  4  5  6  7  8  9

All version are in those number directories. Actual version info is in versioning_db.fws

[Expert@CP-Management]# cd database/ 

[Expert@CP-Management]# lsversioning_db.fws

12. Change user cli between BASH and CLISH

HostName> set user admin shell /bin/bash
HostName> save config

[Expert@HostName]# dbset passwd:admin:shell /etc/cli.sh
[Expert@HostName]# dbset :save

or

chsh -s /bin/bash admin

chsh -s /etc/cli.sh admin

for SPLAT the default shell is /bin/cpshell which is not showing from /etc/shells file.

chsh -s /bin/cpshell admin

[admin@CP-Management ~]$ cat /etc/shells

/bin/sh

/bin/bash

/bin/bash2

/bin/tcsh

/bin/csh

[admin@CP-Management ~]$ chsh

Changing shell for root.

New shell [/bin/cpshell]: /bin/bash

Shell changed.

13. Enable SFTP in Gaia

[Expert@HostName]# vi  /etc/ssh/sshd_config 

Uncomment the 'sftp-server' line by deleting the pound '#' character: 

from

#Subsystem sftp /usr/libexec/openssh/sftp-server
to
Subsystem sftp /usr/libexec/openssh/sftp-server

[Expert@HostName]# /etc/init.d/sshd restart

Note: Please check my previous post: Enable SFTP to Checkpoint Gaia OS System for more details.

14. Installation of Hotfixes on Gaia or SPLAT

[Expert@HostName]# tar -zxvf Check_Point_Hotfix_VERSION_OS_sk104443.tgz
[Expert@HostName]# ./SecurePlatform_HOTFIX_NAME
[Expert@HostName]# reboot

Steps to Installation a Jumbo Hotfix for R77.20 on Cluster Environment:
a. install a hotfix on standby cluster member (CP2) then reboot it
b. failover from active cluster member (CP1) to standby cluster (CP2) after standby cluster finished rebooting
c. install hotfix on CP1 and reboot it.

[Expert@FW-CP2:0]# md5sum Check_Point_R77.20.linux.tgz  d788583cf44389b83b0dd6990cb53f63  Check_Point_R77.20.linux.tgz [Expert@FW-CP2:0]# tar -zxvf Check_Point_R77.20.linux.tgz  Actions/ Actions/cpconfig Actions/CheckPackage Actions/CRSValidator Actions/GetPa ...... [Expert@FW-CP2:0]# ./UnixInstallScript  *********************************************************** Welcome to Check Point R77_20_JUMBO_HF installation  *********************************************************** Verifying installation environment for R77_20_JUMBO_HF...Done! The following components will be installed: * R77_20_JUMBO_HF Installation program is about to stop all Check Point Processes. Do you want to continue (y/n) ? y Stopping Check Point Processes...Done! Installing Security Gateway / Security Management R77_20_JUMBO_HF...Done! Installing GAIA R77_20_JUMBO_HF...Done! Installing Performance Pack R77_20_JUMBO_HF...Done! Installing Mobile Access R77_20_JUMBO_HF...Done! ************************************************************************ Package Name                                                    Status ------------                                                    ------ Security Gateway / Security Management R77_20_JUMBO_HF          Succeeded GAIA R77_20_JUMBO_HF                                            Succeeded Performance Pack R77_20_JUMBO_HF                                Succeeded Mobile Access R77_20_JUMBO_HF                                   Succeeded ************************************************************************ Installation program completed successfully. Do you wish to reboot your machine (y/n) ? y Broadcast message from admin (pts/2) (Mon Oct 26 16:37:44 2015): The system is going down for reboot NOW! Broadcast message from admin (pts/2) (Mon Oct 26 16:37:44 2015): The system is going down for reboot NOW! [Expert@FW-CP2:0]#

15. SSH Timeout Solutions

a. Increasing the timeout
set inactivity-timeout 720

b. Ignore Hangup
[Expert@R76GaiaGate1:0]# fw monitor -e "accept host(172.16.0.1);" -o test.cap & [1] 27524
[Expert@R76GaiaGate1:0]# ps -aux | grep "fw monitor"
admin 27524 0.7 2.1 88268 21256 pts/2 S 14:09 0:00 fw monitor -e accept host(172.16.0.1); -o test.cap

Disown the process with this command, specifying the PID:

disown 27524

Closing, or having the SSH session end due to timeout will no longer send a hangup to this process, since it is no longer a child process of the SSH session.

A new SSH session or console session can be started later, and the process can be killed manually to stop it.

kill 27524

[Expert@R76GaiaGate1:0]# nohup fw ctl kdebug -T -f -o debug.txt -m 10 -s 50000 & [1] 30209
nohup: appending output to 'nohup.out'

This example is creating cyclic debug files, 10 files, with a maximum of 50000Kb. Again, the PID is displayed, and the output of the command is piped to the text file 'nohup.out'.

The SSH session can be ended with "exit" or timed out, and the hangup sent to this child process will be ignored, the debug will continue running until we log in again and manually kill the PID.

16. ClusterXL Troubleshooting

16.1 Force a failover

This creates a pnote (problem notification) that is in problem state at current cluster member and force a failover to another member:

cphaprob -d fail -s problem -t 0 register

Verify it's in problem state with

cphaprob stat

and

cphaprob -i list

(you should see 'fail' in problem state)

Once you've finished your testing, run these two to reset it:

cphaprob -d fail -s ok reportcphaprob -d fail unregister

Reference: CheckPoint HA: How to force a failover (ClusterXL/VRRP)

16.2 cphaprob commands and troubleshooting ClustXL Problem

FW-CP2 is fine. But FW-CP1 shows problem on the clustxl status.

[Expert@FW-CP2:0]# cphaprob -a if  Required interfaces: 5 Required secured interfaces: 1 eth1       UP                    non sync(non secured), multicast eth2       UP                    sync(secured), multicast Mgmt       UP                    non sync(non secured), multicast eth3       UP                    non sync(non secured), multicast  (eth3.106  ) eth3       UP                    non sync(non secured), multicast  (eth3.102  ) Virtual cluster interfaces: 6 eth1            2.13.11.60       eth2            10.1.90.14       Mgmt            10.1.72.14       eth3.104        10.1.104.14       eth3.106        10.1.106.14       eth3.102        10.1.102.14

FW-CP1> cphaprob -i list Built-in Devices: Device Name: Interface Active Check Current state: problem Device Name: HA Initialization Current state: OK Device Name: Recovery Delay Current state: OK Registered Devices: Device Name: Synchronization Registration number: 0 Timeout: none Current state: OK Time since last report: 64196.3 sec Device Name: Filter Registration number: 1 Timeout: none Current state: OK Time since last report: 63492.1 sec Device Name: cphad Registration number: 2 Timeout: none Current state: OK Time since last report: 2.68138e+06 sec Device Name: fwd Registration number: 3 Timeout: none Current state: OK Time since last report: 2.68137e+06 sec Device Name: routed Registration number: 4 Timeout: none Current state: OK Time since last report: 62898.8 sec

Usually it was caused by the connection between firewall interface port and  switch port. UDP port 8116 will help us to find out which one is not sending the keep-alive packets:

Cluster Control Protocol (CCP) runs on UDP port 8116, and allows cluster members to report their own states and learn about the states of other members, by sending keep-alive packets (applies only to ClusterXL clusters). Also CCP keeps cluster member sync state.

Following tcpdump shows cluster member 1 (00:00:00:00:fe:00) and cluster member 2 (00:00:00:00:fe:01) both are sending 8116 CCP packets. That is normal. If you only see one sending, you will have to check another one's switch port vlan configuration. You may miss one vlan on switch trunk port, which has happened to me.

[Expert@FW-CP2:0]# tcpdump -enni eth3.102 port 8116 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth3.102, link-type EN10MB (Ethernet), capture size 96 bytes 11:13:17.497801 00:00:00:00:fe:01 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 92: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 50 11:13:17.597743 00:00:00:00:fe:01 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 76: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 34 11:13:17.676067 00:00:00:00:fe:00 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 76: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 34 11:13:17.676182 00:00:00:00:fe:00 > 01:00:5e:5b:66:0e, ethertype IPv4 (0x0800), length 92: 0.0.0.0.8116 > 10.91.102.0.8116: UDP, length 50

Note: Previous Troubleshooting Post - Checkpoint Cluster Member Down because interfaces show partially up

17. Permanent Change Global Kernel Parameters Value
Global kernel parameters exist to control (customize) the behavior of Security Gateway (kernel parameters are located in $FWDIR/boot/modules/fw*mod* kernel modules).

This control (customization) can be done on-the-fly using the fw ctl set int command (change takes effect immediately). However, the value of the kernel parameter returns to its default value after a reboot. At times, it may be required to control (customize) the behavior of Security Gateway permanently. In addition, it is necessary for some kernel parameters to be changed upon boot. fwkern.conf file is the one which holds all those kernel parameters value. If it is not existing in your system, you will need to create it manually.

The Security Gateway must be rebooted after any change in the $FWDIR/boot/modules/fwkern.conf file.

[Expert@CP1:0]# cat /opt/CPsuite-R77/fw1/boot/modules/fwkern.conf  fwha_mac_magic=40 fwha_mac_forward_magic=41 fw_allow_simultaneous_ping=1 fwha_forw_packet_to_not_active=1

Useful Checkpoint KBs:

1. sk97638 - Check Point Processes and Daemons
2. sk98348 - Best Practices - Security Gateway Performance

Reference:

1. Check Point/SPLAT/Network Debug Cheat Sheet
2. A tcpdump Primer with Examples
3. Check Point fw monitor cheat sheet – 20141028
4. Check Point CLI Reference Card – 20150617 by Jens Roesen
5. Upgrading ClusterXL Deployments(R77)

Checkpoint Gateway SSH Connection Intermittenly Slow Issue - CONFD CPU High

When Gaia released at R75.40 on 2012, our Checkpoint firewalls have been adopted it right away with an upgrade. Since then we have upgraded to R77.10, R77.20 and recently planing to R77.30. The new version's experience was quite good, but just recently we are starting to feel the Gaia CLI and Porttal is getting slower and slower. 

Symptoms:
For example, the ssh login process is taking a couple of minutes to show the prompt. WebUi is consistently showing lost database connection when saving any changes. You will have to re-login again to WebUI. SNMP Monitoring shows your device is up and reachable by ping but could not poll any SNMP information. After a couple of minutes, sometimes, it may take more than 10 minutes or longer, everything goes back normal. It did not happen all the time, just a couple of times per day. Most of times, log in, snmp access are fine.

Also some times, you will find out save config command will cause database timeout issue too.

FW-CP2> save config NMSCFD0026  Timeout waiting for response from database server.

Solutions:
Actually Checkpoint has a couple of sk relating to this issue:  such as sk104761 for this. Based on sk104761 : Each change made in Gaia Clish or in Gaia Portal is saved under a revision in the Gaia Database - /config/db/initial_db file. Once this file becomes large, confd process consumes more CPU to read from this file, or to save new data to this file.

[Expert@CP-DMZ-1:0]# cd /config/db [Expert@CP-DMZ-1:0]# ls -l total 218836 -rw-r--r-- 1 admin root    133250 Sep 20 13:28 initial -rw-r--r-- 1 admin root 223720448 Sep 20 13:28 initial_db

The Initial_db file has been increased to size 220M. So what is Initial_db, and can we delete it? Answer of course is no.

From sk101273, "The /config/db/initial file must be present and valid (in other words, not corrupted) at boot time for IP Series Appliance to get configured. Otherwise, the IP Series Appliance will go into first-time boot mode and attempt to configure itself using DHCP, or wait for the user to configure it through the serial console port."

Lets take a look what is inside:

[Expert@CP-1:0]# cat initial # This file was AUTOMATICALLY GENERATED # Generated by /bin/confd on Sun Sep 20 13:28:32 2015 # # DO NOT EDIT # configurationChange t centrallyManaged t inactto:default 720 # DO\ NOT\ EDIT file was AUTOMATICALLY GENERATED by /bin/confd on Tue 6 16\:16\:12 NOT EDIT resolv:resolver:1 8.8.8.8 ntp:server:10.94.16.5 t ntp:server:10.94.16.5:version 1 ntp:server:10.94.16.5:iburst t ntp:server:10.94.16.5:prefer t ntp:server:10.4.4.27 t ntp:server:10.4.4.27:version 1 ntp:server:10.4.4.27:iburst t ntp:servers:primary 10.94.16.5 ntp:servers:secondary 10.4.4.27 dhcp:dhcpc:interface:eth3 t dhcp:dhcpc:interface:eth3:timeout 60 dhcp:dhcpc:interface:eth3:retry 300 dhcp:dhcpc:interface:eth3:reboot 10 machine:hostname FW-GRU1-CP1 update_upgrade_info:set_counter f 5 17\:23\:44 installer:available_install_packages_number 4 installer:available_download_packages_number 7 installer:category_is_aligned:3 1 installer:category_is_aligned:5 1 installer:category_is_aligned:1 1 installer:category_is_aligned:4 0 installer:ftw_random_res 1 installer:d_weekday Saturday installer:d_hours 17 installer:d_minutes 30 ...

To verify that this is indeed the issue:

1. Log in to Expert mode.
2. Backup the current Gaia configuarion database:
   [Expert@HostName]# cp  /config/db/initial_db  /config/db/initial_db_backup
3. Connect to the Gaia configuration database:
   [Expert@HostName]# sqlite3 /config/db/initial_db
4. Query the database using the SQLite to identify the issue:
   sqlite> select * from revisions where time like "%1969%";
   If any entries are returned, the system is likely experiencing this issue.
5. Exit from SQLite:
   sqlite> .exit

[Expert@FW-GRU1-CP1:0]# sqlite3 /config/db/initial_db  SQLite version 3.6.20 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> sqlite> select * from revisions where time like "%1969%"; Error: near "sqlite": syntax error sqlite> select * from revisions where time like "%1969%"; cluster:shared_feature_lock:admin|0|||||1969-12-31 19:00:00|1 cluster:shared_feature_lock:cadmin|0|||||1969-12-31 19:00:00|1 cdm:per_exec|0|||||1969-12-31 19:00:00|1 cdm:total|0|||||1969-12-31 19:00:00|1 cdm:enable|0|||||1969-12-31 19:00:00|1 lcd:screensaver:mode|0|||||1969-12-31 19:00:00|1 lcd:screensaver:timeout|0|||||1969-12-31 19:00:00|1 lcd:backlight:support|0|||||1969-12-31 19:00:00|1 zoneinfo:Atlantic:Faroe|0|||||1969-12-31 19:00:00|1 zoneinfo:Atlantic:Stanley|0|||||1969-12-31 19:00:00|1 zoneinfo:Atlantic:Canary|0|||||1969-12-31 19:00:00|1 zoneinfo:Atlantic:St_Helena|0|||||1969-12-31 19:00:00|1 zoneinfo:Atlantic:South_Georgia|0|||||1969-12-31 19:00:00|1 ......

Once cause confirmed, contact Checkpoint Support to get a fix patch and apply it.

Some other SKs, sk95238, sk102988  are having similar solution on this issue. Basically, a Jumbo Hotfix will have this to be fixed.

Reference:

sk104761:'confd' process consumes CPU at high level on Gaia OS due to large size of Gaia Database (/config/db/initial_db)

sk98285:Jumbo Hotfix Accumulator for R77.10 (gypsy_hf_base_021)
sk95238 ('confd' daemon consumes the CPU up to 100% when using Gaia Portal)
sk102988 ('monitord' and 'confd' processes consume 100% CPU)
sk102994 (Clish in Gaia OS is very slow when making any changes in Gaia OS configuration)

Use Network Automation Tool Infoblox NetMRI Push Configuration to Multiple Network Devices

Infoblox NetMRI Appliance

For those still do not know what is Infoblox NetMRI product, here is some simple introduction. Actually it can do more than what normal network administrators think.

NetMRI is one of the most important products owned by Infoblox. This product came with the acquisition of Netcordia in 2010. NetMRI provides automatic network discovery, switch port management, network change automation and continuous configuration compliance management for multi-vendor routers, switches and other layer 2 and 3 network devices. NetMRI helps customers move from out-of-date spreadsheets, error-prone manual processes like scripts and CLI access and ad hoc audit teams.

NetMRI Dashboard

The NetMRI also has virtual appliance version which runs on VMware ESXi 5.x/6.0 host(s) helping organizations automate, discover and control network devices.

NetMRI's central console, Operations Center, greatly enhances NetMRI's scalability by coordinating, controlling and collecting data from multiple NetMRI devices. The vendor says one NetMRI Operations Center can monitor a network of 20,000 or more routers, switches and firewalls -- roughly 10 NetMRI appliances worth.

This post is the guide to use NetMRI to push configuration to multiple network devices automatically, which is really helpful to lock down and take control of your environment.

1. Go to Config Management - Job Managment - Scripts Page

You will find out lots of prepared scripts with many examples. In this guide, we will use 'AD Hoc Command Batch' scripts to push configuration to multiple Cisco switches and routers.

2. Move your mouse to the gear icon in front of  'AD Hoc Command Batch'

A dark menu with 'Run now' option will show up. You will just click 'Run now' option.

3. Put your configuration into text box. Then click Next.

For example:

conf t line con 0 login authentication CONAUTH end wr mem

You can ignore Custom Fields error and directly click Next again.

4. Choose proper device group(s) you want to change. Then click Next.

In this example, Routers and Switches have been chosen.

5. Review your configuration and select Run Now.

6. You will be brought to Job History page:

7. Click Job name 'Ad Hoc Job 10/15 21:58' to see the Job Details.

There are some errors because of commands incompatible issue. You may need to change your commands to run it again. Click Error will give you which command caused this error issue.

To sum up, NetMRI is a good product. Infoblox NetMRI identifies, tracks and shows the impact of changes to multi-vendor networks, and automatically compares configurations to gold-standard settings. NetMRI provides critical network visibility even in the most complex, virtualized environments. But it is really expensive to purchase and maintain. It is licensed by network devices numbers. When you purchased the appliance, by default, you will only have 50 network devices licensed. If you want to add another 50 license, it will cost you more than $15k .

Reference:

Infoblox Data Center Automation with NetMRI
Infoblox / Forums / Network Change & Configuration Management Network Change & Configuration Management

Cisco IOS Router Hardening Template

Here is the my router configuration template modified from Team Cymru's. 

!Without service nagle on a Cisco router, each character in a Telnet session is a separate CPU interrupt. Hence, a command such as show tech will force a large number of CPU interrupts, impacting the performance of the router.
service nagle
service tcp-keepalives-in
service tcp-keepalives-out
!
! Show copious timestamps in our logs
service timestamps debug datetime msec show-timezone localtime
service timestamps log datetime msec show-timezone localtime
! Ensures all passwords and secrets are obfuscated when showing configuration
service password-encryption
no service dhcp
no ip bootp server
!
hostname xxxxx
!
boot system flash:c2900-universalk9-mz.SPA.151-4.M.bin
logging buffered 16384 debugging
no logging console
!'secret' ensures MD5 is used when 'service password encryption' is used.
enable secret xxxxx
no enable password
!
! Use TACACS+ for AAA. 
aaa new-model
aaa authentication login default group tacacs+ local-case
aaa authentication enable default group tacacs+ enable
aaa authorization commands 15 default group tacacs+ local
aaa accounting exec default stop-only group tacacs+
aaa accounting commands 15 default stop-only group tacacs+
aaa accounting network default stop-only group tacacs+
tacacs-server host 10.2.2.2
tacacs-server key xxxxx
!
ip dhcp bootp ignore
! In the event that TACACS+ fails, use case-sensitve local authentication.
username radmin1 secret 0 1q2w3e4r!
! localit user only can show running configuration
username localit secret Cisco1234
username localit privilege 15 autocommand show running
! localadmin user can do more troubleshooting and run 'show config'
username localadmin privilege 7 secret Cisco1234
!
privilege exec level 7 show config
! Logging and archive the commands and changes
archive
 log config
  logging enable
  logging size 500
  notify syslog
  hidekeys
  path flash:backup-
  write-memory
  maximum 8
!
! Ensure TCL doesn't use an initilizaion file where available.
no scripting tcl init
no scripting tcl encdir
!
! Enable the netflow top talkers feature.You can see the top N talkers (50 in this example) with the show ip flow top-talkers command. This is a handy ! utility to use during DDoS attacks and traffic issues. You can sort-by either packets or bytes, as you prefer.
ip flow-top-talkers
 top 50
 sort-by packets
!
! Don't run the HTTP server.
no ip http server
no ip http secure-server
!
! Allow us to use the low subnet and go classless
ip subnet-zero
ip classless
!
! Disable noxious services
no service pad
no ip source-route
no ip finger
no ip bootp server
no ip domain-lookup
!
! Block brute force login attempts while maintaining access for legitimate source addresses.
login block-for 100 attempts 15 within 100
login quiet-mode access-class 100
login on-failure log
login on-success log
!
! Catch crash dumps; very important with a "security router."
ip ftp username rooter
ip ftp password <PASSWORD>
! Give our core dump files a unique name.
exception core-file secure-router01-core
exception protocol ftp
exception dump 10.2.2.3
!
! Fire up CEF for both performance and security.
ip cef
!

clock timezone EST -5 0

clock summer-time EDT recurring

ntp authentication-key 6767 md5 <SECRETKEY>
ntp authenticate
ntp update-calendar
ntp server 10.3.3.3
!
! Configure the loopback0 interface as the source of our log  messages. 
int loopback0
 ip address 10.10.10.10 255.255.255.255
 no ip redirects
 no ip unreachables
 no ip proxy-arp
!
! Configure null0 as a place to send naughty packets.
interface null0
 no ip unreachables
!
interface Ethernet2/0
 description Unprotected interface, facing towards Internet
 ip address 199.116.2.14 255.255.255.240
 ! Do we run CEF verify? Yes if the data path is symmetric. No if the data path is asymmetric.
 ip verify unicast reverse-path
 ! Apply our template ACL
 ip access-group 2010 in
 ! Allow UDP to occupy no more than 2 Mb/s of the pipe.
 rate-limit input access-group 150 2010000 250000 250000 conform-action transmit exceed-action drop
 ! Allow ICMP to occupy no more than 500 Kb/s of the pipe.
 rate-limit input access-group 160 500000 62500 62500 conform-action transmit exceed-action drop
 ! Allow multicast to occupy no more than 5 Mb/s of the pipe.
 rate-limit input access-group 170 5000000 375000 375000 conform-action transmit exceed-action drop
 ! Don't send redirects.
 no ip redirects
 ! Don't send unreachable. NOTE WELL that this may break PMTU discovery. For example, if this router is edge for a VPN of any sort, you might need to enable ip unreachable. A typical symptom is ping working but a larger transmission doesn't.
 no ip unreachables
 ! Don't propogate smurf attacks.
 no ip directed-broadcast
 ! Don't pretend to be something you're not.
 no ip proxy-arp
 ! Do not reveal our netmask
 no ip mask-reply
 ! Log all naughty business.
 ip accounting access-violations
 ! If you allow multicast in your network or participate in the
 ! MBONE, the following multicast filtering steps will help to
 ! ensure a secure multicast environment. These must be applied
 ! per interface.
 ip multicast boundary 30
 !
 ! Keep flow data for analysis. If possible, export it to a cflowd server.
 ip route-cache flow
 ! When you configure anything to do with ntp on an IOS box, it will start listening on all interfaces. It is therefore a good idea that interfaces with public addresses have ntp disabled and therefore don't show a socket, unless that is what the interface is intended to do.
 ntp disable
 ! Disable Maintenance Operations Protocol on all interfaces
 no mop enable
!
interface Ethernet2/1
 description Protected interface, facing towards DMZ
 ip address 172.16.2.10 255.255.255.240
 ip verify unicast reverse-path
 ! If we are using RPF, comment out the ACL below.
 ip access-group 115 in
 no ip redirects
 no ip unreachables
 no ip directed-broadcast
 no ip proxy-arp
 ip accounting access-violations
 ip multicast boundary 30
 no ip mask-reply
 ip route-cache flow
 no mop enable
!
! Default route to the Internet (could be a routing protocol instead)
ip route 0.0.0.0 0.0.0.0 199.116.2.1
! Route to network on the other side of the firewall
ip route 10.0.0.0 255.0.0.0 172.16.2.1
! Black hole routes. Do not combine this with TCP Intercept; in fact, don't use TCP Intercept at all.
!
! Bogons
! Team Cymru has removed all static bogon references from this template
! due to the high probability that the application of these bogon filters
! will be a one-time event. Unfortunately many of these templates are applied and never re-visited, despite our dire warnings that bogons do change.
!
! This doesn't mean bogon filtering can't be accomplished in an automated
! manner. Why not consider peering with our globally distributed bogon
! route-server project? Alternately you can obtain a current and well
! maintained bogon feed from our DNS and RADb services. Read more at the
! link below to learn how!
!
! https://www.team-cymru.org/Services/Bogons/
!
! Export our NetFlow data to our NetFlow server, 192.0.2.34. NetFlow
! provides some statistics that can be of use when tracing the true
! source of a spoofed attack.
ip flow-export source loopback0
ip flow-export destination 192.0.2.34 2055
ip flow-export version 5 origin-as
!
! Log anything interesting to the loghost. Capture all of
! the logging output with FACILITY LOCAL5.
logging trap debugging
logging facility local5
logging source-interface loopback0
logging 10.2.2.3
!
! With the ACLs, it is important to log the naughty folks.
! Thus, the implicit drop all ACL is replaced (augmented,
! actually) with an explicit drop all that logs the attempt.
! You may wish to keep a second list (e.g. 2011) that does not
! log. During an attack, the additional logging can impact the
! performance of the router. Simply copy and paste access-list 2010,
! remove the log-input keyword, and name it access-list 2011. Then
! when an attack rages, you can replace access-list 2010 on the
! Internet-facing interface with access-list 2011.
!
! Block SNMP access to all but the loghost
access-list 20 remark SNMP ACL
access-list 20 permit 192.0.2.34
access-list 20 deny any log
!
! Multicast - filter out obviously naughty or needless traffic
access-list 30 remark Multicast filtering ACL 
!Link local
access-list 30 deny 224.0.0.0 0.0.0.255 log
! Locally scoped
access-list 30 deny 239.0.0.0 0.255.255.255 log
! sgi-dogfight
access-list 30 deny host 224.0.1.2 log
! rwhod
access-list 30 deny host 224.0.1.3 log
! ms-srvloc
access-list 30 deny host 224.0.1.22 log
! ms-ds
access-list 30 deny host 224.0.1.24 log
! ms-servloc-da
access-list 30 deny host 224.0.1.35 log
! hp-device-disc
access-list 30 deny host 224.0.1.60 log
! Permit all other multicast traffic
access-list 30 permit 224.0.0.0 15.255.255.255 log
!
! Block access to all but the loghost and the firewall, and log any
! denied access attempts. This also serves to create an audit trail
! of all access to the router. Extended ACLs are used to log some
! additional data.
access-list 100 remark VTY Access ACL
access-list 100 permit tcp host 192.0.2.34 host 0.0.0.0 range 22 23 log-input
access-list 100 permit tcp host 192.0.2.30 host 0.0.0.0 range 22 23 log-input
access-list 100 deny ip any any log-input
!
! Leave one VTY safe for access, just in case. The host
! 192.0.2.40 is a secure host in the NOC. If all the VTYs are
! occupied, this leaves one VTY available.
access-list 105 remark VTY Access ACL
access-list 105 permit tcp host 192.0.2.40 host 0.0.0.0 range 22 23 log-input
access-list 105 deny ip any any log-input
!
! Configure an ACL that prevents spoofing from within our network.
! This ACL assumes that we need to access the Internet only from the
! 192.0.2.32/27 network. If you have additional networks behind
! 192.0.2.32/27, then add them into this ACL.
access-list 115 remark Anti-spoofing ACL
! First, allow our intranet to access the Internet.
access-list 115 permit ip 192.0.2.32 0.0.0.31 any
! Second, allow our firewall to access the Internet. This is useful
! for testing.
access-list 115 permit ip host 192.0.2.30 any
! Now log all other such attempts.
access-list 115 deny ip any any log-input
!
! Rate limit (CAR) ACLs for UDP, ICMP, and multicast.
access-list 150 remark CAR-UDP ACL
access-list 150 permit udp any any
access-list 160 remark CAR-ICMP ACL
access-list 160 permit icmp any any
access-list 170 remark CAR-Multicast ACL
access-list 170 permit ip any 224.0.0.0 15.255.255.255
!
! Deny any packets from the RFC 1918, IANA reserved, test,
! multicast as a source, and loopback netblocks to block
! attacks from commonly spoofed IP addresses.
access-list 2010 remark Anti-bogon ACL
! Claims it came from the inside network, yet arrives on the
! outside (read: Internet) interface. Do not use this if CEF
! has been configured to take care of spoofing.
! access-list 2010 deny ip 192.0.2.16 0.0.0.15 any log-input
! access-list 2010 deny ip 192.0.2.32 0.0.0.31 any log-input
!
! Bogons
! Team Cymru has removed all static bogon references from this template
! due to the high probability that the application of these bogon filters
! will be a one-time event. Unfortunately many of these templates are
! applied and never re-visited, despite our dire warnings that bogons do
! change.
!
! This doesn't mean bogon filtering can't be accomplished in an automated
! manner. Why not consider peering with our globally distributed bogon
! route-server project? Alternately you can obtain a current and well
! maintained bogon feed from our DNS and RADb services. Read more at the
! link below to learn how!
!
! https://www.team-cymru.org/Services/Bogons/
!
! Drop all ICMP fragments
access-list 2010 deny icmp any any fragments log-input
! Allow IP access to the intranet (firewall filters specific ports)
access-list 2010 permit ip any 192.0.2.32 0.0.0.31
! Allow multicast to enter. See also access-list 30 for more
! specific multicast rules.
access-list 2010 permit ip any 224.0.0.0 15.255.255.255
! Our explicit (read: logged) drop all rule
access-list 2010 deny ip any any log-input
!
! Do not share CDP information, which contains key bits about our
! configuration, etc. This command disabled CDP globally. If you
! require CDP on an interface, use cdp run and disable cdp
! (no cdp enable) on the Internet-facing interface.
no cdp run
! SNMP is VERY important, particularly with MRTG.
! Treat the COMMUNITY string as a password - keep it difficult to guess.
! For SNMP versions 1-2
snmp-server community <COMMUNITY> RO 20
!
! Introduce ourselves with an appropriately stern banner.
banner motd %
Access to this device or the attached
networks is prohibited without express written permission.
Violators will be prosecuted to the fullest extent of both civil
and criminal law.

%
!
line con 0
 exec-timeout 5 0
 transport input none
 login authentication CONAUTH
line aux 0
 exec-timeout 5 0
line vty 0 3
 access-class 100 in
 exec-timeout 5 0
! Enable SSH connectivity.
! Obviously, you must have an IOS image that supports SSH, and don't
! forget to generate the key with crypto key generate rsa.
! To enable SSH access to the device, you additionally require a domain
! name to be set via "ip domian name x" before generating RSA keys
 ip domain-name Test.com
! Disable SSHv1
 ip ssh version 2
 transport input ssh
line vty 4
 access-class 105 in
 exec-timeout 5 0
 login authentication VTYAUTH
 transport input ssh
!
! End of the configuration.
end

Reference:

1. Cisco Guide to Harden Cisco IOS Devices
2. Secure IOS Template Version 6.5 19 MAY 2014
3. Hardening Cisco Routers
4. Team CYMRU