1. Find big size files
find . -type f -size +10000 -exec ls -lh {} \;root@FW% find . -type f -size +10000 -exec ls -lh {} \; -rw-r--r-- 1 930 929 134M Jan 5 17:34 ./cf/packages/junos-11.4R6.6-domestic -rw-r--r-- 1 root wheel 139M Sep 8 2011 ./cf/var/log/junos-srxsme-11.2R2.4-domestic.tgz -rw-r----- 1 root wheel 4.9M Feb 11 17:12 ./cf/var/db/idpd/db/secdb_02.db -rw-r----- 1 root wheel 6.7M Feb 11 17:13 ./cf/var/db/idpd/db/secdb_03.db -rw-r----- 1 root wheel 64M Feb 11 17:13 ./cf/var/db/idpd/db/secdb_06.db -rwxr-xr-x 1 admin 20 24M May 23 08:38 ./cf/var/db/idpd/nsm-download/SignatureUpdate.xml -r-xr-xr-x 1 root wheel 5.2M Jan 5 17:33 ./jail/html/dynamic-vpn/client/jam/InstallerComponentSRX.exe -rw-r--r-- 1 root wheel 139M Sep 8 2011 ./jail/var/log/junos-srxsme-11.2R2.4-domestic.tgz -rw-r----- 1 root config 14M Feb 8 22:16 ./mfs/var/run/db/schema.db -rw-r----- 1 root wheel 10M Feb 8 22:19 ./mfs/var/sdb/log.0000000001 -r--r--r-- 1 root wheel 6.5M Jan 5 13:59 ./usr/lib/dd/libjkernel-dd.so -r-xr-xr-x 1 root wheel 13M Jan 5 15:39 ./usr/sbin/authd -r-xr-xr-x 1 root wheel 6.0M Jan 5 16:51 ./usr/sbin/chassisd -r-xr-xr-x 1 root wheel 27M Jan 5 13:05 ./usr/sbin/flowd_octeon -r-xr-xr-x 1 root wheel 34M Jan 5 13:05 ./usr/sbin/flowd_octeon_hm -r-xr-xr-x 1 root wheel 5.5M Jan 5 16:51 ./usr/sbin/kmd -r-xr-xr-x 1 root wheel 13M Jan 5 16:24 ./usr/sbin/rpd |
% find / -size +100000 | xargs ls -lhS find: /mfs/var/spool/opielocks: Permission denied -rw-r--r-- 1 930 929 142M Aug 28 2014 /cf/packages/junos-12.1X44-D40.2-domestic -rw-r----- 1 root wheel 84M Feb 23 21:31 /cf/var/db/idpd/db/secdb_06.db |
2. Connecting to the secondary node from the primary node on an SRX cluster
On the branch SRX devices, this can be achieved by the command:
{primary:node0} lab@host-A> request routing-engine login node 1 --- JUNOS 10.1R3.7 built 2010-011-10 04:15:10 UTC {secondary:node1} lab@host-B> |
On the high-end SRX devices, you will need to be in the shell and run the following command:
root@host-A% rlogin -T node1
3. Juniper system software in-service-upgrade (Maintenance) for less downtime upgrade
JUNOS> request system software in-service-upgrade /var/tmp/junos-srxsme-12.1X44-D35.5-domestic.tgz no-sync no-validate ISSU: Validating package WARNING: in-service-upgrade shall reboot both the nodes in your cluster. Please ignore any subsequent reboot request message ISSU: start downloading software package on secondary node Pushing bundle to node1 Formatting alternate root (/dev/da0s1a)... /dev/da0s1a: 297.9MB (610028 sectors) block size 16384, fragment size 2048 using 4 cylinder groups of 74.47MB, 4766 blks, 9600 inodes. super-block backups (for fsck -b #) at: 32, 152544, 305056, 457568 Installing package '/altroot/cf/packages/install-tmp/junos-12.1X44-D35.5-domestic' ... Verified junos-boot-srxsme-12.1X44-D35.5.tgz signed by PackageProduction_12_1_0 Verified junos-srxsme-12.1X44-D35.5-domestic signed by PackageProduction_12_1_0 JUNOS 12.1X44-D35.5 will become active at next reboot WARNING: A reboot is required to load this software correctly WARNING: Use the 'request system reboot' command WARNING: when software installation is complete Saving state for rollback ... ISSU: finished upgrading on secondary node node1 ISSU: start upgrading software package on primary node Formatting alternate root (/dev/da0s2a)... /dev/da0s2a: 297.9MB (610044 sectors) block size 16384, fragment size 2048 using 4 cylinder groups of 74.47MB, 4766 blks, 9600 inodes. super-block backups (for fsck -b #) at: 32, 152544, 305056, 457568 Installing package '/altroot/cf/packages/install-tmp/junos-12.1X44-D35.5-domestic' ... Verified junos-boot-srxsme-12.1X44-D35.5.tgz signed by PackageProduction_12_1_0 Verified junos-srxsme-12.1X44-D35.5-domestic signed by PackageProduction_12_1_0 JUNOS 12.1X44-D35.5 will become active at next reboot WARNING: A reboot is required to load this software correctly WARNING: Use the 'request system reboot' command WARNING: when software installation is complete Saving state for rollback ... ISSU: failover all redundancy-groups 1...n to primary node Successfully reset all redundancy-groups priority back to configured ones. Redundancy-groups-0 will not be reset and the primaryship remains unchanged. Successfully reset all redundancy-groups priority back to configured ones. Redundancy-groups-0 will not be reset and the primaryship remains unchanged. Initiated manual failover for all redundancy-groups to node0 Redundancy-groups-0 will not failover and the primaryship remains unchanged. ISSU: rebooting Secondary Node Shutdown NOW! [pid 69249] |
4. Copy files between two Cluster Members
root@fw-2> file copy /var/tmp/junos10.3.tgz node0:/var/tmp/junos10.3.tgz |
or use fxp0.0 interface ip to do file transferring:
root@fw-2>file copy /var/tmp/junos10.3.tgz 10.2.8.4:/var/tmp/junos10.3.tgz |
5. Juniper Web Tool :
SRX HA Configuration Generator
6. Monitor Commands to do flow trace without commit
monitor security flow filter incoming-filter protocol icmp destination-prefix 1.1.1.1
monitor security flow filter outgoing-filter protocol icmp source-prefix 1.1.1.1
monitor security flow file logflow.log
show monitor security flow
show log logflow.log
monitor security flow stop
clear monitor security flow filter incoming-filter
clear monitor security flow filter outgoing-filter7. Delete Whole Configuartion
- Delete
root@fw-1# delete This will delete the entire configuration Delete everything under this level? [yes,no] (no) yes root@fw-1# set system root-authentication plain-text-password New password: Retype new password: |
- request services fips zeroize
- request system zeroize
- load factory-default
- zeroize
root@fw-1# request system zeroize |
8. Load the Commands/Configuration through the CLI
{primary:node0}[edit] root@fw-1# load set terminal [Type ^D at a new line to end input] load complete {primary:node0}[edit] root@fw-1# {primary:node0}[edit security zones] root@fw-1# load merge relative terminal [Type ^D at a new line to end input] security-zone C-Clients { tcp-rst; address-book { address n-10.9.20.0 10.9.20.0/24; } interfaces { reth1.0; } } load complete {primary:node0}[edit security zones] |
9. Troubleshoot traffic flowing through SRX
root@fw-srx1> show configuration security flow | display set set security flow traceoptions file flow-trace set security flow traceoptions flag basic-datapath set security flow traceoptions packet-filter f0 source-prefix 10.9.20.14/32 set security flow traceoptions packet-filter f0 destination-prefix 10.9.1.5/32 commit and quit |
10. Disable Juniper SRX Clustering
{primary:node1} user@host> set chassis cluster disable reboot Successfully disabled chassis cluster. Going to reboot now. |
11. Commands to collect logs for JTAC
- request support information | save /var/log/rsi.node0.txt
- file archive compress source /var/log/* destination /var/tmp/varlog.node0
12. Installs the new software package
root@fw-2> request system software add /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz no-validate no-copy partition Checking for free space on /var/tmp/install Copying package /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz to /var/tmp/install Checking for basic sanity of copied package WARNING: Installation with partition option will take several minutes to complete WARNING: Please do not power-cycle the system till the installation is over WARNING: A reboot is required to install the software WARNING: Use the 'request system reboot' command immediately {secondary:node1} root@fw-2> request system reboot Reboot the system ? [yes,no] (no) yes Shutdown NOW! [pid 2098] |
Note: Partition option introduced in the command in Release 10.1. of Junos OS. (Optional) Partition option will do format and re-partition the media before installation.
13. Packet Captures on Juniper SRX
Juniper SRX packet captures
1. Configure the forwarding options and limits
configure shared
set forwarding-options packet-capture file filename JTAC.pcap files 10 size 10000
set forwarding-options packet-capture maximum-capture-size 1500
2. Create your filter
set firewall filter PCAP term capture from source-address 192.168.1.1.32
set firewall filter PCAP term capture from destination-address 10.15.61.45/32
set firewall filter PCAP term capture from protocol tcp
set firewall filter PCAP term capture from destination-port 443
set firewall filter PCAP term capture then accept
set firewall filter PCAP term allow-all-else then accept
3. Define the interface(s) that will capture the trafffic
set interfaces ge-0/0/3 unit 0 family inet filter input PCAP
commit and-quit
**** input indicated to capture packets received. PCAP is the name of our firewall filter*****
4. Read the tcpdump file from the shell
start shell
cd /var/tmp
tcpdump –r pcap.ge-0.0.3
5. Cleanup
cli
configure
delete interfaces ge-0/0/3 unit 0 family inet filter input PCAP (stops the cap)
delete firewall filter PCAP (turns off the filter)
delete forward-options packet-capture
commit and-quit
% rm /var/tmp/pcap.ge-0.0.3.
14. Delete IDP database for System Software Upgrade
Some old SRX series does not have enough storage space to hold the new software release if enabled some other features, such as IDP. IDP database will easily hold 180M space on your system. Based on the information from http://jsrx.juniperwiki.com/index.php?title=IDP#Troubleshooting, to upgrade srx system without enough space, we can delete everything in the idp database, then add your new image and reboot:
root@fw-m-test-1> start shell root@fw-m-test-1% cd /var/db/idpd/db/ root@fw-m-test-1% rm -rfv * |
root@fw-m-test-1% cli {primary:node0} root@fw-m-test-1> {primary:node0} root@fw-m-test-1> request system storage cleanup List of files to delete: Size Date Name 4403B Mar 22 15:54 /cf/var/log/default-log-messages.0.gz 66B Feb 2 21:31 /cf/var/log/idpd_err.20170202 88B Mar 22 01:38 /cf/var/log/idpd_err.20170322 152B Mar 22 15:54 /cf/var/log/interactive-commands.0.gz 9712B Mar 22 15:54 /cf/var/log/messages.0.gz 35.9K Mar 19 12:45 /cf/var/log/messages.1.gz 146B Mar 22 15:54 /cf/var/log/traffic-create.0.gz 144B Mar 22 15:54 /cf/var/log/traffic-deny.0.gz 143B Mar 22 15:54 /cf/var/log/traffic-log.0.gz 28.9K Mar 22 15:53 /cf/var/log/wtmp.0.gz 34.1K Feb 24 20:32 /cf/var/log/wtmp.1.gz 52B Jan 25 21:06 /cf/var/log/wtmp.2.gz 60B Mar 15 00:32 /cf/var/tmp/policy_status 0B Feb 13 21:17 /cf/var/tmp/spu_kmd_init Delete these files ? [yes,no] (no) yes {primary:node0} root@fw-m-test-1> show version node0: -------------------------------------------------------------------------- Hostname: fw-m-test-1 Model: srx240h JUNOS Software Release [12.1X44-D40.2] node1: -------------------------------------------------------------------------- Hostname: fw-m-test-2 Model: srx240h JUNOS Software Release [12.1X44-D40.2] {primary:node0} root@fw-m-test-1> exit root@fw-m-test-1% du -sh /cf/var/db/ 187M /cf/var/db/ root@fw-m-test-1% du -sh /cf/var/db/idpd 167M /cf/var/db/idpd root@fw-m-test-1% cd /var/db/idpd/db/ root@fw-m-test-1% ls __db.register secdb_01.db secdb_03.db secdb_05.db secdb_07.db secdb_09.db secdb_12.db secdb_14.db secdb_16.db secdb_18.db dfa_cache secdb_02.db secdb_04.db secdb_06.db secdb_08.db secdb_10.db secdb_13.db secdb_15.db secdb_17.db root@fw-m-test-1% rm -rfv * __db.register dfa_cache/rexp_grp_cache.db dfa_cache/rexp_cache.db dfa_cache/pcre_cache.db dfa_cache secdb_01.db secdb_02.db secdb_03.db secdb_04.db secdb_05.db secdb_06.db secdb_07.db secdb_08.db secdb_09.db secdb_10.db secdb_12.db secdb_13.db secdb_14.db secdb_15.db secdb_16.db secdb_17.db secdb_18.db root@fw-m-test-1% cli {primary:node0} root@fw-m-test-1> show system storage node0: -------------------------------------------------------------------------- Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 293M 148M 121M 55% / devfs 1.0K 1.0K 0B 100% /dev /dev/md0 390M 390M 0B 100% /junos /cf 293M 148M 121M 55% /junos/cf devfs 1.0K 1.0K 0B 100% /junos/dev/ procfs 4.0K 4.0K 0B 100% /proc /dev/bo0s3e 24M 194K 22M 1% /config /dev/bo0s3f 342M 167M 148M 53% /cf/var /dev/md1 168M 19M 135M 12% /mfs /cf/var/jail 342M 167M 148M 53% /jail/var /cf/var/log 342M 167M 148M 53% /jail/var/log devfs 1.0K 1.0K 0B 100% /jail/dev /dev/md2 39M 4.0K 36M 0% /mfs/var/run/utm /dev/md3 1.8M 4.0K 1.7M 0% /jail/mfs node1: -------------------------------------------------------------------------- Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 293M 148M 121M 55% / devfs 1.0K 1.0K 0B 100% /dev /dev/md0 390M 390M 0B 100% /junos /cf 293M 148M 121M 55% /junos/cf devfs 1.0K 1.0K 0B 100% /junos/dev/ procfs 4.0K 4.0K 0B 100% /proc /dev/bo0s3e 24M 196K 22M 1% /config /dev/bo0s3f 342M 160M 156M 51% /cf/var /dev/md1 168M 19M 135M 12% /mfs /cf/var/jail 342M 160M 156M 51% /jail/var /cf/var/log 342M 160M 156M 51% /jail/var/log devfs 1.0K 1.0K 0B 100% /jail/dev /dev/md2 39M 4.0K 36M 0% /mfs/var/run/utm /dev/md3 1.8M 4.0K 1.7M 0% /jail/mfs {primary:node0} root@fw-m-test-1> request system software add no-copy validate /var/tmp/junos-srxsme-12.1X46-D55.3-domestic.tgz Formatting alternate root (/dev/da0s2a)... /dev/da0s2a: 298.0MB (610284 sectors) block size 16384, fragment size 2048 using 4 cylinder groups of 74.50MB, 4768 blks, 9600 inodes. super-block backups (for fsck -b #) at: 32, 152608, 305184, 457760 Extracting /var/tmp/junos-srxsme-12.1X46-D55.3-domestic.tgz ... Checking compatibility with configuration Initializing... Verified manifest signed by PackageProduction_12_1_0 Verified junos-12.1X44-D40.2-domestic signed by PackageProduction_12_1_0 Using junos-12.1X46-D55.3-domestic from /altroot/cf/packages/install-tmp/junos-12.1X46-D55.3-domestic Copying package ... Verified manifest signed by PackageProductionRSA_2016 Hardware Database regeneration succeeded Validating against /config/juniper.conf.gz Usage: license-check -f "<features>" -m -p -q -M -u -U -V -V verify if release based licenses are present rtslib: WARNING version mismatch for msg notify msg: expected 0 got 98,a reboot or software upgrade may be required rtslib: WARNING version mismatch for msg residx: expected 98 got 0,a reboot or software upgrade may be required rtslib: WARNING version mismatch for msg unknown: expected 98 got 0,a reboot or software upgrade may be required rtslib: WARNING version mismatch for msg notify msg: expected 0 got 98,a reboot or software upgrade may be required rtslib: WARNING version mismatch for msg residx: expected 98 got 0,a reboot or software upgrade may be required rtslib: WARNING version mismatch for msg unknown: expected 98 got 0,a reboot or software upgrade may be required rtslib: WARNING version mismatch for msg notify msg: expected 0 got 98,a reboot or software upgrade may be required rtslib: WARNING version mismatch for msg residx: expected 98 got 0,a reboot or software upgrade may be required rtslib: WARNING version mismatch for msg unknown: expected 98 got 0,a reboot or software upgrade may be required Unified access control daemon: [edit services unified-access-control] Unified access control daemon: 'infranet-controller ic4500' Unified access control daemon: warning: We recommend that you configure the ca-profile to verify infranet controller's certificate in order to prevent a man-in-the-middle attack Connectivity fault management process: rtslib: WARNING version mismatch for msg notify msg: expected 0 got 98,a reboot or software upgrade may be required Connectivity fault management process: Connectivity fault management process: rtslib: WARNING version mismatch for msg residx: expected 98 got 0,a reboot or software upgrade may be required Connectivity fault management process: Connectivity fault management process: rtslib: WARNING version mismatch for msg unknown: expected 98 got 0,a reboot or software upgrade may be required Connectivity fault management process: mgd: commit complete Validation succeeded Validating against /config/rescue.conf.gz Unified access control daemon: [edit services unified-access-control] Unified access control daemon: 'infranet-controller ic4500' Unified access control daemon: warning: We recommend that you configure the ca-profile to verify infranet controller's certificate in order to prevent a man-in-the-middle attack Connectivity fault management process: rtslib: WARNING version mismatch for msg notify msg: expected 0 got 98,a reboot or software upgrade may be required Connectivity fault management process: Connectivity fault management process: rtslib: WARNING version mismatch for msg residx: expected 98 got 0,a reboot or software upgrade may be required Connectivity fault management process: Connectivity fault management process: rtslib: WARNING version mismatch for msg unknown: expected 98 got 0,a reboot or software upgrade may be required Connectivity fault management process: mgd: commit complete Validation succeeded rm: /cf/var/validate/chroot/mfs/var/etc/pam.conf: Operation not permitted rm: /cf/var/validate/chroot/mfs/var/etc: Directory not empty rm: /cf/var/validate/chroot/mfs/var: Directory not empty rm: /cf/var/validate/chroot/mfs: Directory not empty rm: /cf/var/validate/chroot: Directory not empty Installing package '/altroot/cf/packages/install-tmp/junos-12.1X46-D55.3-domestic' ... Verified junos-boot-srxsme-12.1X46-D55.3.tgz signed by PackageProductionRSA_2016 Verified junos-srxsme-12.1X46-D55.3-domestic signed by PackageProductionRSA_2016 JUNOS 12.1X46-D55.3 will become active at next reboot WARNING: A reboot is required to load this software correctly WARNING: Use the 'request system reboot' command WARNING: when software installation is complete {primary:node0} root@fw-m-test-1> request system reboot Reboot the system ? [yes,no] (no) yes Shutdown NOW! [pid 29172] {primary:node0} root@fw-m-test-1> *** FINAL System shutdown message from root@fw-m-test-1 *** System going down IMMEDIATELY The semaphore timeout period has expired. --- JUNOS 12.1X46-D55.3 built 2016-07-08 18:46:54 UTC |
16. Upgrade JunOS from RAM Disk
Here is the steps to upgrade JunOS from RAM disk if you do not have enough available space on your disk, but you have enough memory free to use.
First, make sure you have enough free memory on your system:
Method 1:
john@FW-SRX-0> show system processes extensive
node0:
--------------------------------------------------------------------------
last pid: 15489; load averages: 0.14, 0.08, 0.08 up 10+23:35:46 18:27:00
124 processes: 18 running, 92 sleeping, 2 zombie, 12 waiting
Mem: 160M Active, 32M Inact, 560M Wired, 17M Cache, 112M Buf, 201M Free
Swap:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1114 root 7 76 0 490M 46780K RUN 0 820.3H 281.10% flowd_octeon_hm
22 root 1 171 52 0K 16K RUN 0 226.7H 86.43% idle: cpu0
23 root 1 -20 -139 0K 16K WAIT 0 170:59 0.00% swi7: clock
5 root 1 -84 0 0K 16K rtfifo 0 56:51 0.00% rtfifo_kern_recv
1120 root 1 76 0 4272K 1644K select 0 38:08 0.00% license-check
1117 root 1 76 0 9864K 3088K select 0 12:40 0.00% jsrpd
1119 root 1 76 0 9924K 3544K select 0 11:29 0.00% rtlogd
25 root 1 -40 -159 0K 16K WAIT 0 11:02 0.00% swi2: net
1111 root 2 76 0 20116K 5336K select 0 9:53 0.00% pfed
1104 root 1 76 0 31052K 4920K select 0 9:03 0.00% chassisd
1105 root 1 76 0 7140K 2500K select 0 8:04 0.00% alarmd
48 root 1 -16 0 0K 16K psleep 0 6:57 0.00% vmkmemdaemon
1101 root 1 76 0 2624K 744K select 0 6:31 0.00% bslockd
26 root 1 -16 0 0K 16K - 0 5:41 0.00% yarrow
1167 root 1 76 0 14640K 5500K select 0 2:53 0.00% mib2d
1168 root 1 76 0 12680K 5480K select 0 2:50 0.00% snmpd
41 root 1 20 0 0K 16K vnlrum 0 2:34 0.00% vnlru_
Method 2:
root@FW-1> show chassis routing-engine
node0:
--------------------------------------------------------------------------
Routing Engine status:
Temperature 36 degrees C / 96 degrees F
CPU temperature 34 degrees C / 93 degrees F
Total memory 1024 MB Max 748 MB used ( 73 percent)
Control plane memory 560 MB Max 386 MB used ( 69 percent)
Data plane memory 464 MB Max 362 MB used ( 78 percent)
CPU utilization:
User 10 percent
Background 0 percent
Kernel 5 percent
Interrupt 0 percent
Idle 85 percent
Model RE-SRX240H
Serial ID AABG0967
Start time 2013-10-02 20:50:57 EDT
Uptime 19 minutes, 26 seconds
Last reboot reason 0x200:normal shutdown
Load averages: 1 minute 5 minute 15 minute
0.27 0.21 0.30
The unused part of control plane memory can be used to create a RAM disk. In the above example you can calculate it (560 - 386 = 174 MB free).
Note: Do not try to create a RAM disk if you do not have enough free memory for this operation. The system may halt and stop responding, if you run out of memory while creating or using the RAM disk.
Other steps to create a RAM Disk
0. Login as root to shell.
1. Create tmp directory: "mkdir /var/tmp/disk1"
2. Mount it to RAM: "mount_mfs -s 200m md /var/tmp/disk1/"
3. Copy Junos image to /var/tmp/disk1
4. From CLI run update "request system software add
/var/tmp/disk1/junos-srxsme-11.4R5.5-domestic.tgz no-copy no-validate"
5. If you have managed to upgrade software on node-0 move to node-1 and upgrade Junos on it.
6. Reboot both nodes at same time.
Note: after reboot, the files in RAM disk will disappear automatically.
17. Enable Security Alarms
References:
1. Day One: Junos Tips, Techniques, and Templates 20112. Junos tips and tricks from My work help blog
