Problem when two Checkpoint Clusters Connected on same Cisco Switch

Got mac address flapping messages on Cisco Switch log. 

Dec 22 17:27:16: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe01 in vlan 20 is flapping between port Gi0/12 and port Gi0/11

Dec 22 17:27:16: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe00 in vlan 20 is flapping between port Gi0/15 and port Gi0/16

Dec 22 17:27:31: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe01 in vlan 20 is flapping between port Gi0/12 and port Gi0/11

Dec 22 17:27:31: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe00 in vlan 20 is flapping between port Gi0/16 and port Gi0/15

Dec 22 17:27:46: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe00 in vlan 20 is flapping between port Gi0/16 and port Gi0/15

Dec 22 17:27:46: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe01 in vlan 20 is flapping between port Gi0/11 and port Gi0/12

Dec 22 17:28:01: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe00 in vlan 20 is flapping between port Gi0/16 and port Gi0/15

Dec 22 17:28:01: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe01 in vlan 20 is flapping between port Gi0/11 and port Gi0/12

Dec 22 17:28:16: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe00 in vlan 20 is flapping between port Gi0/16 and port Gi0/15

Dec 22 17:28:16: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe01 in vlan 20 is flapping between port Gi0/12 and port Gi0/11

Dec 22 17:28:31: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe00 in vlan 20 is flapping between port Gi0/16 and port Gi0/15

Dec 22 17:28:31: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe01 in vlan 20 is flapping between port Gi0/12 and port Gi0/11

Dec 22 17:28:46: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe00 in vlan 20 is flapping between port Gi0/16 and port Gi0/15

Dec 22 17:28:46: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe01 in vlan 20 is flapping between port Gi0/11 and port Gi0/12

Dec 22 17:29:01: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe00 in vlan 20 is flapping between port Gi0/16 and port Gi0/15

Dec 22 17:29:01: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.0000.fe01 in vlan 20 is flapping between port Gi0/11 and port Gi0/12

Did remember changed some settings on Checkpoint UTM clusters and messages gone. Not sure why it comes back again. After research, found why.

The environment is Gaia R75.40 now, previous it was SPLAT R70, R71 and R75. Issue was happenning for quite a while. Good thing is not affecting production at all. Maybe just we do not know. Resolved once but solution did not survive the reboot.

Here is the reason why this flapping happened based on the word from
 CP_R75.40_ClusterXL_AdminGuide.pdf:

For CCP traffic:
The first four bytes of the source MAC address are all zero: 00.00.00.00
The fifth byte of the source MAC address is a magic number. By default, the value is 0xfe.
The sixth byte is the ID of the sending cluster member

Thats why the switch has mac add flapping since duplicate CCP traffic mac address if there are two set default setting checkpoint clusters on same vlan.

The solution is quite simple and it take effect immediately with following command on one set of checkpoint cluster:

fw ctl set int fwha_mac_magic 0xee  (This command change the fifth byte MAC address from 0xfe to 0xee.)

Unfortunately this change doesn't survive a reboot. Here is final solution for permanent change:
1. Edit the file $FWDIR/boot/modules/fwkern.conf.
2. Add the line  fwha_mac_magic=0xee
3. Reboot.

Cisco Pre-defined Access-list Port Number

Working on move PIX/ASA migration to Juniper SRX. Some of ports name convention Cisco is using which is different from JunOS. I found following list to map port number to cisco name convention from a

Cisco 2901Router runing "Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.0(1)M4,"

Router(config)#access-list 101 permit tcp any any  eq ?

  <0-65535>    Port number

  bgp          Border Gateway Protocol (179)

  chargen      Character generator (19)

  cmd          Remote commands (rcmd, 514)

  daytime      Daytime (13)

  discard      Discard (9)

  domain       Domain Name Service (53)

  drip         Dynamic Routing Information Protocol (3949)

  echo         Echo (7)

  exec         Exec (rsh, 512)

  finger       Finger (79)

  ftp          File Transfer Protocol (21)

  ftp-data     FTP data connections (20)

  gopher       Gopher (70)

  hostname     NIC hostname server (101)

  ident        Ident Protocol (113)

  irc          Internet Relay Chat (194)

  klogin       Kerberos login (543)

  kshell       Kerberos shell (544)

  login        Login (rlogin, 513)

  lpd          Printer service (515)

  nntp         Network News Transport Protocol (119)

  pim-auto-rp  PIM Auto-RP (496)

  pop2         Post Office Protocol v2 (109)

  pop3         Post Office Protocol v3 (110)

  smtp         Simple Mail Transport Protocol (25)

  sunrpc       Sun Remote Procedure Call (111)

  tacacs       TAC Access Control System (49)

  talk         Talk (517)

  telnet       Telnet (23)

  time         Time (37)

  uucp         Unix-to-Unix Copy Program (540)

  whois        Nicname (43)

  www          World Wide Web (HTTP, 80)

On ASA, there are some new port name which not showing up in previous Router's settings.

ASA/act/pri(config)# access-list 10 permit tcp any any eq ?

configure mode commands/options:

  <1-65535>        Enter port number (1 - 65535)

  aol              

  bgp              

  chargen          

  cifs             

  citrix-ica       

  cmd              

  ctiqbe           

  daytime          

  discard          

  domain           

  echo             

  exec             

  finger           

  ftp              

  ftp-data         

  gopher           

  h323             

  hostname         

  http             

  https            

  ident            

  imap4            

  irc              

  kerberos         

  klogin           

  kshell           

  ldap             

  ldaps            

  login            

  lotusnotes       

  lpd              

  netbios-ssn      

  nfs              

  nntp             

  pcanywhere-data  

  pim-auto-rp      

  pop2             

  pop3             

  pptp             

  rsh              

  rtsp             

  sip              

  smtp             

  sqlnet           

  ssh              

  sunrpc           

  tacacs           

  talk             

  telnet           

  uucp             

  whois            

  www        

Actually through ASDM, there are lots of more details for those pre-defined ports name. 

   

Checkpoint R75 new feature violated PCI rules

My company recently upgraded our firewall UTM from R71 to R75. It was neat and no worries upgrade until today our External Security company sent us a report our public Internet ip scanning report failed on PCI compliance. Report shows there is self-sign checkpoint certification on our Internet facing firewall. Yes, it is right. All checkpoint firewall has a Certification on it which default is self signed. Our policy doesn't allow any http/https access to our firewalls. There is a stealthy rule in place to deny all access. 

Checked the smartview, no logs even show https traffic to be denied or accepted. 

contacted the checkpoint TAC. After hours review issues , finally TAC admit there is new feature introduced into R75 and later which call Multi-portal. KB 66030 explained how it works. 

"

Symptoms

    Even there is no rule that accepts that and the GW/MGMT are not listening to this port; there is http (port 80) access to the GW from the external network.

    From SmartView Tracker, you can see that this get accepted by rule 0 – Implied rules.

    After un-checking all the implied rules and pushing policy, it still get accepted by the implied rules. 

Cause

The gateway is responding to http traffic because of a new feature introduced in R75 called multi portal.

Because there are lots of daemons who to listen on port 80 & 443, the multi portal was introduced in R75.

In general it listens to any request on port 80 & 443 and after the 3 ways handshake is done it's forwarding request to the daemon according to the data context.

If the request to port 80 or 443 is not legitimate then the traffic is dropped.

Solution 1

Change the "multi_portal_allow_redirect" kernel parameter to 0.

Run:

# fw ctl set int multi_portal_allow_redirect 0

Please note: disabling multi-portal with kernel will break software blades other than FW, IPS, and VPN

"

*note that this change will not survive reboot.
Also this command works for R75.20 , but not for R75

To change this parameter permanently, refer to sk26202 for instructions.

"
Setting kernel global parameters permanently is unique for some OS's. 
Warning: Edit the Security Gateways ONLY. 
Note: Verify the parameter first using command fw ctl get int 
Example:   fw ctl get int fwseqvalid_exact_syn_on_rst


 fw ctl get int multi_portal_allow_redirect

Solutions:
Edit the $FWDIR/boot/modules/fwkern.conf file using vi editor or text editor to add line syntax: 
parameter_name=value 

multi_portal_allow_redirect=0

"Value" in syntax must be a valid number Decimal or Hexadecimal, as used in C programming language. 
Example: 
Decimal = a number, such as 10 or 1024.
Hexadecimal = prefixed by 0x, such as 0x10 (which is 16), 0xffff 

Note: Create the fwkern.conf file if not found in the $FWDIR/boot/modules/ directory.

Important Note: The Security Gateway must be rebooted after any change in the fwkern.conf file.
"

Solution 2: 
Another solution from CPUG Phoneboy

Edit $FWDIR/lib/implied_rules.def on the Security Management Server. Find this line: #define ENABLE_PORTAL_HTTP_REDIRECT and comment it out. After the change should look like:

/* #define ENABLE_PORTAL_HTTP_REDIRECT */

Reinstall the Security Policy.

As I said (and can confirm now) this should be addressed in a future release. 

# mpclient list
that will show a list of "processes clients' assotsiiorovannyh with multiportalom. For example, sslvpn.

Fortunately we only use FW, IPS and VPN blades, not others although is thinking to get Identity Awareness for next year. During try fw ctl set command, it doesn't works on R75, actually it works on R75.20. 

it looks we have to get a third party trust certification for our R75 firewall if we do not want to tune the kernel. That means add $600 for each internet facing firewall.

-
Another KB sk58362 for this feature in Other Improvement Section


"You can now use multiple portals over port 443 and port 80. For example, the SecurePlatform Web User interface and the Mobile Access portal can both be on port 443. In the SmartDashboard Gateway properties window, set the Portal URL for the different portals on the portal configuration pages."

Tcpdump or Fw Monitor, which is better ?

FW MONITOR
————
It is said that it captures at 4 important points in the firewall namely i,I,o & O. You would see them in the capture in the same sequence.
i – Preinbound, just where the packet is received on the interface. If you see only this then the packet is dropped by address spoofing or the access rule.
I – Postinbound, where the packet has gone across the incoming interface. If you don't see the the capture after this, you could infer that it's a routing issue.
For both i & I the interface is the incoming, where the packet enters the firewall.
o – Preoutbound, the place where the packet is received at the exit interface within the firewall. If this is the point beyond which the capture is not seen, then it turns it to be a NAT issue.
O- Postoutbound, If you see this then make yourself sure that the packet has left the firewall and the ACL, route and NAT all are correct.





TCPDUMP.
———
It captures at position i & O of firewall monitor, and you can be sure the traffic has left the firewall. It would show the return traffic as well.
This is same as the way you put captures in Cisco PIX/ASA.

So, which one you use ?

Consider you run tcpdump and see the incoming traffic but don't see the traffic leaving the exit interface. You can guess it's a routing or a NAT issue. But to make a sure shot without wasting time looking in routes or the NAT rules you could run fw monitor and know what the issue is.

Why TCPDUMP? simple, Easy to use, industry standard, aslo with layer 2 info such as ARP requests/replies as Nick said in his article Packet Captures on Secure Platform – Part 3

Why not TCPDUMP? In dealing with tcpdump on a firewall, you must really understand your network and choose the interface (-i ethX) wisely, or else you might completely miss the traffic that you are searching for. Packet Captures on Secure Platform – Part 1

What else did fw monitor show us that tcpdump did not? It shows us the routing. Let say that your ping was failing, but the SmartView Tracker logs showed it as being accepted. Packet Captures on Secure Platform – Part 2

IEEE STANDARD 802.3AD - JunOS Configuration

The  802.3ad standard supports aggregation on full duplex, point to point  links,  to form a Link Aggregation Group (LAG), so that a Media Access Control (MAC) Client can treat the LAG as if it was a single link. 

The sublayer defines multiple functions like Link Aggregation Control (LAC), Link Aggregation Control Protocol (LACP).

LAC manages the Link Aggregation sub layer by static information local to the LAG groups, and dynamic information that is exchanged as part of the LACP. Each LAG group maintains config information to control and exchange the attributes of the bundle, maintains the ports connecting and disconnecting to the bundle and also controls the Aggregators collector, distributor functions. 

In JUNOS, by default in a Link aggregation group (LAG), LACP is disabled. Following topics will look into LAG, its advantages and configuration . The following discussion will also cover the LACP function which can be enabled/disabled with LAG bundle. 

The document will also cover  the high availability feature that uses the LAG as redundant interfaces to provide LAG groups with failover properties.

IN 10.2 JUNOS version, LACP was introduced to use its benefits in a redundant interface setting and towards the end of the document it is discussed in detail. It is assumed that the reader of this document has basic understanding of SRX high end platform basic architecture. For details on certain topics and configurations, appropriate links are given in footnotes and endnotes.

WHY LINK AGGREGATION

Link Aggregation is a method which groups the multiple physical links to a single logical links for increased bandwidth. 

This gives the advantage to increase the capacity and availability of communications between network devices using Ethernet technologies. 

Higher link availability avoids the single point of failure of a physical link preventing from any  traffic outage network scenario.  If a member link goes down in a Link aggregation (LAG) group, it reduces the available capacity but traffic continues to flow through the other links without any interruption. 

One of the other advantages with Link aggregation is use of load balancing where the forwarding and processing of traffic is distributed across member links of the LAG group.  This avoids any one single physical link being overwhelmed with traffic.

JUNOS uses per packet load balancing algorithm for load balancing across LAG interfaces. The hash is based on SrcIP, DstIP, Protocol, SrcPort, DstPort,  incoming  interface index, and Type of service (ToS).  The default option is to use Layer 3 header for load balancing hash. Layer 4 header can also be configured. 

JunOS Configuration Walkthrough:

§         Chassis aggregated device counts specifies the number of LAG(AE/reth) groups defined. This config has to be deleted to take the interface out of the AE mode.

chassis {

    aggregated-devices {

        ethernet {

            device-count 2;

        }

    }

}

§         Enable the aggregation on the ge-interfaces, All links are full duplex(1gbps in this eg.)

Map the physical member links to the Aggregated virtual links ae0 or ae1.

interfaces {

    ge-0/0/0 {

        gigether-options {

            802.3ad ae0;

        }

    }

    ge-0/0/1 {

        gigether-options {

            802.3ad ae0;

        }

    }

    ge-0/0/4 {

        gigether-options {

            802.3ad ae1;

        }

    }

    ge-0/0/5 {

        gigether-options {

            802.3ad ae1;

        }

    }

ge-0/0/6 {

        gigether-options {

            802.3ad ae1;

        }

    }

§         Ae0 and ae1 are lag groups 2 ge and 3ge member links resp. AE lags can have ports from only one node.

§         The minimum link is defined as 2 which implies that atleast 2 links must be up for bundle ae0 and ae1 to be up.  The default is 1. The maximum is 16 which means all the member links should be up for AE bundle to be up.

§         Any physical property like link speed, vlan tagging, mtu, traceoptions will be defined within the two Aggregate interfaces, ae0 and ae1.               

§         Aggregated-ether-options under AE interface hierarchy can be used to set some of the bundle properties.

§         AE interfaces can also carry untagged traffic. In the eg, ae0.0 is carrying traffic for  vlan 201 and ae1.0 is carrying traffic for vlan 301.

    ae0 {

        aggregated-ether-options {

         minimum-links 2;

        }

        vlan-tagging;

        unit 0 {

            vlan-id 201;

            family inet {

                address 10.10.10.1/27;

            }

        }

    }

    ae1 {

         aggregated-ether-options {

         minimum-links 2;

        }

        vlan-tagging;

        unit 0 {

            vlan-id 301;

            family inet {

                address 20.20.20.1/27;

            }

        }

    }

§         The ae0 and ae1 have to be defined in security zones and permitted or denied by security policies.

zones {

        security-zone trust {

                  host-inbound-traffic {

                system-services {

                    ping;

                }

            }

            interfaces {

                ae0.0;

            }

        }

        security-zone untrust {

            address-book {

            host-inbound-traffic {

                system-services {

                    ping;

                }

            }

            interfaces {

                ae1.0;

            }

        }

    }

Policies

   {

default-policy {

    permit-all;

}

Check the output of the interface with the following command:

root@ > show interfaces ae0 brief