AWS Gateway Load Balancer (GWLB)

 The goal of this post is to test another AWS Cloud Network product called GWLB (Gateway Load Balancer). 

AWS GWLB will be tested in a pure AWS scenario but we will see at the end of this post some explanations on the integration of AWS GWLB with Aviatrix Firenet.

NOTE: later in the summer, a specific post on Aviatrix Firenet with AWS GWLB integration will be released. You then will be able to see the magnitude of AWS GWLB & Aviatrix Firenet combined.

WHAT is AWS GWLB?

AWS GWLB enables you to deploy, scale & manage virtual appliances (FW, IDS & IPS & DPI (Deep Packet Inspection) systems. It combines a transparent network GW & distributes traffic while scaling your virtual appliances with the demand.

AWS GWLB operates @3rd-4th layer of the OSI model. It listens for all IP packets across all ports & forwards traffic to the target group specified in the listener rule. It maintains stickiness of flows to a specific target appliance using 5-tuple (for TCP / UDP flows) or 3-tuple (for non-TCP / UDP flows)). AWS GWLB & its registered virtual appliance instances exchange application traffic using GENEVE protocol on port UDP/6081.

NOTE: GENEVE encapsulation is a network encapsulation protocol created by IETF in order to unify the efforts made by other initiatives like VXLAN & NVGRE.

AWS GWLB High Level

AWS GWLB acts as a L3 GW (no packet rewrite) but as well as a L4 Load Balancer (elasticity of the appliances, stickiness of flows in both directions).

It also provides Health Checks (if 1 of the appliances goes down, AWS GWLB starts rerouting the flows by encapsulating original traffic in a layer3 header called GENEVE). 

AWS GWLB Components & Behaviour

Components

• GWLB is a managed service for the customers to deploy & manage a fleet of horizontally scalable inline network virtual appliances in a transparent manner. This is the control plane where configurations & rules reside.

• GWLBE is the data plane component of GWLB, pushing the packets & filtering the traffic. GWLBE is similar to AWS Private Link allowing you to place your service across accounts & VPC's without losing centralized control & administration. 

GWLB has 2 sides. The side connecting GWLBE is called frontend. The side connecting appliances is called backend. 

In the backend, GWLB acts as a Load Balancer for routing traffic flows through target appliances. GWLB ensures stickiness of the flows in both directions to target appliances & also reroutes flows if 1 appliance becomes unhealthy. 

Packets sent from source to destination do not contain GWLB IP as destination IP@ but will be routed to GWLB due to Route Tables configuration. To achieve this transparent forwarding behaviour, GWLB encapsulates the original packet using Geneve encapsulation. Appliances also need to decapsulate Geneve TLV's (Type-Length-Value) pairs to process the original packet.

Appliance selection by GWLB for a specific flow

• If TCP/UDP flow, GWLB selects a healthy appliance from a target group based on 5-tuple hash (Source/Destination IP, Protocol, Source/Destination Port)

• If non TCP/UDP flow, GWLB uses 3-tuple (Source/Destination IP, Protocol)

GWLB & Health Checks

• TCP, HTTP or HTTPS

Packet walk

Step1: GWLBE receives packets from source & sends packets to GWLB using AWS Private Link

Step2: GWLB uses 3/5-tuple to send to an appliance. GWLB then encapsulates original packet in Geneve header (outer IPv4 header is with Src IP = GWLB IP & Dst IP = Appliance IP) & adds Geneve TLV's (vpce-id, attach-id & flow cookie).

Step3: GWLB forwards encapsulated to an appliance.

Step4: The appliance: 

    - Decapsulates the packet

    - Reads TLV's & saves flow cookie

    - Drop or Allow packet (depending on Appliances rules)

    - Encapsulates packet back & attaches same TLV's

    

Step5:  Appliance sends packet back to GWLB

Step6: GWLB removes Geneve encapsulation, validates 3/5-tuple & flow cookie & forwards packet to GWLBE.

Step7: GWLBE sends back packet to source using AWS Private Link.

GWLB architecture patterns

Please see this link for more details.

• Single VPC with North / South connectivity: represents a VPC requiring a FW or inline function to be placed between resources inside the VPC & outside the VPC (like Internet). This will be the scenario tested later as the focus is more testing the GWLB itself rather than complex scenarios.

• Many VPC's with centralized North / South connectivity across AWS TGW

• Many VPC's with centralized East / West connectivity across AWS TGW

NOTE: a single VPC requiring East / West inspection between subnets is not supported.

NOTE: VPC Ingress is also supported (see link above).

Best practices for deploying AWS GWLB

Tune TCP-keepalive or timeout values to support long-lived TCP flows

GWLB has a fixed idle timeout of 350 seconds for TCP flows & 120 seconds for non TCP flows. Once the idle timeout is reached for a flow, it is removed from GWLB connection state table. Then, the subsequent packets for that flow are treated as a new flow & may be sent to another healthy instance. This could result in the flow timing out on the client side.

It is recommend to configure TCP keepalive of client / server / FW to less than 120sec/350sec.

Enable 'Appliance mode' on AWS TGW to maintain flow symmetry for E/W inter-VPC traffic inspection

Thus ensures symmetry for forward / reverse flow the same appliance.

Understand when to use Cross-Zone Load Balancing

When enabled, GWLB distributes traffic across all registered & healthy targets regardless of which AZ's these targets are in. 

Choose 1-arm or 2-arm deployment for egress traffic inspection

1-arm mode is the most common deployment method & eliminates dependency on appliance supporting NAT. It also increases performance by offloading NAT.

In 2-arm mode fashion, 1 ENI is in private subnet & the other in public subnet. This requires support from the vendor.

AWS GWLB vs NLB vs ALB

The table below is a very high level of comparison between the 3 types of Elastic Load Balancers that AWS provides & what are the specific use cases.

Single Spoke VPC with N/S connectivity (to Internet) configuration

Architecture build

1. Create Security VPC & its 2 associated subnets (in 2 different AZs)

2. Create Appliances in Security (1 per AZ). Special care about Security Group as follows: Health Check (here, HTTP) & Geneve (UDP/6081) must be allowed.

3. Creation of the Spoke VPC, its 4 associated subnets in 2 different AZs (2 for EC2 in each AZ & 2 for GWLBE in each AZ)

4. Creation of 5 RTs for Spoke VPC:

    - Ingress & do 'Edge Association' with IGW

    - 2 Public  GWLBE RT (1 per GWLBE) & associate the corresponding subnet

    - 2 Private RTs (1 per Spoke EC2) & associate the corresponding subnet

5. Create GWLB (EC2 - Load Balancer - GWLB)

6. Basic Network Configuration (mapping with the 2 AZs in Security VPC)

7. Create Target Group (Step1 / Security VPC; Health Check in HTTP)

8. Create Target Group (Step2 / Register Targets (ie the 2 appliances created in the Security VPC)

9. Associate Target Group to GWLB

10. Configure the Security appliances to allow GENEVE encapsulation, listen to HTTP (for health checks) & pass through the traffic without inspection as per link (good luck & Thanks Mihai :-) )

NOTE: we have to do the steps 10.1 & 10.2 as our EC2 is not a 3rd party Security appliance & does not talk GENEVE, nor respond to HTTP/80 or inspect traffic.

10.1 Downloading & installing necessary tools & programs

10.2 Geneve encapsulation, Traffic pass-through & Health Checks

Check the Target appliances answer the Health Checks

11. Create Endpoint Service (no acceptance required for the sake of this test)

12. Deploy Endpoints in Spoke VPC (search Service Name as specified in the below screenshot): 1 GWLBE per AZ

13. Update the Spoke VPC Route Tables  (as below)

13.1 Ingress RT to return traffic to Spoke EC2 via GWLBE

13.2 Each Private RT to send traffic to Internet via GWLBE

13.3 Each PublicGWLBE RT to send traffic to Internet via IGW

14. Enable Cross Zone Load Balancing (Edit Attributes of GWLB)

Testing

1. Testing in initial mode (Spoke EC2 in AZ1 will talk with 1 of the 2 Appliances (here the one in AZ1)

TCPDUMP on Appliance AZ1

2. Testing in failure mode & validate 'cross zone load balancing' - Stop Appliance in AZ1

TCPDUMP on Appliance AZ2 (validating it takes over the traffic & validating then 'cross zone load balancing')

AVIATRIX & AWS GWLB

Definition

Aviatrix Firenet allows you to seamlessly deploy, scale, service chain & operate NG Firewalls & Security appliances (can be Check Point, F5, Fortinet, Palo Alto, etc..). Aviatrix Firenet connects virtual appliances to Cloud Transit Network. Traffic is routed through these security appliances for inspection to enforce corporate & regulatory security policies.

Aviatrix Firenet leverages AWS GWLB & GWLBE (endpoints) to scale & manage these appliances that support GENEVE encapsulation.

Advantages

• GWLB handles symmetric hashing, connection draining & failover

• High performance connection to virtual appliances

• No need for SNAT

• Allows customers to remove / add appliances for scaling or in response to health checks without impacting existing sessions

• Aviatrix Controller automates attachment of AWS GWLB / GWLBE & all connected appliances

• Automation of the propagation of routes to apps. (ie updates of VPC RTs to direct traffic through GWLB & on the appropriate connected FW appliance)

NOTE: if you can't wait for my post about Aviatrix implementation with AWS GWLB & Aviatrix Firenet, please check out this link.

AWS Network Firewall

 

That has been a while I did not post. The reason is quite simple: I got to know my new colleagues, met my first customers and trained the students for certifications (really high level summary). 

As multi-cloud Solutions Architects in Aviatrix, we don't have only to know Aviatrix solution in and out but also deep dive on the solutions the CSP's provide. And that is the case today.

AWS released sometime ago the AWS Network Firewall but I did not get the chance (yet) to dig a bit on that.

Lessons learned

I am not a beginner in AWS: I have 2 years of experience with that Public Cloud, got certified with Advanced Network Specialty & did the architecture and deployment of the Cloud network infrastructure for SWIFT. But, ho boy, what a complexity. 

I am not talking about the AWS Network Firewall itself (5-tuple & Domain List are straight forward but IPS with Suricata...not!), I am talking about the architecture build and how to be in the position to start the AWS Network Firewall implementation.. And for only a single AZ! (the AWS Network Firewall resiliency was not in scope as I wanted to focus more on the 3 main functionalities: packet filtering, Domain Filtering & IPS). You will see below all the needed steps to get there!

I had to reach out 2 of my colleagues Abdul Rahim & Evgeny Vaganov; 2 former superstars of AWS - ex 'AWS Network Black Belt -  to sort out a routing issue. I even opened a case to AWS Support (with not much result). 

Having said that, let's talk about interesting stuff!

WHAT is AWS Network Firewall?

AWS Network Firewall is a stateful, managed, network firewall and intrusion detection & prevention service for your VPC(s). 

You can filter traffic @perimeter of your VPC. This includes traffic from / to IGW, NATGW, VPN or DX. 

AWS Network Firewall works together with AWS Firewall Manager so you can build policies based on AWS Network Firewall rules and then centrally apply those policies across your VPCs and accounts [out of scope of this post].

WHAT are the functionalities of AWS Network FW?

• Packet Filtering: it allows you to allow / deny traffic based on IP | Protocol | Ports. It can be: 
• Stateless / ACL L3 rules
• Stateful / L4 rules
• any type of traffic pattern (East / West: VPC to VPC ; North / South: VPC to On Prem ; Ingress or Egress Internet)

• FQDN filtering (aka Web filtering or Domain List) 
• HTTP / HTTPS only
• wildcards supported ('.example.com' means all subdomains of example.com)

• AWS Network FW IPS (Intrusion Prevention System) provides active traffic flow inspection with real time network / application layer protections against vulnerability exploits & brute force attacks (threat signatures based on byte sequences or packet anomalies for example). It uses the open source IPS Suricata for stateful inspection.

NOTE: AWS claims AWS Network Firewall supports Deep Packet Inspection (DPI); however it does not support for HTTPS. Therefore, it can't compare on this feature with Next Generation FWs.

WHAT are the other capabilities of AWS Network FW?

• High availability & automated scaling; it offers SLA with an uptime commitment of 99.99%. It enables you to automatically scale your FW capacity up or down based on traffic load to maintain steady, predictable performance to minimize costs.

• Flow logs: provide state information about all traffic flows passing through the AWS Network FW, with 1 line per direction (S3 or Cloud Watch)

• Throughput more than 45 Gbps

• Encryption @rest can be done with KMS

Components of AWS Network Firewall

AWS Network Firewall manages the following resources types

• Firewall - connects the inspection rules in the firewall policy to the VPC that the rules protect (aka Firewall endpoint or vpce)

• FirewallPolicy - set of stateless and stateful rule groups, along with some policy-level behaviour settings. The firewall policy provides the network traffic filtering behaviour for a firewall (can be installed on multiple FWs)

• RuleGroup - collection of criteria for inspecting traffic & for handling packets & traffic flows that match the inspection criteria (stateless & stateful)

Main limitations (Service Quotas) of AWS Network Firewall (per Region)

• Maximum number of FWs: 5 (can be increased)

• Maximum number of FW policies: 20 (can be increased)

• Maximum number of stateful or stateless rule groups: 50 (can be increased)

• Maximum number of stateful or stateless rule groups per FW policy: 20 (can't be changed)

AWS Network Firewall behaviour

When AWS Network FW inspects a packet, it evaluates against the rules in the policy's stateless rule groups first, using the stateless rules engine. Then, depending on that inspection, it might evaluate the packets against the rules in the policy's stateful rule groups, using the stateful rules engine. 

NOTE: if no match in the stateless rules engine for UDP packets, it uses default stateless rule action.

• Default Stateless
• Process UDP packets
• Forward other packets to Stateful engine

• Default Stateful by order of evaluation
• Pass
• Drop
• Alert (if you don't want to impact your traffic but want to see what the FW will do with the packet)

AWS Network Firewall Deployment Models

There are many different deployment models for the AWS Network Firewall. Please refer to this link for having an exhaustive list of these deployments.

Please see the 3 main categories for AWS Network Firewall Deployment models:

• Distributed: AWS Network Firewall is deployed into each individual VPC

• Centralized: AWS Network Firewall is deployed into a centralized VPC (called inspection VPC) for East/West (VPC to VPC) and/or North/South (Internet Ingress, Internet Egress & to On Prem) traffic. 

• Combined: AWS Network Firewall is deployed into a centralized VPC for E/W & N/S (On Prem & Egress) but Internet Ingress is distributed to VPCs which require dedicated inbound access from Internet.

Let's take 2 examples to configure:

• Centralized for E/W traffic

• Centralized for N/S Internet Egress with Web filtering & IPS

AWS Network Firewall can be HA and be deployed in a multi-AZs environment but for readability & simplicity, a single AZ will be configured in the following examples. 

NOTE: although AWS Network Firewall is represented by a single firewall endpoint per AZ, it is highly available and based on AWS hyperplane technology. 

Centralized deployment for E/W inspection model

NOTE: AWS Network Firewall is completely transparent to network traffic. We use 100.64/16 range, meaning Carrier Grade NAT range to preserve IP@.

Also, please be aware that this deployment works because Spoke A & Spoke B VPCs are in the same AZ. If they would have been in different AZs, then 'appliance mode' must be configured. (CLI only :-( )

Architecture Build

1. Creation of VPCs A & B with their 2 associated subnets

2. Creation of Inspection VPC with its 2 associated subnets. Create 2 VPC RT for that VPC as shown in the diagram above.

3. Creation of TGW

4. Creation of the 3 TGW - VPC attachments via TGW subnets

5. Creation of the 2 TGW Routing Tables

6. Add default route towards TGW for VPC Route tables for Spoke VPCs A & B

7. In Spoke Inspection TGW Route Table, create a default route towards Inspection VPC attachment

8. In TGW FW Route Table, create static routes to Spoke VPCs A & B towards their respective Spoke VPC x attachments

9. Associate the TGW-VPC Attachments to their respecting TGW Routing Tables:

    - Spoke VPC A & B attachments to TGW RT Spoke Inspection

    - Inspection attachment to TGW RT FW

NOTE: The TGW attachments are associated by default to the default TGW RT. A 'disassociation' with the default RT is then needed

10. Do explicit subnets associations to the respective VPC RT as shown in the drawing above

11. Add default route towards TGW for VPC FW Route Table

Please see hereafter how the Routing Tables should look like.

a. TGW Spoke Inspection RT

b. TGW FW RT

c. VPC Spoke A RT (Route to IGW is to my laptop)

d. VPC Spoke B RT

e. VPC FW RT

Configure AWS Network Firewall

The configuration of AWS Network Firewall is a 3-step process.

1. Create the FW endpoint and locate it into the Firewall Inspection subnet.

The Firewall endpoint type is GWLB.

2. Create FW policy for East / West traffic

3. Create a default route in the Inspection VPC Route Table pointing to the newly created FW endpoint (vpce-xxx)

NOTE: to delete Firewall, the route towards vpce-xx must be deleted first !!

4. Create Stateful rule Group to Firewall policy

Before configuring Stateful rule Group, it is important to understand the default behaviour of the Stateless engine.

• it processes the UDP (fragmented) packets

• it forwards other packets to the stateful engine

Let's configure now the stateful rules group.

• capacity is the number of maximum rule configured

5. 5-tuple has been chosen to filter on IP | Protocol / Port. In our example below only ICMP from Instance in Spoke VPC A is allowed.

• Rule 1 allows ICMP traffic from Spoke A EC2 to Spoke B1 EC2

• Rule 2 drops ICMP traffic from Spoke A EC2 to Spoke B2 EC2

6. Associate Rule Group to FW policy

7. Configure Logging for stateful rules group (to previously configured CloudWatch log groups

[AWS Network FW generates logs for stateful rule groups].

8. Testing

• ICMP from Spoke A EC2 to Spoke B1 EC2 works as per Rule1

• ICMP from Spoke A EC2 to Spoke B2 EC2 does not work as per Rule2

9. Visualisation with Cloud Watch

Only stateful rule traffic can be visualized. 

We can see with Cloud Watch Log Insights the ICMP packets. (example below)

If the Logging type you have enabled is Flow Logs, you will see all packets traversing across the FW, but you can't see the action for those packets (ie it won't show if allowed or dropped). However if you have enabled Alert Logs as well, then you will see Logs only for the packets which match the rules on which the action has been defined explicitly Alert or Drop.

Please check out how Aviatrix does L4 packet filtering with its embedded L4 Stateful FW in each Aviatrix Gateway @no additional cost.

Centralized deployment for Egress Internet model

The drawing below adds up with the previous deployment.

• Central Egress VPC must be additionally configured (including NATGW & IGW)

• TGW FW RT must have a default route towards Central Egress VPC attachment

• The traffic remains the same as per deployment above up to point 3 included (only point 4 differs)

Architecture build

1. Creation of Central Egress VPC & its 2 associated subnets

2. Creation of TGW VPC attachment for Central Egress VPC

3. Create NATGW in Public subnet

4. Create IGW & attach to Central Egress VPC

5. Create the 2 VPC Central Egress RT as shown in the diagram above ('TGW' & 'Public')

6. Do explicit subnets associations to the respective VPC RT as shown in the drawing above

7. In TGW Inspection RT, associate Central Egress VPC

8. In TGW FW RT, add default route towards Central Egress VPC attachment

9. In TGW RT of Central Egress VPC, add default route towards NATGW

10. In Public RT of Central Egress VPC, add static route to 10/8 towards TGW & default route towards IGW

AWS Network Firewall configuration with domain name filtering

1. Add a stateful rule group to the existing FW policy with 'Domain list' selected

'Domain filtering' can only be configured as Stateful rule group.

2. Allow only all subdomains of google (.google.com) for Spoke A EC2 only

It is recommended to configure 'Source IP type' rather than letting this to default because in a Centralized deployment model, only the VPC where the FW resides will be considered as allowed to initiate traffic with 'default'. Thus, with 'defined', the VPC CIDRs must be configured.

3. Testing with curl to https://www.google.com & another website

Curl to google.com is successful whereas the other is not.

NOTE: keep in mind that 'Domain List' is only for HTTP & HTTPS; meaning other protocol like ping google.com will work. To block this, you have to configure Suricata rules :-(

4. Visualization on CloudWatch of curl to google.com

Please check out how Aviatrix does FQDN filtering.

Using Suricata on AWS for threat detection and response

AWS Network Firewall uses the open source Intrusion Prevention System (IPS) for stateful inspection. This capability allows you to continuously monitor your Cloud private workloads for malicious activity & takes action to prevent it. 

1. Testing before IPS Suricata rules

We can see that the EC2 is able to make the HTTP request to a crypto miner website.

2. Configuration of new rule group called 'ips' with the option 'Suricata compatible IPS rules'

IPS can be only configured in stateful rule group.

3. Definition of Suricata rule to block the previous HTTP request to crypto mining website

• 'HOME_NET' variable is the variable for the source CIDRs where the rule applies (in our case, this is the VPC Spoke A CIDR)

• 'EXTERNAL_PORT' is the variable for the ports to be inspected (in our case, as this is a HTTP request, port 80)

4. Associate the 'ips' rule group to the FW policy

5. Re-test after applying the Suricata rule (properly blocked)

Please check out how Aviatrix does IPS with ThreatIQ / ThreatGuard @no additional cost.

Conclusion

• AWS Network Firewall can do packet filtering for any kind of traffic pattern: East/West North / South (To On prem), Ingress & Egress

• Many different deployment models which can lead to confusion

• Creation of Inspection VPC to locate your Firewall leading to re architecture / re configuration in Brownfield

• No proper tool to visualize the packets allowed or dropped other than Cloud Watch or S3 

• Domain filtering (aka Domain List) limited to http and https

• IPS is not enabled by default and IPS Suricata rules must be configured (not easy to configure)

• Not possible to see in Flow logs if a packet is passed or dropped (you have to enable Alert Logs)

• AWS Network Firewall is costly. Just for the setup above (1 FW endpoint for E/W & Egress + 2 TGW attachments for Inspection VPC & Egress VPC; not counting Spoke VPCs)
• AWS Network FW: $0.395/h*24*365 = 3460$/year + $0.065/GB
• TGW Attachment: ($0.5/h*2)*24*365 = 876$/year + $0.02/GB
• Flow Logs: $0.5/GB
• Total: $4336/year + $0.065/GB (FW) + $0.02/GB (TGW) + $0.5/GB (Flow Logs)

My company wouldn't be so happy that I run this lab for an entire year ! :-)