If you work in Risk and Security organization and your applications or micro-services process PCI or GDPR data, your important priority is to protect data-in-motion (flight) and data-at-rest. Welcome aboard and this post is for you! In the related post, we have covered the basics of “End to end TLS/SSL Offloading with Application Gateway and Kubernetes Ingress” and how to implement the solution in Azure. This post dials into more technical details, challenges and help you solve the puzzles.
You may see some changes in the settings in this update. It’s because I was forced to recreate new instance of AKS. Why? I needed private DNS Zone service and joint it to an existing vNET. Sadly, Microsoft would not let you join the private DNS to an existing vNET if you have VM’s in the vNET. Too silly! You can create A records in private DNS and use the internal domain name, with matching internal/self-signed certificate, when configuring ingress (NGINX).
Let’s revisit the reference architecture before we dive into how it works-
We are going to cover data-in-motion in this post and it all starts from user/client. Let’s use a browser to initiate a request to micro-service deployed in Kubernetes (AKS).
As far as user is concern, user expects to get a response over TLS v1.2 connection and data is encrypted between the browser and Azure App Gateway. User experience is important and user should not be prompted for invalid certificate. So, certificate must match the domain name (api.aspnet4you.com) and it must be valid (non-expired) and trusted (by a well-known certificate authority).
TLS v1.2 is compliant with PCI/GDPR requirement. Compliance is one thing and security is another! Just because the connection is using TLS v1.2, it is not enough and, as security best practice, we have to check if web server is configured with weak ciphers. Let’s do a ssl server test with Qualys SSL Labs.
Too bad that we got B overall rating but don’t be panicked! As you can see, app gateway HTTPS Listener is configured with v1.2 only and all other protocols specially the weakest ones are off. We got B because we are allowing some of the weak ciphers at the app gateway. This is where we have to take an educated decision by removing weak ciphers. However, doing so will prevent some users/clients from establishing TLS connection to the server. Make your choice- security and/or cost of doing business!
We can configure app gateway listener and edit the cipher suites-
Keep in mind, app gateway is exposed over HTTP and HTTPS. User (hacker) will try to utilize HTTP protocol to retrieve data! We must auto redirect all HTTP requests to HTTPS at the server-side. We can do this by telling the HTTP listener to permanently redirect to HTTPS listener with query string and path.
Okay, so far we have secured the connection from one point to another between the browser/client and app gateway. This is good and it’s the most important part to guarantee data encryption while in motion over the public network (internet). How do we secure connection between app gateway and backend pool (Kubernetes Ingress, NGINX) residing inside a vNET? Are we required to use 3rd party trust signed certificate at the NGINX?
If you are dealing with sensitive data, it is better to be encrypted while in motion between app gateway and Kubernetes cluster (at least up to the NGINX proxy). We can use internal or self-signed certificate at NGINX, just make sure key strength is RSA 2048 bits or higher. You can follow my previous post if you are not familiar with how to implement TLS at NGINX. In this post, we will focus on configuring app gateway HTTPS Setting to specify matching public key (.CER). This certificate is used to authenticate with backend pool. If you don’t configure correct public cert, you will get error- Back-end server certificate is not whitelisted for an application gateway.
BTW, while we are on this subject, let me mention the use of custom probe. App gateway listeners will not work unless you configure custom probes!
Check the backend health to make sure probe is working, it’s very important!
Okay, our app gateway is healthy. We are ready to test and validate our requests are indeed following the TLS paths! I used Postman to run few tests and we should have enough logs.
First, we are going to check the logs at the app gateway. Yep, app gateway logs are found in AzureDiagnostics.
Next, we need to see flow logs. Flow log is configured to go into storage account. We will download the file from storage account based on the TimeGenerated at app gateway. We would be looking for flow logs between Postman(client) and app gateway.
Rule : UserRule_InternetToAppGateway defined at the App Gateway NSG flows the traffic from internet to app gateway private ip (x.x.92.217 -> 172.240.16.5) on port 443.
Rule : DefaultRule_AllowVnetOutBound allows traffic between app gateway to Kubernetes ingress (172.240.16.5 -> 172.240.0.88) on port 443. Did you notice, app gateway and Kubernetes are on different subnet? Here, we are able to demonstrate our design indeed matches the implementation. Traffic flow between Kubernetes ingress and Application POD is not going over tls connection.
Hope you enjoyed reading this post and found something educational. Have any question or comment? Feel free to send them on via LinkedIn.