End to end TLS/SSL Offloading with Application Gateway and Kubernetes Ingress

You may have seen my post, about a week ago, on unsuccessful attempt to integrate end to end TLS/SSL offloading with Azure Application Gateway and Azure Kubernetes Ingress. Now that issues are behind, I thought it would be nice to share the experience and to document the lesson learned. You may find related documentations on Microsoft site but none of them satisfy the solution architecture that I am trying to implement with additional security and governance controls.

Reference Architecture of the Solution and High Level Requirements:

Business API need to be exposed to internet via Azure Application Gateway (AG). It’s obvious that AG must be exposed with a public IP but AG should reside in private virtual network (vNET) under it’s own subnet. In other words, AG will have a public IP and a private IP attached to it. Real-world business application should implement Web Application Firewall (i.e. Imperva Incapsula) in front of AG to provide additional web security, DDOS, TLS offloading, load balance, etc. capabilities and configure NSG of AG to allow traffic from WAF only.
Security is important and data in transit must be encrypted using TLS v1.2 or higher. End user application will communicate over HTTPS from internet to AG. Also, data classification (i.e. PCI) requires that communication between AG and Reverse Proxy (AKS NGINX Ingress) should happen over HTTPS. Communication inside the AKS cluster does not have to be encrypted.
Both the NGINX Ingress and API Service are exposed to outside (AG in this case) with Internal IP (Internal Load Balancer). They can’t be exposed with Public IP for security reason.
TLS Certificate exposed at the AG must be signed by a trusted authority. Self-signed or internal CA signed TLS Certificate can be used at the NGINX Ingress which is located inside the AKS cluster.
API must support Cross-Origin Resource Sharing (CORS). This is very important because our front-end application is JavaScript based (Angular 5) and modern browsers would not load the response unless CORS policy is satisfied.
TLS/SSL offloading architecture should look like the one below.

Application Architecture should look like the one below. Paths (2) and (3) involving Azure Functions no longer be used. It was done initially to control CORS policy.

Deploy API Application and expose it with internal load balancer in AKS

Our API application is already deployed in Azure AKS and we will skip the details on it. You can refer to Deploy ASP.NET Core Application in Linux Container in Azure Kubernetes Service (AKS) if you want to deploy your first application in Azure Kubernetes (AKS). Note that aspnet4you-apidemo-front-internal is deployed as service with an internal ip 10.240.0.66 over port 80. As per our requirement, TLS is not required at this point.

Deploy NGINX Reverse Proxy and expose it with internal Load Balancer in AKS

I used Microsoft documentation, Create an HTTPS ingress controller and use your own TLS certificates on Azure Kubernetes Service (AKS), as the base and customized to fit my requirements. Also, we would deploy the NGINX proxy in default namespace (as opposed to kube-system) because our API application is resident of default namespace.

I, personally, don’t like to bring too many dependencies into the solution specially introducing yet another open source component without doing due-diligence. Unfortunately, we are forced to use Helm to install NGINX! Install Helm following helm install doc. If you are on Windows OS, you have to get Helm Package using Chocolatey. What a dependency nightmare! Why can’t I just use Kubectl?

Okay, we got the Helm and we are ready to install. Remember, our NGINX ingress would be exposed with private/internal ip (internal load balancer). To do so, we need to create an input file, internal-ingress.yml, and use this file to install NGINX ingress.

controller:
  service:
    loadBalancerIP: 10.240.0.88
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
  extraArgs:
    default-ssl-certificate: "default/api-aspnet4you-aks-ingress-tls"

Note that annotations azure-load-balancer-internal? This is needed to tell Kubernetes that our service desires an internal ip to be assigned! Another important part is- extraArgs. This is how we tell Helm to use our own TLS (self-signed) certificate. Helm would use the default certificate without this instruction. Problem with default certificate is, we don’t have the PFX or root of the default cert! We can’t run helm install until we create the secret with the certificate in AKS.

Let’s create our own self-signed certificate using Openssl. If you don’t have openssl and you, like me, don’t trust internet, can get openssl as part of Git download. Openssl is located under usr/bin folder. Just add it to your path variable before using it! Be sure to change the file names and domain information.

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout C:\temp\api-aspnet4you-aks.key -out C:\temp\api-aspnet4you-aks.crt -subj "/CN=api.aspnet4you.aks.cecbae12ec774137ab64.eastus.aksapp.io/O=aspnet4you.aks"
openssl pkcs12 -inkey C:\temp\api-aspnet4you-aks.key -in C:\temp\api-aspnet4you-aks.crt -export -out C:\temp\api-aspnet4you-aks.k.pfx

Question is, what domain name should I use for this self-signed certificate? Well, there is a DNS Zone created for you when you created your AKS cluster and you gave it a name. Remember? Don’t worry, you can find it under MC_ resource group. You will need to create an A Record in your DNS Zone and map it to the IP address specified in internal-ingress.yml (10.240.0.88 in my case). I used api.aspnet4you.aks as name and it became the part of cecbae12ec774137ab64.eastus.aksapp.io. Even though DNS Zone is public but our ip is private and it would resolve internally only in the vNET.

We got the keys from our self-signed certificate and we got our FQDN configured. FQDN is not required until we are ready to test but it’s better to create as we go. Let’s create the secret so that we can get back to helm install!

kubectl create secret tls api-aspnet4you-aks-ingress-tls --key C:\temp\api-aspnet4you-aks.key --cert C:\temp\api-aspnet4you-aks.crt

Let’s try to install our ingress with helm-

helm install stable/nginx-ingress --name=aspnet4youdemo-ingress-controller --namespace default -f internal-ingress.yml --set controller.replicaCount=2

Well, you will run into errors! Microsoft instruction does not have –name parameter and it was the first issue that I encountered and solved. Depending on which namespace you are using, if you are using kube-system you will get error like- failed: namespaces “kube-system” is forbidden: User “system:serviceaccount:kube-system:default” cannot get namespaces in the namespace “kube-system”. You get this error because your cluster has RBAC enabled and the server component (Tiller) of Helm does not have permission to do the job! I followed the solution mentioned at https://stackoverflow.com/questions/48556971/unable-to-install-kubernetes-charts-on-specified-namespace. I would not give blanket permission to the ClusterRole in production but I did for this POC just to get going.

I had to delete the ingress and rerun the helm install. It worked!

helm del --purge aspnet4youdemo-ingress-controller
helm install stable/nginx-ingress --name=aspnet4youdemo-ingress-controller --namespace default -f internal-ingress.yml --set controller.replicaCount=2

It’s takes few minutes before external ip address is assigned. You can view the status by running kubectl get services command-

Okay, we got our NGINX Ingress Controller installed and you can see them in your AKS dashboard. Now we need to create the Ingress Route. This route will forward the requests to our API Service and take care of CORS settings. cors-origin takes a single value and that’s a big problem- not scalable. Create a file aspnet4you-demo-ingress.yml and update it with the following contents. Be sure to update settings relevant to your environment.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: aspnet4you-demo-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://azure.aspnet4you.com"
    #nginx.ingress.kubernetes.io/cors-allow-origin: "http://localhost:4200"
    nginx.ingress.kubernetes.io/cors-allow-headers: "Origin, Content-Type, X-Auth-Token, X-Requested-With, Accept, Authorization, X-Custom-Tracer, access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,ocp-apim-subscription-key"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
spec:
  tls:
    - hosts:
      - api.aspnet4you.aks.cecbae12ec774137ab64.eastus.aksapp.io
      secretName: api-aspnet4you-aks-ingress-tls
  rules:
  - host: api.aspnet4you.aks.cecbae12ec774137ab64.eastus.aksapp.io
    http:
      paths:
      - path: /api/ 
        backend:
          serviceName: aspnet4you-apidemo-front-internal
          servicePort: 80

Run the following command to apply the ingress route-

kubectl apply -f aspnet4you-demo-ingress.yml

Our ingress is ready and you can test it using curl command but you have to run the command within the cluster (some POD).

Our next step is the configure Azure Application Gateway so that we can call API from internet.

Configure Azure Application Gateway to use TLS and to connect to AKS Ingress

Configure the backend pool to connect to api.aspnet4you.aks.cecbae12ec774137ab64.eastus.aksapp.io

Configure HTTP Settings to use the public key (CRT) of our self-signed cert. Azure does not like .crt file. Load the crt file into your certificate store and export it as base64 encoded .CER file. Upload the .cer file in HTTP Settings.

Configure Listener for HTTPS and use the PFX of your external domain (api.aspnet4you.com). This certificate should be trust signed and you should have a password for the PFX file. Don;t share this PFX or password to anyone other than authorized personnel within your organization. You will need to enter the password at the listener along with the PFX.

Of course, you will need a rule to connect the backend pool, HTTP Setting and Listener!

Update the NSG rule to allow required ports from internet

This is a simple but very important part of end to end TLS and it gave me a lot of pain! Without this inbound rule to allow port 80,443,65503-65534 from internet to AG subnet, I could not get the HTTPS to work. Port 65503-65534 range is required for AG to do health check.

Well, that was it! WOW! We got the end to end TLS/SSL offloading working. Test it out using the URLs below-

I tried to use Letsencrypt w/o success and it brings a lot of unnecessary complexities and dependencies! I would stay away from it since I have to use trust signed certificate and I can use internally signed cert.

This post would be incomplete if I did not do some testing and very logs at the AG! Yep, visited the urls above and I can see the logs.

Happy end to end TLS/SSL offloading with Azure App Gateway and Kubernetes Ingress. Hope you found this post helpful. Feel free to connect with me at LinkedIn as comments are disabled at the portal.

Reference Architecture of the Solution and High Level Requirements:

Deploy API Application and expose it with internal load balancer in AKS

Deploy NGINX Reverse Proxy and expose it with internal Load Balancer in AKS

Configure Azure Application Gateway to use TLS and to connect to AKS Ingress

Update the NSG rule to allow required ports from internet

Leave a Reply Cancel reply