Lesson learned- EKS w/Fargate and ALB

This post is about lesson learned from earlier effort to get EKS cluster running with fargate compute and expose the apps with aws alb. You may read Run Serverless Kubernetes Pods Using Amazon EKS and AWS Fargate post if you have not gone through the steps. Earlier, everything worked except the ALB!

So, what was wrong? ALB ran into errors because I quoted the values in the args section of alb-ingress-controller. See the corrected version of deployment script below-

I posted the issue at aws-alb-ingress-controller on github and one of the developer pointed out the problem area after I shared the logs from the controller POD.

kubectl logs -f alb-ingress-controller-5db898488b-bqrf6 -n kube-system

E0324 01:30:51.391739 1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to unable to fetch subnets. Error: WebIdentityErr: fa
iled to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post https://sts.'us-east-1'.amazonaws.com/: dial tcp: lookup sts.'us-east-1'.amazonaws.com: no such host" "controller
"="alb-ingress-controller" "request"={"Namespace":"default","Name":"aspnetapp-ingress"}

Once I deleted the alb-ingress-controller and recreated, ALB was provisioned along with listener, rules, targets, security group, etc. based on the ingress resource definition. Now I can see the address and port fields are populated at the ingress.:)-

Advantages of letting aws manage the alb-

Anytime you change the replicateset to scale the number of PODs, alb automatically adjust the targets. ALB knows the underlying POD ip’s.
If/when you change the ingress resources, alb automatically configure rules and targets.
No of hop from alb to app is always one- alb targets are POD ip’s. This is definitely a performance boost- no need to go through another load balancer!

Now, I am going to scale down the app to 2 PODS by changing the replicaset to 2. One of the POD will be terminated.

I have not done anything to alb and targets are auto adjusted to match the number of pods. This is so nice!

How did the alb knows the POD ip and routed the traffic directly from alb to the POD? I guess, it know from the ingress resources where backends are configured as kubernetes services and service knows the PODs! See ingress resource below-

apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
   name: "aspnetapp-ingress"
   namespace: "default"
   annotations:
     kubernetes.io/ingress.class: alb
     alb.ingress.kubernetes.io/scheme: internet-facing
     alb.ingress.kubernetes.io/subnets: subnet-05dcc22c3f08db9be, subnet-06817ac3b3572158a
   labels:
     app: aspnetapp-ingress
 spec:
   rules:
 http:   paths: path: /nginx
 backend:
  serviceName: "nginx-service"
  servicePort: 80
 path: /*
 backend:
  serviceName: "aspnetapp-service"
  servicePort: 80

Proper security groups must be attached to alb for inbound traffic flow from internet to public subnet and public subnet to private subnet. They are automatically created and configured but you will need to check/update based on your security requirements.