Exposing your AKS workloads using External DNS and App Gateway

4 min readDec 5, 2020

Introduction

We’re going to follow on from the last article https://adrianhynes.medium.com/exposing-your-aks-workloads-using-external-dns-and-nginx-ingress-controller-434482ea153b where we deployed the kubernetes community addons External DNS and the Nginx Ingress Controller community edition to expose our AKS Cluster workloads over custom URL’s.

In this article we’re going to add App Gateway as an Ingress option to our AKS cluster.

Simple Architecture

In this simple Architecture we’re going to deploy

Public or Private AKS Cluster
Azure Private DNS (so we can create a custom DNS for this example rather than a public one)
Nginx Ingress Controller
App Gateway
External DNS
VM

The only difference here from the previous article is we are creating an App Gateway in a new subnet in the vnet.

Steps

Ansible Playbook

Apply the Ansible Playbook from here: https://github.com/aido123/ansible/blob/main/azure_ansible_appgateway.yaml subbing in your own values as required

ansible-playbook azure_ansible_appgateway.yaml --extra-vars "resource_group_name=myrsg subscription_id=ABC123 tenant_id=DEF456"

App Gateway Helm Chart

Pull down the latest App Gateway Helm Chart

helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/helm pull application-gateway-kubernetes-ingress/ingress-azuretar -xvf ingress-azure-<version>.tgz

Add the env’s AZURE_TENANT_ID and AZURE_SUBSCRIPTION_ID with your own tenant and subscription to the configmap located in ingress-azure/templates/configmap.yaml

...
{{- if eq .Values.armAuth.type "aadPodIdentity"}}
  AZURE_CLIENT_ID: "{{ .Values.armAuth.identityClientID }}"
  AZURE_TENANT_ID: "ABCD-1234"
  AZURE_SUBSCRIPTION_ID: "WXYZ-9876"
...

Remove the files ingress-azure/templates/aadpodidbinding.yaml and ingress-azure/templates/aadpodidentity.yaml. These are use for configuring pod identity. We’re just attaching our user managed identity to the vmscaleset for demo purposes

Apply the helm chart plugging in your own config

helm install ingress-azure \
     ingress-azure \
     --namespace default \
     --debug \
     --set appgw.resourceGroup=<resource group name> \
     --set appgw.environment=AZUREPUBLICCLOUD \
     --set appgw.subscriptionId=<WXYZ-9876> \
     --set appgw.shared=false \
     --set appgw.name=myApplicationGateway \
     --set appgw.usePrivateIP=true \
     --set armAuth.type=aadPodIdentity \
     --set armAuth.identityClientID=<User Managed Identity Client ID> \
     --set rbac.enabled=true \
     --set verbosityLevel=3 \
     --set aksClusterConfiguration.apiServerAddress=<guid>.hcp.<location>.azmk8s.io

Nginx Ingress Controller

Apply the Nginx Ingress Controller as per my other Article https://adrianhynes.medium.com/exposing-your-aks-workloads-using-external-dns-and-nginx-ingress-controller-434482ea153b

Externdal DNS

Apply External DNS as per my other article https://adrianhynes.medium.com/exposing-your-aks-workloads-using-external-dns-and-nginx-ingress-controller-434482ea153b

App Gateway Ingress Example

Apply the App Gateway Ingress example as per below. Notice the different ingress annotations compared to our previous Nginx example.

kubectl apply -f example-app-gateway.yaml#example-app-gateway.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-two
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-two
  template:
    metadata:
      labels:
        app: aks-helloworld-two
    spec:
      containers:
      - name: aks-helloworld-two
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "AKS Ingress Demo"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-two
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  selector:
    app: aks-helloworld-two
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - host: aido.hynes.pri
    http:
      paths:
      - backend:
          serviceName: aks-helloworld-two
          servicePort: 80
        path: /

Nginx Ingress Example

We’ll also apply the following Nginx Ingress example to show them side by side.

kubectl apply -f example-nginx.yaml#example-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-two
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-two
  template:
    metadata:
      labels:
        app: aks-helloworld-two
    spec:
      containers:
      - name: aks-helloworld-two
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "AKS Ingress Demo"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-three
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-two
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress-1
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - host: bido.hynes.pri
    http:
      paths:
      - backend:
          serviceName: aks-helloworld-three
          servicePort: 80
        path: /(.*)

Test

RDP into the Custom VM and open a terminal. Perform an nslookup of both aido.hynes.pri and bido.hynes.pri.

aido.hynes.pri will resolve to App Gateway’s IP Address

bido.hynes.pri will resolve to the Standard Load Balancer IP Address (created via Nginx Ingress Controller Loadbalancer Service)

Type http://aido.hynes.pri into a browser on the Widnows VM and you will get the following

Type http://bido.hynes.pri into a browser and you will get the following

Conclusion

I hope you found this article useful in setting up an Application Gateway (Via Helm Charts) for your AKS Cluster.

Note, it’s possible to setup App Gateway in your AKS clusters via the az aks cli, which I have not looked at here.