Exposing your AKS workloads using External DNS and Nginx Ingress Controller

Introduction

In this article we’re going to explore kubernetes community addons External DNS and the Nginx Ingress Controller community edition to expose our AKS Cluster workloads over custom URL’s.

I don’t have my own domain name on my Azure Tenant, so I’m going to work with a private made up Domain (hynes.pri), only route-able inside my Azure VNet.

Simple Architecture

In this simple Architecture we’re going to deploy

1. Public or Private AKS Cluster
2. Azure Private DNS (so we can create a custom DNS for this example rather than a public one)
3. Nginx Ingress Controller
4. External DNS
5. Windows Custom VM

Once we have the above deployed, we can simply RDP into the Windows Custom VM we created, open a browser, navigate to aido.hynes.pri and we’ll be able to access our workloads on AKS.

As per the dotted lines above, once you enter the URL aido.hynes.priv into your browser, it will resolve the DNS via Azure Private DNS to the IP Address of the Standard Load Balancer 10.240.0.254, so the request will get forwarded to the Standard Load Balancer. The Standard Load Balancer will just forward the request to the Nginx Ingress Controller(s) Node Port. The Ingress Controller will then decide which Cluster IP to forward the request to, and it will eventually get to the pod.

Azure Infrastructure and Configuration

Let’s setup all our infrastructure and configuration. The following walks through setting up the infrastructure and configuration step by step. If you want to automate this, please see my article on “Orchestrating Azure Resources with Ansible” https://adrianhynes.medium.com/orchestrating-azure-resources-with-ansible-fa82f4e3dfd6

Resource Group
Create a Resource Group in your Subscription e.g. adrian-group

VNet
Create a vnet in the adrian-group resource group e.g. adrian-vnet. CIDR 10.240.0.0/16 with a subnet of 10.240.0.0/24

AKS Cluster
Create a new Cluster in the adrian-group e.g. adrian-cluster.

In the Networking section, select Azure CNI and select the VNet (adrian-vnet) and default subnet created earlier.
In the Authentication section, select System Assigned Managed Identity.
Accept the rest of the defaults.

Azure Private DNS
Create an Azure private DNS called hynes.priv in the adrian-group resource group.

Managed Identity
Create a User Managed Identity e.g. adrian-identity.

Get the managed identity client id (see above)and assign it the Contributor role. (Note: the Contributor role here is a bit excessive as it’ll just need to be able to managed Azure Private DNS records)

az role assignment create --role “Contributor” --assignee 9b1cf06b-137f-4ac0–9526-***** --scope /subscriptions/<subscription id>

Add the managed identity to the vmscaleset

The VMScaleSet will be located in the Shadow Resource Group AKS Service Creates

Cluster System Managed Identity

Enable the Cluster System Managed Identity to obtain an IP Address from our Cluster Subnet when creating a Load Balancer

Get the System Managed Identity Principal ID of the cluster and assign it the Contributor role. (Note: The Contributor role is a bit excessive as we only need the Network Join permission scoped to subnet but we’ll keep it open for now)

az aks show -g adrian-group -n adrian-cluster --query “identity”
{
  "principalId": "ABC-123",
  "tenantId": "XYZ",
  "type": "SystemAssigned",
  "userAssignedIdentities": null
}
az role assignment create --role “Contributor” --assignee ABC-123 --scope /subscriptions/<your subscription ID>

Virtual Network Link to Azure Private DNS

Add a Virtual Network link from our Private DNS to our VNet. This will allow the custom vm we are going to deploy to resolve DNS via our Private DNS

Testing VM

Create a new VM (Windows)in the above VNet (adrian-vnet) with a public IP Address and allow inbound connection over RDP.

Kubernetes Resources

Ingress Controller

Add the k8s ingress nginx as a local helm repo

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

Create a yaml file to specify a static private ip address as well as an annotation to specify that the load balancer will created with a static Private IP

#internal-ingress.yaml
controller:
  service:
    loadBalancerIP: 10.240.0.254
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Install the helm chart specifying the above internal-ingress.yaml to override these properties

helm install nginx-ingress ingress-nginx/ingress-nginx   --namespace default  -f internal-ingress.yaml  --set controller.replicaCount=2 --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux  --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux

To validate, describe the new nginx ingress LoadBalancer Service created. The events section should contain information about the load balancer been ensured.

kubectl describe svc nginx-ingress-ingress-nginx-controller

You will also see a new Load Balancer in your MC_* shadow resource group called kubernetes-internal with the IP Address we specified above

External DNS

Apply the following file to the default namespace

kubectl apply -f external-dns.yaml

Replace Subscription ID, and the User Managed Identity Client ID (adrian-identity we configured above) in the below manifest

#external-dns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"] 
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=hynes.pri
        - --provider=azure-private-dns
        - --azure-resource-group=<Resource Group>
        - --azure-subscription-id=<Subscription ID>
        env:
        - name: AZURE_TENANT_ID
          value: <Tenant ID>
        - name: AZURE_CLIENT_ID
          value: <adrian-identity client id>
        - name: AZURE_SUBSCRIPTION_ID
          value: <Subscription ID>
        image: k8s.gcr.io/external-dns/external-dns:v0.7.0

Deploy a sample Application

Apply the below example to the default namespace

kubectl apply -f example.yaml
#example.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-two  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-two
  template:
    metadata:
      labels:
        app: aks-helloworld-two
    spec:
      containers:
      - name: aks-helloworld-two
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "AKS Ingress Demo"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-two  
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-two
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - host: aido.hynes.pri
    http:
      paths:
      - backend:
          serviceName: aks-helloworld-two
          servicePort: 80
        path: /(.*)

You should be able to verify everything is working ok by getting the ingress and making sure the Load Balancer IP Address we statically set earlier corresponds to the Address below. Next check the external DNS logs to make sure the A and TXT records get added to the Private DNS.

kubectl get ing
NAME                   HOSTS             ADDRESS        PORTS   AGE
hello-world-ingress    aido.hynes.priv   10.240.0.254   80      5h54m

Test

To test, connect to the Windows VM over RDP

Once you get into your Windows VM, open a browser and navigate to http://aido.hynes.pri/ and you should get the following

Conclusion

I hope you found this useful, and it helps you on your Kubernetes Journey. We’ll use this setup for future articles.

References

Create an ingress controller - Azure Kubernetes Service

kubernetes-sigs/external-dns