Exposing your AKS workloads using External DNS and Nginx Ingress Controller

Introduction

In this article we’re going to explore kubernetes community addons External DNS and the Nginx Ingress Controller community edition to expose our AKS Cluster workloads over custom URL’s.

I don’t have my own domain name on my Azure Tenant, so I’m going to work with a private made up Domain (hynes.pri), only route-able inside my Azure VNet.

Simple Architecture

In this simple Architecture we’re going to deploy

  1. Public or Private AKS Cluster

Once we have the above deployed, we can simply RDP into the Windows Custom VM we created, open a browser, navigate to aido.hynes.pri and we’ll be able to access our workloads on AKS.

As per the dotted lines above, once you enter the URL aido.hynes.priv into your browser, it will resolve the DNS via Azure Private DNS to the IP Address of the Standard Load Balancer 10.240.0.254, so the request will get forwarded to the Standard Load Balancer. The Standard Load Balancer will just forward the request to the Nginx Ingress Controller(s) Node Port. The Ingress Controller will then decide which Cluster IP to forward the request to, and it will eventually get to the pod.

Azure Infrastructure and Configuration

Let’s setup all our infrastructure and configuration. The following walks through setting up the infrastructure and configuration step by step. If you want to automate this, please see my article on “Orchestrating Azure Resources with Ansible” https://adrianhynes.medium.com/orchestrating-azure-resources-with-ansible-fa82f4e3dfd6

Resource Group
Create a Resource Group in your Subscription e.g. adrian-group

VNet
Create a vnet in the adrian-group resource group e.g. adrian-vnet. CIDR 10.240.0.0/16 with a subnet of 10.240.0.0/24

AKS Cluster
Create a new Cluster in the adrian-group e.g. adrian-cluster.

In the Networking section, select Azure CNI and select the VNet (adrian-vnet) and default subnet created earlier.
In the Authentication section, select System Assigned Managed Identity.
Accept the rest of the defaults.

Azure Private DNS
Create an Azure private DNS called hynes.priv in the adrian-group resource group.

Managed Identity
Create a User Managed Identity e.g. adrian-identity.

Get the managed identity client id (see above)and assign it the Contributor role. (Note: the Contributor role here is a bit excessive as it’ll just need to be able to managed Azure Private DNS records)

az role assignment create --role “Contributor” --assignee 9b1cf06b-137f-4ac0–9526-***** --scope /subscriptions/<subscription id>

Add the managed identity to the vmscaleset

The VMScaleSet will be located in the Shadow Resource Group AKS Service Creates

Cluster System Managed Identity

Enable the Cluster System Managed Identity to obtain an IP Address from our Cluster Subnet when creating a Load Balancer

Get the System Managed Identity Principal ID of the cluster and assign it the Contributor role. (Note: The Contributor role is a bit excessive as we only need the Network Join permission scoped to subnet but we’ll keep it open for now)

az aks show -g adrian-group -n adrian-cluster --query “identity”
{
"principalId": "ABC-123",
"tenantId": "XYZ",
"type": "SystemAssigned",
"userAssignedIdentities": null
}
az role assignment create --role “Contributor” --assignee ABC-123 --scope /subscriptions/<your subscription ID>

Virtual Network Link to Azure Private DNS

Add a Virtual Network link from our Private DNS to our VNet. This will allow the custom vm we are going to deploy to resolve DNS via our Private DNS

Testing VM

Create a new VM (Windows)in the above VNet (adrian-vnet) with a public IP Address and allow inbound connection over RDP.

Kubernetes Resources

Ingress Controller

Add the k8s ingress nginx as a local helm repo

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

Create a yaml file to specify a static private ip address as well as an annotation to specify that the load balancer will created with a static Private IP

#internal-ingress.yaml
controller:
service:
loadBalancerIP: 10.240.0.254
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Install the helm chart specifying the above internal-ingress.yaml to override these properties

helm install nginx-ingress ingress-nginx/ingress-nginx   --namespace default  -f internal-ingress.yaml  --set controller.replicaCount=2 --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux  --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux

To validate, describe the new nginx ingress LoadBalancer Service created. The events section should contain information about the load balancer been ensured.

kubectl describe svc nginx-ingress-ingress-nginx-controller

You will also see a new Load Balancer in your MC_* shadow resource group called kubernetes-internal with the IP Address we specified above

External DNS

Apply the following file to the default namespace

kubectl apply -f external-dns.yaml

Replace Subscription ID, and the User Managed Identity Client ID (adrian-identity we configured above) in the below manifest

#external-dns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: external-dns
rules:
- apiGroups: [""]
resources: ["services","endpoints","pods"]
verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get","watch","list"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: external-dns-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: external-dns
subjects:
- kind: ServiceAccount
name: external-dns
namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: external-dns
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
args:
- --source=service
- --source=ingress
- --domain-filter=hynes.pri
- --provider=azure-private-dns
- --azure-resource-group=<Resource Group>
- --azure-subscription-id=<Subscription ID>
env:
- name: AZURE_TENANT_ID
value: <Tenant ID>
- name: AZURE_CLIENT_ID
value: <adrian-identity client id>
- name: AZURE_SUBSCRIPTION_ID
value: <Subscription ID>
image: k8s.gcr.io/external-dns/external-dns:v0.7.0

Deploy a sample Application

Apply the below example to the default namespace

kubectl apply -f example.yaml#example.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: aks-helloworld-two
spec:
replicas: 1
selector:
matchLabels:
app: aks-helloworld-two
template:
metadata:
labels:
app: aks-helloworld-two
spec:
containers:
- name: aks-helloworld-two
image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
ports:
- containerPort: 80
env:
- name: TITLE
value: "AKS Ingress Demo"
---
apiVersion: v1
kind: Service
metadata:
name: aks-helloworld-two
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: aks-helloworld-two
---apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: aido.hynes.pri
http:
paths:
- backend:
serviceName: aks-helloworld-two
servicePort: 80
path: /(.*)

You should be able to verify everything is working ok by getting the ingress and making sure the Load Balancer IP Address we statically set earlier corresponds to the Address below. Next check the external DNS logs to make sure the A and TXT records get added to the Private DNS.

kubectl get ing
NAME HOSTS ADDRESS PORTS AGE
hello-world-ingress aido.hynes.priv 10.240.0.254 80 5h54m

Test

To test, connect to the Windows VM over RDP

Once you get into your Windows VM, open a browser and navigate to http://aido.hynes.pri/ and you should get the following

Conclusion

I hope you found this useful, and it helps you on your Kubernetes Journey. We’ll use this setup for future articles.

References

Cloud Platform Architect. Opinions and articles on medium are my own.