Creating a Custom Competitive Kubernetes Training Experience on AKS — Part 1

5 min readDec 25, 2020

Introduction

In this series of articles, I want to show you how you can create your own custom competitive Kubernetes Training experience for your colleagues, on AKS. You could also use it as part of an interview test for potential employees.

Demo

First let’s look at the “finished” (more like a POC) product and then we’ll work back to how we got there.

Goals

Ok let’s look at the initial goals we set for this application.

Self Service and On Demand- The ability to take part should be Self Service. Users should be able to leave and come back and continue where they left off.

Competition - To keep users at different levels interested and active, the solution should have a competitive Streak. The introduction of a Leaderboard will help accomplish this

Integration with Azure Identity - While we can enable users on a certain Azure tenant to take part, we should also have the ability to invite users using their social accounts to also take part.

Eat our own Dog Food - This application itself should be deployed onto an AKS cluster, so we’re eating our own dog food along the way

Security End to End - User Credentials and tokens should not be exposed or saved on the server. Traffic encrypted end to end. Application credentials should be securely delivered to the application etc

Diverse and Extensible Question Set - There should be a diverse set of questions and it should be easy to add more and better still for users to contribute. Hands on questions are a must have.

Design Decisions

In this section, we want to answer a few questions which will drive our design.

What languages and frameworks will we use for the Front end and Back end?

We’ve picked Spring Boot for the back end due to the authors familiarity with it and ReactJS for the front end as it’s ease of setup and documentation available. Note: The author is less familiar with front end technologies.

How will users authenticate to this application?

Users will authenticate themselves via the Microsoft Identity Platform using the built in OAuth2 OIDC flow. This will generate a JWT token containing the identity of the user along with a number of claims about the user.
We can validate the authenticity of this token on the server side to verify the user is who they say they are and they indeed have the correct access.
This token can be stored in our UI application on the Client side and passed across with each request that requires it

Reference for Best Practices for managing tokens client side: https://dev.to/bahdcoder/the-ultimate-guide-to-jwt-client-side-auth-stop-using-local-storage-3an9

How will users be authorized to access this application?

Users will be Authorized to access this Application by been a part of a specific AD Group
In order to allow users in the Azure Tenant or Guest Users to get access to the application, an Administrator will be required to add them to an AD Group
We’ll use Spring Security and Azure AD Integration to make sure users are authentications and authorized to access certain apis.

#application.properties
azure.activedirectory.session-stateless=false
azure.activedirectory.client-id=eac99337-4658-4de4-92bc-b0401f417f7e
azure.activedirectory.client-secret=ABC123
azure.activedirectory.user-group.allowed-groups=CompGroup//QuestionController.java
...
@PreAuthorize("hasRole('ROLE_CompGroup')")
@PostMapping(value = "/api/levels/{id}/questions", produces = MediaType.APPLICATION_JSON_VALUE)
public String levels(@PathVariable("id") long id, Authentication authentication) {

How will users be able to access the cluster to solve questions?

We’ll create an AD Group (CompGroup above) that will be assigned the “Azure Kubernetes Service Cluster User Role” role and scoped to the cluster in question. This will allow users who are a part of this AD Group, the ability to generate a kubeconfig file to access the cluster. Guest users can be added to the Azure Tenant and then added to the AD Group, to enable an audience outside your tenant.

We’ll create a Kubernetes namespace on demand unique to each user for each question. Here is where we’ll create the Role and RoleBinding with the users UPN or Unique ID, so that they get access to the namespace and only the namespace.

What types of questions will we have?

Multiple Choice — This will require no interaction with the cluster. Users will be given a question with several possible answers and they must select the correct answer.
Hands on Multiple Choice — Similar to the multiple choice, users will be given a question and several possible answers. To find the correct answer, they must connect to their namespace in the cluster and perform certain kubectl commands to find the solution
Hands on — In this question, the user will be asked to complete a task i.e. Create a docker secret. They will be required to connect to their namespace, create the docker secret, return to the application and validate. The application will then validate that the task was completed as required.

Questions will be organised into levels.

How do we make it competitive?

Each question answered correctly will be rewarded with 1 point. We’ll keep track of user’s points via a leader board section, where each user can pit themselves against others.
The ability to add levels and questions should be a relatively easy task

What will the UI look like?

Some high level screens of what the UI will look like.

How do we design our data store tier?

We’ll pick a simple Relational Database Design for our Application.

Questions will be grouped by levels.
Multiple choice questions can have many answers
A user will be added once they authenticate to the application for the first time
Points will be tracked via the relationship between the user and a question