1.2.1. Smart Orchestrator

1.2.1.1. Introduction

The Smart Orchestrator simplifies how user interfaces and other enablers interact with the primary components of the kubernetes clusters. This enabler manages the complete lifecycle of Containerized Functions, whether they are network-related or not, from their creation to their termination, enabling deployment on any available k8s cluster.

This enabler has reached a TRL of 6 during the execution of the ASSIST-IoT project.

1.2.1.2. Features

The Smart Orchestrator has the goal of deploying, monitoring, and orchestrating resources that have been instantiated in each of the Kubernetes clusters that have been added to it. To achieve these objectives, the enabler relies on four different technologies: API REST, Prometheus, MongoDB, and mck8s. The Smart Orchestrator includes the following main features:

  • Decision intelligence: The Smart Orchestrator offers Kubernetes decision intelligence by accessing the metrics servers in the other joined clusters to determine the optimal placement of enablers based on the resources available in each cluster.

  • Lifecycle control: The Smart Orchestrator provides lifecycle control, enabling the management of enablers from their deployment to their deletion.

1.2.1.3. Place in architecture

The Smart Orchestrator is part of the Smart Network and Control plane in the ASSIST-IoT architecture. It provides an intelligent and dynamic network infrastructure where nodes work in parallel and communicate seamlessly. The Smart Orchestrator monitors enablers and schedules them efficiently based on CPU and memory resources.

Smart Orchestrator overall architecture

The enabler is composed of these elements:

  • API REST: The entry point for user interaction and responsible for communication with other components to obtain, add, or delete resources such as enablers, clusters, or repositories.

  • Orchestrator: Controls the entire lifecycle of Containerized Network Functions (CNFs), from their instantiation to their termination, allowing deployment in any available k8s cluster.

  • Metrics server: Collects performance metrics from targets (Kubernetes clusters).

  • Scheduler: Provides logic to place enablers based on resources available in the joined Kubernetes clusters. Also, it is responsible for using AI to predict the resources utilized at another time and make a scheduling decision.

  • Multiservice controller: Allows the connectivity from edge services to cloud services based in name service.

Smart Orchestrator enabler architecture

1.2.1.4. Pre-requisites

  • MINIMUM: 2 CPUs, 6 GB RAM, 40GB disk and a single interface with Internet access.

  • RECOMMENDED: 2 CPUs, 8 GB RAM, 40GB disk and a single interface with Internet access.

  • Base image: Ubuntu 20.04 (64-bit variant required).

1.2.1.5. Installation K8s cluster & Smart Orchestrator

1.2.1.5.1. KUBEADM

Install a K8s cluster located in the edge tier of the architecture using Kubeadm.

1. git clone https://gitlab.assist-iot.eu/enablers-registry/public.git
2. cd public/
3. cd smartorchestrator/
4. cd scripts/
5. chmod +x kubernetes.sh

Warning

  • ENSURE THAT ALL NODES ARE ADDED TO THE MAIN CLUSTER (MASTER NODE) PRIOR TO ADDING THE CLUSTER TO THE SMARTORCHESTRATOR.

1.2.1.5.2. Master node & Smart Orchestrator

Install a K8s cluster with a master node.

There are two important flags:

  • t: SERVER or AGENT (in this case SERVER).

  • p: Pod CIDR Network (This MUST be different in each cluster. If you choose 10.216.0.0/16, the other cluster MUST be for instance 10.215.0.0/16).

  • c/f: Install cilium (-c) or flannel (-f).

Warning

  • DO NOT REPEAT POD CIDR NETWORK. - 10.217.0.0/16 IS RESERVED FOR THE SMART ORCHESTRATOR CLUSTER.

sudo ./kubernetes.sh -t SERVER -p 10.216.0.0/16 -c

1.2.1.5.3. Worker node

Install a K8s worker node to add an existing master node.

./kubernetes.sh -t AGENT

Once the worker node is ready, switch to the main cluster (master node) and copy the output of this command:

kubeadm token create --print-join-command

Switch again to the agent node of the cluster and paste the command output as sudo.

Note

  • A KUBEADM node can not be joined to a k3s cluster.

  • A k3s node can not be joined to a KUBEADM cluster.

1.2.1.5.4. K3S

Install a K3s (a lightweight K8s distribution) cluster located in the edge tier of the architecture

1. git clone https://gitlab.assist-iot.eu/enablers-registry/public.git
2. cd public/
3. cd smartorchestrator/
4. cd scripts/
5. chmod +x k3s.sh

Warning

  • ENSURE THAT ALL NODES ARE ADDED TO THE MAIN CLUSTER (MASTER NODE) PRIOR TO ADDING THE CLUSTER TO THE SMARTORCHESTRATOR.

  • WE ARE FACING SOME ISSUES WITH CILIUM AND RPI, PLEASE WAIT UNTIL WE HAVE SOLVED IT.

1.2.1.5.5. Master node

Install a K8s cluster with a master node.

There are three important flags:

  • t: SERVER or AGENT (in this case SERVER).

  • i: Server IP. If the edge is behind a NAT and the Smart Orchestrator or the worker nodes are outside, the value is your Public IP.

  • p: Pod CIDR Network (This MUST be different in each cluster. If you choose 10.216.0.0/16, the other cluster MUST be for instance 10.215.0.0/16).

  • c/f: Install cilium (-c) or flannel (-f).

Warning

  • DO NOT REPEAT POD CIDR NETWORK.

  • 10.217.0.0/16 IS RESERVED FOR THE SMART ORCHESTRATOR CLUSTER.

sudo ./k3s.sh -t SERVER -i serverIP -p 10.213.0.0/16 -c

1.2.1.5.6. Worker node

Install a K8s worker node to add an existing master node.

There are three important flags: - s: Server IP (Master Node IP). - k: The server token can be found on the master node machine, located at the following path: /var/lib/rancher/k3s/server/node-token

sudo ./k3s.sh -t AGENT -i serverIP -k serverToken

Note K8s clusters cannot mix nodes from different K8s distributions (kubeadm, K3s, …) , all the nodes of a cluster must belong to the same distribution. - A KUBEADM node can not be joined to a k3s cluster. - A k3s node can not be joined to a KUBEADM cluster.

1.2.1.6. User guide

The enabler has a management API with a REST interface that allows you to configure certain values. The API will respond with the requested information or the result of the command you executed.

Method

Endpoint

Description

Payload

Information

GET

/clusters

Return K8s clusters

GET

/clusters/:clusterid

Get k8s cluster by id

GET

/clusters/node/:clusterid

Get nodes by k8s cluster

GET

/clusters/cloud/find

Get cluster cloud

POST

/clusters

Add a K8s cluster

{“name”: String, “description”: String, “credentials”: Object, “cloud”: String, “cni”: String}

DELETE

/clusters/:id

Delete a k8s cluster by id

GET

/repos

Return the helm repositories

GET

/repos/charts/:repositoryId

Return the charts in a helm repository

POST

/repos/public

Add a public helm repository

{“name”: String, “description”: String, “url”: String}

POST

/repos/private

Add a private helm repository

{“name”: String, “description”: String, “url”: String, “auth”: { “username”: String, “password”: String }}

POST

/repos/update

Update helm repositories

DELETE

/repos/:id

Delete a helm repository by id

GET

/enabler

Return the instanced enablers

POST

/enabler

Instantiate an enabler

{“name”: String, “helmChart”: String, “values”: Object, “cluster”: String, “version”: String, “timeout”: String, “auto”: Boolean}

PlacementPolicy: worst-fit, best-fit, or traffic-most

POST

/enabler/upgrade/:enablerId

Upgrade an enabler by id

{“values”: Object, “version”: String, “timeout”: String}

DELETE

/enabler/:id

Delete an enabler by id

GET

/enabler/cluster/:clusterId

Get enablers in a cluster by cluster name

DELETE

/enabler/volumes/:enableId

Delete PV and PVC related with an enabler by enabler id

GET

/version

Get Enabler Version

GET

/api-export

Get Enabler OpenAPI

1.2.1.7. Troubleshooting

1.2.1.7.1. Kubectl error

1.2.1.7.1.1. KUBEADM

  1. The connection to the server localhost:8080 was refused - did you specify the right host or port?

  2. Unable to connect to the server: x509: certificate signed by unknown authority

Please use this command:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

1.2.1.7.1.2. K3S

Please use this command:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

1.2.1.7.2. Reset kubernetes

1.2.1.7.2.1. KUBEADM

For reseting a kubernetes kubeadm cluster:

sudo kubeadm reset

1.2.1.7.2.2. K3s

For reseting a kubernetes k3s server node:

/usr/local/bin/k3s-uninstall.sh

For reseting a k3s agent node:

/usr/local/bin/k3s-agent-uninstall.sh

1.2.1.8. Developer guide

The Smart Orchestrator is written in Javascript, using the Express framework, Python and Go. The information about the clusters, enablers and repositories objects is stored in MongoDB.

This code is expected to be executed within a Helm chart, in a Kubernetes-governed platform. In case that developers aims at using the code directly over a given Operating System, non-virtualized, the code has been tested in Ubuntu 20.04 machines in amd64.

This code is open source and can be freely used by the innovation and research community. In case that commits are to be made, the mantainer team (UPV) holds the rights to accept or deny them. Best practices are encouraged in the latter case.

To run it in a development environment, the installation of Node.js, Python, and Go is required. Each of these components is an API, where the paths to the cluster, repository, and enabler services are accessible from the routes specified in the user guide. The MultiCluster Service Controller is a kubernetes controller, the only service that does not works as an API.

1.2.1.9. Version control and release

Version 4.0.0. New features:

  • Auto-Clustermesh

  • MultiCluster Service Controller

  • Acceptance of any helm repository (public or private).

  • Scheduler Policy using AI

1.2.1.10. License

This software is licensed under the Apache 2.0 license.

1.2.1.11. Notice (dependencies)

ASSIST-IoT - Architecture for Scalable, Self-*, human-centric, Intelligent, Se-cure, and Tactile next generation IoT

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957258.

Smart Orchestrator enabler

Copyright 2020-2023 Universitat Politècnica de València

  1. Included Software

  2. Used Software

  1. List of licenses

1.2.1.12. Components