
We have made the decision to shut down Symbiosis as of 28th of August. We’ve had a sharp increase in the amount of fraud over the last couple of month which has put a large dent in our profitability. This has made it increasingly difficult for Symbiosis to stay competitive and we've not had the resources necessary to develop and improve the service.

Additionally, we've lately had multiple networking and storage related issues that have been hard to properly diagnose and mitigate with our small team. Ensuring that our service is stable and that we are able to handle incidents has therefore been challenging. As a result we've decided we can no longer safely keep the service running in its current state.

 We're very grateful to all our users that have supported us and helped shape the service throughout our beta period. We still believe there is tremendous value in making Kubernetes more accessible and easier to use, albeit probably in a different shape or form.

The lights will be kept on until 30th of September, after that time we will shut down any remaining clusters. If you are unable to migrate before that time then please reach out and we might be able to find an alternative solution.


Buster Styren

Sunsetting Symbiosis


<center className="flex justify-center">
  <div className="rounded-2xl overflow-hidden flex">
    <Image src="/images/articles/oauth2-proxy-forbidden.png" width="2875" height="1427" alt="Forbidden page returned by oauth2-proxy to unauthorized users" />
  </div>
</center>

**In this tutorial I will show how you can use Oauth2 Proxy to limit access to your Kubernetes services only to members of your GitHub organization, team or just to yourself.**

If you're like me then you might have a long list of self-hosted third-party services running inside of your Kubernetes clusters, such as Grafana, Sentry.io, Elastic and Jaeger. Some of these services may have their own mechanism for authentication, some can be configured with an external OAuth provider, and some won't have any authentication mechanism at all.

How can we easily expose our services over the internet with a proper audit trail and without letting any unauthorized users in?

One surprisingly simple solution is to create a GitHub OAuth app and configure your ingresses to use a proxy that permits access based on which GitHub teams a user is part of.

That's a lot of words, but I'll show you that setting it up is just minutes of work and the result is protected internal services with just two ingress annotations:

```yaml
nginx.ingress.kubernetes.io/auth-signin: https://oauth2.symbiosis.host/oauth2/start?rd=https://$host$uri
nginx.ingress.kubernetes.io/auth-url: https://oauth2.symbiosis.host/oauth2/auth
```

**Requirements**
- A Kubernetes cluster
- Helm
- Kubectl
- [NGINX ingress controller](https://kubernetes.github.io/ingress-nginx/)
- [Cert-manager](https://cert-manager.io/docs/) (to generate TLS certs for OAuth2 Proxy, optional)

# What is OAuth2 Proxy?

[OAuth2 Proxy](https://oauth2-proxy.github.io/oauth2-proxy/docs/) is a reverse proxy that authenticates users for a whole range of different Oauth identity providers, such as Keycloak, Google, GitHub, OpenID Connect, [and more](https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/oauth_provider).

Different providers have different configurations. What makes this powerful in the case of GitHub is the ability to allow or reject requests based on properties such as GitHub organization membership, team membership, email domain and others.

A running Oauth2 Proxy will expose endpoints for signing in, signing out, callbacks used by the OAuth provider, and more. These endpoints can in turn be used to configure NGINX Ingress to ensure that requests to a specific ingress are only performed by a user that is properly signed in and has the proper authority.

However, before we can install and configure OAuth2 Proxy we first have to configure the OAuth provider.

# Creating an OAuth app on GitHub

Sign in to GitHub and browse to [Settings → Developer settings → OAuth Apps → New OAuth App](https://github.com/settings/developers), fill in the name and page of your application.

The authorization callback URL is the URL that GitHub will redirect to when the authorization is finished. This URL will not be reachable as we haven't configured the OAuth2 Proxy ingress yet. However, it should have the following structure, with `example.com` replaced by your domain name:

https://example.com/oauth2/callback

Once the app is created generate a client secret and save the generated value. We will use this value soon to configure OAuth2 Proxy.

# Installing OAuth2 Proxy with Helm

I've used Helm to install OAuth2 Proxy and found it to be pretty convenient. Check out the [Helm chart](https://github.com/oauth2-proxy/manifests/tree/main/helm/oauth2-proxy) for the full list of parameters.

Below is an example `values.yaml` file that permits access to any user that is part of the "example-org" GitHub organization, with TLS certificates generated by cert-manager.

```yaml
config:
  clientID: example-client-id  # Replace with your GitHub OAuth app client ID
  configFile: |
    provider = "github"
    scope = "user:email read:org"
    github_org = "example-org"  # Replace with your GitHub org name, or github_team=team_name to limit access to a specific team
    email_domains = [ "*" ]  # Replace with [ "example.com" ] to limit access to users with specific email domain
    cookie_domains = [ "example.com" ]  # Replace with domain names that the proxy is allowed to redirect to after auth, prepend . for wildcards, i.e. ".example.com"
    whitelist_domains = [ "example.com" ]  # Same as above

ingress:
  enabled: true
  path: /oauth2
  pathType: Prefix
  className: nginx
  annotations:
    acme.cert-manager.io/http01-edit-in-place: "true"
    cert-manager.io/cluster-issuer: letsencrypt  # Replace with name of your cert-manager ClusterIssuer
    kubernetes.io/tls-acme: "true"
  hosts:
  - example.com  # Replace with host address for the oauth2-proxy ingress
  tls:
  - hosts: 
    - example.com  # Same as above
    secretName: oauth2-tls
```

Make sure to replace `example-client-id`, `example-org` and `example.com` with your GitHub OAuth app client ID, GitHub organization and domain name.

You may have to edit the cert-manager annotations based on your own configuration, for example by using the `cert-manager.io/issuer` annotation for namespaced certificate issuers.

We also need to configure a cookie secret that is used by OAuth2 Proxy to encrypt and decrypt user session cookies. Any base64 encoded string is valid, a secure base64 encoded string can be generated using `openssl` with the following command:

```bash
openssl rand -base64 32 | head -c 32 | base64
```

Finally, we can add the OAuth2 Proxy helm repository and install the chart with our values file, cookie and client secrets:

```bash
helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm install oauth2-proxy oauth2-proxy/oauth2-proxy --file values.yaml --set config.cookieSecret=example-cookie-secret --set config.clientSecret=example-client-secret
```

## oauth2-proxy without cert-manager

If you don't have cert-manager installed you can either [create the TLS certificate manually](https://kubernetes.github.io/ingress-nginx/user-guide/tls/) or disable TLS altogether:

```yaml
config:
  clientID: example-client-id  # Replace with your GitHub OAuth app client ID
  configFile: |
    github_org = "example-org"  # Replace with your GitHub org name, or github_team=team_name to limit access to a specific team
    scope = "user:email read:org"
    email_domains = [ "*" ]  # Replace with [ "example.com" ] to limit access to users with specific email domain
    provider = "github"
    cookie_secure = false
    cookie_domains = [ "example.com" ]  # Replace with domain names that the proxy is allowed to redirect to after auth
    whitelist_domains = [ "example.com" ]  # Same as above

ingress:
  enabled: true
  path: /oauth2
  pathType: Prefix
  className: nginx
  hosts:
  - example.com  # Replace with host address for the oauth2-proxy ingress
```

Note that you need to change the schema of the callback and auth URLs to `http://`, both in the GitHub OAuth configuration and in the annotations below.

# Configuring our ingress to use OAuth2 Proxy

Now that OAuth2 Proxy is up and running, we can configure our ingress to protect our internal services with GitHub OAuth.

Add the following annotations to your ingress:

```yaml
nginx.ingress.kubernetes.io/auth-signin: https://example.com/oauth2/start?rd=https://$host$uri
nginx.ingress.kubernetes.io/auth-url: https://example.com/oauth2/auth
```

Make sure to replace `example.com` with your own domain name.

The first annotation `auth-signin` will redirect unauthenticated requests to the OAuth2 Proxy login page. After the user logs in and authorizes the application, they will be redirected back to the original requested URL thanks to the `rd` query parameter that fowards the redirect URL to the proxy.

The second annotation `auth-url` specifies the authentication URL that is used to verify the user's session.

Use the commands below if you don't have a service to expose yet in order to launch a protected hello-world application. Make sure to replace `example.com` with your domain name.

```bash
kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
kubectl expose deployment web --port=8080
kubectl create ingress web --class nginx '--rule=example.com/=web:8080' \
  --annotation 'nginx.ingress.kubernetes.io/auth-signin=https://example.com/oauth2/start?rd=https://$host$uri' \
  --annotation 'nginx.ingress.kubernetes.io/auth-url=https://example.com/oauth2/auth'
```

Accessing an endpoint protected by OAuth2 Proxy will redirect you to a GitHub OAuth sign-in page.

Make sure to grant access to the organization that you are authenticating against for OAuth2 Proxy to be able to find it and verify your authority.

<center className="flex justify-center">
  <div className="rounded-2xl overflow-hidden flex">
    <Image src="/images/articles/oauth2-proxy-authenticate.png" width="2880" height="1600" alt="GitHub OAuth sign-in page" />
  </div>
</center>

If everything is set up correctly you should be redirected to your protected endpoint after signing in.

# Limitations

Unfortunately, OAuth2 Proxy cannot be configured to allow specific GitHub teams or organizations to access a resource on the ingress level. If you wish to limit access to ingress1 only to team1 while also limiting access to ingress2 only to team2 you will have to configure multiple OAuth2 Proxy instances and GitHub OAuth applications.

Your services are still exposed on the internet. A security vulnerability in GitHub OAuth, the OAuth2 Proxy or the configuration might expose your service to the whole wide world. Protecting internal services behind a private network is always a good idea.

# Putting it all together

To conclude, we've set up a GitHub OAuth app that we've configured with OAuth2 Proxy to allow access to layer 7 ingresses only to the members of our GitHub organization or team. We can use the aforementioned annotations to tag any ingress that we wish to only protect to authenticated users.

Oauth2-proxy isn't the silver bullet for securing internal services, rather it is an easy way to secure many different services or UIs without much work. It can also act as a first line of defense even for services that are only routable on a private network, or services that offer their own authentication mechanism, like Grafana.

You can check out OAuth2 Proxy's [official documentation](https://oauth2-proxy.github.io/oauth2-proxy/docs/) for more tips, tricks and config parameters.


Lock down your Kubernetes services with OAuth2 Proxy


We're celebrating the fact that spring has **finally** come to Sweden by offering new users $50 in free credits for their Symbiosis resources in Germany, valid for 1 month.

**Changelog**

- We've changed the theme to a dark mode, including plenty of UI/UX tweaks & improvements
- Support for Kubernetes 1.26 added (and we're phasing out 1.23)
- CSI driver support for increasing capacity of volumes (online/offline)
- We are running a closed beta of object storage, see pinned post in Discord to opt-in

<center className="flex justify-center mx-auto mt-6">
  <div className="border flex shadow-lg rounded-lg overflow-hidden">
    <Image
      alt="New UI"
      width="1958"
      height="1183"
      src="/images/articles/2023-march-updated-uiux.png"
    />
  </div>
</center>
<div className="text-center text-sm mt-2 mb-6">A new & darker UI 🌚</div>

We've disabled the light theme for now as the dark mode just feels that much crisper.

## What's next

Our primary focus right now is to build tools to simplify setting up infrastructure with CI/CD, IaC+GitOps and other best practices in mind. We're planning to release a closed beta of our auto-provisioning service during April.


March product changelog


**What's new?** We've added the ability to copy [Pulumi](https://pulumi.com) manifests in Go, Python or Typescript directly from the UI. We have also made it possible to import your clusters and node pools directly into Pulumi using our "Pulumi Export" button.

Infrastructure-as-code (IaC) is the backbone of any serious cloud deployment. It enables developers to collaborate, review and approve eachothers changes, ensures there's a log of all changes taking place and simplifies the management of resources across multiple cloud vendors.

That's why we're pushing out an even better integration with Pulumi. So you can easily set up Pulumi for provisioning and managing Kubernees infrastructure with us. 🥳

# What's the deal with Pulumi?

Pulumi is an IaC platform that lets you define cloud resources using Python, Go, TypeScript, Java, .NET – even YAML – so you don’t have to adopt a domain-specific language like HCL.

The Pulumi Kubernetes Operator is another feature that separates Pulumi from other IaC tools. With the operator it's possible to manage your Pulumi Stack using CRDs inside of Kubernetes.

The operator is configured by pointing it to a git repository, which the operator uses to ensure that the state is kept consistent with version control. With the operator it's extremely easy to set-up a GitOps workflow in Kubernetes.

All in all, Pulumi is a great tool that makes it dead simple to adhere to the best practices for cloud management, which is why we're thrilled to build out support for it!

# Copy Pulumi manifests directly from the UI

Configure your Kubernetes cluster in our web UI and use the Copy-button to create Go, Typescript or Python stanzas that can be imported directly into your Pulumi project.

<center className="flex justify-center">
  <div className="border shadow-lg rounded overflow-hidden flex">
    <Image
      alt="Copy Pulumi manifests from the console"
      src="/images/articles/pulumi-copy.png"
      width="800"
      height="418"
    />
  </div>
</center>

The copy button make it possible to configure your clusters or node pools with the UI while being able to easily shift it over to IaC once you're done. You might call it... *configuration-without-reading-the-manual*?

# Export/Import directly into Pulumi

Use the export button to download a JSON file of your cluster and node pools. Pulumi can read this JSON and import these reasorces directly into Pulumi. This might be useful if you already have clusters set up but want to quickly import them into a new (or existing) Pulumi project.

<center className="flex justify-center">
  <div className="border shadow-lg rounded overflow-hidden flex">
    <Image
      alt="Copy Pulumi manifests from the console"
      src="/images/articles/pulumi-export.png"
      width="800"
      height="271"
    />
  </div>
</center>

## How to import Symbiosis clusters into Pulumi

First, Create a fresh Pulumi project if you don't already have one. You will be prompted to select a template to start building with, for example `python` to create an empty python project without any predefined resources.

You will also be prompted to name your project and to create a stack. A stack is a specific environment based on our project. Stacks can be used to create separate environments such as dev, staging or production.

```bash
# Select from a template
pulumi new

# Or create a python, go or typescript project directly
pulumi new python
```

With our project set up we can install the Symbiosis Pulumi plugin:

```bash
pulumi plugin install resource symbiosis --server github://api.github.com/symbiosis-cloud
```

Next, we have to define a value for the API key that Pulumi uses to communicate with the Symbiosis API. An API key can be created under **api keys** in the UI or through the CLI.

```bash
pulumi config set symbiosis:apiKey --secret <api-key>
```

Download the exported Pulumi resources from the `settings` tab in web UI. Importing the JSON file into our project:

```bash
pulumi import -f <file>
```

Done! You will be presented with a code snippet that you can copy and paste into your Pulumi code (the correct file will depend on what language you are using) with the correct parameters for your cluster.

# Next steps

You can use Pulumi to manage all aspects of deploying your k8s clusters on Symbiosis – check out the full API docs and further examples of how to use the Symbiosis provider on the [Pulumi registry](https://www.pulumi.com/registry/packages/symbiosis/).

Have any input, suggestions or feedback? Please reach out to us on [Discord](https://discord.gg/bd4DKsgtkS) or [Twitter](https://twitter.com/symbiosiscloud).


Taking our Pulumi integration to the next level ⬆️


If you manage the infrastructure for a small or medium-sized company, you understand that having a scalable, secure, and flexible infrastructure is crucial for the success of your business. 

Kubernetes is a popular open-source container orchestration platform, that is already **used by [61%](https://www.statista.com/statistics/1233945/kubernetes-adoption-level-organization/) of software organizations** globally. 

However, managing Kubernetes can be a complex task, requiring specific expertise and a learning curve, especially at a time when your development team should be focused on making the best application possible. 

At such a critical junction of your business, a complex infrastructure set up can be extremely draining. The most common complaint we've heard is that there are just **too many things that can go wrong**. Whether it is network routing, security controls or just trying to get your team up to speed on the YAML manifests. 

That's where managed Kubernetes services come in, offering a solution that can benefit companies of all sizes.

## Simplified management

A managed Kubernetes service provides you with a simplified approach to Kubernetes management & day-2 operations. It can help you **avoid the complexities of maintaining and updating the infrastructure**, allow configuration of access controls and also provide integration with other managed services like databases. 

Most managed services provide all key benefits of the Kubernetes API like autoscaling, auto-healing and replication. This way, you can spend more time building your application, and less time managing your infrastructure.

## Security

Kubernetes can be challenging to secure, requiring extensive knowledge of container security and network security. Managed Kubernetes services offer security features such as:

- network isolation
- identity and access management
- vulnerability scanning
- multi-region support. 

These features ensure that your infrastructure and data are protected, and that you comply with industry standards and regulations.

## Cost savings

While hyperscalers like Amazon and Google offer Kubernetes services, they can become expensive as soon as the company is out of their free credit program. 

Managed Kubernetes services offer cost-effective options that are tailored to small and medium-sized companies' needs, (for example, Symbiosis can save you up to 82% compared to AWS). 

Managed services providers offer solutions that fit within the company's budget while ensuring a high level of security and scalability.

## Summary

A managed Kubernetes service offers small and medium-sized companies the **scalability, security, and flexibility** they need to grow their businesses.

It can help you save costs, simplify management, and scale efficiently without compromising on security. 

If you're working for a fast-growing company, consider the benefits of using a managed Kubernetes service for your infrastructure.


Sauraj Gambhir

Why an SME should use Managed Kubernetes

This week we experienced a major outage that affected the service of all our users.

Firstly, we want to apologize profusely for the impact this outage may have caused. We’re a small provider with a small but growing team at an early stage. We pride ourselves in offering a competitive service and being reliable is therefore critical to establishing trust with you as an end user.

This post-mortem details how we were made aware of the outage, the steps we took to solve it and the mitigations and learnings that came out of it. We have attached a detailed timeline of all events and the steps we took to resolve the issue. All timestamps are in CET.

## Issue summary

From 12:19 the 10th of March until 05:13 the 11th of March there was an outage of our entire storage cluster, which prohibited any read or writes to nodes and PVs. This was caused by fragmentation of multiple drives that prevented storage nodes from allocating additional 64K memory blocks for metadata, despite affected drive having more than 115GiB of capacity.

The fragmentation and the low amount of drives in the cluster caused the allocation failure to bring the cluster into an inoperable state. We have doubled the amount of drives and we are adding monitoring to track the level of fragmentation.

## Detailed timeline

**On 9th March** we got alerted that our storage cluster was over 85% capacity. In response, we planned to increase the capacity the next day.

At **12:19 on 10th March**, we added a first additional volume to the storage cluster. At this point the storage cluster was at 86% capacity.

Shortly after the first volume was added, two-thirds of our storage nodes went down.

At **12:36** the storage cluster was completely unresponsive to any requests.

We had two initial hypothesis at this point:

- A networking failure was prohibiting hosts from communicating
- The new node caused a rebalancing that moved objects onto already full storage nodes

A networking issue was quickly ruled out as the BGP peering was active and root servers between the affected nodes were reachable over the network used by the storage cluster.

The logs for the storage managers turned up dry, only reporting on the degraded storage pools.

Next we looked at the status for the affected storage nodes. They were all in a restart crash loop. The stack trace indicated an allocation failure for BlueStore, which manages the write-ahead log and the metadata store.

We had now identified the problem to be the inability to allocate more memory for metadata. We had to find a way to either free up space on the storage node, move objects in order to defragment it or find a way to start the node while allocating less memory for metadata.

A storage node is a normal block device, therefore the metadata database uses a free-list to figure out where it can allocate space on the device. If it requests too big of an allocation it may fail if there are no empty spaces to put it. Our goal was to see if:

- we could reduce the amount it allocated
- start it without any allocation being made at all or,
- whether we could defragment the list to introduce additional blocks for allocation.

Our first attempt was to change the type of memory allocator.

**At 13:12** our first attempt was to change the allocation algorithm. We hoped that we could start the nodes and quickly reshuffle storage over to the new node while removing superfluous or redundant objects.

However changing the allocator didn’t have any impact, the nodes were completely jammed and neither defragmentation nor removing objects were possible due to the lack of allocatable memory.

Next, we looked into using Bluestores tooling for repairing and defragmenting the metadata database. We had to be careful to not do any destructive operations that might jeopardise or corrupt user data.

**At 14:22** we called our hosting provider to have additional disks installed on the affected hosts with the goal of copying the complete node over to the new disk and expand it. Our hosting provider required 2 hours to get the new disks installed.

The nodes have a ceiling on the amount of memory that can be written to it before it fails, this is by default set to 95% and is the value that we use as well. **At 15:48** we tried to increase the value to 97% in order to try to create a wedge that would get it to start. It didn’t work as the limitation wasn’t imposed by software but rather by the allocatable memory on the device.

**At 16:02** we had access to new disks on the affected hosts. We cloned the failing nodes onto the new disks, extend the drive and finally expand the metadata database.

Since the storage nodes are placed on logical volumes we had a problem: cloning the devices would mean two identical logical partitions which would confuse the OS. We would have to tread carefully in order for that confusion to not corrupt the disks or even impact the ability for the servers to boot normally.

**At 18:22** we had defined a migration plan set and we booted the servers into rescue mode in order to configure the LVM partitions.

**At 18:45** we restarted the affected servers with the new storage nodes configured. At this point the cluster started properly.

**At 19:10** the storage cluster initialised a global recovery event in order to reshuffle and backfill the storage nodes with the missing data. However, **at 19:15** the third and previously unaffected host had a node put in the same crash loop due to fragmentation.

Now, this should normally be okay if there’s just a single node that is down. But the recovery only succeeded for one of the nodes, which meant that any objects that were only replicated on the node that failed to recover, and the now crashed node, were unavailable. The cluster could not resume operations until these objects were made available. The cluster was once again put out of service.

The recovery event stalled **at circa 20:23**.

At this point it was not entirely clear why the recovery had stopped and whether or not there was any objects missing from the active storage nodes. We had to figure out both why it stopped, if there was any missing data and if there was a way for us to jump start the recovery again.

**At 21:12** we wanted to verify that the recovery wasn’t stalled due to any IO bottleneck. Therefore we stopped any running VMs and other processes that used the storage cluster, however this did not impact the recovery of the degraded, missing or misplaced objects.

We were able to conclusively identify that 28% of all objects in the clusters were missing **at 21:59**. This meant that these objects were not stuck on the failing node.

**At 23:05** we were sure that we now had objects stuck in the failing node without any redundancy at all. We decided to look into how we could backup this volume so that any mistakes during  recovery would not lead to data loss.

However, it would take too long to transfer the data securely to another host. So **at 23:18** we called our hosting provider to ask them to install additional disks in order to perform the same recovery operation as we did for the previously broken nodes. We were told we had to wait up to two more hours in order to get them installed.

**At 00:48** we had received the additional disk and proceeded with cloning the node volume onto it and extending it in order to be able to access it without running into any allocation issues.

**At 01:36** we had identified all objects that were currently unavailable and at risk of being lost. We backed up all missing objects on the root filesystem.

**At 02:25** we attempted to move the extracted objects and import them on one of the healthy nodes. However after moving all objects the import tool incorrectly reported the objects as already present.

At this point we were confident that even though we couldn’t import the objects that they were still properly exported, therefore we could perform the same recovery procedure as the previous time around, as even if we encountered a failure we could still hopefully restore the missing objects later on.

**At 03:19** we booted the host in rescue mode. Just like the previous time we cloned the disk and configured the LVM metadata in order for the storage cluster to properly identify it.

**At 04:18** the cluster was finally reporting itself as healthy.

**At 05:13**, after seeing that nodes started to boot successfully and that there was a low risk of data loss, we sent an email to affected users to notify that the core issue had been resolved.

## Root cause

The adding of a new drive created new backfill jobs, this led to the existing storage nodes allocating additional memory to handle the metadata necessary for the backfill to complete.

Despite the drives having more than 100GiB of storage left they were so heavily fragmented that the allocator wasn’t able to allocate the 64K block it needed, leading to a crash loop.

`debug 2023-03-10T12:15:42.412+0000 7ff3e72b0080  1 bluefs _allocate unable to allocate 0x80000 on bdev 1, allocator name block, allocator type bitmap, capacity 0xdf90000000, block size 0x1000, free 0x1c9e9c0000, fragmentation 1, allocated 0x30000`

Due to the small size of the storage cluster the fragmentation affected multiple nodes at the same time which led to an outage of the full storage cluster. Since nodes and PVs both rely on the storage cluster it caused all IO operations on Kubernetes nodes and control planes to fail.

## Related incidents

When all nodes and control planes were brought up simultaneously it caused some clusters to not boot properly. The reason is that the control planes connect to CoreDNS to resolve service names. However, when CoreDNS was offline its IP was still routable, this resulted in DNS lookups taking very long time before timing out. As a result the cluster couldn’t bootstrap properly.

Also, some nodes that were brought up failed to attach their persistent volumes after the service was brought back up and had to be manually recycled.

## Remedies

In hindsight this problem would have been avoided if there were more nodes in the storage clusters. We have doubled the amount of disks since the incident.

We have added monitoring to automatically track the fragmentation of our storage nodes and alert if the fragmentation crosses a threshold.

We have changed our alerting to trigger once the capacity reaches 80%, as opposed to the previous threshold of 75%.

We are also lowering the timeout of DNS lookups against CoreDNS to ensure that the control planes can quickly default to a working nameserver in case of a cold boot.

## Learnings

This incident could have been avoided if the storage cluster was larger, in which case two nodes failing wouldn’t have caused this global outage. This is a failure on our end. We should’ve made sure the cluster was big enough to have the durability necessary for business critical workloads, even if the overall usage would’ve been low.

We’ve set ourselves out to offer a simpler, faster and cheaper k8s service. And in order to meet that goal we under-provisioned volumes in order to keep costs and prices for users down. We should have evaluated carefully whether the small number of physical volumes could have an impact on the durability of the storage cluster.

Also, we never access root servers directly under normal circumstances. Instead we rely on IaC and CaC to set-up servers and deploy a server daemon to provision and manage VMs, routing and other configuration. In this case we had to “break the glass” and access the servers directly in order to solve the issue. As a result we had to be extremely careful in order to avoid any potential data loss as we were dealing with a very sensitive part of our stack, therefore the incident took a long time to properly resolve.


March 10 incident post mortem


We have added multifactor-authentication (MFA) to Symbiosis. You can configure one-time passwords with authenticator apps or biometric authentication using Yubikeys.

# What is multi-factor authentication?

Multi-factor authentication (MFA) is a security mechanism that adds an additional layer of protection to your accounts. MFA requires two or more forms of identification before gaining access to your accounts, preventing an outside party from accessing your Symbiosis resources without also compromising your authenticator app or physical Yubikey.

# What are time-based one time passwords (TOTP)?

Authenticator apps are small mobile applications that can generate time-based one time passwords (TOTP) to supplement your standard login method.

Many different authenticator apps exist, some examples are Google Authenticator ([Android](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2&hl=sv&gl=US), [IOS](https://apps.apple.com/se/app/google-authenticator/id388497605)) or LastPass Authenticator ([Android](https://play.google.com/store/apps/details?id=com.lastpass.authenticator&hl=en_US), [IOS](https://apps.apple.com/us/app/lastpass-authenticator/id1079110004)).

# Enable MFA using an authenticator app (TOTP)

Enable TOTP by browsing to **My Account** (open the menu in the top-left corner of we web UI and click **My Account**). Search for **Authenticator** and click **Manage**.

<center className="flex justify-center">
  <div className="border shadow-lg rounded overflow-hidden">
    <Image
      alt="Time-based one time password modal under account settings"
      src="/images/articles/mfa-launch-auth-app.png"
      width="559"
      height="497"
    />
  </div>
</center>
<div className="text-center text-sm mt-2">TOTP configuration under account settings</div>

Open up your Authenticator app and follow the instructions for adding a new account. If you're unable to scan the QR code you can alternatively configure it by entering the "authenticator app secret" that is displayed below the QR code.

Finalize the integration by entering the code displayed in the Authenticator app in the "Verify code" input in the Symbiosis UI. You will be prompted to login again to verify.

Voilà! Next time you sign up you will be prompted to add the code from the Authenticator app in order to proceed.

# What are hardware tokens?

Hardware tokens is an alternative MFA method that requires the user to physically activate the Yubikey in order to pass authentication. They are therefore a secure alternative as even if both the device with the authenticator app and the desktop is compromised the hacker will not be able to pass MFA.

A physical Yubikey or other webauthn compatible hardware token is required in order to configure biometric authentication in Symbiosis.

# Enable biometric MFA using a Yubikey

Enable Yubikey MFA by browsing to **My Account** (open the menu in the top-left corner of we web UI and click **My Account**). Click on **Manage** in order to bring up the modal for managing hardware tokens.

<center className="flex justify-center">
  <div className="border shadow-lg rounded overflow-hidden">
    <Image
      alt="Hardware token modal under account settings"
      src="/images/articles/mfa-launch-hw-token-setup.png"
      width="556"
      height="252"
    />
  </div>
</center>
<div className="text-center text-sm mt-2">Hardware token configuration under account settings</div>

Add a name of your key and click on "Add security key". You will be prompted by your browser to select the key and authenticate with it for verification.

Done! Next time you log in you will be prompted for biometric authentication using your hardware token.

# Disabling MFA using an authenticator app

Disabling the integration with an authenticator app by browsing to **My Account** and click on **Manage** under the Authenticator app section. Remove it by clicking "Unlink TOTP Authenticator App". Remember to also unlink the integration in the authenticator app.

# Disabling biometric MFA

Remove a security key by to **My Account** (open the menu in the top-left corner of we web UI and click **My Account**). Click on **Manage** and 'Remove security key "YOUR-KEY"'.

# Future improvements 

In the future, we plan to allow the use of MFA to secure access access to your Kubernetes cluster as well. We are currently investigating the best ways to support that.


Multi-factor authentication (MFA) launched


GitHub has made it possible to run GitHub Actions using your own self-hosted runners. Thanks to the [Actions Runner Controller](https://github.com/actions/actions-runner-controller) it is surprisingly easy to run actions in your Kubernetes clusters.

In this post we will show how to install Actions Runner Controller into an existing Kubernetes cluster to run customized runners at a fraction of the cost.

# Why?

At Symbiosis we run **a lot** of tests on each commit, so we've spent a considerable time to make sure they run quickly and can perform intricate integration tests.

Using GitHub's own runners is therefore not ideal, as the average commit would cost us almost a dollar each and sadly we make a lot of small changes.

So we could either pay $40 for 5000 minutes of a 2 CPU github runner. Or we could pay $2 to rent a 2 CPU 8 GB Kubernetes node for 5000 minutes and run our actions on there instead.

And as we see below, can also customize our runners to add even more flexibility over the default GitHub runners by heavily modifying the runtime.

# Prerequisites

To follow this tutorial you need:

- A Kubernetes cluster
- NGINX ingress (or other ingress controller)
- Certmanager (optional)

# Installing Actions Runner Controller

The Actions Runner Controller (ARC) is the service responsible for monitoring your selected repositories and firing up new runners.

We will show you how to install it using `kubectl` but using helm is [just as easy](https://github.com/actions/actions-runner-controller/blob/master/docs/installing-arc.md).

```bash
kubectl create -f https://github.com/actions-runner-controller/actions-runner-controller/releases/download/v0.25.2/actions-runner-controller.yaml
```

# Connecting Actions Runner Controller to GitHub

So we have the controller running, that's great. Now we need to authenticate so that commits, PRs, comments or any other event can be picked up by the controller and trigger the start of a runner.

We have two options, either we can create a Personal Access Token (PAT) with admin access to our repos or we can [create a github app](https://github.com/actions-runner-controller/actions-runner-controller#deploying-using-github-app-authentication) and install it into the repos instead.

For simplicity we will authenticate using a PAT.

## Personal Access Token (PAT)

Create a token under [Settings > Developer settings > Personal access token](https://github.com/settings/tokens/new). Make sure you have admin access to the repos your runners will run on.

Select the `repo (Full control)` permission, and if your runners will run in an organization you need to select the following permissions as well:

- admin:org (Full control)
- admin:public_key (read:public_key)
- admin:repo_hook (read:repo_hook)
- admin:org_hook (Full control)
- notifications (Full control)
- workflow (Full control)

Next, let's store the token we just created in a secret that our controller can use for authentication.

```
kubectl create secret generic controller-manager \
    --namespace=actions-runner-system \
    --from-literal=github_token=<YOUR PERSONAL ACCESS TOKEN>
```

# Creating a workflow

Before we move on it is perhaps a good time to create an actual workflow that will eventually trigger our self-hosted runner.

```
name: Run test on PRs
on:
  pull_request: {}
jobs:
  test:
    name: "Run tests"
    runs-on: [self-hosted]
    steps:
    - name: Checkout repo
      uses: actions/checkout@master
    - name: Run tests
      run: yarn test
```

This workflow triggers on commits to pull requests and runs `yarn test`. Lets put it into `.github/workflow/test-workflow.yaml` and push the changes to our repository.

Notice the `runs-on: [self-hosted]` option that will instruct GitHub to select any of your own self-hosted runners. Don't worry, you can be more specific about which type of runner to use. More on that later.

# Webhook & Ingress

Runners can trigger based on either push or pull-based mechanics. For example by polling or by configuring webhooks. Most triggers come with certain drawbacks, some spawn too many runners and some spawn too few which may put your actions on a slow moving queue.

However there is the `workflowJob` trigger that has none of these drawbacks, but requires us to create an Ingress and configure a GitHub webhook. So this step isn't strictly necessary but we can assure you it's worth the effort.

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: actions-runner-controller-github-webhook-server
  namespace: actions-runner-system
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  tls:
  - hosts:
    - your.domain.com
    secretName: your-tls-secret-name
  rules:
  - http:
      paths:
      - path: /actions-runner-controller-github-webhook-server
        pathType: Prefix
        backend:
          service:
            name: actions-runner-controller-github-webhook-server
            port:
              number: 80
```

This Ingress is configured for NGINX ingress, so make sure to edit it depending on your ingress controller. It also assumes that cert-manager is configured to automatically provision a TLS certificate.

Next step is to define the webhook in GitHub. Go to `Settings > Webhooks > Add webhook` in your target repository.

First let's set the payload URL to point to the ingress, for example using the details above: https://your.domain.com/actions-runner-controller-github-webhook-server. Set content type to `json` and enable the *Workflow Jobs*  permission.

Once it is done you can create the webhook and go to *Recent Deliveries* to verify that the ingress can be reached successfully.

# Listening on events

We have our controller running, it's authenticated and we have a workflow. The only thing left is to create the actual runners.

Now, we could just create a Runner resource and be done with it, but just like a Pod it wouldn't have any replicas or any autoscaling.

Instead we create a RunnerDeployment and a HorizontalRunnerAutoscaler. And for any Kubernetes user you will notice plenty of similarities to regular Deployments and HPAs.

```yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: actions-runners
spec:
  template:
    spec:
      repository: myorg/myrepo
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: actions-runners
spec:
  minReplicas: 0
  maxReplicas: 5
  scaleTargetRef:
    kind: RunnerDeployment
    name: actions-runners
  scaleUpTriggers:
  - githubEvent:
      workflowJob: {}
    duration: "30m"
```

Applying the above manifest will launch a deployment that will scale up to five concurrent runners. Remember to change the manifest to track the repository of choice (and make sure the access token has access to it).

Voilà! We're now able to create a pull request to verify that the runner is automatically triggered.

# Using labels to identify runners

In a repo with many workflows, for example a monorepo, it may be necessary to run many different runners at once.

In order to more carefully select which runner to use for a specific workflow we can define custom labels:

```yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: actions-runners
spec:
  template:
    spec:
      repository: myorg/myrepo
      labels:
      - my-label
```

With this label we're able to select this runner by setting both `self-hosted` and `my-label` in our workflow:

```
name: Run test on PRs
on:
  pull_request: {}
jobs:
  test:
    name: "Run tests"
    runs-on: [self-hosted, my-label]
    steps:
    - name: Checkout repo
      uses: actions/checkout@master
    - name: Run tests
      run: yarn test
```

# Customizing runners with custom volumes

Runners can be configured to pass through volumes from the host system, or to attach a PVC to runners.

At Symbiosis we use PVCs to expose KVM to our runners, in order to run integrations tests with vitualization enabled. We also use PVCs to attach large images that are used to set up a multi-tenant cloud environment for integration testing.

Custom volumes can also be used for [layer caching](https://github.com/actions/actions-runner-controller/blob/master/docs/using-custom-volumes.md#docker-image-layers-caching), to improve the speed of building OCI images.

The below runner will provision a 10Gi PVC that will be shared. Notice that we're using `RunnerSet` instead of `RunnerDeployment`. This resource functions much like the `StatefulSet` in that it will allocate the runner on a node where the volume can be properly mounted.

```yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: actions-runners
spec:
  template:
    spec:
      repository: myorg/myrepo
      volumeMounts:
      - mountPath: /runner/work
        name: pvc
      volumes:
      - name: pvc
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              resources:
                requests:
                  storage: 10Gi
```

This PVC can be used to store any data that we need between runs without having to store an unnecessarily large amount of data in the GitHub actions cache! At the time of writing this article 100GiB of storage using GitHub runners would costs $24/mo. With cloud providers like Linode, Symbiosis or Scaleway that cost would be closer to $8/mo.

# To summarize

Running your own actions runners requires some upfront configuration but come with a list of benefits such as:

- Reduced costs
- Attaching custom volumes (such as hostpath or PVCs) to your runners
- Customizing images or adding sidecars
- Integrate workflow runs into the existing Kubernetes observability stack

Therefore, we highly recommend running your own runners to save cost, simplify management and increase flexibility by bringing the runners into the Kubernetes ecosystem.


How to save a fortune with self hosted GitHub runners


**We are kicking of the new year with the launch of our CLI, as well a convenient button for exporting your terraform resources directly from the UI. 🥳**

# Symbiosis CLI

We've launched the first version of the Symbiosis CLI (`sym`). The CLI can be used to create, update and remove clusters, adjust node pools, team membership and other settings.

A few examples:

```bash
# Creating a cluster
sym cluster create my-cluster

# Print the config of a node-pool with json
sym node-pool describe my-node-pool -o json

# List all available node types
sym info node-types
```

For additional examples and instructions on how to install it see the [CLI docs](/docs/quickstart/using-the-cli).

## Symbiosis CLI GitHub Action

The CLI can be authorized and installed in GitHub workflows by using the `symbiosis-cloud/action-cli` action.

The action can be useful for creatin ephemeral clusters for testing, or perhaps to configure the kube context for deployments.

```yaml
- name: Install and authorize Symbiosis CLI
  uses: symbiosis-cloud/action-cli@v1
  with:
    api-key: ${{ secrets.SYMBIOSIS_API_KEY }}
```

# Copy IaC manifests directly from the UI

We find that it is often easier to use a UI to configure cloud resources, as opposed to reading the terraform documentation and authoring them manually.

So we added a button that lets you copy IaC definitions directly from the UI in order to get the best of both worlds:

<center>
<Image
alt="AWS logo"
width="800"
height="376"
src="/images/articles/ioc-copy.png"
/>
</center>

The button can be used to copy stanzas for clusters and node-pools. Only Terraform is supported so far, but we are planning to add Crossplane and Pulumi later on.


Stephen Hoogendijk

January 2023 update


Kubernetes nodes are slow to initialize. OS's have to be booted, networks have to be configured, kubelets need to initialize, certificates need to be issued and approved, and so on...

The unfortunate side effect is that cluster autoscaling is limited by the time it takes to add more nodes into the pool. If your environment sees a sudden spike in usage there might not be enough time to scale up to handle the additional load.

This volatility in usage will impact the amount of additional capacity that is necessary for your cluster to function during high stress. For very bursty settings you will need to configure more headroom to account for the hightened variance.

However, the faster nodes initialize the faster your cluster can react to these sudden spikes. So, not only can quick nodes reduce the risk of resource congestion, it also reduces the additional headroom you need to have on hand, leading to lower costs.

In this benchmark we compared initialization time across 8 managed Kubernetes providers.

# The setting

As [last time](/blog/comparing-cluster-creation-times), we ran our benchmark across 8 different Kubernetes providers. We compared the providers by launching a single node 15 times and measuring the time from creation until the it was ready to schedule work.

The averaged time is shown in the chart below and the complete data is available [in this spreadsheet](https://docs.google.com/spreadsheets/d/1PIzXePHNIHBIFyOkjI6-rFA_Lk_XTR3pcZ9_adn1CgI/edit?usp=sharing).

The source code is available on [GitHub](https://github.com/symbiosis-cloud/k8s-cluster-creation-benchmarks).

# The results

<BasicVerticalChart
  items={[
    {
      title: "Google Cloud",
      logo: (
        <Image
          alt="Google Cloud logo"
          width="32"
          height="32"
          src="/images/cloud-icons/gcp.svg"
        />
      ),
      value: 0.8,
    },
    {
      title: "Symbiosis",
      logo: (
        <Image
          alt="Symbiosis logo"
          width="32"
          height="32"
          src="/icon-zoomed.png"
        />
      ),
      value: 1.15,
    },
    {
      title: "AWS",
      logo: (
        <Image
          alt="AWS logo"
          width="32"
          height="32"
          src="/images/cloud-icons/aws.svg"
        />
      ),
      value: 1.34,
    },
    {
      title: "Civo",
      logo: (
        <Image
          alt="Civo logo"
          width="32"
          height="32"
          src="/images/cloud-icons/civo.png"
        />
      ),
      value: 1.7,
    },
    {
      title: "Linode",
      logo: (
        <Image
          alt="Linode logo"
          width="32"
          height="32"
          src="/images/cloud-icons/linode.png"
        />
      ),
      value: 2.27,
    },
    {
      title: "Scaleway",
      logo: (
        <Image
          alt="Scaleway logo"
          width="32"
          height="32"
          src="/images/cloud-icons/scaleway.png"
        />
      ),
      value: 2.56,
    },
    {
      title: "Digital Ocean",
      logo: (
        <Image
          alt="Digital Ocean logo"
          width="32"
          height="32"
          src="/images/cloud-icons/do.svg"
        />
      ),
      value: 3.1,
    },
    {
      title: "Azure",
      logo: (
        <Image
          alt="Azure logo"
          width="32"
          height="32"
          src="/images/cloud-icons/azure.svg"
        />
      ),
      value: 3.74,
    },
  ]}
/>

# Sidenotes

It should be noted that all tests ran at around the same time for each provider. There will be a variance due to congestion and other factors paying into initialization time.

The default (or recommended) settings were used when configuring the nodes.

The OS has a large impact on the time it takes for a node to reach a ready state, it's therefore likely that the initialization time can be improved by changing to a OS optimized for Kubernetes or container-based workloads — if you're not using one already.

As for Symbiosis a significant part of initialization is due to the relatively slow initialization of Ubuntu. Stay tuned for snappier scaling as we move to optimized images.

If you enjoyed this article you can follow us on [twitter](https://twitter.com/symbiosiscloud). More benchmarks to come. 👋


Benchmarking Kubernetes node initialization


**Kubernetes 1.25 is generally available on Symbiosis. We wrote a summary on some of the most important (in our opinion) APIs that got stabilized.**

# Pod Security Admission stabilized

**PodSecurityPolicy have been removed in favor of PodSecurityAdmission. But what are PodSecurityAdmissions?**

PodSecurityAdmission lets users define isolation levels that either enforce (reject), audit or warn when pods violate the selected policy.

The baseline security policy covers most known priviledge escalations and can easily be added to an existing namespace by adding the label, for example with the policy mode enforce to block any pods violating the policy:

```bash
kubectl label ns my-namespace pod-security.kubernetes.io/enforce=baseline
```

The two other standard policies are priviledge (unrestricted) and restricted, the least permissive and covers most best practices.

To apply a policy cluster-wide have a look at [AdmissionConfigurations](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/).

# Ephemeral Containers stabilized

Ephemeral containers can, for example, be used to attach short lived containers to running pods.

One important use-case is to debug a pod running distroless images as they don't include a shell that can be used for debugging.

An ephemeral container can be created using `kubectl debug`.

```bash
kubectl debug -it --image=busybox pod-name
```

Above command will launch an interactive shell to a busybox image running inside pod-name.

Details on how to use the command to debug containers can be found in the official [documentation](https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/).

# Other selected changes

- **[stable]** Ephemeral storage (such as `emptyDir`) can now be limited so that pods get evicted if storage consumption exceed a certain threshold.
- **[beta]** CRD Validation Expression Language adds a declarative syntax for validating custom resources ([read more](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules))
- **[misc]** Size of the kube-proxy image has been decreased by almost half as it's now built using a distroless OS rather than Debian
- **[misc]** Container registry for k8s components has moved from k8s.gcr.io to registry.k8s.io

You can [read the full announcement on kubernetes.io](https://kubernetes.io/blog/2022/08/23/kubernetes-v1-25-release/).

# Deprecation of 1.22

We are also ending support for 1.22. All clusters that are still running 1.22 will automatically update on December 16th. More details at [Control Planes#Version upgrades](/docs/features/control-planes#version-upgrades).


Kubernetes 1.25 is available


**Are you planning to run CI tests in a production-like environment? Perhaps you are running short-lived workloads in separate clusters? If so, you might care about cluster boot times.**

We ran a benchmark across 8 different providers of managed Kubernetes to determine how long they take to initialize.

We compared the providers by creating a cluster with a single node and measuring the time from creation until the node was in a ready state.

The benchmark ran 15 times for each provider. The averaged time is shown in the chart below and the complete data is available [in this spreadsheet](https://docs.google.com/spreadsheets/d/1Qdf4OilsFGR4tCgNh3MwFZiYiFpM_4-QXJS3lmJwnlw/edit?usp=sharing).

The source code is on [GitHub](https://github.com/symbiosis-cloud/k8s-cluster-creation-benchmarks). We will happily accept pull requests if you find a way to improve it.

# Providers tested

- Azure
- AWS
- Civo
- Digital Ocean
- Google Cloud
- Linode
- Scaleway
- Symbiosis

Civo diverges by running the [k3s](https://k3s.io/) distribution, a lightweight Kubernetes built for IoT and edge computing.

# The results

<BasicVerticalChart
  items={[
    {
      title: "Symbiosis",
      logo: (
        <Image
          alt="Symbiosis logo"
          width="32"
          height="32"
          src="/icon-zoomed.png"
        />
      ),
      value: 1.66,
    },
    {
      title: "Civo",
      logo: (
        <Image
          alt="Civo logo"
          width="32"
          height="32"
          src="/images/cloud-icons/civo.png"
        />
      ),
      value: 2.09,
    },
    {
      title: "Scaleway",
      logo: (
        <Image
          alt="Scaleway logo"
          width="32"
          height="32"
          src="/images/cloud-icons/scaleway.png"
        />
      ),
      value: 4.26,
    },
    {
      title: "Azure",
      logo: (
        <Image
          alt="Azure logo"
          width="32"
          height="32"
          src="/images/cloud-icons/azure.svg"
        />
      ),
      value: 4.48,
    },
    {
      title: "Google Cloud",
      logo: (
        <Image
          alt="Google Cloud logo"
          width="32"
          height="32"
          src="/images/cloud-icons/gcp.svg"
        />
      ),
      value: 5.12,
    },
    {
      title: "Digital Ocean",
      logo: (
        <Image
          alt="Digital Ocean logo"
          width="32"
          height="32"
          src="/images/cloud-icons/do.svg"
        />
      ),
      value: 6.29,
    },
    {
      title: "Linode",
      logo: (
        <Image
          alt="Linode logo"
          width="32"
          height="32"
          src="/images/cloud-icons/linode.png"
        />
      ),
      value: 6.41,
    },
    {
      title: "AWS",
      logo: (
        <Image
          alt="AWS logo"
          width="32"
          height="32"
          src="/images/cloud-icons/aws.svg"
        />
      ),
      value: 18.8,
    },
  ]}
/>

Providers give access to the cluster at different stages of initialization. For example, Linode make it possible to download credentials after just 11 seconds, however it took another 4 minutes until workloads were finally schedulable.

In order to remove variance we also disabled high availability (HA) for the providers that made it possible.

# Sidenotes

It should be noted that all tests ran at around the same time for each provider. It's reasonable to assume that the result will differ slightly throughout the day depending on factors like congestion.

AWS was significantly slower to initialize compared to the others. Although we did attempt to create the AWS cluster manually (without the CLI) it didn't have any significant impact on the benchmark.

We hope to publish yet another benchmark on node boot times in the near future and we have already gathered most of the data. With quick boot time your cluster can react to bursts faster and avoid running out of resources.

If you enjoyed this article you can follow us on [twitter](https://twitter.com/symbiosiscloud) or [reach us by email](mailto:info@symbiosis.host). Until next time. 👋


Benchmarking cluster creation time for 8 managed Kubernetes providers


**The most cost-efficient Kubernetes provider just got a glow-up. This month we are introducing cluster autoscaling and analytics.**

# Cluster analytics

There are a million packages and charts that can help you gauge the health and status of your Kubernetes cluster, but sometimes you just want something simple.

<div className="rounded-lg overflow-hidden shadow-xl flex">
  <Image
    src="/images/articles/autoscaler-and-analytics-ui.png"
    height="440"
    width="1201"
  />
</div>

We have incorporated CPU, memory and bandwith usage into the main UI so you can get a quick overview of the resource utilization in your cluster.

The telemetry is collected every minute and the UI is updated in real-time. Since the data is provided by the hypervisor no additional agents need to be installed and the overhead is negligible.

We are hoping to add more telemetry and metrics over time so stay tuned!

# Autoscaler support

There's not much merit to using the cloud if you can't scale things up and down on a whim, so we are finally adding in support for autoscaler so this process can be automated.

Node pools can be configured to scale automatically by setting a minimum and maximum number of nodes, autoscaler does the rest.

No need to configure your cluster for autoscaler support. Symbiosis will pick up that you have a node pool set to autoscale and launch the controller to your cluster automatically.

Autoscaling can be configured on all integrations (UI, Terraform, REST, SDKs).

# In other news

We benchmarked cluster initialization times and as it turns out we are pretty fast. Stay tuned for our post on how we stack up against the competition.

As always, if you want to get in touch you can reach us on [twitter](https://twitter.com/symbiosiscloud) or [by email](mailto:info@symbiosis.host).


Autoscaler & analytics


**Kubernetes 1.24 is generally available for your clusters. We compiled a list of the most important changes for you as an end-user.**

# Selected changes

- The overhead of running pods is included in the resource metrics used for scheduling and runtime limits
- Dockershim has been deprecated. A big step for k8s but a small (or not so relevant) step for Symbiosis since nodes have always been running containerd
- **[beta]** GRPC requests can be used for liveness, readiness and startup probes
- **[stable]** Pod affinity rules can specify a namespaceSelector to select which namespaces a label selector matches against
- **[stable]** PriorityClasses can be set to non-preemptive. Meaning pods will remain in the scheduling queue until sufficient resources are available, even if lower-priority pods are running
- **[stable]** Services can be annotated with `service.kubernetes.io/load-balancer-class` to configure a third-party loadbalancer instead of the default Symbiosis LB

You can [read the full announcement on kubernetes.io](https://kubernetes.io/blog/2022/05/03/kubernetes-1-24-release-announcement/).

# Deprecation of 1.21

With the introduction of 1.24 we are also ending support for 1.21. All clusters that are still running 1.21 will be automatically updated on October 10th. More information available on [Control Planes#Version upgrades](https://docs.symbiosis.host/reference/control-plane).

# Preview of 1.25

Kubernetes 1.25 was release about a month ago. We are launching a closed preview to a small number of our users. Send an email to [beta@symbiosis.host](mailto:beta@symbiosis.host) and we will add you to our wait-list.


Kubernetes 1.24 is available


**We're wrapping up summer '22 by compiling a list of some important changes and new features, with highly-available (HA) clusters and Golang SDK being the most notable.**

We rely heavily on our users to determine what to build and when. Don't hesitate to [contact us](/support) with your feedback and ideas.

Without further ado...

## Highly-available control planes

Control plane replicas (HA) increases the reliability of your cluster by making your Kubernetes API server and data store redundant.

Normally your workloads are not directly impacted by downtime affecting your control plane. However, in case of an outage your cluster might not be able to schedule any new workloads or handle any user requests to the API server.

With highly-available control planes you minimize this risk by removing the single point of failure. We therefore highly recommend enabling HA for your production clusters. Replicas can also reduce response times by distributing the load between your replicas.

Highly-available control planes are available in open preview. Clusters can be configured with highly-available control planes for an additional $32/mo during beta.

## Golang SDK

We have released a first version of our [Golang SDK](https://github.com/symbiosis-cloud/symbiosis-go). The SDK currently supports creating, listing, fetching and deleting clusters, node pools, team memberships and team invitations.

As we've started using it for out terraform plugin and k8s controllers we expect to add more functionality to the SDK over time.

## Labels and taints for node pools

You can add custom taints and labels to node pools. This is useful if you wish to restrict certain pods to a specific node type (like memory optimized nodes) or avoid congestion by separating environments or workloads physically.

Note that it's not yet possible to add default taints or labels to existing node pools.

New node pools are also labeled by their node pool name and ID. Older nodes can be recycled in order to add the new labels.

<center>
  <Image src="/images/tainted-pool-example.png" height="212" width="743" />
</center>

## Miscellaneous changes

- **[Nodes]** Boot scripts improvements to speed up node initialization (down to ~90 sec)
- **[Nodes]** Nodes and persistent volumes are automatically LUKS encrypted
- **[Billing]** Teams can request an increase on their resource quotas (under general settings in UI)
- **[Terraform Plugin]** Added `symbiosis_cluster_service_account` resource
- **[Terraform Plugin]** Clusters expose a root identity to make it easier to, for example, initialize a TF Kubernetes provider (see [example](https://registry.terraform.io/providers/symbiosis-cloud/symbiosis/latest/docs/resources/cluster))


Symbiosis Summer 2022 Update 🌞


**Hey! We are Stephen and Buster, and we have built a Kubernetes provider that's 1/4 the cost of AWS and significantly simpler to set-up and operate.**

The price and complexity of Google Cloud and AWS has made Kubernetes looking much scarier and complicated than it necessarily has to be. For example, just setting up a Kubernetes cluster shouldn't require [17 terraform resources](https://github.com/terraform-aws-modules/terraform-aws-eks).

We have experienced first-hand how frustrating managing Kubernetes can be: lackluster support, upgrades that take long to complete and simple mistakes such as excessive logging or egress traffic that can lead to [devastating bills](https://www.theregister.com/2020/12/10/google_cloud_over_run/).

We offer 2 vCPU nodes with 8 GiB memory for $18/mo, with support for CSI driver, external load balancers, layer 2 VPCs, free support & RCA, terraform plugin and more.

**In other words:** Our goal is to offer the support of providers like Rackspace, the scalability of AWS and a price comparable to co-location.

# Slash your cloud expenses

Since the birth of AWS we have gotten used to paying significant premiums for the convenience of the cloud. With complicated and opaque billing every resource you use is a potential footgun.

We hope to bring prices to par with the real costs of hosting cloud infrastructure, so you can both lower your expenses while still enjoying the benefits of fully managed Kubernetes.

<ViewportObserver>
  {({ hasAppearedInViewport }) => (
    <div className="grid gap-3 lg:mx-5 py-5">
      <div className="flex items-center">
        <div className="w-16 flex items-center">
          <Image
            alt="DO logo"
            width="32"
            height="32"
            src="/images/cloud-icons/do.svg"
          />
        </div>
        <div className="grow">
          <AnimatedHorizontalLoadingBar
            hasAppearedInViewport={hasAppearedInViewport}
            value={2520}
            backgroundColor="#0168ff"
            prefix="$"
            suffix="/mo"
            width="100%"
            delayMs={500}
          />
        </div>
      </div>
      <div className="flex items-center">
        <div className="w-16 flex items-center">
          <Image
            alt="AWS logo"
            width="32"
            height="32"
            src="/images/cloud-icons/aws.svg"
          />
        </div>
        <div className="grow">
          <AnimatedHorizontalLoadingBar
            hasAppearedInViewport={hasAppearedInViewport}
            value={2396}
            backgroundColor="#ff9900"
            prefix="$"
            suffix="/mo"
            width="95.08%"
            delayMs={600}
          />
        </div>
      </div>
      <div className="flex items-center">
        <div className="w-16 flex items-center">
          <Image
            alt="Google Cloud logo"
            width="32"
            height="32"
            src="/images/cloud-icons/gcp.svg"
          />
        </div>
        <div className="grow">
          <AnimatedHorizontalLoadingBar
            hasAppearedInViewport={hasAppearedInViewport}
            value={1957}
            backgroundColor="#4285f4"
            prefix="$"
            suffix="/mo"
            width="77.65%"
            delayMs={700}
          />
        </div>
      </div>
      <div className="flex items-center">
        <div className="w-16 flex items-center">
          <Image
            alt="Civo logo"
            width="32"
            height="32"
            src="/images/cloud-icons/civo.png"
          />
        </div>
        <div className="grow">
          <AnimatedHorizontalLoadingBar
            hasAppearedInViewport={hasAppearedInViewport}
            value={1600}
            backgroundColor="#0f273e"
            prefix="$"
            suffix="/mo"
            width="63.49%"
            delayMs={800}
          />
        </div>
      </div>
      <div className="flex items-center">
        <div className="w-16 flex items-center">
          <Image
            alt="Symbiosis logo"
            width="32"
            height="32"
            src="/icon-zoomed.png"
          />
        </div>
        <div className="grow">
          <AnimatedHorizontalLoadingBar
            hasAppearedInViewport={hasAppearedInViewport}
            value={720}
            backgroundColor="#171717"
            prefix="$"
            suffix="/mo"
            width="28.57%"
            delayMs={900}
          />
        </div>
      </div>
    </div>
  )}
</ViewportObserver>
<div className="text-center text-sm">
  Monthly cost of a Kubernetes cluster with 20 nodes having a total capacity of
  80 vCPU and 320GB RAM.
  <br />
  Note that Digital Ocean, Civo and Symbiosis nodes also include free traffic
  and storage.
</div>

Reserving instances for longer periods of time can significantly reduce your monthly costs with the hyperscale providers. However, use of autoscaler and variable workloads can make it difficult to make full use of these rebates.

# ~~Hostage fees~~ Egress fees

We’ve spoken with countless SREs and DevOps and there’s one billable item that is a constant source of fever dreams and anxiety: egress fees.

Moving a single terabyte from an AWS datacenter will cost you $92. Symbiosis nodes include free traffic, with our smallest nodes including 1TB/mo. If any nodes go over their allotment we charge $0.005/GB of overage, or ~$5/TB.

# Security

Securing a cloud platform is a challenge we take very seriously. Some measures we take include using isolated layer 2 overlay networks for your internal traffic and encrypting your nodes and volumes using LUKS.

For details check out our [security policies](https://symbiosis.host/docs/about/security).

# Feature parity (almost)

The least we can do is offer all key features you're already getting from your current cloud provider...

We cherry-picked some of the features we find most important into this chart. For a more thorough comparison see [LearnK8s’ managed services chart](https://docs.google.com/spreadsheets/d/1RPpyDOLFmcgxMCpABDzrsBYWpPYCIBuvAoUQLwOGoQw).

|                         | Symbiosis  |    Civo     | Digital Ocean | Google Cloud |     AWS     |
| ----------------------- | :--------: | :---------: | :-----------: | :----------: | :---------: |
| Terraform plugin        |     ✅     |     ✅      |      ✅       |      ✅      |     ✅      |
| External load balancers |     ✅     |     ✅      |      ✅       |      ✅      |     ✅      |
| Persistent volumes      |     ✅     |     ✅      |      ✅       |      ✅      |     ✅      |
| VPCs                    |     ✅     |     ✅      |      ✅       |      ✅      |     ✅      |
| Autoscaler              |     ❌     |     ❌      |      ✅       |      ✅      |     ✅      |
| Current stable version  | **1.23.9** | **1.22.11** |  **1.23.9**   | **1.22.10**  | **1.22.10** |

Note that we are currently working on autoscaler support.

# What else...

- Symbiosis is currently in open beta
- Our top priorities for Autumn '22 is Autoscaler support and US launch
- [Sign-up](/#newsletter-signup) for our newsletter
- Get started by reading our [documentation](https://symbiosis.host/docs)
- We appreciate all feedback we receive from our users, please do not hesitate to [contact us](/support)


Documentation

Installation

Tutorials