Kind host not configured for more than two clusters

Kind uses a docker-in-docker technique to create multiple Kubernetes clusters on your host. But, in order for this to work for three or more clusters, the host running Docker typically needs an expanded configuration. This is mostly described in a known issue of kind. However, that document does not mention the additional complexity that arises when the OS running the containers is the guest OS inside a virtual machine on your host (e.g., a Mac, which does not natively run containers and so uses a virtual machine with a Linux guest OS).

Symptoms

Many KubeStellar setup paths check for the needed configuration. When the check fails, you get an error message like the following.

sysctl fs.inotify.max_user_watches is only 155693 but must be at least 524288

If you avoid the check but the configuration is not expanded then the symptom will most likely be setup ceasing to make progress at some point. Or maybe other errors about things not happening or things not existing.

Solution

To resolve this error, you need to increase the value of fs.inotify.max_user_watches and/or fs.inotify.max_user_instances. Follow the steps below:

For Rancher Desktop

1. Open the configuration file:

   vi "~/Library/Application Support/rancher-desktop/lima/_config/override.yaml"
   

2. Add the following script to the provision section:

   provision:
   - mode: system
     script: |
       #!/bin/sh
       sysctl fs.inotify.max_user_watches=524288
       sysctl fs.inotify.max_user_instances=512
   

3. Restart Rancher Desktop.

Docker on Linux

The resolution in the kind known issue can be used directly.

1. Create a new configuration file:

   sudo vi /etc/sysctl.d/99-sysctl.conf
   

2. Add the following lines:

   fs.inotify.max_user_watches=1048576
   fs.inotify.max_user_instances=1024
   

3. Apply the changes:

   sudo sysctl -p /etc/sysctl.d/99-sysctl.conf