MinIO on vSphere - Automated Deployment and Onboarding

In the world of Kubernetes, reliable S3-compliant object storage is essential for tasks like storing backups. However, not everyone has access to a native S3-compatible solution, and setting one up can feel like a daunting task. MinIO, an open-source object storage solution, is a popular choice to fill this gap. Its lightweight, high-performance architecture makes it an excellent option for Kubernetes users seeking quick and reliable storage.

MinIO is also one of the most widely adopted open-source object storage solutions, thanks to its simplicity and S3 compatibility. It’s perfect for Kubernetes environments that need a reliable and scalable storage layer for backups, logs, or other data.

I’ve frequently been asked how to deploy MinIO to vSphere, particularly in scenarios where users want to stand up a solution quickly without compromising reliability. To streamline this process, I created an Ansible-based automation to deploy and configure MinIO on a vSphere virtual machine. This blog post shares the details of the automation, making it easier for others to implement a similar solution.

Deployment and Onboarding

First, clone my GitHub repository, where all the code resides. Then, change directory to the minio-on-vsphere/ansible/deploy-minio directory.

git clone https://github.com/itaytalmi/minio.git
cd minio-on-vsphere/ansible/deploy-minio

The Ansible playbook consists of two roles:

deploy-minio-vm - responsible for the infrastructure parts, such as deploying and configuring the vSphere VM itself. It uses the Ubuntu Cloud Image OVA as a base for the deployment of the VM and templates a Cloud-Init user data configuration to bootstrap the OS configuration.
The open-source ricsanfre.minio Ansible role, which is available on Galaxy for the installation and configuration of MinIO. I leverage this role with some customizations.

The Ansible modules require the pyvmomi, the VMware Ansible collection and the ricsanfre.minio Ansible role. You can use the requirements.txt and the requirements.yml files to install these prerequisites, using pip and ansible-galaxy.

You can simply run the following to install the required packages:

ansible-galaxy install -r requirements.yml
python3 -m pip install --upgrade -r requirements.txt

Next, review and update the vars.yml file.

Most of the parameters are quite self-explanatory, but I’ll provide some references and clarifications on their usage.

vcenter_hostname, vcenter_username and vcenter_password are used for vSphere authentication.
vsphere_datacenter is your vSphere datacenter
vsphere_cluster is your vSphere cluster
vsphere_resource_pool is your vSphere resource pool. Leave as Resources if you don’t have a preference
vsphere_datastore is your vSphere datastore
vsphere_network is your vSphere network/portgroup
vsphere_vm_folder is your vSphere VM folder for the MinIO VM
ubuntu_ova_download_url is the source of the Ubuntu Cloud Image. It shouldn’t be modified, unless you have your own OVA or you would like to use a different Ubuntu version.
vm_name is the desired VM name
vm_additional_user and vm_additional_password will be used by Ansible to connect to the machine and perform the configuration. Note: do not use the username minio as it is reserved for the MinIO service and Ansible will be disabling shell access to the OS for this user.
vm_disk_provisioning is the type of disk for your MinIO VM (thin, thick lazy zeroed, thick eager zeroed, etc.)
vm_os_disk_size_gb, vm_num_cpus and vm_memory_mb are the desired VM resources.
vm_authorized_ssh_keys must contain an SSH key which is present on the machine running the Ansible playbook. Ansible will use this SSH key to connect to the machine.
vm_ip_address, vm_prefix_length, vm_default_gateway, vm_search_domains and vm_dns_servers are the network configuration for the machine.
minio_root_user and minio_root_password are the credentials you will be using to authenticate to MinIO as an admin.
minio_data_disk_size_gb is the desired size of the MinIO data disk. This is the disk which will be used to store the MinIO data.
minio_server_datadir is the path on the OS where the MinIO data will be stored.
minio_server_data_filesystem_disk is the file system on the OS where the MinIO data will be stored.
minio_server_data_filesystem_type is the file system type
minio_alias is the MinIO alias which will be used by the MinIO client CLI to connect to the MinIO instance.
minio_buckets and minio_users can be used to define a list of buckets, users and their access policies to be created.

# vSphere variables
vcenter_hostname: vcsa.cloudnativeapps.cloud
vcenter_username: [email protected]
vcenter_password: YourP@ssw0rd

vsphere_datecenter: Demo-Datacenter
vsphere_cluster: Demo-Cluster
vsphere_resource_pool: Resources
vsphere_datastore: vsanDatastore
vsphere_network: demo-k8s
vsphere_vm_folder: "/{{ vsphere_datecenter }}/vm/minio"

# VM variables
ubuntu_ova_download_url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.ova
vm_name: demo-minio
vm_additional_user: minioadmin
vm_additional_user_password: MinIO1!
vm_disk_provisioning: thin
vm_os_disk_size_gb: 50
vm_num_cpus: 2
vm_memory_mb: 8192

# Note: the authorized ssh keys must contain the SSH key of the machine where the playbook is executed, so it can connect to the MinIO Server node
vm_authorized_ssh_keys:
  - ssh-rsa AAAAB3NzaC1y...

vm_ip_address: 10.100.152.13
vm_prefix_length: 24
vm_default_gateway: 10.100.152.1
vm_search_domains:
  - terasky.demo
vm_dns_servers:
  - 10.100.100.100

# MinIO server config
minio_root_user: admin
minio_root_password: MinIO1!@#
minio_data_disk_size_gb: 100
minio_server_datadir: /minio-data
minio_server_data_filesystem_disk: /dev/sdb
minio_server_data_filesystem_type: xfs
minio_alias: minio

# MinIO buckets and users
minio_buckets:
  - name: k8s-backup
    policy: read-write

minio_users:
- name: k8s-backup
  password: Kubernetes1!
  buckets_acl:
    - name: k8s-backup
      policy: read-write

Note that the ricsanfre.minio Ansible role supports additional configurations and covers many use cases, including multi-node MinIO clusters, site replication, TLS configuration, and more. I will not be covering these here, but feel free to explore on your own. Refer to the ricsanfre/ansible-role-minio repository on GitHub and the ricsanfre.minio Ansible Role on Galaxy.

To run the playbook:

ansible-playbook playbook.yml

Example output:

PLAY [Deploy MinIO] **********************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [deploy-minio-vm : Template VM User Data] *******************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [deploy-minio-vm : Download Ubuntu Cloud Image] *************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [deploy-minio-vm : Power Off VM (only used for redeployment)] ***********************************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [deploy-minio-vm : Delete MinIO VM (only used for redeployment)] ********************************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [deploy-minio-vm : Deploy MinIO VM from Ubuntu Cloud Image] *************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [deploy-minio-vm : Configure VM Resources] ******************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [deploy-minio-vm : Power on VM and wait for IP address] *****************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [deploy-minio-vm : Wait for VMware tools] *******************************************************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [deploy-minio-vm : Remove MinIO VM IP address from known hosts if it exists (to ensure Ansible can successfully connect to it with a new thumbprint)] *******************************************************************************************************************************
changed: [localhost]

TASK [deploy-minio-vm : Add Ansible host] ************************************************************************************************************************************************************************************************************************************************
changed: [localhost]

PLAY [Setup MinIO] ***********************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [Wait for connection] ***************************************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [Update apt cache] ******************************************************************************************************************************************************************************************************************************************************************
changed: [10.100.152.13]

TASK [Create data storage directory filesystem] ******************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [Mount data storage directory filesystem] *******************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [Create MinIO group] ****************************************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [Create MinIO user] *****************************************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [Create the MinIO data storage directory and apply permissions] *********************************************************************************************************************************************************************************************************************
changed: [10.100.152.13]

TASK [ricsanfre.minio : Install server] **************************************************************************************************************************************************************************************************************************************************
included: /home/k8s/.ansible/roles/ricsanfre.minio/tasks/install_server.yml for 10.100.152.13

TASK [ricsanfre.minio : Compose the Minio server download url] ***************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Get the Minio server checksum for architecture amd64] ************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Create Minio group] **********************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Create Minio user] ***********************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Create the Minio config directories] *****************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13] => (item=/etc/minio)
ok: [10.100.152.13] => (item=/etc/minio/ssl)
ok: [10.100.152.13] => (item=/etc/minio/policy)

TASK [ricsanfre.minio : Create the Minio data storage directories] ***********************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13] => (item=/minio-data)
skipping: [10.100.152.13]

TASK [ricsanfre.minio : Download the Minio server] ***************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Generate the Minio server envfile] *******************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Create the Minio server systemd config] **************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Copy SSL private key file] ***************************************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13]

TASK [ricsanfre.minio : Copy cert file] **************************************************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13]

TASK [ricsanfre.minio : Flush handlers] **************************************************************************************************************************************************************************************************************************************************

TASK [ricsanfre.minio : Ensure minio is started at boot] *********************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Set secure minio url if not defined] *****************************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13]

TASK [ricsanfre.minio : Set unsecure minio_url if not defined] ***************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Install client] **************************************************************************************************************************************************************************************************************************************************
included: /home/k8s/.ansible/roles/ricsanfre.minio/tasks/install_client.yml for 10.100.152.13

TASK [ricsanfre.minio : Compose the Minio client download base url] **********************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Get the Minio client checksum for architecture amd64] ************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Download the Minio client] ***************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Configure server] ************************************************************************************************************************************************************************************************************************************************
included: /home/k8s/.ansible/roles/ricsanfre.minio/tasks/configure_server.yml for 10.100.152.13

TASK [ricsanfre.minio : Set insecure mc command] *****************************************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13]

TASK [ricsanfre.minio : Set secure mc command] *******************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Configure minio connection alias] ********************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Create Minio Buckets] ********************************************************************************************************************************************************************************************************************************************
included: /home/k8s/.ansible/roles/ricsanfre.minio/tasks/create_minio_buckets.yml for 10.100.152.13

TASK [ricsanfre.minio : Ensure PIP is installed.] ****************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Install pip packages] ********************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Create Buckets] **************************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13] => (item={'name': 'k8s-backup', 'policy': 'read-write'})

TASK [ricsanfre.minio : Create Minio Users] **********************************************************************************************************************************************************************************************************************************************
included: /home/k8s/.ansible/roles/ricsanfre.minio/tasks/create_minio_user.yml for 10.100.152.13 => (item={'name': 'k8s-backup', 'password': 'Kubernetes1!', 'buckets_acl': [{'name': 'k8s-backup', 'policy': 'read-write'}]})

TASK [ricsanfre.minio : Check whether user is already configured] ************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Create users] ****************************************************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13]

TASK [ricsanfre.minio : Create users policy file] ****************************************************************************************************************************************************************************************************************************************
ok: [10.100.152.13]

TASK [ricsanfre.minio : Add user policy] *************************************************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13]

TASK [ricsanfre.minio : Apply user policy] ***********************************************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13]

TASK [ricsanfre.minio : Configure Minio Site Replication] ********************************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13]

TASK [ricsanfre.minio : Create Prometheus bearer token] **********************************************************************************************************************************************************************************************************************************
skipping: [10.100.152.13]

PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************************************
10.100.152.13              : ok=33   changed=2    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0
localhost                  : ok=9    changed=3    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

Once the Playbook completes, you should be able to access the MinIO console from your browser at http://<MINIO_IP>:9202. Login using the credentials you specified for the minio_root_user and minio_root_password parameters earlier. For example, admin and MinIO1!@#.

You should be able to see the bucket and user and policy you specified in your vars.yml configuration, which were also created by Ansible during provisioning.

You can also log in to the MinIO console using this user. Note that it has minimal permissions and can only access this specific bucket due to the policy it’s assigned to.

Using the MinIO Client

If you wish to interact with MinIO from your own machine (outside of the MinIO VM), you have to install the mc MinIO client.

If you use the Homebrew the package manager, you can use brew install minio/stable/mc to install it.

If you are on a Linux machine without Homebrew installed, you can use the following to install it:

sudo wget https://dl.minio.io/client/mc/release/linux-amd64/mc -O /usr/local/bin/mc \
&& sudo chmod +x /usr/local/bin/mc

Verify the installation by running mc --version.

Example output:

mc version RELEASE.2024-11-17T19-35-25Z (commit-id=bb4ff4951a3e54bbee6ac75cfaf387c521e98709)
Runtime: go1.23.3 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

Next, create a MinIO alias for the connection to the server. For example:

mc alias set minio http://10.100.152.13:9091 admin 'MinIO1!@#'

Output:

Added `minio` successfully.

Verify the connection to the server by running any command against the server. For example:

mc admin info minio

Example output:

10.100.152.13:9091
   Uptime: 1 day
   Version: 2024-11-07T00:52:20Z
   Network: 1/1 OK
   Drives: 1/1 OK
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 1.9% (total: 100 GiB) │ 1                   │ 1            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

0 B Used, 1 Bucket, 0 Objects
1 drive online, 0 drives offline, EC:0

Another example is to list the buckets. Note: minio is the alias name.

mc ls minio

Example output:

[2024-11-17 20:06:58 UTC]     0B k8s-backup/

Wrap Up

You should now be all set to store your data on MinIO, whether it be used for Kubernetes backups, or anything else.

This post is a reference for a quick and easy deployment of MinIO in an on-premise vSphere environment, which is an incredibly common scenario. However, there are many ways to run and host MinIO on-premise, in the cloud, or even on Kubernetes itself. In a future post, I will provide references for hosting and managing MinIO on Kubernetes using the MinIO operator.