Last updated:

Set up a GlusterFS Server

As an open-source distributed file system, GlusterFS allows you to mount glusterfs volumes to your Pods. If a glusterfs volume is pre-populated with data, they can be shared among your Pods in a Kubernetes cluster.

This tutorial demonstrates how to configure GlusterFS on three server machines and install Heketi to manage your GlusterFS cluster.

Once you have GlusterFS and Heketi set up, you can install GlusterFS on your client machine and use KubeKey to create a KubeSphere cluster with GlusterFS as a storage class.

Prepare GlusterFS Nodes

There are three server machines of Ubuntu 16.04 in this example with each having one attached disk.

Hostname	IP Address	Operating System	Device
server1	192.168.0.2	Ubuntu 16.04, 4 Cores, 4 GB of Memory	/dev/vdd 300 GB
server2	192.168.0.3	Ubuntu 16.04, 4 Cores, 4 GB of Memory	/dev/vdd 300 GB
server3	192.168.0.4	Ubuntu 16.04, 4 Cores, 4 GB of Memory	/dev/vdd 300 GB

Note

Heketi will be installed on server1, which provides a RESTful management interface to manage the lifecycle of GlusterFS volumes. You can install it on a separate machine as well.
Attach more block storage disks to your server machine if you need more storage space.
Data will be saved to /dev/vdd (block device), which must be original without partitioning or formatting.

Log in to server1 and switch to the root user.
```
sudo -i
```
Change the root user password:
```
passwd
```
Note

Make sure password authentication is enabled in the file /etc/ssh/sshd_config (the value of PasswordAuthentication should be yes).
Change the root user password of server2 and server3 as well.

Add hosts file entries

Configure your DNS or edit the /etc/hosts file on all server machines to add their hostnames and IP addresses:

vi /etc/hosts

# hostname loopback address
192.168.0.2  server1
192.168.0.3  server2
192.168.0.4  server3

Make sure you add the above entries to the hosts file of all server machines.

On server1, create a key by running the following command. Press Enter directly for all the prompts.
```
ssh-keygen
```

Copy the key to all GlusterFS nodes.

ssh-copy-id root@server1

ssh-copy-id root@server2

ssh-copy-id root@server3

Verify that you can access all server machines from server1 through passwordless login.
```
ssh root@server1
```
```
ssh root@server2
```
```
ssh root@server3
```

Install GlusterFS on All Server Machines

On server1, run the following command to install software-properties-common.
```
apt-get install software-properties-common
```

Add the community GlusterFS PPA.

add-apt-repository ppa:gluster/glusterfs-7

Make sure you are using the latest package.
```
apt-get update
```
Install the GlusterFS server.
```
apt-get install glusterfs-server -y
```
Make sure you run the above commands on server2 and server3 as well and verify the version on all machines.
```
glusterfs -V
```

Note

The above commands may be slightly different if you do no install GlusterFS on Ubuntu. For more information, see the Gluster documentation.

Load Kernel Modules

Run the following commands to load three necessary kernel modules on server1.

echo dm_thin_pool | sudo tee -a /etc/modules

echo dm_snapshot | sudo tee -a /etc/modules

echo dm_mirror | sudo tee -a /etc/modules

Intall thin-provisioning-tools.

apt-get -y install thin-provisioning-tools

Make sure you run the above commands on server2 and server3 as well.

Create a GlusterFS Cluster

Run the following command on server1 to add other nodes and create a cluster.
```
gluster peer probe server2
```
```
gluster peer probe server3
```
Verify that all nodes in the cluster are connected successfully.
```
gluster peer status
```

Expected output:

Number of Peers: 2

Hostname: server2
Uuid: e1192d6a-b65e-4ce8-804c-72d9425211a6
State: Peer in Cluster (Connected)

Hostname: server3
Uuid: 9bd733e4-96d4-49d5-8958-6c947a2b4fa6
State: Peer in Cluster (Connected)

Install Heketi

As GlusterFS itself does not provide a way for API calls, you can install Heketi to manage the lifecycle of GlusterFS volumes with a RESTful API for Kubernetes calls. In this way, your Kubernetes cluster can dynamically provision GlusterFS volumes. Heketi v7.0.0 will be installed in this example. For more information about available Heketi versions, see its Release Page.

Download Heketi on server1.

wget https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-v7.0.0.linux.amd64.tar.gz

Note

You can also install Heketi on a separate machine.

Unzip the file.

tar -xf heketi-v7.0.0.linux.amd64.tar.gz

cd heketi

cp heketi /usr/bin

cp heketi-cli /usr/bin

Create a Heketi service file.

vi /lib/systemd/system/heketi.service

[Unit]
Description=Heketi Server
[Service]
Type=simple
WorkingDirectory=/var/lib/heketi
ExecStart=/usr/bin/heketi --config=/etc/heketi/heketi.json
Restart=on-failure
StandardOutput=syslog
StandardError=syslog
[Install]
WantedBy=multi-user.target

Create Heketi folders.

mkdir -p /var/lib/heketi

mkdir -p /etc/heketi

Create a JSON file for Heketi configurations.

vi /etc/heketi/heketi.json

An example file:

{
  "_port_comment": "Heketi Server Port Number",
  "port": "8080",

  "_use_auth": "Enable JWT authorization. Please enable for deployment",
  "use_auth": false,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "123456"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "123456"
    }
  },

  "_glusterfs_comment": "GlusterFS Configuration",
  "glusterfs": {
    "_executor_comment": [
      "Execute plugin. Possible choices: mock, ssh",
      "mock: This setting is used for testing and development.",
      "      It will not send commands to any node.",
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
      "      It will need the values in sshexec to be configured.",
      "kubernetes: Communicate with GlusterFS containers over",
      "            Kubernetes exec api."
    ],
    "executor": "ssh",

    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/root/.ssh/id_rsa",
      "user": "root"
    },

    "_kubeexec_comment": "Kubernetes configuration",
    "kubeexec": {
      "host" :"https://kubernetes.host:8443",
      "cert" : "/path/to/crt.file",
      "insecure": false,
      "user": "kubernetes username",
      "password": "password for kubernetes user",
      "namespace": "Kubernetes namespace",
      "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab"
    },

    "_db_comment": "Database file name",
    "db": "/var/lib/heketi/heketi.db",
    "brick_max_size_gb" : 1024,
	"brick_min_size_gb" : 1,
	"max_bricks_per_volume" : 33,


    "_loglevel_comment": [
      "Set log level. Choices are:",
      "  none, critical, error, warning, info, debug",
      "Default is warning"
    ],
    "loglevel" : "debug"
  }
}

Note

The account admin and its key value must be provided when you install GlusterFS as a storage class of your KubeSphere cluster.

Start Heketi.
```
systemctl start heketi
```

Check the status of Heketi.

systemctl status heketi

If you can see active (running), it means the installation is successful. Expected output:

● heketi.service - Heketi Server
   Loaded: loaded (/lib/systemd/system/heketi.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-03-09 13:04:30 CST; 4s ago
 Main PID: 9282 (heketi)
    Tasks: 8
   Memory: 6.5M
      CPU: 62ms
   CGroup: /system.slice/heketi.service
           └─9282 /usr/bin/heketi --config=/etc/heketi/heketi.json

Mar 09 13:04:30 server1 systemd[1]: Started Heketi Server.
Mar 09 13:04:30 server1 heketi[9282]: Heketi v7.0.0
Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Loaded ssh executor
Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Adv: Max bricks per volume set to 33
Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Adv: Max brick size 1024 GB
Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Adv: Min brick size 1 GB
Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 GlusterFS Application Loaded
Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Started Node Health Cache Monitor
Mar 09 13:04:30 server1 heketi[9282]: Listening on port 8080

Enable Heketi.

systemctl enable heketi

Expected output:

Created symlink from /etc/systemd/system/multi-user.target.wants/heketi.service to /lib/systemd/system/heketi.service.

Create a topology configuration file for Heketi. It contains the information of clusters, nodes, and disks added to Heketi.

vi /etc/heketi/topology.json

An example file:

   {
    "clusters": [
       {
         "nodes": [
           {
             "node": {
               "hostnames": {
                 "manage": [
                   "192.168.0.2" 
                ],
                "storage": [
                  "192.168.0.2" 
                ]
              },
              "zone": 1
            },
            "devices": [
              "/dev/vdd" 
            ]
          },
          {
            "node": {
              "hostnames": {
                "manage": [
                  "192.168.0.3" 
                ],
                "storage": [
                  "192.168.0.3"
                ]
              },
              "zone": 1
            },
            "devices": [
              "/dev/vdd" 
            ]
          },
          {
             "node": {
               "hostnames": {
                 "manage": [
                   "192.168.0.4"
                ],
                "storage": [
                  "192.168.0.4"  
                ]
              },
              "zone": 1
            },
            "devices": [
              "/dev/vdd"
            ]
          }
        ]
      }
    ]
  }

Note

Replace the IP addresses above with your own.
Add your own disk name for devices.

Load the Heketi JSON file.

export HEKETI_CLI_SERVER=http://localhost:8080

heketi-cli topology load --json=/etc/heketi/topology.json

Expected output:

Creating cluster ... ID: 2d9e11adede04fe6d07cb81c5a1a7ea4
	Allowing file volumes on cluster.
	Allowing block volumes on cluster.
	Creating node 192.168.0.2 ... ID: 0a9f240ab6fd96ea014948c5605be675
		Adding device /dev/vdd ... OK
	Creating node 192.168.0.3 ... ID: 2468086cadfee8ef9f48bc15db81c88a
		Adding device /dev/vdd ... OK
	Creating node 192.168.0.4 ... ID: 4c21b33d5c32029f5b7dc6406977ec34
		Adding device /dev/vdd ... OK

The above output displays both your cluster ID and node ID. Run the following command to view your cluster information.

heketi-cli cluster info 2d9e11adede04fe6d07cb81c5a1a7ea4 # Use your own cluster ID.

Expected output:

Cluster id: 2d9e11adede04fe6d07cb81c5a1a7ea4
Nodes:
0a9f240ab6fd96ea014948c5605be675
2468086cadfee8ef9f48bc15db81c88a
4c21b33d5c32029f5b7dc6406977ec34
Volumes:

Block: true

File: true

Feedback

Was this page Helpful?

Receive the latest news, articles and updates from KubeSphere

Thanks for the feedback. If you have a specific question about how to use KubeSphere, ask it on Slack. Open an issue in the GitHub repo if you want to report a problem or suggest an improvement.

What’s on this Page

Set up a GlusterFS Server

Prepare GlusterFS Nodes

Set up Passwordless SSH Login

Configure root login

Add hosts file entries

Configure passwordless SSH login

Install GlusterFS on All Server Machines

Load Kernel Modules

Create a GlusterFS Cluster

Install Heketi