Gluster DFS
Introduction
Gluster (https://www.gluster.org/) is a distributed parallel fault-tolerant file system, which can also be referred to as a clustered file system. I'll refer to it simply as a Distributed File System (DFS).
Gluster, currently owned by Red Hat, is an open source enterprise grade DFS with an active public community. There is also a commercially supported variant known as "Red Hat Gluster Storage" (https://www.redhat.com/en/technologies/storage/gluster).
Online documentation for Gluster can be found at https://docs.gluster.org/en/latest/.
Implementation
Gluster is implemented as a cluster of storage nodes. Each node is an equal peer in the cluster, so administrative commands can be run from any of the nodes.
Each node supplies one or more data partitions, known as 'bricks', that are used to build network accessible cluster volumes. There are a number of different configurations that the bricks can be arranged into to build the volumes. More information about volume types can be found in the documentation at https://docs.gluster.org/en/latest/Administrator Guide/Setting Up Volumes/.
A common implementation uses a configuration known as 'dispersed', where you have data storage and redundancy storage, striped across multiple bricks in the volume. This configuration provides for a file redundant and high availability network file system that is protected against node failure.
The layout is referred to in terms of brick capacity (which should be equal and balanced throughout the nodes), where you have x capacities of data plus y capacities of redundancy. Examples are 3+1 (3 capacities of data plus 1 capacity of redundancy) or 4+2 (4 data plus 2 redundancy). The number of redundancy capacities also indicates the number of nodes that can be lost without impacting on volume availability. For a 3+1 volume, one node could be lost without impact, and for a 4+2 volume, two nodes.
Administrative Commands
As stated above, all nodes in a cluster are equal peers, so administrative commands can be run from any node. All administrative commands are implemented as subcommands of the 'gluster' executable. Information about the commands can be viewed from either the man pages ('man gluster'), or via command line help ('gluster help'). Subcommands also have their own help pages, accessible as 'gluster < subcommand > help' (ie, 'gluster volume help').
Status
Gluster has extensive logging, located in /var/log/glusterfs. Much of the information in the log directory is specific to the node the logs are located on.
Status information at a cluster scope can be obtained using the gluster command. For example, to see the nodes that are currently members of the "Trusted Storage Pool" (TSP), use 'gluster pool list'. To see the status of each of the peer nodes in the cluster, use 'gluster peer status'.
To see what volumes have been defined, use 'gluster volume list'. To see basic info about one or more volumes, use 'gluster volume info' to see all volumes (assuming more than one volume has been defined), or 'gluster volume info < volume >' for information about a specific volume.
Similarly, to see more detailed information about volumes, you can use either 'gluster volume status' or 'gluster volume status < volume >'.
One command of note is 'gluster volume status < volume > clients', which shows all of the client connections to the volume.
Expansion
There are two ways to expand a volume in a Gluster cluster. One way would be to add an additional node to the cluster. This can be a bit complicated, as will be explained below.
Another way is to add more bricks to the volume. There is one caveat that must be considered when expanding a Gluster volume though. Due to the way the volumes are built, bricks must be added in the same quantity as were used to initially setup the volume. For example, in a 3+1 volume, the volume must be expanded in multiples of 4 bricks, and for a 4+2 volume, 6 bricks.
This caveat is why expanding a Gluster volume by adding a new node can be problematic. In order to satisfy the brick count requirement, the new node will likely end up unbalanced with respect to the other node brick counts. Future volume expansion will also be confusing, as new bricks will be added in a different count than the current number of nodes.
A simpler method would be to plan out the cluster size with future expansion in mind, and only expand the volume by adding equal number of bricks to each node.
Example: A six node cluster has a 4+2 volume made up of 1 TB bricks. The initial volume capacity would be 4 TB (4 data brick capacities). To expand this volume, add a 1 TB partition to each node. After formatting and mounting, the new bricks are added to the volume using the 'gluster volume add-brick' command. After adding the new set of bricks to the volume, the volume capacity will now be 8 TB, and in 'gluster volume info' the bricks will be listed as '2 x (4 + 2) = 12'. Future expansion of the volume is accomplished in the same way.
Network Access
Gluster supports two methods to directly access the volume. The first is the GlusterFS native client. This is the best method, and provides for high concurrency, performance and transparent failover.
Gluster also supports NFS v3 to access Gluster volumes.
In the case of sharing the Gluster volume via CIFS (Samba), a samba file server can be stood up with the volume mounted locally via the GlusterFS native client, and then re-shared across the network via CIFS.
Created: 2021-06-04 02:38
Last update: 2021-09-01 02:36