Pacemaker

Pacemaker is "resource manager", designed to provision resources across a cluster of machines according to a set of constraints.

Pacemaker objects

Pacemaker declares several different object types, each of which has a very specific meaning:

A Cluster is a set of Nodes.
Nodes are individual machines within a Cluster.
Quorum is achieved when a majority of Nodes are able to agree about the state of a Resource. Quorum must be reached before the Cluster can take an action (e.g. stop, start or restart) on a Resource.
Resources define the services we want to run in a Cluster. These are modular and defined by resource agents.
A Fence declares a means of disabling one of the Nodes in the event Quorum cannot be reached, thus protecting shared storage from data corruption or destruction.
Constraints declare conditions in which Resources can run:
- Colocation constraints set the neighbouring Resources with which a Resource must be colocated;
- Location constraints set which Nodes they can run on; and
- Order constraints define the order in which they should start and stop.
Master/slave relationships allow us to group sets of Nodes in active/passive pairs.

Cluster manager components

Pacemaker components

The cluster manager is composed of many components:

pcsd (Pacemaker/Corosync Configuration System daemon) manages the configuration across the cluster nodes.
Corosync is the cluster communication manager, providing the framework for notifying the cluster nodes about the health of the services running on it and deciding on the course of corrective action when something's gone wrong.
Pacemaker itself provides the cluster resource manager, coordinating the resource health checking and start/stop operations.
Resource agents interact with the individual services, health checking, starting and stopping them as appropriate. We configure these by declaring resources.
Fence agents are designed to prevent cluster nodes from corrupting or destroying data on shared storage, e.g. by cutting the power or network from the node.

Management

A bunch of different management tools exist, the most common being crmsh and pcs.

Common operations

Finding where a resource is running

Because services are configured to move between the nodes we often need to locate the service we're interested in. To see the complete status of the cluster, see the output of the following command from one of the nodes:

$ sudo pcs status
Cluster name: ubiquitous
Stack: corosync
Current DC: node1.example.com (version 1.1.18-2b07d5c5a9) - partition with quorum
Last updated: Wed Oct  9 14:55:09 2019
Last change: Thu Sep  5 15:27:57 2019 by root via cibadmin on node0.example.com

2 nodes configured
24 resources configured

Online: [ node0.example.com node1.example.com ]

Full list of resources:

 Master/Slave Set: export-data-election [export-data-disks]
     Masters: [ node1.example.com ]
     Slaves: [ node0.example.com ]
 export-data-lvm        (ocf::heartbeat:LVM):   Started node1.example.com
 Resource Group: export-data-shares
     export-data-share-a (ocf::heartbeat:exportfs):      Started node1.example.com
     export-data-share-b  (ocf::heartbeat:exportfs):      Started node1.example.com
 Resource Group: export-data
     export-data-filesystem     (ocf::heartbeat:Filesystem):    Started node1.example.com
     export-data-nfs    (ocf::heartbeat:nfsserver):     Started node1.example.com
     export-data-root   (ocf::heartbeat:exportfs):      Started node1.example.com
 export-data-fence      (stonith:fence_azure_arm):      Stopped

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Each resource will have a line with a status (e.g. Started or Stopped). Resources which are running will have the hostname of the server they're running on next to the status (e.g. node1.* in the above example).

Putting a node into standby

When performing cluster maintenance we often need to take a node out of rotation to enable us to work on it without risking the availability of the service. From the node you wish to take out of rotation:

sudo pcs cluster standby

When the maintenance is complete, bring the node back up:

sudo pcs cluster unstandby