Fencing (computing)

Fencing is isolating a node of a computer cluster when it is malfunctioning. Isolating a node means ensuring that I/O can no longer be done from it. Fencing is typically done automatically, by cluster infrastructure such as shared disk file systems, in order to protect processes from other active nodes modifying the resources during node failures.

Fencing is required because it is impossible to distinguish between a real failure and a temporary hang. If the malfunctioning node is really down, then it cannot do any damage, so theoretically no action would be required (it could simply be brought back into the cluster with the usual join process). However, because there is a possibility that a malfunctioning node could itself consider the rest of the cluster to be the one that is malfunctioning, a race condition could ensue, and cause data corruption. Instead, the system has to assume the worst scenario and always fence in case of problems.

Fencing methods include:
* STONITH, which stands for "Shoot The Other Node In The Head", meaning automatically power off the server
* reserve/release ('R/R')
* persistent reservation (SCSI3)
* SAN Fabric fencing, which is widely used both by Red Hat Global File System (GFS) and the PolyServe File System (PSFS)

Reserve/release by its nature only works with two-node clusters, because one of the two nodes in the cluster, upon detecting that the other node has 'failed', will issue the reserve and grab all the disks for itself. The other node will commit suicide if it tries to do I/O (in case it was temporarily hung). The I/O failure triggers some code to kill the node. In general, in the case of two-node clusters, R/R is sufficient to address the split-brain issue, also.

For clusters greater than two nodes, R/R fencing does not work very well because it would cause all the nodes but one to commit suicide. In those cases persistent reservation is used. Persistent reservation is essentially a match on a key, so the node which has the right key can do I/O, otherwise its I/O fails. Therefore, it is sufficient to change the key on a failure to ensure the right behavior during failure.

External links

* [http://www.redhat.com/docs/manuals/csgfs/admin-guide/ch-fence.html Red Hat GFS 6.0 Administrator's Guide - Using the Fencing System]
* [http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html#QUORUM OCFS2 FAQ - Quorum and fencing]