https://codeberg.org/sanlock/sanlock
sanlock is a lock manager for shared storage environments. It allows applications running on multiple hosts to coordinate access to shared resources, such as data objects on shared storage systems like a SAN, preventing data corruption and ensuring data integrity.
sanlock uses the shared storage itself to hold lock state. The lock state is typically located on the same storage as the application data that the locks are protecting. A distributed application wishing to use sanlock for coordination will reserve space on the shared storage to hold the lock state, and pass the location for lock storage to sanlock.
A sanlock resource is an object which can be locked. It has a name and a location on disk. The resource exists as a 1MB area on shared storage, which holds the lock state for that resource. An application initializes a resource, giving it a name that corresponds to an application-specific entity. The application can acquire and release a lock on the resource to coordinate application activity. Within sanlock, the process of acquiring or releasing the lock involves reading and writing to the resource area on the shared storage. When sanlock acquires a lock on a resource, the local host ID is written as the lock owner in the first sector of the resource area.
A sanlock lockspace is a collection of resources, and a group of hosts that can acquire locks on them. The lockspace has a unique name which is referenced by each resource in the lockspace. The lockspace exists as a 1MB area on shared storage, typically preceding the resource areas (also 1MB each.) The lockspace area consists of 2000 special "host_id leases" (one sector each). This supports up to 2000 hosts using the lockspace. The lease for host_id N exists in sector N-1 of the lockspace disk area.
A lockspace also refers to a group of hosts with common access to the lockspace's shared storage, and using the lockspace's resources. Each host using the lockspace has a unique host name and a unique integer host ID. A host must "join" the lockspace before it can lock resources. The process of joining the lockspace involves acquiring a special "host_id lease" within the lockspace disk area. Acquiring the host_id lease involves writing the host name to the host_id lease. A host_id lease is renewed (once every 20 seconds by default). The host_id lease renewal applies to all resource locks held in the lockspace, i.e. renewing the host_id lease renews all resource locks. During renewal, a host also reads the state of other host_id leases in the lockspace. Monitoring lease renewals enables a host to determine if other hosts are alive.
A host is a machine with a unique name and host ID (an integer in the range 1-2000). The host ID is assigned by the application or user to each host. The host name can be configured by the user in sanlock.conf, otherwise it is taken from the product_uuid of the system. If a name is not assigned, and product_uuid is not available, sanlock generates a new random uuid for the host name each time sanlock is started.
To use sanlock, a host must run the sanlock(8) and wdmd(8) daemons, and have a connection to the shared storage holding lockspaces and resources. Applications use libsanlock to connect to the sanlock daemon, and make API calls to initialize lockspaces and resources, join and leave lockspaces, and acquire and release locks on resources.
When a host has joined a lockspace, and the sanlock daemon has begun renewing the lockspace host_id lease, the reported host state is "LIVE" for the given lockspace. The host's state in other lockspaces depends on whether the application on the host has joined them, and whether sanlock maintains access to the lockspace storage for renewals.
If a host stops renewing its host_id lease, its resource locks in that lockspace will expire, and sanlock will eventually allow those locks to be acquired by other hosts. When a host fails to renew its host_id lease for a period of time (id_renewal_fail_seconds, default 80 seconds), it is initially placed into the "FAIL" state. In this state, sanlock attempts to stop the application from using resources in the lockspace, so that the resource locks are no longer needed.
If the application shutdown is successful, resource locks and the host_id lease are dropped, and sanlock stops attempting to renew them. If not successful, the failed host will be fenced and then enter the "DEAD" state. When dead, a host's unreleased locks will be granted by sanlock on other hosts. sanlock includes built-in watchdog-based fencing, and also supports applications deploying other (often faster) fencing methods, e.g. removing persistent reservations.
Resource locks are implemented using two alternative methods, depending on the capabilities of the storage device:
The SCSI COMPARE AND WRITE (CAW) command. This special-purpose command, supported by certain storage devices, atomically updates a disk sector. sanlock uses this to atomically update the lock owner field in a resource. If multiple hosts attempt to become the lock owner simultaneously, the CAW command will succeed on only one of the hosts.
The Disk Paxos algorithm. All hosts which want to become the lock owner cooperatively determine, using a sequence of reads and writes, which host will become the new lock owner. The resource lock owner is updated with a new value in a way that all hosts recognize the same result.
Lockspace host_id leases are also implemented using two alternative methods:
The SCSI COMPARE AND WRITE (CAW) command.
A time-delay algorithm.
In theory, host_id leases in a lockspace should not be contended, because the host_id should be configured uniquely for all hosts. But, the algorithm for host_id lease ownership attempts to detect conflict in case of misconfiguration. The time delay method is particularly inefficient at this, which can cause long delays joining a lockspace. The CAW method has no delay.
Within sanlock, all resource locks are referred to as "paxos leases" (even when CAW is used), and all lockspace host_id leases are referred to as "delta leases" (regardless of which algorithm is used.)
When the shared storage supports the COMPARE AND WRITE (CAW) command (also known as atomic test and set), it is the most quick and efficient option, and is recommended. When CAW is not supported, the Disk Paxos algorithm is used. The use of CAW is determined when a resource or lockspace is initialized on disk by testing if the storage location for the lease supports CAW.
The amount of I/O required to acquire/release a resource lock depends on:
The I/O required for CAW and Disk Paxos (DP), exclusive and shared locks, assuming the lock is unowned and uncontended:
acquire ex CAW resource lock: read (1MB), ioctl (512b) release ex CAW resource lock: ioctl (512b) acquire sh CAW resource lock: read (1MB), ioctl (512b); write (512b); ioctl (512b) release sh CAW resource lock: write (512b) (ioctl: ioctl submits an SG_IO COMPARE AND WRITE command to a single sector. Two sectors of data, previous and next, are sent to the device.) acquire ex DP resource lock: read (1MB), write (512b), read (1MB), write (512b), read (1MB), write (512b) release ex DP resource lock: read (512b), write (512b) acquire sh DP resource lock read (1MB), write (512b), read (1MB), write (512b), read (1MB), write (512b); write (512b); read (512b), write (512b) release sh DP resource lock: write (512b)
sanlock commonly uses atomic reads and writes of disk sectors. Storage with either 512 or 4096 byte sectors is supported. For 512 byte sectors, resource and lockspace areas are always 1MB (supporting 2000 hosts). With 4096 byte sectors, resource and lockspace areas default to 8MB (also supporting 2000 hosts.) Discussion elsewhere referring to 1MB disk areas should be translated to 8MB in this case. The disk area size of a lockspace or resource is called "align_size" in sanlock interfaces and internally.
For disks with 4K sectors, applications can choose to initialize resource and lockspace areas as 1MB, 2MB, 4MB or 8MB. The reduced sizes of 1/2/4 MB limit the permitted host_id range (and the max number of hosts) to 250/500/1000.
Accepted combinations of sector size and align size, and the corresponding max_hosts (and max host_id) are:
sector_size 512, align_size 1M, max_hosts 2000 sector_size 4096, align_size 1M, max_hosts 250 sector_size 4096, align_size 2M, max_hosts 500 sector_size 4096, align_size 4M, max_hosts 1000 sector_size 4096, align_size 8M, max_hosts 2000
Using sanlock on shared block devices that do host based mirroring or replication is unlikely to work correctly.
sanlock can also be used on shared files, e.g. over NFS, but this is not the primary use case. When using sanlock on shared files, all sanlock io should go to one file server.
/dev/sdx --------------------------------------- Lockspace | Resource | Resource | ... | --------------------------------------- 0 1MB 2MB 3MB Lockspace ------------------------------------------------------------ | delta lease | delta lease | ... | delta lease | unused | | LS name FOO | LS name FOO | ... | LS name FOO | | | host_id 1 | host_id 2 | ... | host_id 2000 | | | generation 1 | generation 1 | ... | generation 0 | | | hostname A | hostname B | ... | hostname - | | | timestamp 44 | timestamp 25 | ... | timestamp 0 | | ------------------------------------------------------------- 0 512 ... 1023488 1MB Resource ------------------------------------------------------------- | paxos lease | | LS name FOO | | RS name FILE0 | | owner: host_id 1 generation 1 | ------------------------------------------------------------- 1MB 2MB Resource ------------------------------------------------------------- | paxos lease | | LS name FOO | | RS name FILE1 | | owner: host_id 2 generation 1 | ------------------------------------------------------------- 2MB 3MB
Lockspace "FOO" at the start of the shared disk is followed by two resources "FILE0" and "FILE1".
Two hosts: "A" using host_id 1 and "B" using host_id 2 have joined the lockspace.
In joining the lockspace, A and B have written their hostname to the host_id lease, and have begun updating the timestamp field in each renewal (current times on A/B were 44/25 at their last renewals.)
A has locked FILE0, and B has locked FILE1.
The following sanlock commands will produce the state shown in the diagram. These steps illustrate, using command line interfaces to libsanlock APIs, what an application would typically perform using the libsanlock APIs.
Shared disk /dev/sdx is attached to hosts A and B. host A initializes disk areas for one lockspace and two resources: $ sanlock init -s FOO:0:/dev/sdx:0 $ sanlock init -r FOO:FILE0:/dev/sdx:1M $ sanlock init -r FOO:FILE1:/dev/sdx:2M host A joins lockspace FOO using host_id 1: $ sanlock add_lockspace -s FOO:1:/dev/sdx:0 host B joins lockspace FOO using host_id 2: $ sanlock add_lockspace -s FOO:2:/dev/sdx:0 host A acquires lock on FILE0 (lock associated with the sleep process): $ sanlock command -r FOO:FILE0:/dev/sdx:1M -c /bin/sleep 10 host B acquires lock on FILE1 (lock associated with the sleep process): $ sanlock command -r FOO:FILE1:/dev/sdx:2M -c /bin/sleep 10 host A fails to acquire lock on FILE1 (owned by host B): $ sanlock command -r FOO:FILE1:/dev/sdx:1M -c /bin/sleep 10 host B fails to acquire lock on FILE0 (owned by host A): $ sanlock command -r FOO:FILE0:/dev/sdx:1M -c /bin/sleep 10 When the sleep processes exit, their associated locks (FILE0 and FILE1) are automatically released. (The sleep process is a trivial replacement for what would typically be a long running application process holding locks.) host A and B leave the lockspace: $ sanlock rem_lockspace -s FOO:1:/dev/sdx:0 $ sanlock rem_lockspace -s FOO:2:/dev/sdx:0
application process failure
If a process using sanlock exits, sanlock automatically drops the resource locks that the process was holding. This response assumes that if the process is not running, it can no longer write to the shared storage, so the locks it held are no longer required. However, if an application may be writing to shared storage from a process other than the one connected to sanlock, then PERSISTENT resource locks should be used, which sanlock will not automatically drop if the connection to the application process is closed.
sanlock daemon failure
If the sanlock daemon process exits abnormally, e.g. it crashes or is killed, it will no longer be renewing host_id leases in lockspaces that had been joined. To other hosts this will appear to be, and will be handled as, a host failure. The host with the failed sanlock daemon will be reset by its watchdog (since watchdog renewals are tied to lockspace lease renewals), and/or other hosts will apply a form of external fencing, e.g. removing persistent reservations from the host with the crashed/killed sanlock daemon.
wdmd daemon failure
sanlock uses the wdmd process to manage and update the local /dev/watchdog device. If the wdmd process exits abnormally, e.g. it crashes or is killed, it will no longer be updating the watchdog, which will lead to the host being reset by its watchdog. Other hosts wait until the watchdog has fired, or until external fencing has occurred, before considering the host dead.
host system freeze
If a host system freezes, sanlock and wdmd will no longer be updating the watchdog, causing the system to be reset by the watchdog. Other hosts wait until the watchdog expiration time, or until external fencing has occurred, before considering the host dead.
host system failure
If a host system fails, e.g. loss of power, other hosts cannot distinguish the failure from a host system freeze, and will wait for the watchdog expiration time, or for external fencing to occur, before considering the host dead.
renewal i/o failure
Renewing host_id leases in a lockspace (periodically writing a new timestamp to the lease) is how other hosts determine liveness. If renewal I/O fails to the host_id lease on the shared storage, the host will no longer appear to be alive. Other hosts cannot distinguish this condition from a host system failure or freeze.
Once sanlock has failed to renew a host_id lease for specific period of time (id_renewal_fail_seconds, default 80 seconds), it begins responding:
Graceful shutdown. sanlock will execute a "graceful shutdown" program if the application has specified one to be run. The shutdown program tells the application to shut down because its leases are expiring. The application must respond by stopping its activities and releasing its leases (or exit).
SIGTERM. If an application does not specify a graceful shutdown program, sanlock sends SIGTERM to the process instead. The process must release its leases or exit in a prescribed amount of time, or sanlock proceeds to the next method of stopping. (An application may specify that SIGTERM is not appropriate for shutdown and should not be tried.)
SIGKILL. sanlock sends SIGKILL to the process using the expiring leases. The process has a fixed amount of time to exit after receiving SIGKILL (note: a process could be stuck waiting for I/O in the kernel and not killable.) If it does not exit in time, sanlock will proceed to the next method. As with SIGTERM, this method may be disabled by applications for which SIGKILL is not appropriate, e.g. those that modify shared storage from processes other than the process connected to sanlock.
Watchdog reset. Eventually, if an application has not voluntarily released all of its locks, sanlock intentionally allows the local watchdog to do a forced reset of the host. sanlock carefully manages the timing of the watchdog device so that it fires shortly before any other host could take over the resource locks held by local processes.
External fencing. Other hosts may still use an external fencing method against a failed host. External fencing may finish faster than the watchdog reset, allowing faster takeover of locks/resources from a failed host. The watchdog reset can be optionally disabled with external fencing.
sanlock uses the wdmd(8) daemon to access /dev/watchdog. wdmd multiplexes multiple timeouts onto the single watchdog timer. This is required because host_id leases for each lockspace are renewed and expire independently.
sanlock maintains a wdmd connection for each lockspace host_id lease being renewed. Each connection has an expiry time for some seconds in the future. After each successful host_id lease renewal, the expiry time is renewed for the associated wdmd connection. If wdmd finds any connection expired, it will not renew the /dev/watchdog timer. Given enough successive failed renewals, the watchdog device will fire and reset the host. (Given the multiplexing nature of wdmd, shorter overlapping renewal failures from multiple lockspaces could cause spurious watchdog firing.)
The direct link between host_id lease renewals and watchdog renewals provides a predictable watchdog firing time based on host_id lease renewal timestamps that are visible from other hosts. sanlock knows the time the watchdog on another host has fired based on the host_id lease time. Furthermore, if the watchdog device on another host fails to fire when it should, the continuation of host_id lease renewals from the other host will make this evident and prevent leases from being taken from the failed host.
If sanlock is able to stop/kill all processing using an expiring lockspace, the associated wdmd connection for that lockspace is removed. The expired wdmd connection will no longer block /dev/watchdog renewals, and the host should avoid being reset.
External fencing allows an application to control when failed hosts are declared dead. When sanlock indicates that a host has entered the failed state (after missing renewals for id_renewal_fail_seconds), the application can detect this, and respond by performing a custom form of fencing that it manages itself.
The custom fencing method would protect the shared storage that the application is using (sanlock itself is not aware of all the shared storage that the application may be writing to, it is only aware of the shared storage used by the lockspace and resources.) One example of custom fencing is the application removing persistent reservations of the failed host.
After the application has successfully fenced the failed host, it notifies sanlock that the host is dead. sanlock records this in the host_id lease of the dead host (the DEAD_EXT flag), which in turn allows resource locks owned by the dead host to be granted by other hosts.
If the application wants to use external fencing only, and does not want the watchdog expiration to place a host into the DEAD state, it can pass the NO_TIMEOUT flag when initializing the lockspace. With this flag encoded in the lockspace host_id leases, the host_id leases will enter the FAIL state based on missing renewals, but the host will only enter the DEAD state based on application's "external dead" API call.
The sanlock timing parameters control host_id lease renewal frequency, failure detection time, and total recovery time. These parameters work together to balance responsiveness to failures (allowing other hosts to take over resources from failed hosts) against tolerance for temporary storage slowdowns (avoid resetting hosts that are temporarily slow.)
io_timeout
This is the maximum time allowed for a single I/O operation to complete. This includes I/O used to renew host_id leases, and I/O used to acquire resource locks. If I/O times out while acquiring a resource lock, an error is returned to the application's lock request. If I/O times out while renewing a host_id lease, the renewal I/O continues to be retried until successful or until the lockspace is removed.
The default io_timeout is 10 seconds. This value can be configured by the application when initializing a lockspace, or can be configured in sanlock.conf. The io_timeout should be set based on the expected I/O performance of the shared storage, including potential transient delays.
id_renewal_seconds
This is the interval between successful host_id lease renewals. It represents the maximum time that a single host_id lease renewal can take (when each I/O operation takes the maximum io_timeout).
id_renewal_seconds = 2 * io_timeout
With the default io_timeout of 10 seconds, id_renewal_seconds is 20 seconds. This is the typical interval seen between renewal timestamps written to disk when I/O completes quickly. Each host_id lease renewal consists of one read and one write.
If a renewal completes in less than id_renewal_seconds, sanlock will delay before beginning the next renewal attempt, so that renewals occur at approximately id_renewal_seconds intervals. If a renewal takes the full id_renewal_seconds to complete, the next renewal attempt will begin immediately.
id_renewal_fail_seconds
This is the time from the last successful host_id lease renewal until sanlock enters recovery mode and begins attempting to stop local processes using the expiring lockspace.
id_renewal_fail_seconds = 8 * io_timeout
With the default io_timeout of 10 seconds, id_renewal_fail_seconds is 80 seconds.
When sanlock has been unable to renew a host_id lease for id_renewal_fail_seconds, it begins recovery procedures: running graceful shutdown programs, sending SIGTERM/SIGKILL, and ultimately allowing the watchdog to reset the host if the application using the lockspace does not shut down in time.
watchdog_fire_timeout
This is the time the /dev/watchdog device will wait without being renewed before firing and resetting the host. The watchdog_fire_timeout is a constant, typically 60 seconds.
All hosts using sanlock must use the same watchdog_fire_timeout value. Mismatched values will break the protection provided by the watchdog.
Some watchdog devices support configurable timeouts. When using a configurable watchdog, common alternative values are 30 seconds and 10 seconds (typically paired with io_timeout values of 5 and 2 seconds respectively.)
host_dead_seconds
(Note: host_dead_seconds does not apply to host_id leases with NO_TIMEOUT set. Hosts using NO_TIMEOUT rely on an external host dead call from the application before they are considered dead. An external host dead indicator may often be faster than waiting for host_dead_seconds.)
This is the time from the last successful host_id lease renewal until other hosts can be certain that the host has been reset by its watchdog and is dead (no longer writing to storage).
host_dead_seconds = id_renewal_fail_seconds + watchdog_fire_timeout
With the default values of io_timeout 10 and watchdog_fire_timeout 60, host_dead_seconds is 140 seconds (80 + 60).
When a host attempts to acquire a resource lock that is held by another host, it checks the host_id lease of the owner host. If the host_id lease has not been renewed for host_dead_seconds, the resource lock is considered expired and can be acquired. Before this time, the resource lock remains owned by the other host, even if that host has stopped renewing its host_id lease.
https://lists.fedorahosted.org/admin/lists/sanlock-devel.lists.fedorahosted.org/
Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »
© Red Hat, Inc. and others.