Application leader election
This topic describes the process of building client-side leader elections for service instances by using Consul's session mechanism for building distributed locks and Consul KV, which is Consul's key/value datastore.
Tip
The content of this guide also applies to Consul clusters hosted on HashiCorp Cloud (HCP).
Note
This tutorial is not related to Consul's leader election. If you are interested in the leader election used internally by Consul, please refer to the consensus protocol documentation instead.
Background
Some distributed applications, like HDFS or ActiveMQ, require setting up one instance as a leader to ensure application data is current and stable.
Consul's support for sessions and watches allows you to build a client-side leader election process where clients use a lock on a key in the KV datastore to ensure mutual exclusion and to gracefully handle failures.
All service instances that are participating should coordinate on a key format. We recommend the following pattern:
service/<service name>/leader
Requirements
A running Consul server
A path in the Consul KV datastore to acquire locks and to store infromation about the leader. For this guide, the key is
service/leader
.If ACLs are enabled, a token with the following permissions:
session:write
permissions over the service session namekey:write
permissions over the agreed key for this topic.
You expose the token using the
CONSUL_HTTP_TOKEN
environment variable.The
curl
command
Client-side leader election procedure
The workflow for building a client-side leader election process has the following steps:
For each client trying to acquire the lock:
- Create a session associated with the client node.
- Try to acquire the lock on the agreed key in the KV store using the
acquire
parameter. - Watch the KV key to verify if the lock was released and if no lock is present, try to acquire a lock.
For the client that acquires the lock
- Periodically, renew the session to avoid expiration.
- Optionally, release the lock.
For other services
- Watch the KV key to verify there is at least one process holding the lock.
- Use the values written under the KV path to identify the leader and update configuration accordingly.
Create a new session
Create a configuration for the session. The minimum viable configuration is shown in the following example and only requires that you specify the session name.
{
"Name": "session_name"
}
Create a session using the Session HTTP API. In this example, the node's hostname
is the session name.
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--data '{"Name": "'`hostname`'"}' \
--request PUT \
http://127.0.0.1:8500/v1/session/create | jq
The command returns a JSON object containing the ID of the newly created session:
{
"ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6"
}
Verify created session info
Use the /v1/session/list
endpoint to retrieve existing sessions.
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--request GET \
http://127.0.0.1:8500/v1/session/list | jq
The command returns a JSON array containing all available sessions in the system.
[
{
"ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6",
"Name": "hashicups-db-0",
"Node": "hashicups-db-0",
"LockDelay": 15000000000,
"Behavior": "release",
"TTL": "",
"NodeChecks": [
"serfHealth"
],
"ServiceChecks": null,
"CreateIndex": 11956,
"ModifyIndex": 11956
}
]
You can verify from the output that the session is associated with the hashicups-db-0
node, which is the client agent where the API request was made.
With the exception of the Name
parameter, all other settings are left to the default value. The session is created without a TTL
value, which means that it never expires and requires an explicit deletion.
Depending on your needs you can create sessions specifying more parameters such as:
TTL
- If provided, the session is invalidated and deleted if it is not renewed before the TTL expires.ServiceChecks
- Specifies a list of service checks to monitor. The session is invalidated if the checks get into a critical state.
By setting these extra parameters, you can create a client-side leader election workflow that automatically releases the lock after a certain period of time after the latest renew or that automatically releases locks when the service holding them fails.
For a full list of parameters available refer to the Create Session API documentation.
Acquire the lock
Create the data object to associate to the lock request.
The data of the request should be a JSON object representing the local instance. This value is opaque to Consul, but it should contain whatever information clients require to communicate with your application. For example, it could be a JSON object that contains the node's name and the application's port.
{
"Node": "node-name",
"Port": "8080"
}
Acquire a lock for a given key using the PUT method on a KV entry with the
?acquire=<session>
query parameter.
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--data '{"Node": "'`hostname`'"}' \
--request PUT \
http://localhost:8500/v1/kv/service/leader?acquire=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq
This either returns true
or false
. If true
, the lock has been acquired and
the local service instance is now the leader. If false
is returned, some other node has acquired
the lock.
In the example, you used the node's hostname
as the key data. This can be used by the other services to create the configuration files.
Note
This locking system is purely advisory. There is no enforcement that clients must acquire a lock to perform any operation. Any client can read, write, and delete a key without owning the corresponding lock. It is not the goal of Consul to protect against misbehaving clients.
Watch the KV key for locks
Existing locks need to be monitored by all nodes involved in the client-side leader elections, as well as by the other nodes that need to know the identity of the leader.
- Lock holders need to monitor the lock because the session might get invalidated by an operator.
- Other services that want to acquire the lock need to monitor it to check if the lock is released so they can try acquire the lock.
- Other nodes need to monitor the lock to see if the value of the key changed and update their configuration accordingly.
Monitor the lock using the GET method on a KV entry with the blocking query enabled.
First, verify the latest index for the current value.
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--request GET \
http://127.0.0.1:8500/v1/kv/service/leader?index=1 | jq
The command outputs the key data, including the ModifyIndex
for the object.
[
{
"LockIndex": 0,
"Key": "service/leader",
"Flags": 0,
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=",
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6",
"CreateIndex": 12399,
"ModifyIndex": 13061
}
]
Using the value of the ModifyIndex
, run a blocking query against the lock.
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--request GET \
http://127.0.0.1:8500/v1/kv/service/leader?index=13061 | jq
The command hangs until a change is made on the KV path and after that the path data prints on the console.
[
{
"LockIndex": 0,
"Key": "service/leader",
"Flags": 0,
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0xIn0=",
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6",
"CreateIndex": 12399,
"ModifyIndex": 13329
}
]
For automation purposes, add logic to the blocking query mechanism to trigger a command every time a change is returned.
A better approach is to use the CLI command consul watch
.
From the output, notice that once the lock is acquired, the Session
parameter contains the ID of the session that holds the lock.
Renew a session
If a session is created with a TTL
value set, you need to renew the session before the TTL expires.
Use the /v1/session/renew
endpoint to renew existing sessions.
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--request PUT \
http://127.0.0.1:8500/v1/session/renew/f027470f-2759-6b53-542d-066ae4185e67 | jq
If the command succeeds, the session information in JSON format is printed.
[
{
"ID": "f027470f-2759-6b53-542d-066ae4185e67",
"Name": "test",
"Node": "consul-server-0",
"LockDelay": 15000000000,
"Behavior": "release",
"TTL": "30s",
"NodeChecks": [
"serfHealth"
],
"ServiceChecks": null,
"CreateIndex": 11842,
"ModifyIndex": 11842
}
]
Release a lock
A lock associated with a session with no TTL
value set might never be released, even when the service holding it fails.
In such cases, you need to manually release the lock.
$ curl --silent \
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
--data '{"Node": "'`hostname`'"}' \
--request PUT \
http://localhost:8500/v1/kv/service/leader?release=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq
The command prints true
on success.
After a lock is released, the key data do not show a value for Session
in the results.
Other clients can use this as a way to coordinate their lock requests.