Read Consistency

You do not need to know the information on this page to use rqlite well, it’s mostly for advanced users. rqlite has also been run through Jepsen-style testing. You can read about that here.

Even though serving queries does not require Raft consensus (because the database is not changed), queries should generally be served by the cluster Leader. Why is this? Because, without this check, queries on a node could return results that are out-of-date i.e. stale. This could happen for one, or both, of the following two reasons:

• The node that received your request , while still part of the cluster, has fallen behind the Leader in terms of updates to its underlying database.
• The node is no longer part of the cluster, and has stopped receiving Raft log updates.

This is why rqlite offers selectable read consistency levels of weak (the default), strong, and none. Each is explained below, and examples of each are shown at the end of this page.

Weak

Weak consistency is used if you don’t specify any level, or if an unrecognized level is specified – and it’s probably the right choice for your application.

Weak instructs the node receiving the read request to check that it is the Leader, before querying the local SQLite file. Checking Leader state only involves checking state local to the node, so is very fast. There is, however, a very small window of time (milliseconds by default) during which the node may return stale data if a Leader-election is in progress. This is because after the local Leader check, but before the local SQLite database is read, another node could be elected Leader and make changes to the cluster. As result the node may not be quite up-to-date with the rest of cluster.

If the node determines it is not the Leader, the node will transparently forward the request to the Leader, which will in turn perform a Weak read of its database. The node then waits for the response from the Leader, and then returns that response to the client.

Strong

To avoid even the issues associated with weak consistency, rqlite also offers strong. In this mode, the node receiving the request sends the query through the Raft consensus system, ensuring that the cluster Leader remains the Leader at all times during the processing of the query. When using strong you can be sure that the database reflects every change sent to it prior to the query. However, this will involve the Leader contacting at least a quorum of nodes, and will therefore increase query response times.

If a query request is sent to a Follower, and strong consistency is specified, the Follower will transparently forward the request to the Leader. The Follower waits for the response from the Leader, and then returns that response to the client.

None

With none, the node receiving your read request simply queries its local SQLite database, and does not perform any Leadership check – in fact, the node could be completely disconnected from the rest of the cluster, but the query will still be successful. This offers the fastest query response, but suffers from the potential issues outlined above, whereby there is a chance of Stale Reads if the Leader changes during the query, or if the node has become disconneted from the cluster.

Limiting read staleness

You can tell the receiving node not to return results if the node has been disconnected from the cluster for longer than a specified duration. If a read request sets the query parameter freshness to a Go duration string, the node serving the read will check that less time has passed since it was last in contact with the Leader, than that specified via freshness. If more time has passed the node will return an error. This approach can be useful if you want to maximize successful query operations, but are willing to tolerate occassional, short-lived networking issues between nodes.

It’s important to node that the freshness does not guarantee that the node is caught up with the Leader, only that it is contact with the Leader. But if a node is in contact with the Leader, it’s usually caught up with all changes that have taken place on the cluster. To check if node is actually caught up with Leader, use freshness_strict in addition to the freshness query parameter.

The freshness parameter is always ignored if the node serving the query is the Leader. Any read, when served by the Leader, is always going to be within any possible freshness bound. freshness is also ignored for all consistency levels except none, and is also ignored if set to zero.

If you decide to deploy read-only nodes however, none combined with freshness can be a particularly effective at adding read scalability to your system. You can use lots of read-only nodes, yet be sure that a given node serving a request has not been disconnected from the cluster for too long.

freshness_strict

As explained above freshness just checks that that the node has been in contact with the Leader within the specified time. If you also want the node to check that the data it last received is not out-of-date by (at most) the freshness interval, you should also add the freshness_strict flag as a query parameter. Note that this check works by comparing timestamps generated by the Leader to those generated by the node receiving the read request. Any clock skew between the nodes may therefore affect the correctness of the data returned by the node. You are responsible for controlling the amount of clock skew across your rqlite cluster.

See the examples at the end of this page to learn how to control freshness.

Which should I use?

Weak is almost certainly sufficient for your application, and is the default read consistency level. Unless your cluster Leader is continually changing while you’re actually executing queries there will be never be any difference between weak and strong – but using strong will result in much slower queries, which is not what most people want.

How do I specify read consistency?

To explicitly select consistency, set the query param level to the desired level. However, you should use none with read-only nodes, unless you want those nodes to actually forward the query to the Leader.

Example queries

Examples of enabling each read consistency level for a simple query is shown below.

# Default query options. The read request will be served by the node if it believes
# it is the leader, otherwise it transparently forwards the request to the Leader, and
# waits for a response from the Leader. Same as weak.
curl -G 'localhost:4001/db/query' --data-urlencode 'q=SELECT * FROM foo'

# The read request will be served by the node if it believes it is the Leader,
# otherwise it wil forward the request to the Leader. This is the default if
# no read consistency is specified.
curl -G 'localhost:4001/db/query?level=weak' --data-urlencode 'q=SELECT * FROM foo'

# Query the node, telling it simply to read the SQLite database directly.
# No guarantees on how old the data is. In fact, the node may not even be
# connected to the cluster. Provides the fastest possible query response.
curl -G 'localhost:4001/db/query?level=none' --data-urlencode 'q=SELECT * FROM foo'

# Query the node, telling it simply to read the SQLite database directly.
# The read request will be successful only if the node last heard from the
# Leader no more than 1 second ago. This provides very fast reads, but sets
# an upper bound of 1 second on how long the node may have been disconnected
# from the cluster.
curl -G 'localhost:4001/db/query?level=none&freshness=1s' --data-urlencode 'q=SELECT * FROM foo'

# Query the node, telling it simply to read the SQLite database directly.
# The read request will be successful only if the node last heard from the
# Leader no more than 1 second ago, and if the most recently received data
# was appended by the Leader to the log within 1 second ago, relative to the
# node's local clock.
curl -G 'localhost:4001/db/query?level=none&freshness=1s&freshness_strict' --data-urlencode 'q=SELECT * FROM foo'

# The read request will be processed by the Leader and will be successful
# only if the Leader maintained leadership during the entirety of query
# processing. Zero chance of stale reads but query processing will be
# relatively slow. If the node receiving the query is not the the Leader,
# the request will be transparently forwarded to the Leader.
curl -G 'localhost:4001/db/query?level=strong' --data-urlencode 'q=SELECT * FROM foo'