Monitoring rqlite

How to monitor rqlite

Status API

rqlite serves diagnostic and statistical information, as well as basic information about the underlying Raft system, at /status. Assuming the rqlite node is started with default settings you can issue a curl command like so to retrieve this information:

curl localhost:4001/status?pretty

The use of the URL param pretty is optional, and results in pretty-printed JSON responses. The output of this endpoint could be periodically written to a monitoring system, allowing the performance of rqlite to be tracked over time.

You can also request the same status information via the rqlite shell:

$ ./rqlite 
Welcome to the rqlite CLI. Enter ".help" for usage hints.
127.0.0.1:4001> .status
build:
  build_time: unknown
  commit: unknown
  version: 5
  branch: unknown
http:
  addr: 127.0.0.1:4001
  auth: disabled
  redirect: 
node:
  start_time: 2019-12-23T22:34:46.215507011-05:00
  uptime: 16.963009139s
runtime:
  num_goroutine: 9
  version: go1.13

Nodes API

The nodes API returns basic information for nodes in the cluster, as seen by the node receiving the nodes request. The receiving node will also check whether it can actually connect to all other nodes in the cluster. This is an effective way to determine the cluster leader, and the leader’s HTTP API address. It can also be used to check if the cluster is basically running – if the other nodes are reachable, it probably is.

By default, the node only checks if voting nodes are contactable.

curl localhost:4001/nodes?pretty

# Also check non-voting nodes.
curl localhost:4001/nodes?nonvoters&pretty

# Give up if all nodes don't respond within 5 seconds. Default is 30 seconds.
curl localhost:4001/nodes?timeout=5s

You can also request the same nodes information via the rqlite shell:

$ ./rqlite
Welcome to the rqlite CLI. Enter ".help" for usage hints.
127.0.0.1:4001> .nodes
1:
  api_addr: http://localhost:4001
  addr: 127.0.0.1:4002
  reachable: true
  leader: true
2:
  api_addr: http://localhost:4003
  addr: 127.0.0.1:4004
  reachable: true
  leader: false
3:
  api_addr: http://localhost:4005
  addr: 127.0.0.1:4006
  reachable: true
  leader: false

Readiness checks

rqlite nodes serve a “ready” status at /readyz. The endpoint will return HTTP 200 OK if the node is ready to respond to database requests and cluster management operations. An example access is shown below.

$ curl localhost:4001/readyz
[+]node ok
[+]leader ok

Note that a subsequent request to the cluster may still fail even if /readyz returns HTTP 200 OK. This is because there is a window of time (determined by the Raft-related timings configured at start-up) when a node may still be able to contact the Leader, but the Leader has not yet detected that the cluster cannot reach consensus due to failed Followers. This window of time is quite small (less than 1 second) by default, however.

If you wish to check if the node is running, and responding to HTTP requests, regardless of Leader status, add noleader to the URL. This form may be more useful for automated deployments, which simply need to know if the node is responsive.

$ curl localhost:4001/readyz?noleader
[+]node ok

expvar support

rqlite also exports expvar information, which are mostly counters of various rqlite activity. The standard expvar information, as well as some custom information, is exposed. This data can be retrieved like so (assuming the node is started in its default configuration):

curl localhost:4001/debug/vars

You can optionally set the query parmameter key if you wish to retrieve just a subsection of the expvar output e.g. the URL localhost:4001/debug/vars?key=http will return just HTTP information.

You can also request the same expvar information via the CLI:

$ rqlite
127.0.0.1:4001> .expvar
cmdline: [./rqlited data]
db:
  execute_transactions: 0
  execution_errors: 1
  executions: 1
  queries: 0
  query_transactions: 0
http:
  backups: 0
  executions: 0
  queries: 0
memstats:
  Mallocs: 8950
  HeapSys: 2.588672e+06
  StackInuse: 557056
  LastGC: 0...

Similar to the status output, the output of this endpoint could be periodically written to a monitoring system, allowing the performance of rqlite to be tracked over time.

pprof support

pprof information is available by default and can be accessed as follows:

curl localhost:4001/debug/pprof/cmdline
curl localhost:4001/debug/pprof/profile
curl localhost:4001/debug/pprof/symbol
Last modified January 6, 2023: Fix typos (9ae80d6)