Hermes operators guide to using metrics

This section is a basic guide on how Hermes metrics can be used to observe both the current state of the Hermes relayer and the networks it is connected to.

General remarks about the metrics

• All Hermes metrics are tracked and updated from the moment the Hermes service (i.e., start) starts up. Metrics are automatically reset if the service is restarted.
• For maximum reliability, it is advised to combine monitoring of your Hermes service with monitoring of your full nodes.
• Some metrics require specific configurations to be enabled, this is described in the Configuration Dependencies column.

Table of Contents

Hermes' metrics are designed to be able to answer four basic questions:

1. Is Hermes active (i.e., submitting any transactions to any network)?
2. Are Hermes transactions successful (i.e., confirmed and included in the network)?
3. What is the overall IBC status of each network?
4. How efficient, and how secure is the IBC status on each network?
5. Am I getting fee rewards from ICS29 incentivized packets?

For each of this question, there is a dedicated subsection:

• Is Hermes active?
• Are Hermes transactions successful?
• What is the overall IBC status of each network?
• How efficient and how secure is the IBC status on each network?
• Am I getting fee rewards?

Is Hermes active?

By active, we mean specifically: is Hermes submitting any transactions to any network? The metrics in the table below are design to answer this question on multiple dimensions.

Notes & more details below:

What is a worker?

• A worker is a separate thread of execution and there are five types of workers:
  • Client: The worker that refreshed a client periodically and detects misbehaviour.
  • Connection: The worker that handles connection open handshake that may be incomplete.
  • Channel: The worker that handles channel open handshake that may be incomplete.
  • Packet: The worker that handles packet relaying.
  • Wallet: The worker that periodically queries for the balance of each wallet that Hermes is using and updates wallet_balance metric.
• For example, if your metrics show that you have 0 packet workers (workers{type="packet"} 0), that is a clear indication that Hermes is not relaying any packets at the moment.

How do we define the latency of a submitted transaction? The latency is defined as the difference between the moment when Hermes received an event (through the websocket) until the moment when the corresponding transaction(s) were submitted into a full node's mempool.

• If a transaction is submitted it does not mean it was confirmed, see below for more details.
• This metric is tracked per chain, counterparty chain, channel and port.

A note on wallet balances. For the wallet_balance, we convert from a String into a f64, which can lead to a loss in precision in the displayed value.

latency histogram The tx_latency_submitted and tx_latency_confirmed are displayed with histogram buckets which each contain the number of values less or equal to their bucket label. This means that if there are 5 buckets with label 500, 2000, 3000, 4000 and 5000 and 2 tx_latency_submitted were recorded of respectively 1800ms and 3100ms then the tx_latency_submitted will look like this:

tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="500"} 0
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="2000"} 1
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="3000"} 1
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="4000"} 2
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="5000"} 2
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="+Inf"} 2

The range of the buckets can be configured using the latency_submitted and latency_confirmed seen here

Are Hermes transactions successful?

This table shows the metrics for Hermes performance. Importantly, these metrics are only displayed if the configuration tx_confirmation = true is set in your config.toml.

How do we define the latency of a confirmed transaction? This is the difference between the moment when Hermes received an event until the corresponding transaction(s) were confirmed.

• Similarly to tx_latency_submitted, this metrics is tracked per chain, counterparty chain, channel and port.
• This metrics usually contains strictly larger values than tx_latency_submitted, because Hermes first submits transactions into the network's mempool, and then it takes some more time elapses until the network includes those transactions in a block.

What is the overall IBC status of each network?

These metrics are not specific to your Hermes instance. These are metrics that capture the activity of all IBC relayers.

‼️ Important: Your Hermes instance produces these metrics based on the events it receives via a websocket to the full nodes of each network. If these events are not being updated, that is a good indication that either:

• The network has no IBC activity, or
• The websocket connection to that network is broken.

Notes:

• Except for ws_reconnect_total, all these metrics should typically increase regularly in the common-case. That is an indication that the network is regularly producing new blocks and there is ongoing IBC activity, eg send_packet, acknowledgment, and timeout.
• The metric ws_reconnect_total signals that the websocket connection was broken and Hermes had to re-establish that. It is usually an indication that your full node may be falling behind or is experiencing instability.

Since Hermes v1, we also introduced 3 metrics that sketch the backlog status of IBC relaying.

Notes:

• The backlog_size defines how many IBC packets users sent and were not yet relayed (i.e., received on the destination network, or timed-out). If this metric is increasing, it signals that the packet queue is increasing and there may be some errors in the Hermes logs that need your attention.
• If the backlog_oldest_sequence remains unchanged for more than a few minutes, that means that the packet with the respective sequence number is likely blocked and cannot be relayed. To understand for how long the packet is block, Hermes will populate backlog_oldest_timestamp with the local time when it first observed the backlog_oldest_sequence that is blocked.

How efficient and how secure is the IBC status on each network?

Notes:

• The two metrics cleared_send_packet_count_total and cleared_acknowledgment_count_total are only populated if tx_confirmation = true. These two metrics usually correlate with backlog_* metrics. They are an indication that IBC packet relaying may be unsuccessful and that Hermes periodically finds packets to clear (i.e., unblock).
• queries_total and queries_cache_hits_total values are complementary. For the total number of queries, the two metrics should be summed for a specific query type.

For security, we only expose one metric, described in the table below. Note that this metrics is disabled if misbehaviour = false in your Hermes config.toml.

Am I getting fee rewards?