Hermes Guide (v0.15.0)

Hermes is a an open-source Rust implementation of a relayer for the Inter-Blockchain Communication protocol (IBC).

This guide can help you setup, configure, and operate Hermes to transfer packets between two or more IBC-enabled chains.

Sections

What is Hermes

• Explains what Hermes is about.

Features

• This section discusses what features to expect from Hermes, as well as a comparison between the Cosmos Go relayer and Hermes.

Getting Started

• The getting started section can help you setup, configure, and run Hermes.

Tutorials

• This section provides some tutorials on how to operate and test Hermes.

Commands Reference

• The commands let you interact with Hermes using its command line interface.

Help

• This part provides guidelines regarding troubleshooting and general resources for getting help.

Glossary

• This section provides some definitions of terms used throughout the guide

Other References and Useful Links:

• Hermes Github repository — The official Github repository for Hermes.
• IBC Github repository
  • The official repository for the Inter-blockchain protocol (IBC).

Disclaimer

This project is undergoing heavy development, use at your own risk.

What is Hermes?

Hermes is an open-source Rust implementation of a relayer for the Inter-Blockchain Communication protocol (IBC), released under the ibc-relayer-cli crate.

The Inter-Blockchain Communication protocol is an end-to-end, connection-oriented, stateful protocol for reliable, ordered, and authenticated communication between modules on separate distributed ledgers. ¹

An IBC relayer is an off-chain process responsible for relaying IBC datagrams between any two chains. The way it does so is by scanning chain states, building transactions based on these states, and submitting the transactions to the chains involved in the network.

The relayer is a central element in the IBC network architecture. This is because chain modules in this architecture are not directly sending messages to each other over networking infrastructure, but instead they create and store the data to be retrieved and used by a relayer to build the IBC datagrams.

We sometimes refer to Hermes as "IBC Relayer CLI", to make it clear that this is a relayer CLI (i.e., a binary) and distinguish it from the relayer core library (that is the crate called ibc-relayer).

Hermes is actively developed and maintained by Informal Systems in the ibc-rs repository.

Features

This section includes a summary of the supported and planned features. A feature matrix and comparison between the Rust and Go relayer implementations can be found in the Feature Matrix

Cosmos SDK compatibility: Hermes supports Cosmos SDK chains implementing the IBC v1.1 protocol specification. Cosmos SDK versions 0.41.3 to 0.44.x are officially supported. In case Hermes finds an incompatible SDK version, it will output a log warning.

Supported Features

• Basic features
  • create and update clients
  • refresh clients to prevent expiration
  • establish connections with new or existing clients
  • establish channels with new or existing connection
  • channel closing handshake
  • relay packets, acknowledgments, timeout and timeout-on-close packets, with zero or non-zero delay.
  • queries for all objects
• Packet relaying over:
  • multiple paths, for the chains in config.toml
• Restart support
  • clear packets
  • resume channel handshake if configured to relay all
  • resume connection handshake if configured to relay all
• Client upgrade
  • upgrading clients after a counterparty chain has performed an upgrade for IBC breaking changes
• Packet delay:
  • establish path over non-zero delay connection
  • relay all packets with the specified delay
• Monitor and submit misbehaviour for clients
  • monitor client updates for misbehaviour (fork and BFT time violation)
  • submit misbehaviour evidence to the on-chain IBC client.

  misbehaviour submission to full node not yet supported

• Individual commands that build and send transactions for:
  • creating and updating IBC Tendermint light clients
  • sending connection open handshake datagrams
  • sending channel open handshake datagrams
  • sending channel closing handshake datagrams
  • initiating a cross chain transfer (mainly for testing)
  • relaying sent packets, acknowledgments and timeouts
  • client upgrade
• Channel handshake for existing channel that is not in Open state
• Connection handshake for existing connection that is not in Open state
• Telemetry support

Upcoming / Unsupported Features

Planned features:

• Full Passive mode: relay from all IBC events
  • Connection handshake for existing connection that is not in Open state
• Relayer support for management application (add RPC server)
• Dynamic configuration management

Not planned:

• Relayer management application
• Create clients with user chosen parameters (such as UpgradePath)
• Use IBC light clients other than Tendermint such as Solo Machine
• Support non cosmos-SDK chains

Feature Matrix

This section gives more details about the features and implementation status of Hermes in comparison with the cosmos-go-relayer.

Legend:

Feature comparison between Hermes and the Go relayer

Getting Started

In order to run Hermes, please make sure you have all the pre-requisites installed on your machine.

Once you have these pre-requisites, you can build and run Hermes.

The instructions in this guide have been tested on Linux and MacOS environments. Most of the commands should work on both environments. Even though you can build and run the relayer on Windows (since we develop it in Rust and it supports cross platform compilation) we have not tested the relayer on Windows and we do not support this operating system at this time.

Pre-requisites

1. Rust

The IBC Relayer is developed with the Rust programming language. In order to build and run the relayer you need to install and configure Rust on your machine.

Fresh Rust installation

For instructions on how to install Rust on your machine please follow the official Notes about Rust Installation.

The provided instructions will install all the Rust toolchain including rustc, cargo, and rustup that are required to build the project.

Version requirements

Hermes is developed and tested using the latest version of Rust, 1.60 at the moment. To check that your toolchain is up-to-date run:

rustc --version

In case you already had installed the Rust toolchain in the past, you can update your installation by running rustup update.

Testing the installation

After you install the Rust toolchain you can execute the following command:

cargo version

This should display the cargo version and confirm the proper installation.

2. Golang

You will also need the Go programming language installed and configured on your machine. This is a requirement for the the section Installing Gaia in the Two Local Chains tutorial.

To install and configure Golang on your machine please follow the Golang official documentation.

Next Steps

Next, go to the Installation section to learn how to build Hermes.

Install the relayer

There are two main approaches for obtaining Hermes:

1. Installation:

   1. If you are running on a Unix machine (Linux/MacOS), then the simplest option is to download the latest binary.
   2. You can also install via Cargo.

2. Alternatively, build Hermes directly from source.

Install by downloading

Simply head to the GitHub Releases page and download the latest version of Hermes binary matching your platform:

• MacOS: hermes-v0.15.0-x86_64-apple-darwin.tar.gz (or .zip),
• Linux: hermes-v0.15.0-x86_64-unknown-linux-gnu.tar.gz (or .zip).

The step-by-step instruction below should carry you through the whole process:

1. Make the directory where we'll place the binary:

   mkdir -p $HOME/.hermes/bin
   

2. Extract the binary archive:

   tar -C $HOME/.hermes/bin/ -vxzf $ARCHIVE_NAME
   

3. Update your path, by adding this line in your .bashrc or .zshrc shell configuration file:

   export PATH="$HOME/.hermes/bin:$PATH"
   

NOTE: The binary may be initially prevented from running if you're on MacOS. See the "Open Anyway" instructions from this support forum if that is the case.

You should now be able to run Hermes by invoking the hermes executable.

hermes version

hermes 0.15.0

Install via Cargo

NOTE: This approach assumes you have installed all the pre-requisites on your machine.

Hermes is packaged in the ibc-relayer-cli Rust crate. To install the latest release of Hermes, run the following command in a terminal:

cargo install ibc-relayer-cli --bin hermes --locked

This will download and build the crate ibc-relayer-cli, and install the hermes binary in $HOME/.cargo/bin.

If you have not installed Rust and Cargo via rustup.rs, you may need to add the $HOME/.cargo/bin directory to your PATH environment variable. For most shells, this can be done by adding the following line to your .bashrc or .zshrc configuration file:

export PATH="$HOME/.cargo/bin:$PATH"

You should now be able to run Hermes by invoking the hermes executable.

hermes version

hermes 0.15.0

Build from source

Clone the repository

Open a terminal and clone the ibc-rs repository:

git clone https://github.com/informalsystems/ibc-rs.git

Change to the repository directory

cd ibc-rs

Checkout the latest release

Go to the ibc-rs releases page to see what is the most recent release.

Then checkout the release, for example if the most recent release is v0.15.0 then execute the command:

git checkout v0.15.0

Building with cargo build

This command builds all the crates from the ibc-rs repository, namely: the ibc modules crate, ibc-relayer crate, ibc-proto crate, and the ibc-relayer-cli crate. The last of these crates contains the hermes binary.

cargo build --release --bin hermes

By default, Hermes bundles a telemetry service and server. To build Hermes without telemetry support, and get a smaller executable, supply the --no-default-features flag to cargo build:

cargo build --release --no-default-features --bin hermes

If the build is successful, the hermes executable will be located in the following location:

./target/release/hermes

Troubleshooting: In case the cargo build command above fails, as a first course of action we recommend trying to run the same command with the additional locked flag:

cargo build --release --bin hermes --locked

Running for the first time

If you run the hermes without any additional parameters you should see the usage and help information:

./target/release/hermes

hermes 0.15.0
Informal Systems <hello@informal.systems>

USAGE:
    hermes <SUBCOMMAND>

SUBCOMMANDS:
    help       Get usage information
    config     Validate Hermes configuration file
    keys       Manage keys in the relayer for each chain
    create     Create objects (client, connection, or channel) on chains
    update     Update objects (clients) on chains
    upgrade    Upgrade objects (clients) after chain upgrade
    start      Start the relayer
    query      Query objects from the chain
    tx         Create and send IBC transactions
    listen     Listen to and display IBC events emitted by a chain
    misbehaviour Listen to client update IBC events and handles misbehaviour
    version    Display version information

Creating an alias for the executable

It might be easier to create an alias for hermes so you can just run it by specifying the executable name instead of the whole path. In order to create an alias execute the following command:

alias hermes='cargo run --release --bin hermes --'

Shell auto-completions

The completions subcommand of Hermes can be used to output a completion script for a choice of widely used command-line shells. Refer to hermes completions --help for the list. Some shell-specific examples of setting up auto-completion with this command are provided below; check your shell configuration to decide on the suitable directory in which to install the script and any further necessary modifications to the shell's startup files.

Bash

hermes completions bash > ~/.local/share/bash-completion/completions/hermes

On a MacOS installation with Homebrew bash-completion formula installed, use

hermes completions bash > $(brew --prefix)/etc/bash_completion.d/hermes.bash-completion

Zsh

hermes completions zsh > ~/.zfunc/_hermes

To make the shell load the script on initialization, add the directory to fpath in your ~/.zshrc before compinit:

fpath+=~/.zfunc

Next Steps

Go to the Configuration section to learn how to create a configuration file to be used by Hermes.

Configuration

In order to run Hermes, you will need to have a configuration file.

The format supported for the configuration file is TOML.

By default, Hermes expects the configuration file to be located at $HOME/.hermes/config.toml.

This can be overridden by supplying the -c flag when invoking hermes, before the name of the command to run, eg. hermes -c my_config.toml query connection channels ibc-1 connection-1.

The current version of Hermes does not support managing the configuration file programmatically. You will need to use a text editor to create the file and add content to it.

hermes [-c CONFIG_FILE] COMMAND

Table of contents

• Configuration
• Adding private keys
• Connecting via TLS
• Support for Interchain Accounts
• Update the configuration without restarting Hermes
• Next steps

Configuration

The configuration file must have one global section, and one chains section for each chain.

Note: As of 0.6.0, the Hermes configuration file is self-documented. Please read the configuration file config.toml itself for the most up-to-date documentation of parameters.

By default, Hermes will relay on all channels available between all the configured chains. In this way, every configured chain will act as a source (in the sense that Hermes listens for events) and as a destination (to relay packets that others chains have sent).

For example, if there are only two chains configured, then Hermes will only relay packets between those two, i.e. the two chains will serve as a source for each other, and likewise as a destination for each other's relevant events. Hermes will ignore all events that pertain to chains which are unknown (ie. not present in config.toml).

To restrict relaying on specific channels, or uni-directionally, you can use packet filtering policies.

Adding private keys

For each chain configured you need to add a private key for that chain in order to submit transactions, please refer to the Keys sections in order to learn how to add the private keys that are used by the relayer.

Connecting via TLS

Hermes supports connection via TLS for use-cases such as connecting from behind a proxy or a load balancer. In order to enable this, you'll want to set the rpc_addr, grpc_addr, or websocket_addr parameters to specify a TLS connection via HTTPS using the following scheme (note that the port number 443 is just used for example):

rpc_addr = 'https://domain.com:443'
grpc_addr = 'https://domain.com:443'
websocket_addr = 'wss://domain.com:443/websocket'

Support for Interchain Accounts

As of version 0.13.0, Hermes supports relaying on Interchain Accounts channels.

If the packet_filter option in the chain configuration is disabled, then Hermes will relay on all existing and future channels, including ICA channels.

There are two kinds of ICA channels:

1. The host channels, whose port is icahost
2. The controller channels, whose port starts with icacontroller- followed by the owner account address. See the spec for more details.

If you wish to only relay on a few specific standard channels (here channel-0 and channel-1), but also relay on all ICA channels, you can specify the following packet filter:

Note the use of wildcards in the port and channel identifiers (['ica*', '*']) to match over all the possible ICA ports.

[chains.packet_filter]
policy = 'allow'
list = [
  ['ica*', '*'], # allow relaying on all channels whose port starts with `ica`
  ['transfer', 'channel-0'],
  ['transfer', 'channel-1'],
  # Add any other port/channel pairs you wish to relay on
]

If you wish to relay on all channels but not on ICA channels, you can use the following packet filter configuration:

[chains.packet_filter]
policy = 'deny'
list = [
  ['ica*', '*'], # deny relaying on all channels whose port starts with `ica`
]

Update the configuration without restarting Hermes

⚠️ This feature has been removed in Hermes v0.12.0.

Before Hermes 0.6.1, the only way to get Hermes to pick up a change in the configuration was to stop and restart Hermes.

As of version 0.6.1, Hermes will react to receiving a SIGHUP signal by reloading the [chains] section of the configuration, and stopping, starting or restarting the affected workers.

Warning: the configuration reload feature only supports adding, removing, or updating configuration of chains. It does not support dynamically changing global features, such as the filtering mechanism or logging level.

For example, say you start with the configuration given in the previous section in ~/.hermes/config.toml, ie. with two chains ibc-0 and ibc-1.

1. Start three chains ibc-0, ibc-1 and ibc-2:

   ./scripts/dev-env ibc-0 ibc-1 ibc-2
   

2. Start Hermes

   hermes start
   

3. Add the configuration for the chain ibc-2 to the configuration file:

   [[chains]]
   id = 'ibc-2'
   rpc_addr = 'http://127.0.0.1:26457'
   grpc_addr = 'http://127.0.0.1:9092'
   websocket_addr = 'ws://127.0.0.1:26457/websocket'
   rpc_timeout = '10s'
   account_prefix = 'cosmos'
   key_name = 'testkey'
   store_prefix = 'ibc'
   max_gas = 20000000
   gas_price = { price = 0.001, denom = 'stake' }
   clock_drift = '5s'
   trusting_period = '14days'
   

4. Change the configuration of the chain ibc-0, eg. the max_gas property.

5. Send a SIGHUP signal to the hermes process:

   ⚠️ Warning: the command below will send a SIGHUP signal to the first process in the list emitted by ps aux which contains the string hermes. Alternatively, you can look up the process ID (PID) of the hermes process you want to target and use kill -SIGHUP PID.

   ps aux | rg hermes | awk '{ print $2 }' | head -n1 | xargs -I{} kill -SIGHUP {}
   

6. Watch the output of Hermes, it will show that Hermes has picked up the changes in the config. Hermes is now relaying between the three chains and using the new maximum amount of gas specified for ibc-0.

   ...
   
   INFO reloading configuration (triggered by SIGHUP)
   INFO configuration successfully reloaded
   INFO updating existing chain chain.id=ibc-1
   INFO adding new chain chain.id=ibc-2
   

To make sure Hermes ends up in the expected state, check out the documentation on inspecting the relayer state.

Next steps

Now that you learned how to build the relayer and how to create a configuration file, you can go to the Two Chains tutorial to learn how to perform some local testing connecting the relayer to two local chains.

Example Configuration File

Here is a full example of a configuration file with two chains configured:

# The global section has parameters that apply globally to the relayer operation.
[global]

# Specify the verbosity for the relayer logging output. Default: 'info'
# Valid options are 'error', 'warn', 'info', 'debug', 'trace'.
log_level = 'info'


# Specify the mode to be used by the relayer. [Required]
[mode]

# Specify the client mode.
[mode.clients]

# Whether or not to enable the client workers. [Required]
enabled = true

# Whether or not to enable periodic refresh of clients. [Default: true]
# This feature only applies to clients that underlie an open channel.
# For Tendermint clients, the frequency at which Hermes refreshes them is 2/3 of their
# trusting period (e.g., refresh every ~9 days if the trusting period is 14 days).
# Note: Even if this is disabled, clients will be refreshed automatically if
#      there is activity on a connection or channel they are involved with.
refresh = true

# Whether or not to enable misbehaviour detection for clients. [Default: false]
misbehaviour = true

# Specify the connections mode.
[mode.connections]

# Whether or not to enable the connection workers for handshake completion. [Required]
enabled = false

# Specify the channels mode.
[mode.channels]

# Whether or not to enable the channel workers for handshake completion. [Required]
enabled = false

# Specify the packets mode.
[mode.packets]

# Whether or not to enable the packet workers. [Required]
enabled = true

# Parametrize the periodic packet clearing feature.
# Interval (in number of blocks) at which pending packets
# should be periodically cleared. A value of '0' will disable
# periodic packet clearing. [Default: 100]
clear_interval = 100

# Whether or not to clear packets on start. [Default: false]
clear_on_start = true

# Toggle the transaction confirmation mechanism.
# The tx confirmation mechanism periodically queries the `/tx_search` RPC
# endpoint to check that previously-submitted transactions
# (to any chain in this config file) have been successfully delivered.
# If they have not been, and `clear_interval = 0`, then those packets are
# queued up for re-submission.
# Experimental feature. Affects telemetry if set to false.
# [Default: true]
tx_confirmation = true

# The REST section defines parameters for Hermes' built-in RESTful API.
# https://hermes.informal.systems/rest.html
[rest]

# Whether or not to enable the REST service. Default: false
enabled = true

# Specify the IPv4/6 host over which the built-in HTTP server will serve the RESTful
# API requests. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the restful API
# requests. Default: 3000
port = 3000


# The telemetry section defines parameters for Hermes' built-in telemetry capabilities.
# https://hermes.informal.systems/telemetry.html
[telemetry]

# Whether or not to enable the telemetry service. Default: false
enabled = true

# Specify the IPv4/6 host over which the built-in HTTP server will serve the metrics
# gathered by the telemetry service. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the metrics gathered
# by the telemetry service. Default: 3001
port = 3001


# A chains section includes parameters related to a chain and the full node to which
# the relayer can send transactions and queries.
[[chains]]

# Specify the chain ID. Required
id = 'ibc-0'

# Specify the RPC address and port where the chain RPC server listens on. Required
rpc_addr = 'http://127.0.0.1:26657'

# Specify the GRPC address and port where the chain GRPC server listens on. Required
grpc_addr = 'http://127.0.0.1:9090'

# Specify the WebSocket address and port where the chain WebSocket server
# listens on. Required
websocket_addr = 'ws://127.0.0.1:26657/websocket'

# Specify the maximum amount of time (duration) that the RPC requests should
# take before timing out. Default: 10s (10 seconds)
# Note: Hermes uses this parameter _only_ in `start` mode; for all other CLIs,
# Hermes uses a large preconfigured timeout (on the order of minutes).
rpc_timeout = '10s'

# Specify the prefix used by the chain. Required
account_prefix = 'cosmos'

# Specify the name of the private key to use for signing transactions. Required
# See the Adding Keys chapter for more information about managing signing keys:
#   https://hermes.informal.systems/commands/keys/index.html#adding-keys
key_name = 'testkey'

# Specify the address type which determines:
# 1) address derivation;
# 2) how to retrieve and decode accounts and pubkeys;
# 3) the message signing method.
# The current configuration options are for Cosmos SDK and Ethermint.
#
# Example configuration for chains based on Ethermint library:
#
# address_type = { derivation = 'ethermint', proto_type = { pk_type = '/ethermint.crypto.v1.ethsecp256k1.PubKey' } }
#
# Default: { derivation = 'cosmos' }, i.e. address derivation as in Cosmos SDK.
# Warning: This is an advanced feature! Modify with caution.
address_type = { derivation = 'cosmos' }

# Specify the store prefix used by the on-chain IBC modules. Required
# Recommended value for Cosmos SDK: 'ibc'
store_prefix = 'ibc'

# Specify the default amount of gas to be used in case the tx simulation fails,
# and Hermes cannot estimate the amount of gas needed.
# Default: 100 000
default_gas = 100000

# Specify the maximum amount of gas to be used as the gas limit for a transaction.
# Default: 400 000
max_gas = 400000

# Specify the price per gas used of the fee to submit a transaction and
# the denomination of the fee. Required
gas_price = { price = 0.001, denom = 'stake' }

# Specify the ratio by which to increase the gas estimate used to compute the fee,
# to account for potential estimation error. Default: 0.1, ie. 10%.
# Valid range: 0.0 to 1.0 (inclusive)
gas_adjustment = 1.0

# Specify how many IBC messages at most to include in a single transaction.
# Default: 30
max_msg_num = 30

# Specify the maximum size, in bytes, of each transaction that Hermes will submit.
# Default: 2097152 (2 MiB)
max_tx_size = 2097152

# Specify the maximum amount of time to tolerate a clock drift.
# The clock drift parameter defines how much new (untrusted) header's time
# can drift into the future. Default: 5s
clock_drift = '5s'

# Specify the maximum time per block for this chain.
# The block time together with the clock drift are added to the source drift to estimate
# the maximum clock drift when creating a client on this chain. Default: 30s
# For cosmos-SDK chains a good approximation is `timeout_propose` + `timeout_commit`
# Note: This MUST be the same as the `max_expected_time_per_block` genesis parameter for Tendermint chains.
max_block_time = '30s'

# Specify the amount of time to be used as the light client trusting period.
# It should be significantly less than the unbonding period
# (e.g. unbonding period = 3 weeks, trusting period = 2 weeks).
# Default: 2/3 of the `unbonding period` for Cosmos SDK chains
trusting_period = '14days'

# Specify the trust threshold for the light client, ie. the maximum fraction of validators
# which have changed between two blocks.
# Default: { numerator = '1', denominator = '3' }, ie. 1/3.
# Warning: This is an advanced feature! Modify with caution.
trust_threshold = { numerator = '1', denominator = '3' }

# Specify a string that Hermes will use as a memo for each transaction it submits
# to this chain. The string is limited to 50 characters. Default: '' (empty).
# Note: Hermes will append to the string defined here additional
# operational debugging information, e.g., relayer build version.
memo_prefix = ''

# This section specifies the filters for policy based relaying.
#
# Default: no policy / filters, allow all packets on all channels.
#
# Only packet filtering based on channel identifier can be specified.
# A channel filter has two fields:
# 1. `policy` - one of two types are supported:
#       - 'allow': permit relaying _only on_ the port/channel id in the list below,
#       - 'deny': permit relaying on any channel _except for_ the list below.
# 2. `list` - the list of channels specified by the port and channel identifiers.
#             Optionally, each element may also contains wildcards, for eg. 'ica*'
#             to match all identifiers starting with 'ica' or '*' to match all identifiers.
#
# Example configuration of a channel filter, only allowing packet relaying on
# channel with port ID 'transfer' and channel ID 'channel-0', as well as on
# all ICA channels.
#
# [chains.packet_filter]
# policy = 'allow'
# list = [
#   ['ica*', '*'],
#   ['transfer', 'channel-0'],
# ]

# Specify that the transaction fees should be payed from this fee granter's account.
# Optional. If unspecified (the default behavior), then no fee granter is used, and
# the account specified in `key_name` will pay the tx fees for all transactions
# submitted to this chain.
# fee_granter = ''

[[chains]]
id = 'ibc-1'
rpc_addr = 'http://127.0.0.1:26557'
grpc_addr = 'http://127.0.0.1:9091'
websocket_addr = 'ws://127.0.0.1:26557/websocket'
rpc_timeout = '10s'
account_prefix = 'cosmos'
key_name = 'testkey'
store_prefix = 'ibc'
default_gas = 100000
max_gas = 400000
gas_price = { price = 0.001, denom = 'stake' }
gas_adjustment = 0.1
max_msg_num = 30
max_tx_size = 2097152
clock_drift = '5s'
max_block_time = '30s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }
address_type = { derivation = 'cosmos' }

Telemetry

Since version 0.4.0.

To gain a better understanding of the status and activity of the relayer, Hermes features a built-in telemetry service based on the OpenTelemetry observability framework, whose metrics can be exposed over HTTP for integration with the Prometheus monitoring system.

The official Hermes builds for Linux and macOS come with telemetry support since version 0.4.0, and can be downloaded directly from the GitHub Releases page.

Configuration

The telemetry service is not active by default, and must be enabled in the relayer configuration:

[telemetry]
enabled = true
host    = '127.0.0.1'
port    = 3001

Please see the relevant section in the Configuration page for details about the configuration options.

Metrics

The following table describes the metrics currently tracked by the telemetry service:

Integration with Prometheus

With the settings , the telemetry service will be enabled and will serve the metrics using the Prometheus encoder over HTTP at http://localhost:3001/metrics.

After starting Hermes with hermes start, and letting it run for a while to relay packets, open http://localhost:3001/metrics in a browser, you should see Prometheus-encoded metrics.

For example, with two channels and after transferring some tokens between the chains:

# HELP cache_hits Number of cache hits for queries emitted by the relayer, per chain and query type
# TYPE cache_hits counter
cache_hits{chain="ibc-0",query_type="query_channel"} 276
cache_hits{chain="ibc-0",query_type="query_client_state"} 177
cache_hits{chain="ibc-0",query_type="query_connection"} 160
cache_hits{chain="ibc-1",query_type="query_channel"} 240
cache_hits{chain="ibc-1",query_type="query_client_state"} 173
cache_hits{chain="ibc-1",query_type="query_connection"} 160
# HELP ibc_acknowledgment_packets Number of acknowledgment packets relayed per channel
# TYPE ibc_acknowledgment_packets counter
ibc_acknowledgment_packets{src_chain="ibc-0",src_channel="channel-0",src_port="transfer"} 0
ibc_acknowledgment_packets{src_chain="ibc-0",src_channel="channel-1",src_port="transfer"} 42
ibc_acknowledgment_packets{src_chain="ibc-1",src_channel="channel-0",src_port="transfer"} 110
ibc_acknowledgment_packets{src_chain="ibc-1",src_channel="channel-1",src_port="transfer"} 0
# HELP ibc_receive_packets Number of receive packets relayed per channel
# TYPE ibc_receive_packets counter
ibc_receive_packets{src_chain="ibc-0",src_channel="channel-0",src_port="transfer"} 110
ibc_receive_packets{src_chain="ibc-0",src_channel="channel-1",src_port="transfer"} 0
ibc_receive_packets{src_chain="ibc-1",src_channel="channel-0",src_port="transfer"} 0
ibc_receive_packets{src_chain="ibc-1",src_channel="channel-1",src_port="transfer"} 42
# HELP ibc_timeout_packets Number of timeout packets relayed per channel
# TYPE ibc_timeout_packets counter
ibc_timeout_packets{src_chain="ibc-0",src_channel="channel-0",src_port="transfer"} 0
ibc_timeout_packets{src_chain="ibc-0",src_channel="channel-1",src_port="transfer"} 0
ibc_timeout_packets{src_chain="ibc-1",src_channel="channel-0",src_port="transfer"} 0
ibc_timeout_packets{src_chain="ibc-1",src_channel="channel-1",src_port="transfer"} 0
# HELP msg_num How many messages Hermes submitted to the chain, per chain
# TYPE msg_num counter
msg_num{chain="ibc-0"} 168
msg_num{chain="ibc-1"} 156
# HELP queries Number of queries emitted by the relayer, per chain and query type
# TYPE queries counter
queries{chain="ibc-0",query_type="query_application_status"} 23
queries{chain="ibc-0",query_type="query_channel"} 88
queries{chain="ibc-0",query_type="query_client_connections"} 2
queries{chain="ibc-0",query_type="query_client_state"} 383
queries{chain="ibc-0",query_type="query_clients"} 1
queries{chain="ibc-0",query_type="query_connection"} 2
queries{chain="ibc-0",query_type="query_connection_channels"} 2
queries{chain="ibc-0",query_type="query_consensus_state"} 392
queries{chain="ibc-0",query_type="query_consensus_states"} 2
queries{chain="ibc-0",query_type="query_latest_height"} 1
queries{chain="ibc-0",query_type="query_packet_acknowledgements"} 5
queries{chain="ibc-0",query_type="query_packet_commitments"} 10
queries{chain="ibc-0",query_type="query_staking_params"} 2
queries{chain="ibc-0",query_type="query_txs"} 76
queries{chain="ibc-0",query_type="query_unreceived_acknowledgements"} 241
queries{chain="ibc-0",query_type="query_unreceived_packets"} 127
queries{chain="ibc-1",query_type="query_application_status"} 20
queries{chain="ibc-1",query_type="query_channel"} 224
queries{chain="ibc-1",query_type="query_client_connections"} 2
queries{chain="ibc-1",query_type="query_client_state"} 387
queries{chain="ibc-1",query_type="query_clients"} 1
queries{chain="ibc-1",query_type="query_connection"} 2
queries{chain="ibc-1",query_type="query_connection_channels"} 2
queries{chain="ibc-1",query_type="query_consensus_state"} 394
queries{chain="ibc-1",query_type="query_consensus_states"} 3
queries{chain="ibc-1",query_type="query_latest_height"} 1
queries{chain="ibc-1",query_type="query_packet_acknowledgements"} 5
queries{chain="ibc-1",query_type="query_packet_commitments"} 10
queries{chain="ibc-1",query_type="query_staking_params"} 2
queries{chain="ibc-1",query_type="query_txs"} 56
queries{chain="ibc-1",query_type="query_unreceived_acknowledgements"} 127
queries{chain="ibc-1",query_type="query_unreceived_packets"} 292
# HELP tx_latency_confirmed The latency for all transactions submitted to a specific chain, i.e. the difference between the moment when Hermes received a batch of events until the corresponding transaction(s) were confirmed. Milliseconds.
# TYPE tx_latency_confirmed histogram
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",le="0.5"} 0
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",le="0.9"} 0
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",le="0.99"} 0
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",le="+Inf"} 4
tx_latency_confirmed_sum{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer"} 22466
tx_latency_confirmed_count{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer"} 4
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer",le="0.5"} 0
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer",le="0.9"} 0
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer",le="0.99"} 0
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer",le="+Inf"} 1
tx_latency_confirmed_sum{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer"} 4256
tx_latency_confirmed_count{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer"} 1
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",le="0.5"} 0
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",le="0.9"} 0
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",le="0.99"} 0
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",le="+Inf"} 2
tx_latency_confirmed_sum{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} 9408
tx_latency_confirmed_count{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} 2
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer",le="0.5"} 0
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer",le="0.9"} 0
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer",le="0.99"} 0
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer",le="+Inf"} 1
tx_latency_confirmed_sum{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer"} 3173
tx_latency_confirmed_count{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer"} 1
# HELP tx_latency_submitted The latency for all transactions submitted to a specific chain, i.e. the difference between the moment when Hermes received a batch of events and when it submitted the corresponding transaction(s). Milliseconds.
# TYPE tx_latency_submitted histogram
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",le="0.5"} 0
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",le="0.9"} 0
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",le="0.99"} 0
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",le="+Inf"} 5
tx_latency_submitted_sum{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer"} 14428
tx_latency_submitted_count{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer"} 5
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer",le="0.5"} 0
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer",le="0.9"} 0
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer",le="0.99"} 0
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer",le="+Inf"} 1
tx_latency_submitted_sum{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer"} 729
tx_latency_submitted_count{chain="ibc-0",channel="channel-1",counterparty="ibc-1",port="transfer"} 1
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",le="0.5"} 0
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",le="0.9"} 0
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",le="0.99"} 0
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",le="+Inf"} 2
tx_latency_submitted_sum{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} 1706
tx_latency_submitted_count{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} 2
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer",le="0.5"} 0
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer",le="0.9"} 0
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer",le="0.99"} 0
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer",le="+Inf"} 1
tx_latency_submitted_sum{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer"} 791
tx_latency_submitted_count{chain="ibc-1",channel="channel-1",counterparty="ibc-0",port="transfer"} 1
# HELP wallet_balance The balance in each wallet that Hermes is using, per wallet, denom and chain
# TYPE wallet_balance gauge
wallet_balance{account="cosmos1934akx97773lsjjs9x74dr03uuam29hcc9grp3",chain="ibc-0",denom="stake"} 99999970473
wallet_balance{account="cosmos1hngzqscyg476nd68qggxps8r2aq56lne45ps8n",chain="ibc-1",denom="stake"} 99999978431
# HELP workers Number of workers per object
# TYPE workers gauge
workers{type="client"} 4
workers{type="packet"} 4
workers{type="wallet"} 2
# HELP ws_events How many IBC events did Hermes receive via the WebSocket subscription, per chain
# TYPE ws_events counter
ws_events{chain="ibc-0"} 443
ws_events{chain="ibc-1"} 370

REST API

Since version 0.7.0.

Hermes features a built-in HTTP server which exposes information about the relayer configuration and state via a REST API.

Table of Contents

• Configuration
• Endpoints
  • GET /version
  • GET /chains
  • GET /chain/:id
  • GET /state

Configuration

The REST API is not active by default, and must be enabled in the relayer configuration:

[rest]
enabled = true
host    = '127.0.0.1'
port    = 3000

Please see the relevant section in the Configuration page for details about the configuration options.

Endpoints

GET /version

This endpoint returns the version of the Hermes (under the ibc-relayer key) as well as the version of the REST server itself (under the ibc-relayer-rest key).

Example

❯ curl -s -X GET 'http://127.0.0.1:3000/version' | jq

[
  {
    "name": "ibc-relayer",
    "version": "0.15.0"
  },
  {
    "name": "ibc-relayer-rest",
    "version": "0.1.0"
  }
]

GET /chains

This endpoint return the identifiers of the chains that Hermes is connected to. Those identifiers can be used with the /chain/:id endpoint to gather more information about each chain's configuration. See the next section for more details.

Example

❯ curl -s -X GET 'http://127.0.0.1:3000/chains' | jq

{
  "status": "success",
  "result": [
    "ibc-0",
    "ibc-1"
  ]
}

GET /chain/:id

This endpoint returns the configuration of the chain with the given identifier, where :id stands for the identififer.

Example

❯ curl -s -X GET 'http://127.0.0.1:3000/chain/ibc-0' | jq

{
  "status": "success",
  "result": {
    "id": "ibc-0",
    "rpc_addr": "http://127.0.0.1:26657/",
    "websocket_addr": "ws://127.0.0.1:26657/websocket",
    "grpc_addr": "http://127.0.0.1:9090/",
    "rpc_timeout": "10s",
    "account_prefix": "cosmos",
    "key_name": "testkey",
    "store_prefix": "ibc",
    "max_gas": 900000000,
    "gas_adjustment": null,
    "max_msg_num": 60,
    "max_tx_size": 2097152,
    "clock_drift": "5s",
    "trusting_period": "14days",
    "trust_threshold": {
      "numerator": "1",
      "denominator": "3"
    },
    "gas_price": {
      "price": 0.001,
      "denom": "stake"
    },
    "packet_filter": {
      "policy": "allowall"
    }
  }
}

GET /state

This endpoint returns the current state of the relayer, namely which chains it is connected to, as well as a description of all the workers which are currently active.

❯ curl -s -X GET 'http://127.0.0.1:3000/state' | jq

{
  "status": "success",
  "result": {
    "chains": [
      "ibc-0",
      "ibc-1"
    ],
    "workers": {
      "Client": [
        {
          "id": 3,
          "object": {
            "type": "Client",
            "dst_chain_id": "ibc-1",
            "dst_client_id": "07-tendermint-0",
            "src_chain_id": "ibc-0"
          }
        },
        {
          "id": 4,
          "object": {
            "type": "Client",
            "dst_chain_id": "ibc-1",
            "dst_client_id": "07-tendermint-1",
            "src_chain_id": "ibc-0"
          }
        },
        {
          "id": 1,
          "object": {
            "type": "Client",
            "dst_chain_id": "ibc-0",
            "dst_client_id": "07-tendermint-0",
            "src_chain_id": "ibc-1"
          }
        },
        {
          "id": 2,
          "object": {
            "type": "Client",
            "dst_chain_id": "ibc-0",
            "dst_client_id": "07-tendermint-1",
            "src_chain_id": "ibc-1"
          }
        }
      ]
    }
  }
}

Tutorials

This section includes tutorials for some common relayer uses cases and commands. You can also refer to the Commands Reference section to learn more about individual commands.

Basic tutorials

Two Local Chains

In this tutorial you will learn how to start two local Cosmos Gaia chains that support the IBC protocol and start relaying packets between them.

Tutorial: Relayer with two local chains

In this tutorial we will show how you can test the relayer against two chains, we provide a script that can start two separate chains and configure them automatically. This is the easiest way to get started.

The script starts two gaia chains that support the IBC protocol.

Follow the steps in this tutorial section starting with the Install Gaia section.# Local chains

Install Gaia

The script to start the chains requires gaia to be installed.

NOTE: This assumes you have Golang programming language installed on your machine. If not, please ensure you install before proceeding. See more details in the Pre-requisites section.

Clone gaia

Clone the repository from Github:

git clone https://github.com/cosmos/gaia.git ~/go/src/github.com/cosmos/gaia

Build and Install

Run the make command to build and install gaiad

cd ~/go/src/github.com/cosmos/gaia
git checkout v4.2.1
make install

If the command above is successful you can run the following command to ensure it was properly installed:

gaiad version --log_level error --long | head -n4

Output:

name: gaia
server_name: gaiad
version: v4.2.1
commit: dbd8a6fb522c571debf958837f9113c56d418f6b

Next Steps

In the next section you will learn how to start two local chains

Start the local chains

In this chapter, you will learn how to spawn two Gaia chains, and use Hermes to relay packets between them. To spawn the chains and configure Hermes accordingly, we will make use of script bundled in the ibc-rs repository.

To this end, clone the ibc-rs repository and check out the current version:

git clone git@github.com:informalsystems/ibc-rs.git
cd ibc-rs
git checkout v0.15.0

Stop existing gaiad processes

If this is not the first time you are running the script, you can manually stop the two gaia instances executing the following command to kill all gaiad processes:

killall gaiad

NOTE: If you have any Docker containers running that might be using the same ports as gaiad (e.g. port 26657 or port 9090), please ensure you stop them first before proceeding to the next step.

Configuration file

In order to run the script, you will need a TOML configuration file to be passed as a parameter. Please check the Configuration section for more information about the relayer configuration file.

The following configuration file in the ibc-rs repository folder can be used for running the local chains:

config.toml

# The global section has parameters that apply globally to the relayer operation.
[global]

# Specify the verbosity for the relayer logging output. Default: 'info'
# Valid options are 'error', 'warn', 'info', 'debug', 'trace'.
log_level = 'info'


# Specify the mode to be used by the relayer. [Required]
[mode]

# Specify the client mode.
[mode.clients]

# Whether or not to enable the client workers. [Required]
enabled = true

# Whether or not to enable periodic refresh of clients. [Default: true]
# This feature only applies to clients that underlie an open channel.
# For Tendermint clients, the frequency at which Hermes refreshes them is 2/3 of their
# trusting period (e.g., refresh every ~9 days if the trusting period is 14 days).
# Note: Even if this is disabled, clients will be refreshed automatically if
#      there is activity on a connection or channel they are involved with.
refresh = true

# Whether or not to enable misbehaviour detection for clients. [Default: false]
misbehaviour = true

# Specify the connections mode.
[mode.connections]

# Whether or not to enable the connection workers for handshake completion. [Required]
enabled = false

# Specify the channels mode.
[mode.channels]

# Whether or not to enable the channel workers for handshake completion. [Required]
enabled = false

# Specify the packets mode.
[mode.packets]

# Whether or not to enable the packet workers. [Required]
enabled = true

# Parametrize the periodic packet clearing feature.
# Interval (in number of blocks) at which pending packets
# should be periodically cleared. A value of '0' will disable
# periodic packet clearing. [Default: 100]
clear_interval = 100

# Whether or not to clear packets on start. [Default: false]
clear_on_start = true

# Toggle the transaction confirmation mechanism.
# The tx confirmation mechanism periodically queries the `/tx_search` RPC
# endpoint to check that previously-submitted transactions
# (to any chain in this config file) have been successfully delivered.
# If they have not been, and `clear_interval = 0`, then those packets are
# queued up for re-submission.
# Experimental feature. Affects telemetry if set to false.
# [Default: true]
tx_confirmation = true

# The REST section defines parameters for Hermes' built-in RESTful API.
# https://hermes.informal.systems/rest.html
[rest]

# Whether or not to enable the REST service. Default: false
enabled = true

# Specify the IPv4/6 host over which the built-in HTTP server will serve the RESTful
# API requests. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the restful API
# requests. Default: 3000
port = 3000


# The telemetry section defines parameters for Hermes' built-in telemetry capabilities.
# https://hermes.informal.systems/telemetry.html
[telemetry]

# Whether or not to enable the telemetry service. Default: false
enabled = true

# Specify the IPv4/6 host over which the built-in HTTP server will serve the metrics
# gathered by the telemetry service. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the metrics gathered
# by the telemetry service. Default: 3001
port = 3001


# A chains section includes parameters related to a chain and the full node to which
# the relayer can send transactions and queries.
[[chains]]

# Specify the chain ID. Required
id = 'ibc-0'

# Specify the RPC address and port where the chain RPC server listens on. Required
rpc_addr = 'http://127.0.0.1:26657'

# Specify the GRPC address and port where the chain GRPC server listens on. Required
grpc_addr = 'http://127.0.0.1:9090'

# Specify the WebSocket address and port where the chain WebSocket server
# listens on. Required
websocket_addr = 'ws://127.0.0.1:26657/websocket'

# Specify the maximum amount of time (duration) that the RPC requests should
# take before timing out. Default: 10s (10 seconds)
# Note: Hermes uses this parameter _only_ in `start` mode; for all other CLIs,
# Hermes uses a large preconfigured timeout (on the order of minutes).
rpc_timeout = '10s'

# Specify the prefix used by the chain. Required
account_prefix = 'cosmos'

# Specify the name of the private key to use for signing transactions. Required
# See the Adding Keys chapter for more information about managing signing keys:
#   https://hermes.informal.systems/commands/keys/index.html#adding-keys
key_name = 'testkey'

# Specify the address type which determines:
# 1) address derivation;
# 2) how to retrieve and decode accounts and pubkeys;
# 3) the message signing method.
# The current configuration options are for Cosmos SDK and Ethermint.
#
# Example configuration for chains based on Ethermint library:
#
# address_type = { derivation = 'ethermint', proto_type = { pk_type = '/ethermint.crypto.v1.ethsecp256k1.PubKey' } }
#
# Default: { derivation = 'cosmos' }, i.e. address derivation as in Cosmos SDK.
# Warning: This is an advanced feature! Modify with caution.
address_type = { derivation = 'cosmos' }

# Specify the store prefix used by the on-chain IBC modules. Required
# Recommended value for Cosmos SDK: 'ibc'
store_prefix = 'ibc'

# Specify the default amount of gas to be used in case the tx simulation fails,
# and Hermes cannot estimate the amount of gas needed.
# Default: 100 000
default_gas = 100000

# Specify the maximum amount of gas to be used as the gas limit for a transaction.
# Default: 400 000
max_gas = 400000

# Specify the price per gas used of the fee to submit a transaction and
# the denomination of the fee. Required
gas_price = { price = 0.001, denom = 'stake' }

# Specify the ratio by which to increase the gas estimate used to compute the fee,
# to account for potential estimation error. Default: 0.1, ie. 10%.
# Valid range: 0.0 to 1.0 (inclusive)
gas_adjustment = 1.0

# Specify how many IBC messages at most to include in a single transaction.
# Default: 30
max_msg_num = 30

# Specify the maximum size, in bytes, of each transaction that Hermes will submit.
# Default: 2097152 (2 MiB)
max_tx_size = 2097152

# Specify the maximum amount of time to tolerate a clock drift.
# The clock drift parameter defines how much new (untrusted) header's time
# can drift into the future. Default: 5s
clock_drift = '5s'

# Specify the maximum time per block for this chain.
# The block time together with the clock drift are added to the source drift to estimate
# the maximum clock drift when creating a client on this chain. Default: 30s
# For cosmos-SDK chains a good approximation is `timeout_propose` + `timeout_commit`
# Note: This MUST be the same as the `max_expected_time_per_block` genesis parameter for Tendermint chains.
max_block_time = '30s'

# Specify the amount of time to be used as the light client trusting period.
# It should be significantly less than the unbonding period
# (e.g. unbonding period = 3 weeks, trusting period = 2 weeks).
# Default: 2/3 of the `unbonding period` for Cosmos SDK chains
trusting_period = '14days'

# Specify the trust threshold for the light client, ie. the maximum fraction of validators
# which have changed between two blocks.
# Default: { numerator = '1', denominator = '3' }, ie. 1/3.
# Warning: This is an advanced feature! Modify with caution.
trust_threshold = { numerator = '1', denominator = '3' }

# Specify a string that Hermes will use as a memo for each transaction it submits
# to this chain. The string is limited to 50 characters. Default: '' (empty).
# Note: Hermes will append to the string defined here additional
# operational debugging information, e.g., relayer build version.
memo_prefix = ''

# This section specifies the filters for policy based relaying.
#
# Default: no policy / filters, allow all packets on all channels.
#
# Only packet filtering based on channel identifier can be specified.
# A channel filter has two fields:
# 1. `policy` - one of two types are supported:
#       - 'allow': permit relaying _only on_ the port/channel id in the list below,
#       - 'deny': permit relaying on any channel _except for_ the list below.
# 2. `list` - the list of channels specified by the port and channel identifiers.
#             Optionally, each element may also contains wildcards, for eg. 'ica*'
#             to match all identifiers starting with 'ica' or '*' to match all identifiers.
#
# Example configuration of a channel filter, only allowing packet relaying on
# channel with port ID 'transfer' and channel ID 'channel-0', as well as on
# all ICA channels.
#
# [chains.packet_filter]
# policy = 'allow'
# list = [
#   ['ica*', '*'],
#   ['transfer', 'channel-0'],
# ]

# Specify that the transaction fees should be payed from this fee granter's account.
# Optional. If unspecified (the default behavior), then no fee granter is used, and
# the account specified in `key_name` will pay the tx fees for all transactions
# submitted to this chain.
# fee_granter = ''

[[chains]]
id = 'ibc-1'
rpc_addr = 'http://127.0.0.1:26557'
grpc_addr = 'http://127.0.0.1:9091'
websocket_addr = 'ws://127.0.0.1:26557/websocket'
rpc_timeout = '10s'
account_prefix = 'cosmos'
key_name = 'testkey'
store_prefix = 'ibc'
default_gas = 100000
max_gas = 400000
gas_price = { price = 0.001, denom = 'stake' }
gas_adjustment = 0.1
max_msg_num = 30
max_tx_size = 2097152
clock_drift = '5s'
max_block_time = '30s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }
address_type = { derivation = 'cosmos' }

Saving the configuration file

Create the config.toml file

mkdir -p $HOME/.hermes && touch $HOME/.hermes/config.toml

Add content to the configuration file:

You can use your preferred text editor. If using vi you can run:

vi ~/.hermes/config.toml

Then just copy the content for config.toml above and paste into this file.

Generating private keys

Next, we will need to associate a private key with chains ibc-0 and ibc-1 which hermes will use to sign transactions. There are two steps involved. For each chain,

1. Generate a Key Seed file using gaiad
2. Use the Key Seed file to associate the corresponding private key with the chain

In this tutorial, we will only generate a single Key Seed file, which we will use with both chains.

Generate a Key Seed file

We will generate the Key Seed file for a key that we will call testkey.

gaiad keys add testkey --output json

This will generate an output similar to the one below (albeit all on the same line):

{
  "name": "testkey",
  "type": "local",
  "address": "cosmos1tc3vcuxyyac0dmayf887t95tdg7qpyql48w7gj",
  "pubkey": "cosmospub1addwnpepqgg7ng4ycm60pdxfzdfh4hjvkwcr3da59mr8k883vsstx60ruv7kur4525u",
  "mnemonic": "[24 words mnemonic]"
}

Next, copy and paste the output to a new file called key_seed.json. This file contains all the information necessary for hermes to derive a private key from.

Associate a private key with each chain

Our config file specifies two chains: ibc-0 and ibc-1. We will need to specify a private key that hermes will use for each chain. As previously mentioned, in this tutorial we will use the same private key for both chains.

hermes keys add ibc-0 -f key_seed.json
hermes keys add ibc-1 -f key_seed.json

If successful, both commands should show an output similar to:

Success: Added key testkey ([ADDRESS]) on [CHAIN ID] chain

And that's it! hermes will now be able to sign transactions to be sent to both chains. key_seed.json can safely be disposed of.

Running the script to start the chains

From the ibc-rs repository folder run the following script with the parameters below to start the chains (ibc-0 and ibc-1) and import the signing keys into the keyring:

./scripts/dev-env ~/.hermes/config.toml ibc-0 ibc-1

NOTE: If the script above prompts you to delete the data folder just answer 'yes'

The script configures and starts two gaiad instances, one named ibc-0 and the other ibc-1

graph TD
    A[dev-env] -->|run| C(start chains)
    C -->|gaiad| D[ibc-0]
    C -->|gaiad| F[ibc-1]

If the script runs successfully you should see a message similar to the one below in the terminal:

GAIA VERSION INFO: v4.2.1
Generating gaia configurations...
Creating gaiad instance: home=./data | chain-id=ibc-0 | p2p=:26656 | rpc=:26657 | profiling=:6060 | grpc=:9090 | samoleans=:100000000000
Change settings in config.toml file...
Start gaia on grpc port: 9090...
Balances for validator 'cosmos15cugtww7rwmayvshfznuxam55jsv23xh3jdeqv' @ 'tcp://localhost:26657'
balances:
- amount: "0"
  denom: stake
pagination:
  next_key: null
  total: "0"
Balances for user 'cosmos1usn8g2rj9q48y245pql9589zf9m8srcpxtzklg' @ 'tcp://localhost:26657'
balances:
- amount: "100000000000"
  denom: samoleans
- amount: "100000000000"
  denom: stake
pagination:
  next_key: null
  total: "0"
Creating gaiad instance: home=./data | chain-id=ibc-1 | p2p=:26556 | rpc=:26557 | profiling=:6061 | grpc=:9091 | samoleans=:100000000000
Change settings in config.toml file...
Start gaia on grpc port: 9091...
Balances for validator 'cosmos1zdmr04w7c04ef4vkuur9c0vyvl78q45qjncmja' @ 'tcp://localhost:26557'
balances:
- amount: "0"
  denom: stake
pagination:
  next_key: null
  total: "0"
Balances for user 'cosmos12p6k2dta0lsd6n80tpz34yepfpv7u7fvedm5mp' @ 'tcp://localhost:26557'
balances:
- amount: "100000000000"
  denom: samoleans
- amount: "100000000000"
  denom: stake
pagination:
  next_key: null
  total: "0"
ibc-0 initialized. Watch file /Users/ancaz/rust/ibc-rs/data/ibc-0.log to see its execution.
ibc-1 initialized. Watch file /Users/ancaz/rust/ibc-rs/data/ibc-1.log to see its execution.
Building the Rust relayer...
Importing keys...
Success: Added key 'testkey' (cosmos1usn8g2rj9q48y245pql9589zf9m8srcpxtzklg) on chain ibc-0
Success: Added key 'testkey' (cosmos12p6k2dta0lsd6n80tpz34yepfpv7u7fvedm5mp) on chain ibc-1
Done!

Data directory

The script creates a data directory in the current directory in order. The data directory contains the chain stores and configuration files.

The data directory has a tree structure similar to the one below:

data
├── ibc-0
│   ├── config
│   ├── data
│   ├── keyring-test
│   ├── user_seed.json
│   ├── user2_seed.json
│   └── validator_seed.json
├── ibc-0.log
├── ibc-1
│   ├── config
│   ├── data
│   ├── keyring-test
│   ├── user_seed.json
│   ├── user2_seed.json
│   └── validator_seed.json
└── ibc-1.log

Tip: You can use the command tree ./data/ -L 2 to view the folder structure above:

$HOME/.hermes directory

By the default hermes expects the configuration file to be in the $HOME/.hermes folder.

It also stores the private keys for each chain in this folder as outlined in the Keys section.

After executing the dev-env script, this is how the folder should look like:

$HOME/.hermes/
├── config.toml
└── keys
    ├── ibc-0
    │   └── keyring-test
    │       └── testkey.json
    └── ibc-1
        └── keyring-test
            └── testkey.json

Next Steps

The next section describes how identifers for clients, connections and channels are allocated, and will walk you through how to pre-allocate some identifers to help matching them with their corresponding chains for the purpose of this tutorial.

Identifiers

A chain allocates identifiers when it creates clients, connections and channels. These identifiers can subsequently be used to refer to existing clients, connections and channels.

NOTE: If you want to ensure you get the same identifiers while following the tutorials, run the each of the three commands below once on ibc-1. This will ensure that when going through the tutorial, a second channel on ibc-1 with identifier channel-1 will created.

Chains allocate identifiers using a chain specific allocation scheme. Currently, cosmos-sdk implementation uses the follow identifiers:

1. Client Identifiers

07-tendermint-<n> for tendermint clients

For example 07-tendermint-0 is assigned to the first client created on ibc-1:

hermes tx raw create-client ibc-1 ibc-0

Success: CreateClient(
   CreateClient(
       Attributes {
           height: Height {
               revision: 1,
               height: 103,
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           client_type: Tendermint,
           consensus_height: Height {
               revision: 0,
               height: 112,
           },
       },
   ),
)

We will create a second client on ibc-1 with identifier 07-tendermint-1 in the client tutorial.

2. Connection Identifiers

connection-<n> for connections

For example connection-0 is assigned to the first connection created on ibc-1:

hermes tx raw conn-init ibc-1 ibc-0 07-tendermint-0 07-tendermint-0

Success: OpenInitConnection(
    OpenInit(
        Attributes {
            height: Height {
                revision: 1,
                height: 119,
            },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: None,
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

We will create a second connection on ibc-1 with identifier connection-1 in the connection tutorial.

3. Channel Identifiers

channel-<n> for channels

For example channel-0 is assigned to the first channel created on ibc-1:

hermes tx raw chan-open-init ibc-1 ibc-0 connection-0 transfer transfer

Success: OpenInitChannel(
    OpenInit(
        Attributes {
            height: Height {
                revision: 1,
                height: 225,
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: None,
        },
    ),
)

In the following tutorials the ibc-0 and ibc-1 chains are setup and configured.

For clarity, the tutorials run on a setup where the identifiers allocated to the client, connection and channel on ibc-0 are 07-tendermint-0, connection-0 and channel-0 respectively. Identifiers allocated to the client, connection and channel on ibc-1 are 07-tendermint-1, connection-1 and channel-1 respectively.

Before going over the next sections, please ensure the commands above are executed.

Next Steps

The following sections describe the commands to connect and relay packets between two chains. You can:

1. use a simplified approach for managing relaying paths, or
2. use individual (raw) transactions to create all the necessary chain objects (clients, connections, channels) and relay packets.

Connect the chains using relay paths

A relay path refers to a specific channel used to interconnect two chains and over which packets are being sent.

Hermes can be started to listen for packet events on the two ends of multiple paths and relay packets over these paths. This can be done over a new path or over existing paths.

• Create a new path
• Packet relaying on multiple paths

Before proceeding to the sections above, please first, make sure you followed the steps in the Identifiers section

Create a new path

Perform client creation, connection and channel handshake to establish a new path between the transfer ports on ibc-0 and ibc-1 chains.

hermes create channel ibc-0 ibc-1 --port-a transfer --port-b transfer

If all the handshakes are performed successfully you should see a message similar to the one below:

Success: Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-0",
        ),
    },
    b_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-1",
        ),
        connection_id: ConnectionId(
            "connection-1",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-1",
        ),
    },
    connection_delay: 0s,
    version: Some(
        "ics20-1",
    ),
}

Note that for each side, a_side (ibc-0) and b_side (ibc-1) there are a client_id, connection_id, channel_id and port_id. With all these established, you have a path that you can relay packets over.

Relay packets on multiple paths

Hermes can relay packets over all current or future paths between the configured set of chains.

Follow the steps below to connect three chains together and relay packets between them:

1. Paste the following configuration in the standard Hermes configuration file at ~/.hermes/config.toml:

   [global]
   log_level = 'info'
   
   [mode]
   
   [mode.clients]
   enabled = true
   refresh = true
   misbehaviour = true
   
   [mode.connections]
   enabled = false
   
   [mode.channels]
   enabled = false
   
   [mode.packets]
   enabled = true
   clear_interval = 100
   clear_on_start = true
   tx_confirmation = true
   
   [[chains]]
   id = 'ibc-0'
   rpc_addr = 'http://127.0.0.1:26657'
   grpc_addr = 'http://127.0.0.1:9090'
   websocket_addr = 'ws://127.0.0.1:26657/websocket'
   rpc_timeout = '10s'
   account_prefix = 'cosmos'
   key_name = 'testkey'
   store_prefix = 'ibc'
   max_gas = 2000000
   gas_price = { price = 0.001, denom = 'stake' }
   gas_adjustment = 0.1
   clock_drift = '5s'
   trusting_period = '14days'
   trust_threshold = { numerator = '1', denominator = '3' }
   
   [[chains]]
   id = 'ibc-1'
   rpc_addr = 'http://127.0.0.1:26557'
   grpc_addr = 'http://127.0.0.1:9091'
   websocket_addr = 'ws://127.0.0.1:26557/websocket'
   rpc_timeout = '10s'
   account_prefix = 'cosmos'
   key_name = 'testkey'
   store_prefix = 'ibc'
   max_gas = 2000000
   gas_price = { price = 0.001, denom = 'stake' }
   gas_adjustment = 0.1
   clock_drift = '5s'
   trusting_period = '14days'
   trust_threshold = { numerator = '1', denominator = '3' }
   
   [[chains]]
   id = 'ibc-2'
   rpc_addr = 'http://127.0.0.1:26457'
   grpc_addr = 'http://127.0.0.1:9092'
   websocket_addr = 'ws://127.0.0.1:26457/websocket'
   rpc_timeout = '10s'
   account_prefix = 'cosmos'
   key_name = 'testkey'
   store_prefix = 'ibc'
   max_gas = 2000000
   gas_price = { price = 0.001, denom = 'stake' }
   gas_adjustment = 0.1
   clock_drift = '5s'
   trusting_period = '14days'
   trust_threshold = { numerator = '1', denominator = '3' }
   

   This configuration has three chains ibc-0, ibc-1 and ibc-2.

2. Run the dev-env script with the parameters below to start three chains:

   ./scripts/dev-env ~/.hermes/config.toml ibc-0 ibc-1 ibc-2
   

   NOTE: The script will prompt you to delete the data folder, double check the path and if it points to the data directory in the current directory, answer 'yes'.

   The script configures and starts three gaiad instances, named ibc-0, and ibc-1, and ibc-2.

3. Create a channel between ibc-0 and ibc-1. Since this is the first time we're connecting these two chains, we'll need to spin up a client and a connection between them as well. The create channel command gives us the convenient option to create a client and a connection. Keep in mind that this is not the default behavior of create channel, but in this case we're making an exception. Execute the following command:

   hermes create channel ibc-0 --chain-b ibc-1 --port-a transfer --port-b transfer --new-client-connection
   

   Then respond 'yes' to the prompt that pops up. Once the command has run to completion, you should see the following among the output logs:

   (...)
   
   Success: Channel {
       ordering: Unordered,
       a_side: ChannelSide {
           chain: ProdChainHandle {
               chain_id: ChainId {
                   id: "ibc-0",
                   version: 0,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           connection_id: ConnectionId(
               "connection-0",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: ChannelId(
               "channel-0",
           ),
       },
       b_side: ChannelSide {
           chain: ProdChainHandle {
               chain_id: ChainId {
                   id: "ibc-1",
                   version: 1,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           connection_id: ConnectionId(
               "connection-0",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: ChannelId(
               "channel-0",
           ),
       },
       connection_delay: 0s,
       version: Some(
           "ics20-1",
       ),
   }
   

   Note that the channel identifier on both ibc-0 and ibc-1 is channel-0.

4. Create a channel between ibc-1 and ibc-2 using the structure of the previous invocation we used to create a channel between ibc-0 and ibc-1:

   hermes create channel ibc-1 --chain-b ibc-2 --port-a transfer --port-b transfer --new-client-connection
   

   (...)
   
   Success: Channel {
       ordering: Unordered,
       a_side: ChannelSide {
           chain: ProdChainHandle {
               chain_id: ChainId {
                   id: "ibc-1",
                   version: 1,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-1",
           ),
           connection_id: ConnectionId(
               "connection-1",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: ChannelId(
               "channel-1",
           ),
       },
       b_side: ChannelSide {
           chain: ProdChainHandle {
               chain_id: ChainId {
                   id: "ibc-2",
                   version: 2,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           connection_id: ConnectionId(
               "connection-0",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: ChannelId(
               "channel-0",
           ),
       },
       connection_delay: 0s,
       version: Some(
           "ics20-1",
       ),
   }
   

   Note that the channel identifier on ibc-1 is channel-1, and on ibc-2 it is channel-0.

5. Start Hermes using the start command:

   hermes start
   

   Hermes will first relay the pending packets that have not been relayed and then start passive relaying by listening to and acting on packet events.

6. In a separate terminal, use the ft-transfer command to send:

   • Two packets from ibc-0 to ibc-1 from source channel channel-0

     hermes tx raw ft-transfer ibc-1 ibc-0 transfer channel-0 9999 -o 1000 -n 2
     

     Success: [
         SendPacket(
             SendPacket {
                 height: revision: 0, height: 3056,
                 packet: PortId("transfer") ChannelId("channel-0") Sequence(3),
             },
         ),
         SendPacket(
             SendPacket {
                 height: revision: 0, height: 3056,
                 packet: PortId("transfer") ChannelId("channel-0") Sequence(4),
             },
         ),
     ]
     

   • Two packets from ibc-1 to ibc-2 from source channel channel-1

     hermes tx raw ft-transfer ibc-2 ibc-1 transfer channel-1 9999 -o 1000 -n 2
     

     Success: [
         SendPacket(
             SendPacket {
                 height: revision: 1, height: 3076,
                 packet: PortId("transfer") ChannelId("channel-1") Sequence(3),
             },
         ),
         SendPacket(
             SendPacket {
                 height: revision: 1, height: 3076,
                 packet: PortId("transfer") ChannelId("channel-1") Sequence(4),
             },
         ),
     ]
     

7. Observe the output on the relayer terminal, verify that the send events are processed, and that the recv_packets are sent out.

   (...)
   
   INFO ibc_relayer::link: [ibc-0 -> ibc-1] result events:
       UpdateClientEv(ev_h:1-3048, 07-tendermint-0(0-3057), )
       WriteAcknowledgementEv(h:1-3048, seq:3, path:channel-0/transfer->channel-0/transfer, toh:1-4045, tos:0))
       WriteAcknowledgementEv(h:1-3048, seq:4, path:channel-0/transfer->channel-0/transfer, toh:1-4045, tos:0))
   INFO ibc_relayer::link: [ibc-0 -> ibc-1] success
   
   (...)
   
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] clearing old packets
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] received from query_txs []
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] finished clearing pending packets
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] generate messages from batch with 2 events
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] scheduling op. data with 2 msg(s) for Destination chain (height 1-3049)
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] relay op. data to Destination, proofs height 1-3048, (delayed by: 2.154603ms) [try 1/10]
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] prepending Destination client update @ height 1-3049
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] assembled batch of 3 message(s)
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] result events:
       UpdateClientEv(ev_h:0-3059, 07-tendermint-0(1-3049), )
       AcknowledgePacketEv(h:0-3059, seq:3, path:channel-0/transfer->channel-0/transfer, toh:1-4045, tos:0))
       AcknowledgePacketEv(h:0-3059, seq:4, path:channel-0/transfer->channel-0/transfer, toh:1-4045, tos:0))
   INFO ibc_relayer::link: [ibc-1 -> ibc-0] success
   
   (...)
   

8. Query the unreceived packets and acknowledgments on ibc-1 and ibc-2 from a different terminal:

   hermes query packet unreceived-packets ibc-1 transfer channel-0
   hermes query packet unreceived-acks ibc-0 transfer channel-0
   hermes query packet unreceived-packets ibc-2 transfer channel-0
   hermes query packet unreceived-acks ibc-1 transfer channel-1
   

   If everything went well, each of these commands should result in:

   Success: []
   

Connecting the chains

In the rest of this section we will show how to create the clients, establish a connection and a channel between the two chains, and relay packets over the channel. But first, make sure you followed the steps in the start the local chains and Identifiers section

Steps to start relaying packets between the two local chains

In order to start relaying packets please follow the steps below:

• Configure Clients
• Connection Handshake
• Open the Channel
• Relay Packets

1. Configuring clients

1.1. create client

First you will need to create a client for each chain:

This command submits a transaction to a destination chain (ibc-0) with a request to create a client for a source chain (ibc-1):

hermes tx raw create-client ibc-0 ibc-1

if the command is successful a message similar to the one below will be displayed status:success:

{
    Success: CreateClient(
        CreateClient(
            Attributes {
                height: Height { revision: 0, height: 43 },
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                client_type: Tendermint,
                consensus_height: Height { revision: 1, height: 32 },
            },
        ),
    )
}

Please note the client_id value returned. You will need that for other commands.

You can also execute a query to view the client state on destination chain ibc-0 by specifying the client_id value 07-tendermint-0:

hermes query client state ibc-0 07-tendermint-0

which show a message similar to the one below:

Success: ClientState {
    chain_id: ChainId {
        id: "ibc-1",
        version: 1,
    },
    trust_level: TrustThresholdFraction {
        numerator: 1,
        denominator: 3,
    },
    trusting_period: 1209600s,
    unbonding_period: 1814400s,
    max_clock_drift: 3s,
    frozen_height: Height {
        revision: 0,
        height: 0,
    },
    latest_height: Height {
        revision: 1,
        height: 38,
    },
    upgrade_path: [
        "upgrade",
        "upgradedIBCState",
    ],
    allow_update_after_expiry: true,
    allow_update_after_misbehaviour: true,
}

Now let's do the same for ibc-1 as the destination chain:

hermes tx raw create-client ibc-1 ibc-0

Take note of the client_id allocated for this client. In the examples we assume is 07-tendermint-1 (this client identity is obtained by creating two clients on ibc-1 for ibc-0).

As before, if the command is successful a message with status:success is displayed:

Success: CreateClient(
    CreateClient(
        Attributes {
            height: Height {
                revision: 1,
                height: 135,
            },
            client_id: ClientId(
                "07-tendermint-1",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 0,
                height: 145,
            },
        },
    ),
)

1.2 update-client

Client states can be updated by sending an update-client transaction:

hermes tx raw update-client ibc-0 07-tendermint-0

hermes tx raw update-client ibc-1 07-tendermint-1

Next Steps

In the next section, we'll establish the Connection Handshake

2. Connection Handshake

2.1 conn-init

Initialize a new connection on ibc-0:

hermes tx raw conn-init ibc-0 ibc-1 07-tendermint-0 07-tendermint-1

Take note of the ID allocated by the chain, e.g. connection-0 on ibc-0 in order to use it in the conn-try command below.

2.2 conn-try

Send a connection try to ibc-1:

hermes tx raw conn-try ibc-1 ibc-0 07-tendermint-1 07-tendermint-0 -s connection-0

Take note of the ID allocated by the chain, e.g. connection-1 on ibc-1. Use in the conn-ack CLI

2.3 conn-ack

Send a connection open acknowledgment to ibc-0:

hermes tx raw conn-ack ibc-0 ibc-1 07-tendermint-0 07-tendermint-1 -d connection-0 -s connection-1

2.4 conn-confirm

Send the open confirmation to ibc-1:

hermes tx raw conn-confirm ibc-1 ibc-0 07-tendermint-1 07-tendermint-0 -d connection-1 -s connection-0

2.5 query connection

To verify that the two ends are in Open state:

hermes query connection end ibc-0 connection-0

hermes query connection end ibc-1 connection-1

Next Steps

In the next section, we'll establish a new channel

3. Channel Handshake

3.1 chan-open-init

Initialize a new unordered channel on ibc-0:

hermes tx raw chan-open-init ibc-0 ibc-1 connection-0 transfer transfer -o UNORDERED

3.2 chan-open-try

Send a channel open try to ibc-1:

hermes tx raw chan-open-try ibc-1 ibc-0 connection-1 transfer transfer -s channel-0

Take note of the ID allocated by the chain, e.g. channel-1 on ibc-1. Use in the chan-open-ack CLI

3.3 chan-open-ack

Send a channel open acknowledgment to ibc-0:

hermes tx raw chan-open-ack ibc-0 ibc-1 connection-0 transfer transfer -d channel-0 -s channel-1

3.4 chan-open-confirm

Send the open confirmation to ibc-1:

hermes tx raw chan-open-confirm ibc-1 ibc-0 connection-1 transfer transfer -d channel-1 -s channel-0

3.5 query channel

To verify that the two ends are in Open state:

hermes query channel end ibc-0 transfer channel-0

hermes query channel end ibc-1 transfer channel-1

Next Steps

In the next section, we'll start to relay packets

4. Relay Packets

4.1 Query balances

• balance at ibc-0

  gaiad --node tcp://localhost:26657 query bank balances $(gaiad --home data/ibc-0 keys --keyring-backend="test" show user -a)
  

• balance at ibc-1

  gaiad --node tcp://localhost:26557 query bank balances $(gaiad --home data/ibc-1 keys --keyring-backend="test" show user -a)
  

Note that the addresses used in the two commands above are configured in dev-env.

4.2 Packet relaying

First, we'll send 9999 samoleans from ibc-0 to ibc-1.

• start the transfer of 9999 samoleans from ibc-0 to ibc-1. This sends a MsgTransfer in a transaction to ibc-0

  hermes tx raw ft-transfer ibc-1 ibc-0 transfer channel-0 9999 -o 1000 -n 1 -d samoleans
  

• query packet commitments on ibc-0

  hermes query packet commitments ibc-0 transfer channel-0
  

• query unreceived packets on ibc-1

  hermes query packet unreceived-packets ibc-1 transfer channel-1
  

• send recv_packet to ibc-1

  hermes tx raw packet-recv ibc-1 ibc-0 transfer channel-0
  

• query unreceived acks on ibc-0

  hermes query packet unreceived-acks ibc-0 transfer channel-0
  

• send acknowledgement to ibc-0

  hermes tx raw packet-ack ibc-0 ibc-1 transfer channel-1
  

Send those samoleans back, from ibc-1 to ibc-0.

hermes tx raw ft-transfer ibc-0 ibc-1 transfer channel-1 9999 -o 1000 -n 1 -d ibc/49D321B40FCF56B0370E5673CF090389C8E9CD185209FBE1BEE5D94E58E69BDC
hermes tx raw packet-recv ibc-0 ibc-1 transfer channel-1
hermes tx raw packet-ack  ibc-1 ibc-0 transfer channel-0

The ibc/49D321B40FCF56B0370E5673CF090389C8E9CD185209FBE1BEE5D94E58E69BDC denominator above can be obtained by querying the balance at ibc-1 after the transfer from ibc-0 to ibc-1 is concluded.

Next we will test the packet timeouts.

• send 1 packet with low timeout height offset to ibc-0

  hermes tx raw ft-transfer ibc-1 ibc-0 transfer channel-0 9999 -o 2 -n 1
  

• send timeout to ibc-0

  hermes tx raw packet-recv ibc-1 ibc-0 transfer channel-0
  

• send 1 packet with 2 second timeout to ibc-0

  hermes tx raw ft-transfer ibc-1 ibc-0 transfer channel-0 9999 -t 2 -n 1
  

• send timeout to ibc-0

  hermes tx raw packet-recv ibc-1 ibc-0 transfer channel-0
  

Commands

The Commands section presents the commands current available in Hermes

Sections

Keys

Commands to manage keys (private keys) for each chain.

Config

Commands to manage configuration file, in particular to validate it.

Path Setup

Commands to manage clients, connections, channels.

Relaying

Commands to start the relayer and relay packets.

Listen Mode

Commands to listen for IBC events

Upgrade

Commands to perform client upgrade

Monitor

Commands to monitor clients and submit evidence of misbehaviour

Queries

Commands to execute queries on configured chains

Raw Transactions

Commands to submit individual transactions to configured chains

Global options

Hermes accepts global options which affect all commands.

hermes 0.15.0
Informal Systems <hello@informal.systems>
Implementation of `hermes`, an IBC Relayer developed in Rust.

FLAGS:
    -c, --config CONFIG       path to configuration file
    -j, --json                enable JSON output

The flags must be specified right after the hermes command and before any subcommand.

Example

To start the relayer using the configuration file at /home/my_chain.toml and enable JSON output:

hermes -c /home/my_chain.toml --json start

JSON output

If the --json option is supplied, all relayer commands will output single-line JSON values instead of plain text.

Log messages will be written to stderr, while the final result will be written to stdout, and everything will be formatted as JSON. This allows processing only the final output using jq. To process all the output using jq, one can redirect stderr to stdout with hermes --json COMMAND 2>&1 | jq.

Example

hermes -c /home/my_chain.toml --json create client ibc-0 ibc-1

{"timestamp":"Apr 13 20:46:31.921","level":"INFO","fields":{"message":"Using default configuration from: '.hermes/config.toml'"},"target":"ibc_relayer_cli::commands"}
{"timestamp":"Apr 13 20:46:31.961","level":"INFO","fields":{"message":"running listener","chain.id":"ibc-1"},"target":"ibc_relayer::event::monitor"}
{"timestamp":"Apr 13 20:46:31.989","level":"INFO","fields":{"message":"running listener","chain.id":"ibc-0"},"target":"ibc_relayer::event::monitor"}
{"result":{"CreateClient":{"client_id":"07-tendermint-1","client_type":"Tendermint","consensus_height":{"revision_height":10060,"revision_number":1},"height":{"revision_height":10072,"revision_number":0}}},"status":"success"}

The first three lines are printed to stderr, while the last line with a "result" key is printed to stdout.

Example

To improve the readability, pipe all of the output to jq:

hermes -c /home/my_chain.toml --json create client ibc-0 ibc-1 2>&1 | jq

{
  "timestamp": "Apr 13 20:52:26.060",
  "level": "INFO",
  "fields": {
    "message": "Using default configuration from: '.hermes/config.toml'"
  },
  "target": "ibc_relayer_cli::commands"
}
{
  "timestamp": "Apr 13 20:52:26.082",
  "level": "INFO",
  "fields": {
    "message": "running listener",
    "chain.id": "ibc-1"
  },
  "target": "ibc_relayer::event::monitor"
}
{
  "timestamp": "Apr 13 20:52:26.088",
  "level": "INFO",
  "fields": {
    "message": "running listener",
    "chain.id": "ibc-0"
  },
  "target": "ibc_relayer::event::monitor"
}
{
  "result": {
    "CreateClient": {
      "client_id": "07-tendermint-5",
      "client_type": "Tendermint",
      "consensus_height": {
        "revision_height": 10364,
        "revision_number": 1
      },
      "height": {
        "revision_height": 10375,
        "revision_number": 0
      }
    }
  },
  "status": "success"
}

Example

To extract the identifer of the newly created client above:

hermes -c /home/my_chain.toml --json create client ibc-0 ibc-1 | jq '.result.CreateClient.client_id'

"07-tendermint-2"

Adding Keys to the Relayer

WARNING: Currently the relayer does NOT support a keyring store to securely store the private key file. The key file will be stored on the local file system in the user $HOME folder under $HOME/.hermes/keys/

BREAKING: As of Hermes v0.2.0, the format of the keys stored on disk has changed, and keys which had been previously configured must now be re-imported using either the keys add or the keys restore commands.

Using the keys command you can add and list keys.

Show usage

To see the available sub-commands for the keys command run:

hermes help keys

Currently there are two sub-commands supported add and list:

USAGE:
    hermes keys <SUBCOMMAND>

DESCRIPTION:
    Manage keys in the relayer for each chain

SUBCOMMANDS:
    help       Get usage information
    add        Adds a key to a configured chain
    delete     Delete key(s) from a configured chain
    list       List keys configured on a chain
    restore    restore a key to a configured chain using a mnemonic

Key Seed file (Private Key)

In order to execute the command below you need a private key file (JSON). The relayer uses the private key file to sign the transactions submitted to the chain.

The private key file can be obtained by using the keys add on a Cosmos chain. For example, the command for gaiad is:

# The `key_name` parameter is the name of the key that will be found in the json output
# For example, in the "Two Local Chains" tutorial, we use "testkey".
gaiad keys add <key_name> --output json

The command outputs a JSON similar to the one below.

{
  "name": "testkey",
  "type": "local",
  "address": "cosmos1tc3vcuxyyac0dmayf887t95tdg7qpyql48w7gj",
  "pubkey": "cosmospub1addwnpepqgg7ng4ycm60pdxfzdfh4hjvkwcr3da59mr8k883vsstx60ruv7kur4525u",
  "mnemonic": "[24 words mnemonic]"
}

You can save this to a file (e.g. key_seed.json) and use it to add to the relayer with hermes keys add <chain_id> -f key_seed.json. See the Adding Keys section for more details.

Adding Keys

Add a private key to a chain from a key file

    hermes keys add <OPTIONS>

DESCRIPTION:
    Adds a key to a configured chain

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain

FLAGS:
    -f, --file FILE           path to the key file
    -n, --name NAME           name of the key (defaults to the `key_name` defined in the config)
    -p, --hd-path HD-PATH     derivation path for this key (default: m/44'/118'/0'/0/0)

To add a private key file to a chain:

hermes -c config.toml keys add [CHAIN_ID] -f [PRIVATE_KEY_FILE]

If the command is successful a message similar to the one below will be displayed:

Success: Added key testkey ([ADDRESS]) on [CHAIN ID] chain

Key name: By default, the key will be named after the key_name property specified in the configuration file. To use a different key name, specify the --name option when invoking keys add.

hermes -c config.toml keys add [CHAINID] -f [PRIVATE_KEY_FILE] -n [KEY_NAME]

Restore a private key to a chain from a mnemonic

USAGE:
    hermes keys restore <OPTIONS>

DESCRIPTION:
    restore a key to a configured chain using a mnemonic

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain

FLAGS:
    -m, --mnemonic MNEMONIC   mnemonic to restore the key from
    -n, --name NAME           name of the key (defaults to the `key_name` defined in the config)
    -p, --hd-path HD-PATH     derivation path for this key (default: m/44'/118'/0'/0/0)

To restore a key from its mnemonic:

hermes -c config.toml keys restore [CHAIN_ID] -m "[MNEMONIC]"

or using an explicit derivation path, for example an Ethereum coin type (used for Evmos, Injective, Umee, Cronos, and possibly other networks):

hermes -c config.toml keys restore --mnemonic <MNEMONIC> --hd-path "m/44'/60'/0'/0/0" <CHAIN_ID>

If the command is successful a message similar to the one below will be displayed:

Success: Restore key testkey ([ADDRESS]) on [CHAIN ID] chain

Key name: By default, the key will be named after the key_name property specified in the configuration file. To use a different key name, specify the --name option when invoking keys restore.

hermes -c config.toml keys restore [CHAINID] -m "[MNEMONIC]" -n [KEY_NAME]

Delete keys

In order to delete the private keys added to chains use the keys delete command

USAGE:
    hermes keys delete <OPTIONS>

DESCRIPTION:
    Delete key(s) from a configured chain

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain

FLAGS:
    -n, --name NAME           name of the key
    -a, --all                 delete all keys

Delete private keys that was previously added to a chain

To delete a single private key by name:

hermes -c config.toml keys delete [CHAIN_ID] -n [KEY_NAME]

Alternatively, to delete all private keys added to a chain:

hermes -c config.toml keys delete [CHAIN_ID] -a

List keys

In order to list the private keys added to chains use the keys list command

USAGE:
    hermes keys list <OPTIONS>

DESCRIPTION:
    List keys configured on a chain

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain

Listing the private key that was added to a chain

To list the private key file that was added to a chain:

hermes -c config.toml keys list [CHAIN_ID]

If the command is successful a message similar to the one below will be displayed:

Success:
- user2 (cosmos1attn9fxrcvjz483w3tu4cfz77ldmlyujly3q3k)
- testkey (cosmos1dw88vdekeeuta5u50p6n5lt5v5c6y2we0pu8nz)

JSON:

hermes --json -c config.toml keys list [CHAIN_ID] | jq

If the command is successful a message similar to the one below will be displayed:

{
  "result": {
    "testkey": {
      "account": "cosmos1dw88vdekeeuta5u50p6n5lt5v5c6y2we0pu8nz",
      "address": [ 107, 142, 118, 55, 54, 206, 120, 190, 211, 148, 120, 117, 58, 125, 116, 101, 49, 162, 41, 217 ],
      "coin_type": 118,
      "private_key": "(snip)",
      "public_key": "xpub6Gc7ZUt2q1BiQYjhUextPv5bZLwosHigZYqEquPD6FkAGmHDrLiBgE5Xnh8XGZp79rAXtZn1Dt3DNQHxxgCgVQqfRMfVsRiXn6mwULBnYq7"
    },
    "user2": {
      "account": "cosmos1attn9fxrcvjz483w3tu4cfz77ldmlyujly3q3k",
      "address": [ 234, 215, 50, 164, 195, 195, 36, 42, 158, 46, 138, 249, 92, 36, 94, 247, 219, 191, 147, 146 ],
      "coin_type": 118,
      "private_key": "(snip)",
      "public_key": "xpub6FmDbeGTWVjSvHrqHfrpnMTZxpPX1V7XFiq5nMuvgwX9jumt1yUuwNAUQo8Nn36unbFShg6iSjkfMBgeY49wik7rF91N2SHvarpX62ByWMf"
    }
  },
  "status": "success"
}

Config

Use the config validate command to perform a quick syntactic validation of your configuration file.

USAGE:
    hermes config validate <OPTIONS>

DESCRIPTION:
    validate the relayer configuration

Example

Validate the default config file, the path inferred automatically to be $HOME/.hermes/config.toml.

hermes config validate

hermes config validate
Jul 12 16:31:07.017  INFO using default configuration from '$HOME/.hermes/config.toml'
Success: "validation passed successfully"

Validate a config file at an arbitrary location:

hermes -c ./config.toml config validate

This one fails validation because we mistakenly added two separate sections for the same chain ibc-1:

hermes -c ./config.toml  config validate
error: hermes fatal error: config error: config file has duplicate entry for the chain with id ibc-1

Path Setup

This section describes a number of commands that can be used to manage clients, connections, channels.

Create

Use the create commands to create new clients, connections, and channels.

USAGE:
    hermes create <SUBCOMMAND>

DESCRIPTION:
    Create objects (client, connection, or channel) on chains

SUBCOMMANDS:
    help       Get usage information
    client     Create a new IBC client
    connection Create a new connection between two chains
    channel    Create a new channel between two chains

Update

Use the update commands to update a client.

USAGE:
    hermes update <SUBCOMMAND>

DESCRIPTION:
    Update objects (clients) on chains

SUBCOMMANDS:
    help       Get usage information
    client     Update an IBC client

Client

Table of Contents

• Create Client
• Update Client

Create Client

Use the create client command to create a new client on a destination chain, tracking the state of the source chain.

USAGE:
    hermes create client [OPTIONS] <DST_CHAIN_ID> <SRC_CHAIN_ID>

ARGS:
    <DST_CHAIN_ID>
            identifier of the destination chain

    <SRC_CHAIN_ID>
            identifier of the source chain

OPTIONS:
    -d, --clock-drift <CLOCK_DRIFT>
            The maximum allowed clock drift for this client.

            The clock drift is a correction parameter. It helps deal with clocks that are only
            approximately synchronized between the source and destination chains of this client. The
            destination chain for this client uses the clock drift parameter when deciding to accept
            or reject a new header (originating from the source chain) for this client. If this
            option is not specified, a suitable clock drift value is derived from the chain
            configurations.

    -p, --trusting-period <TRUSTING_PERIOD>
            Override the trusting period specified in the config.

            The trusting period specifies how long a validator set is trusted for (must be shorter
            than the chain's unbonding period).

    -t, --trust-threshold <TRUST_THRESHOLD>
            Override the trust threshold specified in the configuration.

            The trust threshold defines what fraction of the total voting power of a known and
            trusted validator set is sufficient for a commit to be accepted going forward.

Example

Create a new client on ibc-0 which tracks ibc-1:

hermes create client ibc-0 ibc-1

    CreateClient(
        Attributes {
            height: Height {
                revision: 0,
                height: 286,
            },
            client_id: ClientId(
                "07-tendermint-0",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 1,
                height: 274,
            },
        },
    ),
)

A new client is created with identifier 07-tendermint-1

Update Client

Use the update client command to update an existing client with a new consensus state. Specific update and trusted heights can be specified.

USAGE:
    hermes update client [OPTIONS] <DST_CHAIN_ID> <DST_CLIENT_ID>

ARGS:
    <DST_CHAIN_ID>     identifier of the destination chain
    <DST_CLIENT_ID>    identifier of the client to be updated on destination chain

OPTIONS:
    -h, --help                               Print help information
    -H, --target-height <TARGET_HEIGHT>      the target height of the client update
    -t, --trusted-height <TRUSTED_HEIGHT>    the trusted height of the client update

Update client with latest header

the client on ibc-0 with latest header of ibc-1:

hermes update client ibc-0 07-tendermint-9

Success: UpdateClient(
    UpdateClient {
        common: Attributes {
            height: Height { revision: 0, height: 303 },
            client_id: ClientId(
                "07-tendermint-1",
            ),
            client_type: Tendermint,
            consensus_height: Height { revision: 1, height: 293 },
        },
        header: Some(
            Tendermint(
                 Header {...},
            ),
        ),
    },
)

The client with identifier 07-tendermint-1 has been updated with the consensus state at height 1-293.

Update a client to a specific target height

hermes update client ibc-0 07-tendermint-1 --target-height 320 --trusted-height 293

Success: UpdateClient(
    UpdateClient {
        common: Attributes {
            height: Height { revision: 0, height: 555 },
            client_id: ClientId(
                "07-tendermint-1",
            ),
            client_type: Tendermint,
            consensus_height: Height { revision: 1, height: 320 },
        },
        header: Some(
            Tendermint(
                 Header {...},
            ),
        ),
    },
)

The client with identifier 07-tendermint-1 has been updated with the consensus state at height 1-320, as specified.

Connection

Table of Contents

• Establish Connection
• Examples
  • New connection over new clients
  • New connection over existing clients
• Non-zero Delay Connection

Establish Connection

Use the create connection command to create a new connection.

USAGE:
    hermes create connection <OPTIONS>

DESCRIPTION:
    Create a new connection between two chains

POSITIONAL ARGUMENTS:
    chain_a_id                identifier of the side `a` chain for the new connection
    chain_b_id                identifier of the side `b` chain for the new connection

FLAGS:
    --client-a CLIENT-A       identifier of client hosted on chain `a`; default: None (creates a new client)
    --client-b CLIENT-B       identifier of client hosted on chain `b`; default: None (creates a new client)
    --delay DELAY             delay period parameter for the new connection (seconds) (default: 0)

Examples

New connection over new clients

Create a new connection between ibc-0 and ibc-1 over new clients:

hermes create connection ibc-0 ibc-1

🥂  ibc-0 => OpenInitConnection(
    OpenInit(
        Attributes {
            height: Height { revision: 0, height: 4073 },
            connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-8",
            ),
            counterparty_connection_id: None,
            counterparty_client_id: ClientId(
                "07-tendermint-8",
            ),
        },
    ),
)

🥂  ibc-1 => OpenTryConnection(
    OpenTry(
        Attributes {
            height: Height { revision: 1, height: 4069 },
            connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-8",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-8",
            ),
        },
    ),
)

🥂  ibc-0 => OpenAckConnection(
    OpenAck(
        Attributes {
            height: Height { revision: 0, height: 4081 },
            connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-8",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-8",
            ),
        },
    ),
)

🥂  ibc-1 => OpenConfirmConnection(
    OpenConfirm(
        Attributes {
            height: Height { revision: 1, height: 4073 },
            connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-8",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-8",
            ),
        },
    ),
)

🥂🥂🥂  Connection handshake finished for [Connection {
    delay_period: 0s,
    a_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-8",
        ),
        connection_id: ConnectionId(
            "connection-8",
        ),
    },
    b_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-8",
        ),
        connection_id: ConnectionId(
            "connection-8",
        ),
    },
}]

Success: Connection {
    delay_period: 0s,
    a_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-8",
        ),
        connection_id: ConnectionId(
            "connection-8",
        ),
    },
    b_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-8",
        ),
        connection_id: ConnectionId(
            "connection-8",
        ),
    },
}

New connection over existing clients

Create a new connection between ibc-0 and ibc-1 over existing clients, both with client id 07-tendermint-0:

hermes create connection ibc-0 --client-a 07-tendermint-0 --client-b
07-tendermint-0

Notice that one can omit the destination chain parameter, as Hermes will automatically figure it out by looking up the given client on ibc-0.

Non-zero Delay Connection

A connection can be created with a delay period parameter. This parameter specifies a period of time that must elpase after a successful client state update and before a packet with proofs using its commitment root can pe processed on chain. For more information see how packet delay works and the connection delay specification.

Channel

Table of Contents

• Establish Channel
• Examples
  • New channel over an existing connection
  • New channel over a new connection

Establish Channel

Use the create channel command to establish a new channel.

USAGE:
    hermes create channel [OPTIONS] --port-a <PORT_A> --port-b <PORT_B> <CHAIN_A> [CONNECTION_A]

DESCRIPTION:
    Create a new channel between two chains using a pre-existing connection.
    Alternatively, create a new client and a new connection underlying the new channel if a pre-existing connection is not provided.

POSITIONAL ARGUMENTS:
    <CHAIN_A>                Identifier of the side `a` chain for the new channel
    <CONNECTION_A>           Identifier of the connection on chain `a` to use in creating the new channel

FLAGS:
    -c, --chain-b <CHAIN_B>         Identifier of the side `b` chain for the new channel
    -h, --help                         Print help information
        --new-client-connection        Indicates that a new client and connection will be created underlying the new channel
    -o, --order <ORDER>                The channel ordering, valid options 'unordered' (default) and 'ordered' [default: ORDER_UNORDERED]
        --port-a <PORT_A>              Identifier of the side `a` port for the new channel
        --port-b <PORT_B>              Identifier of the side `b` port for the new channel
    -v, --channel-version <VERSION>    The version for the new channel

Examples

New channel over an existing connection

This is the preferred way to create a new channel, by leveraging an existing connection.

Create a new unordered channel between ibc-0 and ibc-1 over an existing connection, specifically the one we just created in the example above, with port name transfer on both sides:

hermes create channel ibc-0 --connection-a connection-0 --port-a transfer --port-b transfer -o unordered

Notice that one can omit the destination chain parameter, as Hermes will automatically figure it out by looking up the given connection on ibc-0.

🥳  ibc-0 => OpenInitChannel(
    OpenInit(
        Attributes {
            height: Height { revision: 0, height: 129 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-1")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: None
        }
    )
)
🥳  ibc-1 => OpenTryChannel(
    OpenTry(
        Attributes {
            height: Height { revision: 1, height: 126 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-1")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-1"))
        }
    )
)
🥳  ibc-0 => OpenAckChannel(
    OpenAck(
        Attributes {
            height: Height { revision: 0, height: 137 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-1")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-1"))
        }
    )
)
🥳  ibc-1 => OpenConfirmChannel(
    OpenConfirm(
        Attributes {
            height: Height { revision: 1, height: 129 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-1")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-1"))
        }
    )
)
🥳  🥳  🥳  Channel handshake finished for Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-1",
        ),
    },
    b_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-1",
        ),
    },
    connection_delay: 0s,
}
Success: Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-1",
        ),
    },
    b_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-1",
        ),
    },
    connection_delay: 0s,
}

New channel over a new connection

Should you specifically want to create a new client and a new connection as part of the create channel flow, that option exists, though this is the less-preferred option over the previous flow, as creating new clients and connections should only be done in certain specific circumstances so as not to create redundant resources.

Create a new unordered channel between ibc-0 and ibc-1 over a new connection, using port name transfer on both sides and accepting the interactive prompt that pops up notifying you that a new client and a new connection will be initialized as part of the process:

hermes create channel ibc-0 --chain-b ibc-1 --port-a transfer --port-b transfer -o unordered --new-client-connection

🥂  ibc-0 => OpenInitConnection(
    OpenInit(
        Attributes {
            height: Height { revision: 0, height: 66 },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: None,
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

🥂  ibc-1 => OpenTryConnection(
    OpenTry(
        Attributes {
            height: Height { revision: 1, height: 64 },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

🥂  ibc-0 => OpenAckConnection(
    OpenAck(
        Attributes {
            height: Height { revision: 0, height: 76 },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

🥂  ibc-1 => OpenConfirmConnection(
    OpenConfirm(
        Attributes {
            height: Height { revision: 1, height: 68 },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

🥂🥂🥂  Connection handshake finished for [Connection {
    delay_period: 0s,
    a_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
    },
    b_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
    },
}]

🥳  ibc-0 => OpenInitChannel(
    OpenInit(
        Attributes {
            height: Height { revision: 0, height: 78 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-0")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: None
        }
    )
)

🥳  ibc-1 => OpenTryChannel(
    OpenTry(
        Attributes {
            height: Height { revision: 1, height: 70 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-0")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-0"))
        }
    )
)

🥳  ibc-0 => OpenAckChannel(
    OpenAck(
        Attributes {
            height: Height { revision: 0, height: 81 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-0")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-0"))
        }
    )
)

🥳  ibc-1 => OpenConfirmChannel
    OpenConfirm
        Attributes {
            height: Height { revision: 1, height: 73 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-0")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-0"))
        }
    )
)

🥳  🥳  🥳  Channel handshake finished for Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-0",
        ),
    },
    b_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-0",
        ),
    },
    connection_delay: 0s,
}

Success: Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-0",
        ),
    },
    b_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-0",
        ),
    },
    connection_delay: 0s,
}

A new channel with identifier channel-0 on both sides has been established on a new connection with identifier connection-0 on both sides.

Relaying

This section describes the types of relaying that hermes can perform.

Hermes can send transactions triggered by IBC events. It currently handles channel handshake and packet events:

• packets messages only
• channel and packet messages

The start Command

The start command can be used to start hermes in IBC event listen mode.

USAGE:
    hermes start <OPTIONS>

DESCRIPTION:
    Start the relayer in multi-chain mode. Relays packets and channel handshake messages between all chains in the config.

As described in next sub-sections, the type of relaying can be configured in the global section of the configuration file, by specifying different values in strategy field.

Packet Relaying

This section describes the configuration and commands that can be used to start the relayer and relay packets over one or multiple paths.

Table of Contents

• The start Command
• Packet Streaming
• Packet Delay

The start Command

To relay packets only configure the mode section of the configuration file like so:

[global]
log_level = 'info'

[mode]

[mode.clients]
enabled = true
# ...

[mode.connections]
enabled = false

[mode.channels]
enabled = false

[mode.packets]
enabled = true
# ...

Then start hermes using the start command:

hermes start

The relayer sends packet transactions triggered by IBC packet events for all open channels between the configured chains. This is also referred to packet streaming.

Packet Streaming

After the relayer is started using the start command, it listens to IBC packet events emitted by any of the configured chains. Assuming the events are coming from a source chain, the relayer builds packets based on these events, packets that are then sent either to the source chain or the counterparty (destination) chain.

Current events and actions are:

• send_packet: the relayer builds a packet message with the packet obtained from the event and any required proofs obtained from the counterparty of the chain where the message is sent. The concrete packet is:
  • MsgRecvPacket, sent to destination chain if the channel is in open state on the destination chain, and a timeout has not occurred,
  • MsgTimeout, sent to the source chain if the channel is in open state on the destination chain, but a timeout has occurred.
  • MsgTimeoutOnClose, sent to the source chain if the channel is in closed state on the destination chain.
• write_acknowledgement: the relayer builds a MsgAcknowledgement packet that is sent to the destination chain.

In addition to these events, the relayer will also handle channel closing events:

• chan_close_init: the relayer builds a MsgChannelCloseConfirm and sends it to the destination chain

Packet Delay

If the relay path is using a non-zero delay connection, then hermes will delay all packet transactions. The delay is relative to the submission time for the client update at the height required by the packet proof. The delay is used to prevent light client attacks and ensures that misbehavior detection finalizes before the transaction is submitted. For more information on the misbehavior detector see the misbehaviour section.

Relaying of Handshake Messages

This section describes the configuration and commands that can be used to start the relayer and relay both handshake and packets for connections and channels.

The start Command

To relay packets and handshake messages configure the mode section of the configuration file like so:

[global]
log_level = 'info'

[mode]

[mode.clients]
enabled = true
# ...

[mode.connections]
enabled = true

[mode.channels]
enabled = true

[mode.packets]
enabled = true
# ...

Then start hermes using the start command:

hermes start

The relayer sends handshake and packet transactions triggered by IBC events.

Completing Channel Handshakes

After the relayer is started using the start command, it scans the chain state and will resume the handshake for any channels or connections that are not in open state. It then listens to IBC events emitted by any of the configured chains.

Assuming the events are coming from a source chain, the relayer determines the destination chain and builds the handshake messages based on these events. These are then sent to the destination chain.

In addition to the events described in Packet Relaying, the following IBC events may be handled:

• Channels (if mode.channels.enabled=true):

  • chan_open_init: the relayer builds a MsgChannelOpenTry message
  • chan_open_try: the relayer builds a MsgChannelOpenAck message
  • chan_open_ack: the relayer builds a MsgChannelOpenConfirm message
  • chan_open_confirm: no message is sent out, channel opening is finished

• Connections (if mode.connections.enabled=true):

  • conn_open_init: the relayer builds a MsgConnOpenTry message
  • conn_open_try: the relayer builds a MsgConnOpenAck message
  • conn_open_ack: the relayer builds a MsgConnOpenConfirm message
  • conn_open_confirm: no message is sent out, connection opening is finished

Clearing Packets

clear packets

This command clears outstanding packets on a given channel in both directions, by issuing the appropriate packet-recvs and packet-acks.

Usage

Clear outstanding packets (i.e. packet-recv and packet-ack) on a given channel in both directions.

The channel is identified by the chain, port, and channel IDs at one of its ends

USAGE:
    hermes clear packets <CHAIN_ID> <PORT_ID> <CHANNEL_ID>

ARGS:
    <CHAIN_ID>      identifier of the chain
    <PORT_ID>       identifier of the port
    <CHANNEL_ID>    identifier of the channel

OPTIONS:
    -h, --help    Print help information

Example

1. Without Hermes running, send 3 packets over a channel, here channel-13:

❯ hermes tx raw ft-transfer ibc1 ibc0 transfer channel-13 9999 -o 1000 -n 3
2022-02-24T14:16:28.295526Z  INFO ThreadId(01) using default configuration from '/Users/coromac/.hermes/config.toml'
2022-02-24T14:16:28.330860Z  INFO ThreadId(15) send_tx{id=ibc0}: refresh: retrieved account sequence=61 number=1
2022-02-24T14:16:28.350022Z  INFO ThreadId(15) wait_for_block_commits: waiting for commit of tx hashes(s) AE4C3186778488E45670EB7303FA77E69B39F4E7C7494B05EC51E55136A373D6 id=ibc0
Success: [
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 86208,
            },
            packet: Packet {
                sequence: Sequence(
                    14,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 86208,
            },
            packet: Packet {
                sequence: Sequence(
                    15,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 86208,
            },
            packet: Packet {
                sequence: Sequence(
                    16,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
]

2. Because the relayer is not running these packets won't be relayed, as can be seen with the query packet unreceived-packets command:

❯ hermes query packet unreceived-packets ibc1 transfer channel-13
2022-02-24T14:21:28.874190Z  INFO ThreadId(01) using default configuration from '/Users/coromac/.hermes/config.toml'
Success: [
    14,
    15,
    16,
]

3. We can clear them manually using the clear packets command:

❯ hermes clear packets ibc0 transfer channel-13
2022-02-24T14:17:25.748422Z  INFO ThreadId(01) using default configuration from '/Users/coromac/.hermes/config.toml'
2022-02-24T14:17:25.799704Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}: found unprocessed SendPacket events for [Sequence(14), Sequence(15), Sequence(16)] (first 10 shown here; total=3)
2022-02-24T14:17:25.827177Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}: ready to fetch a scheduled op. data with batch of size 3 targeting Destination
2022-02-24T14:17:26.504798Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}:relay{odata=E96CV_cA5P ->Destination @0-86218; len=3}: assembled batch of 4 message(s)
2022-02-24T14:17:26.508873Z  INFO ThreadId(29) send_tx{id=ibc1}: refresh: retrieved account sequence=54 number=1
2022-02-24T14:17:26.561715Z  INFO ThreadId(29) wait_for_block_commits: waiting for commit of tx hashes(s) 07AA83524257105CC476063932A560893BE8F4E94C679BFD00F970FC248647E0 id=ibc1
2022-02-24T14:17:31.948950Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}:relay{odata=E96CV_cA5P ->Destination @0-86218; len=3}: [Sync->ibc1] result events:
    UpdateClientEv(h: 0-86215, cs_h: 07-tendermint-3(0-86219))
    WriteAcknowledgementEv(WriteAcknowledgement - h:0-86215, seq:14, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))
    WriteAcknowledgementEv(WriteAcknowledgement - h:0-86215, seq:15, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))
    WriteAcknowledgementEv(WriteAcknowledgement - h:0-86215, seq:16, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))


2022-02-24T14:17:31.949192Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}:relay{odata=E96CV_cA5P ->Destination @0-86218; len=3}: success
2022-02-24T14:17:31.989215Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}: found unprocessed WriteAcknowledgement events for [Sequence(14), Sequence(15), Sequence(16)] (first 10 shown here; total=3)
2022-02-24T14:17:32.013500Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}: ready to fetch a scheduled op. data with batch of size 3 targeting Destination
2022-02-24T14:17:33.211930Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}:relay{odata=L4fnSXkxL_ ->Destination @0-86215; len=3}: assembled batch of 4 message(s)
2022-02-24T14:17:33.215803Z  INFO ThreadId(15) send_tx{id=ibc0}: refresh: retrieved account sequence=62 number=1
2022-02-24T14:17:33.245229Z  INFO ThreadId(15) wait_for_block_commits: waiting for commit of tx hashes(s) 62C69B1C46AF45182D5D99C6CB5EB301F8A402726772BA4EE067B18C68F2A4D6 id=ibc0
2022-02-24T14:17:37.465489Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}:relay{odata=L4fnSXkxL_ ->Destination @0-86215; len=3}: [Sync->ibc0] result events:
    UpdateClientEv(h: 0-86221, cs_h: 07-tendermint-3(0-86216))
    AcknowledgePacketEv(h:0-86221, seq:14, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))
    AcknowledgePacketEv(h:0-86221, seq:15, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))
    AcknowledgePacketEv(h:0-86221, seq:16, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))


2022-02-24T14:17:37.465802Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}:relay{odata=L4fnSXkxL_ ->Destination @0-86215; len=3}: success
Success: [
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 0,
                    height: 86215,
                },
                client_id: ClientId(
                    "07-tendermint-3",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 0,
                    height: 86219,
                },
            },
            header: Some(
                Tendermint(
                     Header {...},
                ),
            ),
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 0,
                height: 86215,
            },
            packet: Packet {
                sequence: Sequence(
                    14,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
            ack: [ ... ],
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 0,
                height: 86215,
            },
            packet: Packet {
                sequence: Sequence(
                    15,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
            ack: [ ... ],
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 0,
                height: 86215,
            },
            packet: Packet {
                sequence: Sequence(
                    16,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
            ack: [ ... ],
        },
    ),
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 0,
                    height: 86221,
                },
                client_id: ClientId(
                    "07-tendermint-3",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 0,
                    height: 86216,
                },
            },
            header: Some(
                Tendermint(
                     Header {...},
                ),
            ),
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 86221,
            },
            packet: Packet {
                sequence: Sequence(
                    14,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 86221,
            },
            packet: Packet {
                sequence: Sequence(
                    15,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 86221,
            },
            packet: Packet {
                sequence: Sequence(
                    16,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
]

4. The packets have now been successfully relayed:

❯ hermes query packet unreceived-packets ibc1 transfer channel-13
2022-02-24T14:21:28.874190Z  INFO ThreadId(01) using default configuration from '/Users/coromac/.hermes/config.toml'
Success: []

Relayer Listen Mode

The relayer can be started in listen mode to display the events emitted by a given chain. NewBlock and Tx IBC events are shown.

USAGE:
    hermes listen <OPTIONS>

DESCRIPTION:
    Listen to and display IBC events emitted by a chain

POSITIONAL ARGUMENTS:
    chain_id                  Identifier of the chain to listen for events from

FLAGS:
    -e, --event EVENT         Add an event type to listen for, can be repeated. Listen for all events by default (available: Tx, NewBlock)

Example

Start the relayer in listen mode for all ibc-0 events and observe the output:

hermes listen ibc-0

EventBatch {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    height: block::Height(10914),
    events: [
        NewBlock(
            NewBlock {
                height: block::Height(10914),
            },
        ),
    ],
}
EventBatch {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    height: block::Height(10915),
    events: [
        OpenInitConnection(
            OpenInit(
                Attributes {
                    height: block::Height(10915),
                    connection_id: Some(
                        ConnectionId(
                            "connection-3",
                        ),
                    ),
                    client_id: ClientId(
                        "07-tendermint-3",
                    ),
                    counterparty_connection_id: None,
                    counterparty_client_id: ClientId(
                        "07-tendermint-5",
                    ),
                },
            ),
        ),
    ],

...

EventBatch {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    height: block::Height(10919),
    events: [
        UpdateClient(
            UpdateClient(
                Attributes {
                    height: block::Height(10919),
                    client_id: ClientId(
                        "07-tendermint-3",
                    ),
                    client_type: Tendermint,
                    consensus_height: Height {
                        revision: 1,
                        height: 10907,
                    },
                },
            ),
        ),
    ],
}

...

EventBatch {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    height: block::Height(10924),
    events: [
        UpdateClient(
            UpdateClient(
                Attributes {
                    height: block::Height(10924),
                    client_id: ClientId(
                        "07-tendermint-3",
                    ),
                    client_type: Tendermint,
                    consensus_height: Height {
                        revision: 1,
                        height: 10912,
                    },
                },
            ),
        ),
        OpenAckConnection(
            OpenAck(
                Attributes {
                    height: block::Height(10924),
                    connection_id: Some(
                        ConnectionId(
                            "connection-3",
                        ),
                    ),
                    client_id: ClientId(
                        "07-tendermint-3",
                    ),
                    counterparty_connection_id: Some(
                        ConnectionId(
                            "connection-5",
                        ),
                    ),
                    counterparty_client_id: ClientId(
                        "07-tendermint-5",
                    ),
                },
            ),
        ),
    ],
}

Filter events

The listen command accepts an --event flag to specify which event types to listen for.

At the moment, two event types are available:

• NewBlock
• Tx

The --event flag can be repeated to specify more than one event type.

• To listen for only NewBlock events on ibc-0, invoke hermes listen ibc-0 --event NewBlock
• To listen for only Tx events on ibc-0, invoke hermes listen ibc-0 --event Tx
• To listen for both NewBlock and Tx events on ibc-0, invoke hermes listen ibc-0 --e NewBlock --event Tx

If the --event flag is omitted, the relayer will subscribe to all event types.

Client Upgrade

Client Upgrade Command

Use the upgrade client command to upgrade a client after a chain upgrade.

USAGE:
    hermes upgrade client <OPTIONS>

DESCRIPTION:
    Upgrade an IBC client

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    dst_client_id             identifier of the client to be upgraded on destination chain

Example

Here is an example of a chain upgrade proposal submission and client upgrade.

Testing Client Upgrade

Prerequisites

• gaiad (v4.2.*), for example:

gaiad version --log_level error --long | head -n4

name: gaia
server_name: gaiad
version: v4.2.0
commit: 535be14a8bdbfeb0d950914b5baa2dc72c6b081c

Testing procedure

1. Start two gaia instances and initialize hermes:

   ./scripts/dev-env ~/.hermes/config.toml ibc-0 ibc-1
   

   The one-chain script is invoked for each chain and modifies the genesis.json file to use a short window for governance proposals (200s for max_deposit_period and voting_period). Therefore, an upgrade proposal can be submitted, voted on and accepted within a short time.

2. Create one client on ibc-1 for ibc-0:

   hermes create client ibc-1 ibc-0
   

   Success: CreateClient(
      CreateClient(
          Attributes {
              height: Height { revision: 1, height: 9 },
              client_id: ClientId(
                  "07-tendermint-0",
              ),
              client_type: Tendermint,
              consensus_height: Height { revision: 0, height: 18 },
          },
      ),
   )
   

3. Create and submit an upgrade plan for chain ibc-0:

   Use the hermes test command to make an upgrade proposal. In the example below a software upgrade proposal is made for ibc-0, for the height 300 blocks from latest height. 10000000stake is deposited. The proposal includes the upgraded client state constructed from the state of 07-tendermint-0 client on ibc-1 that was created in the previous step. In addition, the unbonding_period of the client is set to some new value (400h)

   hermes tx raw upgrade-chain ibc-0 ibc-1 07-tendermint-0 10000000 300
   

   Success: transaction::Hash(CE98D8D98091BA8016BD852D18056E54C4CB3C4525E7F40DD3C40B4FD0F2482B)
   

   Note that the height offset should be picked such that the proposal plan height is reached after the 200s voting period.

4. Verify that the proposal was accepted:

   Query the upgrade plan to check that it was submitted correctly. Note the height at which the proposal will take effect (chain halts). Also status: PROPOSAL_STATUS_VOTING_PERIOD.

   gaiad query gov proposal 1 --home data/ibc-0/
   

   content:
     '@type': /cosmos.upgrade.v1beta1.SoftwareUpgradeProposal
     description: upgrade the chain software and unbonding period
     plan:
       height: "332"
       info: upgrade the chain software and unbonding period
       name: test
       time: "0001-01-01T00:00:00Z"
       upgraded_client_state:
         '@type': /ibc.lightclients.tendermint.v1.ClientState
         allow_update_after_expiry: true
         allow_update_after_misbehaviour: true
         chain_id: ibc-0
         frozen_height:
           revision_height: "0"
           revision_number: "0"
         latest_height:
           revision_height: "333"
           revision_number: "0"
         max_clock_drift: 0s
         proof_specs:
         - inner_spec:
             child_order:
             - 0
             - 1
             child_size: 33
             empty_child: null
             hash: SHA256
             max_prefix_length: 12
             min_prefix_length: 4
           leaf_spec:
             hash: SHA256
             length: VAR_PROTO
             prefix: AA==
             prehash_key: NO_HASH
             prehash_value: SHA256
           max_depth: 0
           min_depth: 0
         - inner_spec:
             child_order:
             - 0
             - 1
             child_size: 32
             empty_child: null
             hash: SHA256
             max_prefix_length: 1
             min_prefix_length: 1
           leaf_spec:
             hash: SHA256
             length: VAR_PROTO
             prefix: AA==
             prehash_key: NO_HASH
             prehash_value: SHA256
           max_depth: 0
           min_depth: 0
         trust_level:
           denominator: "0"
           numerator: "0"
         trusting_period: 0s
         unbonding_period: 1440000s
         upgrade_path:
         - upgrade
         - upgradedIBCState
     title: upgrade_ibc_clients
   deposit_end_time: "2021-04-12T16:33:37.187389Z"
   final_tally_result:
     abstain: "0"
     "no": "0"
     no_with_veto: "0"
     "yes": "0"
   proposal_id: "1"
   status: PROPOSAL_STATUS_VOTING_PERIOD
   submit_time: "2021-04-12T16:30:17.187389Z"
   total_deposit:
   - amount: "10000000"
     denom: stake
   voting_end_time: "2021-04-12T16:33:37.187389Z"
   voting_start_time: "2021-04-12T16:30:17.187389Z"
   

5. Vote on the proposal

   The parameter 1 should match the proposal_id: from the upgrade proposal submitted at step 3. This command must be issued while the proposal status is PROPOSAL_STATUS_VOTING_PERIOD. Confirm transaction when prompted.

   gaiad tx gov vote 1 yes --home data/ibc-0/data/ --keyring-backend test --keyring-dir data/ibc-0/ --chain-id ibc-0 --from validator
   

   confirm transaction before signing and broadcasting [y/N]: y
   
   {"height":"85","txhash":"AC24D80B1BFE0832769DECFDD3B3DF999A363D5E4390B0B673344FFDED9150B2","codespace":"","code":0,"data":"0A060A04766F7465","raw_log":"[{\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"vote\"},{\"key\":\"module\",\"value\":\"governance\"},{\"key\":\"sender\",\"value\":\"cosmos1srfzw0jkyyn7wf0ps4zy0tuvdaclfj2ufgp6w3\"}]},{\"type\":\"proposal_vote\",\"attributes\":[{\"key\":\"option\",\"value\":\"VOTE_OPTION_YES\"},{\"key\":\"proposal_id\",\"value\":\"1\"}]}]}]","logs":[{"msg_index":0,"log":"","events":[{"type":"message","attributes":[{"key":"action","value":"vote"},{"key":"module","value":"governance"},{"key":"sender","value":"cosmos1srfzw0jkyyn7wf0ps4zy0tuvdaclfj2ufgp6w3"}]},{"type":"proposal_vote","attributes":[{"key":"option","value":"VOTE_OPTION_YES"},{"key":"proposal_id","value":"1"}]}]}],"info":"","gas_wanted":"200000","gas_used":"43716","tx":null,"timestamp":""}
   

6. Wait approximately 200 seconds until the proposal changes status to PROPOSAL_STATUS_PASSED. Note the final tally_result that includes the vote submitted in the previous step.

   gaiad query gov proposal 1 --home data/ibc-0/
   

      content:
        '@type': /cosmos.upgrade.v1beta1.SoftwareUpgradeProposal
        description: upgrade the chain software and unbonding period
      ...
      final_tally_result:
        abstain: "0"
        "no": "0"
        no_with_veto: "0"
        "yes": "100000000000"
      proposal_id: "1"
      status: PROPOSAL_STATUS_PASSED
      submit_time: "2021-04-12T16:30:17.187389Z"
      total_deposit:
      - amount: "10000000"
        denom: stake
      voting_end_time: "2021-04-12T16:33:37.187389Z"
      voting_start_time: "2021-04-12T16:30:17.187389Z"
   

7. Test the upgrade client CLI

   The following command performs the upgrade for client 07-tendermint-0. It outputs two events, one for the updated client state, and another for the upgraded state.

   hermes upgrade client ibc-1 07-tendermint-0
   

   Success: [
       UpdateClient(
           UpdateClient {
               common: Attributes {
                   height: Height { revision: 1, height: 438 },
                   client_id: ClientId(
                       "07-tendermint-0",
                   ),
                   client_type: Tendermint,
                   consensus_height: Height { revision: 0, height: 440 },
               },
               header: Some(
                   Tendermint(..)
               ),
           },
       ),
       UpgradeClient(
           UpgradeClient(
               Attributes {
                   height: Height { revision: 1, height: 438 },
                   client_id: ClientId(
                       "07-tendermint-0",
                   ),
                   client_type: Tendermint,
                   consensus_height: Height { revision: 0, height: 441 },
               },
           ),
       ),
   ]
   

Misbehaviour

Table of Contents

• Monitoring Misbehaviour and Evidence Submission

Monitoring Misbehaviour and Evidence Submission

Use the mishbehaviour command to monitor the updates for a given client, detect certain types of misbehaviour and submit evidence to the chain. If the evidence passes the on-chain validation, the client is frozen. Further packets cannot be relayed using the frozen client.

USAGE:
    hermes misbehaviour <OPTIONS>

DESCRIPTION:
    Listen to client update IBC events and handles misbehaviour

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain where client updates are monitored for misbehaviour
    client_id                 identifier of the client to be monitored for misbehaviour

The misbehaviour monitor starts by analyzing all headers used in prior client updates. Once finished it registers for update client events and checks any new headers for misbehaviour. If it detects evidence of misbehaviour, it submits a transaction with the evidence to the chain. If the chain validates the transaction then the monitor exits.

This is an experimental feature.

The following types of misbehaviour are handled:

1. Fork

   Assumes at least one consensus state before the fork point exists. Let existing consensus states on chain B be: [Sn,.., Sf, Sf-1, S0] with Sf-1 being the most recent state before the fork. Chain A is queried for a header Hf' at Sf.height and if it is different than the Hf in the event for the client update (the one that has generated Sf on chain), then the two headers are included in the evidence and submitted. Note that in this case the headers are different but have the same height.

2. BFT time violation for an unavailable header

   Some header with a height that is higher than the latest height on chain A has been accepted and a consensus state was created on B. Note that this implies that the timestamp of this header must be within the clock_drift of the client. Assume the client on B has been updated with h2(not present on/ produced by chain A) and it has a timestamp of t2 that is at most clock_drift in the future. Then the latest header from A is fetched, let it be h1, with a timestamp of t1. If t1 >= t2 then evidence of misbehavior is submitted to A.

Example

The hermes misbehaviour outputs an error message displaying MISBEHAVIOUR DETECTED:

hermes misbehaviour ibc-0 07-tendermint-0

Apr 13 20:04:03.347  INFO ibc_relayer::foreign_client: checking misbehaviour for consensus state heights [Height { revision: 1, height: 195 }, Height { revision: 1, height: 85 }, Height { revision: 1, height: 28 }]
Apr 13 20:04:04.425 ERROR ibc_relayer::foreign_client: MISBEHAVIOUR DETECTED ClientId("07-tendermint-0") h1: Height { revision: 1, height: 195 }-Height { revision: 1, height: 85 } h2: Height { revision: 1, height: 195 }-Height { revision: 1, height: 85 }, sending evidence
Apr 13 20:04:05.070  INFO ibc_relayer_cli::commands::misbehaviour: evidence submission result [ClientMisbehaviour(ClientMisbehaviour(Attributes { height: Height { revision: 0, height: 1521 }, client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 1, height: 195 } }))]

Success: Some(
    ClientMisbehaviour(
        ClientMisbehaviour(
            Attributes {
                height: Height {
                    revision: 0,
                    height: 1521,
                },
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 1,
                    height: 195,
                },
            },
        ),
    ),
)

Querying client state from this point will show the client is in frozen state, with frozen_height indicating the height at which the client was frozen:

hermes query client state ibc-0 07-tendermint-0 | jq

{
  "result": {
    "allow_update_after_expiry": true,
    "allow_update_after_misbehaviour": true,
    "chain_id": "ibc-1",
    "frozen_height": {
      "revision_height": 16,
      "revision_number": 1
    },
    "latest_height": {
      "revision_height": 16,
      "revision_number": 1
    },
    "max_clock_drift": {
      "nanos": 0,
      "secs": 3
    },
    "trust_level": {
      "denominator": "3",
      "numerator": "1"
    },
    "trusting_period": {
      "nanos": 0,
      "secs": 1209600
    },
    "unbonding_period": {
      "nanos": 0,
      "secs": 1814400
    },
    "upgrade_path": [
      "upgrade",
      "upgradedIBCState"
    ]
  },
  "status": "success"
}

Queries

Hermes supports querying for different objects that exist on a configured chain.

The query command provides the following sub-commands:

Usage

USAGE:
    hermes query <SUBCOMMAND>

DESCRIPTION:
    Query objects from the chain

SUBCOMMANDS:
    client         Query information about clients
    clients        Query clients
    connection     Query information about connections
    connections    Query the identifiers of all connections on a chain
    channel        Query information about channels
    channels       Query the identifiers of all channels on a given chain
    packet         Query information about packets

Table of Contents

• Query Clients
• Query Client Data
  • Query the client state
  • Query the client consensus state
  • Query the identifiers of all connections associated with a given client
  • Query for the header used in a client update at a certain height

Query Clients

Use the query clients command to query the identifiers of all clients on a given chain.

USAGE:
    hermes query clients <OPTIONS>

DESCRIPTION:
    Query the identifiers of all clients on a chain

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query

FLAGS:
    -s, --src-chain-id ID     filter for clients which target a specific chain id (implies '-o')
    -o, --omit-chain-ids      omit printing the source chain for each client (default: false)

Example

Query all clients on ibc-1:

hermes query clients ibc-1

Success: [
    ClientChain {
        client_id: ClientId(
            "07-tendermint-0",
        ),
        chain_id: ChainId {
            id: "ibc-0",
            version: 0,
        },
    },
    ClientChain {
        client_id: ClientId(
            "07-tendermint-1",
        ),
        chain_id: ChainId {
            id: "ibc-2",
            version: 2,
        },
    },
]

Query all clients on ibc-1 having ibc-2 as their source chain:

hermes query clients ibc-1 -s ibc-2

Success: [
    ClientId(
        "07-tendermint-1",
    ),
]

Query Client Data

Use the query client command to query the information about a specific client.

USAGE:
    hermes query client <SUBCOMMAND>

DESCRIPTION:
    Query information about clients

SUBCOMMANDS:
    state      query client full state
    consensus  query client consensus
    connections query client connections

Query the client state

Use the query client state command to query the client state of a client:

USAGE:
    hermes query client state <OPTIONS>

DESCRIPTION:
    Query client full state

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    client_id                 identifier of the client to query

FLAGS:
    -H, --height HEIGHT       the chain height which this query should reflect

Example

Query the state of client 07-tendermint-2 on ibc-1:

hermes query client state ibc-1 07-tendermint-1

Success: ClientState {
    chain_id: ChainId {
        id: "ibc-2",
        version: 2,
    },
    trust_level: TrustThresholdFraction {
        numerator: 1,
        denominator: 3,
    },
    trusting_period: 1209600s,
    unbonding_period: 1814400s,
    max_clock_drift: 3s,
    frozen_height: Height {
        revision: 0,
        height: 0,
    },
    latest_height: Height {
        revision: 2,
        height: 3069,
    },
    upgrade_path: [
        "upgrade",
        "upgradedIBCState",
    ],
    allow_update_after_expiry: true,
    allow_update_after_misbehaviour: true,
}

Query the client consensus state

Use the query client consensus command to query the consensus states of a given client, or the state at a specified height:

USAGE:
    hermes query client consensus <OPTIONS>

DESCRIPTION:
    Query client consensus state

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    client_id                 identifier of the client to query

FLAGS:
    -c, --consensus-height    CONSENSUS-HEIGHT
    -s, --heights-only        show only consensus heights
    -H, --height HEIGHT       the chain height context to be used, applicable only to a specific height

Example

Query the states of client 07-tendermint-0 on ibc-0:

hermes query client consensus ibc-0 07-tendermint-0 --heights-only

Success: [
    Height {
        revision: 1,
        height: 3049,
    },
    Height {
        revision: 1,
        height: 2888,
    },
    Height {
        revision: 1,
        height: 2736,
    },
    Height {
        revision: 1,
        height: 2729,
    },
    Height {
        revision: 1,
        height: 2724,
    },
    Height {
        revision: 1,
        height: 2717,
    },
]

Query ibc-0 at height 2800 for the consensus state for height 2724:

hermes query client consensus ibc-0 07-tendermint-0 -c 2724 -h 2800

Success: ConsensusState {
    timestamp: Time(
        2021-04-13T14:11:20.969154Z
    ),
    root: CommitmentRoot(
        "371DD19003221B60162D42C78FD86ABF95A572F3D9497084584B75F97B05B70C"
    ),
    next_validators_hash: Hash::Sha256(
        740950668B6705A136D041914FC219045B1D0AD1C6A284C626BF5116005A98A7
    ),
}

Query the identifiers of all connections associated with a given client

Use the query client connections command to query the connections associated with a given client:

USAGE:
    hermes query client connections <OPTIONS>

DESCRIPTION:
    Query client connections

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    client_id                 identifier of the client to query

FLAGS:
    -H, --height HEIGHT       the chain height which this query should reflect

Example

Query the connections of client 07-tendermint-0 on ibc-0:

hermes query client connections ibc-0 07-tendermint-0

Success: [
    ConnectionId("connection-0"),
    ConnectionId("connection-1"),
]

Query for the header used in a client update at a certain height

USAGE:
    hermes query client header <OPTIONS>

DESCRIPTION:
    Query for the header used in a client update at a certain height

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    client_id                 identifier of the client to query
    consensus_height          height of header to query

FLAGS:
    -H, --height HEIGHT       the chain height context for the query

Example

Query for the header used in the 07-tendermint-0 client update at height 2724 on ibc-0:

hermes query client header ibc-0 07-tendermint-0 2724

Success: [
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 0,
                    height: 0,
                },
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 1,
                    height: 2724,
                },
            },
            header: Some(
                Tendermint(...),
            ),
        },
    ),
]

Table of Contents

• Query Connections
• Query Connection Data
  • Query the connection end data
  • Query the identifiers of all channels associated with a given connection

Query Connections

Use the query connections command to query the identifiers of all connections on a given chain.

USAGE:
    hermes query connections <OPTIONS>

DESCRIPTION:
    Query the identifiers of all connections on a chain

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query

Example

Query all connections on ibc-1:

hermes query connections ibc-1

Success: [
    ConnectionId(
        "connection-0",
    ),
    ConnectionId(
        "connection-1",
    ),
]

Query Connection Data

Use the query connection commands to query a specific connection.

USAGE:
    hermes query connection <SUBCOMMAND>

DESCRIPTION:
    Query information about connection(s)

SUBCOMMANDS:
    end        query connection end
    channels   query connection channels

Query the connection end data

Use the query connection end command to query the connection end:

USAGE:
    hermes query connection end <OPTIONS>

DESCRIPTION:
    query connection end

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    connection_id             identifier of the connection to query

FLAGS:
    -H, --height HEIGHT       height of the state to query

Example

Query the connection end of connection connection-1 on ibc-1:

hermes query connection end ibc-1 connection-1

Success: ConnectionEnd {
    state: Open,
    client_id: ClientId(
        "07-tendermint-1",
    ),
    counterparty: Counterparty {
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: Some(
            ConnectionId(
                "connection-0",
            ),
        ),
        prefix: ibc,
    },
    versions: [
        Version {
            identifier: "1",
            features: [
                "ORDER_ORDERED",
                "ORDER_UNORDERED",
            ],
        },
    ],
    delay_period: 0s,
}

Query the identifiers of all channels associated with a given connection

Use the query connection channels command to query the identifiers of the channels associated with a given connection:

USAGE:
    hermes query connection channels <OPTIONS>

DESCRIPTION:
    query connection channels

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    connection_id             identifier of the connection to query

Example

Query the channels associated with connection connection-1 on ibc-1:

hermes query connection channels ibc-1 connection-1

Success: [
    PortChannelId {
        channel_id: ChannelId(
            "channel-0",
        ),
        port_id: PortId(
            "transfer",
        ),
    },
    PortChannelId {
        channel_id: ChannelId(
            "channel-1",
        ),
        port_id: PortId(
            "transfer",
        ),
    },
]

Table of Contents

• Query Channels
• Query Channel Data
  • Query the channel end data
  • Query the channel data for both ends of a channel

Query Channels

Use the query channels command to query the identifiers of all channels on a given chain.

USAGE:
    hermes query channels <OPTIONS>

DESCRIPTION:
    Query the identifiers of all channels on a given chain

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query

Example

Query all channels on ibc-1:

hermes query channels ibc-1

Success: [
    PortChannelId {
        channel_id: ChannelId(
            "channel-0",
        ),
        port_id: PortId(
            "transfer",
        ),
    },
    PortChannelId {
        channel_id: ChannelId(
            "channel-1",
        ),
        port_id: PortId(
            "transfer",
        ),
    },
]

Query Channel Data

Use the query channel commands to query the information about a specific channel.

USAGE:
    hermes query channel <SUBCOMMAND>

DESCRIPTION:
    Query information about channels

SUBCOMMANDS:
    end        Query channel end
    ends       Query channel ends and underlying connection and client objects

Query the channel end data

Use the query channel end command to query the channel end:

USAGE:
    hermes query channel end <OPTIONS>

DESCRIPTION:
    Query channel end

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    port_id                   identifier of the port to query
    channel_id                identifier of the channel to query

FLAGS:
    -H, --height HEIGHT       height of the state to query

Example

Query the channel end of channel channel-1 on port transfer on ibc-1:

hermes query channel end ibc-1 transfer channel-1

Success: ChannelEnd {
    state: Open,
    ordering: Unordered,
    remote: Counterparty {
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-0",
            ),
        ),
    },
    connection_hops: [
        ConnectionId(
            "connection-1",
        ),
    ],
    version: "ics20-1",
}

Query the channel data for both ends of a channel

Use the query channel ends command to obtain both ends of a channel:

USAGE:
    hermes query channel ends <OPTIONS>

DESCRIPTION:
    Query channel ends and underlying connection and client objects

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    port_id                   identifier of the port to query
    channel_id                identifier of the channel to query

FLAGS:
    -H, --height HEIGHT       height of the state to query
    -v, --verbose             enable verbose output, displaying all details of channels, connections & clients

Example

Query the channel end of channel channel-1 on port transfer on ibc-0:

hermes query channel ends ibc-0 transfer channel-1

Success: ChannelEndsSummary {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    client_id: ClientId(
        "07-tendermint-1",
    ),
    connection_id: ConnectionId(
        "connection-1",
    ),
    channel_id: ChannelId(
        "channel-1",
    ),
    port_id: PortId(
        "transfer",
    ),
    counterparty_chain_id: ChainId {
        id: "ibc-2",
        version: 2,
    },
    counterparty_client_id: ClientId(
        "07-tendermint-1",
    ),
    counterparty_connection_id: ConnectionId(
        "connection-1",
    ),
    counterparty_channel_id: ChannelId(
        "channel-1",
    ),
    counterparty_port_id: PortId(
        "transfer",
    ),
}

Passing the -v flag will additionally print all the details of the channel, connection, and client on both ends.

Packet Queries

Use the query packet commands to query information about packets.

USAGE:
    hermes query packet <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    ack                   Query packet acknowledgment
    acks                  Query packet acknowledgments
    commitment            Query packet commitment
    commitments           Query packet commitments
    pending               Output a summary of pending packets in both directions
    unreceived-acks       Query unreceived acknowledgments
    unreceived-packets    Query unreceived packets
    help                  Print this message or the help of the given subcommand(s)

Table of Contents

• Pending Packets
• Packet Commitments
• Packet Commitment with Sequence
• Packet Acknowledgments
• Packet Acknowledgment with Sequence
• Unreceived Packets
• Unreceived Acknowledgments

Pending Packets

Use the query packet pending command to query the sequence numbers of all packets that have not yet been received or acknowledged, at both ends of a channel.

USAGE:
    hermes query packet pending <CHAIN_ID> <PORT_ID> <CHANNEL_ID>

ARGS:
    <CHAIN_ID>      identifier of the chain at one end of the channel
    <PORT_ID>       port identifier on the chain given by <CHAIN_ID>
    <CHANNEL_ID>    channel identifier on the chain given by <CHAIN_ID>

Example

Query the sequence numbers of all packets that either not yet been received or not yet been acknowledged, at both ends of the channel channel-1.

$ hermes query packet pending ibc-0 tranfer channel-1

Success: Summary {
    forward: PendingPackets {
        unreceived_packets: [
            2203,
            ...
            2212,
        ],
        unreceived_acks: [
           2183,
           ...
           2202,
        ],
    },
    reverse: PendingPackets {
        unreceived_packets: [
           14,
           ...
           23,
        ],
        unreceived_acks: [
           4,
           ...
           13,
        ],
    },
}

Packet Commitments

Use the query packet commitments command to query the sequence numbers of all packets that have been sent but not yet acknowledged (these are the packets that still have their commitments stored).

USAGE:
    hermes query packet commitments <OPTIONS>

DESCRIPTION:
    Query packet commitments

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    port_id                   identifier of the port to query
    channel_id                identifier of the channel to query

Example

Query ibc-0 for the sequence numbers of packets that still have commitments on ibc-0 and that were sent on transfer port and channel-0:

hermes query packet commitments ibc-0 transfer channel-0

Success: PacketSeqs {
    height: Height {
        revision: 0,
        height: 9154,
    },
    seqs: [
        1,
        2,
        3
    ],
}

Packet Commitment with Sequence

Use the query packet commitment command to query the commitment value of a packet with a given sequence number.

USAGE:
    hermes query packet commitment <OPTIONS>

DESCRIPTION:
    Query packet commitment

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    port_id                   identifier of the port to query
    channel_id                identifier of the channel to query
    sequence                  sequence of packet to query

FLAGS:
    -H, --height HEIGHT       height of the state to query

Example

Query ibc-0 for the commitment of packet with sequence 3 sent on transfer port and channel-0:

hermes query packet commitment ibc-0 transfer channel-0 3

Success: "F9458DC7EBEBCD6D18E983FCAB5BD752CC2A74532BBD50B812DB229997739EFC"

Packet Acknowledgments

Use the query packet acknowledgments command to query the sequence numbers of all packets that have been acknowledged.

USAGE:
    hermes query packet acks <OPTIONS>

DESCRIPTION:
    Query packet acknowledgments

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    port_id                   identifier of the port to query
    channel_id                identifier of the channel to query

Example

Query ibc-1 for the sequence numbers of packets acknowledged that were received on transfer port and channel-1:

hermes query packet acks ibc-1 transfer channel-1

Success: PacketSeqs {
    height: Height {
        revision: 1,
        height: 9547,
    },
    seqs: [
        1,
        2,
        3
    ],
}

Packet Acknowledgment with Sequence

Use the query packet acknowledgment command to query the acknowledgment value of a packet with a given sequence number.

USAGE:
    hermes query packet ack <OPTIONS>

DESCRIPTION:
    Query packet acknowledgment

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    port_id                   identifier of the port to query
    channel_id                identifier of the channel to query
    sequence                  sequence of packet to query

FLAGS:
    -H, --height HEIGHT       height of the state to query

Example

Query ibc-1 for the acknowledgment of packet with sequence 2 received on transfer port and channel-1:

hermes query packet ack ibc-1 transfer channel-1 2

Success: "08F7557ED51826FE18D84512BF24EC75001EDBAF2123A477DF72A0A9F3640A7C"

Unreceived Packets

Use the query packet unreceived-packets command to query the sequence numbers of all packets that have been sent on the source chain but not yet received on the destination chain.

USAGE:
    hermes query packet unreceived-packets <OPTIONS>

DESCRIPTION:
    Query unreceived packets

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain for the unreceived sequences
    port_id                   port identifier
    channel_id                channel identifier

Example

Query transfer port and channel-1 on ibc-1 for the sequence numbers of packets sent on ibc-0 but not yet received:

hermes query packet unreceived-packets ibc-1 transfer channel-1

Success: [
    1,
    2,
    3
]

Unreceived Acknowledgments

Use the query packet unreceived-acks command to query the sequence numbers of all packets that have not yet been acknowledged.

USAGE:
    hermes query packet unreceived-acks <OPTIONS>

DESCRIPTION:
    Query unreceived acknowledgments

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query the unreceived acknowledgments
    port_id                   port identifier
    channel_id                channel identifier

Example

Query transfer port and channel-0 on ibc-0 for the sequence numbers of packets received by ibc-1 but not yet acknowledged on ibc-0:

hermes query packet unreceived-acks ibc-0 transfer channel-0

Success: [
    1,
    2,
    3
]

Tx Queries

Use the query tx command to query information about transaction(s).

USAGE:
    hermes query tx <SUBCOMMAND>

DESCRIPTION:
    Query information about transactions

SUBCOMMANDS:
    events     Query the events emitted by transaction

Table of Contents

• Transaction Events

Transaction Events

Use the query tx events command to obtain a list of events that a chain generated as a consequence of delivering a transaction.

USAGE:
    hermes query tx events <OPTIONS>

DESCRIPTION:
    Query the events emitted by transaction

POSITIONAL ARGUMENTS:
    chain_id                  identifier of the chain to query
    hash                      transaction hash to query

Example

Query chain ibc-0 for the events emitted due to transaction with hash 6EDBBCBCB779F9FC9D6884ACDC4350E69720C4B362E4ACE6C576DE792F837490:

hermes query tx events ibc-0 6EDBBCBCB779F9FC9D6884ACDC4350E69720C4B362E4ACE6C576DE792F837490

Success: [
    SendPacket(
        SendPacket {
            height: Height {
                revision: 4,
                height: 6628239,
            },
            packet: PortId("transfer") ChannelId("channel-139") Sequence(2),
        },
    ),
]

Raw Transactions

There are a number of simple commands that perform minimal validation, build and send IBC transactions.

The tx raw command provides the following sub-commands:

The main purpose of these commands is to support development and testing, and continuous integration. These CLIs take quite a few parameters and they are explained in the individual sub-sections.

At a high level, most commands follow this template:

hermes tx raw <ibc-datagram> <dst-chain-id> <src-chain-id> [-d <dst-obj-id> -s <src-obj-id>]*

In the command template above:

• ibc-datagram - identifies the "main" IBC message that is being sent, e.g. conn-init, conn-try, chan-open-init, etc. To ensure successful processing on the receiving chain, the majority of these commands build and send two messages: one UpdateClient message followed by the actual IBC message. These two messages are included in a single transaction. This is done for all IBC datagrams that include proofs collected from the source chain.

  The messages that do not require proofs are:

  • MsgCreateClient (create-client command),
  • MsgConnectionOpenInit (conn-open-init command),
  • MsgChannelOpenInit (chan-open-init command),
  • MsgChannelCloseInit (chan-close-init command) and
  • MsgTransfer (ft-transfer command)

• dst-chain-id - is the identifier of the chain where the transaction will be sent.

• src-chain-id - is the identifier of the chain that is queried for the data that is included in the transaction, e.g. connection data, client proofs, etc. To ensure correct on-chain state, the relayer also queries the destination chain, however it does not include this information in the Tx to the destination chain.

• dst-obj-id - the identifier of an object on destination chain required by the datagram, e.g. the client-id associated with the connection on destination chain in connection datagrams. Or the connection-id in a ConnOpenAck datagram.

• src-obj-id - the identifier of an object on the source chain, required by the datagram, e.d. the client-id of the connection on source chain.

• More details about the tx raw commands can be found in the following sections:

  • Client
  • Connection
  • Channel Open
  • Channel Close
  • Packet

Usage

USAGE:
    hermes tx raw <SUBCOMMAND>

DESCRIPTION:
    Raw commands for sending transactions to a configured chain.

SUBCOMMANDS:
    help                Get usage information
    create-client       Create a client for source chain on destination chain
    update-client       Update the specified client on destination chain
    conn-init           Initialize a connection (ConnectionOpenInit)
    conn-try            Relay the connection attempt (ConnectionOpenTry)
    conn-ack            Relay acknowledgment of a connection attempt (ConnectionOpenAck)
    conn-confirm        Confirm opening of a connection (ConnectionOpenConfirm)
    chan-open-init      Initialize a channel (ChannelOpenInit)
    chan-open-try       Relay the channel attempt (ChannelOpenTry)
    chan-open-ack       Relay acknowledgment of a channel attempt (ChannelOpenAck)
    chan-open-confirm   Confirm opening of a channel (ChannelOpenConfirm)
    chan-close-init     Initiate the closing of a channel (ChannelCloseInit)
    chan-close-confirm  Confirm the closing of a channel (ChannelCloseConfirm)
    ft-transfer         Send a fungible token transfer test transaction (ICS20 MsgTransfer)
    packet-recv         Relay receive or timeout packets
    packet-ack          Relay acknowledgment packets

Client

The tx raw commands can be used to create and update the on-chain IBC clients.

Table of Contents

• Create Client
• Update Client

Create Client

Use the create-client command to create a new client.

USAGE:
    hermes tx raw create-client <OPTIONS>

DESCRIPTION:
    Create a client for source chain on destination chain

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain

Example

Create a new client of ibc-1 on ibc-0:

hermes tx raw create-client ibc-0 ibc-1

{
    Success: CreateClient(
        CreateClient(
            Attributes {
                height: Height { revision: 0, height: 43 },
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                client_type: Tendermint,
                consensus_height: Height { revision: 1, height: 32 },
            },
        ),
    )
}

A new client is created with identifier 07-tendermint-0

Update Client

Use the update-client command to update an existing client with a new consensus state. Specific update and trusted heights can be specified.

USAGE:
    hermes tx raw update-client <OPTIONS>

DESCRIPTION:
    Update the specified client on destination chain

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    dst_client_id             identifier of the client to be updated on destination chain

FLAGS:
    -H, --target-height TARGET-HEIGHT
    -t, --trusted-height TRUSTED-HEIGHT

Example

Update the client on ibc-0 with latest header of ibc-1

hermes tx raw update-client ibc-0 07-tendermint-0

Success: UpdateClient(
    UpdateClient {
        common: Attributes {
            height: Height { revision: 0, height: 110 },
            client_id: ClientId(
                "07-tendermint-0",
            ),
            client_type: Tendermint,
            consensus_height: Height { revision: 1, height: 109 },
        },
        header: Some(
            Tendermint(
                 Header {...},
            ),
        ),
    },
)

The client with identifier 07-tendermint-0 has been updated with the consensus state at height 1-273.

Connection Handshake

The tx raw commands can be used to establish a connection between two clients.

sequenceDiagram
    autonumber
    participant A as ibc-1
    participant B as ibc-0
    Note over A, B: No connection
    A->>B: ConnectionOpenInit
    Note over B: connection: connection-0
    Note over B: counterparty: none
    B->>A: ConnectionOpenTry
    Note over A: connection: connection-1
    Note over A: counterparty: connection-0
    A->>B: ConnectionOpenAck
    note over B: connection: connection-0
    note over B: counterparty: connection-1
    B->>A: ConnectionOpenConfirm
    Note over A, B: Connection open

Table of Contents

• Connection Init
• Connection Try
• Connection Ack
• Connection Confirm

Connection Init

Use the conn-init command to initialize a new connection on a chain.

USAGE:
    hermes tx raw conn-init <OPTIONS>

DESCRIPTION:
    Initialize a connection (ConnectionOpenInit)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_client_id             identifier of the destination client
    src_client_id             identifier of the source client

Example

Given that two clients were previously created with identifier 07-tendermint-0 on chain ibc-0 and identifier 07-tendermint-1 on chain ibc-1, we can initialize a connection between the two clients.

First, let's initialize the connection on ibc-0:

hermes tx raw conn-init ibc-0 ibc-1 07-tendermint-0 07-tendermint-1

Success: OpenInitConnection(
    OpenInit(
        Attributes {
            height: Height {
                revision: 0,
                height: 73,
            },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: None,
            counterparty_client_id: ClientId(
                "07-tendermint-1",
            ),
        },
    ),
)

A new connection has been initialized on ibc-0 with identifier connection-0.

Note that the counterparty_connection_id field is currently empty.

Connection Try

Use the conn-try command to establish a counterparty to the connection on the other chain.

USAGE:
    hermes tx raw conn-try <OPTIONS>

DESCRIPTION:
    Relay the connection attempt (ConnectionOpenTry)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_client_id             identifier of the destination client
    src_client_id             identifier of the source client

FLAGS:
    -s, --src-conn-id ID      identifier of the source connection (required)

Example

Let's now create the counterparty to connection-0 on chain ibc-1:

hermes tx raw conn-try ibc-1 ibc-0 07-tendermint-1 07-tendermint-0 -s connection-0

Success: OpenTryConnection(
    OpenTry(
        Attributes {
            height: Height {
                revision: 1,
                height: 88,
            },
            connection_id: Some(
                ConnectionId(
                    "connection-1",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-1",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

A new connection has been created on ibc-1 with identifier connection-1.

Note that the field counterparty_connection_id points to the connection on ibc-0.

Connection Ack

Use the conn-ack command to acknowledge the connection on the initial chain.

USAGE:
    hermes tx raw conn-ack <OPTIONS>

DESCRIPTION:
    Relay acknowledgment of a connection attempt (ConnectionOpenAck)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_client_id             identifier of the destination client
    src_client_id             identifier of the source client

FLAGS:
    -d, --dst-conn-id ID      identifier of the destination connection (required)
    -s, --src-conn-id ID      identifier of the source connection (required)

Example

We can now acknowledge on ibc-0 that ibc-1 has accepted the connection attempt:

hermes tx raw conn-ack ibc-0 ibc-1 07-tendermint-0 07-tendermint-1 -d connection-0 -s connection-1

Success: OpenAckConnection(
    OpenAck(
        Attributes {
            height: Height {
                revision: 0,
                height: 206,
            },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-1",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-1",
            ),
        },
    ),
)

Note that the field counterparty_connection_id now points to the connection on ibc-1.

Connection Confirm

Use the conn-confirm command to confirm that the connection has been acknowledged, and finish the handshake, after which the connection is open on both chains.

USAGE:
    hermes tx raw conn-confirm <OPTIONS>

DESCRIPTION:
    Confirm opening of a connection (ConnectionOpenConfirm)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_client_id             identifier of the destination client
    src_client_id             identifier of the source client

FLAGS:
    -d, --dst-conn-id ID      identifier of the destination connection (required)
    -s, --src-conn-id ID      identifier of the source connection (required)

Example

Confirm on ibc-1 that ibc-0 has accepted the connection attempt.

hermes tx raw conn-confirm ibc-1 ibc-0 07-tendermint-1 07-tendermint-0 -d connection-1 -s connection-0

Success: OpenConfirmConnection(
    OpenConfirm(
        Attributes {
            height: Height {
                revision: 1,
                height: 239,
            },
            connection_id: Some(
                ConnectionId(
                    "connection-1",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-1",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

We have now successfully established a connection between the two chains.

Channel Open Handshake

The tx raw commands can be used to establish a channel for a given connection. Only unordered channels are currently supported.

sequenceDiagram
    autonumber
    participant A as ibc-1
    participant B as ibc-0
    Note over A, B: No channel
    A->>B: ChannelOpenInit
    Note over B: channel: channel-0
    Note over B: channel: counterparty: none
    B->>A: ChannelOpenTry
    Note over A: channel: channel-1
    Note over A: channel: counterparty: channel-0
    A->>B: ChannelOpenAck
    note over B: channel: channel-0
    note over B: counterparty: channel-1
    B->>A: ChannelOpenConfirm
    Note over A, B: Channel open

Table of Contents

• Channel Open Init
• Channel Open Try
• Channel Open Ack
• Channel Open Confirm

Channel Open Init

Use the chan-open-init command to initialize a new channel.

USAGE:
    hermes tx raw chan-open-init <OPTIONS>

DESCRIPTION:
    Initialize a channel (ChannelOpenInit)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_conn_id               identifier of the destination connection
    dst_port_id               identifier of the destination port
    src_port_id               identifier of the source port

FLAGS:
    -o, --order ORDER         the channel ordering, valid options 'unordered' (default) and 'ordered'

Example

First, let's initialize the channel on ibc-0 using an existing connection identified by connection-0:

hermes tx raw chan-open-init ibc-0 ibc-1 connection-0 transfer transfer

Success: OpenInitChannel(
    OpenInit(
        Attributes {
            height: Height {
                revision: 0,
                height: 3091
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: None,
        },
    ),
)

A new channel has been initialized on ibc-1 with identifier channel-0.

Note that the counterparty_channel_id field is currently empty.

Channel Open Try

Use the chan-open-try command to establish a counterparty to the channel on the other chain.

USAGE:
    hermes tx raw chan-open-try <OPTIONS>

DESCRIPTION:
    Relay the channel attempt (ChannelOpenTry)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_conn_id               identifier of the destination connection
    dst_port_id               identifier of the destination port
    src_port_id               identifier of the source port

FLAGS:
    -s, --src-chan-id ID      identifier of the source channel (required)

Example

Let's now create the counterparty to channel-0 on chain ibc-1:

hermes tx raw chan-open-try ibc-1 ibc-0 connection-1 transfer transfer -s channel-0

Success: OpenTryChannel(
    OpenTry(
        Attributes {
            height: Height {
                revision: 1,
                height: 3213
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
            connection_id: ConnectionId(
                "connection-1",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
        },
    ),
)

A new channel has been created on ibc-1 with identifier channel-1.

Note that the field counterparty_channel_id points to the channel on ibc-0.

Channel Open Ack

Use the chan-open-ack command to acknowledge the channel on the initial chain.

USAGE:
    hermes tx raw chan-open-ack <OPTIONS>

DESCRIPTION:
    Relay acknowledgment of a channel attempt (ChannelOpenAck)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_conn_id               identifier of the destination connection
    dst_port_id               identifier of the destination port
    src_port_id               identifier of the source port

FLAGS:
    -d, --dst-chan-id ID      identifier of the destination channel (required)
    -s, --src-chan-id ID      identifier of the source channel (required)

Example

We can now acknowledge on ibc-0 that ibc-1 has accepted the opening of the channel:

hermes tx raw chan-open-ack ibc-0 ibc-1 connection-0 transfer transfer -d channel-0 -s channel-1

Success: OpenAckChannel(
    OpenAck(
        Attributes {
            height: Height {
                revision: 0,
                height: 3301
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
        },
    ),
)

Note that the field counterparty_channel_id now points to the channel on ibc-1.

Channel Open Confirm

Use the chan-open-confirm command to confirm that the channel has been acknowledged, and finish the handshake, after which the channel is open on both chains.

USAGE:
    hermes tx raw chan-open-confirm <OPTIONS>

DESCRIPTION:
    Confirm opening of a channel (ChannelOpenConfirm)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_conn_id               identifier of the destination connection
    dst_port_id               identifier of the destination port
    src_port_id               identifier of the source port

FLAGS:
    -d, --dst-chan-id ID      identifier of the destination channel (required)
    -s, --src-chan-id ID      identifier of the source channel (required)

Example

Confirm on ibc-1 that ibc-0 has accepted the opening of the channel, after which the channel is open on both chains.

hermes tx raw chan-open-confirm ibc-1 ibc-0 connection-1 transfer transfer -d channel-1 -s channel-0

    OpenConfirm(
        Attributes {
            height: Height {
                revision: 1,
                height: 3483
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
            connection_id: ConnectionId(
                "connection-1",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
        },
    ),
)

We have now successfully opened a channel over an existing connection between the two chains.

Channel Close Handshake

The channel close handshake involves two steps: init and confirm.

Table of Contents

• Channel Close Init
• Channel Close Confirm

Channel Close Init

Use the chan-close-init command to initialize the closure of a channel.

USAGE:
    hermes tx raw chan-close-init <OPTIONS>

DESCRIPTION:
    Initiate the closing of a channel (ChannelCloseInit)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_conn_id               identifier of the destination connection
    dst_port_id               identifier of the destination port
    src_port_id               identifier of the source port

FLAGS:
    -d, --dst-chan-id ID      identifier of the destination channel (required)
    -s, --src-chan-id ID      identifier of the source channel (required)

Example

hermes tx raw chan-close-init ibc-0 ibc-1 connection-0 transfer transfer -d channel-0 -s channel-1

Success: CloseInitChannel(
    CloseInit(
        Attributes {
            height: Height {
                revision: 0,
                height: 77,
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
        },
    ),
)

Channel Close Confirm

Use the chan-close-confirm command to confirm the closure of a channel.

USAGE:
    hermes tx raw chan-close-confirm <OPTIONS>

DESCRIPTION:
    Confirm the closing of a channel (ChannelCloseConfirm)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    dst_conn_id               identifier of the destination connection
    dst_port_id               identifier of the destination port
    src_port_id               identifier of the source port

FLAGS:
    -d, --dst-chan-id ID      identifier of the destination channel (required)
    -s, --src-chan-id ID      identifier of the source channel (required)

Example

hermes tx raw chan-close-confirm ibc-1 ibc-0 connection-1 transfer transfer -d channel-1 -s channel-0

Success: CloseConfirmChannel(
    CloseConfirm(
        Attributes {
            height: Height {
                revision: 1,
                height: 551,
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
            connection_id: ConnectionId(
                "connection-1",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
        },
    ),
)

NOTE: The cosmos-sdk transfer module implementation does not allow the user (hermes in this case) to initiate the closing of channels. Therefore, when using the Gaia release image, the chan-close-init command fails as the MsgChannelCloseInit message included in the transaction is rejected. To be able to test channel closure, you need to patch your gaia deployments.

Packet Tx Commands

Table of Contents

• Fungible token transfer
• Relay receive and timeout packets
• Relay acknowledgment packets

Fungible token transfer

Use the tx raw ft-transfer command to send ICS-20 fungible token transfer packets. NOTE: This command is mainly used for testing the packet features of the relayer.

USAGE:
    hermes tx raw ft-transfer <OPTIONS>

DESCRIPTION:
    Send a fungible token transfer test transaction (ICS20 MsgTransfer)

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    src_port_id               identifier of the source port
    src_channel_id            identifier of the source channel
    amount                    amount of coins (samoleans, by default) to send (e.g. `100000`)

FLAGS:
    -o, --timeout-height-offset TIMEOUT-HEIGHT-OFFSET  timeout in number of blocks since current
    -t, --timeout-seconds TIMEOUT-SECONDS              timeout in seconds since current
    -r, --receiver RECEIVER                            receiving account address on the destination chain
    -d, --denom DENOM                                  denomination of the coins to send (default: samoleans)
    -n, --number-msgs NUMBER-MSGS                      number of messages to send
    -k, --key KEY                                      use the given signing key (default: `key_name` config)

Example

Send two transfer packets from the transfer module and channel-0 of ibc-0 to ibc-1. Each transfer if for 9999 samoleans (default denomination) and a timeout offset of 10 blocks. The transfer fee is paid by the relayer account on ibc-1.

hermes tx raw ft-transfer ibc-1 ibc-0 transfer channel-0 9999 -o 1000 -n 2

Success: [
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 431,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(4),
        },
    ),
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 431,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(5),
        },
    ),
]

The transfer packets are stored on ibc-0 and can be relayed.

To send transfer packets with a custom receiver address use the --receiver | -r flag.

hermes tx raw ft-transfer ibc-1 ibc-0 transfer channel-0 9999 -o 1000 -n 1 -r board:1938586739

Success: [
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 546,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(7),
        },
    ),
]

Relay receive and timeout packets

Use the tx raw packet-recv command to relay the packets sent but not yet received. If the sent packets have timed out then a timeout packet is sent to the source chain.

USAGE:
    hermes tx raw packet-recv <OPTIONS>

DESCRIPTION:
    Relay receive or timeout packets

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    src_port_id               identifier of the source port
    src_channel_id            identifier of the source channel

Example

Send the two transfer packets to the ibc-1 module bound to the transfer port and the channel-0's counterparty.

NOTE: The relayer prepends a client update message before the receive messages.

hermes tx raw packet-recv ibc-1 ibc-0 transfer channel-0

Success: [
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 1,
                    height: 439,
                },
                client_id: ClientId(
                    "07-tendermint-1",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 0,
                    height: 449,
                },
            },
            header: Some(
                Tendermint(...),
            ),
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 1,
                height: 439,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(4),
            ack: [
                123,
                34,
                114,
                101,
                115,
                117,
                108,
                116,
                34,
                58,
                34,
                65,
                81,
                61,
                61,
                34,
                125,
            ],
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 1,
                height: 439,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(5),
            ack: [
                123,
                34,
                114,
                101,
                115,
                117,
                108,
                116,
                34,
                58,
                34,
                65,
                81,
                61,
                61,
                34,
                125,
            ],
        },
    ),
]

Both packets have been relayed to ibc-1 and acknowledged.

Relay acknowledgment packets

Use the tx raw packet-ack command to relay acknowledgments to the original source of the packets.

USAGE:
    hermes tx raw packet-ack <OPTIONS>

DESCRIPTION:
    Relay acknowledgment packets

POSITIONAL ARGUMENTS:
    dst_chain_id              identifier of the destination chain
    src_chain_id              identifier of the source chain
    src_port_id               identifier of the source port
    src_channel_id            identifier of the source channel

Example

Send the acknowledgments to the ibc-0 module bound to the transfer port and the channel-1's counterparty.

NOTE: The relayer prepends a client update message before the acknowledgments.

hermes tx raw packet-ack ibc-0 ibc-1 transfer channel-1

Success: [
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 0,
                    height: 495,
                },
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 1,
                    height: 483,
                },
            },
            header: Some(
                Tendermint(...),
            ),
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 495,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(4),
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 495,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(5),
        },
    ),
]

Both acknowledgments have been received on ibc-0.

Help

This section provides guidelines regarding troubleshooting and general resources for getting help with hermes. For this purpose, we recommend a few ideas that could be of help:

• hermes help command, providing a CLI documentation for all hermes commands.
• profile your relayer binary to identify slow methods;
• configure the log_level to help with debugging;
• patch your local gaia chain(s) to enable some corner-case methods (e.g., channel close);

And if the above options do not address your specific problem:

• you can request a new feature;
• or consult the list of reported issues and search by relevant keywords to see if you're dealing with a known problem;
• we would be grateful if you can submit a bug report discussing any problem you find, and from there on we can look at the problem together;

Lastly, for general questions, you can reach us at hello@informal.systems, or on Twitter @informalinc.

Table of contents

• Help command
• Parametrizing the log output level
• Overriding the tracing filter using RUST_LOG
• Inspecting the relayer state
• Patching gaia to support ChanCloseInit
• New Feature Request
• Profiling
  • Setup
  • Example
    • Output
    • Example output for tx raw conn-init command

Help command

The CLI comprises a special help command, which accepts as parameter other commands, and provides guidance on what is the correct way to invoke those commands.

For instance,

hermes help create

will provide details about all the valid invocations of the create CLI command.

USAGE:
    hermes create <SUBCOMMAND>

DESCRIPTION:
    Create objects (client, connection, or channel) on chains

SUBCOMMANDS:
    help       Get usage information
    client     Create a new IBC client
    connection Create a new connection between two chains
    channel    Create a new channel between two chains

This can provide further specific guidance if we add additional parameters, e.g.,

hermes help create channel

USAGE:
    hermes create channel <OPTIONS>

DESCRIPTION:
    Create a new channel between two chains

POSITIONAL ARGUMENTS:
    chain_a_id                identifier of the side `a` chain for the new channel
    chain_b_id                identifier of the side `b` chain for the new channel (optional)

FLAGS:
    -c, --connection-a CONNECTION-A
    --port-a PORT-A           identifier of the side `a` port for the new channel
    --port-b PORT-B           identifier of the side `b` port for the new channel
    -o, --order ORDER         the channel ordering, valid options 'unordered' (default) and 'ordered'
    -v, --channel-version VERSION     the version for the new channel

Additionally, the -h/--help flags typical for CLI applications work on all commands.

Parametrizing the log output level

The relayer configuration file permits parametrization of output verbosity via the knob called log_level. This file is loaded by default from $HOME/.hermes/config.toml, but can be overridden in all commands with the -c flag, eg. hermes -c ./path/to/my/config.toml some command.

Relevant snippet:

[global]
log_level = 'error'

Valid options for log_level are: 'error', 'warn', 'info', 'debug', 'trace'. These levels correspond to the tracing sub-component of the relayer-cli, see here.

The relayer will always print a last line summarizing the result of its operation for queries or transactions. In addition to this last line, arbitrary debug, info, or other outputs may be produced.

Overriding the tracing filter using RUST_LOG

For debugging purposes, we may want to inspect which RPC queries the relayer is making. The relayer makes use of the tendermint-rpc library to issue RPC queries, but the output of this library is by default turned off in order to keep the logs more readable.

Using the RUST_LOG environment variable, we can turn logging on for the tendermint-rpc library, as follows:

RUST_LOG=tendermint-rpc=debug,info hermes start

Setting the RUST_LOG environment variable to tendermint_rpc=debug,info instructs the relayer to set the log level of the tendermint_rpc crate to debug and otherwise use the info log level.

Note: While the tendermint-rpc contains a dash in its name, the logging filter expects a module name, which can only contain alphanumeric characters and underscores, hence why the filter above is written tendermint_rpc=debug.

Example:

❯ RUST_LOG=tendermint_rpc=debug,info hermes start
2022-02-24T14:32:14.039555Z  INFO ThreadId(01) using default configuration from '/Users/coromac/.hermes/config.toml'
2022-02-24T14:32:14.043500Z  INFO ThreadId(01) telemetry service running, exposing metrics at http://127.0.0.1:3001/metrics
2022-02-24T14:32:14.043542Z  INFO ThreadId(01) [rest] address not configured, REST server disabled
2022-02-24T14:32:14.049759Z DEBUG ThreadId(01) Incoming response: {
  "jsonrpc": "2.0",
  "id": "143b4580-c49e-47c1-81b2-4e7090f6e762",
  "result": {
    "node_info": {
      "protocol_version": {
        "p2p": "8",
        "block": "11",
        "app": "0"
      },
      "id": "73f9134539f9845cd253dc302e36d48ee4c0f32d",
      "listen_addr": "tcp://0.0.0.0:27003",
      "network": "ibc0",
      "version": "v0.34.14",
      "channels": "40202122233038606100",
      "moniker": "ibc0",
      "other": {
        "tx_index": "on",
        "rpc_address": "tcp://0.0.0.0:27000"
      }
    },
    "sync_info": {
      "latest_block_hash": "8396B93E355AD80EED8167A04BB9858A315A8BEB482547DE16A6CD82BC11551B",
      "latest_app_hash": "22419E041D6997EE75FF66F7F537A3D36122B220EAB89A9C246FEF680FB1C97A",
      "latest_block_height": "86392",
      "latest_block_time": "2022-02-24T14:32:08.673989Z",
      "earliest_block_hash": "0A73CFE8566D4D4FBFE3178D9BCBAD483FD689854CA8012FF1457F8EC4598132",
      "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
      "earliest_block_height": "1",
      "earliest_block_time": "2022-01-20T09:04:21.549736Z",
      "catching_up": false
    },
    "validator_info": {
      "address": "6FD56E6AA1EEDAD227AFAB6B9DE631719D4A3691",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "mR5V/QWOv/mJYyNmlsl3mfxKy1PNaOzdztyas4NF2BA="
      },
      "voting_power": "10"
    }
  }
}
2022-02-24T14:32:14.052503Z DEBUG ThreadId(21) Incoming response: {
  "jsonrpc": "2.0",
  "id": "0ca35e64-ea98-4fbf-bd66-c3291128ace9",
  "result": {}
}

...

The two DEBUG log lines above were emitted by the tendermint-rpc crate.

Inspecting the relayer state

To get a little bit of insight into the state of the relayer, Hermes will react to a SIGUSR1 signal by dumping its state to the console, either in plain text form or as a JSON object if Hermes was started with the --json option.

To send a SIGUSR1 signal to Hermes, look up its process ID (below PID) and use the following command:

kill -SIGUSR1 PID

Hermes will print some information about the workers which are currently running.

For example, with three chains configured and one channel between each pair of chains:

INFO Dumping state (triggered by SIGUSR1)
INFO
INFO * Chains: ibc-0, ibc-1, ibc-2
INFO * Client workers:
INFO   - client::ibc-0->ibc-1:07-tendermint-0 (id: 5)
INFO   - client::ibc-0->ibc-2:07-tendermint-0 (id: 9)
INFO   - client::ibc-1->ibc-0:07-tendermint-0 (id: 1)
INFO   - client::ibc-1->ibc-2:07-tendermint-1 (id: 11)
INFO   - client::ibc-2->ibc-0:07-tendermint-1 (id: 3)
INFO   - client::ibc-2->ibc-1:07-tendermint-1 (id: 7)
INFO * Packet workers:
INFO   - packet::channel-0/transfer:ibc-0->ibc-1 (id: 2)
INFO   - packet::channel-0/transfer:ibc-1->ibc-0 (id: 6)
INFO   - packet::channel-0/transfer:ibc-2->ibc-0 (id: 10)
INFO   - packet::channel-1/transfer:ibc-0->ibc-2 (id: 4)
INFO   - packet::channel-1/transfer:ibc-1->ibc-2 (id: 8)
INFO   - packet::channel-1/transfer:ibc-2->ibc-1 (id: 12)

or in JSON form (prettified):

{
  "timestamp": "Jul 12 17:04:37.244",
  "level": "INFO",
  "fields": {
    "message": "Dumping state (triggered by SIGUSR1)"
  }
}
{
  "chains": [
    "ibc-0",
    "ibc-1",
    "ibc-2"
  ],
  "workers": {
    "Client": [
      {
        "id": 5,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-1",
          "dst_client_id": "07-tendermint-0",
          "src_chain_id": "ibc-0"
        }
      },
      {
        "id": 9,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-2",
          "dst_client_id": "07-tendermint-0",
          "src_chain_id": "ibc-0"
        }
      },
      {
        "id": 1,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-0",
          "dst_client_id": "07-tendermint-0",
          "src_chain_id": "ibc-1"
        }
      },
      {
        "id": 11,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-2",
          "dst_client_id": "07-tendermint-1",
          "src_chain_id": "ibc-1"
        }
      },
      {
        "id": 3,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-0",
          "dst_client_id": "07-tendermint-1",
          "src_chain_id": "ibc-2"
        }
      },
      {
        "id": 7,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-1",
          "dst_client_id": "07-tendermint-1",
          "src_chain_id": "ibc-2"
        }
      }
    ],
    "Packet": [
      {
        "id": 2,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-1",
          "src_chain_id": "ibc-0",
          "src_channel_id": "channel-0",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 6,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-0",
          "src_chain_id": "ibc-1",
          "src_channel_id": "channel-0",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 10,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-0",
          "src_chain_id": "ibc-2",
          "src_channel_id": "channel-0",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 4,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-2",
          "src_chain_id": "ibc-0",
          "src_channel_id": "channel-1",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 8,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-2",
          "src_chain_id": "ibc-1",
          "src_channel_id": "channel-1",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 12,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-1",
          "src_chain_id": "ibc-2",
          "src_channel_id": "channel-1",
          "src_port_id": "transfer"
        }
      }
    ]
  }
}

Patching gaia to support ChanCloseInit

The guide below refers specifically to patching your gaia chain so that the relayer can initiate the closing of channels by submitting a ChanCloseInit message. Without this modification, the transaction will be rejected. We also describe how to test the channel closing feature.

• Clone the Cosmos SDK

  git clone https://github.com/cosmos/cosmos-sdk.git ~/go/src/github.com/cosmos/cosmos-sdk
  cd ~/go/src/github.com/cosmos/cosmos-sdk
  

• Apply these diffs:

     --- a/x/ibc/applications/transfer/module.go
     +++ b/x/ibc/applications/transfer/module.go
     @@ -305,7 +305,7 @@ func (am AppModule) OnChanCloseInit(
             channelID string,
      ) error {
             // Disallow user-initiated channel closing for transfer channels
     -       return sdkerrors.Wrap(sdkerrors.ErrInvalidRequest, "user cannot close channel")
     +       return nil
      }
  

• Append the line below (watch for the placeholder <your>) as the last line in your go.mod in the gaia clone:

replace github.com/cosmos/cosmos-sdk => /Users/<your>/go/src/github.com/cosmos/cosmos-sdk

• Now make build and make install your local copy of gaia

In order to test the correct operation during the channel close, perform the steps below.

• the channel should be in state open-open:

• transfer of 5555 samoleans from ibc-1 to ibc-0. This results in a Tx to ibc-1 for a MsgTransfer packet. Make sure you're not relaying this packet (the relayer should not be running on this path).

  hermes tx raw ft-transfer ibc-0 ibc-1 transfer channel-1 5555 -o 1000 -n 1 -d samoleans
  

• now do the first step of channel closing: the channel will transition to close-open:

  hermes -c config.toml tx raw chan-close-init ibc-0 ibc-1 connection-0 transfer transfer channel-0 channel-1
  

• trigger timeout on close to ibc-1

  hermes -c config.toml tx raw packet-recv ibc-0 ibc-1 transfer channel-1
  

• close-close

  hermes -c config.toml tx raw chan-close-confirm ibc-1 ibc-0 connection-1 transfer transfer channel-1 channel-0
  

• verify that the two ends are in Close state:

  hermes -c config.toml query channel end ibc-0 transfer channel-0
  hermes -c config.toml query channel end ibc-1 transfer channel-1
  

New Feature Request

If you would like a feature to be added to hermes, don't hesitate to open a discussion about that via the feature request issue template.

Note that Hermes is packaged as part of the ibc-relayer-cli crate.

Profiling

The relayer crate provides a time! macro which can be used to measure how much time is spent between the invocation of the macro and the end of the enclosing scope.

Setup

The time! macro has no effect unless the profiling feature of the relayer crate is enabled.

To enable it, one must compile the relayer-cli crate with the --features=profiling flag.

a) One way is to build the relayer binary and update the hermes alias to point to the executable:

cd relayer-cli/
cargo build --features=profiling

b) Alternatively, one can use the cargo run command and update the alias accordingly:

alias hermes='cargo run --features=profiling --manifest-path=relayer-cli/Cargo.toml --'

The --manifest-path=relayer-cli/Cargo.toml flag is needed for cargo run to accept the --features flag.

Example

#![allow(unused)]
fn main() {
fn my_function(x: u32) -> u32 {
    time!("myfunction: x={}", x); // A

    std::thread::sleep(Duration::from_secs(1));

    {
        time!("inner operation"); // B

        std::thread::sleep(Duration::from_secs(2));

        // timer B ends here
    }

    x + 1

    // timer A ends here
}
}

Output

Jan 20 11:28:46.841  INFO relayer::macros::profiling: ⏳ myfunction: x=42 - start
Jan 20 11:28:47.842  INFO relayer::macros::profiling:    ⏳ inner operation - start
Jan 20 11:28:49.846  INFO relayer::macros::profiling:    ⏳ inner operation - elapsed: 2004ms
Jan 20 11:28:49.847  INFO relayer::macros::profiling: ⏳ myfunction: x=42 - elapsed: 3005ms

Profiling is useful for tracking down unusually slow methods. Each transaction or query usually consists of multiple lower-level methods, and it's often not clear which of these are the culprit for low performance. With profiling enabled, hermes will output timing information for individual methods involved in a command.

NOTE: To be able to see the profiling output, the realyer needs to be compiled with the profiling feature and the log level should be info level or lower.

Example output for tx raw conn-init command

hermes -c   config.toml tx raw conn-init ibc-0 ibc-1 07-tendermint-0 07-tendermint-0

Apr 13 20:58:21.225  INFO ibc_relayer::macros::profiling: ⏳ init_light_client - start
Apr 13 20:58:21.230  INFO ibc_relayer::macros::profiling: ⏳ init_light_client - elapsed: 4ms
Apr 13 20:58:21.230  INFO ibc_relayer::macros::profiling: ⏳ init_event_monitor - start
Apr 13 20:58:21.235  INFO ibc_relayer::macros::profiling: ⏳ init_event_monitor - elapsed: 5ms
Apr 13 20:58:21.235  INFO ibc_relayer::event::monitor: running listener chain.id=ibc-1
Apr 13 20:58:21.236  INFO ibc_relayer::macros::profiling: ⏳ init_light_client - start
Apr 13 20:58:21.239  INFO ibc_relayer::macros::profiling: ⏳ init_light_client - elapsed: 2ms
Apr 13 20:58:21.239  INFO ibc_relayer::macros::profiling: ⏳ init_event_monitor - start
Apr 13 20:58:21.244  INFO ibc_relayer::macros::profiling: ⏳ init_event_monitor - elapsed: 4ms
Apr 13 20:58:21.244  INFO ibc_relayer::event::monitor: running listener chain.id=ibc-0
Apr 13 20:58:21.244  INFO ibc_relayer::macros::profiling: ⏳ get_signer - start
Apr 13 20:58:21.246  INFO ibc_relayer::macros::profiling: ⏳ get_signer - elapsed: 1ms
Apr 13 20:58:21.246  INFO ibc_relayer::macros::profiling: ⏳ query_latest_height - start
Apr 13 20:58:21.246  INFO ibc_relayer::macros::profiling:    ⏳ block_on - start
Apr 13 20:58:21.248  INFO ibc_relayer::macros::profiling:    ⏳ block_on - elapsed: 1ms
Apr 13 20:58:21.249  INFO ibc_relayer::macros::profiling: ⏳ query_latest_height - elapsed: 3ms
Apr 13 20:58:21.250  INFO ibc_relayer::macros::profiling: ⏳ unbonding_period - start
Apr 13 20:58:21.250  INFO ibc_relayer::macros::profiling:    ⏳ block_on - start
Apr 13 20:58:21.251  INFO ibc_relayer::macros::profiling:    ⏳ block_on - elapsed: 0ms
Apr 13 20:58:21.270  INFO ibc_relayer::macros::profiling:    ⏳ block_on - start
Apr 13 20:58:21.273  INFO ibc_relayer::macros::profiling:    ⏳ block_on - elapsed: 2ms
Apr 13 20:58:21.273  INFO ibc_relayer::macros::profiling: ⏳ unbonding_period - elapsed: 23ms
Apr 13 20:58:21.279  INFO ibc_relayer::macros::profiling: ⏳ build_consensus_state - start
Apr 13 20:58:21.280  INFO ibc_relayer::macros::profiling: ⏳ build_consensus_state - elapsed: 0ms
Apr 13 20:58:21.280  INFO ibc_relayer::macros::profiling: ⏳ send_msgs - start
Apr 13 20:58:21.280  INFO ibc_relayer::macros::profiling:    ⏳ send_tx - start
Apr 13 20:58:21.282  INFO ibc_relayer::macros::profiling:       ⏳ PK "03f17d2c094ee68cfcedb2c2f2b7dec6cd82ea158ac1c32d3de0ca8b288a3c8bfa" - start
Apr 13 20:58:21.282  INFO ibc_relayer::macros::profiling:          ⏳ block_on - start
Apr 13 20:58:21.285  INFO ibc_relayer::macros::profiling:          ⏳ block_on - elapsed: 3ms
Apr 13 20:58:21.296  INFO ibc_relayer::macros::profiling:             ⏳ block_on - start
Apr 13 20:58:22.664  INFO ibc_relayer::macros::profiling:             ⏳ block_on - elapsed: 1367ms
Apr 13 20:58:22.664  INFO ibc_relayer::macros::profiling:       ⏳ PK "03f17d2c094ee68cfcedb2c2f2b7dec6cd82ea158ac1c32d3de0ca8b288a3c8bfa" - elapsed: 1382ms
Apr 13 20:58:22.664  INFO ibc_relayer::macros::profiling:    ⏳ send_tx - elapsed: 1384ms
Apr 13 20:58:22.664  INFO ibc_relayer::macros::profiling: ⏳ send_msgs - elapsed: 1384ms
Success: CreateClient(
    CreateClient(
        Attributes {
            height: Height {
                revision: 0,
                height: 10675,
            },
            client_id: ClientId(
                "07-tendermint-7",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 1,
                height: 10663,
            },
        },
    ),
)

Glossary

These are some of the definitions used in this guide: