Hermes Guide (v1.6.0)

Overview

Hermes is an open-source Rust implementation of a relayer for the Inter-Blockchain Communication protocol (IBC) released under the ibc-relayer-cli crate. It provides a CLI to relay packets between Cosmos SDK chains, exposes Prometheus metrics and offers a REST API.

This guide can help you set up, configure, operate and monitor Hermes to relay packets between two or more IBC-enabled chains.

About Hermes

An IBC relayer is an off-chain process responsible for relaying IBC datagrams between any two chains. The way it does so is by scanning chain states, building transactions based on these states, and submitting the transactions to the chains involved in the network.

The relayer is a central element in the IBC network architecture. This is because chain modules in this architecture are not directly sending messages to each other over networking infrastructure, but instead they create and store the data to be retrieved and used by a relayer to build the IBC datagrams.

We sometimes refer to Hermes as "IBC Relayer CLI", to make it clear that this is a relayer CLI (i.e., a binary) and distinguish it from the relayer core library (that is the crate called ibc-relayer).

Hermes is actively developed and maintained by Informal Systems in the informalsystems/hermes repository.

Note that Hermes is packaged as part of the ibc-relayer-cli crate and not the hermes crate, which is unrelated.

Where to go

• Glossary

  • This section provides some definitions of terms used throughout the guide.

• Quick start

  • This section helps you install Hermes.

• Tutorials

  • Prerequisites for local chains
    • Install Gaia and gm (Gaia Manager) for tutorials using local chains.
  • Two Local Chains
    • Start two local Cosmos Gaia chains that support the IBC protocol and learn the fundamentals of IBC.
  • More Local Chains
    • Learn how to relay on an arbitrary topology of more than two chains by using packet filters and to run multiple instances of Hermes.
  • Relaying in production
    • Learn how to set up, configure and run hermes on IBC-enabled chains in production.

• Advanced

  • Features
    • This section summarizes Hermes' features and includes a comparison between the Cosmos Go relayer and Hermes.
  • Troubleshooting
    • Learn the general guidelines regarding troubleshooting.

• Documentation

  • Configuration
    • This section includes everything you need to know to configure Hermes.
  • Telemetry
    • This section describes all Prometheus metrics and how to use them efficiently.
  • REST API
    • This section presents Hermes' REST API.
  • Commands Reference
    • This section describes the command line interface from which you can interact with Hermes.

• Educational resources

  • About IBC
    • The official IBC-Go documentation.
  • Cosmos network tutorial
    • Learn the basics of IBC in the official tutorial.
  • Connect IBC enabled chains with Hermes
    • Video demonstration of Hermes at Hackatom 2021.

• Useful links

  • Hermes FAQ Page
    • The official FAQ of Hermes.
  • Hermes GitHub repository
    • The official GitHub repository for Hermes.
  • IBC GitHub repository
    • The official repository for the Inter-blockchain protocol (IBC).
  • IBC Protocol
    • The official IBC protocol page.

Contact

• Request a new feature via the feature request issue template;
• Consult the list of reported issues and search by relevant keywords to see if you're dealing with a known problem;
• We would be grateful if you can submit a bug report discussing any problem you find, and from there on we can look at the problem together;
• Reach Hermes developers and other relayer operators in the #hermes channel on the IBC Gang Discord.

Lastly, for general questions, you can reach us at hello@informal.systems, or on Twitter @informalinc.

Disclaimer As with all software offered by Informal Systems, Hermes is offered on an “as is” basis and you use at your own risk. Informal makes no warranties of any kind, express or implied, relating to Hermes, including no warranties of a “bug-free” nature.

Quick Start

In order to run Hermes, please make sure you have all the prerequisites installed on your machine.

Once you have these prerequisites, you can build and run Hermes.

The instructions in this guide have been tested on Linux and MacOS environments. Most of the commands should work on both environments. It is currently impossible to build and run Hermes on Windows.

Sections

• Prerequisites
  • Install the latest versions of Rust and Golang.
• Installation
  • Install Hermes.

Pre-requisites

1. Rust

The IBC Relayer is developed with the Rust programming language. In order to build and run the relayer you need to install and configure Rust on your machine.

Fresh Rust installation

For instructions on how to install Rust on your machine please follow the official Notes about Rust Installation.

The provided instructions will install all the Rust tool chain including rustc, cargo, and rustup that are required to build the project.

Version requirements

Hermes is developed and tested using the latest version of Rust, 1.70 at the moment. To check that your tool chain is up-to-date run:

rustc --version

In case you already had installed the Rust tool chain in the past, you can update your installation by running rustup update.

Testing the installation

After you install the Rust tool chain you can execute the following command:

cargo version

This should display the cargo version and confirm the proper installation.

2. Golang

You will also need the Go programming language installed and configured on your machine. This is a requirement for the section Installing Gaia.

To install and configure Golang on your machine please follow the Golang official documentation.

Next Steps

Next, go to the Installation section to learn how to build Hermes.

Install Hermes

There are two main approaches for obtaining Hermes:

1. Installation:

   1. If you are running on a Unix machine (Linux/macOS), then the simplest option is to download the latest binary.
   2. You can also install via Cargo.

2. Alternatively, build Hermes directly from source.

Install by downloading

Simply head to the GitHub Releases page and download the latest version of Hermes binary matching your platform:

• macOS: hermes-v1.6.0-x86_64-apple-darwin.tar.gz (or .zip),
• Linux: hermes-v1.6.0-x86_64-unknown-linux-gnu.tar.gz (or .zip).

The step-by-step instruction below should carry you through the whole process:

1. Make the directory where we'll place the binary:

   mkdir -p $HOME/.hermes/bin
   

2. Extract the binary archive:

   tar -C $HOME/.hermes/bin/ -vxzf $ARCHIVE_NAME
   

3. Update your path, by adding this line in your .bashrc or .zshrc shell configuration file:

   export PATH="$HOME/.hermes/bin:$PATH"
   

NOTE: The binary may be initially prevented from running if you're on macOS. See the "Open Anyway" instructions from this support forum if that is the case.

You should now be able to run Hermes by invoking the hermes executable.

hermes version

Which should be:

hermes v1.6.0

Install via Cargo

NOTE: This approach assumes you have installed all the prerequisites on your machine.

Hermes is packaged in the ibc-relayer-cli Rust crate. To install the latest release of Hermes, run the following command in a terminal:

cargo install ibc-relayer-cli --bin hermes --locked

This will download and build the crate ibc-relayer-cli, and install the hermes binary in $HOME/.cargo/bin.

If you have not installed Rust and Cargo via rustup.rs, you may need to add the $HOME/.cargo/bin directory to your PATH environment variable. For most shells, this can be done by adding the following line to your .bashrc or .zshrc configuration file:

export PATH="$HOME/.cargo/bin:$PATH"

You should now be able to run Hermes by invoking the hermes executable.

hermes version

Which should be:

hermes v1.6.0

Build from source

Clone the repository

Open a terminal and clone the hermes repository:

git clone https://github.com/informalsystems/hermes.git

Change to the repository directory

cd hermes

Checkout the latest release

Go to the hermes releases page to see what is the most recent release.

Then checkout the release, for example if the most recent release is v1.6.0 then execute the command:

git checkout v1.6.0

Building with cargo build

This command builds all the crates from the hermes repository, namely: the ibc-relayer crate, and the ibc-relayer-cli crate. The last of these crates contains the hermes binary.

cargo build --release --bin hermes

By default, Hermes bundles a telemetry service and server. To build Hermes without telemetry support, and get a smaller executable, supply the --no-default-features flag to cargo build:

cargo build --release --no-default-features --bin hermes

If the build is successful, the hermes executable will be located in the following location:

./target/release/hermes

Troubleshooting: In case the cargo build command above fails, as a first course of action we recommend trying to run the same command with the additional locked flag:

cargo build --release --bin hermes --locked

Running for the first time

If you run the hermes without any additional parameters you should see the usage and help information:

./target/release/hermes

DESCRIPTION:
Informal Systems <hello@informal.systems>
  Hermes is an IBC Relayer written in Rust

USAGE:
    hermes [OPTIONS] [SUBCOMMAND]

OPTIONS:
        --config <CONFIG>    Path to configuration file
        --debug <DEBUG>      Enable debug output for the given section(s), comma separated, can be
                             repeated. [possible values: rpc, profiling, profiling-json]
    -h, --help               Print help information
        --json               Enable JSON output
    -V, --version            Print version information

SUBCOMMANDS:
    clear           Clear objects, such as outstanding packets on a channel
    config          Generate a new Hermes configuration file or validate an existing one
    create          Create objects (client, connection, or channel) on chains
    fee             Interact with the fee middleware
    health-check    Performs a health check of all chains in the the config
    help            Print this message or the help of the given subcommand(s)
    keys            Manage keys in the relayer for each chain
    listen          Listen to and display IBC events emitted by a chain
    misbehaviour    Listen to client update IBC events and handles misbehaviour
    query           Query objects from the chain
    start           Start the relayer in multi-chain mode
    tx              Create and send IBC transactions
    update          Update objects (clients) on chains
    upgrade         Upgrade objects (clients) after chain upgrade
    completions     Generate auto-complete scripts for different shells

Creating an alias for the executable

It might be easier to create an alias for hermes, so you can just run it by specifying the executable name instead of the whole path. In order to create an alias execute the following command:

alias hermes='cargo run --manifest-path $IBCFOLDER/Cargo.toml --release --bin hermes --'

Shell auto-completions

The completions sub-command of Hermes can be used to output a completion script for a choice of widely used command-line shells. Refer to hermes completions --help for the list. Some shell-specific examples of setting up auto-completion with this command are provided below; check your shell configuration to decide on the suitable directory in which to install the script and any further necessary modifications to the shell's startup files.

Bash

hermes completions --shell bash > ~/.local/share/bash-completion/completions/hermes

On a macOS installation with Homebrew bash-completion formula installed, use

hermes completions --shell bash > $(brew --prefix)/etc/bash_completion.d/hermes.bash-completion

Zsh

hermes completions --shell zsh > ~/.zfunc/_hermes

To make the shell load the script on initialization, add the directory to fpath in your ~/.zshrc before compinit:

fpath+=~/.zfunc

Next Steps

Go to the Tutorials section to learn the basics of Hermes.

Tutorials

This section includes tutorials to learn the basics of relaying and the main commands. You can also refer to the Commands Reference section to learn more about individual commands.

Sections

• Prerequisites for local chains
  • Install Gaia and gm (Gaia Manager) for tutorials using local chains.
• Two Local Chains
  • Start two local Cosmos Gaia chains that support the IBC protocol and learn the fundamentals of IBC.
• More Local Chains
  • Learn how to relay on an arbitrary topology of more than two chains by using packet filters and to run multiple instances of Hermes.
• Relaying in production
  • Learn how to set up, configure and run hermes on IBC-enabled chains in production.

Pre-requisites for local chains

In order to follow the tutorials using local chains, please make sure that Gaia and Gaiad manager are installed on your machine.

Sections

• Gaia

  • Install Gaia, the first implementation of the CosmosHub.

• Gaiad Manager

  • Install Gaiad Manager, a command-line tool (CLI) that helps manage local gaiad networks.

Install Gaia

For gm to start the chains, it requires Gaia to be installed.

NOTE: This assumes you have Golang programming language installed on your machine. If not, please ensure you install before proceeding. See more details in the Pre-requisites section.

Follow the instructions below to install Gaia.

Clone Gaia

Clone the repository from GitHub:

git clone https://github.com/cosmos/gaia.git ~/go/src/github.com/cosmos/gaia

Build and Install

Run the make command to build and install gaiad

cd ~/go/src/github.com/cosmos/gaia
git checkout v9.0.3
make install

NOTE: Specific to M1 macOS, there could be some warnings after running make install which can be safely ignored as long as gaiad binaries are built in $HOME/go/bin directory.

Add the path export PATH=$HOME/go/bin:$PATH

If the command is successful, you can run the following command to ensure it was properly installed:

gaiad version --log_level error --long | head -n4

Output:

name: gaia
server_name: gaiad
version: v9.0.3

Next Steps

Go to the next section to install gm, a tool to easily start IBC-enabled local chains.

Install Gaiad Manager

Gaiad manager (gm) is a command-line tool (CLI) that helps manage local gaiad networks.

Follow the instructions below to install and configure gm.

Requirements

• Bourne shell (sh)
• sconfig and stoml installed in your PATH (put them in /usr/local/bin)
• sed, tr
• For shell-completion Bourne Again Shell (bash) for the local user

How to run

1. Install the dependencies.

   On macOS:

   # You might need sudo permissions and create the `usr/local/bin` directory
   
   curl -Lo /usr/local/bin/sconfig https://github.com/freshautomations/sconfig/releases/download/v0.1.0/sconfig_darwin_amd64
   curl -Lo /usr/local/bin/stoml https://github.com/freshautomations/stoml/releases/download/v0.7.0/stoml_darwin_amd64
   chmod 755 /usr/local/bin/sconfig
   chmod 755 /usr/local/bin/stoml
   

   On Linux:

   curl -Lo /usr/local/bin/sconfig https://github.com/freshautomations/sconfig/releases/download/v0.1.0/sconfig_linux_amd64
   curl -Lo /usr/local/bin/stoml https://github.com/freshautomations/stoml/releases/download/v0.7.0/stoml_linux_amd64
   chmod 755 /usr/local/bin/sconfig
   chmod 755 /usr/local/bin/stoml
   

2. Install gm

   git clone https://github.com/informalsystems/gm
   gm/bin/gm install
   

   Alternatively, you can create the folder $HOME/.gm/bin and copy the files from gm/bin in there.

3. Activate gm

• Add source $HOME/.gm/bin/shell-support to a file that executes when a new terminal window comes up ($HOME/.bash_profile or $HOME/.bashrc or $HOME/.zprofile)

• (Optional) Enable auto-completion

  • On macOS:

    # Note: zsh is the default shell on MacOS, so no need to run this unless you explicitly use bash
    brew install bash-completion
    

  • On Linux:

    apt install bash-completion || yum install bash-completion
    

• Restart your terminal

Note: The shell-support script allows bash-completion as well as creating a gm alias, so you don't need to add more entries to your PATH environment variable. If you don't want to use this, you can always just add $HOME/.gm/bin to your path.

The configuration: gm.toml

Where: $HOME/.gm/gm.toml.

Description: This file contains all the high-level node configuration that gm is aware of. Note that all entries under [global] are also valid entries under any [node] header, and can be used to override the global entries for specific nodes/validators.

Entries: All entries are defined and documented in the gm.toml example configuration file.

Copy and paste below to $HOME/.gm/gm.toml and set Hermes' binary path according to your setup.

The following configuration you need to specify 2 gaiad chains. hermes will know about these chains.

[global]
  add_to_hermes = false
  auto_maintain_config = true
  extra_wallets = 2
  gaiad_binary = "~/go/bin/gaiad"
  hdpath = ""
  home_dir = "$HOME/.gm"
  ports_start_at = 27000
  validator_mnemonic = ""
  wallet_mnemonic = ""

  [global.hermes]
    binary = "PATH-TO-HERMES-BINARY" #change this path according to your setup
    config = "$HOME/.hermes/config.toml"
    log_level = "info"
    telemetry_enabled = true
    telemetry_host = "127.0.0.1"
    telemetry_port = 3001

[ibc-0]
  ports_start_at = 27010

[ibc-1]
  ports_start_at = 27020

[node-0]
  add_to_hermes = true
  network = "ibc-0"
  ports_start_at = 27030

[node-1]
  add_to_hermes = true
  network = "ibc-1"
  ports_start_at = 27040

NOTE: Go to this page for more details about Gaiad Manager

Next steps

Now that Gaiad Manager is installed on your machine, visit the first tutorial to learn the basics of Hermes. You will start two local chains and exchange tokens over IBC transfers.

Local chains

In this tutorial, you will test Hermes against two chains using Gaiad manager gm. This is the easiest way to get started.

Using gm you will start two gaia chains that support the IBC protocol.

Make sure that you followed the steps in the Prerequisites for local chains section before moving to the next section.

Sections

• Start Local Chains

  • Start two local chains with gm and set up Hermes.

• Add a new relay path

  • Add a relay path between the two chains you started.

• Start relaying

  • Exchange and relay packets between two local chains.

Start the local chains

In this chapter, you will learn how to spawn two Gaia chains, and use Hermes to relay packets between them. The topology of the network will look like this:

flowchart LR
    A((ibc-0))---B((ibc-1))

To spawn the chains and configure Hermes accordingly, we will make use of Gaiad Manager gm that we installed in the last section Install Gaiad Manager.

Reset your configuration and start the chains

Make sure you have the $HOME/.gm/gm.toml that we configured in the previous section Install Gaiad Manager. If this is not the first time you are running the script, you can manually stop the two gaia instances executing the following command to kill all gaiad processes:

gm stop

Then, reset the configuration of every node and every chain with:

rm -r $HOME/.gm/node-*
rm -r $HOME/.gm/ibc-*

NOTE: If you have any Docker containers running that might be using the same ports as gaiad (e.g. port 27010-27012), please ensure you stop them first before proceeding to the next step.

Finally, start the chains with the start command.

gm start

This configures and starts two gaiad instances, one named ibc-0 and the other named ibc-1:

graph TD
    A[gm] -->|start| C(start chains)
    C -->|gaiad| D[ibc-0]
    C -->|gaiad| F[ibc-1]

If the command runs successfully, it should output something similar to:

Creating ibc-0 config...
ibc-0 started, PID: 11244, LOG: $HOME/.gm/ibc-0/log
Creating ibc-1 config...
ibc-1 started, PID: 11796, LOG: $HOME/.gm/ibc-1/log
Creating node-0 config...
node-0 started, PID: 12342, LOG: $HOME/.gm/node-0/log
Creating node-1 config...
node-1 started, PID: 12885, LOG: $HOME/.gm/node-1/log

Run the following command to check the status of the chains:

gm status

If the command is successful, you should see a message similar to:

NODE               PID    RPC   APP  GRPC  HOME_DIR
ibc-0            11244  27010 27011 27012  $HOME/.gm/ibc-0
 node-0          12342  27030 27031 27032  $HOME/.gm/node-0
ibc-1            11796  27020 27021 27022  $HOME/.gm/ibc-1
 node-1          12885  27040 27041 27042  $HOME/.gm/node-1

Configuration file

Gaiad Manager gm takes care of creating the configuration file. Run the command below to create the $HOME/.hermes/config.toml file:

gm hermes config

NOTE: You can visit the Configuration section for more information about the configuration file.

Based on the gm.toml we created in the previous section Install Gaiad Manager, your $HOME/.hermes/config.toml file should look like below :

config.toml

[global]
log_level = 'info'

[mode]

[mode.clients]
enabled = true
refresh = true
misbehaviour = true

[mode.connections]
enabled = true

[mode.channels]
enabled = true

[mode.packets]
enabled = true
clear_interval = 100
clear_on_start = true
tx_confirmation = true

[telemetry]
enabled = true
host = '127.0.0.1'
port = 3001

[[chains]]
id = 'ibc-0'
rpc_addr = 'http://localhost:27030'
grpc_addr = 'http://localhost:27032'
event_source = { mode = 'push', url = 'ws://localhost:27030/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[[chains]]
id = 'ibc-1'
rpc_addr = 'http://localhost:27040'
grpc_addr = 'http://localhost:27042'
event_source = { mode = 'push', url = 'ws://localhost:27040/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

Adding private keys to the chains

gm will automatically generate private keys that will be used by hermes to sign transactions.

To see the keys generated by gm, run the command below

gm keys

This will generate an output similar to the one below (albeit all on the same line):

"$HOME/go/bin/gaiad" keys list --keyring-backend test --keyring-dir "$HOME/.gm/ibc-0"

- name: validator
address: cosmos1a5545h09sdzwgjpraasgkvu0f585lc33k9h4kx
pubkey: cosmospub1addwnpepqw5j24lg0ya34umnrn7akxuks3as2ktggndxg37cnfsx2fl5xkl8ymte6c2
mnemonic: "confirm path season shiver adjust order quarter now empower crystal busy foam pony web chaos bachelor magnet imitate audit wear spike chunk garlic sport"

- name: wallet
address: cosmos14czpvfgzcr06astyylahshcexzwm0j9ne6h5p5
pubkey: cosmospub1addwnpepqdcmngqappsxp6jp53atfx6kt5p7d6vce4un3mfvsa8gtml5n8lj2yh29q9
mnemonic: "brass exhibit artist beef album canvas liar fine water wave bus rose sunny permit strategy eight stove legal sustain vessel offer great book loan"

- name: wallet1
address: cosmos1qs5nmmf7jall4sm38fjssxfw5ay87mfp22p3xm
pubkey: cosmospub1addwnpepqtxfgjxg8xrc9xrzqyfs3ud6svmu7wrt608s80d0t0g93rylu4kd7kpckj6
mnemonic: "puzzle pole beyond announce clip else cause airport index pencil intact camp leisure pole nasty put meat cover garage ripple chief unfair destroy spatial"

- name: wallet2
address: cosmos1n7qyhjkfp8szpy7ury7vlejd5wcfc2ysdd9xlx
pubkey: cosmospub1addwnpepq2nuh2a9x9wd6ad78dcft3e8tuds5xs4ypeterl0zenw9ejt0tdvk38yd3z
mnemonic: "february slab crane panther harbor judge artefact ghost clay torch stay cave enrich narrow sausage expand tomato margin wool repeat squeeze couch fork unhappy"

"$HOME/go/bin/gaiad" keys list --keyring-backend test --keyring-dir "$HOME/.gm/ibc-1"

- name: validator
address: cosmos14eg9y3kjlrepk8lmdavw8u5l472sl8e6xv99yk
pubkey: cosmospub1addwnpepq0q4f0aaaq2wycg7y3x8j8gfacazdf3xlxujkjguy2k3gq654jwuyn58hhq
mnemonic: "clarify concert lens mobile hover lucky bulk home elite fix school jungle draw soul excess siren advice accuse shallow copper model absorb salon mystery"

- name: wallet
address: cosmos120jm7xkv49erxty6ec9trs85j8yfgjwwdlsrtz
pubkey: cosmospub1addwnpepqgs0llcm64e7yrpx7hs9fmzqefnwxzfxnujf3qgysdpv8w5aalu2z2e86gs
mnemonic: "shine again similar wheel also frozen equal win ask grit artist quality subject twenty pet scrub olympic ladder puppy balcony blood exotic buddy gather"

- name: wallet1
address: cosmos18ccme8td0zdktcy7dafhurdhx7x8xxx0s445y2
pubkey: cosmospub1addwnpepq045d9qjrkvfxdx39849qdcrny0zr8z2elx6z7kjkgezrvw2enepx98pyxf
mnemonic: "join skill day disease canal alpha sweet sing icon donor relief little wheat borrow silver allow child silent teach then flower deliver arena library"

- name: wallet2
address: cosmos1x45ucdaa3fegemh3x2xp0qtnxl2gv533e2fg6g
pubkey: cosmospub1addwnpepq0ryrcm08l8x5wskhd5dczrduj535fxs9w7wky04ux97amljcffe6ewxymg
mnemonic: "wish burden unfair subway club pulp wood helmet whip decline between maid defense sniff cash guard cargo travel donor nasty saddle tumble service fringe"

"$HOME/go/bin/gaiad" keys list --keyring-backend test --keyring-dir "$HOME/.gm/node-0"
[]

"$HOME/go/bin/gaiad" keys list --keyring-backend test --keyring-dir "$HOME/.gm/node-1"
[]

Next, we will need to associate a private key with chains ibc-0 and ibc-1 which hermes will use to sign transactions.

gm hermes keys

If successful, the command should show an output similar to:

SUCCESS Added key 'wallet' (cosmos1qsl5sq48r7xdfwq085x9pnlfu9ul5seufu3n03) on chain ibc-0
SUCCESS Added key 'wallet2' (cosmos1haaphqucg2u9g8gwgv6z8jzegvca85r4d7yqh9) on chain ibc-0
SUCCESS Added key 'wallet1' (cosmos1cgjf7m9txsxf2pdekxk60ll6xusx0heznqsnxn) on chain ibc-0
SUCCESS Added key 'wallet' (cosmos1zp3t2rp7tjr23wchp36lmw7vhk77gtvvc7lc5s) on chain ibc-1
SUCCESS Added key 'wallet2' (cosmos1644x9c8pyfwcmg43ch2u3vr6hl4rkmkz2weq39) on chain ibc-1
SUCCESS Added key 'wallet1' (cosmos1dsrj2uqjvtssenkwperuvfkgkg2xvmydvpzswy) on chain ibc-1

TROUBLESHOOTING:

• If the command does not out output anything, make sure the path to Hermes' binary is set in $HOME/.gm/gm.toml.

The $HOME/.gm directory

This directory is created when you install gm and the binaries are stored here but when we start the chains, all the related files and folders are stored here as well.

The $HOME/.gm directory has a tree structure similar to the one below:

.gm
├── bin
│   ├── gm
│   ├── lib-gm
│   └── shell-support
├── gm.toml
├── ibc-0
│   ├── config
│   ├── data
│   ├── init.json
│   ├── keyring-test
│   ├── log
│   ├── pid
│   ├── validator_seed.json
│   ├── wallet1_seed.json
│   ├── wallet2_seed.json
│   └── wallet_seed.json
├── ibc-1
│   ├── config
│   ├── data
│   ├── init.json
│   ├── keyring-test
│   ├── log
│   ├── pid
│   ├── validator_seed.json
│   ├── wallet1_seed.json
│   ├── wallet2_seed.json
│   └── wallet_seed.json
├── node-0
│   ├── config
│   ├── data
│   ├── init.json
│   ├── keyring-test
│   ├── log
│   └── pid
└── node-1
    ├── config
    ├── data
    ├── init.json
    ├── keyring-test
    ├── log
    └── pid

Tip: You can use the command tree $HOME/.gm/ -L 2 to view the folder structure above

The $HOME/.hermes directory

By the default hermes expects the configuration file to be in the $HOME/.hermes folder.

It also stores the private keys for each chain in this folder as outlined in the Keys section.

After executing gm start, this is how the folder should look like:

$HOME/.hermes/
├── config.toml
└── keys
    ├── ibc-0
    │   └── keyring-test
    │       └── wallet.json
    |       └── wallet1.json
    |       └── wallet2.json
    └── ibc-1
        └── keyring-test
            └── wallet.json
            └── wallet1.json
            └── wallet2.json

Next Steps

The next section describes how clients, connections and channels are created and how their identifiers are assigned.

Add a new relay path

In order to connect two IBC-enabled chains, both chains need an on-chain client that keeps track of the other chain. These two clients can be connected by one or multiple connections. Then, channels need to be created, over a connection, to specify the destination module.

WARNING: In production, do not create clients, connections or channels between two chains before checking that a client/connection/channel does not already fulfill the same function.

Identifiers

A chain allocates identifiers when it creates clients, connections and channels. These identifiers can subsequently be used to refer to existing clients, connections, and channels.

NOTE: If you want to ensure you get the same identifiers while following the tutorials, run each of the commands in this page only once or reset the chains as instructed in section Start local chains.

Chains allocate identifiers using a chain-specific allocation scheme. Currently, the cosmos-sdk implementation uses the following identifiers:

• 07-tendermint-<n> for tendermint clients.
• connection-<n> for connections.
• channel-<n> for channels.

It is possible for two chains to use the same identifier to designate two different objects. For example, two different channels, one on the Hub and one on Osmosis, can both be designated with the channel-0 identifier.

Create the relay path

A relay path refers to a specific channel used to interconnect two chains and over which packets are being sent.

Hermes can be started to listen for packet events on the two ends of multiple paths and relay packets over these paths. This can be done over a new path or over existing paths.

NOTE: The following steps decompose every step from the creation of the clients to the channel handshake for educational purposes. More realistically, you'd use the command hermes create channel --a-chain ibc-0 --b-chain ibc-1 --a-port transfer --b-port transfer --new-client-connection in order to create a new client on each chain, establish a connection, and open a channel, all with a single command.

You will need to first create a client on both chains and then establish a connection between them. It is possible to have multiple connections between clients, which can be useful in order to support multiple versions of IBC. Finally, you need to create channels over a connection to identify the source and destination modules. You can learn more in the cosmos academy tutorial.

Create clients

First, create a client on ibc-1 tracking the state of ibc-0. It will be assigned 07-tendermint-0 as its identifier:

hermes create client --host-chain ibc-1 --reference-chain ibc-0

If the command is successful, the output should be similar to:

SUCCESS CreateClient(
    CreateClient(
        Attributes {
            client_id: ClientId(
                "07-tendermint-0",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 0,
                height: 2,
            },
        },
    ),
)

Now, create a client on ibc-0 tracking ibc-1:

hermes create client --host-chain ibc-0 --reference-chain ibc-1

If the command is successful, the output should be similar to:

SUCCESS CreateClient(
    CreateClient(
        Attributes {
            client_id: ClientId(
                "07-tendermint-0",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 1,
                height: 3,
            },
        },
    ),
)

As you can see, the identifier is also 07-tendermint-0 because the client-id is local to a chain.

2. Create connections

After creating clients on both chains, you have to establish a connection between them. Both chains will assign connection-0 as the identifier of their first connection:

hermes create connection --a-chain ibc-0 --a-client 07-tendermint-0 --b-client 07-tendermint-0

NOTE: The command does not take --b-chain as argument as --a-client can only track one chain (ibc-1).

If the command runs successfully, it should output something similar to:

2022-08-29T11:16:39.833467Z  INFO ThreadId(01) using default configuration from '$HOME/.hermes/config.toml'
2022-08-29T11:16:39.838071Z  INFO ThreadId(01) Creating a new connection with pre-existing clients 07-tendermint-0 and 07-tendermint-0
2022-08-29T11:16:39.843103Z  INFO ThreadId(15) wait_for_block_commits: waiting for commit of tx hashes(s) F87AE29F8BA86EA9F6533C0CE8A34101C90948B824446E0B4889C4F953A9E094 id=ibc-0
2022-08-29T11:16:41.047867Z  INFO ThreadId(01) 🥂 ibc-0 => IbcEventWithHeight {
    event: OpenInitConnection(
        OpenInit(
            Attributes {
                connection_id: Some(
                    ConnectionId(
                        "connection-0",
                    ),
                ),
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                counterparty_connection_id: None,
                counterparty_client_id: ClientId(
                    "07-tendermint-0",
                ),
            },
        ),
    ),
    height: Height {
        revision: 0,
        height: 29,
    },
}

2022-08-29T11:16:44.061620Z  INFO ThreadId(15) wait_for_block_commits: waiting for commit of tx hashes(s) AEEAE5846991C6748248ECD81A5B8D83E7E0388322202900788C72518649EF7B id=ibc-0
2022-08-29T11:16:51.249114Z  INFO ThreadId(41) wait_for_block_commits: waiting for commit of tx hashes(s) BFED59B2EBE5D75A19C1CBB1FB931FF6FC81EF02F872CEB3D37AA40DDA5101B4 id=ibc-1
2022-08-29T11:16:52.452619Z  INFO ThreadId(01) 🥂 ibc-1 => IbcEventWithHeight {
    event: OpenTryConnection(
        OpenTry(
            Attributes {
                connection_id: Some(
                    ConnectionId(
                        "connection-0",
                    ),
                ),
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                counterparty_connection_id: Some(
                    ConnectionId(
                        "connection-0",
                    ),
                ),
                counterparty_client_id: ClientId(
                    "07-tendermint-0",
                ),
            },
        ),
    ),
    height: Height {
        revision: 1,
        height: 31,
    },
}

2022-08-29T11:16:55.459367Z  WARN ThreadId(01) [ibc-0 -> ibc-1:07-tendermint-0] resolving trusted height from the full list of consensus state heights for target height 0-31; this may take a while
2022-08-29T11:16:55.469498Z  INFO ThreadId(41) wait_for_block_commits: waiting for commit of tx hashes(s) D232FCF03549B692604A06AFC1D82494FB1D466E61880E9A8653FEFC2F41BA69 id=ibc-1
2022-08-29T11:17:02.248045Z  INFO ThreadId(15) wait_for_block_commits: waiting for commit of tx hashes(s) 0ABC352714048C0873537CCEBE31393E1CB09F810B5AAE495833436A8F9447C0 id=ibc-0
2022-08-29T11:17:06.159408Z  INFO ThreadId(01) 🥂 ibc-0 => IbcEventWithHeight {
    event: OpenAckConnection(
        OpenAck(
            Attributes {
                connection_id: Some(
                    ConnectionId(
                        "connection-0",
                    ),
                ),
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                counterparty_connection_id: Some(
                    ConnectionId(
                        "connection-0",
                    ),
                ),
                counterparty_client_id: ClientId(
                    "07-tendermint-0",
                ),
            },
        ),
    ),
    height: Height {
        revision: 0,
        height: 34,
    },
}

2022-08-29T11:17:11.202362Z  INFO ThreadId(41) wait_for_block_commits: waiting for commit of tx hashes(s) F5A344056C7F8775620581756985C2C5DB43F396A18956C017E56EFB4A8FF616 id=ibc-1
2022-08-29T11:17:12.407373Z  INFO ThreadId(01) 🥂 ibc-1 => IbcEventWithHeight {
    event: OpenConfirmConnection(
        OpenConfirm(
            Attributes {
                connection_id: Some(
                    ConnectionId(
                        "connection-0",
                    ),
                ),
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                counterparty_connection_id: Some(
                    ConnectionId(
                        "connection-0",
                    ),
                ),
                counterparty_client_id: ClientId(
                    "07-tendermint-0",
                ),
            },
        ),
    ),
    height: Height {
        revision: 1,
        height: 35,
    },
}

2022-08-29T11:17:15.409868Z  INFO ThreadId(01) connection handshake already finished for Connection {
    delay_period: 0ns,
    a_side: ConnectionSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: Some(
            ConnectionId(
                "connection-0",
            ),
        ),
    },
    b_side: ConnectionSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: Some(
            ConnectionId(
                "connection-0",
            ),
        ),
    },
}

SUCCESS Connection {
    delay_period: 0ns,
    a_side: ConnectionSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: Some(
            ConnectionId(
                "connection-0",
            ),
        ),
    },
    b_side: ConnectionSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: Some(
            ConnectionId(
                "connection-0",
            ),
        ),
    },
}

3. Channel identifiers

Finally, after the connection has been established, you can now open a new channel on top of it. Both chains will assign channel-0 as the identifier of their first channel:

hermes create channel --a-chain ibc-0 --a-connection connection-0 --a-port transfer --b-port transfer

NOTE: Again, you do not need to specify the counterparty chain as a connection can only be established with a single counterparty. The port specifies the protocol which will be used on this channel.

If the command runs successfully, it should output something similar to:

2022-08-29T11:26:28.027659Z  INFO ThreadId(01) using default configuration from '$HOME/.hermes/config.toml'
2022-08-29T11:26:28.040558Z  INFO ThreadId(15) wait_for_block_commits: waiting for commit of tx hashes(s) A7B19D0BB98DD6724B7E41A2CAD8381989D38C8D9E8C141D111DBF9DB5C20DC1 id=ibc-0
2022-08-29T11:26:33.455062Z  INFO ThreadId(01) 🎊  ibc-0 => IbcEventWithHeight {
    event: OpenInitChannel(
        OpenInit {
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: None,
        },
    ),
    height: Height {
        revision: 0,
        height: 147,
    },
}

2022-08-29T11:26:38.199410Z  INFO ThreadId(41) wait_for_block_commits: waiting for commit of tx hashes(s) 31CBCFAA6806315A5A6D96C71AEBFDFD71757F823914037B51893F123332282D id=ibc-1
2022-08-29T11:26:39.704788Z  INFO ThreadId(01) 🎊  ibc-1 => IbcEventWithHeight {
    event: OpenTryChannel(
        OpenTry {
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
        },
    ),
    height: Height {
        revision: 1,
        height: 148,
    },
}

2022-08-29T11:26:44.242127Z  INFO ThreadId(15) wait_for_block_commits: waiting for commit of tx hashes(s) 0B6EAF8ABCC7E807EDBD65E73EEE32CEE736BE787D2791C49D1436F2BA810F37 id=ibc-0
2022-08-29T11:26:48.455749Z  INFO ThreadId(01) 🎊  ibc-0 => IbcEventWithHeight {
    event: OpenAckChannel(
        OpenAck {
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
        },
    ),
    height: Height {
        revision: 0,
        height: 150,
    },
}

2022-08-29T11:26:53.297494Z  INFO ThreadId(41) wait_for_block_commits: waiting for commit of tx hashes(s) 005B0105B4E1541F3ABF56CF5AB340EDA4DE0A81939CF379F1FEA272160C47EE id=ibc-1
2022-08-29T11:26:54.501966Z  INFO ThreadId(01) 🎊  ibc-1 => IbcEventWithHeight {
    event: OpenConfirmChannel(
        OpenConfirm {
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
        },
    ),
    height: Height {
        revision: 1,
        height: 151,
    },
}

2022-08-29T11:26:57.503582Z  INFO ThreadId(01) channel handshake already finished for Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-0",
            ),
        ),
        version: None,
    },
    b_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-0",
            ),
        ),
        version: None,
    },
    connection_delay: 0ns,
}

SUCCESS Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-0",
            ),
        ),
        version: None,
    },
    b_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-0",
            ),
        ),
        version: None,
    },
    connection_delay: 0ns,
}

Visualize the current network

You can visualize the topology of the current network with:

hermes query channels --show-counterparty --chain ibc-0

If all the commands were successful, this command should output :

ibc-0: transfer/channel-0 --- ibc-1: transfer/channel-0

The chains ibc-0 and ibc-1 are now set up and configured as so:

Relay path:

flowchart LR
    A((ibc-0))---B(transfer<br>channel-0)---C(transfer<br>channel-0)---D((ibc-1))

Before going over the next sections, please ensure the commands above are executed.

Next Steps

The following section describes how to relay packets over the relay path you just created.

Start relaying

In the previous section, you created clients, established a connection between them, and opened a channel on top of it. Now you can start relaying on this path.

Relay path:

flowchart LR
    A((ibc-0))---B(transfer<br>channel-0)---C(transfer<br>channel-0)---D((ibc-1))

Query balances

Use the following commands to query balances on your local chains:

• Balances on ibc-0:

  gaiad --node tcp://localhost:27030 query bank balances $(gaiad --home ~/.gm/ibc-0 keys --keyring-backend="test" show wallet -a)
  

• Balances on ibc-1:

  gaiad --node tcp://localhost:27040 query bank balances $(gaiad --home ~/.gm/ibc-1 keys --keyring-backend="test" show wallet -a)
  

NOTE the RPC addresses used in the two commands above are configured in ~/.hermes/config.toml file. It can also be found with gm status

At this point in the tutorial, the two commands should output something similar to:

balances:
- amount: "100000000"
  denom: samoleans
- amount: "99994088"
  denom: stake
pagination:
  next_key: null
  total: "0"

NOTE: Some stake tokens were used during the connection and channel handshakes.

Exchange packets

Now, let's exchange samoleans between two chains.

• Open a new terminal and start Hermes using the start command :

  hermes start
  

  Hermes will first relay the pending packets that have not been relayed and then start passively relaying by listening for and acting on packet events.

• In a separate terminal, use the ft-transfer command to send 100000 samoleans from ibc-0 to ibc-1 over channel-0:

  hermes tx ft-transfer --timeout-seconds 1000 --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-0 --amount 100000
  

• Wait a few seconds, then query balances on ibc-1 and ibc-0. You should observe something similar to:

  • Balances at ibc-0:

    balances:
    - amount: "99900000"
    denom: samoleans
    - amount: "99992054"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  • Balances at ibc-1:

    balances:
    - amount: "100000"
    denom: ibc/C1840BD16FCFA8F421DAA0DAAB08B9C323FC7685D0D7951DC37B3F9ECB08A199
    - amount: "100000000"
    denom: samoleans
    - amount: "99989196"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  The samoleans were transferred to ibc-1 and are visible under the denomination ibc/C1840.... The exact denomination you see might be different, make sure to use the denomination assigned in your case in the following.

• Transfer back these tokens to ibc-0:

  hermes tx ft-transfer --timeout-seconds 10000 --denom ibc/C1840BD16FCFA8F421DAA0DAAB08B9C323FC7685D0D7951DC37B3F9ECB08A199 --dst-chain ibc-0 --src-chain ibc-1 --src-port transfer --src-channel channel-0 --amount 100000
  

• Wait a few seconds then query balances on ibc-1 and ibc-0 again. You should observe something similar to:

  • Balances on ibc-0:

    balances:
    - amount: "100000000"
    denom: samoleans
    - amount: "99987927"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  • Balances on ibc-1:

    balances:
    - amount: "0"
    denom: ibc/C1840BD16FCFA8F421DAA0DAAB08B9C323FC7685D0D7951DC37B3F9ECB08A199
    - amount: "100000000"
    denom: samoleans
    - amount: "99983879"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

• Open your browser and open http://localhost:3001/metrics. At this point, you should observe that the wallet_balance metric corresponds to what you observed in the previous step. All the metrics can be useful and are described in the Telemetry section. We will describe a way to use them in the tutorial Relaying in production.

Stop relaying and stop the chains

• Stop Hermes by pressing Ctrl+C on the terminal running hermes start.

• Stop the chains with gm stop.

Next steps

In this tutorial, you learned the basics of relaying by:

• Creating clients on two chains.
• Establishing a connection between them.
• Opening a channel.
• Visualizing your network.
• Exchanging packets.

In the next tutorial, you will learn how to relay between multiple chains with multiple instances.

Even more local chains

In this tutorial, you will test Hermes against four chains using Gaiad manager gm connected in an arbitrary topology of IBC channels.

Using gm you will start four gaia chains that support the IBC protocol.

Make sure that you followed the steps in the Prerequisites for local chains section before moving to the next section.

Sections

• Start Local Chains

  • Start four local chains with gm and set up Hermes.

• Build the topology

  • Add a relay path between every chain and relay on an arbitrary topology with packet filters.

• Start relaying

  • Exchange and relay packets between these chains.

• Add new instances of Hermes

  • Add new instances of Hermes to start relaying on the paths filtered out by the first instance.

Start the local chains

In this chapter, you will learn how to spawn four Gaia chains, connect them in an arbitrary topology and use Hermes to transfer tokens between them.

flowchart LR
    ibc0((ibc-0))
    ibc1((ibc-1))
    ibc2((ibc-2))
    ibc3((ibc-3))

    ibc0---ibc1
    ibc1---ibc2
    ibc2---ibc3
    ibc0---ibc3
    

As for the Local chains tutorial, we will make use of Gaiad Manager gm that we installed in Install Gaiad Manager.

Reset your configuration

First, make sure that no chain is currently running by killing all gaiad processes.

gm stop

Then, make sure that your folder $HOME/.gm does not contain any ibc-* or node-* file. You can remove them with

rm -r $HOME/.gm/node-*
rm -r $HOME/.gm/ibc-*

Copy and paste the configuration below to $HOME/.gm/gm.toml and set Hermes' binary path according to your setup. The following contains the configuration of 4 IBC-enabled chains.

gm.toml

[global]
  add_to_hermes = false
  auto_maintain_config = true
  extra_wallets = 2
  gaiad_binary = "~/go/bin/gaiad"
  hdpath = ""
  home_dir = "~/.gm"
  ports_start_at = 27000
  validator_mnemonic = ""
  wallet_mnemonic = ""

  [global.hermes]
    binary = "$HOME/hermes/target/release/hermes" # change this path according to your setup
    config = "~/.hermes/config.toml"
    log_level = "info"
    telemetry_enabled = true
    telemetry_host = "127.0.0.1"
    telemetry_port = 3001

[ibc-0]
  ports_start_at = 27010

[ibc-1]
  ports_start_at = 27020

[ibc-2]
  ports_start_at = 27030

[ibc-3]
  ports_start_at = 27040

[node-0]
  add_to_hermes = true
  network = "ibc-0"
  ports_start_at = 27050

[node-1]
  add_to_hermes = true
  network = "ibc-1"
  ports_start_at = 27060

[node-2]
  add_to_hermes = true
  network = "ibc-2"
  ports_start_at = 27070

[node-3]
  add_to_hermes = true
  network = "ibc-3"
  ports_start_at = 27080

NOTE: If you have any Docker containers running that might be using the same ports as gaiad (e.g. port 27010-27012), please ensure you stop them first before proceeding to the next step.

Finally, start the chains with the start command.

gm start

This configures and starts four gaiad instances.

graph TD
    A[gm] -->|start| C(start chains)
    C -->|gaiad| D[ibc-0]
    C -->|gaiad| E[ibc-1]
    C -->|gaiad| F[ibc-2]
    C -->|gaiad| G[ibc-3]
    

If the command runs successfully, it should output something similar to:

Creating ibc-0 config...
ibc-0 started, PID: 21330, LOG: $HOME/.gm/ibc-0/log
Creating ibc-1 config...
ibc-1 started, PID: 21888, LOG: $HOME/.gm/ibc-1/log
Creating ibc-2 config...
ibc-2 started, PID: 22443, LOG: $HOME/.gm/ibc-2/log
Creating ibc-3 config...
ibc-3 started, PID: 22999, LOG: $HOME/.gm/ibc-3/log
Creating node-0 config...
node-0 started, PID: 23547, LOG: $HOME/.gm/node-0/log
Creating node-1 config...
node-1 started, PID: 24101, LOG: $HOME/.gm/node-1/log
Creating node-2 config...
node-2 started, PID: 24649, LOG: $HOME/.gm/node-2/log
Creating node-3 config...
node-3 started, PID: 25194, LOG: $HOME/.gm/node-3/log

Run the following command to check the status of the chains:

gm status

If the command is successful, you should see a message similar to:

NODE               PID    RPC   APP  GRPC  HOME_DIR
ibc-0            21330  27010 27011 27012  $HOME/.gm/ibc-0
 node-0          23547  27050 27051 27052  $HOME/.gm/node-0
ibc-1            21888  27020 27021 27022  $HOME/.gm/ibc-1
 node-1          24101  27060 27061 27062  $HOME/.gm/node-1
ibc-2            22443  27030 27031 27032  $HOME/.gm/ibc-2
 node-2          24649  27070 27071 27072  $HOME/.gm/node-2
ibc-3            22999  27040 27041 27042  $HOME/.gm/ibc-3
 node-3          25194  27080 27081 27082  $HOME/.gm/node-3

Hermes' configuration file

Gaiad Manager gm takes care of creating the configuration file. Run the command below to create the $HOME/.hermes/config.toml file:

gm hermes config

NOTE: You can visit the Configuration section for more information about the configuration file.

Based on the gm.toml above, your $HOME/.hermes/config.toml file should look like:

config.toml

[global]
log_level = 'info'

[mode]

[mode.clients]
enabled = true
refresh = true
misbehaviour = true

[mode.connections]
enabled = true

[mode.channels]
enabled = true

[mode.packets]
enabled = true
clear_interval = 100
clear_on_start = true
tx_confirmation = true

[telemetry]
enabled = true
host = '127.0.0.1'
port = 3001

[[chains]]
id = 'ibc-0'
rpc_addr = 'http://localhost:27050'
grpc_addr = 'http://localhost:27052'
event_source = { mode = 'push', url = 'ws://localhost:27050/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[[chains]]
id = 'ibc-1'
rpc_addr = 'http://localhost:27060'
grpc_addr = 'http://localhost:27062'
event_source = { mode = 'push', url = 'ws://localhost:27060/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[[chains]]
id = 'ibc-2'
rpc_addr = 'http://localhost:27070'
grpc_addr = 'http://localhost:27072'
event_source = { mode = 'push', url = 'ws://localhost:27070/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[[chains]]
id = 'ibc-3'
rpc_addr = 'http://localhost:27080'
grpc_addr = 'http://localhost:27082'
event_source = { mode = 'push', url = 'ws://localhost:27080/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

Adding private keys to the chains

Next, we will need to associate a private key to every chain which hermes will use to sign transactions. gm will automatically generate and associate them with:

gm hermes keys

If successful, the command should show an output similar to:

SUCCESS Added key 'wallet' (cosmos1qsl5sq48r7xdfwq085x9pnlfu9ul5seufu3n03) on chain ibc-0
SUCCESS Added key 'wallet2' (cosmos1haaphqucg2u9g8gwgv6z8jzegvca85r4d7yqh9) on chain ibc-0
SUCCESS Added key 'wallet1' (cosmos1cgjf7m9txsxf2pdekxk60ll6xusx0heznqsnxn) on chain ibc-0
SUCCESS Added key 'wallet' (cosmos1zp3t2rp7tjr23wchp36lmw7vhk77gtvvc7lc5s) on chain ibc-1
SUCCESS Added key 'wallet2' (cosmos1644x9c8pyfwcmg43ch2u3vr6hl4rkmkz2weq39) on chain ibc-1
SUCCESS Added key 'wallet1' (cosmos1dsrj2uqjvtssenkwperuvfkgkg2xvmydvpzswy) on chain ibc-1
SUCCESS Added key 'wallet' (cosmos1k6c6le34zsmz34yez84a7tquedy3mkc3hy7wg8) on chain ibc-2
SUCCESS Added key 'wallet2' (cosmos1murv55h3utv5ck0a2tk5ue3n88wgglhlhyzyq8) on chain ibc-2
SUCCESS Added key 'wallet1' (cosmos1r8sq88n4k8ajsmq3sscnsd8829lqxvsmue2gf7) on chain ibc-2
SUCCESS Added key 'wallet' (cosmos1eykzqwq20sqdgvhf0tmz6xjq9mlcluwxed77gj) on chain ibc-3
SUCCESS Added key 'wallet2' (cosmos1lz6df9uggl9459z2vusw9tknpy3xn2v7yq60k9) on chain ibc-3
SUCCESS Added key 'wallet1' (cosmos15jxyjskrx7s8yqpfn3xddlrx7qcq0f8r69mp4g) on chain ibc-3

TROUBLESHOOTING:

• If the command does not out output anything, make sure the path to Hermes' binary is set in $HOME/.gm/gm.toml.

The $HOME/.gm directory

This directory is created when you install gm and the binaries are stored here but when we start the chains, all the related files and folders are stored here as well.

The $HOME/.gm directory has a tree structure similar to:

.gm
├── bin
│   ├── gm
│   ├── lib-gm
│   └── shell-support
├── gm.toml
├── ibc-0
│   ├── config
│   ├── data
│   ├── init.json
│   ├── keyring-test
│   ├── log
│   ├── pid
│   ├── validator_seed.json
│   ├── wallet1_seed.json
│   ├── wallet2_seed.json
│   └── wallet_seed.json
├── ibc-1
│   ├── config
│   ├── data
│   ├── init.json
│   ├── keyring-test
│   ├── log
│   ├── pid
│   ├── validator_seed.json
│   ├── wallet1_seed.json
│   ├── wallet2_seed.json
│   └── wallet_seed.json
├── ibc-2
│   ├── config
│   ├── data
│   ├── init.json
│   ├── keyring-test
│   ├── log
│   ├── pid
│   ├── validator_seed.json
│   ├── wallet1_seed.json
│   ├── wallet2_seed.json
│   └── wallet_seed.json
├── ibc-3
│   ├── config
│   ├── data
│   ├── init.json
│   ├── keyring-test
│   ├── log
│   ├── pid
│   ├── validator_seed.json
│   ├── wallet1_seed.json
│   ├── wallet2_seed.json
│   └── wallet_seed.json
├── node-0
│   ├── config
│   ├── data
│   ├── init.json
│   ├── log
│   └── pid
├── node-1
│   ├── config
│   ├── data
│   ├── init.json
│   ├── log
│   └── pid
├── node-2
│   ├── config
│   ├── data
│   ├── init.json
│   ├── log
│   └── pid
└── node-3
    ├── config
    ├── data
    ├── init.json
    ├── log
    └── pid

Tip: You can use the command tree $HOME/.gm/ -L 2 to view the folder structure above

The $HOME/.hermes directory

By default, hermes expects the configuration file to be in the $HOME/.hermes folder.

It also stores the private keys for each chain in this folder as outlined in the Keys section.

After executing gm start, this is how the folder should look like:

$HOME/.hermes/
├── config.toml
└── keys
    ├── ibc-0
    │   └── keyring-test
    │       ├── wallet.json
    │       ├── wallet1.json
    │       └── wallet2.json
    ├── ibc-1
    │   └── keyring-test
    │       ├── wallet.json
    │       ├── wallet1.json
    │       └── wallet2.json
    ├── ibc-2
    │   └── keyring-test
    │       ├── wallet.json
    │       ├── wallet1.json
    │       └── wallet2.json
    └── ibc-3
        └── keyring-test
            ├── wallet.json
            ├── wallet1.json
            └── wallet2.json

Next Steps

The next section describes how to create an arbitrary topology between these chains before relaying packets.

Build the topology

At this point in the tutorial, you should have four chains running and Hermes correctly configured. You can perform a health-check with the command :

hermes health-check

If the command runs successfully, it should output something similar to:

    2022-08-23T15:54:58.150005Z  INFO ThreadId(01) using default configuration from '$HOME/.hermes/config.toml'
    2022-08-23T15:54:58.150179Z  INFO ThreadId(01) [ibc-0] performing health check...
    2022-08-23T15:54:58.163298Z  INFO ThreadId(01) chain is healthy chain=ibc-0
    2022-08-23T15:54:58.163323Z  INFO ThreadId(01) [ibc-1] performing health check...
    2022-08-23T15:54:58.169132Z  INFO ThreadId(01) chain is healthy chain=ibc-1
    2022-08-23T15:54:58.169154Z  INFO ThreadId(01) [ibc-2] performing health check...
    2022-08-23T15:54:58.178418Z  INFO ThreadId(01) chain is healthy chain=ibc-2
    2022-08-23T15:54:58.178445Z  INFO ThreadId(01) [ibc-3] performing health check...
    2022-08-23T15:54:58.184615Z  INFO ThreadId(01) chain is healthy chain=ibc-3
    SUCCESS performed health check for all chains in the config

In the following tutorial, we will connect all of these chains in a full mesh topology, then use Packet filters to simulate the topology given at the beginning of the previous section.

NOTE: It is also possible to only create the channels that you want. However, in production, anyone can open channels and recreate a fully-connected topology.

Connect all the chains

Execute the following command:

gm hermes cc 

If this command runs successfully, it should output the following:

"$HOME/hermes/target/release/hermes" create channel --a-chain ibc-0 --b-chain ibc-1 --a-port transfer --b-port transfer --new-client-connection
"$HOME/hermes/target/release/hermes" create channel --a-chain ibc-0 --b-chain ibc-2 --a-port transfer --b-port transfer --new-client-connection
"$HOME/hermes/target/release/hermes" create channel --a-chain ibc-0 --b-chain ibc-3 --a-port transfer --b-port transfer --new-client-connection
"$HOME/hermes/target/release/hermes" create channel --a-chain ibc-1 --b-chain ibc-2 --a-port transfer --b-port transfer --new-client-connection
"$HOME/hermes/target/release/hermes" create channel --a-chain ibc-1 --b-chain ibc-3 --a-port transfer --b-port transfer --new-client-connection
"$HOME/hermes/target/release/hermes" create channel --a-chain ibc-2 --b-chain ibc-3 --a-port transfer --b-port transfer --new-client-connection

Executing these commands will:

• For every pair of chains, create a client on both chain tracking the state of the counterparty chain.
• Create a connection between these two clients.
• Create a transfer channel over this connection.

Use the flag --exec flag to execute these commands:

gm hermes cc --exec

At this point, your network should be fully connected. It is now time to filter channels. The following chart shows the current state of the network. The channels that we want to filter out are filled in red while the channels we want to relay on are filled in green:

Network topology

flowchart TD 
    ibc0((ibc-0))
    ibc0ibc1[[channel-0]]
    ibc0ibc2[[channel-1]]
    ibc0ibc3[[channel-2]]

    ibc1((ibc-1))
    ibc1ibc0[[channel-0]]
    ibc1ibc2[[channel-1]]
    ibc1ibc3[[channel-2]]

    ibc2((ibc-2))
    ibc2ibc0[[channel-0]]
    ibc2ibc1[[channel-1]]
    ibc2ibc3[[channel-2]]

    ibc3((ibc-3))
    ibc3ibc0[[channel-0]]
    ibc3ibc1[[channel-1]]
    ibc3ibc2[[channel-2]]

    classDef deny fill:#AA0000,color:#000000;
    classDef allow fill:#00AA00,color:#000000;
    class ibc0ibc1 allow;
    class ibc1ibc0 allow;
    class ibc0ibc3 allow;
    class ibc3ibc0 allow;
    class ibc2ibc1 allow;
    class ibc1ibc2 allow;
    class ibc2ibc3 allow;
    class ibc3ibc2 allow;


    class ibc1ibc3 deny;
    class ibc3ibc1 deny;
    class ibc0ibc2 deny;
    class ibc2ibc0 deny;

    ibc0---ibc0ibc1---ibc1ibc0---ibc1
    ibc0---ibc0ibc2---ibc2ibc0---ibc2
    ibc0---ibc0ibc3---ibc3ibc0---ibc3
    ibc1---ibc1ibc2---ibc2ibc1---ibc2
    ibc1---ibc1ibc3---ibc3ibc1---ibc3
    ibc2---ibc2ibc3---ibc3ibc2---ibc3

You can verify that everything is correct with the commands:

hermes query channels --show-counterparty --chain ibc-0
hermes query channels --show-counterparty --chain ibc-1
hermes query channels --show-counterparty --chain ibc-2
hermes query channels --show-counterparty --chain ibc-3

Which should normally output:

ibc-0: transfer/channel-0 --- ibc-1: transfer/channel-0
ibc-0: transfer/channel-1 --- ibc-2: transfer/channel-0
ibc-0: transfer/channel-2 --- ibc-3: transfer/channel-0

ibc-1: transfer/channel-0 --- ibc-0: transfer/channel-0
ibc-1: transfer/channel-1 --- ibc-2: transfer/channel-1
ibc-1: transfer/channel-2 --- ibc-3: transfer/channel-1

ibc-2: transfer/channel-0 --- ibc-0: transfer/channel-1
ibc-2: transfer/channel-1 --- ibc-1: transfer/channel-1
ibc-2: transfer/channel-2 --- ibc-3: transfer/channel-2

ibc-3: transfer/channel-0 --- ibc-0: transfer/channel-2
ibc-3: transfer/channel-1 --- ibc-1: transfer/channel-2
ibc-3: transfer/channel-2 --- ibc-2: transfer/channel-2

Add packet filters

Let's use packet filters to relay only on the green paths specified in the chart. In order to add filters, open your default configuration file $HOME/.hermes/config.toml and add:

• Under ibc-0's config:

  [chains.packet_filter]
  policy = 'allow'
  list = [
      ['transfer', 'channel-0'],
      ['transfer', 'channel-2'],
  ]
  

• Under ibc-1's config:

  [chains.packet_filter]
  policy = 'allow'
  list = [
      ['transfer', 'channel-0'],
      ['transfer', 'channel-1'],
  ]
  

• Under ibc-2's config:

  [chains.packet_filter]
  policy = 'allow'
  list = [
      ['transfer', 'channel-1'],
      ['transfer', 'channel-2'],
  ]
  

• Under ibc-3's config:

  [chains.packet_filter]
  policy = 'allow'
  list = [
      ['transfer', 'channel-0'],
      ['transfer', 'channel-2'],
  ]
  

NOTE: It is also possible to use a deny policy to filter out the channels you do not want to relay on. However, if other channels exist or are created, Hermes will also relay on them.

At this point, your config file should look like this:

[global]
log_level = 'info'

[mode]

[mode.clients]
enabled = true
refresh = true
misbehaviour = true

[mode.connections]
enabled = true

[mode.channels]
enabled = true

[mode.packets]
enabled = true
clear_interval = 100
clear_on_start = true
tx_confirmation = true

[telemetry]
enabled = true
host = '127.0.0.1'
port = 3001

[[chains]]
id = 'ibc-0'
rpc_addr = 'http://localhost:27050'
grpc_addr = 'http://localhost:27052'
event_source = { mode = 'push', url = 'ws://localhost:27050/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[chains.packet_filter]
policy = 'allow'
list = [
    ['transfer', 'channel-0'],
    ['transfer', 'channel-2'],
]

[[chains]]
id = 'ibc-1'
rpc_addr = 'http://localhost:27060'
grpc_addr = 'http://localhost:27062'
event_source = { mode = 'push', url = 'ws://localhost:27060/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }


[chains.packet_filter]
policy = 'allow'
list = [
    ['transfer', 'channel-0'],
    ['transfer', 'channel-1'],
]

[[chains]]
id = 'ibc-2'
rpc_addr = 'http://localhost:27070'
grpc_addr = 'http://localhost:27072'
event_source = { mode = 'push', url = 'ws://localhost:27070/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[chains.packet_filter]
policy = 'allow'
list = [
    ['transfer', 'channel-1'],
    ['transfer', 'channel-2'],
]

[[chains]]
id = 'ibc-3'
rpc_addr = 'http://localhost:27080'
grpc_addr = 'http://localhost:27082'
event_source = { mode = 'push', url = 'ws://localhost:27080/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[chains.packet_filter]
policy = 'allow'
list = [
    ['transfer', 'channel-0'],
    ['transfer', 'channel-2'],
]

It is also possible to check that the configuration file is valid with the command:

hermes config validate

If the command runs successfully, the output should be:

SUCCESS "configuration is valid"

Next Steps

The following section describes how to relay packets between any chain with this topology.

Start relaying

In the previous tutorial, you learned about how to relay packets between a pair of chains on a relay path. Now, you will learn how to relay packets on an arbitrary topology.

WARNING Before proceeding to the sections below, please first, make sure you followed the steps in the Build the topology section.

The chains should be fully connected and the relayer should be setup to relay on these channels:

flowchart TD 
ibc0((ibc-0))
ibc0ibc1[[channel-0]]
ibc0ibc3[[channel-2]]

ibc1((ibc-1))
ibc1ibc0[[channel-0]]
ibc1ibc2[[channel-1]]

ibc2((ibc-2))
ibc2ibc1[[channel-1]]
ibc2ibc3[[channel-2]]

ibc3((ibc-3))
ibc3ibc0[[channel-0]]
ibc3ibc2[[channel-2]]

ibc0---ibc0ibc1---ibc1ibc0---ibc1
ibc0---ibc0ibc3---ibc3ibc0---ibc3
ibc1---ibc1ibc2---ibc2ibc1---ibc2
ibc2---ibc2ibc3---ibc3ibc2---ibc3

Query balances

• Balances on ibc-0:

  gaiad --node tcp://localhost:27050 query bank balances $(gaiad --home ~/.gm/ibc-0 keys --keyring-backend="test" show wallet -a)
  

• Balances on ibc-1:

  gaiad --node tcp://localhost:27060 query bank balances $(gaiad --home ~/.gm/ibc-1 keys --keyring-backend="test" show wallet -a)
  

• Balances on ibc-2:

  gaiad --node tcp://localhost:27070 query bank balances $(gaiad --home ~/.gm/ibc-2 keys --keyring-backend="test" show wallet -a)
  

• Balances on ibc-3:

  gaiad --node tcp://localhost:27080 query bank balances $(gaiad --home ~/.gm/ibc-3 keys --keyring-backend="test" show wallet -a)
  

NOTE the RPC addresses used in the two commands above are configured in ~/.hermes/config.toml file. It can also be found with gm status

At this point in the tutorial, every command should output something similar to:

balances:
- amount: "100000000"
  denom: samoleans
- amount: "99982481"
  denom: stake
pagination:
  next_key: null
  total: "0"

Start relaying

Now, let's exchange samoleans between chains.

• Open a new terminal and start Hermes using the start command :

  hermes start
  

  Hermes will first relay the pending packets that have not been relayed and then start passively relaying by listening for and acting on packet events.

• Let's transfer 1000000 samoleans from ibc-1 to ibc-3. There is a direct path between ibc-1 (channel-2) and ibc-3 (channel-1). Let's attempt the transfer on this path.

  • In a separate terminal, use the ft-transfer command to send 1000000 samoleans from ibc-1 to ibc-3 from channel-2:

  hermes tx ft-transfer --timeout-seconds 10000 --dst-chain ibc-3 --src-chain ibc-1 --src-port transfer --src-channel channel-2 --amount 1000000
  

  • Wait a few seconds then query balances on ibc-1 and ibc-3. You should observe that ibc-1 lost 1000000 samoleans but ibc-3 did not receive any:

    • Balances at ibc-1 :

      balances:
      - amount: "99000000"
      denom: samoleans
      - amount: "99980646"
      denom: stake
      pagination:
      next_key: null
      total: "0"
      

    • Balances at ibc-3 :

      balances:
      - amount: "100000000"
      denom: samoleans
      - amount: "99979803"
      denom: stake
      pagination:
      next_key: null
      total: "0"
      

  • Observe the output on the relaying terminal and verify that no event is processed.

  If you correctly created the packet filters in the previous section, Hermes does not relay on this path. So what happened to the 1000000 samoleans you just sent ? It is stuck until a relayer decides to relay this packet to ibc-3. For now, let's forget about these samoleans. We can get as many as we want anyway.

  It might be impossible to send packets directly from ibc-1 to ibc-3, however, it is possible to use ibc-2 as a bridge:

graph LR;
    A((ibc-1)) --> B((ibc-2)) --> C((ibc-3))

• In a separate terminal, use the ft-transfer command to send 1000000 samoleans from ibc-1 to ibc-2 from channel-1:

  hermes tx ft-transfer --timeout-seconds 10000 --dst-chain ibc-2 --src-chain ibc-1 --src-port transfer --src-channel channel-1 --amount 1000000
  

• Wait a few seconds then query balances on ibc-1 and ibc-2. You should observe something similar to:

  • Balances on ibc-1:

    balances:
    - amount: "98000000"
    denom: samoleans
    - amount: "99979707"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  • Balances at ibc-2:

    balances:
    - amount: "1000000"
    denom: ibc/C1840BD16FCFA8F421DAA0DAAB08B9C323FC7685D0D7951DC37B3F9ECB08A199
    - amount: "100000000"
    denom: samoleans
    - amount: "99979154"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  The samoleans were transferred to ibc-1 and are visible under the denomination ibc/C1840... (it might be a different one for you).

• Transfer these tokens to ibc-3:

  hermes tx ft-transfer --timeout-seconds 10000 --denom ibc/C1840BD16FCFA8F421DAA0DAAB08B9C323FC7685D0D7951DC37B3F9ECB08A199 --dst-chain ibc-3 --src-chain ibc-2 --src-port transfer --src-channel channel-2 --amount 1000000
  

• Wait a few seconds then query balances on ibc-2 and ibc-3. You should observe something similar to:

  • Balances on ibc-2:

    balances:
    - amount: "0"
    denom: ibc/C1840BD16FCFA8F421DAA0DAAB08B9C323FC7685D0D7951DC37B3F9ECB08A199
    - amount: "100000000"
    denom: samoleans
    - amount: "99977059"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  • Balances on ibc-3:

    balances:
    - amount: "1000000"
    denom: ibc/C658F0EB9DE176E080B586D634004141239C3E55676462C976266DB54C56EBE4
    - amount: "100000000"
    denom: samoleans
    - amount: "99978251"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  The tokens were correctly received by ibc-3 under the denomination ibc/C658....

Send back the tokens

Now let's send some of these coins back to ibc-1. We will dedicate half of them to learn a valuable lesson while the other half will be correctly transferred back.

The wrong way

Let's start with a common mistake and send back these coins on a different path than the one they were received on:

Another path to ibc-1

graph LR;
A((ibc-3)) --> B((ibc-0)) --> C((ibc-1));

• Use the ft-transfer command to transfer 500000 ibc/C658... tokens from ibc-3 to ibc-0:

  hermes tx ft-transfer --timeout-seconds 10000 --denom ibc/C658F0EB9DE176E080B586D634004141239C3E55676462C976266DB54C56EBE4 --dst-chain ibc-0 --src-chain ibc-3 --src-port transfer --src-channel channel-0 --amount 500000
  

• Wait a few seconds, then query balances on ibc-0 and ibc-3. You should observe something similar to:

  • Balances on ibc-0:

    balances:
    - amount: "500000"
    denom: ibc/563FDAE5A0D8C15013E4485134A2D2EE3317452278B56B2ED63DDB4EB677DF84
    - amount: "100000000"
    denom: samoleans
    - amount: "99980928"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  • Balances on ibc-3:

    balances:
    - amount: "500000"
    denom: ibc/C658F0EB9DE176E080B586D634004141239C3E55676462C976266DB54C56EBE4
    - amount: "100000000"
    denom: samoleans
    - amount: "99976153"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  The tokens were correctly received by ibc-0 under the denomination ibc/563....

• Transfer the ibc/563... tokens from ibc-0 to ibc-1:

  hermes tx ft-transfer --timeout-seconds 10000 --denom ibc/563FDAE5A0D8C15013E4485134A2D2EE3317452278B56B2ED63DDB4EB677DF84 --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-0 --amount 500000
  

• Wait a few seconds then query balances on ibc-0 and ibc-3. You should observe something similar to:

  • Balances on ibc-0:

    balances:
    - amount: "0"
    denom: ibc/563FDAE5A0D8C15013E4485134A2D2EE3317452278B56B2ED63DDB4EB677DF84
    - amount: "100000000"
    denom: samoleans
    - amount: "99978828"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  • Balances on ibc-1:

    balances:
    - amount: "500000"
    denom: ibc/8F3641F853A1D075C549E733AB624BA8607C8D2FFC26B32717DE660AE6A34A73
    - amount: "98000000"
    denom: samoleans
    - amount: "99978141"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  The tokens were successfully received by ibc-1 under the denomination ibc/8F3... while they should be recognized as samoleans. Indeed, it is impossible to transfer back tokens from a different channel than the one they were received from.

  You could get the tokens back to samoleans by transferring them back via the route you sent them from, but instead let's forget about these tokens.

The right way

Now that you have seen the wrong way to transfer back tokens, let's see the right way. Tokens should be transferred back on the same path they were received.

Correct Path:

graph LR;
A((ibc-3))-->B((ibc-2))-->C((ibc-1));

• Use the ft-transfer command to transfer 500000 ibc/C658... tokens from ibc-3 to ibc-2:

  hermes tx ft-transfer --timeout-seconds 10000 --denom ibc/C658F0EB9DE176E080B586D634004141239C3E55676462C976266DB54C56EBE4 --dst-chain ibc-2 --src-chain ibc-3 --src-port transfer --src-channel channel-2 --amount 500000
  

• Wait a few seconds then query balances on ibc-2 and ibc-3. You should observe something similar to:

  • Balances at ibc-2:

    balances:
    - amount: "500000"
    denom: ibc/C1840BD16FCFA8F421DAA0DAAB08B9C323FC7685D0D7951DC37B3F9ECB08A199
    - amount: "100000000"
    denom: samoleans
    - amount: "99975713"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  • Balances at ibc-3:

    balances:
    - amount: "0"
    denom: ibc/C658F0EB9DE176E080B586D634004141239C3E55676462C976266DB54C56EBE4
    - amount: "100000000"
    denom: samoleans
    - amount: "99973935"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  The tokens were correctly received by ibc-2 under the denomination ibc/C184.... The tokens retrieved the denomination they had before they were transferred to ibc-3.

• Transfer the ibc/C184... tokens from ibc-2 to ibc-1:

  hermes tx ft-transfer --timeout-seconds 10000 --denom ibc/C1840BD16FCFA8F421DAA0DAAB08B9C323FC7685D0D7951DC37B3F9ECB08A199 --dst-chain ibc-1 --src-chain ibc-2 --src-port transfer --src-channel channel-1 --amount 500000
  

• Wait a few seconds, then query balances on ibc-1 and ibc-2. You should observe something similar to:

  • Balances on ibc-1:

    balances:
    - amount: "500000"
    denom: ibc/8F3641F853A1D075C549E733AB624BA8607C8D2FFC26B32717DE660AE6A34A73
    - amount: "98500000"
    denom: samoleans
    - amount: "99975741"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  • Balances on ibc-2:

    balances:
    - amount: "0"
    denom: ibc/C1840BD16FCFA8F421DAA0DAAB08B9C323FC7685D0D7951DC37B3F9ECB08A199
    - amount: "100000000"
    denom: samoleans
    - amount: "99974602"
    denom: stake
    pagination:
    next_key: null
    total: "0"
    

  The tokens were successfully received by ibc-1 under the denomination samoleans.

Next Steps

In the next section, you will start new instances of Hermes to relay packets over the channels that were filtered out by the first instance.

Add new instances of Hermes

In the previous section, you attempted a direct transfer between ibc-1 and ibc-3 which failed because your current instance of Hermes does not relay on that path.

In the following section, you will start new instances of Hermes to relay on the paths which are currently disabled:

flowchart LR
A((ibc-0))---B[[channel-1]]---C[[channel-0]]---D((ibc-2))

E((ibc-1))---G[[channel-2]]---H[[channel-1]]---F((ibc-3))

classDef deny fill:#AA0000,color:#000000;

class B deny;
class C deny;
class G deny;
class H deny;

Running multiple instances of Hermes can have many advantages and disadvantages. It allows for fine-grained control over every channel and can be more stable than running a single instance. However, you will also need to manage more wallets.

Create a new config file

First, you will have to create a new configuration file with:

• New packets filters.

  In order to enable the new paths.

• Different wallets.

  Two instances of Hermes can not share the same wallet.

• A different telemetry port.

  Two processes can not share the same port for any of their services.

Create the following configuration file at $HOME/hermes_second_instance.toml:

hermes_second_instance.toml

[global]
log_level = 'info'

[mode]

[mode.clients]
enabled = true
refresh = true
misbehaviour = true

[mode.connections]
enabled = true

[mode.channels]
enabled = true

[mode.packets]
enabled = true
clear_interval = 100
clear_on_start = true
tx_confirmation = true

[telemetry]
enabled = true
host = '127.0.0.1'
port = 3002

[[chains]]
id = 'ibc-0'
rpc_addr = 'http://localhost:27050'
grpc_addr = 'http://localhost:27052'
event_source = { mode = 'push', url = 'ws://localhost:27050/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet1'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[chains.packet_filter]
policy = 'allow'
list = [
    ['transfer', 'channel-1'],
]

[[chains]]
id = 'ibc-1'
rpc_addr = 'http://localhost:27060'
grpc_addr = 'http://localhost:27062'
event_source = { mode = 'push', url = 'ws://localhost:27060/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet1'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }


[chains.packet_filter]
policy = 'allow'
list = [
    ['transfer', 'channel-2'],
]

[[chains]]
id = 'ibc-2'
rpc_addr = 'http://localhost:27070'
grpc_addr = 'http://localhost:27072'
event_source = { mode = 'push', url = 'ws://localhost:27070/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet1'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[chains.packet_filter]
policy = 'allow'
list = [
    ['transfer', 'channel-0'],
]

[[chains]]
id = 'ibc-3'
rpc_addr = 'http://localhost:27080'
grpc_addr = 'http://localhost:27082'
event_source = { mode = 'push', url = 'ws://localhost:27080/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet1'
store_prefix = 'ibc'
gas_price = { price = 0.01, denom = 'stake' }
max_gas = 10000000
clock_drift = '5s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }

[chains.packet_filter]
policy = 'allow'
list = [
    ['transfer', 'channel-1'],
]

In order to make use of this config, specify it with the --config flag:

hermes --config $HOME/hermes_second_instance.toml <COMMAND>

Query pending packets

Let's find the packet that was lost in the first step of the previous section with the query packet command:

hermes query packet pending --chain ibc-1 --port transfer --channel channel-2

NOTE: You do not need to specify the configuration file as long as ibc-1 and ibc-3 are in the default config file.

If the command runs successfully, it should output:

SUCCESS Summary {
    src: PendingPackets {
        unreceived_packets: [
            Sequence(
                1,
            ),
        ],
        unreceived_acks: [],
    },
    dst: PendingPackets {
        unreceived_packets: [],
        unreceived_acks: [],
    },
}

Clear the packet

Now that we have retrieved this packet, let's clear it manually with the command hermes clear packets:

hermes  --config $HOME/hermes_second_instance.toml clear packets --chain ibc-1 --port transfer --channel channel-2

NOTE: We are using the second config to avoid using the same wallets as the running instance of Hermes. You could also simply use the key-name and counterparty-key-name flags to set another wallet. If you do not use it, you will observe a few account_sequence_mismatch errors on the terminal running hermes start but Hermes will automatically recover.

If the command runs successfully, it should output:

SUCCESS [
    UpdateClient(
        cs_h: 07-tendermint-1(1-364),
    ),
    WriteAcknowledgement(
        WriteAcknowledgement - seq:1, path:channel-2/transfer->channel-1/transfer, toh:no timeout, tos:Timestamp(2022-08-29T18:29:44.733494709Z)),
    ),
    UpdateClient(
        cs_h: 07-tendermint-2(3-365),
    ),
    AcknowledgePacket(
        AcknowledgePacket - seq:1, path:channel-2/transfer->channel-1/transfer, toh:no timeout, tos:Timestamp(2022-08-29T18:29:44.733494709Z)),
    ),
]

NOTE: It can also output a TimeoutPacket if you execute it after the packet times out (10000 seconds in this case).

You can verify that the packet was correctly relayed by querying balances or directly querying packets:

hermes query packet pending --chain ibc-1 --port transfer --channel channel-2

If the command runs successfully, it should output:

SUCCESS Summary {
    src: PendingPackets {
        unreceived_packets: [],
        unreceived_acks: [],
    },
    dst: PendingPackets {
        unreceived_packets: [],
        unreceived_acks: [],
    },
}

As you can see, there is currently no stuck packet between ibc-1 and ibc-3.

Make stuck packets

For the sake of learning, let's make new stuck packets on the ibc-0<>ibc-2 channel and the ibc-1<>ibc-3 channel.

hermes tx ft-transfer --timeout-seconds 10000 --dst-chain ibc-3 --src-chain ibc-1 --src-port transfer --src-channel channel-2 --amount 1000000
hermes tx ft-transfer --timeout-seconds 10000 --dst-chain ibc-2 --src-chain ibc-0 --src-port transfer --src-channel channel-1 --amount 1000000

If both commands run successfully, they should output a SUCCESS message.

Now, let's verify that these packets are indeed stuck with the query packet command:

• On ibc-0:

  hermes query packet pending --chain ibc-0 --port transfer --channel channel-1
  

  Which should output:

  SUCCESS Summary {
      src: PendingPackets {
          unreceived_packets: [
              Sequence(
                  1,
              ),
          ],
          unreceived_acks: [],
      },
      dst: PendingPackets {
          unreceived_packets: [],
          unreceived_acks: [],
      },
  }
  

• On ibc-1:

  hermes query packet pending --chain ibc-1 --port transfer --channel channel-2
  

  Which should output:

  SUCCESS Summary {
      src: PendingPackets {
          unreceived_packets: [
              Sequence(
                  2,
              ),
          ],
          unreceived_acks: [],
      },
      dst: PendingPackets {
          unreceived_packets: [],
          unreceived_acks: [],
      },
  }   
  

You have pending packets on the two paths filtered out by our running instance.

NOTE: You can also verify that Hermes is still relaying on the other paths by sending a packet from ibc-1 to ibc-2:

hermes tx ft-transfer --timeout-seconds 10000 --dst-chain ibc-2 --src-chain ibc-1 --src-port transfer --src-channel channel-1 --amount 1000000

Wait a few seconds then verify that no packet is pending with:

hermes query packet pending --chain ibc-1 --port transfer --channel channel-1

Start your second instance to clear packets

Instead of clearing packets manually again, you can just start Hermes with the new config file you created in a new terminal:

hermes  --config $HOME/hermes_second_instance.toml start

At launch, Hermes will clear pending packets before moving into passive mode.

• Wait a few seconds. You should observe logs produced on the terminal running the second instance of Hermes.

• Query for pending packets at ibc-0 on channel-1 and ibc-1 on channel-2 again with the query packet pending command. Both should output:

  SUCCESS Summary {
      src: PendingPackets {
          unreceived_packets: [],
          unreceived_acks: [],
      },
      dst: PendingPackets {
          unreceived_packets: [],
          unreceived_acks: [],
      },
  }
  

You can now send packets between any pair of chains. One of your two instances will relay it. Feel free to exchange more packets and observe the logs.

Stop relaying and stop the chains

• Stop Hermes by pressing Ctrl+C on the terminals running hermes start.

• Stop the chains with gm stop.

Next Steps

In the next tutorial, you will learn how to set up Hermes in production.

Production

In this tutorial, you will learn how to set up Hermes to relay between the Cosmos Hub chain and the Osmosis chain.

You will monitor Hermes' activity with a Grafana dashboard from which it is possible to visualize both the logs and the metrics produced by Hermes.

The next section will contain the steps to install all the dependencies of the monitoring platform.

Sections

• Set up Grafana

  • Learn how to set up Grafana's monitoring stack for Hermes.

• Set up Hermes

  • Learn how to configure Hermes on production chains (Cosmos Hub <> Osmosis).

• Start relaying

  • Exchange and relay packets between production chains.

Setup Grafana

Hermes provides many metrics to monitor its activity. You can find a detailed description of all the metrics in the Telemetry section. In this chapter, you will install Grafana components which will ingest the data produced by Hermes and provide both analytics and visualization.

Install Docker

You will need Docker installed and configured on your machine. We provide a Compose file to install Grafana and all its dependencies through Docker.

To install and configure Docker, please follow the Docker official documentation.

Tools

Grafana Dashboard

Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. It can be used to monitor the health of an application and the data it produces. In the following tutorial, we will use a Grafana Dashboard to visualize the Prometheus metrics and the logs.

Prometheus

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using an HTTP pull model, with flexible queries and real-time alerting. Hermes can expose Prometheus metrics. The Prometheus server will pull them and Grafana will use this server as a data source for data visualization.

Grafana Loki

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It will be used to aggregate the logs produced by Hermes.

Promtail

Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored. You will use it to ship Hermes' logs to Loki.

NOTE: You will redirect hermes' output to /var/log/hermes.log. The configuration we provide ships every log file in /var/log to Loki.

Setup Grafana

Installation

• Download docker-compose.yaml, prometheus.yml and grafana_template.json and place them in the same repository.

• Run the following command in your command line to start Grafana, Prometheus, Loki, and Promtail.

  docker-compose -f docker-compose.yaml up
  

Sign in to Grafana

• Open your web browser and go to http://localhost:3000/.
• On the sign-in page, enter admin for the username and password.
• Click Sign in. If successful, you will see a prompt to change the password.
• Click OK on the prompt and change your password.

Add Prometheus

• In the sidebar, hover your cursor over the Configuration (gear) icon, and then select Data Sources.
• Click Add data source.
• In the list of data sources, select Prometheus.
• In the URL box, enter http://prometheus:9090.
• Click Save & Test. Prometheus is now available as a data source in Grafana.

Add Loki

• Add another data source, however, this time, select Loki.
• In the URL box, enter http://loki:3100.
• Click Save & Test. Loki is now available as a data source in Grafana.

Set up the dashboard

• Download the Grafana template we provide.
• In the sidebar, hover your cursor over the + icon, and then click Import.
• Click on Upload JSON file and select the Grafana template you just downloaded.
• On the Import page, enter Hermes dashboard template as a name, enter your data sources and click Import.
• In the top right corner, next to the refresh dashboard button, select 5s to automatically query Prometheus and Loki every 5s.

Next steps

In the next section, you will learn how to set up Hermes on production chains.

Setup Hermes

In this section, you will learn how to set up Hermes to relay between the Hub and Osmosis. You will relay on channels that are already created. It is strongly advised not to create any channels between two chains if another one with the same port already exists.

Setup accounts

First, you need a wallet with enough funds on both chains. This tutorial assumes that you already have wallets created on the chains you want to relay on, and that these wallets have funds allocated to each of them.

Adding a private key

You can add a private key using one of two different ways:

• If you have a key-seed file, use the commands :

  hermes keys add --chain cosmoshub-4 --key-file key_file_hub.json
  hermes keys add --chain osmosis-1 --key-file key_file_osmosis.json
  

NOTE: Do not confuse the chain-name and the chain-id which follows the format chain_name-version.

• If you have a mnemonic, you can restore a private key from a mnemonic-file. The following steps create a mnemonic-file and restore its key for each chain under names keyhub and keyosmosis :

  echo word1 ... word12or24 > mnemonic_file_hub
  hermes keys add --key-name keyhub --chain cosmoshub-4 --mnemonic-file mnemonic_file_hub.json
  rm mnemonic_file_hub
  echo word1 ... word12or24 > mnemonic_file_osmosis
  hermes keys add --key-name keyosmosis --chain osmosis-1 --mnemonic-file mnemonic_file_osmosis.json
  rm mnemonic_file_osmosis
  

Configuration file

Then, you need to create a configuration file for Hermes (more details in the documentation).

The command hermes config auto provides a way to automatically generate a configuration file for chains in the chain-registry:

hermes config auto --output $HOME/.hermes/config.toml --chains cosmoshub:keyhub osmosis:keyosmosis

NOTE: This command also automatically finds IBC paths and generates packet filters from the _IBC folder in the chain-registry.

If the command runs successfully, it should output:

2022-08-26T11:40:35.164371Z  INFO ThreadId(01) using default configuration from '$HOME/.hermes/config.toml'
2022-08-26T11:40:35.165353Z  INFO ThreadId(01) Fetching configuration for chains: ["cosmoshub", "osmosis"]
2022-08-26T11:40:36.253328Z  WARN ThreadId(01) cosmoshub-4: uses key "keyhub"
2022-08-26T11:40:36.253704Z  WARN ThreadId(01) osmosis-1: uses key "keyosmosis"
2022-08-26T11:40:36.253860Z  WARN ThreadId(01) Gas parameters are set to default values.
SUCCESS "Config file written successfully : $HOME/.hermes/config.toml."

And generate the following configuration :

config.toml

[global]
log_level = 'info'
[mode.clients]
enabled = true
refresh = true
misbehaviour = true

[mode.connections]
enabled = false

[mode.channels]
enabled = false

[mode.packets]
enabled = true
clear_interval = 100
clear_on_start = true
tx_confirmation = false

[rest]
enabled = false
host = '127.0.0.1'
port = 3000

[telemetry]
enabled = false
host = '127.0.0.1'
port = 3001

[[chains]]
id = 'cosmoshub-4'
type = 'CosmosSdk'
rpc_addr = 'https://rpc.cosmoshub.strange.love/'
event_source = { mode = 'push', url = 'wss://rpc.cosmoshub.strange.love/websocket', batch_delay = '500ms' }
grpc_addr = 'https://grpc-cosmoshub-ia.notional.ventures/'
rpc_timeout = '10s'
account_prefix = 'cosmos'
key_name = 'keyhub'
key_store_type = 'Test'
store_prefix = 'ibc'
default_gas = 100000
max_gas = 400000
gas_multiplier = 1.1
max_msg_num = 30
max_tx_size = 2097152
clock_drift = '5s'
max_block_time = '30s'
memo_prefix = ''
sequential_batch_tx = false

[chains.trust_threshold]
numerator = '1'
denominator = '3'

[chains.gas_price]
price = 0.1
denom = 'uatom'

[chains.packet_filter]
policy = 'allow'
list = [[
    'transfer',
    'channel-141',
]]

[chains.address_type]
derivation = 'cosmos'

[[chains]]
id = 'osmosis-1'
type = 'CosmosSdk'
rpc_addr = 'https://rpc.osmosis.interbloc.org/'
event_source = { mode = 'push', url = 'wss://rpc.osmosis.interbloc.org/websocket', batch_delay = '500ms' }
grpc_addr = 'https://grpc-osmosis-ia.notional.ventures/'
rpc_timeout = '10s'
account_prefix = 'osmo'
key_name = 'keyosmosis'
key_store_type = 'Test'
store_prefix = 'ibc'
default_gas = 100000
max_gas = 400000
gas_multiplier = 1.1
max_msg_num = 30
max_tx_size = 2097152
clock_drift = '5s'
max_block_time = '30s'
memo_prefix = ''
sequential_batch_tx = false

[chains.trust_threshold]
numerator = '1'
denominator = '3'

[chains.gas_price]
price = 0.1
denom = 'uosmo'

[chains.packet_filter]
policy = 'allow'
list = [[
    'transfer',
    'channel-0',
]]

[chains.address_type]
derivation = 'cosmos'

NOTE: You might not have the same RPC and gRPC endpoints in your configuration file as they are randomly selected in the chain-registry.

The command created packet filters so Hermes will only relay on channel-0 for osmosis-1 and channel-141 for cosmoshub-4. It uses RPC and gRPC endpoints found in the chain registry. If you also run a full node, you can replace the endpoints with your own. It has many advantages as you can accept transactions with lower gas.

WARNING: It is difficult to estimate how much gas you will spend as it depends on many parameters like:

• The volume of transactions. More congestion means higher gas prices.
• The transaction's size. Bigger transactions need more gas.
• The volume of IBC messages to relay.

We cannot provide a way to precisely set those parameters. However, you can refer to other operators' configuration. You can also find IBC transfers on mintscan.io to observe how much other operators are spending. But remember that if the gas wanted is too low, the transactions will fail. If the gas price is too high gas will be wasted, but the transactions will have a higher priority.

For the tutorial, we will follow the example of Crypto Crew and set the gas parameters as follows.

• For Cosmoshub:

default_gas = 2000000
max_gas = 10000000
gas_multiplier = 1.1
max_msg_num = 25
# ...
[chains.gas_price]
price = 0.005
denom = 'uatom'

• For Osmosis:

default_gas = 5000000
max_gas = 15000000
gas_multiplier = 1.1
max_msg_num = 20
# ...
[chains.gas_price]
price = 0.0026
denom = 'uosmo'

NOTE: max_msg_nums defines the number of messages that can be sent in the same transaction.

DISCLAIMER: These parameters need to be tuned. We can not guarantee that they will always work and kept up to date.

Health-check

Finally, perform a health-check to verify that your setup is correct with:

hermes health-check

If the command runs successfully, it should output:

2022-08-26T15:54:21.321683Z  INFO ThreadId(01) using default configuration from '$HOME/.hermes/config.toml'
2022-08-26T15:54:21.321882Z  INFO ThreadId(01) [cosmoshub-4] performing health check...
2022-08-26T15:54:22.909339Z  WARN ThreadId(01) chain is healthy chain=cosmoshub-4
2022-08-26T15:54:22.909374Z  INFO ThreadId(01) [osmosis-1] performing health check...
2022-08-26T15:54:23.954362Z  INFO ThreadId(01) chain is healthy chain=osmosis-1
SUCCESS performed health check for all chains in the config

WARNING: In the previous tutorials, after setting up Hermes, we started by creating a new relay path. In production, the relay path most likely already exists and does not need to be created. Do not create channels between the Hub and Osmosis.

Next steps

You are now ready to relay. In the next chapter, you will start relaying and monitoring Hermes with Grafana.

Start relaying

In section Setup Grafana, you did set up a Grafana dashboard which is now waiting to receive data produced by hermes and running on port 3000. You also configured Hermes in section Setup Hermes and added the keys you will be using.

Create an empty log file

Promtail is shipping every log file from /var/log to Loki. Follow the steps below to create an empty log file for Hermes:

sudo touch /var/log/hermes.log 
sudo chown $(whoami) /var/log/hermes.log 

You should now have an empty hermes.log file that you can access and see on the Grafana Dashboard, on the explore page, if you select Loki as a data source. You can query the label filename=hermes.log.

Start relaying

Follow the steps to get started :

• Open your dashboard. Make sure it gets refreshed every 5s.

• In a new terminal, run hermes start &> hermes.log.

If the command runs successfully, you should be able to see the metrics panels displaying data on the Grafana Dashboard and you should also be able to see the logs on the Logs panel at the top of the dashboard. You can also explore them on the explore page.

You can now inspect the logs to verify whether the gas parameters are set correctly and tune them as possible. However, remember to restart Hermes when you modify the configuration.

Finally, Hermes is designed to relay without any intervention, however, you might have to manually trigger hermes clear packets to clear outstanding packets that Hermes failed to relay.

NOTE: It is not possible to share a wallet between two instances of Hermes.

Next steps

Visit the Telemetry section to learn how to use the metrics and the Avanced section to learn about Hermes' features and general guidelines for troubleshooting.

You can also learn more about Grafana's features and learn how to create a Grafana Managed Alert.

NOTE: In the future, Hermes might implement functionalities to trigger the commands through a REST API. It might become possible to manipulate the relayer through Grafana Alerts.

Configuration

This section includes everything you need to know to configure Hermes.

Sections

• Configure Hermes

  • Learn how to configure Hermes and some supported features.

• Description of the parameters

  • Detailed description of every parameter of Hermes

• Filter incentivized packets

  • Examples on how to configure Hermes in order to filter incentivized packets

• Performance Tuning
  • Learn about configurations allowing more refined performance tuning.

Configure Hermes

In order to run Hermes, you will need to have a configuration file.

The format supported for the configuration file is TOML.

By default, Hermes expects the configuration file to be located at $HOME/.hermes/config.toml.

This can be overridden by supplying the --config flag when invoking hermes, before the name of the command to run, e.g. hermes --config my_config.toml query connection channels --chain ibc-1 --connection connection-1.

The current version of Hermes does not support managing the configuration file programmatically. You will need to use a text editor to create the file and add content to it.

hermes [--config CONFIG_FILE] COMMAND

Table of contents

• Configuration
  • Automatically Generating A Config File
  • Tips for Manually Configuring Hermes
• Adding Private Keys
• Connecting via TLS
• Configuring Support for Interchain Accounts
• Configuring Support for Consumer Chains that Utilize Cross-Chain Validation
• Connecting to a full node protected by HTTP Basic Authentication
• Configuring Support for Wasm Relaying

Configuration

Automatically Generating A Config File

The simplest way to configure Hermes for a given chain is by running the command

hermes config auto --output ~/<OUTPUT_PATH>/config.toml --chains <CHAIN_1> <CHAIN_2>

This will generate a config.toml file for some specified chains. Note, however, that the configuration is generated by pulling the chain data from the Cosmos chain registry. The specified chain(s) must exist in the registry for the command to work. Check out this section of the Hermes commands reference to find more information on the config auto command.

Tips for Manually Configuring Hermes

For relaying use-cases that require some more bespoke configuration, you'll have to manually edit the config.toml file. The following are some rules of thumb to follow when manually configuring Hermes.

The configuration file must have one global section, and one chains section for each chain.

Note: As of 0.6.0, the Hermes configuration file is self-documented. Please read the configuration file config.toml itself for the most up-to-date documentation of parameters.

By default, Hermes will relay on all channels available between all the configured chains. In this way, every configured chain will act as a source (in the sense that Hermes listens for events) and as a destination (to relay packets that others chains have sent).

For example, if there are only two chains configured, then Hermes will only relay packets between those two, i.e. the two chains will serve as a source for each other, and likewise as a destination for each other's relevant events. Hermes will ignore all events that pertain to chains which are unknown (i.e. not present in config.toml).

To restrict relaying on specific channels, or uni-directionally, you can use packet filtering policies.

Check out the example config.toml file in the Hermes repo to see how the different parameters can be configured.

Adding Private Keys

For each chain configured you need to add a private key for that chain in order to submit transactions, please refer to the Keys sections in order to learn how to add the private keys that are used by Hermes.

Connecting via TLS

Hermes supports connection via TLS for use-cases such as connecting from behind a proxy or a load balancer. In order to enable this, you'll want to set the rpc_addr, grpc_addr, or event_source parameters to specify a TLS connection via HTTPS using the following scheme (note that the port number 443 is just used for example):

rpc_addr = 'https://domain.com:443'
grpc_addr = 'https://domain.com:443'
event_source = { mode = 'push', url = 'wss://domain.com:443/websocket', batch_delay = '500ms' }

Configuring Support for Interchain Accounts

As of version 0.13.0, Hermes supports relaying on Interchain Accounts channels.

If the packet_filter option in the chain configuration is disabled, then Hermes will relay on all existing and future channels, including ICA channels.

There are two kinds of ICA channels:

1. The host channels, whose port is icahost
2. The controller channels, whose port starts with icacontroller- followed by the owner account address. See the spec for more details.

If you wish to only relay on a few specific standard channels (here channel-0 and channel-1), but also relay on all ICA channels, you can specify the following packet filter:

Note the use of wildcards in the port and channel identifiers (['ica*', '*']) to match over all the possible ICA ports.

[chains.packet_filter]
policy = 'allow'
list = [
  ['ica*', '*'], # allow relaying on all channels whose port starts with `ica`
  ['transfer', 'channel-0'],
  ['transfer', 'channel-1'],
  # Add any other port/channel pairs you wish to relay on
]

If you wish to relay on all channels but not on ICA channels, you can use the following packet filter configuration:

[chains.packet_filter]
policy = 'deny'
list = [
  ['ica*', '*'], # deny relaying on all channels whose port starts with `ica`
]

Configuring Support for Consumer Chains that Utilize Cross-Chain Validation

As of version 1.4.1, Hermes supports relaying for consumer chains that utilize cross-chain validation (CCV).

Note: A consumer chain is essentially a regular Cosmos-SDK based chain that uses the interchain security module to achieve economic security by stake deposited on a provider chain instead of on the consumer chain itself. Consumer chains are bound to their provider chains by the provider's validator set. By being bound together in this way, consumer chains inherit the economic security guarantees of the provider chain.

You can read more about consumer chains, and Interchain Security more generally, at https://cosmos.github.io/interchain-security.

If you are configuring Hermes in order to relay for a consumer chain, set ccv_consumer_chain = true under its [[chains]] section in the config.toml file. By default, this option is set to false. It should ONLY be toggled on for CCV consumer chains, NOT for sovereign chains.

This parameter is required because consumer chains do not utilize the same staking module as sovereign chains. Consumer chains must query a different gRPC endpoint in order to fetch the relevant ccvconsumer parameters that Hermes needs in order to relay on behalf of consumer chains.

Connecting to a full node protected by HTTP Basic Authentication

To connect to a full node protected by HTTP Basic Authentication, specify the username and password in the rpc_addr, grpc_addr and event_source settings under the chain configuration in config.toml.

Here is an example with username hello and password world, assuming the RPC, WebSocket and gRPC servers listen on domain mydomain.com with TLS enabled (HTTPS/WSS).

[[chains]]
id = 'my-chain-0'

# ...

rpc_addr = 'https://hello:world@mydomain.com:26657'
grpc_addr = 'https://hello:world@mydomain.com:9090'
event_source = { mode = 'push', url = 'wss://hello:world@mydomain.com:26657/websocket', batch_delay = '500ms' }

# ...

Caution: The "Basic" authentication scheme sends the credentials encoded but not encrypted. This would be completely insecure unless the exchange was over a secure connection (HTTPS/TLS).

Configuring Support for Wasm Relaying

As of version 1.6.0, Hermes supports the relaying of wasm messages natively. This is facilitated by configuring Hermes to use pull-based relaying by polling for IBC events via the /block_results RPC endpoint. Set the event_source parameter to pull mode in config.toml:

event_source = 'poll'

The default interval at which Hermes polls the RPC endpoint is 1 second. If you need to change the interval, you can do so like this:

event_source = { mode = 'pull', interval = '2s' }

The pull model of relaying is in contrast with Hermes' default push model, where IBC events are received over WebSocket.

Note: This mode should only be used in situations where Hermes misses events that it should be receiving, such as when relaying for CosmWasm-enabled blockchains which emit IBC events without the message attribute. Without this attribute, the WebSocket is not able to catch these events to stream to Hermes, so the /block_results RPC endpoint must be used instead.

Description of the parameters

This page provides a full example of a configuration file with two chains configured:

NOTE: Visit the Relaying section to learn more about Hermes' modes.

# This is an example configuration for Hermes. It is meant to be
# used as a reference, _NOT_ for configuring a production relayer. 
# If you're looking to configure a production relayer for some chains, 
# try using the `hermes config auto` command to generate a config 
# file that serves as the starting point for configuring Hermes.

# The global section has parameters that apply globally to the relayer operation.
[global]

# Specify the verbosity for the relayer logging output. Default: 'info'
# Valid options are 'error', 'warn', 'info', 'debug', 'trace'.
log_level = 'debug'


# Specify the mode to be used by the relayer. [Required]
[mode]

# Specify the client mode.
[mode.clients]

# Whether or not to enable the client workers. [Required]
enabled = true

# Whether or not to enable periodic refresh of clients. [Default: true]
# This feature only applies to clients that underlie an open channel.
# For Tendermint clients, the frequency at which Hermes refreshes them is 2/3 of their
# trusting period (e.g., refresh every ~9 days if the trusting period is 14 days).
# Note: Even if this is disabled, clients will be refreshed automatically if
#      there is activity on a connection or channel they are involved with.
refresh = true

# Whether or not to enable misbehaviour detection for clients. [Default: true]
misbehaviour = true

# Specify the connections mode.
[mode.connections]

# Whether or not to enable the connection workers for handshake completion. [Required]
enabled = true

# Specify the channels mode.
[mode.channels]

# Whether or not to enable the channel workers for handshake completion. [Required]
enabled = true

# Specify the packets mode.
[mode.packets]

# Whether or not to enable the packet workers. [Required]
enabled = true

# Parametrize the periodic packet clearing feature.
# Interval (in number of blocks) at which pending packets
# should be periodically cleared. A value of '0' will disable
# periodic packet clearing. [Default: 100]
clear_interval = 100

# Whether or not to clear packets on start. [Default: true]
clear_on_start = true

# Toggle the transaction confirmation mechanism.
# The tx confirmation mechanism periodically queries the `/tx_search` RPC
# endpoint to check that previously-submitted transactions
# (to any chain in this config file) have been successfully delivered.
# If they have not been, and `clear_interval = 0`, then those packets are
# queued up for re-submission.
# If set to `false`, the following telemetry metrics will be disabled:
# `acknowledgment_packets_confirmed`, `receive_packets_confirmed` and `timeout_packets_confirmed`.
# [Default: false]
tx_confirmation = false

# Auto register the counterparty payee on a destination chain to
# the relayer's address on the source chain. This can be used
# for simple configuration of the relayer to receive fees for
# relaying RecvPacket on fee-enabled channels.
# For more complex configuration, turn this off and use the CLI
# to manually register the payee addresses.
# [Default: false]
auto_register_counterparty_payee = false

# The REST section defines parameters for Hermes' built-in RESTful API.
# https://hermes.informal.systems/rest.html
[rest]

# Whether or not to enable the REST service. Default: false
enabled = false

# Specify the IPv4/6 host over which the built-in HTTP server will serve the RESTful
# API requests. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the restful API
# requests. Default: 3000
port = 3000


# The telemetry section defines parameters for Hermes' built-in telemetry capabilities.
# https://hermes.informal.systems/telemetry.html
[telemetry]

# Whether or not to enable the telemetry service. Default: false
enabled = false

# Specify the IPv4/6 host over which the built-in HTTP server will serve the metrics
# gathered by the telemetry service. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the metrics gathered
# by the telemetry service. Default: 3001
port = 3001

[telemetry.buckets]
# Specify the range of the 10 histogram buckets in ms for the `tx_latency_submitted` metric.
# Default: { start = 500, end = 10000, buckets = 10 }
# The default will give the following buckets:
# [500, 2450, 4400, 6350, 8300, 10250, 12200, 14150, 16100, 18050, 20000]
# latency_submitted = { start = 500, end = 20000, buckets = 10 }

# Specify the range of the 10 histogram buckets in ms for the `tx_latency_confirmed` metric.
# Default: { start = 1000, end = 20000, buckets = 10 }
# The default will give the following buckets:
# [1000, 3900, 6800, 9700, 12600, 15500, 18400, 21300, 24200, 27100, 30000]
# latency_confirmed = { start = 1000, end = 30000, buckets = 10 }


# A chains section includes parameters related to a chain and the full node to which
# the relayer can send transactions and queries.
[[chains]]

# Specify the chain ID. Required
id = 'ibc-0'

# Whether or not this is a CCV consumer chain. Default: false
# Only specifiy true for CCV consumer chain, but NOT for sovereign chains.
ccv_consumer_chain = false

# Specify the RPC address and port where the chain RPC server listens on. Required
rpc_addr = 'http://127.0.0.1:26657'

# Specify the GRPC address and port where the chain GRPC server listens on. Required
grpc_addr = 'http://127.0.0.1:9090'

# The type of event source to use for getting events from the chain.
#
# This setting can take two types of values, as an inline table:
# 
# a) Push: for receiving IBC events over WebSocket
#
#      `{ mode = 'push', url = 'ws://127.0.0.1:26657/websocket', batch_delay = '500ms' }`
#
#    where
#
#    - `url` is the WebSocket URL to connect to. Required
#    - `batch_delay` is the delay until event batch is
#      emitted in the absence of NewBlock event. Default: 500ms
#      Lower values will result in faster event processing, improving the latency of Hermes,
#      but may split the events into more batches than necessary, requiring more client updates
#      to be submitted, yielding higher costs. Higher values will result in slower event
#      processing, increasing the latency of Hermes, but are more likely to batch events together.
#      The default value provides good latency while minimizing the number of client updates needed.

# b) Pull: for polling for IBC events via the `/block_results` RPC endpoint
#
#     `{ mode = 'pull', interval = '1s' }`
#
#    where
#
#    - `interval` is the interval at which to poll for blocks. Default: 1s
#
event_source = { mode = 'push', url = 'ws://127.0.0.1:26657/websocket', batch_delay = '500ms' }

# Specify the maximum amount of time (duration) that the RPC requests should
# take before timing out. Default: 10s (10 seconds)
# Note: Hermes uses this parameter _only_ in `start` mode; for all other CLIs,
# Hermes uses a large preconfigured timeout (on the order of minutes).
rpc_timeout = '10s'

# Experimental: Whether or not the full node is trusted.
#
# If not trusted, Hermes will verify headers included in the `ClientUpdate` message using the light client.
#
# Note: If the full node is configured as trusted then, in addition to headers not being verified,
#       the verification traces will not be provided.
#       This may cause failure in client updates after significant change in validator sets.
#
# Default: false
trusted_node = false

# Specify the prefix used by the chain. Required
account_prefix = 'cosmos'

# Specify the name of the private key to use for signing transactions. Required
# See the Adding Keys chapter for more information about managing signing keys:
#   https://hermes.informal.systems/commands/keys/index.html#adding-keys
key_name = 'testkey'

# Specify the folder used to store the keys. Optional
# If this is not specified then the hermes home folder is used.
# key_store_folder = '$HOME/.hermes/keys'

# Specify the address type which determines:
# 1) address derivation;
# 2) how to retrieve and decode accounts and pubkeys;
# 3) the message signing method.
# The current configuration options are for Cosmos SDK and Ethermint.
#
# Example configuration for chains based on Ethermint library:
#
# address_type = { derivation = 'ethermint', proto_type = { pk_type = '/ethermint.crypto.v1.ethsecp256k1.PubKey' } }
#
# Default: { derivation = 'cosmos' }, i.e. address derivation as in Cosmos SDK.
# Warning: This is an advanced feature! Modify with caution.
address_type = { derivation = 'cosmos' }

# Specify the store prefix used by the on-chain IBC modules. Required
# Recommended value for Cosmos SDK: 'ibc'
store_prefix = 'ibc'

# Gas Parameters
# 
# The term 'gas' is used to denote the amount of computation needed to execute
# and validate a transaction on-chain. It can be thought of as fuel that gets 
# spent in order to power the on-chain execution of a transaction.
#
# Hermes attempts to simulate how much gas a transaction will expend on its 
# target chain. From that, it calculates the cost of that gas by multiplying the
# amount of estimated gas by the `gas_multiplier` and the `gas_price`
# (estimated gas * `gas_multiplier` * `gas_price`) in order to compute the 
# total fee to be deducted from the relayer's wallet.
# 
# The `simulate_tx` operation does not always correctly estimate the appropriate
# amount of gas that a transaction requires. In those cases when the operation 
# fails, Hermes will attempt to submit the transaction using the specified 
# `default_gas` and `max_gas` parameters. In the case that a transaction would 
# require more than `max_gas`, it doesn't get submitted and a 
# `TxSimulateGasEstimateExceeded` error is returned.

# Specify the default amount of gas to be used in case the tx simulation fails,
# and Hermes cannot estimate the amount of gas needed.
# Default: 100 000
default_gas = 100000

# Specify the maximum amount of gas to be used as the gas limit for a transaction.
# If `default_gas` is unspecified, then `max_gas` will be used as `default_gas`.
# Default: 400 000
max_gas = 400000

# Specify the price per gas used of the fee to submit a transaction and
# the denomination of the fee. 
# 
# The specified gas price should always be greater or equal to the `min-gas-price`
# configured on the chain. This is to ensure that at least some minimal price is 
# paid for each unit of gas per transaction.
# 
# Required
gas_price = { price = 0.001, denom = 'stake' }

# Multiply this amount with the gas estimate, used to compute the fee
# and account for potential estimation error.
#
# The purpose of multiplying by `gas_multiplier` is to provide a bit of a buffer
# to catch some of the cases when the gas estimation calculation is on the low
# end. 
# 
# Example: With this setting set to 1.1, then if the estimated gas
# is 80_000, then gas used to compute the fee will be adjusted to
# 80_000 * 1.1 = 88_000.
#
# Default: 1.1, ie. the gas is increased by 10%
# Minimum value: 1.0
gas_multiplier = 1.1

# Specify how many IBC messages at most to include in a single transaction.
# Default: 30
max_msg_num = 30

# Specify the maximum size, in bytes, of each transaction that Hermes will submit.
# Default: 2097152 (2 MiB)
max_tx_size = 2097152

# Specify the maximum amount of time to tolerate a clock drift.
# The clock drift parameter defines how much new (untrusted) header's time
# can drift into the future. Default: 5s
clock_drift = '5s'

# Specify the fallback value for the maximum time per block for this chain.
# The block time together with the clock drift are added to the source drift to estimate
# the maximum clock drift when creating a client on this chain. Default: 30s
# When validating the configuration, Hermes will query the /genesis RPC endpoint in order
# to retrieve the `max_expected_time_per_block` and use it as the `max_block_time`.
# If the query fails, the configured `max_block_time` is used instead.
# For cosmos-SDK chains a good approximation is `timeout_propose` + `timeout_commit`
# Note: This MUST be the same as the `max_expected_time_per_block` genesis parameter for Tendermint chains.
max_block_time = '30s'

# Specify the amount of time to be used as the light client trusting period.
# It should be significantly less than the unbonding period
# (e.g. unbonding period = 3 weeks, trusting period = 2 weeks).
# Default: 2/3 of the `unbonding period` for Cosmos SDK chains
trusting_period = '14days'

# Specify the trust threshold for the light client, ie. the minimum fraction of validators
# which must overlap across two blocks during light client verification.
# Default: { numerator = '2', denominator = '3' }, ie. 2/3.
# Warning: This is an advanced feature! Modify with caution.
trust_threshold = { numerator = '2', denominator = '3' }

# Specify a string that Hermes will use as a memo for each transaction it submits
# to this chain. The string is limited to 50 characters. Default: '' (empty).
# Note: Hermes will append to the string defined here additional
# operational debugging information, e.g., relayer build version.
memo_prefix = ''

# This section specifies the filters for policy based relaying.
#
# Default: no policy / filters, allow all packets on all channels.
#
# Only packet filtering based on channel identifier can be specified.
# A channel filter has two fields:
# 1. `policy` - one of two types are supported:
#       - 'allow': permit relaying _only on_ the port/channel id in the list below,
#       - 'deny': permit relaying on any channel _except for_ the list below.
# 2. `list` - the list of channels specified by the port and channel identifiers.
#             Optionally, each element may also contains wildcards, for eg. 'ica*'
#             to match all identifiers starting with 'ica' or '*' to match all identifiers.
#
# Example configuration of a channel filter, only allowing packet relaying on
# channel with port ID 'transfer' and channel ID 'channel-0', as well as on
# all ICA channels.
#
# [chains.packet_filter]
# policy = 'allow'
# list = [
#   ['ica*', '*'],
#   ['transfer', 'channel-0'],
# ]

# This section specifies the filters for incentivized packet relaying.
# Default: no filters, will relay all packets even if they
# are not incentivized.
#
# It is possible to specify the channel or use wildcards for the
# channels.
# The only fee which can be parametrized is the `recv_fee`.
#
# Example configuration of a filter which will only relay incentivized
# packets, with no regards for channel and amount.
#
# [chains.packet_filter.min_fees.'*']
# recv = [ { amount = 0 } ]
#
# Example configuration of a filter which will only relay packets if they are
# from the channel 'channel-0', and they have a `recv_fee` of at least 20 stake
# or 10 uatom.
#
# [chains.packet_filter.min_fees.'channel-0']
# recv = [ { amount = 20, denom = 'stake' }, { amount = 10, denom = 'uatom' } ]

# Specify that the transaction fees should be payed from this fee granter's account.
# Optional. If unspecified (the default behavior), then no fee granter is used, and
# the account specified in `key_name` will pay the tx fees for all transactions
# submitted to this chain.
# fee_granter = ''

[[chains]]
id = 'ibc-1'
rpc_addr = 'http://127.0.0.1:26557'
grpc_addr = 'http://127.0.0.1:9091'
event_source = { mode = 'push', url = 'ws://127.0.0.1:26557/websocket', batch_delay = '500ms' }
rpc_timeout = '10s'
trusted_node = false
account_prefix = 'cosmos'
key_name = 'testkey'
store_prefix = 'ibc'
default_gas = 100000
max_gas = 400000
gas_price = { price = 0.001, denom = 'stake' }
gas_multiplier = 1.1
max_msg_num = 30
max_tx_size = 2097152
clock_drift = '5s'
max_block_time = '30s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }
address_type = { derivation = 'cosmos' }

Filter incentivized packets

Hermes can be configured in order to only relay packets which are incentivized. This is done by using the [[chain.packet_filter.min_fees]] setting.

When this filter is configured, Hermes will only relay send_packet events when they meet the configured requirements. This configuration can be set per channel or for a set of channels using a wildcard expression.

WARNING: This configuration is experimental. Packet clearing will be disabled for the channels which have a fee filter configured, and some send_packet events might not be relayed if the incentivized event is not in the same batch of events.

Examples

Channel, amount and denom specific

This example will configure Hermes so it will ignore send_packet events from channel-0 which do not have at least 10 uatoms as the recv_fee.

[chains.packet_filter.min_fees.'channel-0']
  recv = [{ amount = 10, denom = 'uatom' }]

Amount and denom specific

This example will configure Hermes so it will ignore send_packet events from any channel which do not have at least 10 uatoms as the recv_fee.

[chains.packet_filter.min_fees.'*']
  recv = [{ amount = 10, denom = 'uatom' }]

Amount only

This example will configure Hermes so it will only relay send_packet events sent with incentivized events.

[chains.packet_filter.min_fees.'*']
  recv    = [{ amount = 0 }]

Multiple filters

This example will configure Hermes so it will ignore send_packet events from any channel which starts with ics, does not have at least 10 uatom or 20 stake as the recv_fee.

[chains.packet_filter.min_fees.'ics*']
  recv    = [{ amount = 10, denom = 'uatom' }, { amount = 20, denom = 'stake' }]

Performance tuning

Table of contents

• Overview
• Configuration Options
  • 1. Trusted Node
  • 2. Batch Delay
  • 3. Slow start
    • 3.1 Specify an allow list in the packet filter
    • 3.2 Disable clients, connections, channels workers and packet clearing on startup
• Conclusion

Overview

Hermes provides several configuration options that users can tweak to optimize its performance to suit specific requirements. This guide provides an overview of these options, and suggests ways to modify them for different scenarios.

The two per-chain configuration options you can use to tune the performance of Hermes as of version 1.5 are trusted_node and batch_delay.

Configuration Options

1. Trusted Node

The trusted_node setting is an experimental option that determines whether or not the full node is trusted.

trusted_node = false

When set to true, Hermes trusts the full node and does not verify headers included in the ClientUpdate message using the light client. This could lead to faster processing as it bypasses the verification step.

However, it's important to note that when the full node is configured as trusted, the verification traces will not be provided. This could potentially lead to failure in client updates after a significant change in validator sets.

If you prefer security over speed, or if the validator set changes frequently, consider leaving this setting as false, which is also the default.

2. Batch Delay

The batch_delay setting dictates the delay until an event batch is emitted if no NewBlock events have been received yet.

batch_delay = '500ms'

Lower batch_delay values will result in faster event processing, improving the latency of Hermes. However, setting it too low could sometimes cause events to be split across more batches than necessary, which will then cause Hermes to send more client updates than otherwise required. Conversely, higher values will increase the latency of Hermes, but will minimize the number of client updates.

If you prioritize processing speed and can tolerate the potentially slightly higher costs, consider setting a lower batch_delay. For backup relayer or settings where latency is not as important, consider a higher batch_delay.

The default 500ms provides a good balance between speed and reliability, while still minimizing the number of client updates to send.

3. Slow start

On blochains with many open channels, connections and/or clients, Hermes may take a long while to start. That is because Hermes needs to perform a scan of all available clients, connections and channels on that blockchain in order to refresh these clients, complete the handshakes of partially open channels and connections. If Hermes takes more than a couple minutes to start, that may be because there are too many clients, connections and/or channels.

To alleviate this issue, there are two potential solutions:

3.1 Specify an allow list in the packet filter

Add the end of the [[chains]] section for affected blockchain, you can specify a packet filter with an allow list. This will ensure Hermes will only scan for the listed channels, and gather the corresponding connections and clients for these channels only.

For example, to only relay on two channels named channel-0 and channel-1, on port transfer, you can add the following packet filter:

 [chains.packet_filter]
 policy = 'allow'
 list = [
   ['transfer', 'channel-0'],
   ['transfer', 'channel-1'],
 ]

Caveat: the allow list cannot contain any wildcards, otherwise Hermes will have to perform a full scan to gather all channels and subsequently filter them against the allow list.

For example, the following configuration would cause Hermes to perform such a full scan, and therefore potentially slow down its startup time:

 [chains.packet_filter]
 policy = 'allow'
 list = [
   ['ica*', '*'],
   ['transfer', 'channel-0'],
 ]

3.2 Disable clients, connections, channels workers and packet clearing on startup

If you wish to relay on all channels, or to use wildcards in the packet filter, then another option to speed up the startup is to disable scanning altogether.

At the moment, there is no single setting to do this, but by disabling the clients, connections and channels workers and setting clear_on_start to false under the mode.packets section, Hermes will not need to perform a scan and will only relay packets on active channels, provided they match the packet filter, if present. Otherwise Hermes will relay on all active channels.

Please note that because these settings are globa, they will affect the behaviour of Hermes for all chains listed in its configuration.

Here is how the configuration file should look like in order to disable scanning altogether.

# ...

[mode.clients]
enabled = false

# ...

[mode.connnections]
enabled = false

# ...

[mode.channels]
enabled = false

# ...

[mode.packets]
enabled = true
clear_on_start = false

Conclusion

The tuning of Hermes performance relies on the balance between processing speed and reliability. Keep in mind that tuning these configurations according to your needs could significantly improve the performance of your Hermes instance. Please thoroughly test any changes in a controlled environment before implementing them in a production setting.

Remember, every blockchain and network has unique characteristics that can affect the performance of the relayer, so there's no one-size-fits-all configuration. Feel free to experiment and fine-tune these settings to achieve optimal performance for your specific use case.

Telemetry

To gain a better understanding of the status and activity of the relayer, Hermes features a built-in telemetry service based on the OpenTelemetry observability framework, whose metrics can be exposed over HTTP for integration with the Prometheus monitoring system.

The official Hermes builds for Linux and macOS come with telemetry support since version v0.4.0. See the installation instructions for how to obtain the latest version of Hermes.

Configuration

The telemetry service is not active by default, and must be enabled in Hermes' configuration:

[telemetry]
enabled = true                                                  # default = false
host    = '127.0.0.1'                                           # default value
port    = 3001                                                  # default value
[telemetry.buckets]                                             # default value
latency_submitted = { start = 5000, end = 10000, buckets = 10 } # default value
latency_confirmed = { start = 5000, end = 10000, buckets = 10 } # default value

Please see the relevant section for Configuration for more general details about Hermes configuration options.

Hermes operators guide to using metrics

This section is a basic guide on how Hermes metrics can be used to observe both the current state of the Hermes relayer and the networks it is connected to.

General remarks about the metrics

• All Hermes metrics are tracked and updated from the moment the Hermes service (i.e., start) starts up. Metrics are automatically reset if the service is restarted.
• For maximum reliability, it is advised to combine monitoring of your Hermes service with monitoring of your full nodes.
• Some metrics require specific configurations to be enabled, this is described in the Configuration Dependencies column.

Table of Contents

Hermes' metrics are designed to be able to answer four basic questions:

1. Is Hermes active (i.e., submitting any transactions to any network)?
2. Are Hermes transactions successful (i.e., confirmed and included in the network)?
3. What is the overall IBC status of each network?
4. How efficient, and how secure is the IBC status on each network?
5. Am I getting fee rewards from ICS29 incentivized packets?

For each of this question, there is a dedicated subsection:

• Is Hermes active?
• Are Hermes transactions successful?
• What is the overall IBC status of each network?
• How efficient and how secure is the IBC status on each network?
• Am I getting fee rewards?

Is Hermes active?

By active, we mean specifically: is Hermes submitting any transactions to any network? The metrics in the table below are design to answer this question on multiple dimensions.

Notes & more details below:

What is a worker?

• A worker is a separate thread of execution and there are five types of workers:
  • Client: The worker that refreshed a client periodically and detects misbehaviour.
  • Connection: The worker that handles connection open handshake that may be incomplete.
  • Channel: The worker that handles channel open handshake that may be incomplete.
  • Packet: The worker that handles packet relaying.
  • Wallet: The worker that periodically queries for the balance of each wallet that Hermes is using and updates wallet_balance metric.
• For example, if your metrics show that you have 0 packet workers (workers{type="packet"} 0), that is a clear indication that Hermes is not relaying any packets at the moment.

How do we define the latency of a submitted transaction? The latency is defined as the difference between the moment when Hermes received an event (through the websocket) until the moment when the corresponding transaction(s) were submitted into a full node's mempool.

• If a transaction is submitted it does not mean it was confirmed, see below for more details.
• This metric is tracked per chain, counterparty chain, channel and port.

A note on wallet balances. For the wallet_balance, we convert from a String into a f64, which can lead to a loss in precision in the displayed value.

latency histogram The tx_latency_submitted and tx_latency_confirmed are displayed with histogram buckets which each contain the number of values less or equal to their bucket label. This means that if there are 5 buckets with label 500, 2000, 3000, 4000 and 5000 and 2 tx_latency_submitted were recorded of respectively 1800ms and 3100ms then the tx_latency_submitted will look like this:

tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="500"} 0
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="2000"} 1
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="3000"} 1
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="4000"} 2
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="5000"} 2
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="+Inf"} 2

The range of the buckets can be configured using the latency_submitted and latency_confirmed seen here

Are Hermes transactions successful?

This table shows the metrics for Hermes performance. Importantly, these metrics are only displayed if the configuration tx_confirmation = true is set in your config.toml.

How do we define the latency of a confirmed transaction? This is the difference between the moment when Hermes received an event until the corresponding transaction(s) were confirmed.

• Similarly to tx_latency_submitted, this metrics is tracked per chain, counterparty chain, channel and port.
• This metrics usually contains strictly larger values than tx_latency_submitted, because Hermes first submits transactions into the network's mempool, and then it takes some more time elapses until the network includes those transactions in a block.

What is the overall IBC status of each network?

These metrics are not specific to your Hermes instance. These are metrics that capture the activity of all IBC relayers.

‼️ Important: Your Hermes instance produces these metrics based on the events it receives via a websocket to the full nodes of each network. If these events are not being updated, that is a good indication that either:

• The network has no IBC activity, or
• The websocket connection to that network is broken.

Notes:

• Except for ws_reconnect_total, all these metrics should typically increase regularly in the common-case. That is an indication that the network is regularly producing new blocks and there is ongoing IBC activity, eg send_packet, acknowledgment, and timeout.
• The metric ws_reconnect_total signals that the websocket connection was broken and Hermes had to re-establish that. It is usually an indication that your full node may be falling behind or is experiencing instability.

Since Hermes v1, we also introduced 3 metrics that sketch the backlog status of IBC relaying.

Notes:

• The backlog_size defines how many IBC packets users sent and were not yet relayed (i.e., received on the destination network, or timed-out). If this metric is increasing, it signals that the packet queue is increasing and there may be some errors in the Hermes logs that need your attention.
• If the backlog_oldest_sequence remains unchanged for more than a few minutes, that means that the packet with the respective sequence number is likely blocked and cannot be relayed. To understand for how long the packet is block, Hermes will populate backlog_oldest_timestamp with the local time when it first observed the backlog_oldest_sequence that is blocked.

How efficient and how secure is the IBC status on each network?

Notes:

• The two metrics cleared_send_packet_count_total and cleared_acknowledgment_count_total are only populated if tx_confirmation = true. These two metrics usually correlate with backlog_* metrics. They are an indication that IBC packet relaying may be unsuccessful and that Hermes periodically finds packets to clear (i.e., unblock).
• queries_total and queries_cache_hits_total values are complementary. For the total number of queries, the two metrics should be summed for a specific query type.

For security, we only expose one metric, described in the table below. Note that this metrics is disabled if misbehaviour = false in your Hermes config.toml.

Am I getting fee rewards?

Integration with Prometheus

With the enabled = true setting for telemetry in your config.toml, the telemetry service will be enabled and will serve the metrics using the Prometheus encoder over HTTP at http://localhost:3001/metrics.

After starting Hermes with hermes start, and letting it run for a while to relay packets, open http://localhost:3001/metrics in a browser, you should see Prometheus-encoded metrics.

For example, with two channels and after transferring some tokens between the chains:

# HELP acknowledgement_events_total Number of WriteAcknowledgement events received
# TYPE acknowledgement_events_total counter
acknowledgement_events_total{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 4
acknowledgement_events_total{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
# HELP acknowledgment_packets_confirmed_total Number of confirmed acknowledgment packets. Available if relayer runs with Tx confirmation enabled
# TYPE acknowledgment_packets_confirmed_total counter
acknowledgment_packets_confirmed_total{dst_chain="ibc-0",dst_channel="channel-0",dst_port="transfer",service_name="unknown_service",src_chain="ibc-1",src_channel="channel-0",src_port="transfer",otel_scope_name="hermes",otel_scope_version=""} 1
acknowledgment_packets_confirmed_total{dst_chain="ibc-1",dst_channel="channel-0",dst_port="transfer",service_name="unknown_service",src_chain="ibc-0",src_channel="channel-0",src_port="transfer",otel_scope_name="hermes",otel_scope_version=""} 0
# HELP backlog_oldest_sequence Sequence number of the oldest SendPacket event in the backlog
# TYPE backlog_oldest_sequence gauge
backlog_oldest_sequence{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
backlog_oldest_sequence{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
# HELP backlog_oldest_timestamp Local timestamp for the oldest SendPacket event in the backlog
# TYPE backlog_oldest_timestamp gauge
backlog_oldest_timestamp{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
backlog_oldest_timestamp{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
# HELP backlog_size Total number of SendPacket events in the backlog
# TYPE backlog_size gauge
backlog_size{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
backlog_size{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
# HELP client_updates_submitted_total Number of client update messages submitted
# TYPE client_updates_submitted_total counter
client_updates_submitted_total{client="07-tendermint-0",dst_chain="ibc-0",service_name="unknown_service",src_chain="ibc-1",otel_scope_name="hermes",otel_scope_version=""} 2
client_updates_submitted_total{client="07-tendermint-0",dst_chain="ibc-1",service_name="unknown_service",src_chain="ibc-0",otel_scope_name="hermes",otel_scope_version=""} 2
# HELP ics29_period_fees Amount of ICS29 fees rewarded over the past 7 days
# TYPE ics29_period_fees gauge
ics29_period_fees{chain="ibc-0",denom="stake",receiver="cosmos1j6z6q9d2gf2suav88z8g3zf726vz9ehg4hkr8x",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
ics29_period_fees{chain="ibc-1",denom="stake",receiver="cosmos1340jyu3hawjzusu4jfwh29prpglkju5rlkpesn",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
ics29_period_fees{chain="ibc-2",denom="stake",receiver="cosmos1yxzuet72f4qlks8tzrna6y2q4wchur02gqs5al",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
ics29_period_fees{chain="ibc-3",denom="stake",receiver="cosmos1fk5ykcvkgr4yzlzfyegnaaqyc0ulv8wv8u9hjl",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
# HELP messages_submitted_total Number of messages submitted to a specific chain
# TYPE messages_submitted_total counter
messages_submitted_total{chain="ibc-0",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 7
messages_submitted_total{chain="ibc-1",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 7
messages_submitted_total{chain="ibc-2",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
messages_submitted_total{chain="ibc-3",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
# HELP otel_scope_info Instrumentation Scope metadata
# TYPE otel_scope_info gauge
otel_scope_info{otel_scope_name="hermes",otel_scope_version=""} 1
# HELP queries_cache_hits_total Number of cache hits for queries submitted by Hermes
# TYPE queries_cache_hits_total counter
queries_cache_hits_total{chain="ibc-0",query_type="query_channel",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 9
queries_cache_hits_total{chain="ibc-0",query_type="query_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 4
queries_cache_hits_total{chain="ibc-0",query_type="query_connection",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 11
queries_cache_hits_total{chain="ibc-0",query_type="query_latest_height",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_cache_hits_total{chain="ibc-1",query_type="query_channel",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 12
queries_cache_hits_total{chain="ibc-1",query_type="query_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 4
queries_cache_hits_total{chain="ibc-1",query_type="query_connection",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 11
queries_cache_hits_total{chain="ibc-1",query_type="query_latest_height",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_cache_hits_total{chain="ibc-2",query_type="query_channel",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_cache_hits_total{chain="ibc-2",query_type="query_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_cache_hits_total{chain="ibc-2",query_type="query_connection",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_cache_hits_total{chain="ibc-2",query_type="query_latest_height",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_cache_hits_total{chain="ibc-3",query_type="query_channel",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_cache_hits_total{chain="ibc-3",query_type="query_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_cache_hits_total{chain="ibc-3",query_type="query_connection",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_cache_hits_total{chain="ibc-3",query_type="query_latest_height",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
# HELP queries_total Number of queries submitted by Hermes
# TYPE queries_total counter
queries_total{chain="ibc-0",query_type="query_application_status",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 10
queries_total{chain="ibc-0",query_type="query_block",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="query_blocks",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="query_channel",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 9
queries_total{chain="ibc-0",query_type="query_channel_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="query_channels",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="query_client_connections",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-0",query_type="query_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 6
queries_total{chain="ibc-0",query_type="query_clients",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-0",query_type="query_commitment_prefix",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="query_config_params",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-0",query_type="query_connection",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-0",query_type="query_connection_channels",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-0",query_type="query_connections",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="query_consensus_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-0",query_type="query_consensus_states",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="query_latest_height",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-0",query_type="query_next_sequence_receive",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="query_packet_acknowledgements",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-0",query_type="query_packet_commitments",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-0",query_type="query_packet_events",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="query_staking_params",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-0",query_type="query_txs",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 28
queries_total{chain="ibc-0",query_type="query_unreceived_acknowledgements",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-0",query_type="query_unreceived_packets",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 8
queries_total{chain="ibc-0",query_type="query_upgraded_consensus_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-0",query_type="status",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-1",query_type="query_application_status",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 10
queries_total{chain="ibc-1",query_type="query_block",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="query_blocks",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="query_channel",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 3
queries_total{chain="ibc-1",query_type="query_channel_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="query_channels",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="query_client_connections",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-1",query_type="query_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 6
queries_total{chain="ibc-1",query_type="query_clients",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-1",query_type="query_commitment_prefix",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="query_config_params",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-1",query_type="query_connection",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-1",query_type="query_connection_channels",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-1",query_type="query_connections",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="query_consensus_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-1",query_type="query_consensus_states",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="query_latest_height",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-1",query_type="query_next_sequence_receive",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="query_packet_acknowledgements",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-1",query_type="query_packet_commitments",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-1",query_type="query_packet_events",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="query_staking_params",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-1",query_type="query_txs",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 20
queries_total{chain="ibc-1",query_type="query_unreceived_acknowledgements",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 8
queries_total{chain="ibc-1",query_type="query_unreceived_packets",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-1",query_type="query_upgraded_consensus_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-1",query_type="status",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-2",query_type="query_application_status",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_block",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_blocks",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_channel",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_channel_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_channels",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_client_connections",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_clients",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-2",query_type="query_commitment_prefix",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_config_params",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-2",query_type="query_connection",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_connection_channels",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_connections",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_consensus_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_consensus_states",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_latest_height",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-2",query_type="query_next_sequence_receive",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_packet_acknowledgements",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_packet_commitments",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_packet_events",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_staking_params",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-2",query_type="query_txs",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_unreceived_acknowledgements",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_unreceived_packets",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="query_upgraded_consensus_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-2",query_type="status",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-3",query_type="query_application_status",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_block",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_blocks",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_channel",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_channel_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_channels",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_client_connections",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_client_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_clients",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-3",query_type="query_commitment_prefix",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_config_params",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-3",query_type="query_connection",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_connection_channels",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_connections",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_consensus_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_consensus_states",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_latest_height",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
queries_total{chain="ibc-3",query_type="query_next_sequence_receive",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_packet_acknowledgements",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_packet_commitments",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_packet_events",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_staking_params",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
queries_total{chain="ibc-3",query_type="query_txs",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_unreceived_acknowledgements",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_unreceived_packets",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="query_upgraded_consensus_state",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
queries_total{chain="ibc-3",query_type="status",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
# HELP receive_packets_confirmed_total Number of confirmed receive packets. Available if relayer runs with Tx confirmation enabled
# TYPE receive_packets_confirmed_total counter
receive_packets_confirmed_total{dst_chain="ibc-0",dst_channel="channel-0",dst_port="transfer",service_name="unknown_service",src_chain="ibc-1",src_channel="channel-0",src_port="transfer",otel_scope_name="hermes",otel_scope_version=""} 4
receive_packets_confirmed_total{dst_chain="ibc-1",dst_channel="channel-0",dst_port="transfer",service_name="unknown_service",src_chain="ibc-0",src_channel="channel-0",src_port="transfer",otel_scope_name="hermes",otel_scope_version=""} 1
# HELP send_packet_events_total Number of SendPacket events received
# TYPE send_packet_events_total counter
send_packet_events_total{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
send_packet_events_total{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 4
# HELP timeout_events_total Number of TimeoutPacket events received
# TYPE timeout_events_total counter
timeout_events_total{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
timeout_events_total{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
# HELP timeout_packets_confirmed_total Number of confirmed timeout packets. Available if relayer runs with Tx confirmation enabled
# TYPE timeout_packets_confirmed_total counter
timeout_packets_confirmed_total{dst_chain="ibc-0",dst_channel="channel-0",dst_port="transfer",service_name="unknown_service",src_chain="ibc-1",src_channel="channel-0",src_port="transfer",otel_scope_name="hermes",otel_scope_version=""} 0
timeout_packets_confirmed_total{dst_chain="ibc-1",dst_channel="channel-0",dst_port="transfer",service_name="unknown_service",src_chain="ibc-0",src_channel="channel-0",src_port="transfer",otel_scope_name="hermes",otel_scope_version=""} 0
# HELP tx_latency_confirmed The latency for all transactions submitted & confirmed to a specific chain, i.e. the difference between the moment when Hermes received a batch of events until the corresponding transaction(s) were confirmed. Milliseconds.
# TYPE tx_latency_confirmed histogram
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="1000"} 0
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="5000"} 0
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="9000"} 2
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="13000"} 2
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="17000"} 2
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="20000"} 2
tx_latency_confirmed_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="+Inf"} 2
tx_latency_confirmed_sum{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 11980
tx_latency_confirmed_count{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="1000"} 0
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="5000"} 0
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="9000"} 1
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="13000"} 1
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="17000"} 1
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="20000"} 1
tx_latency_confirmed_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="+Inf"} 1
tx_latency_confirmed_sum{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 6021
tx_latency_confirmed_count{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 1
# HELP tx_latency_submitted The latency for all transactions submitted to a specific chain, i.e. the difference between the moment when Hermes received a batch of events and when it submitted the corresponding transaction(s). Milliseconds.
# TYPE tx_latency_submitted histogram
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="200"} 0
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="500"} 2
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="1000"} 2
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="2000"} 2
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="5000"} 2
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="10000"} 2
tx_latency_submitted_bucket{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="+Inf"} 2
tx_latency_submitted_sum{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 599
tx_latency_submitted_count{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="200"} 0
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="500"} 1
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="1000"} 2
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="2000"} 2
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="5000"} 2
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="10000"} 2
tx_latency_submitted_bucket{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version="",le="+Inf"} 2
tx_latency_submitted_sum{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 837
tx_latency_submitted_count{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 2
# HELP wallet_balance The balance of each wallet Hermes uses per chain. Please note that when converting the balance to f64 a loss in precision might be introduced in the displayed value
# TYPE wallet_balance gauge
wallet_balance{account="cosmos1340jyu3hawjzusu4jfwh29prpglkju5rlkpesn",chain="ibc-1",denom="stake",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 99994729
wallet_balance{account="cosmos1fk5ykcvkgr4yzlzfyegnaaqyc0ulv8wv8u9hjl",chain="ibc-3",denom="stake",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 100000000
wallet_balance{account="cosmos1j6z6q9d2gf2suav88z8g3zf726vz9ehg4hkr8x",chain="ibc-0",denom="stake",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 99993392
wallet_balance{account="cosmos1yxzuet72f4qlks8tzrna6y2q4wchur02gqs5al",chain="ibc-2",denom="stake",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 100000000
# HELP workers Number of workers
# TYPE workers gauge
workers{service_name="unknown_service",type="packet",otel_scope_name="hermes",otel_scope_version=""} 2
workers{service_name="unknown_service",type="wallet",otel_scope_name="hermes",otel_scope_version=""} 4
# HELP ws_events_total How many IBC events did Hermes receive via the websocket subscription
# TYPE ws_events_total counter
ws_events_total{chain="ibc-0",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 17
ws_events_total{chain="ibc-1",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 15
ws_events_total{chain="ibc-2",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 9
ws_events_total{chain="ibc-3",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 9
# HELP ws_reconnect_total Number of times Hermes reconnected to the websocket endpoint
# TYPE ws_reconnect_total counter
ws_reconnect_total{chain="ibc-0",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
ws_reconnect_total{chain="ibc-1",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
ws_reconnect_total{chain="ibc-2",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0
ws_reconnect_total{chain="ibc-3",service_name="unknown_service",otel_scope_name="hermes",otel_scope_version=""} 0

REST API

Since version 0.7.0.

Hermes features a built-in HTTP server which exposes information about the configuration and state via a REST API.

Table of Contents

• Configuration
• Endpoints
  • GET /version
  • GET /chains
  • GET /chain/:id
  • GET /state

Configuration

The REST API is not active by default, and must be enabled in the configuration:

[rest]
enabled = true
host    = '127.0.0.1'
port    = 3000

Endpoints

GET /version

This endpoint returns the version of the Hermes (under the ibc-relayer key) as well as the version of the REST server itself (under the ibc-relayer-rest key).

Example

❯ curl -s -X GET 'http://127.0.0.1:3000/version' | jq

[
  {
    "name": "ibc-relayer",
    "version": "v1.6.0"
  },
  {
    "name": "ibc-relayer-rest",
    "version": "0.1.0"
  }
]

GET /chains

This endpoint return the identifiers of the chains that Hermes is connected to. Those identifiers can be used with the /chain/:id endpoint to gather more information about each chain's configuration. See the next section for more details.

Example

❯ curl -s -X GET 'http://127.0.0.1:3000/chains' | jq

{
  "status": "success",
  "result": [
    "ibc-0",
    "ibc-1"
  ]
}

GET /chain/:id

This endpoint returns the configuration of the chain with the given identifier, where :id stands for the identifier.

Example

❯ curl -s -X GET 'http://127.0.0.1:3000/chain/ibc-0' | jq

{
  "status": "success",
  "result": {
    "id": "ibc-0",
    "rpc_addr": "http://127.0.0.1:26657/",
    "websocket_addr": "ws://127.0.0.1:26657/websocket",
    "grpc_addr": "http://127.0.0.1:9090/",
    "rpc_timeout": "10s",
    "account_prefix": "cosmos",
    "key_name": "testkey",
    "store_prefix": "ibc",
    "max_gas": 900000000,
    "gas_multiplier": 1.0,
    "max_msg_num": 60,
    "max_tx_size": 2097152,
    "clock_drift": "5s",
    "trusting_period": "14days",
    "trust_threshold": {
      "numerator": "1",
      "denominator": "3"
    },
    "gas_price": {
      "price": 0.001,
      "denom": "stake"
    },
    "packet_filter": {
      "policy": "allowall"
    }
  }
}

GET /state

This endpoint returns the current state of Hermes, namely which chains it is connected to, as well as a description of all the workers which are currently active.

❯ curl -s -X GET 'http://127.0.0.1:3000/state' | jq

{
  "status": "success",
  "result": {
    "chains": [
      "ibc-0",
      "ibc-1"
    ],
    "workers": {
      "Client": [
        {
          "id": 3,
          "object": {
            "type": "Client",
            "dst_chain_id": "ibc-1",
            "dst_client_id": "07-tendermint-0",
            "src_chain_id": "ibc-0"
          }
        },
        {
          "id": 4,
          "object": {
            "type": "Client",
            "dst_chain_id": "ibc-1",
            "dst_client_id": "07-tendermint-1",
            "src_chain_id": "ibc-0"
          }
        },
        {
          "id": 1,
          "object": {
            "type": "Client",
            "dst_chain_id": "ibc-0",
            "dst_client_id": "07-tendermint-0",
            "src_chain_id": "ibc-1"
          }
        },
        {
          "id": 2,
          "object": {
            "type": "Client",
            "dst_chain_id": "ibc-0",
            "dst_client_id": "07-tendermint-1",
            "src_chain_id": "ibc-1"
          }
        }
      ]
    }
  }
}

Advanced

Acquire advanced knowledge about hermes. In this section, we present a summary of the Hermes' features compared to other relayer implementations, and we provide general guidelines for troubleshooting.

Sections

• Features

  • Learn about Hermes' features and how it compares to another relayer implementation.

• Troubleshooting

  • Learn the general guidelines regarding troubleshooting.

Features

This section includes a summary of the supported and planned features. It also includes a feature matrix which compares hermes to the cosmos-go-relayer.

Cosmos SDK & IBC compatibility: Hermes supports Cosmos SDK chains implementing the IBC protocol v1 protocol specification. Cosmos SDK versions 0.44.0 through 0.47.x are officially supported. IBC-go versions 1.1.* thorough 7.* are officially supported. In case Hermes finds an incompatible SDK or IBC-go version, it will output a log warning upon initialization as part of the start command or upon health-check command.

Supported Features

• Basic features
  • Create and update clients.
  • Refresh clients to prevent expiration.
  • Establish connections with new or existing clients.
  • Establish channels with new or existing connection.
  • Channel closing handshake.
  • Relay packets, acknowledgments, timeout and timeout-on-close packets, with zero or non-zero delay.
  • Queries for all objects.
• Packet relaying over:
  • multiple paths, for the chains in config.toml.
• Restart support:
  • Clear packets.
  • Resume channel handshake if configured to relay all.
  • Resume connection handshake if configured to relay all.
• Client upgrade:
  • Upgrading clients after a counterparty chain has performed an upgrade for IBC breaking changes.
• Packet delay:
  • Establish path over non-zero delay connection.
  • Relay all packets with the specified delay.
• Interchain Accounts & Interchain Security

  Relaying between Interchain Security-enabled chains requires Hermes v1.2+.

• Monitor and submit misbehaviour for clients
  • Monitor client updates for misbehaviour (fork and BFT time violation).
  • Submit misbehaviour evidence to the reference chain via /broadcast_evidence tendermint RPC.
  • Submit misbehaviour evidence to the on-chain IBC client in a transaction that includes the MsgMisbehaviour message.

  Misbehaviour submission to full node not yet supported.

• Individual commands that build and send transactions for:
  • Creating and updating IBC Tendermint light clients.
  • Sending connection open handshake messages.
  • Sending channel open handshake messages.
  • Sending channel closing handshake messages.
  • Initiating a cross chain transfer (mainly for testing).
  • Relaying sent packets, acknowledgments and timeouts.
  • Automatically generate a configuration file from the chain-registry
  • Client upgrade.
• Channel handshake for existing channel that is not in Open state.
• Connection handshake for existing connection that is not in Open state.
• Telemetry support.

Upcoming / Unsupported Features

Planned features:

• Interchain Queries
• Non-SDK support
• Relay from all IBC events, including governance upgrade proposal
• Dynamic & automatic configuration management

Features matrix

Legend:

Feature comparison between Hermes and the Go relayer

Troubleshooting

This section provides guidelines regarding troubleshooting.

Sections

• Help Command
  • Learn about hermes help command, providing a CLI documentation for all hermes commands.
• Profiling
  • Learn how to profile your Hermes binary to identify slow methods and bottlenecks.
• Parametrize the log level
  • Learn how to configure the log-level to help with debugging.
• Patch Gaia
  • Learn how to patch your local gaia chain(s) to enable some corner-case methods (e.g., channel close).
• Inspecting the relayer state
  • Learn how to inspect the state of Hermes.
• Cross-stack misconfiguration
  • Learn how to configure Hermes, Tendermint, and the SDK such that they play well with Hermes.
• Genesis restart without IBC upgrade proposal
  • Learn how to update a client after a chain undergoes a genesis restart without an IBC upgrade proposal.

Help command

The CLI comprises a special help command, which accepts as parameter other commands, and provides guidance on what is the correct way to invoke those commands.

NOTE: This special help command is preferred as it will display the full help message.

For instance,

hermes help create

will provide details about all the valid invocations of the create CLI command.

DESCRIPTION:
Create objects (client, connection, or channel) on chains

USAGE:
    hermes create <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    channel       Create a new channel between two chains
    client        Create a new IBC client
    connection    Create a new connection between two chains
    help          Print this message or the help of the given subcommand(s)

This can provide further specific guidance if we add additional parameters, e.g.,

hermes help create channel

DESCRIPTION:
Create a new channel between two chains.

Can create a new channel using a pre-existing connection or alternatively, create a new client and a
new connection underlying the new channel if a pre-existing connection is not provided.

USAGE:
    hermes create channel [OPTIONS] --a-chain <A_CHAIN_ID> --a-connection <A_CONNECTION_ID> --a-port <A_PORT_ID> --b-port <B_PORT_ID>

    hermes create channel [OPTIONS] --a-chain <A_CHAIN_ID> --b-chain <B_CHAIN_ID> --a-port <A_PORT_ID> --b-port <B_PORT_ID> --new-client-connection

OPTIONS:
        --channel-version <VERSION>
            The version for the new channel
            
            [aliases: chan-version]

    -h, --help
            Print help information

        --new-client-connection
            Indicates that a new client and connection will be created underlying the new channel
            
            [aliases: new-client-conn]

        --order <ORDER>
            The channel ordering, valid options 'unordered' (default) and 'ordered'
            
            [default: ORDER_UNORDERED]

        --yes
            Skip new_client_connection confirmation

FLAGS:
        --a-chain <A_CHAIN_ID>
            Identifier of the side `a` chain for the new channel

        --a-connection <A_CONNECTION_ID>
            Identifier of the connection on chain `a` to use in creating the new channel
            
            [aliases: a-conn]

        --a-port <A_PORT_ID>
            Identifier of the side `a` port for the new channel

        --b-chain <B_CHAIN_ID>
            Identifier of the side `b` chain for the new channel

        --b-port <B_PORT_ID>
            Identifier of the side `b` port for the new channel

Additionally, the -h/--help flags typical for CLI applications work on all commands.

Profiling

The relayer crate provides a time! macro which can be used to measure how much time is spent between the invocation of the macro and the end of the enclosing scope.

Setup

The time! macro has no effect unless the --debug=profiling global flag is specified on the command-line:

$ hermes --debug=profiling start

Example

#![allow(unused)]
fn main() {
fn my_function(x: u32) -> u32 {
    time!("myfunction: x={}", x); // A

    std::thread::sleep(Duration::from_secs(1));

    {
        time!("inner operation"); // B

        std::thread::sleep(Duration::from_secs(2));

        // timer B ends here
    }

    x + 1

    // timer A ends here
}
}

Console output

Jan 20 11:28:46.841  INFO relayer::macros::profiling: ⏳ myfunction: x=42 - start
Jan 20 11:28:47.842  INFO relayer::macros::profiling:    ⏳ inner operation - start
Jan 20 11:28:49.846  INFO relayer::macros::profiling:    ⏳ inner operation - elapsed: 2004ms
Jan 20 11:28:49.847  INFO relayer::macros::profiling: ⏳ myfunction: x=42 - elapsed: 3005ms

Profiling is useful for tracking down unusually slow methods. Each transaction or query usually consists of multiple lower-level methods, and it's often not clear which of these are the culprit for low performance. With profiling enabled, hermes will output timing information for individual methods involved in a command.

NOTE: To be able to see the profiling output, the [log level][log-level] should be info level or lower.

Example output for tx conn-init command

hermes tx conn-init --dst-chain ibc-0 --src-chain ibc-1 --dst-client 07-tendermint-0 --src-client 07-tendermint-0

Apr 13 20:58:21.225  INFO ibc_relayer::macros::profiling: ⏳ init_light_client - start
Apr 13 20:58:21.230  INFO ibc_relayer::macros::profiling: ⏳ init_light_client - elapsed: 4ms
Apr 13 20:58:21.230  INFO ibc_relayer::macros::profiling: ⏳ init_event_monitor - start
Apr 13 20:58:21.235  INFO ibc_relayer::macros::profiling: ⏳ init_event_monitor - elapsed: 5ms
Apr 13 20:58:21.235  INFO ibc_relayer::event::monitor: running listener chain.id=ibc-1
Apr 13 20:58:21.236  INFO ibc_relayer::macros::profiling: ⏳ init_light_client - start
Apr 13 20:58:21.239  INFO ibc_relayer::macros::profiling: ⏳ init_light_client - elapsed: 2ms
Apr 13 20:58:21.239  INFO ibc_relayer::macros::profiling: ⏳ init_event_monitor - start
Apr 13 20:58:21.244  INFO ibc_relayer::macros::profiling: ⏳ init_event_monitor - elapsed: 4ms
Apr 13 20:58:21.244  INFO ibc_relayer::event::monitor: running listener chain.id=ibc-0
Apr 13 20:58:21.244  INFO ibc_relayer::macros::profiling: ⏳ get_signer - start
Apr 13 20:58:21.246  INFO ibc_relayer::macros::profiling: ⏳ get_signer - elapsed: 1ms
Apr 13 20:58:21.246  INFO ibc_relayer::macros::profiling: ⏳ query_latest_height - start
Apr 13 20:58:21.246  INFO ibc_relayer::macros::profiling:    ⏳ block_on - start
Apr 13 20:58:21.248  INFO ibc_relayer::macros::profiling:    ⏳ block_on - elapsed: 1ms
Apr 13 20:58:21.249  INFO ibc_relayer::macros::profiling: ⏳ query_latest_height - elapsed: 3ms
Apr 13 20:58:21.250  INFO ibc_relayer::macros::profiling: ⏳ unbonding_period - start
Apr 13 20:58:21.250  INFO ibc_relayer::macros::profiling:    ⏳ block_on - start
Apr 13 20:58:21.251  INFO ibc_relayer::macros::profiling:    ⏳ block_on - elapsed: 0ms
Apr 13 20:58:21.270  INFO ibc_relayer::macros::profiling:    ⏳ block_on - start
Apr 13 20:58:21.273  INFO ibc_relayer::macros::profiling:    ⏳ block_on - elapsed: 2ms
Apr 13 20:58:21.273  INFO ibc_relayer::macros::profiling: ⏳ unbonding_period - elapsed: 23ms
Apr 13 20:58:21.279  INFO ibc_relayer::macros::profiling: ⏳ build_consensus_state - start
Apr 13 20:58:21.280  INFO ibc_relayer::macros::profiling: ⏳ build_consensus_state - elapsed: 0ms
Apr 13 20:58:21.280  INFO ibc_relayer::macros::profiling: ⏳ send_msgs - start
Apr 13 20:58:21.280  INFO ibc_relayer::macros::profiling:    ⏳ send_tx - start
Apr 13 20:58:21.282  INFO ibc_relayer::macros::profiling:       ⏳ PK "03f17d2c094ee68cfcedb2c2f2b7dec6cd82ea158ac1c32d3de0ca8b288a3c8bfa" - start
Apr 13 20:58:21.282  INFO ibc_relayer::macros::profiling:          ⏳ block_on - start
Apr 13 20:58:21.285  INFO ibc_relayer::macros::profiling:          ⏳ block_on - elapsed: 3ms
Apr 13 20:58:21.296  INFO ibc_relayer::macros::profiling:             ⏳ block_on - start
Apr 13 20:58:22.664  INFO ibc_relayer::macros::profiling:             ⏳ block_on - elapsed: 1367ms
Apr 13 20:58:22.664  INFO ibc_relayer::macros::profiling:       ⏳ PK "03f17d2c094ee68cfcedb2c2f2b7dec6cd82ea158ac1c32d3de0ca8b288a3c8bfa" - elapsed: 1382ms
Apr 13 20:58:22.664  INFO ibc_relayer::macros::profiling:    ⏳ send_tx - elapsed: 1384ms
Apr 13 20:58:22.664  INFO ibc_relayer::macros::profiling: ⏳ send_msgs - elapsed: 1384ms
Success: CreateClient(
    CreateClient(
        Attributes {
            height: Height {
                revision: 0,
                height: 10675,
            },
            client_id: ClientId(
                "07-tendermint-7",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 1,
                height: 10663,
            },
        },
    ),
)

JSON output

Additionally, if the --debug=profiling-json flag is specified, Hermes will output profiling information in JSON format in a file named hermes-YYYY-MM-DD-HHMMSS-prof.json, in the directory specified in the PROFILING_DIR env variable, or the current directory otherwise.

NOTE: Outputting profiling information in JSON format in a file is only available for the hermes start command. This debug option won't do anything with the other CLIs.

{"name":"fetch_node_info","src_chain":"ibc-0","elapsed":6}
{"name":"chain_status","src_chain":"ibc-0","elapsed":12}
{"name":"query_config_params","src_chain":"ibc-0","elapsed":3}
{"name":"min_gas_price","src_chain":"ibc-0","elapsed":3}
{"name":"query_staking_params","src_chain":"ibc-0","elapsed":159}
{"name":"historical_entries","src_chain":"ibc-0","elapsed":329}
{"name":"query_staking_params","src_chain":"ibc-0","elapsed":121}
{"name":"unbonding_period","src_chain":"ibc-0","elapsed":12}
{"name":"query_latest_height","src_chain":"ibc-0","elapsed":8}
{"name":"fetch_node_info","src_chain":"ibc-1","elapsed":9}
{"name":"chain_status","src_chain":"ibc-1","elapsed":43}

Parametrize the log level

This section explains how to parametrize the log output level of hermes.

The configuration file permits parametrization of output verbosity via the knob called log_level. This file is loaded by default from $HOME/.hermes/config.toml, but can be overridden in all commands with the --config flag, e.g. hermes --config $CONFIGPATH subcommand.

Relevant snippet:

[global]
log_level = 'error'

Valid options for log_level are: 'error', 'warn', 'info', 'debug', 'trace'. These levels correspond to the tracing subcomponent of the relayer-cli, see here.

Hermes will always print a last line summarizing the result of its operation for queries or transactions. In addition to this last line, arbitrary debug, info, or other outputs may be produced.

Overriding the tracing filter using RUST_LOG

For debugging purposes, we may want to inspect which RPC queries Hermes is making. Hermes makes use of the tendermint-rpc library to issue RPC queries, but the output of this library is by default turned off in order to keep the logs more readable.

Using the RUST_LOG environment variable, we can turn logging on for the tendermint-rpc library, as follows:

RUST_LOG=tendermint-rpc=debug,info hermes start

Setting the RUST_LOG environment variable to tendermint_rpc=debug,info instructs Hermes to set the log level of the tendermint_rpc crate to debug and otherwise use the info log level.

Note: While the tendermint-rpc contains a dash in its name, the logging filter expects a module name, which can only contain alphanumeric characters and underscores, hence why the filter above is written tendermint_rpc=debug.

Example:

❯ RUST_LOG=tendermint_rpc=debug,info hermes start
2022-02-24T14:32:14.039555Z  INFO ThreadId(01) using default configuration from '/Users/coromac/.hermes/config.toml'
2022-02-24T14:32:14.043500Z  INFO ThreadId(01) telemetry service running, exposing metrics at http://127.0.0.1:3001/metrics
2022-02-24T14:32:14.043542Z  INFO ThreadId(01) [rest] address not configured, REST server disabled
2022-02-24T14:32:14.049759Z DEBUG ThreadId(01) Incoming response: {
  "jsonrpc": "2.0",
  "id": "143b4580-c49e-47c1-81b2-4e7090f6e762",
  "result": {
    "node_info": {
      "protocol_version": {
        "p2p": "8",
        "block": "11",
        "app": "0"
      },
      "id": "73f9134539f9845cd253dc302e36d48ee4c0f32d",
      "listen_addr": "tcp://0.0.0.0:27003",
      "network": "ibc0",
      "version": "v0.34.14",
      "channels": "40202122233038606100",
      "moniker": "ibc0",
      "other": {
        "tx_index": "on",
        "rpc_address": "tcp://0.0.0.0:27000"
      }
    },
    "sync_info": {
      "latest_block_hash": "8396B93E355AD80EED8167A04BB9858A315A8BEB482547DE16A6CD82BC11551B",
      "latest_app_hash": "22419E041D6997EE75FF66F7F537A3D36122B220EAB89A9C246FEF680FB1C97A",
      "latest_block_height": "86392",
      "latest_block_time": "2022-02-24T14:32:08.673989Z",
      "earliest_block_hash": "0A73CFE8566D4D4FBFE3178D9BCBAD483FD689854CA8012FF1457F8EC4598132",
      "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
      "earliest_block_height": "1",
      "earliest_block_time": "2022-01-20T09:04:21.549736Z",
      "catching_up": false
    },
    "validator_info": {
      "address": "6FD56E6AA1EEDAD227AFAB6B9DE631719D4A3691",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "mR5V/QWOv/mJYyNmlsl3mfxKy1PNaOzdztyas4NF2BA="
      },
      "voting_power": "10"
    }
  }
}
2022-02-24T14:32:14.052503Z DEBUG ThreadId(21) Incoming response: {
  "jsonrpc": "2.0",
  "id": "0ca35e64-ea98-4fbf-bd66-c3291128ace9",
  "result": {}
}

...

The two DEBUG log lines above were emitted by the tendermint-rpc crate.

Patch Gaia

The guide below refers specifically to patching your gaia chain so that the relayer can initiate the closing of channels by submitting a ChanCloseInit message. Without this modification, the transaction will be rejected. We also describe how to test the channel closing feature.

• Clone the Cosmos SDK

  git clone https://github.com/cosmos/cosmos-sdk.git ~/go/src/github.com/cosmos/cosmos-sdk
  cd ~/go/src/github.com/cosmos/cosmos-sdk
  

• Apply these diffs:

     --- a/x/ibc/applications/transfer/module.go
     +++ b/x/ibc/applications/transfer/module.go
     @@ -305,7 +305,7 @@ func (am AppModule) OnChanCloseInit(
             channelID string,
      ) error {
             // Disallow user-initiated channel closing for transfer channels
     -       return sdkerrors.Wrap(sdkerrors.ErrInvalidRequest, "user cannot close channel")
     +       return nil
      }
  

• Append the line below (watch for the placeholder <your>) as the last line in your go.mod in the gaia clone:

replace github.com/cosmos/cosmos-sdk => /Users/<your>/go/src/github.com/cosmos/cosmos-sdk

• Now make build and make install your local copy of gaia

In order to test the correct operation during the channel close, perform the steps below.

• The channel should be in state open-open:

• Transfer of 5555 samoleans from ibc-1 to ibc-0. This results in a Tx to ibc-1 for a MsgTransfer packet. Make sure you're not relaying this packet (Hermes should not be running on this path).

  hermes tx ft-transfer --timeout-height-offset 1000 --denom samoleans --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-1 --amount 5555
  

• Now do the first step of channel closing: the channel will transition to close-open:

  hermes tx chan-close-init --dst-chain ibc-0 --src-chain ibc-1 --dst-connection connection-0 --dst-port transfer --src-port transfer --dst-channel channel-0 --src-channel channel-1
  

• Trigger timeout on close to ibc-1

  hermes tx packet-recv --dst-chain ibc-0 --src-chain ibc-1 --src-port transfer --src-channel channel-1
  

• Close the channel

  hermes tx chan-close-confirm --dst-chain ibc-1 --src-chain ibc-0 --dst-connection connection-1 --dst-port transfer --src-port transfer --dst-channel channel-1 --src-channel channel-0
  

• Verify that the two ends are in Close state:

  hermes query channel end --chain ibc-0 --port transfer --channel channel-0
  hermes query channel end --chain ibc-1 --port transfer --channel channel-1
  

Inspecting the relayer state

To get some insight into the state of Hermes, Hermes will react to a SIGUSR1 signal by dumping its state to the console, either in plain text form or as a JSON object if Hermes was started with the --json option.

To send a SIGUSR1 signal to Hermes, look up its process ID (below PID) and use the following command:

kill -SIGUSR1 PID

Hermes will print some information about the workers which are currently running.

For example, with three chains configured and one channel between each pair of chains:

INFO Dumping state (triggered by SIGUSR1)
INFO
INFO * Chains: ibc-0, ibc-1, ibc-2
INFO * Client workers:
INFO   - client::ibc-0->ibc-1:07-tendermint-0 (id: 5)
INFO   - client::ibc-0->ibc-2:07-tendermint-0 (id: 9)
INFO   - client::ibc-1->ibc-0:07-tendermint-0 (id: 1)
INFO   - client::ibc-1->ibc-2:07-tendermint-1 (id: 11)
INFO   - client::ibc-2->ibc-0:07-tendermint-1 (id: 3)
INFO   - client::ibc-2->ibc-1:07-tendermint-1 (id: 7)
INFO * Packet workers:
INFO   - packet::channel-0/transfer:ibc-0->ibc-1 (id: 2)
INFO   - packet::channel-0/transfer:ibc-1->ibc-0 (id: 6)
INFO   - packet::channel-0/transfer:ibc-2->ibc-0 (id: 10)
INFO   - packet::channel-1/transfer:ibc-0->ibc-2 (id: 4)
INFO   - packet::channel-1/transfer:ibc-1->ibc-2 (id: 8)
INFO   - packet::channel-1/transfer:ibc-2->ibc-1 (id: 12)

or in JSON form (prettified):

{
  "timestamp": "Jul 12 17:04:37.244",
  "level": "INFO",
  "fields": {
    "message": "Dumping state (triggered by SIGUSR1)"
  }
}
{
  "chains": [
    "ibc-0",
    "ibc-1",
    "ibc-2"
  ],
  "workers": {
    "Client": [
      {
        "id": 5,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-1",
          "dst_client_id": "07-tendermint-0",
          "src_chain_id": "ibc-0"
        }
      },
      {
        "id": 9,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-2",
          "dst_client_id": "07-tendermint-0",
          "src_chain_id": "ibc-0"
        }
      },
      {
        "id": 1,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-0",
          "dst_client_id": "07-tendermint-0",
          "src_chain_id": "ibc-1"
        }
      },
      {
        "id": 11,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-2",
          "dst_client_id": "07-tendermint-1",
          "src_chain_id": "ibc-1"
        }
      },
      {
        "id": 3,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-0",
          "dst_client_id": "07-tendermint-1",
          "src_chain_id": "ibc-2"
        }
      },
      {
        "id": 7,
        "object": {
          "type": "Client",
          "dst_chain_id": "ibc-1",
          "dst_client_id": "07-tendermint-1",
          "src_chain_id": "ibc-2"
        }
      }
    ],
    "Packet": [
      {
        "id": 2,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-1",
          "src_chain_id": "ibc-0",
          "src_channel_id": "channel-0",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 6,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-0",
          "src_chain_id": "ibc-1",
          "src_channel_id": "channel-0",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 10,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-0",
          "src_chain_id": "ibc-2",
          "src_channel_id": "channel-0",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 4,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-2",
          "src_chain_id": "ibc-0",
          "src_channel_id": "channel-1",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 8,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-2",
          "src_chain_id": "ibc-1",
          "src_channel_id": "channel-1",
          "src_port_id": "transfer"
        }
      },
      {
        "id": 12,
        "object": {
          "type": "Packet",
          "dst_chain_id": "ibc-1",
          "src_chain_id": "ibc-2",
          "src_channel_id": "channel-1",
          "src_port_id": "transfer"
        }
      }
    ]
  }
}

Hermes versus App/SDK/Tendermint Configuration

This table summarizes the different configuration parameter combinations that may cause Hermes to raise errors. It gives some information on the failure type and refers to the relevant section of the guide for more information. Following notations are used:

• tendermint.<parameter>:
  • tendermint refers to the Tendermint configuration file config.toml
  • <parameter> refers to the parameter in this file.
• app.<parameter>:
  • app refers to the application configuration file app.toml
  • <parameter> refers to the parameter in this file.
• genesis.<parameter>:
• genesis refers to the application configuration file genesis.json
• <parameter> refers to the parameter in this file.

Hermes vs other configuration parameters that may cause Hermes failures

Recheck

When relaying packets, Hermes may send up multiple transactions to the full node's mempool. Hermes uses the broadcast_tx_sync RPC which does some basic verification and then returns the Tx hash back.

Unless configured with sequential_batch_tx = true, Hermes does not wait for a transaction to be included in a block before sending the next transaction. For this to be possible, Hermes keeps track of the account sequence number locally, incrementing it after each succesfull broadcast_tx_sync RPC.

During peak periods, it is possible that not all Tx-es in the mempool are included in a block. In order for new transactions to be accepted along with the pending Tx-es, the full node must be configured with recheck = true. Otherwise, Hermes may get the following error:

2022-10-25T13:52:51.369822Z  WARN ThreadId(18) send_messages_and_wait_commit
  {chain=ibc-0 tracking_id=ft-transfer}:send_tx_with_account_sequence_retry{chain=ibc-0 account.sequence=88}: 
    failed to broadcast tx because of a mismatched account sequence number, refreshing account sequence number and 
      retrying once response=Response { code: Err(32), data: Data([]), log: Log("account sequence mismatch, 
        expected 69, got 88: incorrect account sequence"), 
      hash: transaction::Hash(DFC53B04CE095CD045E4E89D7CEB095BF977B876FD8D3FB1A7F0AC288B58B9C4) }

Fix

Ensure that the full node is configured with recheck = true. This ensures that the mempool rechecks the Tx-es left in the mempool before accepting new incoming transactions, therefore maintaining the order of transactions.

Minimum Gas Price

Hermes sends transactions using the gas_price parameter from the chain section in the Hermes config.toml configuration file. The full node will not accept any transactions with a gas price smaller than what is configured for the applications (app.minimum-gas-prices) and Hermes will log an insufficient fees error:

2022-10-27T12:45:07.820543Z ERROR ThreadId(18) send_messages_and_wait_commit{chain=ibc-0 tracking_id=ft-transfer}:
  send_tx_with_account_sequence_retry{chain=ibc-0 account.sequence=48}: failed to broadcast tx with unrecoverable error
     response=Response { code: Err(13), data: Data([]), log: Log("insufficient fees; got: 99stake required: 198stake: insufficient fee"), 
       hash: transaction::Hash(AFB9FE23DE9108D349B8679561D7F00DF00863749D7827C3972DFB391CF8E526) } 
         diagnostic=the price configuration for this chain may be too low! please check the `gas_price.price` Hermes config.toml
ERROR transfer error: tx response event consists of an error: check_tx (broadcast_tx_sync) on chain ibc-0 for 
  Tx hash AFB9FE23DE9108D349B8679561D7F00DF00863749D7827C3972DFB391CF8E526 reports error: 
    vcode=Err(13), log=Log("insufficient fees; got: 99stake required: 198stake: insufficient fee")

Fix

Ensure that the gas_price.price parameter in the chain section of the Hermes config.toml configuration file is greater than or equal to the app.minimum-gas-prices.

Out of Gas

Before Hermes sends a transaction, it estimates how much gas the transaction will require by calling the SimulateTx() gRPC. This returns the amount of gas that the Tx requires given the application state at the time of the call. It is possible that by the time the transaction is sent and checked by the application, the amount of gas required has increased. To help alleviate this, the gas estimation is adjusted upward by multiplying it by the gas_multiplier parameter: gas_multiplier * simulated_gas.

If the adjusted amount of gas ends up still being not enough for the transaction to be successfully submitted, e.g., the gas_multiplier parameter is set to 1.0 such that no adjustment is actually performed, Hermes returns the following error:

ERROR transfer error: tx response event consists of an error: deliver_tx for 496835FF5A2F73F38ADA416506F7F1143BBD570E77217DC309CAD979924F0E70 reports error: code=Err(11), log=Log("out of gas in location: WriteFlat; gasWanted: 990000, gasUsed: 990724: out of gas")

Fix

Ensure that the gas_multiplier parameter in the chain section of the Hermes config.toml configuration file is configured such that it allows some increase over the simulated gas. A good value is for example 1.1.

Maximum Tx Size

When Hermes relays packets or handshake messages, it will build multi-message Tx-es with up to max_num_msgs number of messages or up to a Tx size of max_tx_size bytes. The full node accepts only Tx-es with size up to the max_tx_bytes parameter in its config.toml. If a Tx is too large, the full node rejects them with the following error:

2022-10-27T13:59:43.650251Z ERROR ThreadId(18) send_messages_and_wait_commit{chain=ibc-0 tracking_id=ft-transfer}:send_tx_with_account_sequence_retry{chain=ibc-0 account.sequence=159}: gas estimation failed or encountered another unrecoverable error error=RPC error to endpoint http://127.0.0.1:26657/: response error: Internal error: Tx too large. Max size is 500, but got 615 (code: -32603)
ERROR transfer error: failed while submitting the Transfer message to chain ibc-0: RPC error to endpoint http://127.0.0.1:26657/: response error: Internal error: Tx too large. Max size is 500, but got 615 (code: -32603)

Fix

Ensure that the max_tx_size parameter configured for Hermes is smaller than the max_tx_bytes parameter configured for the full node.

Allowed Clients

The SDK chain genesis file contains a list of allowed clients that can be created on a chain. If the chain is configured with genesis.app_state.ibc.client_genesis.allowed_clients that doesn't include 07-tendermint, then the chain will not allow tendermint IBC clients to be created and Hermes cannot open an IBC connection with this chain. When attempting to create a client in this scenario, Hermes will get the following error:

2022-10-27T14:48:04.632320Z ERROR ThreadId(35) send_messages_and_wait_commit{chain=ibc-0 tracking_id=create client}:send_tx_with_account_sequence_retry{chain=ibc-0 account.sequence=0}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 0: client state type 07-tendermint is not registered in the allowlist: invalid client type [cosmos/ibc-go/v3@v3.0.0/modules/core/02-client/keeper/client.go:22] With gas wanted: '0' and gas used: '64671' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "9"} }

Fix

Ensure that the genesis.app_state.ibc.client_genesis.allowed_clients parameter in the chain genesis file includes 07-tendermint.

Historical Entries

The presence of recent consensus states on a chain is required when processing some connection handshake messages, such as MsgConnectionOpenTry and MsgConnectionOpenAck. Such message types require proof verification that the counterparty chain has a valid client for this chain. If the chain is configured with genesis.app_state.staking.params.historical_entries = 0, then the chain will not store historical consensus state information and Hermes cannot open IBC connections with this chain. It will report an error message which looks like this:

2022-10-27T14:54:51.076598Z ERROR ThreadId(18) send_messages_and_wait_commit{chain=ibc-0 tracking_id=ConnectionOpenAck}:send_tx_with_account_sequence_retry{chain=ibc-0 account.sequence=6}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: connection handshake open ack failed: self consensus state not found for height 0-88: no historical info found at height 88: not found [cosmos/ibc-go/v3@v3.0.0/modules/core/02-client/keeper/keeper.go:256] With gas wanted: '0' and gas used: '130807' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "90"} }

The chain genesis misconfiguration is caught when the health check runs, e.g. hermes health-check, hermes start. A warning message such as the following is printed:

2022-10-27T15:00:14.583244Z  WARN ThreadId(40) health_check{chain=ibc-0}: Health checkup for chain 'ibc-0' failed
2022-10-27T15:00:14.583308Z  WARN ThreadId(40) health_check{chain=ibc-0}:     Reason: staking module for chain 'ibc-0' does not maintain any historical entries (`historical_entries` staking params is set to 0)
2022-10-27T15:00:14.583343Z  WARN ThreadId(40) health_check{chain=ibc-0}:     Some Hermes features may not work in this mode!

Fix

Ensure that the genesis.app_state.staking.params.historical_entries parameter in the chain genesis file is greater than 0. Ideal values are > 100.

Maximum Block Time

There are two cases where misconfiguration of the maximum block time may cause Hermes to fail. They are described in the following sections.

Header in the Future

When a client for chain ref_chain is created by Hermes on chain tgt_chain, its max_clock_drift is computed as: ref_chain.clock_drift + tgt_chain.clock_drift + tgt_chain.max_block_time Hermes may delay a client update with header for one block if it determines that sending it immediately would cause the chain to reject the update as being in the future.

Hermes sends the update in the case when header.timestamp <= dst_timestamp + max_clock_drift where dst_timestamp is the last block time on the destination chain. Otherwise, it waits for the next block and retries the update. If the check fails again, an error is returned:

2022-10-28T08:42:37.423739Z  WARN ThreadId(01) foreign_client.build_update_client_and_send{client=ibc-1->ibc-0:07-tendermint-0 target_query_height=latest height}:foreign_client.wait_and_build_update_client_with_trusted{client=ibc-1->ibc-0:07-tendermint-0 target_height=1-2182}:foreign_client.build_update_client_with_trusted{client=ibc-1->ibc-0:07-tendermint-0 target_height=1-2182}:foreign_client.wait_for_header_validation_delay{client=ibc-1->ibc-0:07-tendermint-0}: src header Timestamp(2022-10-28T08:42:35.600792Z) is after dst latest header Timestamp(2022-10-28T08:42:17.864603Z) + client state drift 3s,wait for next height on ibc-0
ERROR foreign client error: update header from ibc-1 with height 1-2182 and time Timestamp(2022-10-28T08:42:35.600792Z) is in the future compared with latest header on ibc-0 with height 0-4 and time Timestamp(2022-10-28T08:42:31.09872Z), adjusted with drift 3s

Block Delay not Reached

An IBC packet sent to the channel end on a destination chain with max_block_delay, channel using a connection with delay, must be received:

• after the time delay: delay has elapsed relative to the block time that included the client update with the packet commitment root, and
• after the block delay: delay / max_block_delay blocks on the destination chain have been created since the client update with the packet commitment root.

The on-chain packet handler uses genesis.app_state.ibc.connection_genesis.params.max_expected_time_per_block (in nanoseconds) when computing the block delay. Hermes uses its config max_block_delay to compute the block delay. If max_block_delay > genesis.app_state.ibc.connection_genesis.params.max_expected_time_per_block, then Hermes may send a packet that is received too early on the destination chain, resulting in an error like the following:

ERROR link error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 0: receive packet verification failed: couldn't verify counterparty packet commitment: failed packet commitment verification for client (07-tendermint-1): cannot verify packet until height: 0-54, current height: 0-46: packet-specified delay period has not been reached [cosmos/ibc-go/v3@v3.0.0/modules/light-clients/07-tendermint/types/client_state.go:536] With gas wanted: '0' and gas used: '78238' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "46"} }

Fix

To avoid errors for both cases described above:

• ensure that the max_block_delay parameter in the Hermes configuration file is equal to the max_expected_time_per_block parameter in the chain genesis file:

genesis.app_state.ibc.connection_genesis.params.max_expected_time_per_block

• ensure that the genesis max_expected_time_per_block parameter is 2 or 3 times the average block time which can be estimates from the tendermint.consensus.timeout* parameters. An example of a good estimation is:

       tendermint.consensus.timeout_propose +
       tendermint.consensus.timeout_prevote +
       tendermint.consensus.timeout_precommit +
       tendermint.consensus.timeout_commit

Insufficient Funds

If the wallet configured in Hermes' config.toml is empty or doesn't have enough funds, any transfer will result in the following error:

ERROR ThreadId(11) send_messages_and_wait_commit{chain=ibc-0 tracking_id=ft-transfer}:send_tx_with_account_sequence_retry{chain=ibc-0 account.sequence=25}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 0: 20stake is smaller than 108stake: insufficient funds [cosmos/cosmos-sdk@v0.46.3/x/bank/keeper/send.go:191] With gas wanted: '18446744073709551615' and gas used: '52497' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "377"} }

Fix

In order to fix the error above, use one of the following two solutions:

• add enough funds to the wallet configured by key_name in Hermes' config.toml.
• change the wallet configured by key_name in Hermes' config.toml to a wallet which has enough funds.

Uncleared Pending Packets

When Hermes starts, it retrieves the sequences for the unrelayed receive and acknowledgment packets from the application. Since only the packet commitments are stored in the application state, Hermes then queries tendermint for the IBC events with those sequence numbers and obtains the packet data from these events. The IBC events are retrieved from transaction and block indexes maintained by tendermint nodes.

In some cases these queries fail to obtain the packet data due to the fact that the state that contained those events has been pruned from the tendermint node. In this case Hermes will not be able to relay the packet. One example can be seen below. Hermes queries the application and finds 222 unreceived acknowledgment packets. Then it queries tendermint for the packet data but fails to find it (pulled packet data for 0 events). In this case the command returns zero relayed packets (SUCCESS []):

INFO ThreadId(01) relay_ack_packet_messages{src_chain=ibc-1 src_port=transfer src_channel=channel-0 dst_chain=ibc-0}: 222 unreceived acknowledgements found: 222
INFO ThreadId(01) relay_ack_packet_messages{src_chain=ibc-1 src_port=transfer src_channel=channel-0 dst_chain=ibc-0}: pulled packet data for 0 events; events.total=222 events.left=172
INFO ThreadId(01) relay_ack_packet_messages{src_chain=ibc-1 src_port=transfer src_channel=channel-0 dst_chain=ibc-0}: pulled packet data for 0 events; events.total=222 events.left=122
INFO ThreadId(01) relay_ack_packet_messages{src_chain=ibc-1 src_port=transfer src_channel=channel-0 dst_chain=ibc-0}: pulled packet data for 0 events; events.total=222 events.left=72
INFO ThreadId(01) relay_ack_packet_messages{src_chain=ibc-1 src_port=transfer src_channel=channel-0 dst_chain=ibc-0}: pulled packet data for 0 events; events.total=222 events.left=22
INFO ThreadId(01) relay_ack_packet_messages{src_chain=ibc-1 src_port=transfer src_channel=channel-0 dst_chain=ibc-0}: pulled packet data for 0 events; events.total=222 events.left=0
SUCCESS []

State Pruning Background

In order to keep the disk storage low, Tendermint nodes may be configured to prune "old" data.

Application State Pruning

The app.toml file defines a few parameters that control the application state pruning window:

pruning = <strategy>

# These are applied if and only if the pruning strategy is custom.
pruning-keep-recent = w
pruning-keep-every = x
pruning-interval = y

Tendermint State Pruning

The app.toml file also defines parameters to control the tendermint block pruning window:

min-retain-blocks = z

The evidence genesis parameters that also influence the actual size of the tendermint block pruning window are:

{
...
  "consensus_params": {
    "block": {
     ...
    },
    "evidence": {
      "max_age_num_blocks": b,
      "max_age_duration": d,
      ...
    },

These parameters are used to determine if a block (and associated state) should be pruned:

• a block with height h and time t is pruned if h < max(z, b) && t < now - d

Additional tendermint state may be maintained for state-sync purposes.

Debug

The unrelayed packet sequences can be retrieved using the following command:

hermes query packet pending --chain <CHAIN_ID> --channel <CHANNEL_ID> --port <PORT_ID>

There are two RPC endpoints, tx_search and block_search, that can be used to check if the full node still has the event information.

For example, if a send_packet with sequence 6 has not been relayed and the packet was sent in a transaction, the tx_search RPC endpoint can be used:

http://localhost:26657/tx_search?query="send_packet.packet_sequence='6' AND send_packet.packet_src_channel='channel-0' AND send_packet.packet_src_port='transfer' AND send_packet.packet_dst_channel='channel-0' AND send_packet.packet_dst_port='transfer'"

If the packet was sent via begin or end blocker, the block_search RPC endpoint can be used:

http://localhost:26657/block_search?query="send_packet.packet_sequence='6' AND send_packet.packet_src_channel='channel-0' AND send_packet.packet_src_port='transfer' AND send_packet.packet_dst_channel='channel-0' AND send_packet.packet_dst_port='transfer'"

If these two queries return an empty result, the event information has been pruned from the full node.

Fix

Depending on how old the missing events are:

• use an archive node, or
• adjust the pruning parameters of the full node to increase the pruning window and,
  • if state-sync is used, specify an initial state with a height that is smaller than the height at which the event occurred.
    • in order to find out the height of the missing events, use an archive node or a node with bigger pruning window and do the same queries as above
  • restart the node

Unimplemented gRPC for Cosmos Staking Service

When relaying for a consumer chain, the ccv_consumer_chain parameter must be set to true under its [[chains]] section in the config.toml file. If it is not, then when Hermes attempts to relay for a consumer chain, it will typically report an error related to a gRPC endpoint being unimplemented for the staking service:

ERROR error raised while creating client for chain: failed when building client state: gRPC call failed with status: status: Unimplemented, message: "unknown service cosmos.staking.v1beta1.Query", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "content-length": "0", "date": "Thu, 16 Feb 2023 10:42:14 GMT", "server": "Caddy"} }

This error occurs because consumer chains do not utilize the same staking module as sovereign chains. The ccv_consumer_chain parameter must be set to true so that Hermes knows to query a different gRPC endpoint for the relevant ccvconsumer parameters that it needs in order to relay on behalf of a consumer chain.

Fix

Set ccv_consumer_chain = true in config.toml.

Updating a client after a Genesis restart withtout IBC upgrade proposal

If a chain went through a genesis restart without an IBC upgrade proposal updating the client can result in an error due to blocks at lower heights not being available.

For example if we have two chains ibc-0 and ibc-1, a client 07-tendermint-0 hosted on chain ibc-1 referencing ibc-0 and an Archive Node with its RPC address http://127.0.0.1:28000. After ibc-0 goes through a genesis restart without an IBC upgrade proposal, trying to update client 07-tendermint-0 will result in an error:

ERROR failed during a client operation: error raised while updating client: failed
building header with error: Light client error for RPC address network1-1:
Internal error: height 26 is not available, lowest height is 1101 (code: -32603):

In this case, to update the client at a height higher than 1101 for example 1102, the following command can be used.

hermes update client --height 1102 --archive-address http://127.0.0.1:28000 --restart-height 1101 --host-chain ibc-1 --client 07-tendermint-0

Once this command is successful, updating the client should then work as usual.

Handling Clock Drift

IBC light client security model requires that the timestamp of a header included in client updates for some client is within [now - client.trusting_period, now + client.max_clock_drift).

The trusting_period is a parameter that is configurable under the [[chains]] section and its default value is two thirds of the chain unbonding_period. The rest of this section describes the configuration parameters that determine the max_clock_drift.

IBC light client security model requires that the clocks of the reference and host chains are roughly synchronized. Hermes uses the clock_drift and max_block_time configuration parameters to determine how much clock drift is tolerable between the reference and host chains.

• clock_drift is a correction parameter that specifies how far in the future or past a chain's clock may be from the current time..
• max_block_time is the maximum amount of time that can occur before a new block is created on the chain.

Note: For Cosmos SDK chains, a good approximation for max_block_time is C * (timeout_propose + timeout_commit), where C is a constant to allow for variation in block times, mainly due to tx execution time which is outside of consensus params. To allow for some variance in block times while still detecting forward lunatic attacks, it is recommended to set C to be in the 3..5 range. Hermes uses the default value of 30s which is a good approximation for most Cosmos SDK chains that have 6-10sec block times.

The clock_drift parameter values on both the reference and host chains, and max_block_time of the host chain are summed to get the max_clock_drift when creating a client on the host chain. This can be summarized more succinctly in the following equation:

client.max_clock_drift = reference.clock_drift + host.max_block_time + host.clock_drift

Thus, when configuring these values in Hermes' config.toml, keep in mind that this is how these parameters will be used. If the total clock drift is too small, then we run the risk of client updates being rejected because a new block won't have been created yet. It's better to err on the side of total clock drift being larger than smaller, however, if this value ends up being too large, then this becomes a security vulnerability.

For a more concrete example of what could happen when clock drift is mis-configured, take a look at the Mishandling Clock Drift troubleshooting section.

Commands

The Commands section presents the commands current available in Hermes

Sections

• Keys

  • Commands to manage keys (private keys) for each chain.

• Config

  • Commands to manage configuration file, in particular to validate it.

• Path Setup

  • Commands to manage clients, connections, channels.

• Relaying

  • Commands to start the relayer and relay packets.

• Listen Mode

  • Commands to listen for IBC events

• Upgrade

  • Commands to perform client upgrade

• Monitor

  • Commands to monitor clients and submit evidence of misbehaviour

• Queries

  • Commands to execute queries on configured chains

• Transactions

  • Commands to submit individual transactions to configured chains

Global options

Hermes accepts global options which affect all commands.

hermes v1.6.0
Informal Systems <hello@informal.systems>
Implementation of `hermes`, an IBC Relayer developed in Rust.

FLAGS:
        --config <CONFIG>    Path to configuration file
        --json               Enable JSON output

Ordering of command-line options

The global options must be specified right after the hermes command and before any sub-command. The non-global options have to be specified after the sub-command.

hermes [GLOBAL_OPTIONS] <SUBCOMMAND> [OPTIONS]

Example

To start Hermes using the configuration file at /home/my_chain.toml and enable JSON output:

hermes  --config $HOME/my_chain.toml --json start

To query all clients on a chain while enabling JSON output:

hermes  --json query clients --host-chain ibc-1

JSON output

If the --json option is supplied, all commands will output single-line JSON values instead of plain text.

Log messages will be written to stderr, while the final result will be written to stdout, and everything will be formatted as JSON. This allows processing only the final output using jq. To process all the output using jq, one can redirect stderr to stdout with hermes --json COMMAND 2>&1 | jq.

Example

hermes  --json create client --host-chain ibc-0 --reference-chain ibc-1

{"timestamp":"Apr 13 20:46:31.921","level":"INFO","fields":{"message":"Using default configuration from: '.hermes/config.toml'"},"target":"ibc_relayer_cli::commands"}
{"timestamp":"Apr 13 20:46:31.961","level":"INFO","fields":{"message":"running listener","chain.id":"ibc-1"},"target":"ibc_relayer::event::monitor"}
{"timestamp":"Apr 13 20:46:31.989","level":"INFO","fields":{"message":"running listener","chain.id":"ibc-0"},"target":"ibc_relayer::event::monitor"}
{"result":{"CreateClient":{"client_id":"07-tendermint-1","client_type":"Tendermint","consensus_height":{"revision_height":10060,"revision_number":1},"height":{"revision_height":10072,"revision_number":0}}},"status":"success"}

The first three lines are printed to stderr, while the last line with a "result" key is printed to stdout.

Example

To improve the readability, pipe all the output to jq:

hermes  --json create client --host-chain ibc-0 --reference-chain ibc-1 2>&1 | jq

{
  "timestamp": "Apr 13 20:52:26.060",
  "level": "INFO",
  "fields": {
    "message": "Using default configuration from: '.hermes/config.toml'"
  },
  "target": "ibc_relayer_cli::commands"
}
{
  "timestamp": "Apr 13 20:52:26.082",
  "level": "INFO",
  "fields": {
    "message": "running listener",
    "chain.id": "ibc-1"
  },
  "target": "ibc_relayer::event::monitor"
}
{
  "timestamp": "Apr 13 20:52:26.088",
  "level": "INFO",
  "fields": {
    "message": "running listener",
    "chain.id": "ibc-0"
  },
  "target": "ibc_relayer::event::monitor"
}
{
  "result": {
    "CreateClient": {
      "client_id": "07-tendermint-5",
      "client_type": "Tendermint",
      "consensus_height": {
        "revision_height": 10364,
        "revision_number": 1
      },
      "height": {
        "revision_height": 10375,
        "revision_number": 0
      }
    }
  },
  "status": "success"
}

Example

To extract the identifier of the newly created client above:

hermes  --json create client --host-chain ibc-0 --reference-chain ibc-1 | jq '.result.CreateClient.client_id'

Which should output:

"07-tendermint-2"

Adding Keys to Hermes

WARNING: Currently Hermes does NOT support a keyring store to securely store the private key file. The key file will be stored on the local file system in the folder set by the configuration key_store_folder which defaults to key_store_folder = '$HOME/.hermes/keys'.

BREAKING: As of Hermes v1.0.0, the sub-command keys restore has been removed. Please use the sub-command keys add in order to restore a key.

Using the keys command you can add and list keys.

Show usage

To see the available sub-commands for the keys command run:

hermes help keys

The available sub-commands are the following:

DESCRIPTION:
Manage keys in the relayer for each chain

USAGE:
    hermes keys <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    add        Adds key to a configured chain or restores a key to a configured chain using a
                   mnemonic
    balance    Query balance for a key from a configured chain. If no key is given, the key is
                   retrieved from the configuration file
    delete     Delete key(s) from a configured chain
    help       Print this message or the help of the given subcommand(s)
    list       List keys configured on a chain

Key Seed file (Private Key)

In order to execute the command below you need a private key file (JSON). Hermes uses the private key file to sign the transactions submitted to the chain.

The private key file can be obtained by using the keys add on a Cosmos chain. For example, the command for gaiad is:

# The `key_name` parameter is the name of the key that will be found in the json output
# For example, in the "Two Local Chains" tutorial, we use "testkey".
gaiad keys add <key_name> --output json

The command outputs a JSON similar to the one below.

{
  "name": "testkey",
  "type": "local",
  "address": "cosmos1tc3vcuxyyac0dmayf887t95tdg7qpyql48w7gj",
  "pubkey": "cosmospub1addwnpepqgg7ng4ycm60pdxfzdfh4hjvkwcr3da59mr8k883vsstx60ruv7kur4525u",
  "mnemonic": "[24 words mnemonic]"
}

You can save this to a file (e.g. key_seed.json) and use it to add to Hermes with hermes keys add --chain <CHAIN_ID> --key-file key_seed.json. See the Adding Keys section for more details.

Adding and restoring Keys

The command keys add has two exclusive flags, --key-file and --mnemonic-file which are respectively used to add and restore a key.
If a key with the same key_name already exists, the flag --overwrite must be passed in order to overwrite the existing key or else the command will abort.

DESCRIPTION:
Adds key to a configured chain or restores a key to a configured chain using a mnemonic

USAGE:
    hermes keys add [OPTIONS] --chain <CHAIN_ID> --key-file <KEY_FILE>

    hermes keys add [OPTIONS] --chain <CHAIN_ID> --mnemonic-file <MNEMONIC_FILE>

OPTIONS:
    -h, --help                   Print help information
        --hd-path <HD_PATH>      Derivation path for this key [default: m/44'/118'/0'/0/0]
        --key-name <KEY_NAME>    Name of the key (defaults to the `key_name` defined in the config)
        --overwrite              Overwrite the key if there is already one with the same key name

FLAGS:
        --chain <CHAIN_ID>                 Identifier of the chain
        --key-file <KEY_FILE>              Path to the key file
        --mnemonic-file <MNEMONIC_FILE>    Path to file containing mnemonic to restore the key from

Add a private key to a chain from a key file

hermes keys add --chain <CHAIN_ID> --key-file <PRIVATE_KEY_FILE>

The content of the file key should have the same format as the output of the gaiad keys add command:

{
  "name": "testkey",
  "type": "local",
  "address": "cosmos1tc3vcuxyyac0dmayf887t95tdg7qpyql48w7gj",
  "pubkey": "cosmospub1addwnpepqgg7ng4ycm60pdxfzdfh4hjvkwcr3da59mr8k883vsstx60ruv7kur4525u",
  "mnemonic": "[24 words mnemonic]"
}

If the command is successful a message similar to the one below will be displayed:

Success: Added key testkey (<ADDRESS>) on <CHAIN_ID> chain

Key name: By default, the key will be named after the key_name property specified in the configuration file. To use a different key name, specify the --key-name option when invoking keys add.

hermes keys add --key-name [KEY_NAME] --chain <CHAIN_ID> --key-file <PRIVATE_KEY_FILE>

Restore a private key to a chain from a mnemonic

hermes keys add --chain <CHAIN_ID> --mnemonic-file <MNEMONIC_FILE>

or using an explicit derivation path, for example an Ethereum coin type (used for Evmos, Injective, Umee, Cronos, and possibly other networks):

hermes keys add --hd-path "m/44'/60'/0'/0/0" --chain <CHAIN_ID> --mnemonic-file <MNEMONIC_FILE>

The mnemonic file needs to have the 24 mnemonic words on the same line, separated by a white space. So the content should have the following format:

word1 word2 word3 ... word24

If the command is successful a message similar to the one below will be displayed:

Success: Restore key testkey (<ADDRESS>) on <CHAIN_ID> chain

Key name: By default, the key will be named after the key_name property specified in the configuration file. To use a different key name, specify the --key-name option when invoking keys add.

hermes keys add --key-name <KEY_NAME> --chain <CHAIN_ID> --mnemonic-file <MNEMONIC_FILE>

Delete keys

In order to delete the private keys added to chains use the keys delete command

DESCRIPTION:
Delete key(s) from a configured chain

USAGE:
    hermes keys delete --chain <CHAIN_ID> --key-name <KEY_NAME>

    hermes keys delete --chain <CHAIN_ID> --all

OPTIONS:
    -h, --help    Print help information

FLAGS:
        --all                    Delete all keys
        --chain <CHAIN_ID>       Identifier of the chain
        --key-name <KEY_NAME>    Name of the key

Delete private keys that was previously added to a chain

To delete a single private key by name:

hermes keys delete --chain <CHAIN_ID> --key-name <KEY_NAME>

Alternatively, to delete all private keys added to a chain:

hermes --config config.toml keys delete --chain <CHAIN_ID> --all

List keys

In order to list the private keys added to chains use the keys list command

DESCRIPTION:
List keys configured on a chain

USAGE:
    hermes keys list --chain <CHAIN_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>    Identifier of the chain

Listing the private key that was added to a chain

To list the private key file that was added to a chain:

hermes keys list --chain <CHAIN_ID>

If the command is successful a message similar to the one below will be displayed:

Success:
- user2 (cosmos1attn9fxrcvjz483w3tu4cfz77ldmlyujly3q3k)
- testkey (cosmos1dw88vdekeeuta5u50p6n5lt5v5c6y2we0pu8nz)

JSON:

hermes  --json keys list --chain <CHAIN_ID> | jq

If the command is successful a message similar to the one below will be displayed:

{
  "result": {
    "testkey": {
      "account": "cosmos1dw88vdekeeuta5u50p6n5lt5v5c6y2we0pu8nz",
      "address": [ 107, 142, 118, 55, 54, 206, 120, 190, 211, 148, 120, 117, 58, 125, 116, 101, 49, 162, 41, 217 ],
      "coin_type": 118,
      "private_key": "(snip)",
      "public_key": "xpub6Gc7ZUt2q1BiQYjhUextPv5bZLwosHigZYqEquPD6FkAGmHDrLiBgE5Xnh8XGZp79rAXtZn1Dt3DNQHxxgCgVQqfRMfVsRiXn6mwULBnYq7"
    },
    "user2": {
      "account": "cosmos1attn9fxrcvjz483w3tu4cfz77ldmlyujly3q3k",
      "address": [ 234, 215, 50, 164, 195, 195, 36, 42, 158, 46, 138, 249, 92, 36, 94, 247, 219, 191, 147, 146 ],
      "coin_type": 118,
      "private_key": "(snip)",
      "public_key": "xpub6FmDbeGTWVjSvHrqHfrpnMTZxpPX1V7XFiq5nMuvgwX9jumt1yUuwNAUQo8Nn36unbFShg6iSjkfMBgeY49wik7rF91N2SHvarpX62ByWMf"
    }
  },
  "status": "success"
}

Query balance

In order to retrieve the balance of an account associated with a key use the keys balance command

DESCRIPTION:
Query balance for a key from a configured chain. If no key is given, the key is retrieved from the
configuration file

USAGE:
    hermes keys balance [OPTIONS] --chain <CHAIN_ID>

OPTIONS:
        --all                    (optional) query the balance for all denom. This flag overwrites
                                 the `--denom` flag (defaults to false)
        --denom <DENOM>          (optional) query the balance for the given denom (defaults to the
                                 `denom` defined in the config for the gas price)
    -h, --help                   Print help information
        --key-name <KEY_NAME>    (optional) name of the key (defaults to the `key_name` defined in
                                 the config)

REQUIRED:
        --chain <CHAIN_ID>    Identifier of the chain

If the command is successful a message with the following format will be displayed:

Success: balance for key `KEY_NAME`: 100000000000 stake

JSON:

hermes  --json keys balance --chain <CHAIN_ID>

If the command is successful a message with the following format will be displayed:

{
  "result": {
    "amount": "99989207",
    "denom": "stake"
  },
  "status": "success"
}

Generating and Validating Config Files

Show usage

To see the available sub-commands for the config command run:

hermes help config

The available sub-commands are the following:

DESCRIPTION:
Generate a new Hermes configuration file or validate an existing one

USAGE:
    hermes config <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    auto        Automatically generate a config.toml for the specified chain(s)
    help        Print this message or the help of the given subcommand(s)
    validate    Validate the relayer configuration

Automatically generate configuration files for specified chains

Use config auto to automatically generate a configuration file from the chain-registry.

WARNING: Currently, default_gas and max_gas parameters are set to default values; these should be manually reset. The gas_price parameter is set as the average gas price listed for the chain in the chain registry.

DESCRIPTION:
Automatically generate a config.toml for the specified chain(s)

USAGE:
    hermes config auto [OPTIONS] --output <PATH> --chains <CHAIN_NAME:OPTIONAL_KEY_NAME>

OPTIONS:
        --commit <COMMIT_HASH>    Commit hash from which the chain configs will be generated. If
                                  it's not set, the latest commit will be used.
    -h, --help                    Print help information

REQUIRED:
        --chains <CHAIN_NAME:OPTIONAL_KEY_NAME>...
            Names of the chains to include in the config. Every chain must be in the chain registry.

        --output <PATH>
            Path to the configuration file

Example

Use config auto to generate a configuration file that is able to relay between cosmoshub and osmosis. This command assumes the existence of a key file for cosmoshub-4 and osmosis-1 in $HOME/.hermes/keys.

hermes config auto --output ~/example_config.toml --chains cosmoshub osmosis

2022-08-16T17:27:26.966233Z  INFO ThreadId(01) using default configuration from '~/.hermes/config.toml'
2022-08-16T17:27:27.800213Z  INFO ThreadId(01) cosmoshub-4: uses key "key_cosmoshub"
2022-08-16T17:27:27.841167Z  INFO ThreadId(01) osmosis-1: uses key "key_osmosis"
2022-08-16T17:27:27.841890Z  WARN ThreadId(01) Gas parameters are set to default values.
SUCCESS Config file written successfully at '~/example_config.toml'

It is also possible to manually specify a key name for any chain.

hermes config auto --output ~/example_config.toml --chains cosmoshub:random_key osmosis

2022-08-16T17:29:56.902499Z  INFO ThreadId(01) using default configuration from '~/.hermes/config.toml'
2022-08-16T17:29:57.288874Z  INFO ThreadId(01) cosmoshub-4: uses key "random_key"
2022-08-16T17:29:57.289728Z  INFO ThreadId(01) osmosis-1: uses key "key_osmosis"
2022-08-16T17:29:57.290314Z  WARN ThreadId(01) Gas parameters are set to default values.
SUCCESS "Config file written successfully : ~/example_config.toml."

WARNING : Do not forget to modify the gas settings before relaying !

Validate an existing configuration file

Use config validate to perform a quick syntactic validation of your configuration file.

DESCRIPTION:
Validate the relayer configuration

USAGE:
    hermes config validate

OPTIONS:
    -h, --help    Print help information

Example

Validate the default config file, the path inferred automatically to be $HOME/.hermes/config.toml.

hermes config validate

Which should output something similar to:

Jul 12 16:31:07.017  INFO using default configuration from '$HOME/.hermes/config.toml'
SUCCESS: "validation passed successfully"

Validate a config file at an arbitrary location:

hermes  --config $CONFIGPATH config validate

This one should fail validation because we mistakenly added two separate sections for the same chain ibc-1:

error: hermes fatal error: config error: config file has duplicate entry for the chain 'ibc-1'

Path Setup

This section describes a number of commands that can be used to manage clients, connections, channels.

Create

Use the create commands to create new clients, connections, and channels.

DESCRIPTION:
Create objects (client, connection, or channel) on chains

USAGE:
    hermes create <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    channel       Create a new channel between two chains
    client        Create a new IBC client
    connection    Create a new connection between two chains
    help          Print this message or the help of the given subcommand(s)

Update

Use the update commands to update a client.

DESCRIPTION:
Update objects (clients) on chains

USAGE:
    hermes update <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    client    Update an IBC client
    help      Print this message or the help of the given subcommand(s)

Client

Table of Contents

• Create Client
• Update Client

Create Client

Use the create client command to create a new client on a destination chain, tracking the state of the source chain.

DESCRIPTION:
Create a new IBC client

USAGE:
    hermes create client [OPTIONS] --host-chain <HOST_CHAIN_ID> --reference-chain <REFERENCE_CHAIN_ID>

OPTIONS:
        --clock-drift <CLOCK_DRIFT>
            The maximum allowed clock drift for this client.
            
            The clock drift is a correction parameter. It helps deal with clocks that are only
            approximately synchronized between the source and destination chains of this client. The
            destination chain for this client uses the clock drift parameter when deciding to accept
            or reject a new header (originating from the source chain) for this client. If this
            option is not specified, a suitable clock drift value is derived from the chain
            configurations.

    -h, --help
            Print help information

        --trust-threshold <TRUST_THRESHOLD>
            Override the trust threshold specified in the configuration.
            
            The trust threshold defines what fraction of the total voting power of a known and
            trusted validator set is sufficient for a commit to be accepted going forward.

        --trusting-period <TRUSTING_PERIOD>
            Override the trusting period specified in the config.
            
            The trusting period specifies how long a validator set is trusted for (must be shorter
            than the chain's unbonding period).

REQUIRED:
        --host-chain <HOST_CHAIN_ID>
            Identifier of the chain that hosts the client

        --reference-chain <REFERENCE_CHAIN_ID>
            Identifier of the chain targeted by the client

Example

Create a new client on ibc-0 which tracks ibc-1:

hermes create client --host-chain ibc-0 --reference-chain ibc-1

    CreateClient(
        Attributes {
            height: Height {
                revision: 0,
                height: 286,
            },
            client_id: ClientId(
                "07-tendermint-0",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 1,
                height: 274,
            },
        },
    ),
)

A new client is created with identifier 07-tendermint-1

Update Client

Use the update client command to update an existing client with a new consensus state. Specific update and trusted heights can be specified.

DESCRIPTION:
Update an IBC client

USAGE:
    hermes update client [OPTIONS] --host-chain <HOST_CHAIN_ID> --client <CLIENT_ID>

OPTIONS:
        --archive-address <ARCHIVE_ADDRESS>
            The RPC address of the archive node to use to fetch headers from before the restart.
            Requires --restart-height if used. [aliases: archive-addr]

    -h, --help
            Print help information

        --height <REFERENCE_HEIGHT>
            The target height of the client update. Leave unspecified for latest height.

        --restart-height <RESTART_HEIGHT>
            The height that the chain underwent a genesis restart at. Requires --archive-address if
            used.

        --trusted-height <REFERENCE_TRUSTED_HEIGHT>
            The trusted height of the client update. Leave unspecified for latest height.

REQUIRED:
        --client <CLIENT_ID>            Identifier of the client to update
        --host-chain <HOST_CHAIN_ID>    Identifier of the chain that hosts the client

Update client with latest header

the client on ibc-0 with the latest header of ibc-1:

hermes update client --host-chain ibc-0 --client 07-tendermint-9

Success: UpdateClient(
    UpdateClient {
        common: Attributes {
            height: Height { revision: 0, height: 303 },
            client_id: ClientId(
                "07-tendermint-1",
            ),
            client_type: Tendermint,
            consensus_height: Height { revision: 1, height: 293 },
        },
        header: Some(
            Tendermint(
                 Header {...},
            ),
        ),
    },
)

The client with identifier 07-tendermint-1 has been updated with the consensus state at height 1-293.

Update a client to a specific target height

hermes update client --height 320 --trusted-height 293 --host-chain ibc-0 --client 07-tendermint-1

Success: UpdateClient(
    UpdateClient {
        common: Attributes {
            height: Height { revision: 0, height: 555 },
            client_id: ClientId(
                "07-tendermint-1",
            ),
            client_type: Tendermint,
            consensus_height: Height { revision: 1, height: 320 },
        },
        header: Some(
            Tendermint(
                 Header {...},
            ),
        ),
    },
)

The client with identifier 07-tendermint-1 has been updated with the consensus state at height 1-320, as specified.

Connection

Table of Contents

• Establish Connection
• Examples
  • New connection over new clients
  • New connection over existing clients
• Non-zero Delay Connection

Establish Connection

Use the create connection command to create a new connection.

DESCRIPTION:
Create a new connection between two chains

USAGE:
    hermes create connection [OPTIONS] --a-chain <A_CHAIN_ID> --b-chain <B_CHAIN_ID>

    hermes create connection [OPTIONS] --a-chain <A_CHAIN_ID> --a-client <A_CLIENT_ID> --b-client <B_CLIENT_ID>

OPTIONS:
        --delay <DELAY>    Delay period parameter for the new connection (seconds) [default: 0]
    -h, --help             Print help information

FLAGS:
        --a-chain <A_CHAIN_ID>      Identifier of the side `a` chain for the new connection
        --a-client <A_CLIENT_ID>    Identifier of client hosted on chain `a`; default: None (creates
                                    a new client)
        --b-chain <B_CHAIN_ID>      Identifier of the side `b` chain for the new connection
        --b-client <B_CLIENT_ID>    Identifier of client hosted on chain `b`; default: None (creates
                                    a new client)

Examples

New connection over new clients

Create a new connection between ibc-0 and ibc-1 over new clients:

hermes create connection --a-chain ibc-0 --b-chain ibc-1

🥂  ibc-0 => OpenInitConnection(
    OpenInit(
        Attributes {
            height: Height { revision: 0, height: 4073 },
            connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-8",
            ),
            counterparty_connection_id: None,
            counterparty_client_id: ClientId(
                "07-tendermint-8",
            ),
        },
    ),
)

🥂  ibc-1 => OpenTryConnection(
    OpenTry(
        Attributes {
            height: Height { revision: 1, height: 4069 },
            connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-8",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-8",
            ),
        },
    ),
)

🥂  ibc-0 => OpenAckConnection(
    OpenAck(
        Attributes {
            height: Height { revision: 0, height: 4081 },
            connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-8",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-8",
            ),
        },
    ),
)

🥂  ibc-1 => OpenConfirmConnection(
    OpenConfirm(
        Attributes {
            height: Height { revision: 1, height: 4073 },
            connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-8",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-8",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-8",
            ),
        },
    ),
)

🥂🥂🥂  Connection handshake finished for [Connection {
    delay_period: 0s,
    a_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-8",
        ),
        connection_id: ConnectionId(
            "connection-8",
        ),
    },
    b_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-8",
        ),
        connection_id: ConnectionId(
            "connection-8",
        ),
    },
}]

Success: Connection {
    delay_period: 0s,
    a_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-8",
        ),
        connection_id: ConnectionId(
            "connection-8",
        ),
    },
    b_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-8",
        ),
        connection_id: ConnectionId(
            "connection-8",
        ),
    },
}

New connection over existing clients

Create a new connection between ibc-0 and ibc-1 over existing clients, both with client id 07-tendermint-0:

hermes create connection --a-chain ibc-0 --a-client 07-tendermint-0 --b-client 07-tendermint-0

Notice that one can omit the destination chain parameter, as Hermes will automatically figure it out by looking up the given client on ibc-0.

Non-zero Delay Connection

A connection can be created with a delay period parameter. This parameter specifies a period of time that must elpase after a successful client state update and before a packet with proofs using its commitment root can pe processed on chain. For more information see how packet delay works and the connection delay specification.

Channel

Table of Contents

• Establish Channel
• Examples
  • New channel over an existing connection
  • New channel over a new connection

Establish Channel

Use the create channel command to establish a new channel.

DESCRIPTION:
Create a new channel between two chains.

Can create a new channel using a pre-existing connection or alternatively, create a new client and a
new connection underlying the new channel if a pre-existing connection is not provided.

USAGE:
    hermes create channel [OPTIONS] --a-chain <A_CHAIN_ID> --a-connection <A_CONNECTION_ID> --a-port <A_PORT_ID> --b-port <B_PORT_ID>

    hermes create channel [OPTIONS] --a-chain <A_CHAIN_ID> --b-chain <B_CHAIN_ID> --a-port <A_PORT_ID> --b-port <B_PORT_ID> --new-client-connection

OPTIONS:
        --channel-version <VERSION>
            The version for the new channel
            
            [aliases: chan-version]

    -h, --help
            Print help information

        --new-client-connection
            Indicates that a new client and connection will be created underlying the new channel
            
            [aliases: new-client-conn]

        --order <ORDER>
            The channel ordering, valid options 'unordered' (default) and 'ordered'
            
            [default: ORDER_UNORDERED]

        --yes
            Skip new_client_connection confirmation

FLAGS:
        --a-chain <A_CHAIN_ID>
            Identifier of the side `a` chain for the new channel

        --a-connection <A_CONNECTION_ID>
            Identifier of the connection on chain `a` to use in creating the new channel
            
            [aliases: a-conn]

        --a-port <A_PORT_ID>
            Identifier of the side `a` port for the new channel

        --b-chain <B_CHAIN_ID>
            Identifier of the side `b` chain for the new channel

        --b-port <B_PORT_ID>
            Identifier of the side `b` port for the new channel

Examples

New channel over an existing connection

This is the preferred way to create a new channel, by leveraging an existing connection.

Create a new unordered channel between ibc-0 and ibc-1 over an existing connection, specifically the one we just created in the example above, with port name transfer on both sides:

hermes create channel --order unordered --a-chain ibc-0 --a-connection connection-0 --a-port  transfer --b-port transfer

Notice that one can omit the destination chain parameter, as Hermes will automatically figure it out by looking up the given connection on ibc-0.

🥳  ibc-0 => OpenInitChannel(
    OpenInit(
        Attributes {
            height: Height { revision: 0, height: 129 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-1")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: None
        }
    )
)
🥳  ibc-1 => OpenTryChannel(
    OpenTry(
        Attributes {
            height: Height { revision: 1, height: 126 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-1")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-1"))
        }
    )
)
🥳  ibc-0 => OpenAckChannel(
    OpenAck(
        Attributes {
            height: Height { revision: 0, height: 137 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-1")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-1"))
        }
    )
)
🥳  ibc-1 => OpenConfirmChannel(
    OpenConfirm(
        Attributes {
            height: Height { revision: 1, height: 129 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-1")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-1"))
        }
    )
)
🥳  🥳  🥳  Channel handshake finished for Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-1",
        ),
    },
    b_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-1",
        ),
    },
    connection_delay: 0s,
}
Success: Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-1",
        ),
    },
    b_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-1",
        ),
    },
    connection_delay: 0s,
}

New channel over a new connection

Should you specifically want to create a new client and a new connection as part of the create channel flow, that option exists, though this is the less-preferred option over the previous flow, as creating new clients and connections should only be done in certain specific circumstances so as not to create redundant resources.

Create a new unordered channel between ibc-0 and ibc-1 over a new connection, using port name transfer on both sides and accepting the interactive prompt that pops up notifying you that a new client and a new connection will be initialized as part of the process:

hermes create channel --order unordered --a-chain ibc-0 --b-chain ibc-1 --a-port  transfer --b-port transfer --new-client-connection

🥂  ibc-0 => OpenInitConnection(
    OpenInit(
        Attributes {
            height: Height { revision: 0, height: 66 },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: None,
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

🥂  ibc-1 => OpenTryConnection(
    OpenTry(
        Attributes {
            height: Height { revision: 1, height: 64 },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

🥂  ibc-0 => OpenAckConnection(
    OpenAck(
        Attributes {
            height: Height { revision: 0, height: 76 },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

🥂  ibc-1 => OpenConfirmConnection(
    OpenConfirm(
        Attributes {
            height: Height { revision: 1, height: 68 },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

🥂🥂🥂  Connection handshake finished for [Connection {
    delay_period: 0s,
    a_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
    },
    b_side: ConnectionSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
    },
}]

🥳  ibc-0 => OpenInitChannel(
    OpenInit(
        Attributes {
            height: Height { revision: 0, height: 78 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-0")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: None
        }
    )
)

🥳  ibc-1 => OpenTryChannel(
    OpenTry(
        Attributes {
            height: Height { revision: 1, height: 70 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-0")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-0"))
        }
    )
)

🥳  ibc-0 => OpenAckChannel(
    OpenAck(
        Attributes {
            height: Height { revision: 0, height: 81 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-0")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-0"))
        }
    )
)

🥳  ibc-1 => OpenConfirmChannel
    OpenConfirm
        Attributes {
            height: Height { revision: 1, height: 73 },
            port_id: PortId("transfer"),
            channel_id: Some(ChannelId("channel-0")),
            connection_id: ConnectionId("connection-0"),
            counterparty_port_id: PortId("transfer"),
            counterparty_channel_id: Some(ChannelId("channel-0"))
        }
    )
)

🥳  🥳  🥳  Channel handshake finished for Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-0",
        ),
    },
    b_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-0",
        ),
    },
    connection_delay: 0s,
}

Success: Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-0",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-0",
        ),
    },
    b_side: ChannelSide {
        chain: ProdChainHandle {
            chain_id: ChainId {
                id: "ibc-1",
                version: 1,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: ChannelId(
            "channel-0",
        ),
    },
    connection_delay: 0s,
}

A new channel with identifier channel-0 on both sides has been established on a new connection with identifier connection-0 on both sides.

Relaying

This section describes the types of relaying that hermes can perform.

Hermes can send transactions triggered by IBC events. It currently handles channel handshake and packet events:

• packets messages only
• channel and packet messages

The start Command

The start command can be used to start Hermes in IBC event listen mode.

DESCRIPTION:
Start the relayer in multi-chain mode.

Relays packets and open handshake messages between all chains in the config.

USAGE:
    hermes start [OPTIONS]

OPTIONS:
        --full-scan
            Force a full scan of the chains for clients, connections and channels

    -h, --help
            Print help information

As described in next subsections, the type of relaying can be configured in the global section of the configuration file, by specifying different values in strategy field.

Packet Relaying

This section describes the configuration and commands that can be used to start Hermes and relay packets over one or multiple paths.

Table of Contents

• The start Command
• Packet Streaming
• Packet Delay

The start Command

To relay packets only configure the mode section of the configuration file like so:

[global]
log_level = 'info'

[mode]

[mode.clients]
enabled = true
# ...

[mode.connections]
enabled = false

[mode.channels]
enabled = false

[mode.packets]
enabled = true
# ...

Then start Hermes using the start command:

hermes start

Hermes sends packet transactions triggered by IBC packet events for all open channels between the configured chains. This is also referred to packet streaming.

Packet Streaming

After Hermes is started using the start command, it listens to IBC packet events emitted by any of the configured chains. Assuming the events are coming from a source chain, Hermes builds packets based on these events, packets that are then sent either to the source chain or the counterparty (destination) chain.

Current events and actions are:

• send_packet: Hermes builds a packet message with the packet obtained from the event and any required proofs obtained from the counterparty of the chain where the message is sent. The concrete packet is:
  • MsgRecvPacket, sent to destination chain if the channel is in open state on the destination chain, and a timeout has not occurred,
  • MsgTimeout, sent to the source chain if the channel is in open state on the destination chain, but a timeout has occurred.
  • MsgTimeoutOnClose, sent to the source chain if the channel is in closed state on the destination chain.
• write_acknowledgement: Hermes builds a MsgAcknowledgement packet that is sent to the destination chain.

In addition to these events, Hermes will also handle channel closing events:

• chan_close_init: Hermes builds a MsgChannelCloseConfirm and sends it to the destination chain

Packet Delay

If the relay path is using a non-zero delay connection, then hermes will delay all packet transactions. The delay is relative to the submission time for the client update at the height required by the packet proof. The delay is used to prevent light client attacks and ensures that misbehavior detection finalizes before the transaction is submitted. For more information on the misbehavior detector see the misbehaviour section.

Relaying of Handshake Messages

This section describes the configuration and commands that can be used to start Hermes and relay both handshake and packets for connections and channels.

The start Command

To relay packets and handshake messages configure the mode section of the configuration file like so:

[global]
log_level = 'info'

[mode]

[mode.clients]
enabled = true
# ...

[mode.connections]
enabled = true

[mode.channels]
enabled = true

[mode.packets]
enabled = true
# ...

Then start Hermes using the start command:

hermes start

Hermes sends handshake and packet transactions triggered by IBC events.

Completing Channel Handshakes

After Hermes is started using the start command, it scans the chain state and will resume the handshake for any channels or connections that are not in open state. It then listens to IBC events emitted by any of the configured chains.

Assuming the events are coming from a source chain, Hermes determines the destination chain and builds the handshake messages based on these events. These are then sent to the destination chain.

In addition to the events described in Packet Relaying, the following IBC events may be handled:

• Channels (if mode.channels.enabled=true):

  • chan_open_init: Hermes builds a MsgChannelOpenTry message
  • chan_open_try: Hermes builds a MsgChannelOpenAck message
  • chan_open_ack: Hermes builds a MsgChannelOpenConfirm message
  • chan_open_confirm: no message is sent out, channel opening is finished

• Connections (if mode.connections.enabled=true):

  • conn_open_init: Hermes builds a MsgConnOpenTry message
  • conn_open_try: Hermes builds a MsgConnOpenAck message
  • conn_open_ack: Hermes builds a MsgConnOpenConfirm message
  • conn_open_confirm: no message is sent out, connection opening is finished

Clearing Packets

clear packets

This command clears outstanding packets on a given channel in both directions, by issuing the appropriate packet-recvs and packet-acks.

Usage

DESCRIPTION:
Clear outstanding packets (i.e., packet-recv and packet-ack) on a given channel in both directions.
The channel is identified by the chain, port, and channel IDs at one of its ends

USAGE:
    hermes clear packets [OPTIONS] --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

OPTIONS:
        --counterparty-key-name <COUNTERPARTY_KEY_NAME>
            use the given signing key for the counterparty chain (default: `counterparty_key_name`
            config)

    -h, --help
            Print help information

        --key-name <KEY_NAME>
            use the given signing key for the specified chain (default: `key_name` config)

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain
        --channel <CHANNEL_ID>    Identifier of the channel
        --port <PORT_ID>          Identifier of the port

Example

1. Without Hermes running, send 3 packets over a channel, here channel-13:

hermes tx ft-transfer --timeout-height-offset 1000 --number-msgs 3 --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-13 --amount 9999

Which should output something similar to:

2022-02-24T14:16:28.295526Z  INFO ThreadId(01) using default configuration from '$HOME/.hermes/config.toml'
2022-02-24T14:16:28.330860Z  INFO ThreadId(15) send_tx{id=ibc0}: refresh: retrieved account sequence=61 number=1
2022-02-24T14:16:28.350022Z  INFO ThreadId(15) wait_for_block_commits: waiting for commit of tx hashes(s) AE4C3186778488E45670EB7303FA77E69B39F4E7C7494B05EC51E55136A373D6 id=ibc0
Success: [
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 86208,
            },
            packet: Packet {
                sequence: Sequence(
                    14,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 86208,
            },
            packet: Packet {
                sequence: Sequence(
                    15,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 86208,
            },
            packet: Packet {
                sequence: Sequence(
                    16,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
]

2. Because Hermes is not running these packets won't be relayed, as can be seen with the query packet pending-sends command:

hermes query packet pending-sends --chain ibc-1 --port transfer --channel channel-13

Which should output something similar to:

Success: [
    14,
    15,
    16,
]

3. We can clear them manually using the clear packets command:

hermes clear packets --chain ibc-0 --port transfer --channel channel-13

Which should output something similar to:

2022-02-24T14:17:25.748422Z  INFO ThreadId(01) using default configuration from '$HOME/.hermes/config.toml'
2022-02-24T14:17:25.799704Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}: found unprocessed SendPacket events for [Sequence(14), Sequence(15), Sequence(16)] (first 10 shown here; total=3)
2022-02-24T14:17:25.827177Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}: ready to fetch a scheduled op. data with batch of size 3 targeting Destination
2022-02-24T14:17:26.504798Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}:relay{odata=E96CV_cA5P ->Destination @0-86218; len=3}: assembled batch of 4 message(s)
2022-02-24T14:17:26.508873Z  INFO ThreadId(29) send_tx{id=ibc1}: refresh: retrieved account sequence=54 number=1
2022-02-24T14:17:26.561715Z  INFO ThreadId(29) wait_for_block_commits: waiting for commit of tx hashes(s) 07AA83524257105CC476063932A560893BE8F4E94C679BFD00F970FC248647E0 id=ibc1
2022-02-24T14:17:31.948950Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}:relay{odata=E96CV_cA5P ->Destination @0-86218; len=3}: [Sync->ibc1] result events:
    UpdateClientEv(h: 0-86215, cs_h: 07-tendermint-3(0-86219))
    WriteAcknowledgementEv(WriteAcknowledgement - h:0-86215, seq:14, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))
    WriteAcknowledgementEv(WriteAcknowledgement - h:0-86215, seq:15, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))
    WriteAcknowledgementEv(WriteAcknowledgement - h:0-86215, seq:16, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))


2022-02-24T14:17:31.949192Z  INFO ThreadId(01) PacketRecvCmd{src_chain=ibc0 src_port=transfer src_channel=channel-13 dst_chain=ibc1}:relay{odata=E96CV_cA5P ->Destination @0-86218; len=3}: success
2022-02-24T14:17:31.989215Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}: found unprocessed WriteAcknowledgement events for [Sequence(14), Sequence(15), Sequence(16)] (first 10 shown here; total=3)
2022-02-24T14:17:32.013500Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}: ready to fetch a scheduled op. data with batch of size 3 targeting Destination
2022-02-24T14:17:33.211930Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}:relay{odata=L4fnSXkxL_ ->Destination @0-86215; len=3}: assembled batch of 4 message(s)
2022-02-24T14:17:33.215803Z  INFO ThreadId(15) send_tx{id=ibc0}: refresh: retrieved account sequence=62 number=1
2022-02-24T14:17:33.245229Z  INFO ThreadId(15) wait_for_block_commits: waiting for commit of tx hashes(s) 62C69B1C46AF45182D5D99C6CB5EB301F8A402726772BA4EE067B18C68F2A4D6 id=ibc0
2022-02-24T14:17:37.465489Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}:relay{odata=L4fnSXkxL_ ->Destination @0-86215; len=3}: [Sync->ibc0] result events:
    UpdateClientEv(h: 0-86221, cs_h: 07-tendermint-3(0-86216))
    AcknowledgePacketEv(h:0-86221, seq:14, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))
    AcknowledgePacketEv(h:0-86221, seq:15, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))
    AcknowledgePacketEv(h:0-86221, seq:16, path:channel-13/transfer->channel-12/transfer, toh:0-87203, tos:Timestamp(NoTimestamp)))


2022-02-24T14:17:37.465802Z  INFO ThreadId(01) PacketAckCmd{src_chain=ibc1 src_port=transfer src_channel=channel-12 dst_chain=ibc0}:relay{odata=L4fnSXkxL_ ->Destination @0-86215; len=3}: success
Success: [
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 0,
                    height: 86215,
                },
                client_id: ClientId(
                    "07-tendermint-3",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 0,
                    height: 86219,
                },
            },
            header: Some(
                Tendermint(
                     Header {...},
                ),
            ),
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 0,
                height: 86215,
            },
            packet: Packet {
                sequence: Sequence(
                    14,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
            ack: [ ... ],
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 0,
                height: 86215,
            },
            packet: Packet {
                sequence: Sequence(
                    15,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
            ack: [ ... ],
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 0,
                height: 86215,
            },
            packet: Packet {
                sequence: Sequence(
                    16,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [ ... ],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
            ack: [ ... ],
        },
    ),
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 0,
                    height: 86221,
                },
                client_id: ClientId(
                    "07-tendermint-3",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 0,
                    height: 86216,
                },
            },
            header: Some(
                Tendermint(
                     Header {...},
                ),
            ),
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 86221,
            },
            packet: Packet {
                sequence: Sequence(
                    14,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 86221,
            },
            packet: Packet {
                sequence: Sequence(
                    15,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 86221,
            },
            packet: Packet {
                sequence: Sequence(
                    16,
                ),
                source_port: PortId(
                    "transfer",
                ),
                source_channel: ChannelId(
                    "channel-13",
                ),
                destination_port: PortId(
                    "transfer",
                ),
                destination_channel: ChannelId(
                    "channel-12",
                ),
                data: [],
                timeout_height: Height {
                    revision: 0,
                    height: 87203,
                },
                timeout_timestamp: Timestamp {
                    time: None,
                },
            },
        },
    ),
]

4. The packets have now been successfully relayed:

hermes query packet pending-sends --chain ibc-1 --port transfer --channel channel-13

Which should output something similar to:

2022-02-24T14:21:28.874190Z  INFO ThreadId(01) using default configuration from '$HOME/.hermes/config.toml'
Success: []

Listen Mode

Hermes can be started in listen mode to display the events emitted by a given chain. NewBlock and Tx IBC events are shown.

DESCRIPTION:
Listen to and display IBC events emitted by a chain

USAGE:
    hermes listen [OPTIONS] --chain <CHAIN_ID>

OPTIONS:
        --events <EVENT>...    Add an event type to listen for, can be repeated. Listen for all
                               events by default (available: Tx, NewBlock)
    -h, --help                 Print help information

REQUIRED:
        --chain <CHAIN_ID>    Identifier of the chain to listen for events from

Example

Start Hermes in listen mode for all ibc-0 events and observe the output:

hermes listen --chain ibc-0

EventBatch {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    height: block::Height(10914),
    events: [
        NewBlock(
            NewBlock {
                height: block::Height(10914),
            },
        ),
    ],
}
EventBatch {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    height: block::Height(10915),
    events: [
        OpenInitConnection(
            OpenInit(
                Attributes {
                    height: block::Height(10915),
                    connection_id: Some(
                        ConnectionId(
                            "connection-3",
                        ),
                    ),
                    client_id: ClientId(
                        "07-tendermint-3",
                    ),
                    counterparty_connection_id: None,
                    counterparty_client_id: ClientId(
                        "07-tendermint-5",
                    ),
                },
            ),
        ),
    ],

...

EventBatch {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    height: block::Height(10919),
    events: [
        UpdateClient(
            UpdateClient(
                Attributes {
                    height: block::Height(10919),
                    client_id: ClientId(
                        "07-tendermint-3",
                    ),
                    client_type: Tendermint,
                    consensus_height: Height {
                        revision: 1,
                        height: 10907,
                    },
                },
            ),
        ),
    ],
}

...

EventBatch {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    height: block::Height(10924),
    events: [
        UpdateClient(
            UpdateClient(
                Attributes {
                    height: block::Height(10924),
                    client_id: ClientId(
                        "07-tendermint-3",
                    ),
                    client_type: Tendermint,
                    consensus_height: Height {
                        revision: 1,
                        height: 10912,
                    },
                },
            ),
        ),
        OpenAckConnection(
            OpenAck(
                Attributes {
                    height: block::Height(10924),
                    connection_id: Some(
                        ConnectionId(
                            "connection-3",
                        ),
                    ),
                    client_id: ClientId(
                        "07-tendermint-3",
                    ),
                    counterparty_connection_id: Some(
                        ConnectionId(
                            "connection-5",
                        ),
                    ),
                    counterparty_client_id: ClientId(
                        "07-tendermint-5",
                    ),
                },
            ),
        ),
    ],
}

Filter events

The listen command accepts a --events flag to specify which event types to listen for.

At the moment, two event types are available:

• NewBlock
• Tx

The --events flag can be repeated to specify more than one event type.

• To listen for only NewBlock events on ibc-0, invoke hermes listen --events NewBlock --chain ibc-0
• To listen for only Tx events on ibc-0, invoke hermes listen --events Tx --chain ibc-0
• To listen for both NewBlock and Tx events on ibc-0, invoke hermes listen --events NewBlock Tx --chain ibc-0

If the --events flag is omitted, Hermes will subscribe to all event types.

Upgrading Clients

If IBC clients need to be upgraded after their reference chains went through an upgrade, the following CLIs may be used.

Upgrade Client Command

Use the upgrade client command to upgrade a specific IBC client after a chain upgrade.

DESCRIPTION:
Upgrade an IBC client

USAGE:
    hermes upgrade client --host-chain <HOST_CHAIN_ID> --client <CLIENT_ID> --upgrade-height <REFERENCE_UPGRADE_HEIGHT>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --client <CLIENT_ID>
            Identifier of the client to be upgraded

        --host-chain <HOST_CHAIN_ID>
            Identifier of the chain that hosts the client

        --upgrade-height <REFERENCE_UPGRADE_HEIGHT>
            The height at which the reference chain halts for the client upgrade

Upgrade Clients Command

Use the upgrade clients command to upgrade all IBC clients that target a specific (upgraded) chain.

DESCRIPTION:
Upgrade all IBC clients that target a specific chain

USAGE:
    hermes upgrade clients [OPTIONS] --reference-chain <REFERENCE_CHAIN_ID> --upgrade-height <REFERENCE_UPGRADE_HEIGHT>

OPTIONS:
    -h, --help                          Print help information
        --host-chain <HOST_CHAIN_ID>    Identifier of the chain hosting the clients to be upgraded

REQUIRED:
        --reference-chain <REFERENCE_CHAIN_ID>
            Identifier of the chain that underwent an upgrade; all clients targeting this chain will
            be upgraded

        --upgrade-height <REFERENCE_UPGRADE_HEIGHT>
            The height at which the reference chain halts for the client upgrade

Example

Here is an example of a chain upgrade proposal submission and client upgrade.

Testing Client Upgrade

Prerequisites

• Gaiad (v7.0.*), for example:

gaiad version --log_level error --long | head -n4

Testing procedure

Setup using Gaia manager

Note: The gm.toml file that we're using here looks like this:

[global]
  add_to_hermes = true
  auto_maintain_config = true
  extra_wallets = 2
  gaiad_binary = "$GOPATH/bin/gaiad"
  hdpath = ""
  home_dir = "$HOME/.gm"
  ports_start_at = 27040
  validator_mnemonic = ""
  wallet_mnemonic = ""

  [global.hermes]
    binary = "<path/to/hermes>"
    config = "$HOME/.hermes/config.toml"
    log_level = "info"
    telemetry_enabled = true
    telemetry_host = "127.0.0.1"
    telemetry_port = 3001

[ibc-0]
  ports_start_at = 27000

[ibc-1]
  ports_start_at = 27010

• Run the command gm start
• Go to the file $HOME/.gm/ibc-0/config/genesis.json and change max_deposit_period and voting_period to a lower value, such as 120s
• Run the commands: gm reset, gm hermes config and gm hermes keys

Test upgrading chain and client

1. Create one client on ibc-1 for ibc-0:

   hermes create client --host-chain ibc-1 --reference-chain ibc-0
   

   Success: CreateClient(
      CreateClient(
          Attributes {
              height: Height { revision: 1, height: 9 },
              client_id: ClientId(
                  "07-tendermint-0",
              ),
              client_type: Tendermint,
              consensus_height: Height { revision: 0, height: 18 },
          },
      ),
   )
   

2. Create and submit an upgrade plan for chain ibc-0:

   Use test command to make an upgrade proposal. In the example below a software upgrade proposal is made for ibc-0, for the height 300 blocks from the latest height. 10000000stake is deposited. The proposal includes the upgraded client state constructed from the state of 07-tendermint-0 client on ibc-1 that was created in the previous step.

   hermes tx upgrade-chain --reference-chain ibc-0 --host-chain ibc-1 --host-client 07-tendermint-0 --amount 10000000 --height-offset 60
   

   Success: transaction::Hash(CE98D8D98091BA8016BD852D18056E54C4CB3C4525E7F40DD3C40B4FD0F2482B)
   

3. Verify that the proposal was accepted by querying the upgrade plan to check that it was submitted correctly.

   Note: You can find the RPC port used to query the local node by running gm ports in order to see a list of the ports being used.

   gaiad --node tcp://localhost:27000 query gov proposal 1 --home $HOME/.gm/ibc-0/
   

   If successful, you should see output like this. Note that the status of the proposal near the bottom of the output should be PROPOSAL_STATUS_VOTING_PERIOD indicating that the voting period is still ongoing.

   content:
     '@type': /ibc.core.client.v1.UpgradeProposal
     description: upgrade the chain software and unbonding period
     plan:
       height: "65"
       info: ""
       name: plan
       time: "0001-01-01T00:00:00Z"
       upgraded_client_state: null
     title: proposal 0
     upgraded_client_state:
       '@type': /ibc.lightclients.tendermint.v1.ClientState
       allow_update_after_expiry: false
       allow_update_after_misbehaviour: false
       chain_id: ibc-0
       frozen_height:
         revision_height: "0"
         revision_number: "0"
       latest_height:
         revision_height: "66"
         revision_number: "0"
       max_clock_drift: 0s
       proof_specs:
       - inner_spec:
           child_order:
           - 0
           - 1
           child_size: 33
           empty_child: null
           hash: SHA256
           max_prefix_length: 12
           min_prefix_length: 4
         leaf_spec:
           hash: SHA256
           length: VAR_PROTO
           prefix: AA==
           prehash_key: NO_HASH
           prehash_value: SHA256
         max_depth: 0
         min_depth: 0
       - inner_spec:
           child_order:
           - 0
           - 1
           child_size: 32
           empty_child: null
           hash: SHA256
           max_prefix_length: 1
           min_prefix_length: 1
         leaf_spec:
           hash: SHA256
           length: VAR_PROTO
           prefix: AA==
           prehash_key: NO_HASH
           prehash_value: SHA256
         max_depth: 0
         min_depth: 0
       trust_level:
         denominator: "0"
         numerator: "0"
       trusting_period: 0s
       unbonding_period: 1814400s
       upgrade_path:
       - upgrade
       - upgradedIBCState
   deposit_end_time: "2022-07-06T15:14:38.993051Z"
   final_tally_result:
     abstain: "0"
     "no": "0"
     no_with_veto: "0"
     "yes": "0"
   proposal_id: "1"
   status: PROPOSAL_STATUS_VOTING_PERIOD
   submit_time: "2022-07-06T15:12:38.993051Z"
   total_deposit:
   - amount: "10000000"
     denom: stake
   voting_end_time: "2022-07-06T15:14:38.993051Z"
   voting_start_time: "2022-07-06T15:12:38.993051Z"
   

4. Vote on the proposal

   The parameter 1 should match the proposal_id: from the upgrade proposal submitted at step 3. This command must be issued while the proposal status is PROPOSAL_STATUS_VOTING_PERIOD. Confirm transaction when prompted.

   gaiad --node tcp://localhost:27000 tx gov vote 1 yes --home $HOME/.gm/ibc-0/data/ --keyring-backend test --keyring-dir $HOME/.gm/ibc-0/ --chain-id ibc-0 --from validator
   

   confirm transaction before signing and broadcasting [y/N]: y
   
   txhash: 50CC1C39FBB14F99580A916ADE7F02883FFCC35D7862153F16BE86138151E17C
   

5. Test the upgrade client CLI

   The following command waits for the reference chain ibc-0 to halt and then performs the upgrade for client 07-tendermint-0 on ibc-1. It outputs two events, one for the updated client state, and another for the upgraded state. The --upgrade-height 65 value is taken from the height in the upgrade plan output.

   hermes upgrade client --host-chain ibc-1 --client 07-tendermint-0 --upgrade-height 65
   

   Success: [
       UpdateClient(
           h: 1-68, cs_h: 07-tendermint-0(0-65),
       ),
       UpgradeClient(
           UpgradeClient(
               Attributes {
                   height: Height {
                       revision: 1,
                       height: 68,
                   },
                   client_id: ClientId(
                       "07-tendermint-0",
                   ),
                   client_type: Tendermint,
                   consensus_height: Height {
                       revision: 0,
                       height: 66,
                   },
               },
           ),
       ),
   ]
   

IBC Packet Forwarding

This section covers IBC packet forwarding introduced in Gaia v6.0.x

Overview

v3.0.0+

Packet forwarding middleware introduced

Packet forwarding allows a Chain to send a packet to another chain without having a direct channel. This means that if there are three Chains A, B and C, with the following channels:

flowchart LR
    A((ibc-0))---B((ibc-1))---C((ibc-2))

Then if the IBC packet forward middleware is active, Chain A can send a packet to Chain C, by including the following memo:

{
  "forward": {
    "receiver": "chain-c-bech32-address",
    "port": "transfer",
    "channel": "channel-123"
  }
}

If there are four Chains A, B, C and D, with the following channels:

flowchart LR
    A((ibc-0))---B((ibc-1))---C((ibc-2))---D((ibc-3))

Then if the IBC packet forward middleware is active, Chain A can send a packet to Chain D, by including the following memo:

{
  "forward": {
    "receiver": "chain-c-bech32-address",
    "port": "transfer",
    "channel": "channel-a-to-b",
    "timeout": "10m",
    "retries": 2,
    "next": {
      "forward": {
        "receiver": "chain-d-bech32-address",
        "port": "transfer",
        "channel":"channel-c-to-d",
        "timeout":"10m",
        "retries": 2
      }
    }
  }
}

Legacy method

Before the packet forward middleware v3.0.0 the receiver address was used to forward packets. In order for Chain A to send a packet to Chain C, the receiver of the packet had to be set as following:

{intermediate_refund_address}|{foward_port}/{forward_channel}:{final_destination_address}

As specified in the packet-forward-middleware module implementation, packet-forward-middleware.

Important notice

Depending on which major version of Gaia is used, the behaviour of packet forwarding will change.

Gaia v7.0.x

The IBC packet forward middleware is disabled on this version. This will cause the sender to be refunded when trying to transfer tokens using packet forwarding.

Gaia v6.0.x

Since this version uses the packet-forward-middleware v1.0.1 and the atomic forward feature was only introduced in v3, if the destination address is invalid, then the intermediary chain will be refunded instead of the sender. And it uses the overloaded receiver method to forward packets.

Gaia v8.0.x

Everything seems to be working as expected in this version, using the memo field to forward packets.

Example

Here is an example of token transfer from Chain A to Chain C, going through Chain B using the memo field.

Legacy Example

Here is an example of token transfer from Chain A to Chain C, going through Chain B using the overloaded receiver.

Testing Packet Forwarding

Prerequisites

• Gaiad at least (v8.0.0). The version can be checked with:

gaiad version --log_level error --long | head -n4

Testing procedure

Setup using Gaia manager

Note: The gm.toml file that we're using here looks like this:

[global]
  add_to_hermes = true
  auto_maintain_config = true
  extra_wallets = 2
  gaiad_binary = "/Users/luca/go/bin/gaiad"
  hdpath = ""
  home_dir = "/Users/luca/.gm"
  ports_start_at = 27000
  validator_mnemonic = ""
  wallet_mnemonic = ""

  [global.hermes]
    binary = "$HOME/.hermes/bin/hermes"
    config = "$HOME/.hermes/config.toml"
    log_level = "trace"
    telemetry_enabled = true
    telemetry_host = "127.0.0.1"
    telemetry_port = 3001

[ibc-0]
  ports_start_at = 27000

[ibc-1]
  ports_start_at = 27010

[ibc-2]
  ports_start_at = 27020

• Run the command gm start
• Run the commands: gm hermes config and gm hermes keys

Test packet forwarding

1. Create a channel between ibc-0 and ibc-1, and another between ibc-1 and ibc-2:

   hermes create channel --a-chain ibc-0 --b-chain ibc-1 --a-port transfer --b-port transfer --new-client-connection
   

   SUCCESS Channel {
       ordering: Unordered,
       a_side: ChannelSide {
           chain: BaseChainHandle {
               chain_id: ChainId {
                   id: "ibc-0",
                   version: 0,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           connection_id: ConnectionId(
               "connection-0",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: Some(
               ChannelId(
                   "channel-0",
               ),
           ),
           version: None,
       },
       b_side: ChannelSide {
           chain: BaseChainHandle {
               chain_id: ChainId {
                   id: "ibc-1",
                   version: 1,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           connection_id: ConnectionId(
               "connection-0",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: Some(
               ChannelId(
                   "channel-0",
               ),
           ),
           version: None,
       },
       connection_delay: 0ns,
   }
   

   hermes create channel --a-chain ibc-1 --b-chain ibc-2 --a-port transfer --b-port transfer --new-client-connection
   

   SUCCESS Channel {
       ordering: Unordered,
       a_side: ChannelSide {
           chain: BaseChainHandle {
               chain_id: ChainId {
                   id: "ibc-1",
                   version: 1,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-1",
           ),
           connection_id: ConnectionId(
               "connection-1",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: Some(
               ChannelId(
                   "channel-1",
               ),
           ),
           version: None,
       },
       b_side: ChannelSide {
           chain: BaseChainHandle {
               chain_id: ChainId {
                   id: "ibc-2",
                   version: 2,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           connection_id: ConnectionId(
               "connection-0",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: Some(
               ChannelId(
                   "channel-0",
               ),
           ),
           version: None,
       },
       connection_delay: 0ns,
   }
   

2. Obtain the addresses of the wallets on each chain:

   hermes keys list --chain ibc-0
   

   SUCCESS 
   - wallet2 (cosmos179ld56nmany7nqmsdjz684rx5t4r5gxspn6hgr)
   - wallet (cosmos1gz507egejvz3ukg3xwr3v04n3xcny7vcnkjw32)
   - wallet1 (cosmos14cgtalvczzm6xuaa086g5tx6sss6e85j55vqrd)
   

   hermes keys list --chain ibc-1
   

   SUCCESS 
   - wallet2 (cosmos1pmzq62tewxla9z7gpntcnvszyrkygnk4mesauy)
   - wallet (cosmos1jwr34yvnkqkc0ddndnh9y8t94hlhn7rapfyags)
   - wallet1 (cosmos1at4nj238c3ltlj0wymwgfjmdjctlvstwj8xl2s)
   

   hermes keys list --chain ibc-2
   

   SUCCESS 
   - wallet2 (cosmos1xpezl2vvwg9fhdmksvne6lygd7dwz4vf65v6ye)
   - wallet (cosmos1nsztzzhl553avufxhqa204908l4dndafqph4tw)
   - wallet1 (cosmos1csdnmydggcyvjd7z8l64z9lpdgmgyr4v7hw5r8)
   

3. (Optional) Check the balance of the wallets before transfering tokens:

   hermes keys balance --all --chain ibc-0
   

   SUCCESS Balances for key `wallet`:
           100000000 samoleans
           99992294 stake
   

   hermes keys balance --all --chain ibc-1
   

   SUCCESS Balances for key `wallet`:
           100000000 samoleans
           99983377 stake
   

   hermes keys balance --all --chain ibc-2
   

   SUCCESS Balances for key `wallet`:
           100000000 samoleans
           99990916 stake
   

4. (Optional) Confirm the name of the channels used for the transfer:

   hermes query channels --counterparty-chain ibc-1 --chain ibc-0
   

   SUCCESS [
       PortChannelId {
           channel_id: ChannelId(
               "channel-0",
           ),
           port_id: PortId(
               "transfer",
           ),
       },
   ]
   

   hermes query channels --counterparty-chain ibc-2 --chain ibc-1
   

   SUCCESS [
       PortChannelId {
           channel_id: ChannelId(
               "channel-1",
           ),
           port_id: PortId(
               "transfer",
           ),
       },
   ]
   

5. In a separate terminal, start an instance of Hermes:

   hermes start
   

6. Transfer token using the special receiver:

   hermes tx ft-transfer --denom samoleans --receiver cosmos1jwr34yvnkqkc0ddndnh9y8t94hlhn7rapfyags --memo '{"forward": {"receiver": "cosmos1al3csagycya3l7ze3dk4345czw9vwgtjtsezut", "port": "transfer", "channel": "channel-1"}}' --timeout-seconds 120  --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-0 --amount 2500
   

   SUCCESS [
       IbcEventWithHeight {
           event: SendPacket(
               SendPacket {
                   packet: Packet {
                       sequence: Sequence(
                           2,
                       ),
                       source_port: PortId(
                           "transfer",
                       ),
                       source_channel: ChannelId(
                           "channel-0",
                       ),
                       destination_port: PortId(
                           "transfer",
                       ),
                       destination_channel: ChannelId(
                           "channel-0",
                       ),
                       data: [123, 34, 97, 109, 111, 117, 110, 116, 34, 58, 34, 50, 53, 48, 48, 34, 44, 34, 100, 101, 110, 111, 109, 34, 58, 34, 115, 97, 109, 111, 108, 101, 97, 110, 115, 34, 44, 34, 114, 101, 99, 101, 105, 118, 101, 114, 34, 58, 34, 99, 111, 115, 109, 111, 115, 49, 106, 119, 114, 51, 52, 121, 118, 110, 107, 113, 107, 99, 48, 100, 100, 110, 100, 110, 104, 57, 121, 56, 116, 57, 52, 104, 108, 104, 110, 55, 114, 97, 112, 102, 121, 97, 103, 115, 124, 116, 114, 97, 110, 115, 102, 101, 114, 47, 99, 104, 97, 110, 110, 101, 108, 45, 49, 58, 99, 111, 115, 109, 111, 115, 49, 110, 115, 122, 116, 122, 122, 104, 108, 53, 53, 51, 97, 118, 117, 102, 120, 104, 113, 97, 50, 48, 52, 57, 48, 56, 108, 52, 100, 110, 100, 97, 102, 113, 112, 104, 52, 116, 120, 34, 44, 34, 115, 101, 110, 100, 101, 114, 34, 58, 34, 99, 111, 115, 109, 111, 115, 49, 103, 122, 53, 48, 55, 101, 103, 101, 106, 118, 122, 51, 117, 107, 103, 51, 120, 119, 114, 51, 118, 48, 52, 110, 51, 120, 99, 110, 121, 55, 118, 99, 110, 107, 106, 119, 51, 50, 34, 125],
                       timeout_height: Never,
                       timeout_timestamp: Timestamp {
                           time: Some(
                               Time(
                                   2022-11-10 16:05:13.409228,
                               ),
                           ),
                       },
                   },
               },
           ),
           height: Height {
               revision: 0,
               height: 59,
           },
       },
   ]
   

7. (Optional) Check the balances:

   hermes keys balance --all --chain ibc-0
   

   SUCCESS Balances for key `wallet`:
           99997500 samoleans
           99985136 stake
   

   hermes keys balance --all --chain ibc-1
   

   SUCCESS Balances for key `wallet`:
           100000000 samoleans
           99972551 stake
   

   hermes keys balance --all --chain ibc-2
   

   SUCCESS Balances for key `wallet`:
           2500 ibc/F47F0D7C9B4F7D971DF647A75A80CB8D905D3230262FEF2996340664D3A12D48
           100000000 samoleans
           99987055 stake
   

Testing Packet Forwarding

Prerequisites

• Gaiad (v6.*.*). The version can be checked with:

gaiad version --log_level error --long | head -n4

Testing procedure

Setup using Gaia manager

Note: The gm.toml file that we're using here looks like this:

[global]
  add_to_hermes = true
  auto_maintain_config = true
  extra_wallets = 2
  gaiad_binary = "/Users/luca/go/bin/gaiad"
  hdpath = ""
  home_dir = "/Users/luca/.gm"
  ports_start_at = 27000
  validator_mnemonic = ""
  wallet_mnemonic = ""

  [global.hermes]
    binary = "$HOME/.hermes/bin/hermes"
    config = "$HOME/.hermes/config.toml"
    log_level = "trace"
    telemetry_enabled = true
    telemetry_host = "127.0.0.1"
    telemetry_port = 3001

[ibc-0]
  ports_start_at = 27000

[ibc-1]
  ports_start_at = 27010

[ibc-2]
  ports_start_at = 27020

• Run the command gm start
• Run the commands: gm hermes config and gm hermes keys

Test packet forwarding

1. Create a channel between ibc-0 and ibc-1, and another between ibc-1 and ibc-2:

   hermes create channel --a-chain ibc-0 --b-chain ibc-1 --a-port transfer --b-port transfer --new-client-connection
   

   SUCCESS Channel {
       ordering: Unordered,
       a_side: ChannelSide {
           chain: BaseChainHandle {
               chain_id: ChainId {
                   id: "ibc-0",
                   version: 0,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           connection_id: ConnectionId(
               "connection-0",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: Some(
               ChannelId(
                   "channel-0",
               ),
           ),
           version: None,
       },
       b_side: ChannelSide {
           chain: BaseChainHandle {
               chain_id: ChainId {
                   id: "ibc-1",
                   version: 1,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           connection_id: ConnectionId(
               "connection-0",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: Some(
               ChannelId(
                   "channel-0",
               ),
           ),
           version: None,
       },
       connection_delay: 0ns,
   }
   

   hermes create channel --a-chain ibc-1 --b-chain ibc-2 --a-port transfer --b-port transfer --new-client-connection
   

   SUCCESS Channel {
       ordering: Unordered,
       a_side: ChannelSide {
           chain: BaseChainHandle {
               chain_id: ChainId {
                   id: "ibc-1",
                   version: 1,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-1",
           ),
           connection_id: ConnectionId(
               "connection-1",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: Some(
               ChannelId(
                   "channel-1",
               ),
           ),
           version: None,
       },
       b_side: ChannelSide {
           chain: BaseChainHandle {
               chain_id: ChainId {
                   id: "ibc-2",
                   version: 2,
               },
               runtime_sender: Sender { .. },
           },
           client_id: ClientId(
               "07-tendermint-0",
           ),
           connection_id: ConnectionId(
               "connection-0",
           ),
           port_id: PortId(
               "transfer",
           ),
           channel_id: Some(
               ChannelId(
                   "channel-0",
               ),
           ),
           version: None,
       },
       connection_delay: 0ns,
   }
   

2. Obtain the addresses of the wallets on each chain:

   hermes keys list --chain ibc-0
   

   SUCCESS 
   - wallet2 (cosmos179ld56nmany7nqmsdjz684rx5t4r5gxspn6hgr)
   - wallet (cosmos1gz507egejvz3ukg3xwr3v04n3xcny7vcnkjw32)
   - wallet1 (cosmos14cgtalvczzm6xuaa086g5tx6sss6e85j55vqrd)
   

   hermes keys list --chain ibc-1
   

   SUCCESS 
   - wallet2 (cosmos1pmzq62tewxla9z7gpntcnvszyrkygnk4mesauy)
   - wallet (cosmos1jwr34yvnkqkc0ddndnh9y8t94hlhn7rapfyags)
   - wallet1 (cosmos1at4nj238c3ltlj0wymwgfjmdjctlvstwj8xl2s)
   

   hermes keys list --chain ibc-2
   

   SUCCESS 
   - wallet2 (cosmos1xpezl2vvwg9fhdmksvne6lygd7dwz4vf65v6ye)
   - wallet (cosmos1nsztzzhl553avufxhqa204908l4dndafqph4tw)
   - wallet1 (cosmos1csdnmydggcyvjd7z8l64z9lpdgmgyr4v7hw5r8)
   

3. (Optional) Check the balance of the wallets before transfering tokens:

   hermes keys balance --all --chain ibc-0
   

   SUCCESS Balances for key `wallet`:
           100000000 samoleans
           99992294 stake
   

   hermes keys balance --all --chain ibc-1
   

   SUCCESS Balances for key `wallet`:
           100000000 samoleans
           99983377 stake
   

   hermes keys balance --all --chain ibc-2
   

   SUCCESS Balances for key `wallet`:
           100000000 samoleans
           99990916 stake
   

4. (Optional) Confirm the name of the channels used for the transfer:

   hermes query channels --counterparty-chain ibc-1 --chain ibc-0
   

   SUCCESS [
       PortChannelId {
           channel_id: ChannelId(
               "channel-0",
           ),
           port_id: PortId(
               "transfer",
           ),
       },
   ]
   

   hermes query channels --counterparty-chain ibc-2 --chain ibc-1
   

   SUCCESS [
       PortChannelId {
           channel_id: ChannelId(
               "channel-1",
           ),
           port_id: PortId(
               "transfer",
           ),
       },
   ]
   

5. In a separate terminal, start an instance of Hermes:

   hermes start
   

6. Transfer token using the special receiver:

   hermes tx ft-transfer --denom samoleans --receiver 'cosmos1jwr34yvnkqkc0ddndnh9y8t94hlhn7rapfyags|transfer/channel-1:cosmos1nsztzzhl553avufxhqa204908l4dndafqph4tw' --timeout-seconds 120 --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-0 --amount 2500
   

   SUCCESS [
       IbcEventWithHeight {
           event: SendPacket(
               SendPacket {
                   packet: Packet {
                       sequence: Sequence(
                           2,
                       ),
                       source_port: PortId(
                           "transfer",
                       ),
                       source_channel: ChannelId(
                           "channel-0",
                       ),
                       destination_port: PortId(
                           "transfer",
                       ),
                       destination_channel: ChannelId(
                           "channel-0",
                       ),
                       data: [123, 34, 97, 109, 111, 117, 110, 116, 34, 58, 34, 50, 53, 48, 48, 34, 44, 34, 100, 101, 110, 111, 109, 34, 58, 34, 115, 97, 109, 111, 108, 101, 97, 110, 115, 34, 44, 34, 114, 101, 99, 101, 105, 118, 101, 114, 34, 58, 34, 99, 111, 115, 109, 111, 115, 49, 106, 119, 114, 51, 52, 121, 118, 110, 107, 113, 107, 99, 48, 100, 100, 110, 100, 110, 104, 57, 121, 56, 116, 57, 52, 104, 108, 104, 110, 55, 114, 97, 112, 102, 121, 97, 103, 115, 124, 116, 114, 97, 110, 115, 102, 101, 114, 47, 99, 104, 97, 110, 110, 101, 108, 45, 49, 58, 99, 111, 115, 109, 111, 115, 49, 110, 115, 122, 116, 122, 122, 104, 108, 53, 53, 51, 97, 118, 117, 102, 120, 104, 113, 97, 50, 48, 52, 57, 48, 56, 108, 52, 100, 110, 100, 97, 102, 113, 112, 104, 52, 116, 120, 34, 44, 34, 115, 101, 110, 100, 101, 114, 34, 58, 34, 99, 111, 115, 109, 111, 115, 49, 103, 122, 53, 48, 55, 101, 103, 101, 106, 118, 122, 51, 117, 107, 103, 51, 120, 119, 114, 51, 118, 48, 52, 110, 51, 120, 99, 110, 121, 55, 118, 99, 110, 107, 106, 119, 51, 50, 34, 125],
                       timeout_height: Never,
                       timeout_timestamp: Timestamp {
                           time: Some(
                               Time(
                                   2022-11-10 16:05:13.409228,
                               ),
                           ),
                       },
                   },
               },
           ),
           height: Height {
               revision: 0,
               height: 59,
           },
       },
   ]
   

7. (Optional) Check the balances:

   hermes keys balance --all --chain ibc-0
   

   SUCCESS Balances for key `wallet`:
           99997500 samoleans
           99985136 stake
   

   hermes keys balance --all --chain ibc-1
   

   SUCCESS Balances for key `wallet`:
           100000000 samoleans
           99972551 stake
   

   hermes keys balance --all --chain ibc-2
   

   SUCCESS Balances for key `wallet`:
           2500 ibc/F47F0D7C9B4F7D971DF647A75A80CB8D905D3230262FEF2996340664D3A12D48
           100000000 samoleans
           99987055 stake
   

Misbehaviour

Table of Contents

• Monitoring Misbehaviour and Evidence Submission

Monitoring Misbehaviour and Evidence Submission

Use the misbehaviour command to monitor the updates for a given client, detect certain types of misbehaviour and submit evidence to the chain. If the evidence passes the on-chain validation, the client is frozen. Further packets cannot be relayed using the frozen client.

DESCRIPTION:
Listen to client update IBC events and handles misbehaviour

USAGE:
    hermes misbehaviour --chain <CHAIN_ID> --client <CLIENT_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>      Identifier of the chain where client updates are monitored for
                                misbehaviour
        --client <CLIENT_ID>    Identifier of the client to be monitored for misbehaviour

The misbehaviour monitor starts by analyzing all headers used in prior client updates. Once finished it registers for update client events and checks any new headers for misbehaviour. If evidence of misbehaviour is found, it submits:

• the tendermint evidence to the reference chain
• the IBC Misbehaviour message with the evidence to the host chain If the chain validates the transaction then the monitor exits.

The following types of misbehaviour are handled:

1. Fork

   Assumes at least one consensus state before the fork point exists. Let existing consensus states on chain B be: [Sn,.., Sf, Sf-1, S0] with Sf-1 being the most recent state before the fork. Chain A is queried for a header Hf' at Sf.height and if it is different from the Hf in the event for the client update (the one that has generated Sf on chain), then the two headers are included in the evidence and submitted. Note that in this case the headers are different but have the same height.

2. BFT time violation for an unavailable header

   Some header with a height that is higher than the latest height on chain A has been accepted and, a consensus state was created on B. Note that this implies that the timestamp of this header must be within the clock_drift of the client. Assume the client on B has been updated with h2(not present on/ produced by chain A) and it has a timestamp of t2 that is at most clock_drift in the future. Then the latest header from A is fetched, let it be h1, with a timestamp of t1. If t1 >= t2 then evidence of misbehavior is submitted to A.

Example

The misbehaviour command outputs an error message displaying MISBEHAVIOUR DETECTED:

hermes misbehaviour --chain ibc-0 --client 07-tendermint-0

Apr 13 20:04:03.347  INFO ibc_relayer::foreign_client: checking misbehaviour for consensus state heights [Height { revision: 1, height: 195 }, Height { revision: 1, height: 85 }, Height { revision: 1, height: 28 }]
Apr 13 20:04:04.425 ERROR ibc_relayer::foreign_client: MISBEHAVIOUR DETECTED ClientId("07-tendermint-0") h1: Height { revision: 1, height: 195 }-Height { revision: 1, height: 85 } h2: Height { revision: 1, height: 195 }-Height { revision: 1, height: 85 }, sending evidence
Apr 13 20:04:05.070  INFO ibc_relayer_cli::commands::misbehaviour: evidence submission result [ClientMisbehaviour(ClientMisbehaviour(Attributes { height: Height { revision: 0, height: 1521 }, client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 1, height: 195 } }))]

Success: Some(
    ClientMisbehaviour(
        ClientMisbehaviour(
            Attributes {
                height: Height {
                    revision: 0,
                    height: 1521,
                },
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 1,
                    height: 195,
                },
            },
        ),
    ),
)

Querying client state from this point will show the client is in frozen state, with frozen_height indicating the height at which the client was frozen:

hermes query client state --chain ibc-0 --client 07-tendermint-0 | jq

Which should output:

{
  "result": {
    "allow_update_after_expiry": true,
    "allow_update_after_misbehaviour": true,
    "chain_id": "ibc-1",
    "frozen_height": {
      "revision_height": 16,
      "revision_number": 1
    },
    "latest_height": {
      "revision_height": 16,
      "revision_number": 1
    },
    "max_clock_drift": {
      "nanos": 0,
      "secs": 3
    },
    "trust_threshold": {
      "denominator": "3",
      "numerator": "1"
    },
    "trusting_period": {
      "nanos": 0,
      "secs": 1209600
    },
    "unbonding_period": {
      "nanos": 0,
      "secs": 1814400
    },
    "upgrade_path": [
      "upgrade",
      "upgradedIBCState"
    ]
  },
  "status": "success"
}

Queries

Hermes supports querying for different objects that exist on a configured chain.

The query command provides the following sub-commands:

Usage

DESCRIPTION:
Query objects from the chain

USAGE:
    hermes query <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    channel        Query information about channels
    channels       Query the identifiers of all channels on a given chain
    client         Query information about clients
    clients        Query the identifiers of all clients on a chain
    connection     Query information about connections
    connections    Query the identifiers of all connections on a chain
    help           Print this message or the help of the given subcommand(s)
    packet         Query information about packets
    transfer       Query information about token transfers
    tx             Query information about transactions

Table of Contents

• Query Clients
• Query Client Data
  • Query the client state
  • Query the client consensus state
  • Query the identifiers of all connections associated with a given client
  • Query for the header used in a client update at a certain height
  • Query for the status of client (active, frozen, or expired)

Query Clients

Use the query clients command to query the identifiers of all clients on a given chain, called the host chain.

DESCRIPTION:
Query the identifiers of all clients on a chain

USAGE:
    hermes query clients [OPTIONS] --host-chain <HOST_CHAIN_ID>

OPTIONS:
    -h, --help
            Print help information

        --omit-chain-ids
            Omit printing the reference (or target) chain for each client

        --reference-chain <REFERENCE_CHAIN_ID>
            Filter for clients which target a specific chain id (implies '--omit-chain-ids')

REQUIRED:
        --host-chain <HOST_CHAIN_ID>    Identifier of the chain to query

Example

Query all clients on ibc-1:

hermes query clients --host-chain ibc-1

Success: [
    ClientChain {
        client_id: ClientId(
            "07-tendermint-0",
        ),
        chain_id: ChainId {
            id: "ibc-0",
            version: 0,
        },
    },
    ClientChain {
        client_id: ClientId(
            "07-tendermint-1",
        ),
        chain_id: ChainId {
            id: "ibc-2",
            version: 2,
        },
    },
]

Query all clients on ibc-1 having ibc-2 as their reference chain:

hermes query clients --reference-chain ibc-2 --host-chain ibc-1

Success: [
    ClientId(
        "07-tendermint-1",
    ),
]

Query Client Data

Use the query client command to query the information about a specific client.

DESCRIPTION:
Query information about clients

USAGE:
    hermes query client <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    connections    Query the client connections
    consensus      Query the client consensus state
    header         Query for the header used in a client update at a certain height
    help           Print this message or the help of the given subcommand(s)
    state          Query the client state
    status         Query the client status (frozen, expired or active)

Query the client state

Use the query client state command to query the client state of a client:

DESCRIPTION:
Query the client state

USAGE:
    hermes query client state [OPTIONS] --chain <CHAIN_ID> --client <CLIENT_ID>

OPTIONS:
    -h, --help               Print help information
        --height <HEIGHT>    The chain height context for the query

REQUIRED:
        --chain <CHAIN_ID>      Identifier of the chain to query
        --client <CLIENT_ID>    Identifier of the client to query

Example

Query the state of client 07-tendermint-2 on ibc-1:

hermes query client state --chain ibc-1 --client 07-tendermint-1

Success: ClientState {
    chain_id: ChainId {
        id: "ibc-2",
        version: 2,
    },
    trust_threshold: TrustThresholdFraction {
        numerator: 1,
        denominator: 3,
    },
    trusting_period: 1209600s,
    unbonding_period: 1814400s,
    max_clock_drift: 3s,
    frozen_height: Height {
        revision: 0,
        height: 0,
    },
    latest_height: Height {
        revision: 2,
        height: 3069,
    },
    upgrade_path: [
        "upgrade",
        "upgradedIBCState",
    ],
    allow_update_after_expiry: true,
    allow_update_after_misbehaviour: true,
}

Query the client consensus state

Use the query client consensus command to query the consensus states of a given client, or the state at a specified height:

DESCRIPTION:
Query the client consensus state

USAGE:
    hermes query client consensus [OPTIONS] --chain <CHAIN_ID> --client <CLIENT_ID>

OPTIONS:
        --consensus-height <CONSENSUS_HEIGHT>
            Height of the client's consensus state to query

    -h, --help
            Print help information

        --height <HEIGHT>
            The chain height context to be used, applicable only to a specific height

REQUIRED:
        --chain <CHAIN_ID>      Identifier of the chain to query
        --client <CLIENT_ID>    Identifier of the client to query

Example

Query the states of client 07-tendermint-0 on ibc-0:

hermes query client consensus --chain ibc-0 --client 07-tendermint-0

Success: [
    Height {
        revision: 1,
        height: 3049,
    },
    Height {
        revision: 1,
        height: 2888,
    },
    Height {
        revision: 1,
        height: 2736,
    },
    Height {
        revision: 1,
        height: 2729,
    },
    Height {
        revision: 1,
        height: 2724,
    },
    Height {
        revision: 1,
        height: 2717,
    },
]

Query ibc-0 at height 2800 for the consensus state for height 2724:

hermes query client consensus --consensus-height 2724 --height 2800 --chain ibc-0 --client 07-tendermint-0

Success: ConsensusState {
    timestamp: Time(
        2021-04-13T14:11:20.969154Z
    ),
    root: CommitmentRoot(
        "371DD19003221B60162D42C78FD86ABF95A572F3D9497084584B75F97B05B70C"
    ),
    next_validators_hash: Hash::Sha256(
        740950668B6705A136D041914FC219045B1D0AD1C6A284C626BF5116005A98A7
    ),
}

Query the identifiers of all connections associated with a given client

Use the query client connections command to query the connections associated with a given client:

DESCRIPTION:
Query the client connections

USAGE:
    hermes query client connections [OPTIONS] --chain <CHAIN_ID> --client <CLIENT_ID>

OPTIONS:
    -h, --help               Print help information
        --height <HEIGHT>    The chain height which this query should reflect

REQUIRED:
        --chain <CHAIN_ID>      Identifier of the chain to query
        --client <CLIENT_ID>    Identifier of the client to query

Example

Query the connections of client 07-tendermint-0 on ibc-0:

hermes query client connections --chain ibc-0 --client 07-tendermint-0

Success: [
    ConnectionId("connection-0"),
    ConnectionId("connection-1"),
]

Query for the header used in a client update at a certain height

DESCRIPTION:
Query for the header used in a client update at a certain height

USAGE:
    hermes query client header [OPTIONS] --chain <CHAIN_ID> --client <CLIENT_ID> --consensus-height <CONSENSUS_HEIGHT>

OPTIONS:
    -h, --help               Print help information
        --height <HEIGHT>    The chain height context for the query. Leave unspecified for latest
                             height.

REQUIRED:
        --chain <CHAIN_ID>                       Identifier of the chain to query
        --client <CLIENT_ID>                     Identifier of the client to query
        --consensus-height <CONSENSUS_HEIGHT>    Height of header to query

Example

Query for the header used in the 07-tendermint-0 client update at height 2724 on ibc-0:

hermes query client header --chain ibc-0 --client 07-tendermint-0 --consensus-height 2724

Success: [
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 0,
                    height: 0,
                },
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 1,
                    height: 2724,
                },
            },
            header: Some(
                Tendermint(...),
            ),
        },
    ),
]

Query for the status of client (active, frozen, or expired)

This command queries the status of a client, ie. whether it is active, frozen or expired.

DESCRIPTION:
Query the client status (frozen, expired or active)

USAGE:
    hermes query client status --chain <CHAIN_ID> --client <CLIENT_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>      Identifier of the chain to query
        --client <CLIENT_ID>    Identifier of the client to query

Example

Query for the status of the client 07-tendermint-0 on ibc-0:

hermes query client status --chain ibc-0 --client 07-tendermint-0

SUCCESS Active

Table of Contents

• Query Connections
• Query Connection Data
  • Query the connection end data
  • Query the identifiers of all channels associated with a given connection

Query Connections

Use the query connections command to query the identifiers of all connections on a given chain.

DESCRIPTION:
Query the identifiers of all connections on a chain

USAGE:
    hermes query connections [OPTIONS] --chain <CHAIN_ID>

OPTIONS:
        --counterparty-chain <COUNTERPARTY_CHAIN_ID>
            Filter the query response by the counterparty chain

    -h, --help
            Print help information

        --verbose
            Enable verbose output, displaying the client for each connection in the response

REQUIRED:
        --chain <CHAIN_ID>    Identifier of the chain to query

Example

Query all connections on ibc-1:

hermes query connections --chain ibc-1

Success: [
    ConnectionId(
        "connection-0",
    ),
    ConnectionId(
        "connection-1",
    ),
]

Query Connection Data

Use the query connection commands to query a specific connection.

DESCRIPTION:
Query information about connections

USAGE:
    hermes query connection <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    channels    Query connection channels
    end         Query connection end
    help        Print this message or the help of the given subcommand(s)

Query the connection end data

Use the query connection end command to query the connection end:

DESCRIPTION:
Query connection end

USAGE:
    hermes query connection end [OPTIONS] --chain <CHAIN_ID> --connection <CONNECTION_ID>

OPTIONS:
    -h, --help               Print help information
        --height <HEIGHT>    Height of the state to query. Leave unspecified for latest height.

REQUIRED:
        --chain <CHAIN_ID>              Identifier of the chain to query
        --connection <CONNECTION_ID>    Identifier of the connection to query [aliases: conn]

Example

Query the connection end of connection connection-1 on ibc-1:

hermes query connection end --chain ibc-1 --connection connection-1

Success: ConnectionEnd {
    state: Open,
    client_id: ClientId(
        "07-tendermint-1",
    ),
    counterparty: Counterparty {
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: Some(
            ConnectionId(
                "connection-0",
            ),
        ),
        prefix: ibc,
    },
    versions: [
        Version {
            identifier: "1",
            features: [
                "ORDER_ORDERED",
                "ORDER_UNORDERED",
            ],
        },
    ],
    delay_period: 0s,
}

Query the identifiers of all channels associated with a given connection

Use the query connection channels command to query the identifiers of the channels associated with a given connection:

DESCRIPTION:
Query connection channels

USAGE:
    hermes query connection channels --chain <CHAIN_ID> --connection <CONNECTION_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>              Identifier of the chain to query
        --connection <CONNECTION_ID>    Identifier of the connection to query [aliases: conn]

Example

Query the channels associated with connection connection-1 on ibc-1:

hermes query connection channels --chain ibc-1 --connection connection-1

Success: [
    PortChannelId {
        channel_id: ChannelId(
            "channel-0",
        ),
        port_id: PortId(
            "transfer",
        ),
    },
    PortChannelId {
        channel_id: ChannelId(
            "channel-1",
        ),
        port_id: PortId(
            "transfer",
        ),
    },
]

Table of Contents

• Query Channels
• Query Channel Data
  • Query the channel end data
  • Query the channel data for both ends of a channel
  • Query the channel client state

Query Channels

Use the query channels command to query the identifiers of all channels on a given chain.

DESCRIPTION:
Query the identifiers of all channels on a given chain

USAGE:
    hermes query channels [OPTIONS] --chain <CHAIN_ID>

OPTIONS:
        --counterparty-chain <COUNTERPARTY_CHAIN_ID>
            Filter the query response by the this counterparty chain

    -h, --help
            Print help information

        --show-counterparty
            Show the counterparty chain, port, and channel

        --verbose
            Enable verbose output, displaying the client and connection ids for each channel in the
            response

REQUIRED:
        --chain <CHAIN_ID>    Identifier of the chain to query

Example

Query all channels on ibc-1:

hermes query channels --chain ibc-1

Success: [
    PortChannelId {
        channel_id: ChannelId(
            "channel-0",
        ),
        port_id: PortId(
            "transfer",
        ),
    },
    PortChannelId {
        channel_id: ChannelId(
            "channel-1",
        ),
        port_id: PortId(
            "transfer",
        ),
    },
]

Query Channel Data

Use the query channel commands to query the information about a specific channel.

DESCRIPTION:
Query information about channels

USAGE:
    hermes query channel <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    client    Query channel's client state
    end       Query channel end
    ends      Query channel ends and underlying connection and client objects
    help      Print this message or the help of the given subcommand(s)

Query the channel end data

Use the query channel end command to query the channel end:

DESCRIPTION:
Query channel end

USAGE:
    hermes query channel end [OPTIONS] --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

OPTIONS:
    -h, --help               Print help information
        --height <HEIGHT>    Height of the state to query

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain to query
        --channel <CHANNEL_ID>    Identifier of the channel to query [aliases: chan]
        --port <PORT_ID>          Identifier of the port to query

Example

Query the channel end of channel channel-1 on port transfer on ibc-1:

hermes query channel end --chain ibc-1 --port transfer --channel channel-1

Success: ChannelEnd {
    state: Open,
    ordering: Unordered,
    remote: Counterparty {
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-0",
            ),
        ),
    },
    connection_hops: [
        ConnectionId(
            "connection-1",
        ),
    ],
    version: "ics20-1",
}

Query the channel data for both ends of a channel

Use the query channel ends command to obtain both ends of a channel:

DESCRIPTION:
Query channel ends and underlying connection and client objects

USAGE:
    hermes query channel ends [OPTIONS] --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

OPTIONS:
    -h, --help               Print help information
        --height <HEIGHT>    Height of the state to query
        --verbose            Enable verbose output, displaying all details of channels, connections
                             & clients

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain to query
        --channel <CHANNEL_ID>    Identifier of the channel to query [aliases: chan]
        --port <PORT_ID>          Identifier of the port to query

Example

Query the channel end of channel channel-1 on port transfer on ibc-0:

hermes query channel ends --chain ibc-0 --port transfer --channel channel-1

Success: ChannelEndsSummary {
    chain_id: ChainId {
        id: "ibc-0",
        version: 0,
    },
    client_id: ClientId(
        "07-tendermint-1",
    ),
    connection_id: ConnectionId(
        "connection-1",
    ),
    channel_id: ChannelId(
        "channel-1",
    ),
    port_id: PortId(
        "transfer",
    ),
    counterparty_chain_id: ChainId {
        id: "ibc-2",
        version: 2,
    },
    counterparty_client_id: ClientId(
        "07-tendermint-1",
    ),
    counterparty_connection_id: ConnectionId(
        "connection-1",
    ),
    counterparty_channel_id: ChannelId(
        "channel-1",
    ),
    counterparty_port_id: PortId(
        "transfer",
    ),
}

Passing the --verbose flag will additionally print all the details of the channel, connection, and client on both ends.

Query the channel client state

Use the query channel client command to obtain the channel's client state:

DESCRIPTION:
Query channel's client state

USAGE:
    hermes query channel client --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain to query
        --channel <CHANNEL_ID>    Identifier of the channel to query [aliases: chan]
        --port <PORT_ID>          Identifier of the port to query

If the command is successful a message with the following format will be displayed:

Success: Some(
    IdentifiedAnyClientState {
        client_id: ClientId(
            "07-tendermint-0",
        ),
        client_state: Tendermint(
            ClientState {
                chain_id: ChainId {
                    id: "network2",
                    version: 0,
                },
                trust_threshold: TrustThreshold {
                    numerator: 1,
                    denominator: 3,
                },
                trusting_period: 1209600s,
                unbonding_period: 1814400s,
                max_clock_drift: 40s,
                latest_height: Height {
                    revision: 0,
                    height: 2775,
                },
                proof_specs: ProofSpecs(
                    [
                        ProofSpec(
                            ProofSpec {
                                leaf_spec: Some(
                                    LeafOp {
                                        hash: Sha256,
                                        prehash_key: NoHash,
                                        prehash_value: Sha256,
                                        length: VarProto,
                                        prefix: [
                                            0,
                                        ],
                                    },
                                ),
                                inner_spec: Some(
                                    InnerSpec {
                                        child_order: [
                                            0,
                                            1,
                                        ],
                                        child_size: 33,
                                        min_prefix_length: 4,
                                        max_prefix_length: 12,
                                        empty_child: [],
                                        hash: Sha256,
                                    },
                                ),
                                max_depth: 0,
                                min_depth: 0,
                            },
                        ),
                        ProofSpec(
                            ProofSpec {
                                leaf_spec: Some(
                                    LeafOp {
                                        hash: Sha256,
                                        prehash_key: NoHash,
                                        prehash_value: Sha256,
                                        length: VarProto,
                                        prefix: [
                                            0,
                                        ],
                                    },
                                ),
                                inner_spec: Some(
                                    InnerSpec {
                                        child_order: [
                                            0,
                                            1,
                                        ],
                                        child_size: 32,
                                        min_prefix_length: 1,
                                        max_prefix_length: 1,
                                        empty_child: [],
                                        hash: Sha256,
                                    },
                                ),
                                max_depth: 0,
                                min_depth: 0,
                            },
                        ),
                    ],
                ),
                upgrade_path: [
                    "upgrade",
                    "upgradedIBCState",
                ],
                allow_update: AllowUpdate {
                    after_expiry: true,
                    after_misbehaviour: true,
                },
                frozen_height: None,
            },
        ),
    },
)

JSON output:

hermes  --json query channel client --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

If the command is successful a message with the following format will be displayed:

{
    "result":
    {
        "client_id":"07-tendermint-0",
        "client_state":
        {
            "allow_update":
            {
                "after_expiry":true,
                "after_misbehaviour":true
            },
            "chain_id":"network2",
            "frozen_height":null,
            "latest_height":
            {
                "revision_height":2775,
                "revision_number":0
            },
            "max_clock_drift":
            {
                "nanos":0,
                "secs":40
            },
            "proof_specs":
            [
                {
                    "inner_spec":
                    {
                        "child_order":[0,1],
                        "child_size":33,
                        "empty_child":"",
                        "hash":1,
                        "max_prefix_length":12,
                        "min_prefix_length":4
                    },
                    "leaf_spec":
                    {
                        "hash":1,
                        "length":1,
                        "prefix":"AA==",
                        "prehash_key":0,
                        "prehash_value":1
                    },
                    "max_depth":0,
                    "min_depth":0
                },
                {
                    "inner_spec":
                    {
                        "child_order":[0,1],
                        "child_size":32,
                        "empty_child":"",
                        "hash":1,
                        "max_prefix_length":1,
                        "min_prefix_length":1
                    },
                    "leaf_spec":
                    {
                        "hash":1,
                        "length":1,
                        "prefix":"AA==",
                        "prehash_key":0,
                        "prehash_value":1
                    },
                    "max_depth":0,
                    "min_depth":0
                }
            ],
            "trust_threshold":
            {
                "denominator":3,
                "numerator":1
            },
            "trusting_period":
            {
                "nanos":0,
                "secs":1209600
            },
            "type":"Tendermint",
            "unbonding_period":
            {
                "nanos":0,
                "secs":1814400
            },
            "upgrade_path":["upgrade","upgradedIBCState"]
        },
        "type":"IdentifiedAnyClientState"
    },
    "status":"success"
}

Packet Queries

Use the query packet commands to query information about packets.

DESCRIPTION:
Query information about packets

USAGE:
    hermes query packet <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    ack              Query packet acknowledgment
    acks             Query packet acknowledgments
    commitment       Query packet commitment
    commitments      Query packet commitments
    help             Print this message or the help of the given subcommand(s)
    pending          Output a summary of pending packets in both directions
    pending-acks     Query pending acknowledgments
    pending-sends    Query pending send packets

Table of Contents

• Pending Packets
• Packet Commitments
• Packet Commitment with Sequence
• Packet Acknowledgments
• Packet Acknowledgment with Sequence
• Unreceived Packets
• Unreceived Acknowledgments

Pending Packets

Use the query packet pending command to query the sequence numbers of all packets that have not yet been received or acknowledged, at both ends of a channel.

DESCRIPTION:
Output a summary of pending packets in both directions

USAGE:
    hermes query packet pending --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain at one end of the channel
        --channel <CHANNEL_ID>    Channel identifier on the chain given by <CHAIN_ID> [aliases:
                                  chan]
        --port <PORT_ID>          Port identifier on the chain given by <CHAIN_ID>

Example

Query the sequence numbers of all packets that either not yet been received or not yet been acknowledged, at both ends of the channel channel-1.

hermes query packet pending --chain ibc-0 --port transfer --channel channel-1

Success: Summary {
    forward: PendingPackets {
        unreceived_packets: [
            2203,
            ...
            2212,
        ],
        unreceived_acks: [
           2183,
           ...
           2202,
        ],
    },
    reverse: PendingPackets {
        unreceived_packets: [
           14,
           ...
           23,
        ],
        unreceived_acks: [
           4,
           ...
           13,
        ],
    },
}

Packet Commitments

Use the query packet commitments command to query the sequence numbers of all packets that have been sent but not yet acknowledged (these are the packets that still have their commitments stored).

DESCRIPTION:
Query packet commitments

USAGE:
    hermes query packet commitments --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain to query
        --channel <CHANNEL_ID>    Identifier of the channel to query [aliases: chan]
        --port <PORT_ID>          Identifier of the port to query

Example

Query ibc-0 for the sequence numbers of packets that still have commitments on ibc-0 and that were sent on transfer port and channel-0:

hermes query packet commitments --chain ibc-0 --port transfer --channel channel-0

Success: PacketSeqs {
    height: Height {
        revision: 0,
        height: 9154,
    },
    seqs: [
        1,
        2,
        3
    ],
}

Packet Commitment with Sequence

Use the query packet commitment command to query the commitment value of a packet with a given sequence number.

DESCRIPTION:
Query packet commitment

USAGE:
    hermes query packet commitment [OPTIONS] --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID> --sequence <SEQUENCE>

OPTIONS:
    -h, --help               Print help information
        --height <HEIGHT>    Height of the state to query. Leave unspecified for latest height.

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain to query
        --channel <CHANNEL_ID>    Identifier of the channel to query [aliases: chan]
        --port <PORT_ID>          Identifier of the port to query
        --sequence <SEQUENCE>     Sequence of packet to query [aliases: seq]

Example

Query ibc-0 for the commitment of packet with sequence 3 sent on transfer port and channel-0:

hermes query packet commitment --chain ibc-0 --port transfer --channel channel-0 --sequence 3

Success: "F9458DC7EBEBCD6D18E983FCAB5BD752CC2A74532BBD50B812DB229997739EFC"

Packet Acknowledgments

Use the query packet acks command to query the sequence numbers of all packets that have been acknowledged.

DESCRIPTION:
Query packet acknowledgments

USAGE:
    hermes query packet acks --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain to query
        --channel <CHANNEL_ID>    Identifier of the channel to query [aliases: chan]
        --port <PORT_ID>          Identifier of the port to query

Example

Query ibc-1 for the sequence numbers of packets acknowledged that were received on transfer port and channel-1:

hermes query packet acks --chain ibc-1 --port transfer --channel channel-1

Success: PacketSeqs {
    height: Height {
        revision: 1,
        height: 9547,
    },
    seqs: [
        1,
        2,
        3
    ],
}

Packet Acknowledgment with Sequence

Use the query packet acknowledgment command to query the acknowledgment value of a packet with a given sequence number.

DESCRIPTION:
Query packet acknowledgment

USAGE:
    hermes query packet ack [OPTIONS] --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID> --sequence <SEQUENCE>

OPTIONS:
    -h, --help               Print help information
        --height <HEIGHT>    Height of the state to query. Leave unspecified for latest height.

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain to query
        --channel <CHANNEL_ID>    Identifier of the channel to query [aliases: chan]
        --port <PORT_ID>          Identifier of the port to query
        --sequence <SEQUENCE>     Sequence of packet to query [aliases: seq]

Example

Query ibc-1 for the acknowledgment of packet with sequence 2 received on transfer port and channel-1:

hermes query packet ack --chain ibc-1 --port transfer --channel channel-1 --sequence 2

Success: "08F7557ED51826FE18D84512BF24EC75001EDBAF2123A477DF72A0A9F3640A7C"

Unreceived Packets

Use the query packet pending-sends command to query the sequence numbers of all packets that have been sent on the source chain but not yet received on the destination chain.

DESCRIPTION:
Query pending send packets

USAGE:
    hermes query packet pending-sends --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain for the unreceived sequences
        --channel <CHANNEL_ID>    Channel identifier [aliases: chan]
        --port <PORT_ID>          Port identifier

Example

Query transfer port and channel-1 on ibc-1 for the sequence numbers of packets sent on ibc-0 but not yet received:

hermes query packet pending-sends --chain ibc-1 --port transfer --channel channel-1

Success: [
    1,
    2,
    3
]

Unreceived Acknowledgments

Use the query packet pending-acks command to query the sequence numbers of all packets that have not yet been acknowledged.

DESCRIPTION:
Query pending acknowledgments

USAGE:
    hermes query packet pending-acks --chain <CHAIN_ID> --port <PORT_ID> --channel <CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>        Identifier of the chain to query the unreceived acknowledgments
        --channel <CHANNEL_ID>    Channel identifier [aliases: chan]
        --port <PORT_ID>          Port identifier

Example

Query transfer port and channel-0 on ibc-0 for the sequence numbers of packets received by ibc-1 but not yet acknowledged on ibc-0:

hermes query packet pending-acks --chain ibc-0 --port transfer --channel channel-0

Success: [
    1,
    2,
    3
]

Tx Queries

Use the query tx command to query information about transaction(s).

DESCRIPTION:
Query information about transactions

USAGE:
    hermes query tx <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    events    Query the events emitted by transaction
    help      Print this message or the help of the given subcommand(s)

Table of Contents

• Transaction Events

Transaction Events

Use the query tx events command to obtain a list of events that a chain generated as a consequence of delivering a transaction.

DESCRIPTION:
Query the events emitted by transaction

USAGE:
    hermes query tx events --chain <CHAIN_ID> --hash <HASH>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>    Identifier of the chain to query
        --hash <HASH>         Transaction hash to query

Example

Query chain ibc-0 for the events emitted due to transaction with hash 6EDBBCBCB779F9FC9D6884ACDC4350E69720C4B362E4ACE6C576DE792F837490:

hermes query tx events --chain ibc-0 --hash 6EDBBCBCB779F9FC9D6884ACDC4350E69720C4B362E4ACE6C576DE792F837490

Success: [
    SendPacket(
        SendPacket {
            height: Height {
                revision: 4,
                height: 6628239,
            },
            packet: PortId("transfer") ChannelId("channel-139") Sequence(2),
        },
    ),
]

Transfer Queries

Use the query transfer command to query information about transfer(s).

DESCRIPTION:
Query information about token transfers

USAGE:
    hermes query transfer <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    denom-trace    Query the denomination trace info from a trace hash
    help           Print this message or the help of the given subcommand(s)

Table of Contents

• Denomination Trace

Denomination Trace

Use the query transfer denom-trace command to obtain the path and base denomination of a given trace hash.

DESCRIPTION:
Query the denomination trace info from a trace hash

USAGE:
    hermes query transfer denom-trace --chain <CHAIN_ID> --hash <HASH>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --chain <CHAIN_ID>    Identifier of the chain
        --hash <HASH>         Trace hash to query

Example

Query chain ibc-1 for the path and base denomination of the trace hash 27A6394C3F9FF9C9DCF5DFFADF9BB5FE9A37C7E92B006199894CF1824DF9AC7C:

hermes query transfer denom-trace --chain ibc-1 --hash 27A6394C3F9FF9C9DCF5DFFADF9BB5FE9A37C7E92B006199894CF1824DF9AC7C

Success: base_denom: samoleans
 path: transfer/channel-0

Or with a JSON output:

hermes  --json query transfer denom-trace --chain ibc-1 --hash 27A6394C3F9FF9C9DCF5DFFADF9BB5FE9A37C7E92B006199894CF1824DF9AC7C

{
    "result":{
        "base_denom":"samoleans",
        "path":"transfer/channel-0"
    },
    "status":"success"
}

Transactions

There are a number of simple commands that perform minimal validation, build and send IBC transactions.

The tx command provides the following sub-commands:

The main purpose of these commands is to support development and testing, and continuous integration. These CLIs take quite a few parameters, and they are explained in the individual subsections.

At a high level, most commands follow this template:

hermes tx <IBC-MESSAGE> --dst-chain-id <CHAIN-ID> --src-chain-id <CHAIN-id> --dst-obj-id <OBJ-ID> --src-obj-id <SRC-OBJ-ID>

In the command template above:

• ibc-message - identifies the "main" IBC message that is being sent, e.g. conn-init, conn-try, chan-open-init, etc. To ensure successful processing on the receiving chain, the majority of these commands build and send two messages: one UpdateClient message followed by the actual IBC message. These two messages are included in a single transaction. This is done for all IBC messages that include proofs collected from the source chain.

  The messages that do not require proofs are:

  • MsgConnectionOpenInit (conn-open-init command),
  • MsgChannelOpenInit (chan-open-init command),
  • MsgChannelCloseInit (chan-close-init command) and
  • MsgTransfer (ft-transfer command)

• dst-chain-id - is the identifier of the chain where the transaction will be sent.

• src-chain-id - is the identifier of the chain that is queried for the data that is included in the transaction, e.g. connection data, client proofs, etc. To ensure correct on-chain state, the relayer also queries the destination chain, however it does not include this information in the Tx to the destination chain.

• dst-obj-id - the identifier of an object on destination chain required by the message, e.g. the client-id associated with the connection on destination chain in connection messages. Or the connection-id in a ConnOpenAck message.

• src-obj-id - the identifier of an object on the source chain, required by the message, e.d. the client-id of the connection on source chain.

• More details about the tx commands can be found in the following sections:

  • Connection
  • Channel Open
  • Channel Close
  • Packet
  • Upgrade

Usage

DESCRIPTION:
Create and send IBC transactions

USAGE:
    hermes tx <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    chan-close-confirm    Confirm the closing of a channel (ChannelCloseConfirm)
    chan-close-init       Initiate the closing of a channel (ChannelCloseInit)
    chan-open-ack         Relay acknowledgment of a channel attempt (ChannelOpenAck)
    chan-open-confirm     Confirm opening of a channel (ChannelOpenConfirm)
    chan-open-init        Initialize a channel (ChannelOpenInit)
    chan-open-try         Relay the channel attempt (ChannelOpenTry)
    conn-ack              Relay acknowledgment of a connection attempt (ConnectionOpenAck)
    conn-confirm          Confirm opening of a connection (ConnectionOpenConfirm)
    conn-init             Initialize a connection (ConnectionOpenInit)
    conn-try              Relay the connection attempt (ConnectionOpenTry)
    ft-transfer           Send a fungible token transfer test transaction (ICS20 MsgTransfer)
    help                  Print this message or the help of the given subcommand(s)
    packet-ack            Relay acknowledgment packets
    packet-recv           Relay receive or timeout packets
    upgrade-chain         Send an IBC upgrade plan

Connection Handshake

The tx commands can be used to establish a connection between two clients.

sequenceDiagram
    autonumber
    participant A as ibc-1
    participant B as ibc-0
    Note over A, B: No connection
    A->>B: ConnectionOpenInit
    Note over B: connection: connection-0
    Note over B: counterparty: none
    B->>A: ConnectionOpenTry
    Note over A: connection: connection-1
    Note over A: counterparty: connection-0
    A->>B: ConnectionOpenAck
    note over B: connection: connection-0
    note over B: counterparty: connection-1
    B->>A: ConnectionOpenConfirm
    Note over A, B: Connection open

Table of Contents

• Connection Init
• Connection Try
• Connection Ack
• Connection Confirm

Connection Init

Use the conn-init command to initialize a new connection on a chain.

DESCRIPTION:
Initialize a connection (ConnectionOpenInit)

USAGE:
    hermes tx conn-init --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-client <DST_CLIENT_ID> --src-client <SRC_CLIENT_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --dst-chain <DST_CHAIN_ID>      Identifier of the destination chain
        --dst-client <DST_CLIENT_ID>    Identifier of the destination client
        --src-chain <SRC_CHAIN_ID>      Identifier of the source chain
        --src-client <SRC_CLIENT_ID>    Identifier of the source client

Example

Given that two clients were previously created with identifier 07-tendermint-0 on chain ibc-0 and identifier 07-tendermint-1 on chain ibc-1, we can initialize a connection between the two clients.

First, let's initialize the connection on ibc-0:

hermes tx conn-init --dst-chain ibc-0 --src-chain ibc-1 --dst-client 07-tendermint-0 --src-client 07-tendermint-1

Success: OpenInitConnection(
    OpenInit(
        Attributes {
            height: Height {
                revision: 0,
                height: 73,
            },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: None,
            counterparty_client_id: ClientId(
                "07-tendermint-1",
            ),
        },
    ),
)

A new connection has been initialized on ibc-0 with identifier connection-0.

Note that the counterparty_connection_id field is currently empty.

Connection Try

Use the conn-try command to establish a counterparty to the connection on the other chain.

DESCRIPTION:
Relay the connection attempt (ConnectionOpenTry)

USAGE:
    hermes tx conn-try [OPTIONS] --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-client <DST_CLIENT_ID> --src-client <SRC_CLIENT_ID> --src-connection <SRC_CONNECTION_ID>

OPTIONS:
        --dst-connection <DST_CONNECTION_ID>
            Identifier of the destination connection (optional) [aliases: dst-conn]

    -h, --help
            Print help information

REQUIRED:
        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --dst-client <DST_CLIENT_ID>
            Identifier of the destination client

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-client <SRC_CLIENT_ID>
            Identifier of the source client

        --src-connection <SRC_CONNECTION_ID>
            Identifier of the source connection (required) [aliases: src-conn]

Example

Let's now create the counterparty to connection-0 on chain ibc-1:

hermes tx conn-try --dst-chain ibc-1 --src-chain ibc-0 --dst-client 07-tendermint-1 --src-client 07-tendermint-0 --src-connection connection-0

Success: OpenTryConnection(
    OpenTry(
        Attributes {
            height: Height {
                revision: 1,
                height: 88,
            },
            connection_id: Some(
                ConnectionId(
                    "connection-1",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-1",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

A new connection has been created on ibc-1 with identifier connection-1.

Note that the field counterparty_connection_id points to the connection on ibc-0.

Connection Ack

Use the conn-ack command to acknowledge the connection on the initial chain.

DESCRIPTION:
Relay acknowledgment of a connection attempt (ConnectionOpenAck)

USAGE:
    hermes tx conn-ack --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-client <DST_CLIENT_ID> --src-client <SRC_CLIENT_ID> --dst-connection <DST_CONNECTION_ID> --src-connection <SRC_CONNECTION_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --dst-client <DST_CLIENT_ID>
            Identifier of the destination client

        --dst-connection <DST_CONNECTION_ID>
            Identifier of the destination connection (required) [aliases: dst-conn]

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-client <SRC_CLIENT_ID>
            Identifier of the source client

        --src-connection <SRC_CONNECTION_ID>
            Identifier of the source connection (required) [aliases: src-conn]

Example

We can now acknowledge on ibc-0 that ibc-1 has accepted the connection attempt:

hermes tx conn-ack --dst-chain ibc-0 --src-chain ibc-1 --dst-client 07-tendermint-0 --src-client 07-tendermint-1 --dst-connection connection-0 --src-connection connection-1

Success: OpenAckConnection(
    OpenAck(
        Attributes {
            height: Height {
                revision: 0,
                height: 206,
            },
            connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-0",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-1",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-1",
            ),
        },
    ),
)

Note that the field counterparty_connection_id now points to the connection on ibc-1.

Connection Confirm

Use the conn-confirm command to confirm that the connection has been acknowledged, and finish the handshake, after which the connection is open on both chains.

DESCRIPTION:
Confirm opening of a connection (ConnectionOpenConfirm)

USAGE:
    hermes tx conn-confirm --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-client <DST_CLIENT_ID> --src-client <SRC_CLIENT_ID> --dst-connection <DST_CONNECTION_ID> --src-connection <SRC_CONNECTION_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --dst-client <DST_CLIENT_ID>
            Identifier of the destination client

        --dst-connection <DST_CONNECTION_ID>
            Identifier of the destination connection (required) [aliases: dst-conn]

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-client <SRC_CLIENT_ID>
            Identifier of the source client

        --src-connection <SRC_CONNECTION_ID>
            Identifier of the source connection (required) [aliases: src-conn]

Example

Confirm on ibc-1 that ibc-0 has accepted the connection attempt.

hermes tx conn-confirm --dst-chain ibc-1 --src-chain ibc-0 --dst-client 07-tendermint-1 --src-client 07-tendermint-0 --dst-connection connection-1 --src-connection connection-0

Success: OpenConfirmConnection(
    OpenConfirm(
        Attributes {
            height: Height {
                revision: 1,
                height: 239,
            },
            connection_id: Some(
                ConnectionId(
                    "connection-1",
                ),
            ),
            client_id: ClientId(
                "07-tendermint-1",
            ),
            counterparty_connection_id: Some(
                ConnectionId(
                    "connection-0",
                ),
            ),
            counterparty_client_id: ClientId(
                "07-tendermint-0",
            ),
        },
    ),
)

We have now successfully established a connection between the two chains.

Channel Open Handshake

The tx commands can be used to establish a channel for a given connection. Only unordered channels are currently supported.

sequenceDiagram
    autonumber
    participant A as ibc-1
    participant B as ibc-0
    Note over A, B: No channel
    A->>B: ChannelOpenInit
    Note over B: channel: channel-0
    Note over B: channel: counterparty: none
    B->>A: ChannelOpenTry
    Note over A: channel: channel-1
    Note over A: channel: counterparty: channel-0
    A->>B: ChannelOpenAck
    note over B: channel: channel-0
    note over B: counterparty: channel-1
    B->>A: ChannelOpenConfirm
    Note over A, B: Channel open

Table of Contents

• Channel Open Init
• Channel Open Try
• Channel Open Ack
• Channel Open Confirm

Channel Open Init

Use the chan-open-init command to initialize a new channel.

DESCRIPTION:
Initialize a channel (ChannelOpenInit)

USAGE:
    hermes tx chan-open-init [OPTIONS] --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-connection <DST_CONNECTION_ID> --dst-port <DST_PORT_ID> --src-port <SRC_PORT_ID>

OPTIONS:
    -h, --help             Print help information
        --order <ORDER>    The channel ordering, valid options 'unordered' (default) and 'ordered'
                           [default: ORDER_UNORDERED]

REQUIRED:
        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --dst-connection <DST_CONNECTION_ID>
            Identifier of the destination connection [aliases: dst-conn]

        --dst-port <DST_PORT_ID>
            Identifier of the destination port

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-port <SRC_PORT_ID>
            Identifier of the source port

Example

First, let's initialize the channel on ibc-0 using an existing connection identified by connection-0:

hermes tx chan-open-init --dst-chain ibc-0 --src-chain ibc-1 --dst-connection connection-0 --dst-port transfer --src-port transfer

Success: OpenInitChannel(
    OpenInit(
        Attributes {
            height: Height {
                revision: 0,
                height: 3091
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: None,
        },
    ),
)

A new channel has been initialized on ibc-1 with identifier channel-0.

Note that the counterparty_channel_id field is currently empty.

Channel Open Try

Use the chan-open-try command to establish a counterparty to the channel on the other chain.

DESCRIPTION:
Relay the channel attempt (ChannelOpenTry)

USAGE:
    hermes tx chan-open-try [OPTIONS] --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-connection <DST_CONNECTION_ID> --dst-port <DST_PORT_ID> --src-port <SRC_PORT_ID> --src-channel <SRC_CHANNEL_ID>

OPTIONS:
        --dst-channel <DST_CHANNEL_ID>
            Identifier of the destination channel (optional) [aliases: dst-chan]

    -h, --help
            Print help information

REQUIRED:
        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --dst-connection <DST_CONNECTION_ID>
            Identifier of the destination connection [aliases: dst-conn]

        --dst-port <DST_PORT_ID>
            Identifier of the destination port

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-channel <SRC_CHANNEL_ID>
            Identifier of the source channel (required) [aliases: src-chan]

        --src-port <SRC_PORT_ID>
            Identifier of the source port

Example

Let's now create the counterparty to channel-0 on chain ibc-1:

hermes tx chan-open-try --dst-chain ibc-1 --src-chain ibc-0 --dst-connection connection-1 --dst-port transfer --src-port transfer --src-channel channel-0

Success: OpenTryChannel(
    OpenTry(
        Attributes {
            height: Height {
                revision: 1,
                height: 3213
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
            connection_id: ConnectionId(
                "connection-1",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
        },
    ),
)

A new channel has been created on ibc-1 with identifier channel-1.

Note that the field counterparty_channel_id points to the channel on ibc-0.

Channel Open Ack

Use the chan-open-ack command to acknowledge the channel on the initial chain.

DESCRIPTION:
Relay acknowledgment of a channel attempt (ChannelOpenAck)

USAGE:
    hermes tx chan-open-ack --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-connection <DST_CONNECTION_ID> --dst-port <DST_PORT_ID> --src-port <SRC_PORT_ID> --dst-channel <DST_CHANNEL_ID> --src-channel <SRC_CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --dst-channel <DST_CHANNEL_ID>
            Identifier of the destination channel (required) [aliases: dst-chan]

        --dst-connection <DST_CONNECTION_ID>
            Identifier of the destination connection [aliases: dst-conn]

        --dst-port <DST_PORT_ID>
            Identifier of the destination port

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-channel <SRC_CHANNEL_ID>
            Identifier of the source channel (required) [aliases: src-chan]

        --src-port <SRC_PORT_ID>
            Identifier of the source port

Example

We can now acknowledge on ibc-0 that ibc-1 has accepted the opening of the channel:

hermes tx chan-open-ack --dst-chain ibc-0 --src-chain ibc-1 --dst-connection connection-0 --dst-port transfer --src-port transfer --dst-channel channel-0 --src-channel channel-1

Success: OpenAckChannel(
    OpenAck(
        Attributes {
            height: Height {
                revision: 0,
                height: 3301
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
        },
    ),
)

Note that the field counterparty_channel_id now points to the channel on ibc-1.

Channel Open Confirm

Use the chan-open-confirm command to confirm that the channel has been acknowledged, and finish the handshake, after which the channel is open on both chains.

DESCRIPTION:
Confirm opening of a channel (ChannelOpenConfirm)

USAGE:
    hermes tx chan-open-confirm --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-connection <DST_CONNECTION_ID> --dst-port <DST_PORT_ID> --src-port <SRC_PORT_ID> --dst-channel <DST_CHANNEL_ID> --src-channel <SRC_CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --dst-channel <DST_CHANNEL_ID>
            Identifier of the destination channel (required) [aliases: dst-chan]

        --dst-connection <DST_CONNECTION_ID>
            Identifier of the destination connection [aliases: dst-conn]

        --dst-port <DST_PORT_ID>
            Identifier of the destination port

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-channel <SRC_CHANNEL_ID>
            Identifier of the source channel (required) [aliases: src-chan]

        --src-port <SRC_PORT_ID>
            Identifier of the source port

Example

Confirm on ibc-1 that ibc-0 has accepted the opening of the channel, after which the channel is open on both chains.

hermes tx chan-open-confirm --dst-chain ibc-1 --src-chain ibc-0 --dst-connection connection-1 --dst-port transfer --src-port transfer --dst-channel channel-1 --src-channel channel-0

    OpenConfirm(
        Attributes {
            height: Height {
                revision: 1,
                height: 3483
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
            connection_id: ConnectionId(
                "connection-1",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
        },
    ),
)

We have now successfully opened a channel over an existing connection between the two chains.

Channel Close Handshake

The channel close handshake involves two steps: init and confirm.

Table of Contents

• Channel Close Init
• Channel Close Confirm

Channel Close Init

Use the chan-close-init command to initialize the closure of a channel.

DESCRIPTION:
Initiate the closing of a channel (ChannelCloseInit)

USAGE:
    hermes tx chan-close-init --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-connection <DST_CONNECTION_ID> --dst-port <DST_PORT_ID> --src-port <SRC_PORT_ID> --dst-channel <DST_CHANNEL_ID> --src-channel <SRC_CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --dst-channel <DST_CHANNEL_ID>
            Identifier of the destination channel (required) [aliases: dst-chan]

        --dst-connection <DST_CONNECTION_ID>
            Identifier of the destination connection [aliases: dst-conn]

        --dst-port <DST_PORT_ID>
            Identifier of the destination port

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-channel <SRC_CHANNEL_ID>
            Identifier of the source channel (required) [aliases: src-chan]

        --src-port <SRC_PORT_ID>
            Identifier of the source port

Example

hermes tx chan-close-init --dst-chain ibc-0 --src-chain ibc-1 --dst-connection connection-0 --dst-port transfer --src-port transfer --dst-channel channel-0 --src-channel channel-1

Success: CloseInitChannel(
    CloseInit(
        Attributes {
            height: Height {
                revision: 0,
                height: 77,
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
            connection_id: ConnectionId(
                "connection-0",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
        },
    ),
)

Channel Close Confirm

Use the chan-close-confirm command to confirm the closure of a channel.

DESCRIPTION:
Confirm the closing of a channel (ChannelCloseConfirm)

USAGE:
    hermes tx chan-close-confirm --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --dst-connection <DST_CONNECTION_ID> --dst-port <DST_PORT_ID> --src-port <SRC_PORT_ID> --dst-channel <DST_CHANNEL_ID> --src-channel <SRC_CHANNEL_ID>

OPTIONS:
    -h, --help    Print help information

REQUIRED:
        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --dst-channel <DST_CHANNEL_ID>
            Identifier of the destination channel (required) [aliases: dst-chan]

        --dst-connection <DST_CONNECTION_ID>
            Identifier of the destination connection [aliases: dst-conn]

        --dst-port <DST_PORT_ID>
            Identifier of the destination port

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-channel <SRC_CHANNEL_ID>
            Identifier of the source channel (required) [aliases: src-chan]

        --src-port <SRC_PORT_ID>
            Identifier of the source port

Example

hermes tx chan-close-confirm --dst-chain ibc-1 --src-chain ibc-0 --dst-connection connection-1 --dst-port transfer --src-port transfer --dst-channel channel-1 --src-channel channel-0

Success: CloseConfirmChannel(
    CloseConfirm(
        Attributes {
            height: Height {
                revision: 1,
                height: 551,
            },
            port_id: PortId(
                "transfer",
            ),
            channel_id: Some(
                ChannelId(
                    "channel-1",
                ),
            ),
            connection_id: ConnectionId(
                "connection-1",
            ),
            counterparty_port_id: PortId(
                "transfer",
            ),
            counterparty_channel_id: Some(
                ChannelId(
                    "channel-0",
                ),
            ),
        },
    ),
)

NOTE: The cosmos-sdk transfer module implementation does not allow the user (hermes in this case) to initiate the closing of channels. Therefore, when using the Gaia release image, the chan-close-init command fails as the MsgChannelCloseInit message included in the transaction is rejected. To be able to test channel closure, you need to patch your gaia deployments.

Packet Tx Commands

Table of Contents

• Fungible token transfer
• Relay receive and timeout packets
• Relay acknowledgment packets

Fungible token transfer

Use the tx ft-transfer command to send ICS-20 fungible token transfer packets. NOTE: This command is mainly used for testing the packet features of Hermes.

DESCRIPTION:
Send a fungible token transfer test transaction (ICS20 MsgTransfer)

USAGE:
    hermes tx ft-transfer [OPTIONS] --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --src-port <SRC_PORT_ID> --src-channel <SRC_CHANNEL_ID> --amount <AMOUNT>

OPTIONS:
        --denom <DENOM>
            Denomination of the coins to send [default: samoleans]

    -h, --help
            Print help information

        --key-name <KEY_NAME>
            Use the given signing key name (default: `key_name` config)

        --memo <MEMO>
            Optional memo included in the transfer

        --number-msgs <NUMBER_MSGS>
            Number of messages to send

        --receiver <RECEIVER>
            The account address on the destination chain which will receive the tokens. If omitted,
            the relayer's wallet on the destination chain will be used

        --timeout-height-offset <TIMEOUT_HEIGHT_OFFSET>
            Timeout in number of blocks since current [default: 0]

        --timeout-seconds <TIMEOUT_SECONDS>
            Timeout in seconds since current [default: 0]

REQUIRED:
        --amount <AMOUNT>
            Amount of coins (samoleans, by default) to send (e.g. `100000`)

        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-channel <SRC_CHANNEL_ID>
            Identifier of the source channel [aliases: src-chan]

        --src-port <SRC_PORT_ID>
            Identifier of the source port

Example

Send two transfer packets from the transfer module and channel-0 of ibc-0 to ibc-1. Each transfer if for 9999 samoleans (default denomination) and a timeout offset of 10 blocks. The transfer fee is paid by the associated account on ibc-1.

hermes tx ft-transfer --timeout-height-offset 1000 --number-msgs 2 --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-0 --amount 9999

Success: [
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 431,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(4),
        },
    ),
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 431,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(5),
        },
    ),
]

The transfer packets are stored on ibc-0 and can be relayed.

To send transfer packets with a custom receiver address use the --receiver flag.

hermes tx ft-transfer --timeout-height-offset 1000 --number-msgs 1 --receiver board:1938586739 --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-0 --amount 9999

Success: [
    SendPacket(
        SendPacket {
            height: Height {
                revision: 0,
                height: 546,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(7),
        },
    ),
]

Relay receive and timeout packets

Use the tx packet-recv command to relay the packets sent but not yet received. If the packets sent have timed out then a timeout packet is sent to the source chain.

DESCRIPTION:
Relay receive or timeout packets

USAGE:
    hermes tx packet-recv [OPTIONS] --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --src-port <SRC_PORT_ID> --src-channel <SRC_CHANNEL_ID>

OPTIONS:
    -h, --help
            Print help information

        --packet-data-query-height <PACKET_DATA_QUERY_HEIGHT>
            Exact height at which the packet data is queried via block_results RPC

REQUIRED:
        --dst-chain <DST_CHAIN_ID>        Identifier of the destination chain
        --src-chain <SRC_CHAIN_ID>        Identifier of the source chain
        --src-channel <SRC_CHANNEL_ID>    Identifier of the source channel [aliases: src-chan]
        --src-port <SRC_PORT_ID>          Identifier of the source port

Example

Send the two transfer packets to the ibc-1 module bound to the transfer port and the channel-0's counterparty.

NOTE: Hermes prepends a Client Update message before the Receive messages.

hermes tx packet-recv --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-0

Success: [
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 1,
                    height: 439,
                },
                client_id: ClientId(
                    "07-tendermint-1",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 0,
                    height: 449,
                },
            },
            header: Some(
                Tendermint(...),
            ),
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 1,
                height: 439,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(4),
            ack: [
                123,
                34,
                114,
                101,
                115,
                117,
                108,
                116,
                34,
                58,
                34,
                65,
                81,
                61,
                61,
                34,
                125,
            ],
        },
    ),
    WriteAcknowledgement(
        WriteAcknowledgement {
            height: Height {
                revision: 1,
                height: 439,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(5),
            ack: [
                123,
                34,
                114,
                101,
                115,
                117,
                108,
                116,
                34,
                58,
                34,
                65,
                81,
                61,
                61,
                34,
                125,
            ],
        },
    ),
]

Both packets have been relayed to ibc-1 and acknowledged.

Relay acknowledgment packets

Use the tx packet-ack command to relay acknowledgments to the original source of the packets.

DESCRIPTION:
Relay acknowledgment packets

USAGE:
    hermes tx packet-ack [OPTIONS] --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --src-port <SRC_PORT_ID> --src-channel <SRC_CHANNEL_ID>

OPTIONS:
    -h, --help
            Print help information

        --packet-data-query-height <PACKET_DATA_QUERY_HEIGHT>
            Exact height at which the packet data is queried via block_results RPC

REQUIRED:
        --dst-chain <DST_CHAIN_ID>        Identifier of the destination chain
        --src-chain <SRC_CHAIN_ID>        Identifier of the source chain
        --src-channel <SRC_CHANNEL_ID>    Identifier of the source channel [aliases: src-chan]
        --src-port <SRC_PORT_ID>          Identifier of the source port

Example

Send the acknowledgments to the ibc-0 module bound to the transfer port and the channel-1's counterparty.

NOTE: The relayer prepends a client update message before the acknowledgments.

hermes tx packet-ack --dst-chain ibc-0 --src-chain ibc-1 --src-port transfer --src-channel channel-1

Success: [
    UpdateClient(
        UpdateClient {
            common: Attributes {
                height: Height {
                    revision: 0,
                    height: 495,
                },
                client_id: ClientId(
                    "07-tendermint-0",
                ),
                client_type: Tendermint,
                consensus_height: Height {
                    revision: 1,
                    height: 483,
                },
            },
            header: Some(
                Tendermint(...),
            ),
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 495,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(4),
        },
    ),
    AcknowledgePacket(
        AcknowledgePacket {
            height: Height {
                revision: 0,
                height: 495,
            },
            packet: PortId("transfer") ChannelId("channel-0") Sequence(5),
        },
    ),
]

Both acknowledgments have been received on ibc-0.

Upgrade Tx Commands

Table of Contents

• Upgrade Chain

Upgrade Chain

Use this to make an upgrade proposal.

DESCRIPTION:
Send an IBC upgrade plan

USAGE:
    hermes tx upgrade-chain [OPTIONS] --reference-chain <REFERENCE_CHAIN_ID> --host-chain <HOST_CHAIN_ID> --host-client <HOST_CLIENT_ID> --amount <AMOUNT> --height-offset <HEIGHT_OFFSET>

OPTIONS:
        --denom <DENOM>
            Denomination for the deposit (default: 'stake')

    -h, --help
            Print help information

        --new-chain <CHAIN_ID>
            New chain identifier to assign to the upgrading chain (optional)

        --new-unbonding <UNBONDING_PERIOD>
            New unbonding period to assign to the upgrading chain, in seconds (optional)

        --upgrade-name <UPGRADE_NAME>
            A string to name the upgrade proposal plan (default: 'plan')

REQUIRED:
        --amount <AMOUNT>
            Amount of stake

        --height-offset <HEIGHT_OFFSET>
            Upgrade height offset in number of blocks since current

        --host-chain <HOST_CHAIN_ID>
            Identifier of the host chain

        --host-client <HOST_CLIENT_ID>
            Identifier of the client on the host chain from which the plan is created

        --reference-chain <REFERENCE_CHAIN_ID>
            Identifier of the chain to upgrade

Example

An upgrade proposal is made for ibc-0, for height 300 blocks from the latest height, with 10000000stake deposited. The proposal will include the upgraded client state constructed from the state of 07-tendermint-0 client on ibc-1.

hermes tx upgrade-chain --reference-chain ibc-0 --host-chain ibc-1 --host-client 07-tendermint-0 --amount 10000000 --height-offset 300

Success: transaction::Hash(779713508B6103E37FADE60483BEE964A90BD67E5F20037B2CC4AE0E90B707C3)

Fee

Hermes supports querying for different objects that exist on a configured chain.

The query command provides the following sub-commands:

Usage

DESCRIPTION:
Interact with the fee middleware

USAGE:
    hermes fee <SUBCOMMAND>

OPTIONS:
    -h, --help    Print help information

SUBCOMMANDS:
    help                           Print this message or the help of the given subcommand(s)
    register-counterparty-payee    Register a counterparty payee for a channel
    register-payee                 Register a payee for a channel
    transfer                       Perform a token transfer supported with a fee

Register Counterparty Payee

Use this command in order to specify the address which will receive the recv_fee from incentivised packets relayed by the specified chain on the specified channel.

NOTE: If the Hermes configuration parameter auto_register_counterpary_payee = true is set, make sure to use the hermes fee register-counterparty-payee command after calling hermes start, otherwise auto_register_counterparty_payee will overwrite the address registered using hermes fee register-counterparty-payee.

DESCRIPTION:
Register a counterparty payee for a channel

USAGE:
    hermes fee register-counterparty-payee --chain <CHAIN_ID> --channel <CHANNEL_ID> --port <PORT_ID> --counterparty-payee <COUNTERPARTY_PAYEE_ADDRESS>

OPTIONS:
    -h, --help    Print help information

FLAGS:
        --chain <CHAIN_ID>
            Identifier of the chain

        --channel <CHANNEL_ID>
            Identifier of the channel [aliases: chan]

        --counterparty-payee <COUNTERPARTY_PAYEE_ADDRESS>
            Address of the counterparty payee

        --port <PORT_ID>
            Identifier of the port

Example

Register the address cosmos10h9stc5v6ntgeygf5xf945njqq5h32r53uquvw for the chain ibc-1 on channel channel-0:

hermes fee register-counterparty-payee --chain ibc-1 --channel channel-0 --port transfer --counterparty-payee cosmos10h9stc5v6ntgeygf5xf945njqq5h32r53uquvw

SUCCESS Successfully registered counterparty payee

Register Payee

Use this command in order to specify the address which will receive the ack_fee and timeout_fee from incentivised packets relayed by the specified chain on the specified channel. By default this is the address of the reverse relayer's wallet.

WARNING: Use this command with caution as some chains do not allow relayer address and payee to be equal. So reverting the payee address to the relayer address might be difficult after using this command.

DESCRIPTION:
Register a payee for a channel

USAGE:
    hermes fee register-payee --chain <CHAIN_ID> --channel <CHANNEL_ID> --port <PORT_ID> --payee <PAYEE_ADDRESS>

OPTIONS:
    -h, --help    Print help information

FLAGS:
        --chain <CHAIN_ID>         Identifier of the chain
        --channel <CHANNEL_ID>     Identifier of the channel [aliases: chan]
        --payee <PAYEE_ADDRESS>    Address of the payee
        --port <PORT_ID>           Identifier of the port

Example

Register the address cosmos10h9stc5v6ntgeygf5xf945njqq5h32r53uquvw for the chain ibc-1 on channel channel-0:

hermes fee register-payee --chain ibc-1 --channel channel-0 --port transfer --payee cosmos10h9stc5v6ntgeygf5xf945njqq5h32r53uquvw

SUCCESS Successfully registered payee

Fungible token transfer with fees

Use the fee transfer command to send an IncentivizedPacket.

DESCRIPTION:
Perform a token transfer supported with a fee

USAGE:
    hermes fee transfer [OPTIONS] --dst-chain <DST_CHAIN_ID> --src-chain <SRC_CHAIN_ID> --src-port <SRC_PORT_ID> --src-channel <SRC_CHANNEL_ID> --amount <AMOUNT>

OPTIONS:
        --ack-fee <ACK_FEE>
            Fee to pay for the Ack message. Default: 0 [default: 0]

        --denom <DENOM>
            Denomination of the coins to send. Default: samoleans [default: samoleans]

    -h, --help
            Print help information

        --key-name <KEY_NAME>
            Use the given signing key name (default: `key_name` config)

        --memo <MEMO>
            Optional memo included in the transfer

        --number-msgs <NUMBER_MSGS>
            Number of messages to send

        --receive-fee <RECEIVE_FEE>
            Fee to pay for the Recv message. Default: 0 [default: 0]

        --recipient <RECIPIENT>
            The account address on the destination chain which will receive the tokens. If omitted,
            the relayer's wallet on the destination chain will be used

        --timeout-fee <TIMEOUT_FEE>
            Fee to pay for the Timeout message. Default: 0 [default: 0]

        --timeout-height-offset <TIMEOUT_HEIGHT_OFFSET>
            Timeout in number of blocks since current. Default: 0 [default: 0]

        --timeout-seconds <TIMEOUT_SECONDS>
            Timeout in seconds since current. Default: 0 [default: 0]

FLAGS:
        --amount <AMOUNT>
            Amount of coins (samoleans, by default) to send (e.g. `100000`)

        --dst-chain <DST_CHAIN_ID>
            Identifier of the destination chain

        --src-chain <SRC_CHAIN_ID>
            Identifier of the source chain

        --src-channel <SRC_CHANNEL_ID>
            Identifier of the source channel [aliases: src-chan]

        --src-port <SRC_PORT_ID>
            Identifier of the source port

Example

Send a transfer packet from the transfer module and channel-0 of ibc-0 to ibc-1. Each transfer is for 9999 samoleans (default denomination), ICS29 fees of 50 samoleans for recv_fee, 25 samoleans for ack_fee, 10 samoleans for timeout_fee and a timeout offset of 10 blocks. The transfer fee is paid by the associated account on ibc-1.

hermes fee transfer  --receive-fee 50 --ack-fee 25 --timeout-fee 10 --timeout-height-offset 1000 --dst-chain ibc-1 --src-chain ibc-0 --src-port transfer --src-channel channel-0 --amount 9999

SUCCESS [
    IbcEventWithHeight {
        event: IncentivizedPacket(
            IncentivizedPacket {
                port_id: PortId(
                    "transfer",
                ),
                channel_id: ChannelId(
                    "channel-0",
                ),
                sequence: Sequence(
                    8,
                ),
                total_recv_fee: [
                    Coin {
                        denom: "stake",
                        amount: Amount(
                            50,
                        ),
                    },
                ],
                total_ack_fee: [
                    Coin {
                        denom: "stake",
                        amount: Amount(
                            25,
                        ),
                    },
                ],
                total_timeout_fee: [
                    Coin {
                        denom: "stake",
                        amount: Amount(
                            10,
                        ),
                    },
                ],
            },
        ),
        height: Height {
            revision: 1,
            height: 1574,
        },
    },
    IbcEventWithHeight {
        event: SendPacket(
            SendPacket {
                packet: Packet {
                    sequence: Sequence(
                        8,
                    ),
                    source_port: PortId(
                        "transfer",
                    ),
                    source_channel: ChannelId(
                        "channel-0",
                    ),
                    destination_port: PortId(
                        "transfer",
                    ),
                    destination_channel: ChannelId(
                        "channel-0",
                    ),
                    data: [123, 34, 97, 109, 111, 117, 110, 116, 34, 58, 34, 49, 48, 48, 48, 34, 44, 34, 100, 101, 110, 111, 109, 34, 58, 34, 115, 116, 97, 107, 101, 34, 44, 34, 114, 101, 99, 101, 105, 118, 101, 114, 34, 58, 34, 99, 111, 115, 109, 111, 115, 49, 52, 122, 115, 50, 120, 51, 56, 108, 109, 107, 119, 52, 101, 113, 118, 108, 51, 108, 112, 109, 108, 53, 108, 56, 99, 114, 122, 97, 120, 110, 54, 109, 55, 119, 117, 122, 110, 120, 34, 44, 34, 115, 101, 110, 100, 101, 114, 34, 58, 34, 99, 111, 115, 109, 111, 115, 49, 109, 57, 108, 51, 53, 56, 120, 117, 110, 104, 104, 119, 100, 115, 48, 53, 54, 56, 122, 97, 52, 57, 109, 122, 104, 118, 117, 120, 120, 57, 117, 120, 114, 101, 53, 116, 117, 100, 34, 125],
                    timeout_height: Never,
                    timeout_timestamp: Timestamp {
                        time: Some(
                            Time(
                                2023-03-22 11:49:54.491498,
                            ),
                        ),
                    },
                },
            },
        ),
        height: Height {
            revision: 1,
            height: 1574,
        },
    },
]

Glossary

These are some definitions used in this guide: