Polymesh Node Operator Guide
Introduction
Operators perform critical functions for the network and have strict uptime requirements. This document provides information about the recommended setup and maintenance of a Polymesh operator node. The intended audience is the operator's IT team; however, some business considerations have been included for completeness and to provide the necessary context.
Governance
Polymesh is a permissioned network, meaning potential operators must undergo a governance process to be permitted to work with the Polymesh network. This governance process is on-chain and managed via the Polymesh Improvement Proposal (PIP) mechanism.
Becoming an Operator
The node operator role on Polymesh requires a permission to be assigned. All operators must satisfy selection criteria determined by Polymesh and be approved by the Polymesh Governing Council.
To become an operator on Polymesh, you also need to bond (lock) POLYX in the system. This facilitates the economic incentives on which Polymesh's security relies. The account that stores your bonded POLYX is called the Stash account. New bonds no longer support pointing a separate Controller account at the stash — the Stash key itself always manages the bonded funds (see Stash Key below). Rewards generated for running an operator node can be sent to the Stash account or another specified account.
You do not need to bond all the POLYX in your Stash account and can bond more later. However, withdrawing any bonded POLYX requires waiting for the unbonding period, currently set at 28 days.
Key Management
NB: It is recommended that operators use Ledger Nano S Plus, Nano X, Flex or Stax devices to store their keys. The discontinued Ledger Nano S should NOT be used.
The Nano S does not support setting session keys and is therefore unsuitable for operators.
There are two main types of keys that an operator must manage:
- Session keys
- Stash key
For new bonds, the Stash key itself performs all controller-type actions (see Stash Key below) — there is no separate Controller key to manage. Some stashes bonded before this change still use a legacy separate Controller key (see Legacy Controller Key).
The session keys are the only type of key that the operator node needs access to. The stash key (and any legacy controller key) should be securely stored in a supported hardware wallet.
Session Keys
Session keys are used by the operator node to sign data necessary for consensus. These keys are stored on the operator node itself. Although session keys do not hold any funds, they can be used to perform actions that could result in penalties, such as double signing. It is therefore critical to keep these keys secure.
Session keys can either be generated offline and injected into the operator node or generated within the operator node by calling the appropriate RPC method. Once generated, session keys should be persisted.
Session keys must either be stored within the client or mounted from secure storage via external methods.
Permissioned Identity
For validator node operations, stash keys must be linked to a DID that is assigned the operator role, making it a permissioned identity. Only keys linked to that permissioned identity can be used as Stash keys.
Primary key
This key is a special key under the onchain identity. It has additional capabilities of adding and removing secondary keys from an identity. It cannot access POLYX on secondary keys. It is recommended to keep the Primary key as in a cold wallet and NOT use it as a stash key.
Secondary keys
Secondary keys are authorized by a primary key to join an identity. For the purposes of staking/POLYX transfers they are no different to primary keys. Secondary keys can be given permissions to perform specific onchain actions. e.g. they may be restricted to specific transactions, to specific portfolios (which governs their ability to move and transfer assets), or to administering specific assets as an external agent.
Stash Key
The stash key is the account that holds the POLYX bonded by the operator and, optionally, receives operator rewards. This should be a cold or semi-cold wallet and must never be attached to the operator node. It is recommended to use a multisig account or a supported hardware wallet for the stash key.
Stash keys can be either a primary or a secondary key but MUST be linked to the permissioned DID of the operator.
For new bonds, the stash key itself always manages the bonded funds — there is no separate controller to designate. The stash key can:
- Bond POLYX
- Bond extra POLYX
- Set/update session keys
- Set commission and validate
- Set a destination for reward payments - i.e. stash with automatically stake, stash unstaked, or other account.
- Commence unbonding of stash tokens
- Withdraw unbonded tokens after the 28 day waiting period to the stash account.
- Rebond tokens which are in the process of unbonding
- "Chill" the node i.e. gracefully stop validating after the last era the node was elected to ends.
- Purge Session keys
Because the stash key now directly performs these actions in addition to holding the bonded funds, securing it appropriately (hardware wallet or multisig) is especially important — treat it as at minimum a semi-cold wallet, not purely cold storage, since it needs to sign these operational transactions.
Legacy Controller Key
Stashes bonded before this change (Polymesh v8) may still have a separate Controller key on file, which manages the bonded funds instead of the stash key performing these actions directly. This is a legacy pattern only — it cannot be set up for new bonds, and calling set_controller on such a stash resets its controller back to the stash key permanently (it can no longer be pointed at a different key afterwards).
Where a legacy separate controller is still in use, it should be treated the same way the stash key is treated above: use a multisig account or a supported hardware wallet, since it can hold funds and directly control bonded funds. This key is not directly required by the operator node and should never be shared with it.
Legacy controller keys can be a Primary or Secondary key and can be associated with any identity. They do not need to be associated with the permissioned identity of the Stash. Where a legacy controller exists, it (not the stash key) is the one that performs the actions listed above for the Stash Key.
Network Architecture
The recommended secure operator setup for Testnet/Mainnet includes the following:
- An active operator node configured with session keys
- A warm spare operator node configured like an operator node but without session keys
A minimum setup consists of a single operator node.
The --reserved-only flag, combined with the --reserved-nodes parameter, allows you to restrict connections to a whitelist of nodes that the operator node may peer with.
Firewall Traffic
To function properly, Polymesh nodes require the following traffic to be whitelisted:
-
All Nodes:
-
Port 443 Egress (HTTPS) (optional but recommended): Used to send basic telemetry to Polymesh servers. Enabling telemetry allows your node to appear on the Polymesh Telemetry page.
-
NTP Egress: System clock drift can cause a node to fail to produce blocks due to mismatched timestamps. Ensure your nodes are synchronized with a reliable NTP server.
-
-
Operator Nodes:
- Libp2p Ingress/Egress: Operator nodes must send and receive P2P events from the WAN or a trusted set of nodes with WAN connectivity. (Default Port: 30333)
High Availability
Operator Node High Availability
The network tolerates simultaneous outages of multiple operator nodes, provided a critical threshold is maintained. An operator node can experience brief downtime for maintenance or upgrades. However, frequent or prolonged downtime—or outages of multiple nodes simultaneously—risks incurring slashing penalties.
Only one operator node may be active with the same session keys at a time. If multiple operator nodes with identical session keys are online simultaneously, they may sign conflicting blocks, resulting in equivocation penalties. Therefore, automatic failover is discouraged. Instead, maintain a warm spare node for supervised failover.
Two failover methods are available:
- Shared Session Key: The session keys are shared with the warm spare, which is activated if the primary node fails. The primary node must not come back online. The penalty for equivocation is much higher than for being offline.
- Unique Session Key: Each operator node instance has a unique session key. If the primary node fails, the stash key (or legacy controller key, if one is in use) must update the session keys on-chain for the secondary node. As key changes take effect in the next session, there may still be a period where the node is offline for one session. This method eliminates the risk of equivocation penalties.
Storing the stash or controller key on a server for automated key change transactions is not recommended. However, you may pre-sign an immortal transaction (without a timeout) and store it on a server to broadcast if the primary node fails. See Upgrading or Replacing a Node for details.
Getting the Polymesh Node Software
All Polymesh nodes use the same binary, differing only in runtime parameters.
There are several ways to obtain the node binary:
-
Prebuilt Container Images: Fetch from the Polymesh Docker Hub repository. Two flavors are available:
debian(with a shell for easier debugging) anddistroless(reduced attack surface, no shell). Images are tagged as<flavor>or<flavor>-<version>. Use versioned tags for deterministic updates. See our sample Docker Compose files. Refer to the Polymesh Developer Documentation for a guide to running a Polymesh node with Docker. -
Precompiled Binary: Download from the GitHub releases page. Releases include the Polymesh binary, its checksum, and optional runtime archives. The runtimes are already included in the binary.
-
Build from Source: Clone the release branch and follow the repository instructions to compile the binary.
Node Resource Requirements
At the time of writing, each Polymesh node should have the following minimum resources:
| Resource | Minimum Value | Recommended Value |
|---|---|---|
| CPU | 2 CPUs | 4 CPUs |
| RAM | 8 GB | 8+ GB |
| Storage | 250 GB SSD | 400+ GB low-latency SSD (e.g., local NVMe) |
As the blockchain grows, storage requirements will increase. Ensure sufficient spare storage or expandable volumes and monitor disk usage. A long-running node accumulates write-ahead logs (WAL) in the database directory. These logs are compacted upon node restart. Reserve an additional 40 GB of disk space for WAL.
Running multiple nodes on shared resources is not recommended. For example, it is preferable to run two 2-CPU/8-GB RAM instances (one node each) than a single 4-CPU/16-GB RAM instance hosting two nodes.
Securing the Instances
Best practices for securing your instances should always be followed. These include (but are not limited to):
- Disabling password-based SSH access
- Setting up and enabling a network firewall
- Only opening required ports
- Disabling unnecessary services
- Avoiding the use of the root user and disabling root login
- Keeping your system up to date
- Enabling SELinux
- Monitoring logs and metrics for signs of malicious activity
- Running periodic CIS benchmarks against your systems
Be advised: Due to the constantly evolving cybersecurity landscape, the above list is not, and cannot be, comprehensive. Node operators are responsible for ensuring that their nodes remain secure and adhere to current best practices.
Upgrading or Replacing a Node
Operator Node Upgrades
The recommended process for upgrading operator nodes is to perform a failover to the warm spare operator node. As mentioned in the High Availability section, the unique key approach is preferable to the shared-key approach.
Begin by upgrading the warm spare operator node. Since this node is not actively validating, you can stop the Polymesh client, perform the necessary upgrade, and then resume operation.
Once the warm spare node is upgraded and fully synchronized, make it the active node by submitting a change of session keys associated with your stash to those stored in the warm spare's keystore, using your stash key (or legacy controller key, if one is in use).
To do so:
- (If not already done) Generate a new set of session keys for the warm spare operator node.
- Navigate to Staking > Account Actions.
- Click "Set Session Key" for your bonding account.
- Enter the
keysvalue from the warm spare node and provide the associatedproof, then click "Set Session Key."
See Running an Operator Node for instructions on using the author_rotateKeysWithOwner RPC method to generate session keys and ownership proof.
The change in operator session keys will only take effect in the next session. For safety, we recommend waiting at least two sessions before proceeding. If the current session is N, wait until session N + 2 before continuing.
At this point, the warm spare and active operator nodes will have switched roles: the previous warm spare becomes the active node, and vice versa. Ensure they are treated accordingly going forward. Alternatively, you may perform the failover operation again to restore the original active node as the current active node and the original warm spare as the current warm spare.
If a warm spare is unavailable, you may perform an in-place upgrade by stopping the Polymesh client, performing the necessary client upgrade, and restarting the node client. We do not recommend in-place upgrades due to the risk of encountering a failure during the upgrade.
Backing Up a Node
Since Polymesh is a public blockchain, node backups are not strictly required, as you can always synchronize from scratch. However, syncing from scratch can be time-consuming. To minimize the delay between node creation and readiness, you may choose to back up the full blockchain database regularly.
This does not need to be done for every node—a single database backup can be used across nodes, provided they use the same pruning settings. In v8, all nodes including operators use the same default pruning: --state-pruning 256 and --blocks-pruning archive-canonical. These defaults are appropriate for consensus participation. If you create backups with these settings, they can be shared across operator nodes.
Database backups should be performed on an offline node. A typical approach is:
- Stop the Polymesh process on the backup node.
- Snapshot the database directory.
- Restart the Polymesh process.
- Sync the database snapshot to offsite storage.
The database snapshot contains no confidential information as long as only the db directory is backed up (e.g., /<base path>/chains/mainnet/db).
Because the database uses file-based storage, stopping/starting Polymesh may create partial files. Excessive file accumulation can cause performance issues. We recommend limiting snapshots to daily intervals and periodically resetting the backup node's database with a fresh sync from the chain.
Auto-Restarting Nodes
Nodes should automatically restart in the event of an intermittent failure.
Container-Based Nodes
For container-based nodes, use your container runtime's features:
restart_policy.condition: anyfordocker-composerestartPolicy: Alwaysforkubernetes
Binary-Based Nodes
For binary-based nodes, we recommend using a supervisor process. Most modern Linux distributions use systemd, which we will focus on, though other options are also viable.
Setting Up systemd
-
Create a new unit file called
polymesh.servicein/etc/systemd/system/with the following content:[Unit]Description=Polymesh NodeAfter=network.target[Service]User=polymeshGroup=polymeshType=simpleRestart=alwaysRestartSec=5ExecStart=/usr/local/bin/polymesh --chain mainnet --name <your-node-name> --validator[Install]WantedBy=multi-user.target -
Enable automatic startup with:
sudo systemctl enable polymesh -
Manage the service with commands such as:
sudo systemctl start polymeshsudo systemctl stop polymeshsudo systemctl restart polymesh -
View logs using
journalctl:sudo journalctl -u polymesh -f
Refer to the journalctl man pages for additional details.
Common Parameters for Running a Polymesh Node
Recommended options for running a Polymesh node include:
--name <name>(optional): Human-readable name reported to telemetry services.--chain mainnet: Run a Mainnet node (default is Testnet if omitted).--wasm-execution compiled: Use compiled WASM for better performance.--base-path <path>(optional): Specify the location for DB files and the keystore.--node-key-file <path>(optional): Use an explicit node key path when you manage node key outside the default "network" folder.--db-cache <cache size in MiB>(optional): Increase in-memory cache for better performance. On a node with 8GB of available RAM, a reasonable value is4096. (default128MiB)--db-max-total-wal-size <WAL database size in MiB>(optional): Limit the total storage capacity that the database can use for WAL files. Recommended minimum value1024.--state-pruningand--blocks-pruning(optional): Configure pruning behavior. Refer to Configuring Pruning for details. The v8 defaults (--state-pruning 256and--blocks-pruning archive-canonical) are appropriate for operator nodes.
Note: The <name> parameter will be publicly visible when telemetry is enabled (default setting).
To see a full list of available options and their descriptions, run the --help command.
Running an Operator Node
To run an operator node, use the following in addition to the common parameters:
--validator: Enable validator mode.
To connect only to trusted peers, use these options:
--reserved-only: Restrict connections to reserved peers.--reserved-nodes: A space-separated list of libp2p peer addresses in the format/ip4/<IP>/tcp/30333/p2p/<Node ID>or/dns4/<Hostname>/tcp/30333/p2p/<Node ID>. If omitted, peers must be added via thesystem_addReservedPeerRPC method.
For validator operations, keep a stable node key persisted under your --base-path. With default settings, the node key is stored in the chain-specific network folder:
/<base-path>/chains/<chain-name>/network/secret_ed25519
If the key is stored in this default location, you do not need to pass --node-key-file.
Use --node-key-file when you intentionally store the node key in a custom location.
For testing you can avoid separate key generation by using --unsafe-force-node-key-generation. This forces key generation if the node-key-file does not exist on container start, which is useful for testing but not recommended for production.
--unsafe-force-node-key-generation is considered unsafe for active authorities because identity changes can break connectivity with peers that expect a stable peer ID.
Generating and Inspecting Node Keys
Before starting a validator, generate and verify the node key explicitly.
Generate a node key in the default chain network location under your --base-path:
polymesh key generate-node-key --base-path <base-path> --chain mainnet
With --base-path <base-path> and --chain mainnet, this writes the key to:
<base-path>/chains/mainnet/network/secret_ed25519
If you run testnet, the path is:
<base-path>/chains/testnet/network/secret_ed25519
Inspect the key and print the peer ID:
polymesh key inspect-node-key --file <base-path>/chains/mainnet/network/secret_ed25519
If you keep the key in the default location, no explicit node-key argument is required at startup. Use --node-key-file only when you intentionally store the key in a custom path.
Next, generate the node's session keys.