Upgrade Procedure

This document describes how to upgrade your existing installation of the BanyanDB database.

Server and File Versioning

BanyanDB uses a separate version for the server and the file format. The server version is the version of the BanyanDB server binary, and the file version is the version of the data files stored on the disk.

You can use “banyand liaison/data/standalone -v” to check the server version. The file version is stored in “metadata” file laid in each “segment” folder.

The server will check the file version when it starts. If the file version is not compatible with the supported version list in the server, the server will refuse to start. Please check the CHANGELOG.md to ensure that the new version is compatible with the existing data files.

BanyanDB upgrade procedure is a rolling upgrade, except where a release says otherwise: 0.12 does not support rolling upgrades (see Upgrading to 0.12). You can upgrade the BanyanDB cluster without downtime. But you need to follow the instructions carefully to avoid any data loss:

• Perform the upgrades consecutively. You cannot skip versions.
• You must keep the all nodes with the same version. Do not mix versions in the same cluster.

All node roles (liaison and data) can be upgraded using the same procedure.

Upgrade a Single Node

• Stop the server by sending a SIGINT or SIGTERM signal to the process.
• Wait for the server to stop gracefully.
• Replace the binary with the new version.
• Start the server with the same configuration.

Rolling Strategy to Upgrade a Cluster

Before upgrading the cluster, you should check CHANGELOG.md](https://github.com/apache/skywalking-banyandb/tree/master/CHANGES.md) to ensure that there are no breaking changes in the new version.

• Upgrade the nodes one by one.
• Wait for the node to join the cluster and become healthy before upgrading the next node.
• Upgrade “data” nodes first, then “liaison” nodes.
• After upgrading all nodes, the cluster will be running the new version.

Exception — upgrading to 0.11. The vectorized query paths change the liaison<->data wire format, and a data node must never emit a frame that the liaison it answers cannot decode. For 0.11 the order is reversed: upgrade “liaison” nodes first, then “data” nodes. See Vectorized query paths enabled by default.

Exception — upgrading to 0.12. Rolling and mixed-version upgrades are not supported. Use the Minimum Downtime Strategy and see Rolling and mixed-version upgrades are not supported.

To ensure this strategy works, you should have a minimum of one node for each role of node in the cluster. For example, if you have a 2-node cluster, you should have at least one “liaison” and one “data” node.

The cluster should also have enough capacity to handle the load when one node is down for upgrade.

Minimum Downtime Strategy to Upgrade a Cluster

If the new version has breaking changes, you can use the following strategy to upgrade the cluster with minimum downtime:

• Stop “liaison” nodes gracefully in a parallel manner.
• Stop “data” gracefully.
• Start the new version of “data” nodes.
• Start the new version of “liaison” nodes.

The data ingestion and retrieval will be stopped during the upgrade process. The downtime will be the sum of the time taken to stop and start the nodes. All these steps should be running in parallel to minimize the downtime.

If you don’t have enough resource to perform a rolling upgrade or you have a large cluster with many nodes, you can use the minimum downtime strategy.

Upgrading to 0.12

This section describes breaking changes and important behavioral changes when upgrading to BanyanDB 0.12.0.

Rolling and mixed-version upgrades are not supported (breaking)

BanyanDB 0.12.0 does not support rolling upgrades. Upgrade the whole cluster at once:

1. Stop all clients and every BanyanDB node, liaison and data.
2. Replace the binaries with 0.12.
3. Start the 0.12 data nodes, then the 0.12 liaison nodes.
4. Verify schema initialization and ingestion before restoring normal traffic.

A cluster that mixes 0.12 nodes with nodes of an earlier release is not supported, whatever the node order. Two changes make the two sides unable to talk to each other:

• The liaison and the data nodes exchange native columnar frames, and a 0.12 data node always sends them: measure frames are version 4 only, and a 0.12 liaison rejects a measure frame of any other version with an error. A liaison of an earlier release has no frame decoder for it. There is no negotiation and no fallback to protobuf or to the earlier row-oriented measure frame.
• The series index cutover described in CHANGES.md changes data that all nodes read and write.

Rollback. Rollback is a binary downgrade of the whole cluster, not a flag flip: stop every node and follow the Rollback procedure below. Do not run a mixed cluster during rollback either. The on-disk format of the vectorized query paths is unchanged by this release.

Row-based query path removed (breaking)

The row-based query execution path is gone. Stream, measure and trace queries all execute through the vectorized columnar pipeline, and a distributed data node always answers with a native columnar frame. There is no runtime switch back to row execution or to the protobuf wire format.

--stream-vectorized-enabled, --trace-vectorized-enabled and --measure-vectorized-enabled are removed. A node started with any of them fails at startup with an unknown-flag error. Drop them from systemd units, Helm values, container arguments and compose files as part of this upgrade. --stream-vectorized-batch-size, --trace-vectorized-batch-size and the *-vectorized-query-memory-mib flags are unchanged.

Stream timestamp-order queries with criteria spanning multiple segments

Stream queries that are ordered by timestamp, carry a criteria filter, and span more than one segment may now return fewer elements than 0.11.x did. The removed row scan resumed segment by segment: after its first maxElementSize-sized batch it could keep pulling from later segments, accumulating further matches until the requested limit was filled. The vectorized scan does not see segment boundaries, so it caps at maxElementSize and stops there.

The results it returns are correct and correctly ordered — the response is simply under-filled against the limit where the old path would have gone back for more. Index-order (tag-ordered) queries are unaffected: their merge stays uncapped and still fills the limit.

A client that needs the missing elements should re-issue the query over a narrower time range or page forward. This divergence was always documented as one the vectorized path deliberately does not emulate; with the row path gone, the vectorized answer is now the only answer.

Index-order queries that do not project the ordered tag

An index-order stream query whose ordered tag is not in the client’s tag projection now answers over protobuf on a distributed data node instead of over a columnar frame. The scan projects that tag internally to derive its sort key and hides it again before egress, while the frame egress rebuilds its projection from the batch schema and would leak the hidden tag — so this query shape is routed onto the protobuf path. Results are unaffected; the query simply forgoes the raw-wire fast path.

Stream query shapes with no execution path (breaking)

A stream query the vectorized engine cannot plan now fails with an error. The row path served these shapes without complaint, so the query returned results instead.

One shape reaches this: an orderBy naming a tag that no longer resolves against the stream’s schema, such as an index rule left behind after its tag was dropped. The error names the reason. Drop the stale index rule, or order by a tag that still exists.

Data export

0.12 adds the ExportService used by bydbctl data export:

• Each data and standalone server keeps export session snapshots in an export-snapshots directory per catalog under the catalog’s root path (--<catalog>-export-snapshot-path), created when the first session is; nothing is created at start-up. For stream, measure and trace it holds hard links, so it must stay on the data path’s filesystem; the default follows the root path, not the data path, so set the flag explicitly when the data path is on another filesystem. A property session snapshot is a full copy of the property index and needs as much free space as the property data. The server refuses to start when it equals or sits inside any catalog’s backup snapshot directory (<root-path>/<catalog>/snapshots), equals, contains or sits inside any catalog’s data path, or overlaps another catalog’s export snapshot path; the defaults are unchanged.
• With RBAC enabled, ExportService Plan and Sessions require cluster:admin.
• During a rolling upgrade a data node that still runs an older version has no ExportService; an export dry run reports it as unreachable (a coverage gap, exit 2 with --strict-coverage) instead of failing.

Upgrading to 0.11

This section describes breaking changes and important behavioral changes when upgrading to BanyanDB 0.11.0.

API version 0.11

BanyanDB 0.11 uses API version 0.11 for external clients and internal node communication. A cluster containing both BanyanDB 0.10 and 0.11 nodes is not supported.

Use a maintenance window for this upgrade:

1. Stop writes and all other BanyanDB API clients.
2. Stop all BanyanDB 0.10 nodes.
3. Upgrade and start all BanyanDB nodes at 0.11.
4. Upgrade API clients to require version 0.11, then start them.
5. Verify schema initialization and ingestion before restoring normal traffic.

Rollback. Stop all clients and BanyanDB nodes before restoring the previous versions. Do not run a mixed 0.10/0.11 BanyanDB cluster during rollback.

Vectorized query paths enabled by default (breaking for rolling upgrades)

This section describes upgrading to 0.11 only. The flags below were removed in 0.12, and 0.12 does not support rolling or mixed-version upgrades; see Rolling and mixed-version upgrades are not supported.

The vectorized stream and trace query paths are now enabled by default, joining measure (default-on since 0.10). --stream-vectorized-enabled, --trace-vectorized-enabled and --measure-vectorized-enabled all default to true.

This changes the required rolling-upgrade order. Upgrade “liaison” nodes BEFORE “data” nodes.

The reason is that each flag selects the liaison<->data wire format as well as the query engine. A flag-on distributed data node emits a native columnar frame instead of protobuf. Liaison nodes on 0.11 accept both — they dispatch per message on the frame’s leading magic byte and fall back to protobuf — but a liaison still running an older binary has no frame decoder at all and will fail to deserialize the response.

The two orders behave differently:

This is the opposite of the general rolling strategy documented above, which applies to all other upgrades.

If you cannot control node ordering, either use the Minimum Downtime Strategy, or start the new data nodes with the vectorized paths explicitly disabled and enable them after every liaison is upgraded:

banyand data \
  --stream-vectorized-enabled=false \
  --trace-vectorized-enabled=false \
  --measure-vectorized-enabled=false

Standalone deployments are unaffected: the frame is only emitted on a distributed data node, so a standalone server always uses the protobuf path regardless of the flag.

Rollback. Pass --stream-vectorized-enabled=false / --trace-vectorized-enabled=false / --measure-vectorized-enabled=false on the standalone or data-node command line and restart. The row path and the protobuf wire format resume immediately; no data migration is involved, because the flags affect only the query and wire paths, never the on-disk format.

This rollback applies to 0.11 only. The row path and these flags were removed in 0.12.0 — see Row-based query path removed.

Upgrading to 0.10

This section describes breaking changes and important behavioral changes when upgrading to BanyanDB 0.10.0.

Property on-disk path change (breaking)

The property data storage path structure has changed:

• Before: <data-dir>/property/data/shard-<id>/...
• After: <data-dir>/property/data/<group>/shard-<id>/...

The new path includes the <group> directory under property/data/. Existing data at the old path layout will not be automatically migrated. Plan a data migration strategy or accept that existing property data at the old path will not be accessible after upgrade.

Node discovery default changed (breaking)

The default node discovery mode changed to none. Cluster deployments must explicitly set a supported discovery mode (dns or file, e.g., --node-discovery-mode=dns or --node-discovery-mode=file) on each node. Without this flag, the cluster nodes will not discover each other after the upgrade. For configuration details, see Node Discovery.

Standalone deployments are unaffected since they do not use node discovery.

Windows binaries no longer shipped (breaking)

Release binaries and Docker images no longer include Windows builds. Users requiring Windows must build from source.

Bloom filters removed for dictionary-encoded tags

Queries on dictionary-encoded tags no longer use Bloom filters. The engine now uses a more efficient dictionary-based filter for tag filtering. This may change query performance characteristics. See TSDB documentation for storage format details.

Criteria tags no longer required in projection

Tags used in WHERE conditions no longer need to be included in the SELECT projection. This is a backward-compatible improvement. Existing queries continue to work, and new queries can omit criteria tags from projections. See BydbQL for details.

Property repair activated by default

The property background repair mechanism is now enabled by default. This automatically repairs inconsistent property data. To disable or tune this behavior, use flags such as --property-repair-enabled and --property-repair-cron. See Property Repair for full configuration options.

Rollback

If you encounter any issues during the upgrade process, you can rollback to the previous version by following the same procedure as the upgrade process. If file format is backward compatible, you can rollback to the previous version without any data loss. Please check the CHANGELOG.md to ensure that old version is compatible with the new data files.