Package detail

@agoric/swing-store

Agoric14.5kApache-2.00.10.0-u21.0

Persistent storage for SwingSet

readme

banner-1500x500

Agoric Platform SDK

This repository contains most of the packages that make up the upper layers of the Agoric platform, with the endo repository providing the lower layers. If you want to build on top of this platform, you don't need these repositories: instead you should follow our instructions for getting started with the Agoric SDK.

But if you are improving the platform itself, these are the repositories to use.

Prerequisites

Prerequisites are enforced in various places that should be kept synchronized with this section (e.g., repoconfig.sh defines golang_version_check and nodejs_version_check shell functions).

Git
Go ^1.22.12
Node.js ^20.9 or ^22.11
- we generally support the latest LTS release: use nvm to keep your local system up-to-date
Yarn (npm install -g yarn)
gcc >=10, clang >=10, or another compiler with __has_builtin()

Any version of Yarn will do: the .yarnrc file should ensure that all commands use the specific checked-in version of Yarn (stored in .yarn/releases/), which we can update later with PRs in conjunction with any necessary compatibility fixes to our package.json files.

Building on Apple Silicon and Newer Architectures

Some dependencies may not be prebuilt for Apple Silicon and other newer architectures, so it may be necessary to build these dependencies from source and install that package’s native dependencies with your package manager (e.g. Homebrew).

Currently these dependencies are:

Canvas

Additionally, if your package manager utilizes a non-standard include path, you may also need to export the following environment variable before running the commands in the Build section.

export CPLUS_INCLUDE_PATH=/opt/homebrew/include

Finally, you will need the native build toolchain installed to build these items from source.

xcode-select --install

Build

From a new checkout of this repository, run:

yarn install
yarn build

When the yarn install is done, the top-level node_modules/ will contain all the shared dependencies, and each subproject's node_modules/ should contain only the dependencies that are unique to that subproject (e.g. when the version installed at the top level does not meet the subproject's constraints). Our goal is to remove all the unique-to-a-subproject deps.

When one subproject depends upon another, node_modules/ will contain a symlink to the subproject (e.g. ERTP depends upon marshal, so node_modules/@endo/marshal is a symlink to packages/marshal).

Run yarn workspaces info to get a report on which subprojects (aka "workspaces") depend upon which others. The mismatchedWorkspaceDependencies section tells us when symlinks could not be used (generally because e.g. ERTP wants marshal@0.1.0, but packages/marshal/package.json says it's actually 0.2.0). We want to get rid of all mismatched dependencies.

The yarn build step generates kernel bundles.

Test

To run all unit tests (in all packages):

yarn test (from the top-level)

To run the unit tests of just a single package (e.g. eventual-send):

cd packages/eventual-send
yarn test

Run the larger demo

Visit https://docs.agoric.com for getting started instructions.

TL;DR:

yarn link-cli ~/bin/agoric
cd ~
agoric init foo
cd foo
agoric install
agoric start

Then browse to http://localhost:8000

Edit Loop

modify something in e.g. zoe/
run yarn build (at the top level or in zoe/)
re-run tests or agoric start --reset
repeat

Doing a yarn build in zoe creates the "contract facet bundle", a single file that rolls up all the Zoe contract vat sources. This bundle file is needed by all zoe contracts before they can invoke zoe~.install(...). If you don't run yarn build, then changes to the Zoe contract facet will be ignored.

Development Standards

All work should happen on branches. Single-commit branches can land on trunk without a separate merge, but multi-commit branches should have a separate merge commit. The merge commit subject should mention which packages were modified (e.g. (SwingSet,cosmic-swingset) merge 123-fix-persistence)
Keep the history tidy. Avoid overlapping branches. Rebase when necessary.
All work should have an Issue. All branches names should include the issue number as a prefix (e.g. 123-description). Use "Labels" on the Issues to mark which packages are affected.
Add user-visible changes to a new file in the changelogs/ directory, named after the Issue number. See the README in those directories for instructions.
Unless the issue spans multiple packages, each branch should only modify a single package.
Releases should be made as according to MAINTAINERS.md.

changelog

Change Log

All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.

0.10.0-u21.0 (2025-06-19)

⚠ BREAKING CHANGES

swing-store: Remove unsafe hostStorage.setExportCallback method

Features

Add consensus-independent vat snapshot archiving configuration to AG_COSMOS_INIT (ffc594f), closes #10036
Add consensus-independent vat transcript archiving configuration to AG_COSMOS_INIT (d2d5803), closes #10036
add exporter.getHostKV() API (eb564f9), closes #8523
cosmic-swingset: Allow launch to accept an already-open swingStore (c65e5b1)
cosmic-swingset: Introduce inquisitor.mjs (e6d75db)
internal: Generalize single-level pick utility to recursive attenuate (6b36d1e)
swing-store: Add options for opening swing-stores (036053c)
swing-store: budget-limited deletion of snapshot and transcripts (c43bf63), closes #8928
swing-store: faster import of swing-store (0170568)
swing-store: Limit item deletion to the previously-current transcript span (766c1bb), closes #9387
swing-store: prevent SwingSet usage of imported swing-store (6a833eb)
swingstore: add repairMetadata() (33b5c1c), closes #8025 #8025
tool for auditing dangling kindID references (eeadc46), closes #7655

Bug Fixes

misc runtime robustness found by typecheck (a033f26)
performance.now() binding (4a3b59b)
rewrite importSwingStore to preserve metadata properly (38c9efc), closes #8025
swing-store: accept budget=Infinity to allow unlimited deletions (c22b656)
swing-store: add 'replay' artifactMode, make export more strict (9939ea6), closes #8105
swing-store: Create archive files with .tmp suffixes in their final directory (1413af2), closes #10239
swing-store: Delete transcript spans in stopUsingTranscript as in rollover (2d1e478), closes #10054
swing-store: ensure crank savepoint is wrapped in transaction (9d2dd3f)
swing-store: explicitly harden prototypes (86c128a)
swing-store: no completeness check when creating exporter (d4df073)
swing-store: Remove unsafe hostStorage.setExportCallback method (c24e990), closes #10062
swing-store: Update archiveTranscript to wait for write stream conclusion (#11294) (fbea810), closes #11293 #11293
types: template syntax (279b903)
update for [@jessie](https://github.com/jessie).js/safe-await-separator (94c6b3c)

0.9.1 (2023-06-02)

Note: Version bump only for package @agoric/swing-store

0.9.0 (2023-05-19)

⚠ BREAKING CHANGES

xsnap: start xsnap takes snapshot stream
xsnap: makeSnapshot yields snapshot data

Features

swing-store: enforce snapStore consistency (b1343b7)
SwingSet: force reload from snapshot (86de218)
swingset-tools: add tool to extract bundles (0144ec1)
swingstore: add support for b0- bundles (4a3b320), closes #7190
xsnap: makeSnapshot yields snapshot data (348bbd2)
xsnap: start xsnap takes snapshot stream (ed87de1)
Add incarnation number to the transcript store records (5d64be7), closes #7482
convert swing-store from LMDB to Sqlite (579a6c7)
eliminate ephemeralSwingStore in favor of Sqlite :memory: database (0283233)
implement bundleStore (34db767), closes #7089
implement swingStore data export/import in support of state sync (268e62f), closes #6773
Integrate kernel with bundleStore (338556a), closes #7197
move snapstore into SQLite database with the rest of the swingstore (5578834), closes #6742
refactor SwingStore APIs to cleanly distinguish kernel facet from host facet (7126822)
relocate snapshot metadata from kvStore to snapStore (4e0f679), closes #6742
use Sqlite save points for crank commit, integrate activity hash into swing-store (6613d7e)

Bug Fixes

swing-store: add isCurrent SQLite CHECK (90bd76c)
swing-store: noteExport span init (5ba0d31)
swing-store: remove explicit wal_checkpoint (efe6b7e), closes #7069
swing-store: replace getAllState/etc with a debug facet (886528c)
swing-store: replace kvStore.getKeys() with getNextKey() (4b2c29b), closes #6468
swing-store: switch to internal line transform (f4b964a)
swingset-tools: extract vat transcript (edbac04)
CI failures in other packages (071bf89)
eliminate snapStore root parameter (f06a171)
incorporate review feedback (24896ee)

0.8.1 (2022-10-05)

Note: Version bump only for package @agoric/swing-store

0.8.0 (2022-09-20)

Features

swing-store: Update snapshot telemetry to differentiate new-file vs. existing-file (73550f3)
add env to keep old snapshots on disk (96e1077)
Report size and timing data of SnapStore save operations (f0a6026)
Write makeSnapshot telemetry to slog (4cdd2f8), closes #6164
swing-store: Switch to lmdb-js (89adc87)
swingstore: switch to async fs for snapstore (13d443c)

Bug Fixes

swing-store: Depend on better-sqlite3 types (ef8a90f)
swing-store: Infer directory to guarantee file rename atomicity (40bc322)
swing-store: Narrow JSDoc templating (4ee3e32)
swing-store: Narrow the signature of measureSeconds to require an async argument (eaf0a68)
record XS snapshots and file sizes to slog and console (5116ebb), closes #5419
Report new metrics in seconds, and do so accurately (c22309f)
Use new || assert style, but when TS confused use if instead (#6174) (94625d3)
swing-store: disable useWritemap (691a4bd)
swing-store: fix tests to use proper temp dbs (e899f5c)
swing-store: IterableIterator types (48836dd)
swing-store: use sqlite transactions for streamStore operations (33ebe2c), closes #6056
swingset: don't delete heap snapshot if it didn't change (2cbe3a8), closes #5901
swingstore: avoid conditional await on commit (67d846d)
tests use debug settings (#5567) (83d751f)

Performance Improvements

swing-store: Improve the efficiency of writing snapshots (ef78e7d), closes #6225

0.7.0 (2022-05-28)

Features

Accept path for swingStore trace (63a209c)
swingset: Add swing store trace option (25c7e79)

Bug Fixes

swing-store: check fs streams are ready (deaaa13)
swingset: fsync snapshots (#5451) (396e4c6)

0.6.6 (2022-04-18)

Note: Version bump only for package @agoric/swing-store

0.6.5 (2022-02-24)

Note: Version bump only for package @agoric/swing-store

0.6.4 (2022-02-21)

Features

swing-store: enable LMDB_MAP_SIZE and SOLO_LMDB_MAP_SIZE (77f67a8)

0.6.3 (2021-12-02)

Note: Version bump only for package @agoric/swing-store

0.6.2 (2021-10-13)

Note: Version bump only for package @agoric/swing-store

0.6.1 (2021-09-23)

Note: Version bump only for package @agoric/swing-store

0.6.0 (2021-09-15)

⚠ BREAKING CHANGES

clean up organization of swing-store

Bug Fixes

swing-store: be resilient to Node.js 16.x fs.rmSync (990f909)

Code Refactoring

clean up organization of swing-store (3c7e57b)

0.5.11 (2021-08-18)

Note: Version bump only for package @agoric/swing-store-lmdb

0.5.10 (2021-08-17)

Note: Version bump only for package @agoric/swing-store-lmdb

0.5.9 (2021-08-15)

Bug Fixes

swingset: delete unused snapshots (#3505) (317959d), closes #3374 #3431

0.26.10 (2021-07-28)

0.5.8 (2021-08-14)

Bug Fixes

swingset: delete unused snapshots (#3505) (317959d), closes #3374 #3431

0.26.10 (2021-07-28)

0.5.7 (2021-07-28)

Note: Version bump only for package @agoric/swing-store-lmdb

0.5.6 (2021-07-01)

Bug Fixes

repair stream store self-interference problem (948d837), closes #3437

0.5.5 (2021-06-28)

Note: Version bump only for package @agoric/swing-store-lmdb

0.5.4 (2021-06-25)

Note: Version bump only for package @agoric/swing-store-lmdb

0.5.3 (2021-06-24)

Features

sqlite-based transcript store (#3402) (960b013), closes #3405

0.5.2 (2021-06-23)

Note: Version bump only for package @agoric/swing-store-lmdb

0.5.1 (2021-06-16)

Note: Version bump only for package @agoric/swing-store-lmdb

0.5.0 (2021-06-15)

⚠ BREAKING CHANGES

swing-store-lmdb: This includes a renaming of the constructors to acknowledge that the different SwingStore constructors are not polymorphic.

Features

swing-store-lmdb: enable configuration of LMDB database size limit (6f7cefa)
greater paranoia about concurrent access (e67c4ef)
move transcripts out of key-value store and into stream stores (a128e93)
overhaul stream store API to better fit actual use in kernel (c5cc00a)
provide streamStore implementations (e094914)
remove .jsonlines hack from simple swing store (ef87997)