seamless-database

seamless-database is the checksum-based metadata and caching service for the Seamless framework. It acts as the distributed computation cache that allows Seamless workflows to avoid recomputing identical transformations, both within a single session and across the entire cluster.

How it works

Seamless uses content-addressed storage: every piece of data (buffers, code, parameters) is identified by its checksum. When a transformation (computation) is submitted, its inputs are hashed into a transformation checksum. Before executing the computation, Seamless components (such as seamless-dask) query the database: "has this transformation been computed before?" If a cached result is found, the result checksum is returned immediately, skipping the computation entirely.

The database stores the following kinds of records:

Table Purpose
Transformation Maps a transformation checksum to its result checksum
RevTransformation Reverse lookup: finds which transformations produced a given result
HashType Stores the packed checksum classification word used by new HashType validation and conversion checks
SyntacticToSemantic Maps between syntactic and semantic checksums per celltype
Expression Caches expression evaluation results keyed by (input_checksum, path, input_celltype, celltype)
MetaData Stores a canonical execution record for each successful, non-probe transformation
IrreproducibleTransformation Records transformations whose results are not reproducible; metadata is preserved on migration

All data is persisted in a single SQLite file (typically seamless.db). The current protocol version is 2.3.

Expression cache schema

Expression input_celltype describes the input checksum's interpretation; celltype describes the produced result. Both are part of the composite cache key. Database, jobserver, remote client, and Dask payloads use those same names. The unreleased old celltype/target_celltype keys are not accepted aliases. Existing development caches need these statements, in order, or a recreated expression cache table:

ALTER TABLE expression RENAME COLUMN celltype TO input_celltype;
ALTER TABLE expression RENAME COLUMN target_celltype TO celltype;

Execution records

MetaData stores one canonical execution record per successful, non-probe transformation, keyed by tf_checksum. Records are write-once: subsequent calls to a recorded transformation hit the normal Transformation cache and do not re-enter the record path.

Two record sizes coexist under the same schema:

  • Minimal record (default) — schema_version, tf_checksum, result_checksum, seamless_version, execution_mode, remote_target, wall_time_seconds, cpu_time_user_seconds, cpu_time_system_seconds, memory_peak_bytes, gpu_memory_peak_bytes. The hot path pays only timing/memory capture and one write.
  • Full record (record: true in seamless.profile.yaml) — adds environment fingerprints (content-addressed node, environment, queue sub-buffer checksums), compilation context, validation snapshots, contract-violation lists, execution envelope (requested cluster/queue/node, scratch/fingertip flags, resolved __env__), and per-job freshness/retry/worker fields.

The validator on PUT metadata checks identity only — record syntax (integer schema_version, body tf_checksum/result_checksum matching the request, sane checksum_fields if present) — not the full payload schema. Identical duplicates are idempotent successes; differing bodies are rejected. Once IrreproducibleTransformation rows exist for a tf_checksum, PUT metadata for that checksum is rejected to avoid silently migrating it back into the normal cache.

When a normal entry moves to IrreproducibleTransformation, its metadata body travels with it unchanged.

Schema upgrade from legacy

The legacy meta_data table had two columns (checksum PRIMARY KEY, metadata JSON). On startup, seamless-database:

  • creates the upgraded table fresh if absent,
  • drops and recreates the legacy table if it is empty,
  • preserves an already-upgraded table on subsequent starts,
  • and fails loudly if a non-empty legacy table is present (it must be migrated explicitly).

Endpoints (request types)

  • PUT metadata — atomically creates missing Transformation and RevTransformation rows alongside the MetaData row. Body: {type: "metadata", checksum: <tf>, result: <result>, value: <record>}.
  • GET metadata — return the canonical record body for a tf_checksum.
  • GET irreproducible — return all rows for a tf_checksum, optionally filtered by result. Each row includes checksum, result, and metadata.
  • PUT irreproducible — move a normal entry into IrreproducibleTransformation, preserving metadata.

The remote-client equivalents are set_execution_record, get_execution_record, and get_irreproducible_records in seamless-remote.

See the agent contract for the full behavioral spec.

Role in the Seamless ecosystem

Other Seamless components interact with the database over HTTP:

  • seamless-dask checks the database cache before scheduling a transformation on the Dask cluster, and writes results back after computation.
  • seamless-remote provides the DatabaseClient / DatabaseLaunchedClient classes that other components use to communicate with the database server.
  • seamless-config defines the launch template for the database server (port range, host, timeout, read/write mode).

The server exposes a JSON-over-HTTP protocol: clients send {"type": "<record_type>", "checksum": "<hex>", ...} via GET (read) or PUT (write) requests.

Installation

pip install seamless-database

Usage

# Start a writable database server on a random port
seamless-database seamless.db --port-range 5520 5530 --writable

# Start a read-only server on a fixed port
seamless-database seamless.db --port 5522

If --port and --port-range are both omitted, seamless-database picks a random free port in the dynamic/private range (49152-65535).

Status-file protocol

seamless-database does not require a status file. If --status-file is omitted, it runs independently.

If --status-file is provided, the file is used for two things:

  1. Report the chosen port, especially when --port-range is used.
  2. Report whether startup succeeded ("running") or failed ("failed").

The status-file protocol is simple:

  1. Wait for the status file to exist and parse it as JSON.
  2. Reuse the existing JSON object as the base payload. An empty JSON object {} is sufficient.
  3. Choose or validate its listening port.
  4. Once the HTTP server is up, rewrite the same file with "status": "running" and the selected "port".
  5. If startup fails before the server is running, rewrite the file with "status": "failed" instead.

If remote-http-launcher is used, it may pre-populate the JSON with fields such as the PID, workdir, or "status": "starting". seamless-database preserves such fields when it writes back the final status.

CLI options

Option Description
database_file Path to the SQLite file (created if it doesn't exist and --writable is set)
--port PORT Fixed network port
--port-range START END Pick a random free port from an inclusive range
--host HOST Bind address (default: 0.0.0.0)
--writable Allow PUT requests; opens the database in read/write mode
--status-file FILE JSON file used to report server status (for process managers)
--timeout SECONDS Stop the server after this many seconds of inactivity

CLI scripts

Installing seamless-database also provides:

  • seamless-database