Import and anonymisation

dynoxide import loads a DynamoDB Export (the S3 export format, JSON Lines) into a Dynoxide database. It's how you get a realistic local dataset without hand-writing fixtures, and the anonymisation rules mean production data can be masked on the way in rather than after it has already landed on your laptop.

It runs in one of two modes. File mode (--output) writes a SQLite file, vacuums it, and optionally compresses it. Serve mode (--serve or --mcp) imports into memory and starts a server on top of it, leaving nothing on disk.

Basic import

dynoxide import \
  --source ./export-data/ \
  --schema schema.json \
  --output snapshot.db

--source points at a directory following the DynamoDB Export layout:

export-data/
├── Users/
│   └── data/
│       └── 00000000.json.gz
└── Orders/
    └── data/
        └── 00000000.json.gz

Each part is JSON Lines, named .json or .jsonl, plain or gzipped. From 1.4.0 the .jsonl and .jsonl.gz names are read too, which saves renaming files that came from somewhere other than an S3 export (#250, thanks @samchungy).

--source can also point at a flat directory of those files for a single table. With no <TableName>/data/ folders inside it, the directory's own name is taken as the table name, so ./Users/ imports into Users.

--schema takes DescribeTable JSON, which is what describe-table gives you:

aws dynamodb describe-table --table-name Users > schema.json

Restrict which tables come across with --tables:

dynoxide import --source ./export/ --schema schema.json --output snapshot.db \
  --tables Users,Orders

Anonymisation

Rules live in a TOML file. Each rule picks the items it applies to with match, names the attribute with path, and says what to do with action:

[[rules]]
match = "attribute_exists(email)"
path = "email"
action = { type = "fake", generator = "safe_email" }

[[rules]]
match = "attribute_exists(phone)"
path = "phone"
action = { type = "mask", keep_last = 4, mask_char = "*" }

[[rules]]
match = "attribute_exists(ssn)"
path = "ssn"
action = { type = "hash", salt_env = "ANON_SALT" }

[[rules]]
match = "attribute_exists(notes)"
path = "notes"
action = { type = "redact" }

[consistency]
fields = ["userId", "email"]

export ANON_SALT="$(openssl rand -base64 24)"

dynoxide import \
  --source ./export/ \
  --schema schema.json \
  --rules rules.toml \
  --output anonymised.db

salt_env and seed_env must name a variable holding at least 16 bytes, and the value wants generating rather than choosing - the salt is the only thing standing between a hashed export and anyone who can guess it. An empty variable is refused outright rather than quietly producing an unsalted hash.

Actions

Action	What it does
`fake`	Replaces the value with generated data. Generators: `safe_email`, `name`, `first_name`, `last_name`, `phone_number`, `address`, `company_name`, `sentence`, `word`
`mask`	Keeps the last N characters and masks the rest. Takes `keep_last` and `mask_char`
`hash`	HMAC-SHA256 keyed on a salt read from an environment variable, over a canonical encoding of the value tagged with its DynamoDB type, so a map, list or set hashes as stably as a string. `salt_env` is required
`redact`	Replaces the value with `[REDACTED]`
`null`	Replaces the value with NULL

mask returns a value of keep_last characters or fewer unchanged, and the import counts those rows as an exposure. The example above keeps four characters of phone, so any four-character value in that column survives in clear.

For an attribute a key is built from, most of these are a bad choice. Prefer hash, or a safe_email with a seed: the other fake generators start repeating within a few hundred items, mask collides on shared tails, redact collapses every item onto one key, and null cannot render a key at all. The import warns up front and names the consequence per action.

Rule options

Beyond match, path and action, a rule takes:

Key	What it does
`values`	Supplies `:name` references, in the shape of ExpressionAttributeValues. A match operand that is not a path has to go through this rather than being written inline
`names`	Supplies `#alias` references, in the shape of ExpressionAttributeNames, so a reserved word such as `name` or `status` is reachable
`tables`	Scopes the rule to named tables. A rule scoped to a table the run leaves out is reported as not applied, rather than as having anonymised nothing
`seed_env`	On a `fake` action, names a secret that makes the generated value a function of the original, so a committed fixture only changes where the source data did

[[rules]]
match = "begins_with(pk, :prefix) AND attribute_exists(#n)"
values = { ":prefix" = "USER#" }
names = { "#n" = "name" }
path = "name"
action = { type = "fake", generator = "name", seed_env = "ANON_SEED" }

The rules file is validated before any data is read, so a reference with nothing behind it, or an operand of the wrong type for its function, fails there rather than part way through an export.

Single-table designs

Rules rewrite whole attribute values, and in a single-table design the sensitive value is usually inside a key too - pk = "CUSTOMER#a.okonkwo@example.co.uk"

where a rule on email never reaches it. The output looks anonymised while every partition key still holds the real address.

Pass the same OneTable schema the MCP server takes as --data-model, and the importer rebuilds keys from the anonymised attributes:

dynoxide import \
  --source ./export/ \
  --schema schema.json \
  --rules rules.toml \
  --data-model onetable.json \
  --output anonymised.db

For each item it resolves the entity, notes which of pk, sk and the GSI keys are built from a template such as user#${email}, applies the rules, then renders those keys again. The prefix survives and the key agrees with the attribute by construction, so a rule on email is all a User entity needs. A key is only rewritten where its template reproduces the value it arrived with; anything else is left alone and counted.

Templates follow OneTable's own forms, including dotted paths (${address.city}) and ${name:length:pad} padding. A model describes one table, so when rules are given a multi-table export needs --tables naming the one the model describes.

Exit codes

An import that leaves an original value in the output exits 3 rather than 0. That covers a rule that anonymised nothing, a key whose template could not rebuild it, items matching no entity, and values a mask kept whole. Each one is printed again at the end of the run, so the reason is in front of you.

--serve and --mcp are gated the same way, before any server starts, so an import that leaked will not hand you a server on the data. --accept-exposure returns such a run to 0 for the cases you have looked at and accept.

Everything else is unchanged: 0 is a clean import, 1 is a failure, and 3 is a run that finished while leaving something behind. A warning on its own does not change the exit code - only a value the run actually saw reach the output does.

Keeping values consistent

Anonymising each attribute independently breaks joins: the same user ID becomes two different values in two tables, and the data stops making sense. Fields listed under [consistency].fields get the same anonymised value everywhere they appear in a single import run.

That works because hash is deterministic - same input plus same salt gives the same output. It also means the salt is what protects the data: reuse a salt across runs and the values stay stable across runs too, which is useful for reproducible fixtures and a liability if the salt leaks. Keep it in an environment variable, and out of the rules file. Dynoxide wipes it from memory once the run is done with it.

hash is pseudonymisation rather than anonymisation. Equal inputs give equal outputs by design, which is what keeps the joins working, but it also means the shape of the original data survives and anyone holding the salt can re-derive every value. Treat a hashed export as sensitive, not anonymous.

Upgrading from 1.1.0 or earlier: every hashed value changed in 1.2.0, when hash moved to HMAC-SHA256. A dataset anonymised by an older release will not join against one anonymised now, so re-import both sides, and rotate the salt while you are there.

Serve mode

Import straight into memory and serve it. Nothing is written to disk, so the data disappears with the process:

# HTTP server on the imported data
dynoxide import --source ./export/ --schema schema.json --serve --port 8000

# stdio MCP server, for a coding agent
dynoxide import --source ./export/ --schema schema.json --mcp

# Both at once
dynoxide import --source ./export/ --schema schema.json --serve --mcp --mcp-port 8100

--serve and --mcp both conflict with --output. Used alone, --mcp starts a stdio MCP server; combined with --serve it starts an HTTP one on --mcp-port (8100 by default). Anonymisation applies the same way, so --rules masks the data before it reaches the in-memory database.

Pairing serve mode with --mcp-read-only gives an agent a realistic, anonymised dataset it cannot write to.

Both refuse to start when the import reports that an original value reached the output, so a run that leaked never becomes a server. See Exit codes.

Other options

Flag	Description
`--log quiet`	Suppress server startup and shutdown messages in serve mode; the import report still prints. Conflicts with `--output`
`--force`	Overwrite an existing output file
`--compress`	Compress the output with zstd, producing `snapshot.db.zst` (requires `--output`)
`--continue-on-error`	Carry on when a batch fails instead of aborting
`--data-model`	OneTable schema whose entity key templates rebuild `pk`, `sk` and index keys after anonymisation
`--accept-exposure`	Exit 0 even when the run reports an original value reaching the output

Every flag is in the CLI reference.

Streams and imported data

If the --schema file carries a StreamSpecification, streams are enabled on the imported table automatically. Imported items themselves generate no stream records - bulk import bypasses recording for speed - so recording starts with the first write after the import completes. See Streams.