command hub · importer rewrite

Atlas importer rewrite

Get the new importer through one shadow week with no unexplained variance. Review the stopped goals and answer beside their evidence.

Plan

atlas-lead

Coordinates the whole rewrite by routing decisions, monitoring progress, and choosing which tasks to focus on.
Replace the XML parser Two workers own separate worktrees; open each worker to inspect its evidence. The stalled deduplication task appears under this goal.
w-1

Parser area owner; focused now on declarations and entity boundaries, but responsible for this whole branch of the plan.
Declarations and entities Three new corpus cases remain. Streaming core Constant memory over the full corpus. CDATA edge cases Eleven legacy exports replay cleanly.
w-2

CDATA specialist: escaped-markup corpus cases.
Deduplicate the corpus snapshot Three hours of worker activity produced no commit. Last worktree: atlas/dedupe-attempt.
w-5

Dedupe specialist, reaped after three hours without a commit; its branch and evidence remain inspectable.

What should the host do with the stalled work?

I recommend restarting, which keeps the durable work and replaces the stalled process.

recommended Restart with a fresh worker Preserve tree-w-5 (atlas/dedupe-attempt), its commits, and comments; reap w-5 and start a new worker on it. Park it for tomorrow Keep tree-w-5 (atlas/dedupe-attempt) and take the goal out of today's plan. Drop it Archive tree-w-5 (atlas/dedupe-attempt); corpus storage stays at 40 GB for this rewrite. Choose the zero-row behavior This policy is consequential enough to wait for an explicit choice.

What should happen when a nightly imports zero rows?

I recommend failing and paging, since zero rows is a failure until proved otherwise.

recommended Fail and page Preserve the export and stop the downstream publish. Warn and continue Record the empty import but publish no data. Reconcile schema and ledger The schema decision blocks the backfill, ledger fixture, and nightly variance report. Choose the additive schema Both columns are nullable; the migration and reverse path are green.
w-8

Schema-choice specialist: migration and reconciliation evidence.

What should the host do with the schema branch?

I recommend landing it now, while the evidence is current.

recommended Land it now Merge tree-w-8 (atlas/schema) from w-8. Tests are green, it unblocks the dependent work, and shadow reads begin before anything depends on it. Hold for the backfill Keep tree-w-8 (atlas/schema) unmerged, leave the dependent work idle, and take one larger review tomorrow. Backfill reconciliation fields Bring one replica-only ledger fixture Run the command below, paste its CSV output into the draft, and remove customer_name and billing_email before saving. Saving the draft tells the orchestrator that w-8 can resume.

Which replica-only ledger fixture should the worker use?

psql "$ATLAS_REPLICA" -c "\copy (select * from ledger_entry
    where refunded_minor is not null limit 1) to stdout with csv header"

Paste the complete CSV row here. Remove customer_name and billing_email before saving.

Paste the complete CSV row here. Remove customer_name and billing_email before saving.
 Nightly variance report Waiting for the schema decision and ledger fixture. Publish the parser contract Review the public API branch. A line comment goes to a new evidence worker and stays attached to the source line. Review the public result type
w-3

Public-contract specialist: result types and compatibility tests.

What should the host do with atlas/public-api?

I recommend landing it, since the public contract is ready.

recommended Land Merge tree-w-3 (atlas/public-api) from w-3. The compatibility suite is green. Request changes Restate this card and start a fresh revision worker on tree-w-3; the request remains in the record. Land as-is — discard revision Merge tree-w-3 (atlas/public-api) over an open change request and reap its reviser. Map structured failures Regenerate the SDK fixture Groundwork Snapshot the import corpus Nine months pinned by content hash. Build the replay harness Old and new importer over the same corpus. Map the legacy schema Every downstream field has an owner. Document the operator path Write the replica runbook Write rollback and evidence capture Cut over the nightly This goal is visibly downstream of both schema and ledger evidence. Shadow both importers for one week Swap with the old path behind a flag