command hub · importer rewrite

Atlas importer rewrite

Get the new importer through one shadow week with no unexplained variance. The tree lists goals of the importer rewrite. The lead is assigned to the whole rewrite, an area owner to one branch, and a specialist to a single task. Open a worker to inspect its latest report and worktree, answer beside the goal that stopped, and use Send & pause when the thread itself should keep new workers away until you resolve it.

Plan

atlas-lead

Coordinates the whole rewrite by routing decisions, monitoring progress, and choosing which tasks to focus on.
Groundwork Snapshot the import corpus Nine months pinned by content hash. Build the replay harness Old and new importer over the same corpus. Map the legacy schema Every downstream field has an owner. Replace the XML parser Two workers own separate worktrees; open each worker to inspect its evidence. The stalled deduplication task appears under this goal.
w-1

Parser area owner; focused now on declarations and entity boundaries, but responsible for this whole branch of the plan.
Declarations and entities Three new corpus cases remain. Streaming core Constant memory over the full corpus. CDATA edge cases Eleven legacy exports replay cleanly.
w-2

CDATA specialist: escaped-markup corpus cases.
Deduplicate the corpus snapshot Three hours of worker activity produced no commit. Last worktree: atlas/dedupe-attempt.
w-5

Dedupe specialist, reaped after three hours without a commit; its branch and evidence remain inspectable.

What should the host do with the stalled work?

Restart with a fresh worker Preserve tree-w-5 (atlas/dedupe-attempt), its commits, and comments; reap w-5 and start a new worker on it. My take: keep the durable work and replace the stalled process. Park it for tomorrow Keep tree-w-5 (atlas/dedupe-attempt) and take the goal out of today's plan. Drop it Archive tree-w-5 (atlas/dedupe-attempt); corpus storage stays at 40 GB for this rewrite. Choose the zero-row behavior This policy is consequential enough to wait for an explicit choice.

What should happen when a nightly imports zero rows?

Fail and page Preserve the export and stop the downstream publish. My take: zero rows is a failure until proved otherwise. Warn and continue Record the empty import but publish no data. Reconcile schema and ledger The schema decision blocks the backfill, ledger fixture, and nightly variance report. Choose the additive schema Both columns are nullable; the migration and reverse path are green.
w-8

Schema-choice specialist: migration and reconciliation evidence.

What should the host do with the schema branch?

Land it now Merge tree-w-8 (atlas/schema) from w-8. Tests are green, it unblocks the dependent work, and shadow reads begin before anything depends on it. My take: land while the evidence is current. Hold for the backfill Keep tree-w-8 (atlas/schema) unmerged, leave the dependent work idle, and take one larger review tomorrow. Backfill reconciliation fields Bring one replica-only ledger fixture Run the command below, paste its CSV output into the draft, and remove customer_name and billing_email before saving. Saving the draft tells the orchestrator that w-8 can resume.

Which replica-only ledger fixture should the worker use?

psql "$ATLAS_REPLICA" -c "\copy (select * from ledger_entry
    where refunded_minor is not null limit 1) to stdout with csv header"

Paste the complete CSV row here. Remove customer_name and billing_email before saving.

Paste the complete CSV row here. Remove customer_name and billing_email before saving.
 Nightly variance report Waiting for the schema decision and ledger fixture. Publish the parser contract Review the public API branch. A line comment goes to a new evidence worker and stays attached to the source line. Review the public result type
w-3

Public-contract specialist: result types and compatibility tests.

What should the host do with atlas/public-api?

Land Merge tree-w-3 (atlas/public-api) from w-3. The compatibility suite is green. My take: the public contract is ready. Request changes Restate this card and start a fresh revision worker on tree-w-3; the request remains in the record. Land as-is — discard revision Merge tree-w-3 (atlas/public-api) over a standing change request and reap its reviser. Map structured failures Regenerate the SDK fixture Document the operator path Write the replica runbook Write rollback and evidence capture Cut over the nightly This goal is visibly downstream of both schema and ledger evidence. Shadow both importers for one week Swap with the old path behind a flag