Debugging SOP

Use: Apply this Standard Operating Procedure (SOP) whenever diagnosing or fixing a defect, failure, regression, crash, incorrect result, integration problem, performance problem, flaky test, packaging failure, or unexpected runtime behavior in Arcane OS. The current rapid-development profile may use the compact path below; the complete evidence sequence is required for security/privilege defects, destructive or persistent failures, unexplained flakes, promotion blockers, and release-candidate work.

Meaning of ASS-U-ME

ASS-U-ME is a warning, not a list of invented steps:

Do not assume. An assumption makes an ass out of you and me.

Debugging starts with facts. Do not assume that the report is correct, the expectation is correct, the code is wrong, the environment is current, a dependency is working, or a proposed cause is relevant. Verify each one.

What a bug is

A reported bug is usually a difference between an observed result and an expected result.

That difference does not prove the implementation is defective. The expectation may be incomplete, outdated, misguided, mistaken, or wrong. The observation may also be misunderstood, stale, or taken from a different environment. The first job is therefore to verify both sides of the comparison.

Use this statement:

Under [specific conditions], I observed [result]. I expected [result] because [verified requirement, contract, design, or user instruction].

If the expectation has no verified basis, do not change the system to satisfy it. Clarify the intended behavior first.

Governing rules

1. Work one proven step at a time.
2. Separate facts, expectations, questions, and possible explanations.
3. Reproduce before changing anything when it is safe to do so.
4. Preserve the failing state and evidence before resetting, rebuilding, restarting, clearing, or repairing it.
5. Inspect the exact point where reality first differs from the verified expectation.
6. Change only one independent variable at a time.
7. Prove the correction manually before encoding it in code whenever the failure permits a manual correction.
8. Retest the original report from a clean state after the code is fixed.
9. Never call a disappearing symptom a root cause.

One step at a time does not mean one file at a time. A coherent fix may require coordinated edits. It means that every action answers one question and produces a result that can be interpreted before the next action is chosen.

Rapid-development path

For an ordinary local development defect with no protected data, privilege, persistent migration, destructive state, or release claim:

1. verify the expected behavior from a contract, test, design, or explicit instruction;
2. reproduce the exact reported case or record why it cannot currently be reproduced;
3. preserve the smallest useful error, log, input, and before-state;
4. inspect the first boundary where observed behavior differs;
5. change one cause, add or update a focused regression check, and rerun the original case;
6. run the nearest affected integration check and npm run check before handoff.

Manual state correction is recommended when it usefully proves the boundary, but it is not ceremony that must be invented for a simple deterministic source defect. A full clean environment is likewise reserved for dependency, packaging, persistent-state, platform, integration, promotion, or release failures where environment state can change the result.

Escalate to the complete process below when the compact path becomes inconclusive or the impact crosses the development boundary in development-lifecycle.md.

Prevent late repairable failures

A repairable checkout prerequisite first discovered after dependency installation, signing, compilation, or package generation is a defect in the setup boundary even when the downstream validator reports the facts correctly. Examples include unresolved Git LFS pointers, a missing supported tool, an unsafe checkout path, or a known checkout migration.

For that class of failure:

1. preserve the downstream error and identify the earliest point at which the prerequisite could have been proven;
2. manually correct that one prerequisite and rerun the original operation to prove the causal boundary;
3. put the permanent inspection or repair in the canonical repository-root setup before expensive work begins;
4. make every interactive first-run or missing-build launcher converge on that setup;
5. retain a downstream integrity check only when it verifies a distinct changed artifact state or trust boundary; reuse immutable hash-bound evidence for unchanged bytes, policy, identity, location, and toolchain instead of repeating an identical check.

Do not fix one named asset when the verified cause applies to an inventory, and do not move an application-specific verifier into Arcane OS. Arcane OS validates only the generic published ARCANE_APP_RELEASE.json admission and inventory contract; the owning application diagnoses and verifies its source, adapters, content, and behavior.

Diagnose performance and apparent hangs

A setup, build, package, test, model, app, or runtime path that appears hung or takes unexpectedly long is a debugging problem, not merely a progress-display problem. Complete the work-amplification review while isolating it. Preserve a representative cold-path timeline and measure the actual operation count, bytes transferred or processed, elapsed time, CPU/disk/network activity, and emitted progress events. State the cardinality and loop multiplier, then classify each expensive operation as invariant, identity-bound per item, or batchable.

Find the first boundary that repeats work without a changing input or identity. Hoist or safely cache invariant downloads, extraction, compilation, hashing, signing, integrity scans, model pulls, service setup, and tests; batch compatible work. Reuse only output whose complete source, toolchain, platform, policy, configuration, signing, and destination identity is verified. Apply one check per trust fact per artifact state: unchanged bytes, policy, identity, location, and toolchain reuse immutable hash-bound evidence, while placement/inventory-only changes verify only that changed boundary. Arcane OS-owned artifacts must not repeat per app, and app-owned compilation or behavior tests must remain in that app's release process.

For a Microsoft NT WebView2 page whose Network panel shows a long Stalled or connection-start phase before an otherwise immediate local response, inspect the virtual-host name before attributing the delay to hashing or package verification. Microsoft documents that a .local top-level domain can delay SetVirtualHostNameToFolderMapping navigation. Arcane's packaged host is https://os.arcane; the legacy https://arcane.local name is not an accepted test configuration. Measure the native executable, not only a development web server. The current host does not migrate legacy origin storage, so preserve the WebView2 profile untouched and expect https://os.arcane to have separate origin-scoped state.

Eliminate redundant scans and operations before adding a spinner, percentage, or heartbeat. Then preserve streaming phase, current item, completed/total count or bytes, elapsed time, throughput when meaningful, and an honest ETA for the necessary work that remains. Record every intentionally repeated expensive operation or check, proof that the underlying state changed or an independent verification is explicitly mandated, the multiplier, why it cannot be batched or reused safely, and its measured cost. Generic defense in depth does not justify repeating an identical check.

Diagnose the verification receipt lifecycle

Find the single verification owner and trace the immutable receipt through every downstream consumer. Confirm that it binds canonical location and filesystem identity, content and inventory hashes, policy, toolchain, platform and architecture, signing identity and trust result, and a receipt generation. Repeated hashing, signature checks, publisher checks, installed-tree walks, or catalog verification against that unchanged state are defects; generic defense in depth is not an exception.

Before treating a matching digest as a security fact, identify who supplied the expected value and whether the payload-replacement attacker can also rewrite it. Adjacent manifests, generated recovery records, same-origin catalogs, checksum sidecars, and digests fetched from the same publication authority are consistency evidence unless a separate signature, trusted binding, or authenticated retained-state owner anchors them. Recomputing both payload and expected digest is the required adversarial reproduction for this class of defect. A successful coherent rewrite proves the hash is not an authenticity boundary; retain it only when its corruption/drift purpose justifies the cost and label that purpose accurately.

Confirm that the owner invalidates the receipt before any bound write, rename, activation, repair, rollback, replacement, identity, location, policy, toolchain, signer, inventory, or generation change. It must stop new consumers, drain or cancel active consumers, close retained file and directory handles only at the mutation boundary, and issue a new receipt only after verifying the new state. A path, timestamp, size, time-to-live, or process-local cache key alone does not prove unchanged state.

For cross-process reuse on Microsoft NT, verify a shared Arcane host, broker, or service retains the verified handles, authenticates pipe peers through kernel process identity, and binds each delegation to the consumer process and receipt generation. Environment variables, command-line claims, unauthenticated pipes, and an administrator-protected receipt file alone are insufficient authority. Record UI or CLI acknowledgement before receipt acquisition or cache-miss verification and preserve the owner's progress, heartbeat, and surfaced failure.

Diagnose the responsiveness invariant

Record timestamps for request receipt, first visible UI or CLI acknowledgement, worker start, first progress event, each heartbeat, cancellation request, and completion or failure. A process existing in Task Manager is not proof that its interface acknowledged or remained responsive.

Inspect the foreground call chain for synchronous network, disk, hashing, compilation, process, model, or integrity waits before UI creation or between event-loop turns. An async/await function that invokes blocking work synchronously has not isolated that work. Verify that CPU-bound or blocking APIs run in an event-driven background worker, genuine asynchronous I/O stays asynchronous through the stack, concurrency is bounded with backpressure, and every task has an owner that observes its error.

Reproduce with a deliberately slow but bounded dependency. Verify that the UI can paint and accept input or that the CLI emits status immediately; progress or heartbeat events remain live; cancellation reaches a documented safe state; retry/resume uses the correct identity; and worker failure becomes visible and produces a nonzero result. Serialize only the isolated trust or transaction boundary, and prove visible status exists before it begins. Unobserved fire-and-forget work is prohibited.

The required process

Follow this order:

Reproduce -> Preserve -> Inspect -> Isolate -> Manually fix -> Manually verify -> Fix the code -> Rebuild -> Retest from a clean state

Do not skip ahead because a likely solution seems obvious.

1. Reproduce

Perform the exact action that is reported to fail.

Record:

• the precise action, input, command, route, or workflow;
• the observed result, including exact error identity where available;
• the claimed expected result;
• the app, component, module, entity, version, host, browser, configuration, and data involved;
• whether the result is consistent, intermittent, timing-dependent, or environment-dependent.

Do not improve, simplify, or substitute a different reproduction before proving the original one. If it cannot be reproduced, say so and improve observability; do not make a speculative fix.

Gate: The original behavior has been observed, or the exact reason it cannot currently be observed is recorded.

2. Preserve

Preserve the evidence and failing state before an action can erase or alter them.

As applicable, save:

• the relevant error, log, screenshot, response, stack, state, or data;
• the exact version and configuration;
• cache, storage, process, network, package, and service state;
• the smallest useful before-state for later comparison;
• unrelated user work in the worktree.

Do not clear a cache, restart a service, reinstall a dependency, rebuild a package, migrate data, delete a record, or reset the environment until the original state is captured. Those actions are experiments, not housekeeping.

Gate: A later reviewer can tell what failed before the investigation changed it.

3. Inspect

Inspect the system as it exists. Prefer read-only checks first.

Verify:

• what code and assets are actually running, not merely what is on disk;
• the input entering the failing operation;
• the output leaving it;
• state immediately before and after it;
• the applicable requirement, interface contract, design decision, or user instruction;
• whether the expectation is correct for this version, environment, and input.

Label every statement as one of:

• Observed fact — directly seen or measured;
• Verified expectation — supported by a requirement, contract, design, or explicit user instruction;
• Question — not yet known;
• Possible explanation — something to test, never a fact.

Do not convert a possible explanation into a conclusion through confident wording.

Gate: The observed behavior and the verified expectation are both known. If the expectation was wrong, correct the expectation or documentation and stop treating the implementation as defective.

4. Isolate

Find the first boundary or step where the observed state differs from the verified expectation.

Move through the workflow one boundary at a time. At each boundary:

1. identify the expected input and output;
2. observe the actual input and output;
3. change or test one independent variable only;
4. inspect and record the result before continuing.

Useful boundaries include user action, event, component, module, entity, storage layer, request, response, service, host bridge, package, and cache.

Do not apply several plausible fixes together. Do not refactor, upgrade dependencies, reset state, and change configuration in the same experiment. A result with several changed variables does not establish which change mattered.

Gate: The first incorrect transition is identified closely enough that a manual correction can test the expected outcome.

5. Manually fix

Before changing the implementation, manually place the isolated boundary into the state the system was expected to produce, when safe and practical.

Examples include supplying the verified input, correcting one record in a disposable copy, invoking the expected operation directly, loading the known-current asset, or performing the missing transition through a diagnostic interface.

The manual correction must be:

• narrow;
• reversible or performed on disposable data;
• recorded exactly;
• limited to the isolated variable.

This is a diagnostic experiment, not the final repair. If a manual correction would be destructive, unsafe, affect real users, or require new authority, stop and request direction.

Gate: One controlled manual change has been made and nothing unrelated was changed with it.

6. Manually verify

Repeat the affected step with the manual correction in place.

Compare the result to both:

• the preserved failing baseline; and
• the verified expectation.

Classify the result:

• Proven useful: the exact failure is corrected and the predicted downstream behavior occurs;
• Disproven: the exact failure remains or the predicted behavior does not occur;
• Inconclusive: the result cannot distinguish the proposed mechanism from another cause.

If disproven or inconclusive, undo the diagnostic change when appropriate, return to Inspect or Isolate, and take one new step. Do not leave a failed experiment in place while stacking another one on top.

Gate: The manual correction has demonstrated what the implementation must do. A temporary disappearance after a restart, retry, delay, cache clear, or broad rebuild is not enough.

7. Fix the code

Only after the manual correction is verified, change the implementation to produce that same proven result.

The fix must:

• address the isolated cause rather than hide the symptom;
• be the smallest coherent correction;
• preserve unrelated behavior and user work;
• include a focused regression check where practical;
• name every new or modified function and callback so failure stacks identify the operation and its exit condition;
• avoid unrelated cleanup, formatting, refactoring, or dependency changes;
• follow the Arcane app-building SOP when it adds or changes a reusable capability.

If the request was diagnosis-only, stop before this step and report the proven cause.

Gate: The code change can be explained directly from the manually verified correction.

8. Rebuild

Rebuild, repackage, restart, or invalidate caches only to the extent required for the changed implementation to become the implementation under test.

Verify which artifact, version, process, module, or cached resource is actually running afterward. A successful build proves only that the build completed; it does not prove the bug is fixed.

Gate: The test environment is known to be running the corrected output.

9. Retest from a clean state

Return to a clean, representative starting state and repeat the original reproduction exactly.

Then verify in this order:

1. the original reproduction;
2. the focused regression or contract check;
3. the nearest integration, browser, runtime, host, or package path;
4. broader checks proportional to the risk.

A task is not complete because a new test passes while the original workflow remains untested. Record any check that could not be performed and why.

Gate: The original observed failure no longer occurs, the verified expectation is met, and the result is attributable to the supported fix.

Compact debugging record

Humans and artificial-intelligence (AI) agents must maintain this record during an investigation:

### ASS-U-ME debugging record

**Reported behavior**
- Conditions/action:
- Observed result:
- Claimed expectation:
- Verified basis for expectation:

**Reproduce**
- Exact reproduction:
- Result:

**Preserve**
- Evidence and failing state saved:

**Inspect**
- Observed facts:
- Verified expectations:
- Questions:
- Possible explanation under test:
- Cardinality and loop multiplier:
- Expensive-operation count/bytes/time:
- Invariant / identity-bound per item / batchable classification:
- Request-to-acknowledgement time and foreground call chain:
- Background owner, concurrency/backpressure, progress/heartbeat, and cancellation path:

**Isolate**
- Boundary:
- One variable tested:
- Result:

**Manual correction**
- One change:
- Manual verification result:
- Classification: Proven useful / Disproven / Inconclusive

**Implementation**
- Root cause:
- Code fix:
- Invariant work hoisted/cached and justified repeated work:
- Responsiveness fix and serialized trust/transaction boundary:
- Rebuild/current artifact proof:

**Clean-state retest**
- Original reproduction after fix:
- Focused regression check:
- Broader checks:
- Remaining uncertainty:

Rules for AI agents

An AI agent must:

• select the rapid-development path or complete high-assurance sequence before beginning diagnostic or fix work;
• use the applicable sequence in this document rather than stacking generic hypotheses;
• never present an assumption as a fact;
• verify the expectation before deciding that observed behavior is a defect;
• state the current step in meaningful progress updates;
• perform only one state-changing diagnostic action at a time;
• inspect its result before choosing the next action;
• preserve unrelated user changes;
• stop before implementation when the request is diagnosis-only;
• report the evidence, root cause, fix, and clean-state retest in the final handoff.

The agent may continue through safe, in-scope steps without pausing for approval after every step. “One step at a time” means one interpretable action at a time, not unnecessary conversational pauses.

Prohibited behavior

Do not:

• begin with “the cause is probably...” and change code to match the guess;
• accept the reported expectation without verifying its basis;
• use the proposed fix as the problem statement;
• make a code change before reproducing and preserving the failure when that is safely possible;
• change multiple independent variables in one experiment;
• erase the original evidence before capturing it;
• treat correlation, timing, or a disappearing symptom as root cause;
• change a test to accept behavior that contradicts the verified requirement;
• weaken security, validation, or error handling to hide a failure;
• mistake a self-authored or same-authority hash for authentication, or regenerate it to bless untrusted bytes;
• suppress an error, ignore an exit code, add an empty catch, or retry forever;
• claim success from compilation or an unrelated test;
• leave temporary logging, bypasses, credentials, unsafe flags, or diagnostic data behind.

Urgent containment

If the failure is actively causing data loss, security exposure, destructive behavior, notification storms, or widespread outage, contain the harm first with the smallest reversible action. Record what was changed, preserve whatever evidence remains, and then begin the normal process at Reproduce.

Containment stops harm. It does not establish the cause or complete the fix.

Completion test

A debugging task is complete only when a reviewer can trace a straight line from:

1. the exact observed result;
2. the verified expectation;
3. the preserved failing state;
4. the first isolated incorrect transition;
5. the manually proven correction;
6. the code change that implements that correction;
7. the original reproduction passing from a clean state.

If that chain contains an assumption, the investigation is not complete.