Update #3Oct 19–25Development
Connecting update feedback and a local research assistant, while keeping failed gates visible
CPU feedback, recovery and task grading advanced; a three-arm comparison found no FULL advantage. Real Ornith powered a read-only research assistant. Retrieval repair improved source coverage at higher cost. After a RAM stop, native 9B restored and resaved older custom state, without reaching research or updates. Autonomy, complete migration and new-task gains remain unverified.
W43 / RESEARCH ATLAS
Return each update outcome to the next inquiry.
Six figures map the engineering connections, capability claims still unproven and the next gates to test.
What reaches the research loop after an update?
First return the actual outcome. Model-driven correction still needs its own test.
Keep the update; do not repeat it
An observation error does not turn a committed update into an uncommitted one or rerun its body.
Restore declared state; retain failure
Declared model, optimizer and RNG state are restored. External I/O and arbitrary Python heap state are outside this evidence.
Observation-on, observation-off and control runs matched raw values at the declared boundaries.
Evidence boundarySpecified CPU paths and host-injected faults; not full 9B/CUDA, diagnosis understanding or autonomous correction.
The complete system has not shown an advantage
Three arms tackled bounded coding tasks, retaining low scores, generation failures and fixed denominators.
Swipe horizontally to see the full chart, or open the data table below.
Read the chart data
| Arm | Condition | Held-out accuracy | Committed updates |
|---|---|---|---|
| FULL | Full system | 2.08% | 0 |
| M0 | No cross-episode memory | 5.99% | 0 |
| W0 | No committed updates | 2.08% | 0 |
Evidence boundaryBounded synthetic coding tasks that did not exercise the complete native research loop; not a general-autonomy or scientific-discovery test.
Finding a symbol is only the start
Retrieval repair recovered more facts. Only one task was fully correct, and costs rose.
complete fact items
complete fact items
Required-source coverage: 4/5 → 5/5. Another 81 extra or interrupted cases remain separate.
Read the chart data
| Metric | Earlier | Repaired |
|---|---|---|
| Complete facts | 1/12 | 7/12 |
| Fully correct | 0/4 | 1/4 |
| Tokens | 12194 | 26626 |
| Seconds | 211.703 | 363.577 |
Evidence boundaryFour exposed source-reading tasks; not a preregistered eight-case, unseen-task or single-factor causal study. Displayed costs exclude the rest of development.
Native 9B reached bounded restore and resave
Keep the memory stop on record, then distinguish the later restore from the steps still not reached.
- 1Initial RAM-guard stopFAIL RETAINED
Custom checkpoint had not yet restored
- 2Older CPU custom state restoredBOUNDED PROGRESS
22 state categories checked; 41 research records retained
- 3Unchanged state saved separatelyBOUNDED PROGRESS
PARTIAL_RESTORE_SAVE_ONLY
- 4Cold-load the new save and reach research updatesUNVERIFIED
Cold load: not run; research: not reached; update: not reached
Strict source inventory: failed. The inventory difference remains unresolved.
Evidence boundaryRestore/resave of older native CPU custom state; not acceptance of the latest CUDA checkpoint, complete source closure or the next research update.
Completed, identical and correct are different claims
Method experiments and basic-answer checks retain separate scopes; different populations are not pooled into a capability score.
Rule comparison within a fixed language
preregistered predictions held
Clock and time-limit predictions failed. Counting always abstained; period-seven answered errors remained near half.
Identical answers can remain incorrect
prompts × states, with identical generated tokens
Evidence boundaryLedger compared author-revised programs in a fixed H/P language. The answer check covered six prompts and three tested adapter states. Neither establishes general autonomy, 9B/True preservation or persistent transfer.
Which open-source tools connect each part?
Full GitHub repository URLs are visible, with executed tools, dependencies and workflow references distinguished.
Read-only boundary: sources and responses can be checked; no generated-code execution, method adoption or native learner updates.
CodeGraph
1.6.2Used the CLI and MCP interface to index sources, locate symbols and trace update, observation and restore connections. Missing and false edges still require source checks.
MIT · upstream license ↗Instructor
1.17.0Produced structured research reviews and retrieval decisions from local Ornith responses. Format and meaning were checked separately; no execution or learner-update authority was granted.
MIT · upstream license ↗llama.cpp
b11669Loaded Ornith GGUF with llama-server for local CPU/Vulkan HTTP inference. This served the external quantized assistant, without porting the native Xiaomai learner or custom checkpoints.
MIT · upstream license ↗ECC
ecc-universal 2.2.3Its iterative-retrieval workflow informed the local staged-retrieval, gap-checking and follow-up implementation through CodeGraph and Instructor. The complete ECC package and global hooks were not installed; this is not parameter learning.
MIT · upstream license ↗Pydantic
2.14.1Provided strict type and field validation for the Instructor integration. A valid structure does not establish correct content.
MIT · upstream license ↗OpenAI Python SDK
3.3.0Called the local OpenAI-compatible HTTP endpoint through Instructor. This does not mean these experiments used an OpenAI cloud model.
Apache-2.0 · upstream license ↗Ornith is credited separately as a model source: ornith-ai/Ornith-1.5-9B-GGUF
Evidence boundaryECC informed a local workflow adaptation; its complete package and global hooks were not installed. The SDK called a local compatible endpoint. This list is not a full software bill of materials.
Continue to the full weekly account for the daily context, original failures and technical limits.
What we got done
Xiaomai studies whether a system can turn observations and contradictions into questions, propose methods, test whether a change helps, and retain useful understanding across restarts and new problems. This direction began with VeriTrade's questions about adaptation and model failure. This week connected one part of that loop: the next research input must receive what actually happened during an update, including success, failed writes, and failures in reporting after a write had already committed.
I connected those outcomes to the bounded CPU small-model research path, with identifiers that lead back to the original events. An update that committed before its observation report failed must not run twice. A failed write must distinguish which declared state was restored. The specified paths now have evidence for these cases, plus new-process restore and duplicate-update prevention. This gives the next inquiry usable facts. It does not establish that the model understands the diagnosis or repairs the problem itself.
Work progressed across the week. Oct 19 joined formal updates, saves, continuation in a different process and SQL post-commit retries. Oct 21 separated successful program execution from correct task answers and repaired a fingerprint that could falsely reject a version after cold restore. A complete second research round in a tiny fixture matched its cold continuation for the declared persistent state. These are better foundations for comparison, within their specified CPU paths.
Observation must also leave the compared computation intact. Across observation-on, observation-off and control runs, the declared CPU scope retained identical bytes for 64 events and 6,082 FP32 values. The checks cover specified forward, backward and update boundaries, rather than every operation in a 9B model. Saving, rollback and source rejection each have separate scopes. Their counts are not added into an overall capability score.
A separate research line asked whether a rule that passes immediate validation remains reliable when answering further questions. On Oct 20, the ledger v5/v6 tested programs compared rule adaptation within a fixed representation language on fresh instances. v6 answered all one-step-state, delayed-input and two-step-output cases correctly, but abstained on every counting case. Period-seven tasks still had nearly half of answered predictions wrong, including cases marked validated. Only four of six preregistered predictions held. These are author-revised programs, not evidence that Xiaomai invented a new representation, but they preserve useful counterexamples for the next investigation.
We also completed a three-arm comparison and retained the disappointing outcome. Across eight new synthetic worlds, FULL, no cross-episode memory and no committed updates attempted the same kinds of coding tasks in 24 sessions and 96 episodes. FULL did not meet its preregistered advantage criteria. More complete engineering connections do not turn those scores into evidence of improved capability.
Over the weekend, a real Ornith model was connected to a local read-only research assistant. CodeGraph locates source material, while Instructor checks response structure. This lets a real model help read code and cite sources. Its external answers remain research material: they do not execute generated code, adopt methods or invoke Xiaomai's updates. Format acceptance and semantic correctness are judged separately, and incomplete original answers remain on record.
Retrieval then addressed a short-context problem: locating a symbol without finding all its dependencies. The first ECC-inspired iterative attempt matched single-round retrieval at just 1/4 fully correct answers while taking more time and tokens. A repair added cross-file dependency material and expanded truncated definitions. In a four-task paired block, the old method scored 0/4 fully correct and the repair 1/4; complete fact items rose from 1/12 to 7/12. This is useful local progress at higher cost, not sufficient evidence to make it the universal default.
The main open-source tools and workflow references used this week are listed below. Each has its own authors and license. This week's Xiaomai work connected them to the specified research tasks. Their use as tools, dependencies or workflow references is identified below.
- CodeGraph v1.6.2, under MIT: we used its CLI and MCP interface to inspect sources, symbols and call relationships around updates and restoration. Missing and false edges still require source checks.
- Instructor 1.17.0, under MIT: it turned local-model responses into structured reviews and retrieval decisions. Format validation and content judgments remain separate; this does not execute generated code or update the native learner.
- llama.cpp b11669, under MIT: llama-server loaded Ornith GGUF for local CPU/Vulkan inference. This served the external quantized assistant without replacing native Xiaomai custom state or research updates.
- ECC, labeled ecc-universal 2.2.3, under MIT: its pinned iterative-retrieval workflow informed our local staged retrieval, gap checks and follow-up reading. We did not install the complete ECC package or enable global hooks. This is not Xiaomai parameter learning.
The integration also used Pydantic 2.14.1 for strict schema validation, under MIT, and OpenAI Python SDK 3.3.0 as a local HTTP client dependency, under Apache-2.0. The SDK called a local compatible endpoint in these experiments; it does not imply use of an OpenAI cloud model. The official Ornith model source is credited separately from GitHub tools. Versions and usage roles are also recorded in this issue's public evidence summary.
What didn't work
Mean held-out accuracy was 2.08%, 5.99% and 2.08% for the three arms respectively. More importantly, every arm committed zero formal parameter updates. No training proposal met the existing requirement to be both valid and fully correct, so the update gate never opened. The experiment therefore did not measure the benefit of successful updates. It does not establish that learning is useless or that memory should be removed. These were bounded synthetic coding tasks; they did not exercise the complete original native research loop or test general autonomous research and scientific discovery.
I also investigated failures on basic questions. The same 0.5B model produced identical generated answers on six fixed prompts with no Xiaomai output adapter, the initial adapter, and an adapter after two public control updates. The unchanged baseline already failed two arithmetic questions. All three states hit the 64-token limit on the coding question. This supports no additional visible answer regression from the tested adapters on those six prompts. It does not establish preserved capability in full Xiaomai, the original 9B model or True, nor does it identify the base model as the sole cause of low scores.
Running the external quantized model did not directly solve native Xiaomai restore. The first Oct 25 native 9B attempt mapped the original BF16 base but hit a RAM guard during ancestor fingerprinting, before loading the custom checkpoint. A later streaming-hash overlay reduced that step's extra memory demand and restored an older native 9B CPU custom state, retaining 41 research records. It then saved the unchanged state. This advances restore and save only: research and parameter updates were not reached, and the new checkpoint has not been cold-loaded by another process.
Performance also depends on the path. A short external-GGUF inference comparison improved after confirming actual GPU loading. In contrast, one paired four-token feature comparison on the restored native custom 9B model took longer on GPU, with substantial per-matrix weight transfers. These paths perform different work. The assistant's token throughput is not the speed of complete Xiaomai research.
A strict source check still failed: the old manifest listed 115 sources, while the new save listed 104. Complete source-closure agreement is therefore unestablished. Complete decoder-gradient localization of the first non-finite value also remains partial. The earlier Oct 14 multilayer FP32 result of 0/14 passes and negative controls of 104/112 retain their original conclusions. This week's passes under different contracts do not erase them.
The retrieval repair used four previously exposed source-reading tasks in one descriptive comparison. An initially larger repeated experiment was shortened to the first completed paired block. That selection was fixed before anonymous scoring, but was not a preregistered eight-case protocol. Another 81 extra or interrupted cases are retained separately. The repaired block increased workflow time from 211.703 to 363.577 seconds and tokens from 12,194 to 26,626. These costs cover that completed block, not the entire development effort.
What's next
The next engineering unit should resolve the new save's source-inventory difference and cold load, then join native research, the next update and saving under one explicit acceptance scope. Source, state and environment mismatches must retain their rejection reasons. Continue with small implementations and limited old-method controls aimed at observed bottlenecks and source-reading gaps. Basic answers also need a same-prompt, same-budget comparison between the native path, the selected modified state and the complete Xiaomai flow, so that model, adapter and controller effects can be separated.
Capability requires a separate experiment. Fresh tests should expose a falsifiable mistaken assumption and observe whether the system proposes competing explanations, selects a discriminating experiment, saves a revision and retains a benefit on unseen instances. Compare the original method, the system's revision and a control with no substantive revision. Fix success criteria, failure criteria and resource limits before the run. This week's formal samples remain frozen.
This issue covers Oct 19–25, with reviewed evidence through Oct 25. Editorial work did not rerun the model. Complete cores and private model assets remain private. The public summary exposes results and limits; source hashes support later byte comparison, not standalone proof of capability.
Where Xiaomai stands
The four research steps from the home page and where each stands this week.
Start from unknowns. Examine how inquiry works.
Not yet tested
The bounded coding comparison does not establish autonomous reframing.
Check explanations against actual computation.
Partly done
Declared CPU observations are checkable; complete gradient fault localization is unfinished.
Turn new understanding into a change that runs.
Partly done
Actual update outcomes reach research inputs; model-driven correction remains unverified.
Select and reuse the change after a restart.
Not yet tested
Older native 9B state was restored/resaved; new-checkpoint cold load and new-task benefit remain unverified.
Technical recordFull figures, scope limits and the daily log, for researchers to check
Daily log
- cold-and-postcommitSave/continuation and SQL post-commit retry; old numerical gates still failPartly passed
- ledger-v6Ledger: four of six predictions met; clock and time-limit gates failPartly passed
- verifier-cold-fixTask grading, fingerprint repair and bounded second-round cold continuationPartly passed
- three-arm-result24-session comparison did not meet FULL advantage criteriaDid not pass
- zero-formal-updatesAll formal updates were zero; failures and checker corrections retainedRecord
- feedback-and-restoreBounded CPU feedback, observation, rollback and cold restorePartly passed
- answer-healthSix answer checks retained baseline errors and truncationPartly passed
- codegraph-toolsCodeGraph queries on fixed sources; known missing edges retainedRecord
- migration-boundaryMigration preconditions checked; formal old-9B migration blockedPartly passed
- ornith-assistantReal Ornith and Instructor connected to a read-only research assistantPartly passed
- retrieval-repairFour-task retrieval repair improves some facts at higher time/token costPartly passed
- native-restore-saveAfter RAM stop, bounded restore/resave of older native 9B custom statePartly passed
Completed this week
CPU update diagnostics carry update identity, before/after versions, commit state and outcome facts into the next research input. In the specified provider path, an observation error after commit does not repeat the update. Fault injection after actual writes checks restoration of the declared model, master, optimizer, gradients, buffers and RNG state. These are host-injected cases, not comprehensive coverage of natural failures.
Saved facts from same-source schema3 small models were restored, queried and used by the next proposal in a new process. Replaying already-applied evidence does not update values again; failed-and-consumed events do not restart. Old checkpoints from different sources remain rejected. Source guards were not waived to obtain a pass.
Oct 19 SQL post-commit faults checked hot retries and cross-process retries after saving without another update. Journals and some episode files still require external material and original paths; this is not a self-contained cross-host package. Oct 21 task grading preserved zero reward for completed-but-wrong execution and removed only specified equivalent compilation/token-encoding rework. A false cold-restore policy rejection came from unstable marshal bytes for the same code. A separate source epoch replaced them with generic semantic-structure fingerprints while retaining rejection of actual code or contract changes. Tiny-model second-round research and cold continuation matched declared persistent state, rather than arbitrary Python process memory.
CodeGraph CLI and MCP queries were used on fixed sources to help engineering authors locate update, observation and restore connections. Known missing dynamic-call edges and same-name false edges remain, so query results require source checks. This is engineering-tool integration, not evidence of Xiaomai understanding itself or a measured efficiency improvement.
On Oct 25, Ornith-1.5-9B-Q4_K_M, llama.cpp and Instructor produced real local responses while the external assistant stayed read-only. ECC informed iterative source location, retrieval and answering; no global hooks were installed. Its developer-agent rule accumulation is not Xiaomai parameter learning. Retrieval repair retained both old and new entry points without rewriting the original core.
The low-memory native alternative used an overlay for streaming tensor-digest computation while leaving the original core unchanged. Guards still reject undeclared migration, additional source mismatches and incompatible Torch versions. The original 9B custom state was actually restored; 22 pre-save state categories matched the original reference, retaining 41 records and existing learner steps/revision. The new save contains unchanged state, with no invented research or parameter change.
Validation and figures
- The observation comparison covers 64 events and 6,082 FP32 values, or 24,328 bytes, at declared boundaries. It includes specified three-layer forward computation, complete small logits outputs, and selected VJP, loss, parameter-gradient and update comparisons. It does not mean 64 kinds of boundary or complete primitive coverage.
- Eight new synthetic worlds each supplied two training and two held-out episodes per arm, giving 24 sessions, 96 episodes and 48 held-out episodes. Accuracy is averaged within each world, then equally across eight worlds. Invalid programs, refusals and preregistered generation failures remain in the fixed denominator.
- FULL, M0 and W0 mean held-out accuracy was 2.08%, 5.99% and 2.08%; all formal committed-update counts were zero. FULL's world-level win/loss/tie counts were 1/2/5 against M0 and 0/0/8 against W0. Neither preregistered positive-advantage criterion passed.
- Separate public mechanism controls made FULL commit two actual updates while W0 performed shadow backward with no commit. These controls are excluded from the formal three-arm scores and do not fill the absence of formal-task updates.
- The answer comparison used the same Qwen2.5-Coder-0.5B-Instruct model, CPU FP32, greedy decoding and a maximum of 64 new tokens per prompt. Three fresh processes ran once each. The initial adapter matched the baseline's first-step logits, generated tokens and answers on all six prompts. After two public control updates, logits changed while generated tokens and answers on the six prompts remained identical.
- Ledger v5 completed all 280 formal sessions. v6 completed 279 of 280 fresh-instance sessions with one timeout; four of six preregistered predictions held, while Q3 and Q6 failed. Unmodified v5 was rerun on the same fresh worlds. Counting abstentions satisfy Q4's conservative-answer branch, not learned counting. Period-seven answered-error rates were about 48–50%; missing timeout predictions stay separate. Scope is adaptation in the author's fixed H/P language, without main-loop integration, save/restart or new representation types.
- The initial ECC comparison gave both single-round and iterative retrieval 1/4 fully correct answers, with higher time and token cost for iteration. The repaired paired block used the same model, 8192 context, temperature 0/seed 42 and 768 final-token limit. Each arm's 11 calls count toward cost. Old/new full correctness was 0/4 to 1/4, complete facts 1/12 to 7/12 and required-source coverage 4/5 to 5/5. Workflow time was 211.703 to 363.577 seconds and tokens 12,194 to 26,626. All 22 responses ended with stop, but incomplete facts remain.
- The external quantized-inference A/B/A short comparison fixed generation at 128 tokens, with actual offload of 28/34, 34/34 and 28/34 layers and identical request/output bytes. Server-reported output throughput was 12.298, 31.547 and 11.693 tokens/s. Only one B sample was taken, with different background load and free RAM. There is no five-run or stable end-to-end speed conclusion. The planned length termination measures throughput, not successful completion of a research answer.
- The restored native 9B four-token prefix compared only 1×32 FP32 features: CPU 24.641 seconds and GPU 30.734 seconds, one CPU-to-GPU pair with two threads. Relative RMSE was about 0.002104 and maximum absolute difference about 0.004307, within preregistered tolerances but not bitwise equal. This is not token generation speed. GPU execution moved about 13.833 GB of weights without a separate copy/compute profile; the entire time difference cannot be attributed to one hardware component.
Failures and unfinished work
The Oct 23 independent acceptance matrix did not pass in full. N3 is PARTIAL: localization covers the observed domain, while complete decoder-gradient coverage and natural joint failures remain untested. M3 was then PARTIAL_BLOCKED_TARGET. Oct 25 later restored an older native 9B state and saved it unchanged, but the source inventory changed from 115 to 104 entries and the strict audit remains false. Missing entries do not mean 11 files were modified, nor establish complete source closure or cold loading of the new save. Complete research/update/save continuation was not reached. Windows saves retain COMMITTED_DURABILITY_UNCONFIRMED; physical power-loss durability is unestablished.
The retrieval block was selected before anonymous scoring, but the original five-round plan was reduced, so this is not a preregistered eight-case experiment. Another 81 extra or interrupted cases are retained separately. Some interrupted calls returned no usage, leaving total development cost partly unknown. Source strategies, prompts and schemas changed together, preventing attribution to one repair. Cache settings and other process loads also changed; earlier RAM pressure cannot be wholly attributed to caching or called a memory leak.
The three-arm source checker originally compared external arm names directly with internal names. A post-run correction addressed that mapping with additional checks; original failures, denominators, statistics and frozen samples remain unchanged. A separate auxiliary checker also retained its original skip-semantics error. These corrections do not mean the preregistered checker passed unchanged on its first attempt. FULL advantage remains unestablished.
All three coding answers in the basic-answer comparison failed to finish with EOS, and their three token-cap failures remain on record. A public self-test found a regex error in the independent scorer, which was corrected in a separate version before model results were read. The original FAIL remains. Its format field requires the whole response to equal the expected answer, so aggregate checks are not a pure format-compliance rate or capability pass rate.
Author and AI contributions
郭俊逸 owns research direction, problem framing, evidence boundaries and project integration. AI agents assisted implementation, experiment execution, source exploration, writing and result review. Different agents handled relevant implementation and verification, with original failures retained. The three-arm examiner and verifier were separate agents from the same model family, not external humans or cross-model third parties. This website update ran no new model experiment, and writing the account is not capability acceptance.
Next steps
Resolve the native save's source inventory and cold load, then advance the next research update through minimal implementation and limited old-method controls. Check basic-answer preservation across native, modified and complete paths. Ledger and retrieval retain their failure denominators; revisions use newly frozen cases. Preregister a separate capability experiment that actually exercises question formation, method revision, discriminating experiments and fresh-task comparison after saving. Complete cores, weights, private checkpoints, raw tensors and full logs remain private. This issue's public summary contains only selected observations, limits and source-hash commitments.