xr-foundation-model

01Why a beginner builds a tiny GPT

Honest reason: my profile said "agentic AI" and my internals said "API caller." You can integrate a provider forever without ever knowing what an attention mask owes you. XRFM is my answer to that gap — not a model that competes with anything, but a complete, measured, reproducible training pipeline where I own every layer: tokenizer → embeddings → blocks → loop → checkpoints → eval → inference engine. A 30-perplexity model can teach you more about causality than a 90th-percentile API wrapper.

02Architecture

• Tokenizer: byte-level BPE written from scratch (tokenizer/bpe.py); the acceptance test is exactness — decode(encode(x)) == x across English, Unicode ("你好") and code.
• Model: pre-norm decoder blocks — RMSNorm, causal MHA with RoPE, SwiGLU FFN, weight tying; explicit causal masking (a lesson learned the hard way, see §04).
• Training: AdamW β=(0.9, 0.999), cosine schedule with warmup, grad accumulation + clip, seeded end-to-end, checkpoints that actually resume (step counter, optimizer, scheduler — verified continuity at step 120→240).
• Inference: KV-cached generation; scripts/evaluate_checkpoint.py can always reproduce output from a committed checkpoint.

03Numbers, with their method

• Corpus: 1.77M tokens of public-domain prose + code; line-boundary splits, dedup, pad-loss-masking so padding can't leak.
• Training smoke: loss 7.66 → 6.45 over 400 steps; final val perplexity 301.55; scaling runs 777.1 → 586.0.
• Tiny overfit test: 600 steps → train loss 0.097, greedy generation reproduces training text verbatim.
• Reproducibility: same seed → identical loss curve (test-enforced).

Perplexity ~301 sounds like a failure until you note the diet: 1.7M tokens. A GPT-2-small needs eight times that to be coherent. The number's purpose is to move predictably when I change something — which is exactly what an eval loop is for.

04The part I'm proudest of: auditing myself, publicly

Version 0.6 had bugs a stranger would find in ten minutes, so I became that stranger. I ran a forensic audit — docs/audit/FORENSIC_AUDIT.md — that logged 49 findings: a broken causal mask, fake bf16 (autocast without an autocast), non-resumable checkpoints, a vocab split-brain between tokenizer and embedding, toy data, a dead API import. Every finding got a fix and a ground-truth test; FINAL_AUDIT.md closes the checklist 18/18 with evidence links. The audit and its transcripts live in the repo — the narrative I want attached to my name is not "first try was right," it's "found every lie, including my own, and made each one a test."

class TestCausality:
    def _probe(self, attn):
        x1 = torch.randn(1, 6, 64)
        x2 = x1.clone()
        x2[0, 4] = 99.0            # perturb a FUTURE position
        o1, _ = attn(x1); o2, _ = attn(x2)
        return (o1[0, 2] - o2[0, 2]).abs().max().item(), \
               (o1[0, 5] - o2[0, 5]).abs().max().item()

    def test_default_path_is_causal(self):
        attn = MultiHeadAttention(d_model=64, n_heads=4, dropout=0.0)
        d2, d5 = self._probe(attn)
        assert d2 == 0.0, "position 2 must not depend on position 4"
        assert d5 > 0.0, "position 5 must depend on position 4"

05Evidence committed, not described

The 76.8 MB trained checkpoint is in the repo tree at checkpoints/ alongside the 5.5 MB corpus — clumsy Git hygiene, deliberate evidentiary discipline. Anyone can load it, run evaluate_checkpoint.py, and get the same completions I claim. That sentence is the entire portfolio thesis in two lines, and this project is where I proved I mean it for ML internals, not just product code.

06Lessons I keep from it

• A number without its method is a vibe. (Every metric above names its script or doc.)
• Shape-tests prove plumbing; only reference-math tests prove semantics. My suite learned both.
• Resume-training continuity is the cheapest deep bug-finder there is.
• Publicly documenting your own 49 failures buys more trust than any polished success story.