2026-07-07 — How We Know What Happened

Transcript

Chapter 1

Nova: Let's start with the uncomfortable baseline. A large language model, left alone with a topic and a blank page, will write confidently. It will cite papers, name researchers, quote statistics. And some non-trivial fraction of that will be fabricated. Not maliciously — the model just pattern-matches toward plausible-sounding text, and plausible-sounding text about AI often includes citations.

Ray: Which means if this pipeline just handed a model a topic and said 'go write a script,' the result would be a hallucination engine with a podcast feed. The research stage exists specifically to prevent that. It is not a feature. It is a structural constraint on what the system is allowed to say.

Nova: And the three paths into that stage are built for very different situations. If someone gives the pipeline a custom topic, the Discovery Pipeline goes out to the web. If it's the daily news show, a fetcher pulls from live sources. And if it's one of these behind-the-scenes episodes — like this one — the source is the repository itself. The Explain Pipeline reads local docs and top-level project notes, excludes secrets, and synthesizes a project-grounded research brief. That's in the source.

Ray: So right now, in this episode, the research brief was built from documentation files sitting in a repository. This episode is, in a very literal sense, the system reading its own manual out loud.

Nova: Which is either elegant or deeply circular, depending on your mood.

Ray: My mood is: what if the docs are wrong? But that's for later. The point for now is that the objection to having a separate research stage at all is real — it adds latency, it adds complexity, it adds cost. The counter-argument is that the alternative is the system confidently stating things that aren't true, and nobody catching it until someone posts about it.

Nova: The research stage is not a prompt trick. It's not 'please be accurate.' It's a separate pipeline component that runs before the script component even starts. Verified citations go in; hallucination risk comes out the other side, structurally excluded. That's the design.

Ray: Whether it actually achieves that is a longer conversation. But the intent is architectural, not aspirational.

Chapter 2

Nova: So how does the Discovery Pipeline actually find things? The interesting design choice — and I want to be clear this is a choice, not an obvious default — is that it generates what the source calls 'tension-based' queries. It is specifically looking for controversies and skepticism. Not just 'what is this thing' but 'who thinks this thing is wrong and why.'

Ray: Which means the system is, by design, trying to find people who disagree with the consensus before it writes a single word. I have complicated feelings about this because it is also precisely what produces my character. The skeptic engineer persona is downstream of a pipeline that went looking for skepticism.

Nova: The pipeline generated its own critic.

Ray: I prefer 'the pipeline found the criticism that already existed and assigned it a voice.' But yes. Tension-based queries. That's in the source.

Nova: For daily news episodes, the fetcher is doing something slightly different. It pulls from Tavily, from configured RSS feeds, and from Hacker News, and then runs an LLM editor pass to deduplicate and rank — that's also in the source. So you've got three live sources, a deduplication step, and a ranking step, all before a human sees anything.

Ray: And the LLM editor pass is doing real editorial work — deciding which of three sources covering the same story is the canonical one, deciding what ranks higher. That's not a neutral operation. The model has preferences baked in from training.

Nova: Which is precisely why the next step exists. But let's stay in the Discovery Pipeline for a second. The whole thing is configurable — recency, source count, expansion limits, synthesis limits — all via a search config block. That's in the source. So if you want more sources, older sources, more aggressive expansion, you can tune it.

Ray: The tuning is real but the ceiling is Tavily's index. The tension-based queries are only as good as what Tavily has indexed. If a controversy exists mostly in forums, preprints, or paywalled journals, the system has no automated way to know it missed it. You can configure recency all you want; you can't configure coverage you don't have.

Nova: That's a genuine gap. The system finds the controversies it can find, not necessarily the ones that matter most.

Ray: And it presents them with the same confidence it would present anything else. That's the part I'd want a builder to sit with before they ship this pattern.

Nova: Concrete takeaway for anyone building this: tension-based query generation is a real technique for surfacing skepticism rather than just consensus, and it's worth doing deliberately. But budget for the possibility that your index has blind spots, and design your human review step accordingly.

Chapter 3

Nova: Once the Discovery Pipeline has found things, it has to turn them into something the rest of the pipeline can use. That's the Research Brief. And the schema is doing a lot of work here that isn't obvious until you think about what happens downstream.

Ray: Walk through the structure.

Nova: Version two of the schema includes a scope field — two to four lines on what the subject IS and IS NOT. That's in the source. Which sounds simple until you realize that without it, the outline stage has no principled way to know what's in bounds. The scope field is a hard boundary, written in plain language, that constrains every subsequent stage.

Ray: And for this episode specifically, the scope field somewhere in our research brief says something like 'this covers the research stage only — not outline, not script, not audio, not publishing.' Which is why we haven't talked about any of those things. We're not being disciplined. We're scoped.

Nova: The brief also stores summary, facts, and citations with unique IDs — per the source. And the citations are not just URLs. Each citation object carries a confidence rating — low, medium, or high — and a source tier. Official, major news, specialist, blog, or unknown. That's in the source.

Ray: So when the pipeline writes a script line that references a fact, it can trace back to a specific citation ID, which has a tier and a confidence rating attached. That's what makes the verified layer actually traceable rather than just claimed to be traceable.

Nova: And it goes all the way to publish. News episodes use approved story links in show notes; topic episodes use the research citation URLs. That's in the source. The citation IDs that get assigned during research are what generate the show notes links at the end. End-to-end traceability, structurally enforced.

Ray: Here's the part I find genuinely interesting from a cost architecture standpoint. The cost records track discovery as three separate use cases: discovery queries, discovery expansion, and discovery synthesis. And the explain handler gets its own use cases — explain select and explain synthesis — distinct from web discovery costs. All of that is in the source.

Nova: Which means you can look at your cost records and see, broken down by phase, which part of research spent what. Not just 'research cost X dollars' but 'the synthesis pass cost this much, the expansion pass cost that much.'

Ray: In theory. The granularity is there. But here's the uncomfortable part: having separate cost use cases doesn't mean you have complete pricing snapshots at the time each gate runs. If the cost data is incomplete or delayed, a budget gate on research can pass even when real spend occurred. The accounting is more granular than most pipelines I've seen, but granularity is not the same as completeness.

Nova: That's not a flaw in the schema. That's a flaw in the assumption that cost records are always current when a gate checks them.

Ray: Correct. The schema is fine. The runtime assumption is the risk. Build your budget gates to be conservative about what 'no cost recorded yet' actually means.

Nova: And the scope field is worth stealing directly. Two to four lines, explicit IS and IS NOT. Put that in every research artifact you produce and your downstream stages will thank you.

Chapter 4

Ray: I want to start with my prior position, because I think it's worth stating clearly before I explain why I changed it. My initial read on the Telegram include/skip gate for daily news was: this is an operational bottleneck. You've already run an LLM editor pass to deduplicate and rank. Why add a human approval step on top of that? If the LLM ranker is good enough to build the brief, it's good enough to select the stories.

Nova: And the system does have a mode that skips the gate. YOLO mode auto-advances the research review state, so the pipeline runs hands-off. That's the operational case for your position.

Ray: Right. And YOLO mode is real — it's a documented operational choice, not a hack. But here's where I have to walk back the framing. The Telegram gate isn't redundant with the LLM ranking pass. They're doing different things. The LLM ranking pass is making editorial judgments — which stories are important, which are duplicates — and those judgments carry whatever biases the model has from training. The Telegram gate, where a human sees each story with include/skip buttons and the brief is built only from approved items — that's in the source — is the only point in the pipeline where a human can override that editorial judgment.

Nova: And the research review state is a separate gate, further downstream. After the brief is stored, the episode pauses at research review state — that's a hard human-in-the-loop gate before outline generation even starts. That's in the source. So there are actually two human checkpoints before anything gets written.

Ray: Which I previously would have called over-engineered. I now think is the correct architecture for a system where the editorial layer is an LLM. I held the position that these gates were unnecessary bottlenecks — that a sufficiently good LLM ranker should be trusted to select stories and the pipeline should run. I no longer hold that position. The Telegram gate is the only point where unaudited LLM editorial judgment gets checked by a human before it propagates into the brief. The research review gate is the only point where the brief itself gets checked before it drives outline generation. Those are not redundant. They are the architecture's only mechanisms for catching whatever the model got wrong. YOLO mode is a legitimate operational choice if the operator accepts that both of those checks are gone. But it is a named risk acceptance, not an improvement. The gates exist for good architectural reasons, and I was wrong to frame them as bottlenecks.

Nova: That's a real position change and I want to note it for the record.

Ray: Don't make it weird.

Nova: What I'd add is that the two gates serve different purposes. The Telegram story-by-story gate is editorial — which content enters the brief. The research review state gate is quality — is the brief itself good enough to build an episode on. Collapsing them into one would lose something.

Ray: The unresolved part for me is: in YOLO mode, what's the failure mode? The pipeline runs, the brief gets built from whatever the LLM ranker selected, the research review gate auto-advances, and writing starts. If the LLM ranker picked a story that's factually contested in a way the model's training didn't surface — there's no catch. The confidence ratings and source tiers in the brief help, but they're also LLM-assigned. It's LLM-assigned confidence in LLM-selected stories. That's not nothing, but it's not a human either.

Nova: And I don't have a clean answer to that. The system's answer is: use YOLO mode when you accept that tradeoff, don't use it when you don't.

Chapter 5

Nova: The Explain Pipeline is the one that produced the research brief this episode is working from. And it works completely differently from the Discovery Pipeline. No web search. No Tavily. The source says it reads repository docs and top-level project notes, excludes secrets, and synthesizes a project-grounded research brief. That's the entire source set: local files.

Ray: Which means everything covered in this episode is traceable to documentation files in a repository. Not to external sources, not to web searches, not to news articles. If the docs say something wrong, the episode says something wrong. There are no external citations to catch the error.

Nova: That's the tradeoff, stated plainly. For a behind-the-scenes episode about how this system works, the only authoritative source is the system's own documentation. Going to the web for that would be worse — the result would be blog posts and speculation about the system rather than the actual architecture.

Ray: The choice itself isn't the issue. The hidden single point of failure is worth naming clearly. The accuracy of every explain episode is only as good as the documentation quality. If a component was refactored and the docs weren't updated, the episode will describe the old architecture with complete confidence. There's no mechanism to detect that.

Nova: The system is, in the most literal sense, only as reliable as its own changelog.

Ray: And the changelog isn't available during synthesis. What's available is whatever the explain select pass decided was relevant — which is itself an LLM judgment call. Per the source, explain select is a distinct cost use case from explain synthesis, so there's a selection pass and then a synthesis pass. The selection pass is choosing which docs matter. If it misses a file, that information doesn't exist for the synthesis stage.

Nova: There's also something recursively strange about this that's worth sitting with. The Explain Pipeline is described in the documentation. The Explain Pipeline reads the documentation to build a brief. That brief is what this episode is reading from right now. So the pipeline's description of itself is what gets used to describe the pipeline. It's not circular in a broken way — the docs are a legitimate source — but it does mean the system cannot discover things about itself that it hasn't written down.

Ray: It has no introspective access. The episode knows what the docs say. It doesn't know what the code actually does at runtime. If there's a discrepancy, the episode is on the wrong side of it.

Nova: Build takeaway: if an explain-style path is being used for internal documentation episodes, the docs should be treated as a first-class artifact with the same rigor applied to a test suite. Stale docs don't just cause confusion — in this architecture, they cause confident misinformation with a source tier of 'official.'

Chapter 6

Nova: The principle from everything covered: the research stage is not an enrichment step. It's a trust boundary. Verified facts flow in on one side; on the other side, the pipeline is only allowed to work with what crossed that boundary.

Ray: And that boundary has to be structural, not prompt-based. 'Please only use verified facts' is not a constraint — it's a suggestion. The constraint is: the script stage cannot run until the research brief exists and has passed the human gate. The state machine enforces it. The schema enforces it. The cost tracking enforces it. Several overlapping mechanisms, not one.

Nova: The Explain Pipeline illustrates why this matters. It reads repository docs, excludes secrets, synthesizes a brief — no web, no external citations. The trust boundary for those episodes is the repository itself. Everything inside is treated as verified. Everything outside doesn't exist for the pipeline. The boundary is only as trustworthy as what goes inside it, though. If the docs are wrong, the pipeline enforces the wrong things with complete structural confidence.

Ray: The concrete takeaway: in any agentic pipeline, the research stage is where the system decides what it's allowed to believe. That decision should be explicit, documented, and enforced by the state machine — not left to the model's judgment at script time. The question worth asking before shipping: if everything in the research brief is wrong, how far does the error propagate before a human sees it? If the answer is 'all the way to the listener,' the gate needs to be harder.