Lightopedia

I built an internal Glean. The hard part was making it honest.

When knowledge is tribal and the product is broad, every question routes to the same two or three people. They answer in Slack, the answer scrolls away, and a week later someone asks the same thing. Nothing ever lands anywhere permanent.

At Light this showed up in order of severity. Sales promised things that didn't exist, or the right thing in the wrong way, because they couldn't find the authoritative answer fast enough and improvised. CS spent its days on the same answerable questions, arriving over and over with no reliable place to point people. New team members couldn't ramp on their own, which pulled even more time from the people already fielding everything.

The tell was that Slack threads had become the source of truth. The team was documenting, just ephemerally, in a format built for conversation rather than retrieval.

Everyone wearing many hats

Tribal knowledge wasn't concentrated in one person. It was scattered across founders, engineers, eventually PMs, anyone who'd been around long enough to know why things worked the way they did. Everyone wore many hats, and everyone fielded questions that pulled them out of whatever they were supposed to be doing.

The people with answers were also the people with the least time to give them. They weren't hoarding anything; they'd answer whatever you asked. But asking meant interrupting someone who was already behind.

We were shipping fast. Faster than Linear hygiene could keep up with, faster than release notes. The codebase was the source of truth, but reading code tells you that something works a certain way, not why. The decisions, the tradeoffs, the "we tried X but it broke Y." That context lived in the people who were there when it happened.

The traditional answer is to write it down. Nobody had time to write it down. They had customer calls and product fires and hiring decisions, and the people who knew the most had the least bandwidth of anyone. The knowledge existed; it was just locked in formats that don't scale. Heads, Slack threads, call recordings, code.

Confident and wrong

I expected the hard problem to be coverage, getting the AI to know enough. That's not what broke first.

What broke first was the AI being confident about things it had no business being confident about. When knowledge is tribal and underdocumented there are real gaps, and an AI without guardrails fills them fluently. It hallucinates in character. It sounds like it knows. It gives answers that are plausible given the product's logic but wrong in the one detail that matters to this customer or this edge case.

A gap is honest. A confident wrong answer is a trust liability, especially in fintech, where precision is the whole point. The failure mode that worried me was never "I don't know." It was a wrong answer delivered well, wrong in a way that takes an expert to catch.

The shadow channel

Before trusting the bot in production we ran a shadow test. For weeks, every question in the main support channel also routed to a parallel channel where Lightopedia answered silently, so we could compare what the human said with what the bot would have said.

This surfaced failure modes a demo never catches. The bot was often close: close enough to sound right, wrong enough to cause problems. "Light automatically reconciles transactions" versus "Light is designed to reconcile transactions" seems minor, until a customer reads "automatically" as zero configuration and files a complaint.

The parallel channel was calibration. It told us where the guardrails needed to be tightest.

The pipeline

The first version ran five stages:

Router. Classifies the question into a mode: capability ("does Light support X?"), enablement ("how do I explain X to a CFO?"), how-to ("how do I configure X?"), followup, or clarify. It's a policy selector; it never answers directly.

Retrieval. Vector search plus keyword search over the help articles, then a rerank. The help article repo is the single source of truth. If something isn't in the articles, the bot can't claim it.

Synthesis. An LLM explains the retrieved evidence in customer-ready language. The model's job is synthesis, not knowledge; it can only work with what retrieval handed it.

Citation gate. Every functional claim has to reference an article that was actually retrieved. If the model cites something it didn't fetch, the response fails validation. This is the hard constraint that keeps hallucination from leaking through.

Render. Formats the answer for Slack, with feedback buttons. Corrections become training signal.

Forbidden phrases

There is a literal blocklist:

• "automatically"
• "out of the box"
• "no setup required"
• "seamlessly"
• "effortlessly"

Each has a safe alternative. "Automatically" becomes "is designed to." "Out of the box" becomes "supports this workflow."

The allowed phrases are specific:

• "Light models this as..."
• "Light supports this workflow by..."
• "Light is designed to handle..."
• "This is represented at the AR / contract / ledger layer"

The distinction matters because the allowed phrases are calibrated to what the system can defend. "Light is designed to" is true: it describes intent and architecture. "Light automatically does" is a promise about behavior that depends on configuration, data quality, and edge cases the bot can't see.

What I learned

Trust came from the system being predictably bounded. People forgive gaps. They don't forgive surprises, especially in operations, where a wrong answer becomes someone else's cleanup.

That's also why the citation gate turned out to be the feature. It's what lets someone in CS or Sales use an answer without re-verifying it themselves, and until they can do that, no load has actually come off anyone.

And I stopped thinking of tribal knowledge as a documentation problem. It's an incentive problem. The people who know things will answer anything you ask; they just answer in the path of least resistance, which is Slack, not a doc. A system that asks them to change that behavior will fail. It has to learn from them without requiring them to teach it, capturing knowledge as a byproduct of work that's already happening, because that's the only version that survives a team at capacity.

What it became

The first version answered from help articles people had written. The rebuild asked a better question: why write them at all?

There is a lot to hold. Light is one platform covering the general ledger, receivables, payables, revenue recognition, procurement, spend, bill pay, subscriptions, and multi-entity reporting. Each of those was its own company once, a whole startup and a whole discipline. Together it's more than any few people can keep in their heads, and it changes every week. Documentation across that much surface doesn't survive being written by hand.

So we grew the knowledge base out of the code, the only place that already held the truth. We pointed an agent at the codebase and it wrote a how-to for every part of the product, from the implementation, in one consistent voice. Lightopedia reopened with 136 articles, none written by hand, living as markdown in a single repository small enough to read front to back.

That number decided the architecture. With 136 articles the bot doesn't need a search engine; it reads the table of contents the way a new hire would. The answering loop runs on GPT-4o across up to five iterations with four tools. knowledge_base pulls the full hierarchy of articles. fetch_articles reads the ones it picked, up to fifteen at once, straight from the repository. search_articles, semantic search over the text, is the fallback for questions phrased in words no title contains. And when the docs genuinely don't cover something, escalate_to_human drafts a structured ticket and hands the gap to a person. Vector search is the safety net, not the road.

The trust machinery survived the rewrite. The loop runs in two phases: the model works the tools and gathers evidence, then a second pass writes the answer from only what was fetched. Every claim carries an inline citation to an article the bot actually read, and a claim without one is rejected before it reaches anyone. The banned phrases stayed banned. The bot reads the whole thread for context, and when someone pastes a screenshot it runs the image through vision first, so a failing screen becomes a grounded answer without anyone typing a word.

The healing loop

The rebuild gave us a better bot. This part is what made it organic.

When the bot hits a question the docs can't answer, it escalates, and a person answers in the thread. That answer doesn't stop there. An agent writes it back into help-articles as a new article, so the next person who asks gets it from the docs and the bot never escalates that question again. The same path catches everything else that surfaces in conversation, like a changed workflow or a correction to something the docs had wrong.

Nobody is assigned to keep Lightopedia current, and it stays current anyway. Every question the company can't yet answer is a hole in the docs that fills itself the first time a human answers it out loud.

At Light we have a name for software built this way: organic software. It builds itself from the source of truth, fixes itself when it's wrong, and grows when the business grows, instead of rotting the day after it ships. Strip away the bot and what's left is a store of structured markdown that gets written and repaired as the company learns. The company's memory, maintained by the same system that reads it.

Growing up

Then it grew past the team. One repository feeds two surfaces now: the same articles answer our own people in Slack and power the knowledge base our customers read, and both update from that single source. Fix an article once and every reader sees it.

Then Lightopedia grew up. It became @light, the assistant our team runs in production, and it kept gaining tools. It searches the web now, not just our own library, and it still heals the library it was born from. We never threw the prototype away; it turned into the production system.

Why the old way rots

A wiki was a bet that the person who knew the answer would stop what they were doing, open the wiki, find the right page, write the answer in the wiki's format, and do this reliably, forever, for free. They never did. Every company has a wiki that was accurate for three months in 2022. The stale page was never a discipline problem: it was software with no way to grow on its own, so the work fell to the person least able to feed it, at the moment they were busiest.

The RAG project in your backlog is the same bet in newer clothes. We started there. Index every stale thread and contradiction you've ever written and you get back a confident average of all of it, and it rots the same way the wiki did.

The 136 articles nobody wrote by hand were just the starting stock. A knowledge base can clearly maintain itself; ours does. What we're working out now is which system gets to carry its own upkeep next.