Active Surveying Beyond the Catalog Perimeter · Part 3
What happens when you actually ask one: prerequisites, what it costs, and how the answer gets published.
In Part 2 we took a single row of the question catalog apart — what a question is made of, which parts of it Egeria governs today, and how one catalog covers five different kinds of resource. This post is about what happens when someone actually asks one. Egeria is the open-source metadata and governance platform Resource Explorer is built on; it is where anything worth keeping ends up.
A note on these blogs. We’re building Resource Explorer as we write this series, so the measurements and limits described here were true when we wrote them, and we expect many of them to be out of date quickly. That’s deliberate. We want to be honest about the gaps we find, because finding an issue is the first step toward fixing it. So read this as a view from the inside — what we think is good, as well as where we know it falls short. Progress marches on.
Working Out What Has to Run First
Ask an advanced question out of order and you hit a practical limit. You can’t look at how values are distributed in a column before you know which tables exist. You can’t scan a repository’s dependencies before reading the files that declare them.
So we let the system work the chain out for you. Each step declares what it needs before it can run, and the budget you’re already working within decides whether we stop and ask. If a prerequisite costs no more than the question you asked, we just run it and tell you we did. If it would cross into a more expensive tier, or needs a clone or a download, or needs credentials, or can only be answered by a person, you get a proposal instead — every step in the chain named at once, with the total, for you to accept or decline. What we don’t do is run half of it and then ask permission for the rest. That would spend your time and then ask for it.
The state behind that is recorded rather than guessed. Every answer records what produced it and when, so the interface can tell run four days ago from never run, and a question whose mechanism doesn’t exist yet shows as a gap with no Run button rather than a silent blank. When we do run something on your behalf, we record that we did, and which question demanded it.
Cost is measured rather than asserted. Every step run is recorded against a fixed set of axes: elapsed time, processor time, bytes fetched, calls made to an external service, calls made to Egeria, and tokens in and out of a language model. Not every axis is wired everywhere yet. On the database path only elapsed and processor time carry real numbers today; the other three read zero, and a zero there means the instrument is not watching rather than that the work was free. If a step has run before, the estimate quotes the median of those runs. If it hasn’t, it falls back to the tier the step declares for itself and says so: not yet measured, estimated from declared tiers. Each real run makes the next estimate better.
Two things complicate that. A step that publishes to Egeria writes a row as well, but it is an action rather than a measurement and carries no cost at all. And a step can now finish partially: our secret scanner reports the files it skipped and why, at zero confidence, rather than returning a clean result it has not earned. So a row in the ledger is not always a measurement, and where it is, it is not always a measurement of a finished job.
We did that work in that order deliberately. Offering to run something on your behalf without being able to say what it’ll cost is how a cheap question quietly triggers an expensive one, which is the economics the funnel exists to protect. Recording cost came first. Acting on it came second.
One honest qualification to all of that. The tiers rank how much attention a resource has been given, not how much it costs, and we now have the measurement to prove those are different things. A single Assessment-tier step — our secret scanner — ran for fifty minutes on one repository, while the whole Analysis-tier definition for a database takes about twelve seconds. The same scanner finished in three and a half minutes on a repository with thirty times as many files, because the cost came from pathological input rather than size: pattern rules restarting at every character of one very long encoded line. We bounded it the same day and fifty minutes became seventy-eight seconds. A gate built on what a step declares about itself can wave something expensive straight through without asking, and only measurement catches it — which is why a median of real runs beats a declared tier, and why we would rather tell you an estimate is unmeasured than let a tier stand in for a number.
How the Spreadsheet Becomes Egeria Content
Perspectives and questions didn’t begin in Egeria. They started as report_specs in pyegeria, became part of the Egeria portal, where you can see them today, and the base questions and perspectives landed in Egeria itself in release 6.1 — refined at each step. What follows is the current stage of that migration rather than a finished pipeline, and some of what still sits on the Resource Explorer side will cross over in time.
That’s the normal condition of this ecosystem rather than a caveat about it. Egeria is a living thing: a research vehicle as well as a production platform, and a place where we deliberately try ideas, re-evaluate them and refine them before they settle. That refinement comes from use cases and from the experts who bring them — a concept earns its keep by being applied to real work by people who know the domain, rather than by being specified in advance. And the evolution is mostly additive: new concepts arrive alongside what already works rather than in place of it, and we treat backward compatibility as a standing consideration. That’s what makes it reasonable to build on a model while it’s still moving.
A spreadsheet is convenient for human authors, but an exploration engine can’t parse raw cells at runtime. The declarative rows compile into two generated outputs — and never the other way around:
- A runtime catalog. question_catalog.yaml backs the Questions checklist the user actually sees, including the parsed answering-analysis kind that decides whether a question can run at all.
- Dr. Egeria markdown, and through it, real Egeria elements. Each row is published as a Question glossary term, linked to its Perspectives and its Funnel Stage. They are queryable in Egeria, not just in our own checklist.
Treating markdown as the step in between means you can review the definitions the way you’d review code: as a diff in a pull request, by someone who can see exactly what changed, using ordinary text tools rather than an opaque configuration screen. The rule that keeps it coherent is blunt. Edit the CSV, then regenerate. Never hand-edit the YAML.
Where This Model Breaks
The characteristic failure of a declarative pipeline isn’t that its artifacts disagree. It’s that they agree perfectly and have drifted from the implementation together, so every internal consistency check passes. We’ve hit this twice, and both are worth describing.
We added a Catalog History column to the CSV without updating the second generator’s list of known lead columns. Both generators identify perspectives by elimination, so the new column became a phantom Perspective on every row, and the next regeneration emitted seventeen “Link Perspective to Question” blocks for a Perspective that doesn’t exist. Someone caught it two days later reading a diff.
The second was larger. Eighteen database questions whose analyses were built, registered and dispatched were still carrying GAP notes we had written before those analyses existed. The interface told you no mechanism existed, and offered no Run button, for work that already ran. The CSV and the YAML agreed with each other exactly; both were wrong about the code. We corrected them at the end of September, and each of these two failures now has a test.
Which is why GAP is the right answer more often than it feels like it is. A question marked as a gap renders an honest “no mechanism exists for this yet” and offers nothing to click. That’s strictly better than a question wired to a number that doesn’t mean what its name says. The framing matters to us: the goal isn’t only to reduce the odds of a confident wrong answer, it’s to make honesty the default when there’s no good answer to give.
Curate: The Stage That Asks for a Decision
Through Scouting to Enrichment, the funnel filters a population of candidates, discarding dead repositories and scratch filesystems to find the handful worth keeping. Curate works differently. It doesn’t filter the population; it filters what inside the surviving resource is worth cataloging today. Two schemas may be worth cataloging this week and three more next week when the purpose changes, and coming back for those three shouldn’t mean re-running the whole analysis over material we’ve already surveyed.
- A qualified database may hold hundreds of tables, but only three schemas carry authoritative business entities; the rest are staging and scratch views.
- A repository may hold dozens of helper scripts and experimental branches, but only its public libraries and interface packages belong in an enterprise catalog.
Enrichment lets you both review and confirm some of the evidence that has surfaced, and add what you know yourself — what a resource is actually for, who really answers for it, what it costs, whether it fits the way you already work. That is knowledge no measurement can easily reach. We save those answers as annotations on the resource, the same shape a measurement takes. They sit beside the measured ones in Resource Explorer’s local store — what Part 1 called a working store rather than a second catalog — and they cross into Egeria when you publish. Some already do: the owner you record here is written onto the element itself at Curate. More will follow as we add annotation types, and some of that means extending Egeria to carry them well.
Curate asks the resource nothing. It runs no further profiling pass and no dependency crawl, and you can see that in the catalog itself, where Curate has no rows at all. That isn’t a gap in coverage. A decision stage has nothing to ask the resource.
The Curate stage is where you decide what to publish to Egeria. It gathers up everything the earlier stages produced, Enrichment’s answers included, and asks for a decision — is this worth putting in the catalog, and how much of it. It takes the commentary that belongs with that decision: a category, documentation, a usefulness rating, notes that are signed, dated and append-only. It takes relationships too, glossary terms against tables and columns and a match to a logical schema, which are claims about meaning rather than facts about storage. That is why the catalog has no Curate rows: here it is the person who answers. And then it publishes.
The choice is a record first. What you select is stored in Resource Explorer, signed and dated, so a reset on the far side can’t erase it. Committing it runs three things in order: we publish the server and database elements along with the owner recorded at Enrichment, we attach one catalog target per schema you chose, and Egeria’s own cataloguer creates that schema’s tables and columns on its next refresh, with Egeria’s survey limited to the schemas you named. We don’t create those elements ourselves. We declare the scope; Egeria’s cataloguer acts on it.
Separating surveying from cataloging is where this series started, and the finest grain is where it matters most. The two put different kinds of content into Egeria. A survey produces a report and annotations — measured values, evidence recorded against the resource. Cataloging asserts: it creates the elements themselves, and the facts and relationships around them. Evidence about a whole database is true of the whole database, so our survey report is published whole and describes every schema it could see; trimming it to the scope would make two reports disagree about one database for no gain. Assertion is the curation step, and you assert only what you chose. Leaving a schema out means no element, not no measurement.
One more thing about that tree. When it says a schema is catalogued, that is because we read Egeria back and found the element — not because the code reached the line that sends the write. Those two come apart more often than you would like, so we stopped letting the code report on its own success. The tree, the summary line above it and the tests all read the same answer, so they cannot tell you different things.
There’s more we could do here, and more we should. Two things in particular, and both follow from that same distinction. What a person tells us at Enrichment is already recorded as an annotation, and some of it already reaches Egeria. The rest needs more annotation types than Egeria carries today, so part of that work is extending Egeria itself — which is the additive evolution described earlier, seen from the end that does the pushing. And what Curate commits ought to be more than elements — the facts and the relationships around the asset, asserted properly, rather than annotations standing in for them. a judgment only we can see is worth less than one in the catalog. Beyond that, today you decide what to publish, and in future we may let you record not only that disposition but the action that should follow: that a resource be evaluated as part of an existing Digital Product, say, or become one in its own right. You’re already making the decision at this point; there’s no reason it has to stop at yes or no.
Looking Ahead to Part 4
Pairing an authoring spreadsheet with dependency-aware execution turns exploration from an assembly line into a question-driven workbench. We’ve now covered what a question is made of, what it costs to answer one, and what happens to the answer afterwards.
In Part 4 we turn to what this foundation makes possible: Context Intelligence and agentic workflows — how deep analysis stops producing static reports and starts contributing to a conversation about a resource.
Links
- Try it. The demo is open to anyone at egeria.pdr-associates.com — register free, no install, and it comes pre-loaded with a complete Egeria environment built around the fictional Coco Pharmaceuticals. It assumes a somewhat technical reader. If you would rather point it at your own estate, Resource Explorer runs from a fresh clone with Postgres and Ollama; you only need an Egeria server when you want to publish what you found.
- Or watch it. Egeria up and running in under two minutes, and accessing the demo environment. A fuller tour of what is in there is in the Egeria Workspaces 6.1 write-up.
- Egeria — the project itself, with the code at odpi/egeria.
- odpi/egeria-trellis — the monorepo where Resource Explorer lives.
- odpi/egeria-python — pyegeria, where report_specs started.
- The question catalog itself: resource_questions.csv — 94 rows, and the source of every number in this post.
- Dr. Egeria: the overview in the Egeria docs.
- The rest of this series: Part 1, Effective Exploration and the Survey Funnel, and Part 2, The Anatomy of a Question.
- Earlier writing on Dr. Egeria and where it came from: The Egeria Advisor — Sharing the Journey and Literate Governance and the Collaborative Canvas. Both predate Resource Explorer, so some details have moved on, but the ideas hold.
As always, feedback is welcome — and if one of these questions matters more to you than it does to us, the analysis behind it is open source and yours to improve.