Active Surveying Beyond the Catalog Perimeter · Part 1
Why surveying has to stay separate from cataloging, and what the funnel actually saves.
Picture a department running eighty databases nobody has written down, a few hundred internal Git repositories, and filesystems that have been growing for a decade. None of it is in a catalog. Someone has just been asked which of it matters for a new AI project, and they have a week.
That is the problem this series is about. Resource Explorer is the tool we are building for it: open source, and built on Egeria, the open-source metadata and governance platform. Egeria is where anything worth keeping ends up; Resource Explorer is how you decide what is worth keeping in the first place.
Enterprise data catalogs are rarely built in a vacuum. Most are created with deliberate purpose: to satisfy a regulatory mandate, support a critical data migration, power an executive analytics initiative, or unblock discovery. For those initial goals, cataloging known, curated assets pays for itself quickly.
The challenge isn’t that initial catalogs were designed poorly; it’s that organizations and technologies never stay still.
New business priorities emerge. Engineering teams spin up new cloud frameworks, AI pipelines, and external integrations. Mergers and acquisitions bring un-indexed digital estates into view. As a consequence, what we govern, how we govern it, and where opportunity resides continually shifts.
Core Principle: Surveying is decoupled from cataloging: exploring what might be useful must never require cluttering the enterprise catalog with what isn’t.
To help organizations navigate this ongoing evolution, our approach to metadata governance with Egeria must adapt to four core realities:
- More is not always better: Flood a catalog with every artifact anyone ever touched and you get noise and stale metadata, and people stop trusting it.
- Exploration demands an asymmetric strategy: Depth doesn’t have to be uniform. When investigating a new use case, examining every candidate in full costs more than the answer is worth. We need an evaluation funnel that starts broad and cheap to winnow the field, reserving deep technical scrutiny for viable contenders.
- Context is in the eye of the stakeholder: A single resource must be evaluated through multiple perspectives—active analytical lenses (such as Security, Architecture, Compliance, or App/AI Building) that care about fundamentally different details.
- Governance requires exploring the unknown: Maintaining what is already registered is essential, but supporting new initiatives requires surveying uncharted territory—scouting candidate assets that live outside the catalog’s existing perimeter.
This is the first in a series exploring the use, design, and implementation of Resource Explorer, with particular emphasis on the role of Context Intelligence.
Concretely, three things. You can look at a candidate resource without putting it in your catalog first. Every answer cites the measurement it came from, so you can check it rather than trust it. And what you decide, and why, is recorded — and travels into Egeria when you decide something is worth keeping.
A note on these blogs. We’re building Resource Explorer as we write this series, so the measurements and limits described here were true when we wrote them, and we expect many of them to be out of date quickly. That’s deliberate. We want to be honest about the gaps we find, because finding an issue is the first step toward fixing it. So read this as a view from the inside — what we think is good, as well as where we know it falls short. Progress marches on.
Egeria Resource Explorer (packages/resource-explorer in the open-source egeria-trellis repository) is designed around this exact discipline: the Survey Funnel.
The Foundation: Active Surveying in Egeria
To see how this works, start with how Egeria already handles discovery.
Egeria introduced native, active resource surveying in version 5.1 Prior to version 5, metadata discovery often operated as an external push or a rigid, periodic crawl. Egeria built active surveying directly into its platform architecture:
- Automated Survey Engines: Dedicated background services running on Egeria engine hosts, scheduled and orchestrated to inspect target digital resources.
- Active Analytical Actions & Annotations: Surveying does far more than passively observe an environment. It executes active analytical actions that produce typed annotations representing concrete measured values—resource metrics, schema extractions, data profiles, quality scores, and classification discoveries. The collection of these measurements is consolidated into an integrated Survey Report.
- Pluggable Survey Types: An Egeria Survey Type is modeled as a governance action process—a defined workflow of process steps implemented by governance actions. Teams can use pre-built survey types or create new ones with custom analytics or that execute external analytic packages (such as dbt models, vulnerability scanners, or custom code analyzers).
Crucially, surveying is decoupled from cataloging. To execute a survey, you point Egeria at a starting point (such as a database server or a repository URL). Running the survey executes analytical steps that observe the resource and record measured annotations, but it does not automatically force every discovered table, folder, or file into the enterprise catalog. You catalog only those elements that prove meaningful, exactly when you need them.
Where Resource Explorer Fits
Egeria provides the foundation for surveying, and developers can easily invoke these services programmatically using Python via pyegeria. But programmatic access alone does not solve the human challenge of exploratory evaluation.
When assessing a pool of new candidates, teams need an interactive environment that supports them through the scouting and surveying lifecycle without forcing a rigid sequence.
Resource Explorer adopts the same underlying Survey Type structure found in Egeria, but packages it for progressive exploration:
- Surveys can run in more than one place: Surveys will run across three flexible modes:
- Entirely within Egeria: Running native Survey Types dispatched across Egeria Engine Hosts.
- Locally within Resource Explorer: Executing surveys in Python (orchestrated as microflows) to leverage rich local analytical libraries without requiring remote service deployment.
- Hybrid: Composing surveys where select steps run inside Egeria while others execute in Resource Explorer.
- A working store, not a second catalog: Resource Explorer maintains a local PostgreSQL database serving as an Operational Data Store (ODS). It caches working findings, powers dynamic visualizations, handles local vector search, and answers rapid interactive queries across candidates mid-evaluation. It is deliberately not a second catalog: the ODS holds working state for resources still under evaluation, it is disposable and rebuildable by re-running the surveys that produced it, and Egeria remains authoritative for anything that has graduated. Nothing in the ODS is a system of record.
- Deterministic analysis first, AI on top of it: Resource Explorer combines deterministic algorithms (AST symbol parsing, commit history analysis, dependency manifest parsing) with grounded LLM synthesis. Traditional code analytics produce verifiable ground truth, while AI models summarize and explain those findings across different user perspectives.
- Answers you can check: Alongside structured metrics, Resource Explorer provides a scoped chat interface. Instead of asserting that the AI “never hallucinates,” the system enforces verifiable grounding: responses cite the exact underlying compiled evidence and analyses allowing human evaluators to audit the evidence directly.
- Graduating into Egeria: Survey results, Survey Types, and annotation types map directly onto Egeria’s open metadata models, so graduation requires no manual re-entry. When an asset is approved, it publishes into Egeria’s knowledge graph through an asynchronous outbox, carrying its complete Survey Report and accumulated classifications with it.
The Repository Survey Funnel in Practice
Organizations often treat code repositories as developer scratchpads rather than governed assets. But repositories define APIs, document system architecture, enforce data contracts, dictate supply-chain exposure, and establish corporate IP. We think they deserve the same attention we give databases and pipelines, and the path from first look to enterprise adoption makes them a good worked example of the funnel.
Discovery follows a natural, asymmetric progression:
- Scouting (Broad & Inexpensive): Fast surface-level signals using remote API metadata alone. Is the repository actively maintained? Who maintains it? What is its declared license and community adoption footprint? Unviable candidates are dismissed before downloading code.
- Discovery (Structural Signatures): Examine basic organizational structure. Does our organization already use or catalog this repository elsewhere in Egeria? What package manifests exist? Here, users can also narrow scope—evaluating the whole repository or focusing on a specific subfolder or module.
- Assessment (Perspective Evaluation): Stakeholders review accumulated findings through specific analytical lenses. Does this meet our baseline criteria for security, architecture, and governance?
- Analysis (Deep Inspection): Intensive analytical passes combining traditional tooling and AI: AST symbol extraction, coupling detection, supply-chain checks against the OSV.dev open-source vulnerability database, secret detection, and telemetry scans to flag hidden tracking endpoints.
- Enrichment (Human Knowledge): Measurement only reaches so far. Enrichment is where people add what the analytics cannot see—local knowledge, the role the resource plays in this organization, and the judgement that turns a set of findings into a position.
- Curate & Graduate: When approved, the asset is curated—linked to business glossary terms, tagged with governance classifications, assigned to governance zones, and published to Egeria along with its complete Survey Report.
What the Funnel Actually Saves
The economics are not really about what any single survey costs. Three things compound.
Surveys differ in cost by orders of magnitude. Reading a repository’s declared licence, commit cadence and contributor count is a handful of remote API calls. Extracting AST symbols across a hundred thousand lines of code, resolving the dependency tree and checking it against public vulnerability databases is a different kind of work entirely—and how different depends on the size and complexity of the resource, which is precisely what you do not yet know.
Surveys build on one another. Later analytics consume what earlier ones recorded. The funnel is not six independent passes; it is a chain in which each stage’s annotations narrow what the next one has to do.
And the saving is multiplicative, not additive. The expensive stages run on however many candidates survive the inexpensive ones. Halving the field during Scouting does not halve the cost of Analysis—it halves the number of times Analysis runs at all.
That is the argument as a claim about cost structure, and it is worth showing rather than asserting. What follows is measured, from the registry’s own step-run records, on one machine: an Apple M3 Max with 96 GB running macOS 27, with PostgreSQL 16.15 on the same machine, so no network sits between Resource Explorer and the database. The resource is AdventureWorks OLTP — 68 base tables, 87 views, 1,236 columns, 761,184 rows, 99 MB.
| Tier | What runs | Wall-clock |
|---|---|---|
| Scouting catalog only | schema and statistics; operations; credential capability | 9.0–9.8 s; 0.9–1.9 s; 1.0–1.4 s whole definition 12.8–14.9 s |
| Discovery zero-fetch | classification, relationship graph, grain, fingerprint, subject, coverage and fit — each from rows Scouting already stored | 0.5–0.7 s each all seven under 5 s |
| Analysis sampling | column profile; nested columns; derived; operations | 3.5 s; 3.6 s; 3.5 s; 1.7 s whole definition 12.2 s |
The distribution is the whole point. The catalog read that establishes what the database is costs about ten seconds, once. Every Discovery-tier question then answers in well under a second, because it reasons over what that read already stored rather than going back to the database. And the first tier that touches actual rows costs roughly what Scouting cost in the first place — which is the moment the ordering starts paying for itself, and the moment it would stop paying if the order were reversed.
Two things these numbers do not show, and both matter. The sample is one to three runs per step, on one machine, against one database, which is why every figure is a range rather than a mean. And the bytes-fetched, API-call and Egeria-call counters read zero for every database step: the cost vector was built for repository fetches and does not yet count SQL rows or bytes. So wall-clock is the only axis here, and it is the axis most sensitive to the machine underneath it.
These are also database figures in a post whose worked example is repositories. The tiers are the same shape either way; the database path is simply where a clean measured run existed first.
Which is why an inexpensive signal can be decisive. A large, complex system might look like exactly what we need and still return a poor OpenSSF Scorecard result. That score costs very little to compute. On its own, it can be enough to set the candidate aside before anyone clones it. We schedule a re-check in three months, in case the score improves, and send the deep analytics to the candidate that scored well instead.
Screening out is not discarding. It is deciding where the next hour of compute goes.
Around the Funnel: Investigation, Understanding, and Automation
Three of Resource Explorer’s capabilities are deliberately not stages, and reading them as stages is the quickest way to misunderstand the tool.
Investigation is the scope a run lives in—the candidate set, the work lists, the dispositions. An investigation can be entirely ad hoc. It can also be tied to an Egeria Project, and then the exploration itself becomes a governed artifact: results and evidence tie back to that project and are shared with its members. Exploratory work stops being somebody’s private scratch pad.
Understanding is a lens rather than a step. It opens over whatever has been measured so far, at any point in the funnel, rather than waiting for a stage to complete—and it is where local dashboards, built from the metrics a particular team actually cares about, will live.
Automate is where scheduled and triggered runs are defined. It re-runs any stage on a cadence; applied to assets that have already graduated, it is continuous surveillance—dependency drift, newly disclosed vulnerabilities, licence changes.
Inside the Interface: Incremental Results and Scoped Chat
(Above: A view of the Resource Explorer interface illustrating the lifecycle navigation, perspective filters, question-driven evidence cards, and the persistent scoped chat panel.)
In the interface, this model translates into a practical, exploratory workspace:
- The navigation follows the funnel: The top navigation tracks the funnel itself—Scouting, Discovery, Assessment, Analysis, Enrichment, Curate—alongside the capabilities that sit outside it: Investigation, Understanding, and Automate.
- Answers come from recorded measurements: Under Assessment, questions like “How mature is it?” or “What explicit license does the repository use?” are answered directly by underlying analytical annotations.
- Ask follow-up questions, scoped to the evidence: The right-hand conversational panel lets users ask open-ended questions about the resource. Answers are constrained to the analytical measurements already recorded in the local ODS, and each response cites the specific annotations it draws on—so an evaluator can open the underlying evidence and check the answer rather than take it on trust.
Looking Ahead: Perspectives and the Resource Taxonomy
The Survey Funnel provides an economically viable way to scout and evaluate candidates, but what we ask about a resource depends on two things: what kind of resource it is, and which perspective is examining it.
While our running example focuses on Git repositories, the questions that make sense for a codebase—commit cadence, branch protections, AST symbols, copyleft terms—are completely different from those needed for a PostgreSQL database, an Apache Kafka topic, or an S3 bucket.
Furthermore, evaluation is rarely one-dimensional. Stakeholders bring concurrent perspectives:
- An architectural perspective examines integration boundaries and coupling.
- A security perspective evaluates vulnerability advisories and supply-chain provenance.
- A governance and financial perspective checks license risk tiers, IP ownership, and maintenance sustainability.
A perspective is not a rigid organizational silo; it is an active, concurrent analytical lens held by an individual or team.
The Spreadsheet as a Customization Vehicle
To support this diversity without hardcoding rigid logic into application code, Resource Explorer organizes discovery around a configurable question matrix.
Resource Explorer provides starter spreadsheets for common resource types (such as code repositories, databases, and file systems). Critically, these spreadsheets are designed to be edited, extended, or replaced by users wanting to tailor the environment to their own organization’s standards. The spreadsheet serves as an accessible vehicle: it captures what stakeholders actually want to know, defines the stage of the funnel where those questions become relevant, and identifies which analytical mechanisms must run to answer them.
In Part 2: The Anatomy of a Question, we take a single row of that spreadsheet apart — what a question is made of, and how one catalog covers five different kinds of resource. Part 3: Asking a Question follows what happens when someone actually asks one: how Resource Explorer works out what has to run first, what that costs, and how the spreadsheet compiles into native Egeria governance artifacts.
Where This Stands Today
This series is candid about gaps, so it is worth saying plainly what already works. Surveying Git repositories and PostgreSQL databases runs today, and the costs in the table above are measured rather than estimated. Questions, perspectives and purposes are authored in a spreadsheet and drive what you see on screen. A question whose prerequisites have not run will tell you what it needs, what that will cost, and run it if you agree. Selective cataloging into Egeria works for databases, schema by schema.
What is not finished: several analyses are declared and not yet built, the cost vector is fully wired on the repository path and only partly on the database path, and some of what a person records during Enrichment stays in Resource Explorer rather than reaching Egeria. The catalog marks those gaps rather than hiding them, which is why you will see them named throughout this series.
Links
- Try it. The demo is open to anyone at egeria.pdr-associates.com — register free, no install, and it comes pre-loaded with a complete Egeria environment built around the fictional Coco Pharmaceuticals. It assumes a somewhat technical reader. If you would rather point it at your own estate, Resource Explorer runs from a fresh clone with Postgres and Ollama; you only need an Egeria server when you want to publish what you found.
- Or watch it. Egeria up and running in under two minutes, and accessing the demo environment. A fuller tour of what is in there is in the Egeria Workspaces 6.1 write-up.
- Egeria — the project itself, with the code at odpi/egeria.
- odpi/egeria-trellis — the monorepo where Resource Explorer lives.
- The question catalog itself: resource_questions.csv — 94 rows, and the source of every number in this series.
- Dr. Egeria: the overview in the Egeria docs.
- The rest of this series: Part 2, The Anatomy of a Question, and Part 3, Asking a Question.
- Earlier writing on Dr. Egeria and where it came from: The Egeria Advisor — Sharing the Journey and Literate Governance and the Collaborative Canvas. Both predate Resource Explorer, so some details have moved on, but the ideas hold.
As always, feedback is welcome — and if one of these questions matters more to you than it does to us, the analysis behind it is open source and yours to improve.
- Egeria version 5 shipped in July 2024. The current release is 6.1 (August 2026), with version 6.2 scheduled for release next month.