Part 1: Active Scouting Beyond the Catalog Perimeter
Enterprise data catalogs are rarely built in a vacuum. Most are created with deliberate purpose: to satisfy a regulatory mandate, support a critical data migration, power an executive analytics initiative, or unblock discovery. For those initial goals, cataloging known, curated digital assets delivers immediate return.
The challenge isn’t that initial catalogs were designed poorly; it’s that organizations and technologies never stay still.
New business priorities emerge. Engineering teams spin up new cloud frameworks, AI pipelines, and external integrations. Mergers and acquisitions bring un-indexed digital estates into view. As a consequence, what we govern, how we govern it, and where opportunity resides continually shifts.
Core Principle: Surveying is decoupled from cataloging: exploring what might be useful must never require cluttering the enterprise catalog with what isn’t.
To help organizations navigate this ongoing evolution, our approach to metadata governance with Egeria must adapt to four core realities:
- More is not always better: Flooding a catalog with every unvetted artifact ever touched produces noise and stale metadata, eroding user trust.
- Exploration demands an asymmetric strategy: Depth need not be uniform. When investigating a new use case, exhaustive analysis of every candidate is cost-prohibitive. We need an evaluation funnel that starts broad and cheap to winnow the field, reserving deep technical scrutiny for viable contenders.
- Context is in the eye of the stakeholder: A single resource must be evaluated through multiple viewpoints—active analytical lenses (such as Security, Architecture, Compliance, or App/AI Building) that care about fundamentally different details.
- Governance requires exploring the unknown: Maintaining what is already registered is essential, but supporting new initiatives requires surveying uncharted territory—scouting candidate assets that live outside the catalog’s existing perimeter.
This is the first in a series exploring the use, design, and implementation of Resource Explorer, with particular emphasis on the role of Context Intelligence.
Egeria Resource Explorer (packages/resource-explorer in the open-source egeria-trellis repository) is designed around this exact discipline: the Survey Funnel.
The Foundation: Active Surveying in Egeria
To understand how Resource Explorer supports progressive qualification, we look to how Egeria handles discovery.
Egeria introduced native, active resource surveying in version 5.1 Prior to version 5, metadata discovery often operated as an external push or a rigid, periodic crawl. Egeria built active surveying directly into its platform architecture:
- Automated Survey Engines: Dedicated background services running on Egeria engine hosts, scheduled and orchestrated to inspect target digital resources.
- Active Analytical Actions & Annotations: Surveying does far more than passively observe an environment. It executes active analytical actions that produce typed annotations representing concrete measured values—resource metrics, schema extractions, data profiles, quality scores, and classification discoveries. The collection of these measurements is consolidated into an integrated Survey Report.
- Pluggable Survey Types: An Egeria Survey Type is modeled as a governance action process—a defined workflow of process steps implemented by governance actions. Teams can use pre-built survey types or create new ones with custom analytics or that execute external analytic packages (such as dbt models, vulnerability scanners, or custom code analyzers).
Crucially, surveying is decoupled from cataloging. To execute a survey, you point Egeria at a starting point (such as a database server or a repository URL). Running the survey executes analytical steps that observe the resource and record measured annotations, but it does not automatically force every discovered table, folder, or file into the enterprise catalog. You catalog only those elements that prove meaningful, exactly when you need them.
Where Resource Explorer Fits
Egeria provides the foundation for surveying, and developers can easily invoke these services programmatically using Python via pyegeria. But programmatic access alone does not solve the human challenge of exploratory evaluation.
When assessing a pool of new candidates, teams need an interactive environment that supports them through the scouting and surveying lifecycle without forcing a rigid sequence.
Resource Explorer adopts the same underlying Survey Type structure found in Egeria, but packages it for progressive exploration:
- Multi-Mode Survey Execution: Surveys will run across three flexible modes:
- Entirely within Egeria: Running native Survey Types dispatched across Egeria Engine Hosts.
- Locally within Resource Explorer: Executing surveys in Python (orchestrated as microflows) to leverage rich local analytical libraries without requiring remote service deployment.
- Hybrid: Composing surveys where select steps run inside Egeria while others execute in Resource Explorer.
- An ODS for Combined Analysis (Not a Second Catalog): Resource Explorer maintains a local PostgreSQL database serving as an Operational Data Store (ODS). It caches working findings, powers dynamic visualizations, handles local vector search, and answers rapid interactive queries across candidates mid-evaluation. It is deliberately not a second catalog: the ODS holds working state for resources still under evaluation, it is disposable and rebuildable by re-running the surveys that produced it, and Egeria remains authoritative for anything that has graduated. Nothing in the ODS is a system of record.
- Combining AI and Traditional Analysis: Resource Explorer combines deterministic algorithms (AST symbol parsing, commit history analysis, dependency manifest parsing) with grounded LLM synthesis. Traditional code analytics produce verifiable ground truth, while AI models summarize and explain those findings across different user viewpoints.
- Interactive Drill-Down with Verifiable AI: Alongside structured metrics, Resource Explorer provides a scoped chat interface. Instead of asserting that the AI “never hallucinates,” the system enforces verifiable grounding: responses cite the exact underlying compiled evidence and analyses allowing human evaluators to audit the evidence directly.
- Seamless Catalog Graduation: Survey results, Survey Types, and annotation types map directly onto Egeria’s open metadata models, so graduation requires no manual re-entry. When an asset is approved, it publishes into Egeria’s knowledge graph through an asynchronous outbox, carrying its complete Survey Report and accumulated classifications with it.
The Repository Survey Funnel in Practice
Organizations often treat code repositories as developer scratchpads rather than governed assets. But repositories define APIs, document system architecture, enforce data contracts, dictate supply-chain exposure, and establish corporate IP. Governing them with the rigor applied to databases and analytical pipelines is overdue—and the progression from initial curiosity to enterprise adoption makes them a vivid illustration of the funnel.
Discovery follows a natural, asymmetric progression:
- Scouting (Broad & Inexpensive): Fast surface-level signals using remote API metadata alone. Is the repository actively maintained? Who maintains it? What is its declared license and community adoption footprint? Unviable candidates are dismissed before downloading code.
- Discovery (Structural Signatures): Examine basic organizational structure. Does our organization already use or catalog this repository elsewhere in Egeria? What package manifests exist? Here, users can also narrow scope—evaluating the whole repository or focusing on a specific subfolder or module.
- Assessment (Viewpoint Evaluation): Stakeholders review accumulated findings through specific analytical lenses. Does this meet our baseline criteria for security, architecture, and governance?
- Analysis (Deep Inspection): Intensive analytical passes combining traditional tooling and AI: AST symbol extraction, coupling detection, supply-chain checks against the OSV.dev open-source vulnerability database, secret detection, and telemetry scans to flag hidden tracking endpoints.
- Enrichment (Human Knowledge): Measurement only reaches so far. Enrichment is where people add what the analytics cannot see—local knowledge, the role the resource plays in this organization, and the judgement that turns a set of findings into a position.
- Curate & Graduate: When approved, the asset is curated—linked to business glossary terms, tagged with governance classifications, assigned to governance zones, and published to Egeria along with its complete Survey Report.
What the Funnel Actually Saves
The economics are not really about what any single survey costs. Three things compound.
Surveys differ in cost by orders of magnitude. Reading a repository’s declared licence, commit cadence and contributor count is a handful of remote API calls. Extracting AST symbols across a hundred thousand lines of code, resolving the dependency tree and checking it against public vulnerability databases is a different kind of work entirely—and how different depends on the size and complexity of the resource, which is precisely what you do not yet know.
Surveys build on one another. Later analytics consume what earlier ones recorded. The funnel is not six independent passes; it is a chain in which each stage’s annotations narrow what the next one has to do.
And the saving is multiplicative, not additive. The expensive stages run on however many candidates survive the inexpensive ones. Halving the field during Scouting does not halve the cost of Analysis—it halves the number of times Analysis runs at all.
Which is why an inexpensive signal can be decisive. A large, complex system might look like exactly what we need and still return a poor OpenSSF Scorecard result. That score costs very little to compute. On its own, it can be enough to set the candidate aside before anyone clones it. We schedule a re-check in three months, in case the score improves, and send the deep analytics to the candidate that scored well instead.
Screening out is not discarding. It is deciding where the next hour of compute goes.
Around the Funnel: Investigation, Understanding, and Automation
Three of Resource Explorer’s capabilities are deliberately not stages, and reading them as stages is the quickest way to misunderstand the tool.
Investigation is the scope a run lives in—the candidate set, the work lists, the dispositions. An investigation can be entirely ad hoc. It can also be tied to an Egeria Project, and then the exploration itself becomes a governed artifact: results and evidence tie back to that project and are shared with its members. Exploratory work stops being somebody’s private scratch pad.
Understanding is a lens rather than a step. It opens over whatever has been measured so far, at any point in the funnel, rather than waiting for a stage to complete—and it is where local dashboards, built from the metrics a particular team actually cares about, will live.
Automate is where scheduled and triggered runs are defined. It re-runs any stage on a cadence; applied to assets that have already graduated, it is continuous surveillance—dependency drift, newly disclosed vulnerabilities, licence changes.
Inside the Interface: Incremental Results and Scoped Chat
(Above: A view of the Resource Explorer interface illustrating the lifecycle navigation, viewpoint filters, question-driven evidence cards, and the persistent scoped chat panel.)
In the interface, this model translates into a practical, exploratory workspace:
- Lifecycle Navigation: The top navigation tracks the funnel itself—Scouting, Discovery, Assessment, Analysis, Enrichment, Curate—alongside the capabilities that sit outside it: Investigation, Understanding, and Automate.
- Evidence-Based Answers: Under Assessment, questions like “How mature is it?” or “What explicit license does the repository use?” are answered directly by underlying analytical annotations.
- Interactive Drill-Down with Scoped AI: The right-hand conversational panel lets users ask open-ended questions about the resource. Answers are constrained to the analytical measurements already recorded in the local ODS, and each response cites the specific annotations it draws on—so an evaluator can open the underlying evidence and check the answer rather than take it on trust.
Looking Ahead: Viewpoints and the Resource Taxonomy
The Survey Funnel provides an economically viable way to scout and evaluate candidates, but what we ask about a resource depends on two things: what kind of resource it is, and which viewpoint is examining it.
While our running example focuses on Git repositories, the questions that make sense for a codebase—commit cadence, branch protections, AST symbols, copyleft terms—are completely different from those needed for a PostgreSQL database, an Apache Kafka topic, or an S3 bucket.
Furthermore, evaluation is rarely one-dimensional. Stakeholders bring concurrent viewpoints:
- An architectural viewpoint examines integration boundaries and coupling.
- A security viewpoint evaluates vulnerability advisories and supply-chain provenance.
- A governance and financial viewpoint checks license risk tiers, IP ownership, and maintenance sustainability.
A viewpoint is not a rigid organizational silo; it is an active, concurrent analytical lens held by an individual or team.
The Spreadsheet as a Customization Vehicle
To support this diversity without hardcoding rigid logic into application code, Resource Explorer organizes discovery around a configurable question matrix.
Resource Explorer provides starter spreadsheets for common resource types (such as code repositories, databases, and file systems). Critically, these spreadsheets are designed to be edited, extended, or replaced by users wanting to tailor the environment to their own organization’s standards. The spreadsheet serves as an accessible vehicle: it captures what stakeholders actually want to know, defines the stage of the funnel where those questions become relevant, and identifies which analytical mechanisms must run to answer them.
In Part 2: Declarative Governance — Compiling Viewpoints into Dr.Egeria, we will explore how Resource Explorer parses these question spreadsheets and compiles them directly into native Egeria governance artifacts—turning a human-authored spreadsheet into an active, automated qualification engine.
- Egeria version 5 shipped in July 2024. The current release is 6.1 (August 2026), with version 6.2 scheduled for release next month.