Skip to content

Auto Classification skips column-name recognizers when sample data is disabled #32606

Description

@stout-yeoman

Affected module

Ingestion Framework — Auto Classification

Describe the bug

Auto Classification proposes no tags when both global sample-data settings are disabled:

readSampleData = false
storeSampleData = false

This also affects column_name recognizers, even though they only require metadata.

For example, a column such as:

project_manager_email_md

should match an enabled email-related column-name recognizer.

The pipeline still completes successfully, but reports zero updates.

Root cause

When sample-data reading and storage are both disabled, the sampler correctly returns:

TableData(columns=[], rows=[])

However, AutoClassificationProcessor in:

ingestion/src/metadata/pii/base_processor.py

iterates record.sample_data.data.columns to decide which fields to classify.

Because that list is empty, no field is passed to the classifier, even though the metadata entity still contains its canonical columns.

The classifier itself already supports column-name recognition with no sample values.

To Reproduce

  1. Configure an enabled column_name recognizer matching a column such as project_manager_email_md.

  2. Enable Auto Classification.

  3. Set globally:

    readSampleData = false
    storeSampleData = false
    
  4. Run Auto Classification.

The pipeline succeeds but proposes no tags.

Expected behavior

When sample-data access is disabled:

  • source values should not be read or stored;
  • content-based recognition should have no evidence;
  • column_name recognizers should still run against metadata fields;
  • matching tags should still be proposed.

Version

Reported on OpenMetadata 1.13.4. The same processor behavior is present on current main.

Additional context

A conservative fix is to preserve the current sampled-field path when samples exist, but fall back to canonical metadata fields with sample_data=[] when the sampler returns no fields.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

  • Status
    Done ✅
  • Status
    Done ✅

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions