[Copilot]

Parquet files were being read twice per file in load_samples — once to check emptiness (loading all columns/rows), then again with the actual filters/columns. Similarly, remove_empty_parquets loaded the entire file just to check row count.

Changes

• load_samples: Read each parquet file once with the target filters and columns, then check if the result is empty:

  # Before: two full reads per file
  if not pd.read_parquet(parquet_file).empty:
      df_sample = pd.read_parquet(parquet_file, filters=filters, columns=load_columns)
  
  # After: one read
  df_sample = pd.read_parquet(parquet_file, filters=filters, columns=load_columns)
  if not df_sample.empty:
      ...

• Broken except block in load_samples: The except block was an exact copy of the try block (with a trailing continue), so on failure it would retry the identical operation before skipping anyway. Simplified to warn + continue.

• remove_empty_parquets: Use columns=[] to read only row-count metadata instead of loading the full file:

  # Before
  if not len(pd.read_parquet(file_path)):
  # After
  if not len(pd.read_parquet(file_path, columns=[])):

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[dprim7]

@copilot change target branch to hh4b-refactor

[Copilot]

@copilot change target branch to hh4b-refactor

I'm unable to change a PR's base branch programmatically with my available tools. You can update the base branch to hh4b-refactor directly from the GitHub PR page — click "Edit" next to the PR title, or use the base branch dropdown near the merge button.