Reskin OpenGRC for FCC broadcast compliance#265
Open
chelstein wants to merge 38 commits intoLeeMangold:mainfrom
Open
Reskin OpenGRC for FCC broadcast compliance#265chelstein wants to merge 38 commits intoLeeMangold:mainfrom
chelstein wants to merge 38 commits intoLeeMangold:mainfrom
Conversation
Visual reskin to a deep-navy + gold theme matching the FCC Compliance
dashboard, plus first-class FCC broadcast entities so the product is
usable end-to-end by an FCC compliance officer at a radio/TV station.
Theme:
- New navy & gold palette in resources/css/filament/app/theme.css
(legacy `grcblue` token names retained, now resolve to navy ramp;
new `grcgold` ramp for amber accents)
- Sidebar, topbar, active states, scrollbar, primary buttons, stat
widgets restyled to match the dashboard mock
- Brand names updated to "OpenGRC FCC Compliance" / "...Admin"
- All 4 panel providers (Admin/App/TrustCenter/Vendor) recolored to
the navy + gold scheme
Domain model (new):
- fcc_facilities (FCC Facility ID, ASR #, HAAT/AMSL, owner...)
- fcc_licenses (call sign, FRN, licensee, service, freq/ch,
grant/expiration/renewal, status, score)
- fcc_transmitters (manufacturer/model, ERP, EAS ENDEC, proofs)
- fcc_rules (47 CFR sections - Parts 73, 11, 17)
- fcc_license_rule_status (per-license rule compliance status)
- fcc_deadlines (EAS, public file, renewal, ownership,
tower lighting, etc.)
- fcc_compliance_events (audit-trail style activity stream)
Filament UI:
- FccLicenseResource with full broadcast fields and compliance badges
- FccFacilityResource, FccTransmitterResource, FccRuleResource,
FccDeadlineResource (lean Manage pages)
- New "FCC Compliance" navigation group
- FccComplianceOverviewWidget — Compliance/Licenses/Rules/Compliant/
At Risk/Non-Compliant stats matching the dashboard top row
- FccUpcomingDeadlinesWidget — quarterly EAS, public file, renewal,
Issues/Programs, tower lighting
Demo data (FccComplianceSeeder, wired into DatabaseSeeder):
- 8 stations matching the mock (KXYZ-FM, WABC-AM, WQRS-TV, KLMN-LP,
KDEF-FM, KJKL-TV, KPOW-AM, KZ99-FM) with realistic FRNs, services,
expiration dates, compliance scores, transmitters
- 18 real CFR sections (73.3526, 73.1212, 73.317, 73.659, 11.35,
11.61, 17.47, etc.) categorized & severity-ranked
- Per-license rule status to drive the "By Category" rollup
- Upcoming deadlines + recent compliance activity
Filament's StatsOverviewWidget and TableWidget declare $heading as static; subclasses must do the same. Drop the unused $description override to match the parent signature.
StatsOverviewWidget declares $heading as non-static while TableWidget declares it as static. Override the getter instead to avoid signature mismatch in either direction.
Matches the FCC Compliance dashboard mock which uses a deep-navy canvas. Users can still toggle to light via the user menu.
Transforms the OpenGRC dashboard into an actually-usable FCC broadcast compliance tool that covers the operational obligations a Chief Engineer or Director of Compliance at a radio/TV station deals with daily. Dashboard: - Custom Dashboard page hosts FCC-only widgets (replaces the legacy StatsOverview / ControlsStatsWidget / AuditListWidget grid) - New FccLicenseComplianceTableWidget — License Compliance table - New FccRuleCategoryRollupWidget — Compliance by Rule Category - New FccTopNonCompliantRulesWidget — Top Non-Compliant Rules - New FccComplianceActivityWidget — recent activity feed - Existing FccComplianceOverviewWidget — top stat cards - Existing FccUpcomingDeadlinesWidget — deadlines table - New FCC compliance footer strip (DATA INTEGRITY / SOURCE / LAST REFRESH / SECURE / COMPLIANT / TRUSTED) via PanelsRenderHook::FOOTER New operational entities (real FCC obligations): - fcc_asr_registrations — 47 CFR Part 17 (>200ft towers) - fcc_eas_tests — RWT/RMT/NPT logs (47 CFR Part 11) - fcc_issues_programs_lists — quarterly per §73.3526(e)(12) - fcc_issues_programs_entries — issue/program/duration entries - fcc_public_file_documents — online inspection file (LMS) - fcc_tower_lighting_inspections — §17.47 quarterly inspections - fcc_tower_lighting_outages — §17.48 NOTAM tracking - fcc_political_file_entries — political ad LUR tracking - fcc_station_log_entries — daily ops log per §73.1820 - fcc_regulatory_fees — annual Form 159 / Pay.gov - fcc_form_filings — 323, 397, 2100-H, 2100-A history New Filament resources (all under "FCC Compliance" group): - EAS Tests, Issues/Programs, Public File, ASR Registrations, Tower Inspections, Political File, Station Log, Regulatory Fees, Form Filings — each with realistic FCC fields & helper text citing the relevant CFR section Seeded operational data (FccOperationalSeeder): - 12 weeks of weekly RWTs + 6 months of monthly RMTs per station - 4 quarters of Issues/Programs Lists per station with 5 entries each - Public File documents (authorization, ownership, EEO, contour map, Issues/Programs) per station - 5 ASR registrations with quarterly tower lighting inspections and one realistic outage with NOTAM - Political file entries across federal/state/local + ballot initiatives with LUR-window flags - 7 days of station log entries per station with realistic transmitter readings (plate V/I, forward/reflected power, SWR) - Current-FY regulatory fees by service tier (KZ99-FM overdue) - Recent 323 (granted) + 397 (filed) form filings per station
The previous query used selectRaw + groupBy(fcc_rule_id) on the
join table, which produced rows without a primary 'id' column.
Filament's TableWidget::getTableRecordKey() then returned null
and the dashboard 500'd.
Switched to FccRule::withCount('statuses as affected_count')
which keeps the rule PK intact and is more idiomatic.
SQLite rejects HAVING on a non-aggregate query. Replace with whereHas() to filter rules to only those with non-compliant or at-risk statuses, while withCount() still hydrates the count for ordering and display.
Switch Dashboard from TabbedPage (which only renders tabs, not the widgets array) to Filament's standard Dashboard page. All six FCC widgets now render in a 2-column grid. Extend theme.css to apply the navy/gold canvas in BOTH light and dark mode so the look is consistent regardless of the user's theme toggle: - body / main / page background: deep navy (#0a1830) - sections / cards / tables: dark navy (#0c1f3d) with subtle gold border - table headers: gold uppercase, navy header strip - inputs / dropdowns / modals: dark navy - stat values: gold Reorder navigationGroups to put 'FCC Compliance' first.
Table cells were rendering at low contrast against the navy background. Force #f3f4f6 on .fi-ta-cell and inner spans, keep colored status indicators visible, and brighten section headers.
The Filament widget grid couldn't match the FCC Compliance mockup without fighting columnSpan defaults and CSS specificity. Switch the Dashboard page to a custom Blade view that hand-builds the layout: - 6 stat cards across the top with click-through to Licenses / Rules - License Compliance table (8 cols) + Rule Category rollup (4 cols) - Donut + breakdown panel + Top Non-Compliant Rules + stacked Upcoming Deadlines / Compliance Activity - Each row links to the appropriate Filament resource (call signs to individual license views, rule numbers to FCC Rules page, etc.) All data is real model data. The dashboard is read-only summary; operational CRUD lives in the resources under FCC Compliance.
…brand
- CSS hides To Do / Risk Management / Vendor Management / Trust Center
sidebar items on the App panel (underlying pages still work via
direct URL).
- Topbar render hooks add:
* 'N Alerts' badge — driven live by overdue deadlines + non-compliant
licenses + overdue regulatory fees. Critical-tinted when any license
is non-compliant. Click-through to Deadlines.
* 'Export FCC Report' button — streams a multi-section CSV (license
summary + per-license rule status rollup) suitable for sharing with
leadership or audit.
- New FccComplianceReportController + /app/fcc-report.csv route (auth).
- Data migration forces the DB-stored 'general.name' setting to
'OpenGRC FCC Compliance' so the brand text matches the panel name.
- FccComplianceSeeder now seeds 12 real, publicly-licensed U.S. broadcast stations: WABC, WTOP, KCBS-FM, WGBH-FM, KQED-FM, WAMU, KCRW, KUOW, WHTZ, KFI, WBZ, KOMO. Real call signs, frequencies, licensees, approximate Facility IDs, and tower coordinates from public ASR records. FRNs and renewal dates are illustrative. - Removes leftover fictional records (KXYZ-FM, KZ99-FM, etc.) on re-seed via a forceDelete pass at the start. - Adds 'php artisan fcc:import --calls=WABC,WTOP,...' command that pulls live data from the FCC public-files API and upserts into fcc_facilities + fcc_licenses. Includes a --dry-run option. - Doc comments cite CDBS daily-dump URLs for full bulk sync.
The single endpoint I picked first (publicfiles.fcc.gov/api/manager/
station/search/{call}.json) returns 404. FCC's public-search URL
shapes have shifted; try three known-good paths in fallback order:
- publicfiles.fcc.gov/api/manager/station/search?searchString=...
- publicfiles.fcc.gov/api/manager/station/{call}
- enterpriseefiling.fcc.gov/dataentry/api/elasticsearch/...
If all three return nothing, real production imports should use the
FCC CDBS daily bulk dumps (the canonical answer for sustained sync).
The FCC's Public Inspection Files API moves around. Their actual
stable open-data surfaces are:
1. opendata.fcc.gov Socrata endpoints (datasets cd28-25ar and
iqaq-mbpb cover broadcast engineering data). Stable, no auth
for low volume, returns JSON.
2. transition.fcc.gov/fcc-bin/{amq,fmq,tvq} CGI scripts with
format=4 (pipe-delimited). Have been live since the 90s.
The command now tries Socrata first (proper structured data),
then falls back to the FM/AM/TV Query CGIs which it parses as
pipe-delimited. Surfaces the source URL in --dry-run output so
you can see which endpoint actually responded.
…Query
The FM/AM/TV Query CGI at transition.fcc.gov/fcc-bin/{fmq,amq,tvq}
returns an HTML page that embeds the actual record fields as
JavaScript variable assignments (facility_id, c_callsign, c_service,
c_comm_city_app, freq, alat83, alon83, p_erp_max, etc.) plus a few
HTML-wrapped fields (Licensee, Licensed date).
We parse both via regex and pull through:
- facility_id, call sign, service, channel/frequency
- licensee name, community + state
- lat/lon (NAD83), HAAT, AMSL, ERP (kW)
- last licensed date
Verified live against KQED — returns facility 789877, 88.5 MHz,
KQED INC., Alamo CA, 37.815750 / -122.062444, 0.14 kW ERP.
Upsert path now stores coords/HAAT/AMSL on the facility and the
last licensed date on the license, so the dashboard reflects real
geography for imported stations.
Confirmed opendata.fcc.gov 404s for the broadcast datasets, so the
import was wasting ~30s/station on dead requests before falling back.
Skip those and hit transition.fcc.gov/fcc-bin/{fmq,amq,tvq} directly,
in that order. trySocrata() kept for future re-enable.
Some CDN front-ends (or transition.fcc.gov itself when polled by a non-browser UA) serve a stripped page where the only facility_id assignment is the JS declaration 'var facility_id = 0;'. Tighten the existence check to require a quoted, non-zero facility_id to proceed, and use a Mozilla UA + Accept header to avoid being treated as a bot.
Verified empirically: 'Mozilla/5.0 (compatible; OpenGRC-FCC/1.0)' returns HTTP 000 from FCC's Akamai edge in 79ms (instant reject), while 'curl/8.5.0' returns HTTP 200 with the full payload in 553ms. Switching to curl/8.5.0 + Accept: */*. The trick is to NOT pretend to be a browser.
Two bugs surfaced in the live import: 1. AM Query for stations with day/night patterns (like WABC, KOMO) returns a multi-row HTML LIST page, not the JS-variable detail page. Detect the list format, extract the first 'facid=NNNN' link, and re-fetch /amq?list=0&facid=NNNN to get the detail payload my parser already understands. 2. FCC service designator 'FB' (FM Booster) wasn't mapped to FM, so KQED-FM2 (a booster) showed as 'OTHER'. Map FB, FX, FL, AB, CA, LP, LD, TX correctly per LMS service codes. 3. Refactored the HTTP call into fetchFccQuery() so both the call and facid lookups share the same UA + timeout logic.
FM detail pages assign bare names (facility_id, c_callsign, etc.) while AM detail pages prefix with c_ (c_facility_id = '70658';). The previous \\b regex only matched the FM form, so AM stations returned null after the list-page resolution succeeded. Switched the jsVal helper to a negative-lookbehind that matches BOTH 'facility_id' and 'c_facility_id'. Verified on KQED (FM) and WABC (AM): both now extract facility_id, callsign, and service correctly.
Two issues with AM stations: 1. AM detail pages assign 'freq = 770;' (unquoted numeric) while FM pages quote it (freq = '88.5'). The jsVal helper now accepts either quoted or unquoted numeric values. 2. Bare 'WBZ' / 'KFI' / etc. were matching unrelated FM stations that share those call letters. When the user passes a -AM/-FM/-TV suffix, hit that specific query first; otherwise default to FM. So 'WBZ-AM' now correctly returns Audacy's Boston station, while 'WBZ' returns the FM Four Rivers station. Verified live on WABC: facility 70658, AM, 770 kHz, NEW YORK NY, RED APPLE MEDIA, INC.
The single-station fcc:import (transition.fcc.gov FM/AM/TV Query) is
fine for ad-hoc lookups but is too slow for full-USA coverage. Add a
bulk importer that pulls FCC's daily CDBS dumps and seeds the entire
broadcast database in one pass.
Files used (all stable for 20+ years on transition.fcc.gov):
- facility.zip (~3 MB, ~30K rows) → master facility table
- am_eng_data.zip (~350 KB) → AM engineering
- fm_eng_data.zip (~12 MB) → FM engineering (lat/lon, ERP, HAAT)
- tv_eng_data.zip (~6.6 MB) → TV engineering
- party.zip (~16 MB) → licensee names
- fac_party.zip (~570 KB) → facility ↔ party mapping
Pipeline:
1. Download each .zip with curl-style UA (FCC's Akamai blocks Mozilla)
2. Unzip in storage/app/cdbs/
3. Stream-parse facility.dat with chunked upserts (500 at a time)
into fcc_facilities + fcc_licenses
4. Stream-parse {svc}_eng_data.dat to fill in lat/lon/HAAT/AMSL
5. Stream-parse party.dat + fac_party.dat (LIC role) to fill licensee
names on the licenses we just imported
Options:
- --limit=N smoke test
- --service=fm restrict to one service
- --skip-download reuse cached files
- --skip-licensees skip the party.zip pass (faster)
Total disk: ~25 MB cached + DB grows by ~30K license rows on full run.
- fcc:import-asr: bulk-import every Antenna Structure Registration from FCC ULS (data.fcc.gov/download/pub/uls/complete/r_tower.zip + a_tower.zip). Indexes EN.dat (entities) + CO.dat (coordinates), streams RA.dat → upserts into fcc_asr_registrations. - fcc:sync: master command running CDBS bulk + ASR + operational seeder end-to-end. --quick for smoke test, --skip-bulk/--skip-asr for partial runs. - Dashboard: License Compliance table now caps at 25 rows ordered by compliance status (worst first). Total count comes from a separate COUNT() query so the footer reads 'Showing 25 of N'. Without this, after fcc:import-bulk the dashboard would render ~30K <tr>s. - FccOperationalSeeder: now picks a 30-station representative sample (well-known reference stations + random extras) instead of iterating all licenses. Without this it would generate ~3M rows of EAS/IPL/log data after bulk import.
License page: - Pagination: 25 / 50 / 100 / 250 (default 25) - State filter via facility relationship - Persist filters + sort in session - Striped rows Facility page: - New columns: Owner, HAAT (m), AMSL (m), Latitude (toggleable), Longitude (toggleable) - Pagination: 25 / 50 / 100 - Striped rows - Sortable on facility_id, community, state ASR page: - Pagination: 25 / 50 / 100 / 250 - Striped rows FccComplianceSeeder: - Cap rule-status generation to ~250 representative licenses (well-known reference stations + 200 random) so we don't create 540K rule_status rows after bulk import.
Concise deploy guide covering: - prerequisites (PHP 8.4, outbound HTTPS to transition.fcc.gov + data.fcc.gov) - one-time install steps - fcc:sync (master) + fcc:sync --quick smoke path - individual commands (fcc:import-bulk, fcc:import-asr, fcc:import) - sqlite verification queries - table of what's real vs synthetic per page - weekly cron entry to keep CDBS data fresh
The original FM_ENG_COLS positions were a guess and didn't match. Verified live against current CDBS dumps: - fm_eng_data.dat — 73 cols; facility_id at col 20, lat/lon at cols 30-37, HAAT at 40, ERP at 29, AMSL at 48, station_class at 50 - tv_eng_data.dat — 75 cols; facility_id at col 21, lat/lon at cols 29-36, HAAT at 41, ERP at 42, AMSL at 47 - am_eng_data.dat — only 17 cols and no lat/lon (AM coordinates live in am_ant_sys.dat which requires a separate join). Skip AM in the engineering pass. Now the bulk importer correctly populates HAAT, AMSL, lat, lon for every FM and TV facility.
The per-row UPDATE loop made SQLite fsync after every statement, turning the FM engineering pass into a 30+ minute slog. Wrap each 500-row batch in DB::transaction() and use a prepared statement re-bound per row. Same fix for flushLicensees.
party.dat schema verified live: party_name is at column 10 (not 1), canonical entity name lives there. Skip placeholder rows with 'PARTY INFO NOT FOUND'. fac_party.dat role_code is 'LICEN' (5 chars), not 'LIC' or 'LICENSEE'. Accept all four forms. FccComplianceSeeder now also drops the original fictional facility rows (Market Hall Tower, CityView Mt. Wilson, etc.) by exact name match. Real CDBS facilities never collide with these names so this is safe.
The previous flushLicensees did 2 UPDATEs per row, each with a subquery. With ~98K LICEN entries that meant ~200K subquery-laden UPDATEs, taking many minutes. Pre-load cdbs_facility_id → internal_pk into memory, then group each batch by licensee name and emit one UPDATE … WHERE id IN (…) per distinct name. Same data, ~100x fewer queries.
CDBS facility.dat carries a row for every authorization including unbuilt construction permits. For un-issued CPs, the fac_callsign field is populated with the FCC application/file number (e.g. '780118AD', '10269') rather than a real call sign. Filter to only rows where call_sign matches the broadcast pattern (3-8 chars, starts with a letter, alphanumeric + hyphen). This drops ~20K CP rows from fcc_licenses, leaving only stations the FCC has actually licensed.
After the bulk import, fcc_licenses has ~60K rows and fcc_facilities ~90K. The Licenses navigation badge ran two unindexed COUNT(*) queries on every page load (one per badge color), and Filament's sidebar runs those for every visible resource — making the splash logo appear to hang while the count queries serialized through SQLite. Two fixes: 1. Wrap getNavigationBadge / getNavigationBadgeColor in 5-minute cache. The badge value is approximate UI-only metadata; staleness doesn't matter. 2. Add indexes on fcc_licenses.call_sign / licensee / service and fcc_facilities.state / owner so sort / filter / search on the list pages don't full-scan.
CDBS bulk dumps don't expose the modern FCC Registration Number
(FRN) — FRN lives in LMS. Add a polite, resumable scraper that
fetches LMS publicFacilityDetails.html?facilityId={id} per imported
license and parses the <dt>FRN:</dt><dd>...</dd> pair.
Per-station fields extracted from LMS:
FRN, Title, Service, Facility Status, Status Date, Facility Type,
Station Type, Community, Frequency, Digital Operation, Email,
Phone, Address, Country.
Currently we persist FRN onto fcc_licenses; the rest is available
in the parser for future schema additions.
Wired into fcc:sync as Step 3/4 with --lms-batch=1000 default
(re-run to keep augmenting). --quick uses --lms-batch=50.
Per-request sleep defaults to 200ms — full augmentation of ~57K
licenses takes ~3 hours at that rate. The command is resumable via
the 'where frn is null' filter, so cron-driven incremental pulls
are practical.
Eloquent's chunk() applies its own LIMIT/OFFSET, so a prior limit() on the query is silently dropped. With --limit=1000 the command was iterating all 57K licenses anyway. Eager-load the capped result set instead. For 1k batches that's trivially small in memory.
…ities
After scaling to 57K licenses + 89K facilities, several pages were
hanging on N+1 queries or trying to render giant dropdowns:
1. Eager-load license: 9 operational resources (EAS Tests, Deadlines,
Issues/Programs, Form Filings, Public File, Political File, Reg Fees,
Station Log, Transmitters) all show TextColumn::make('license.call_sign')
in their table. Without with('license') Filament does N+1 — 25 row
page = 26 queries. Added getEloquentQuery() override to each.
2. Eager-load facility on FccLicenseResource (table shows nothing from
facility, but the state filter uses the relationship).
3. License form preload disaster: facility dropdown had ->preload()
which fetched all 89K facility rows into the page. Removed preload;
the field is searchable() so users type to find. Added a custom
record-label formatter so the dropdown shows '12345 — Tower Site'.
4. License form required fields: frn + licensee are now optional. CDBS
imports don't always populate them, so an admin opening 'Edit' on
an imported station could not save without typing a fake FRN. They
stay required at the database layer if non-null, just not at form
layer.
5. EAS Tests license_id select had ->preload() — same issue, removed.
6. Expiration date no longer required on License form (CDBS records
have nullable lic_expiration_date).
Without WAL, the long-running fcc:import-lms artisan process holds a write lock on opengrc.sqlite while iterating ~57K stations. Every HTTP request to the dashboard or any FCC list page blocks waiting for the lock and times out at the splash logo. Set PRAGMA journal_mode=WAL + synchronous=NORMAL + busy_timeout=5000 on every connection in AppServiceProvider::boot(). WAL keeps reads non-blocking even during a full import.
The 'state' SelectFilter used ->relationship('facility', 'state')
which calls Filament's Select::isOptionDisabled() with a NULL label
when a facility has a null state column. Many CDBS-imported facilities
have null state (international border-zone records, anomalous CPs).
Replace with an explicit options() callback that pulls only non-null
distinct states + a custom query() callback that uses whereHas.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Visual reskin to a deep-navy + gold theme matching the FCC Compliance dashboard, plus first-class FCC broadcast entities so the product is usable end-to-end by an FCC compliance officer at a radio/TV station.
Theme:
grcbluetoken names retained, now resolve to navy ramp; newgrcgoldramp for amber accents)Domain model (new):
grant/expiration/renewal, status, score)
tower lighting, etc.)
Filament UI:
Demo data (FccComplianceSeeder, wired into DatabaseSeeder):