feat: voice event persistence and Discord gateway data lake#259
Merged
Conversation
Voice events stopped being ingested since 2026-03-23 because the real-time handler only updated Redis cache without persisting to the voice_messages table. Adds PersistVoiceStateAction that resolves tenant and identity from Discord guild/user IDs and writes joined/left records, following the same pattern as the ETL importer.
…ta lake Adds a discord_event_logs table and a raw gateway listener that persists every Discord event (MESSAGE_CREATE, VOICE_STATE_UPDATE, GUILD_MEMBER_ADD, etc.) with full payload for future analytics. Registered via Laracord AFTER_BOOT hook on the Discord client.
Suppress false-positive property.notFound on VoiceStateUpdate magic properties and fix unnecessary nullsafe access in RawGatewayEvent.
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR adds Discord gateway event logging and voice channel activity tracking to the bot. The infrastructure captures raw Discord events by registering a gateway event handler that validates dispatch payloads and persists them to a new 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
app-modules/bot-discord/tests/Feature/Actions/VoiceChannel/PersistVoiceStateActionTest.php (1)
26-48: ⚡ Quick winAdd a regression test for tenant lookup provider mismatch.
Given tenant resolution uses
ExternalIdentity, add a case where the sameguild_idexists under a different provider and assert noVoiceis persisted (or correct Discord provider mapping is used). This prevents cross-provider mis-attribution regressions.Also applies to: 112-126
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app-modules/bot-discord/tests/Feature/Actions/VoiceChannel/PersistVoiceStateActionTest.php` around lines 26 - 48, Add a regression test in PersistVoiceStateActionTest (inside the existing beforeEach/setup or a new it/test case) that ensures tenant lookup uses the provider: create an ExternalIdentity with the same guildId value but with a different provider (e.g., IdentityProvider::Slack) for the tenant, then trigger the PersistVoiceStateAction (or the test flow that persists Voice) using the Discord provider and assert that no Voice record is persisted for that mismatched provider (or that the persisted Voice is only created when an ExternalIdentity for IdentityProvider::Discord exists). Reference the test setup variables ($this->tenant, $this->guildId, ExternalIdentity factory, IdentityProvider constants, and the PersistVoiceStateAction) so the new case mirrors the existing setup but swaps the provider to verify provider-specific tenant resolution.app-modules/bot-discord/src/Events/RawGatewayEvent.php (1)
22-22: Plan retention/pruning for raw payload growth.Persisting every dispatch payload will grow
discord_event_logsquickly; add a retention/pruning strategy (or partitioning) before production scale.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app-modules/bot-discord/src/Events/RawGatewayEvent.php` at line 22, The raw payload is being persisted as-is ('payload' => json_decode(json_encode($payload->d), true) in RawGatewayEvent.php) so you must add a retention/pruning strategy before production: add a created_at timestamp to the discord_event_logs records (via migration), make created_at indexed, introduce a configurable retention_days setting, and implement either a periodic cleanup task (cron/queue job) that deletes records older than retention_days or use DB partitioning by date with automated partition drops; update the RawGatewayEvent insert flow to populate created_at and ensure the cleanup job or partitioning is documented and enabled in deployment config.app-modules/bot-discord/tests/Feature/Events/RawGatewayEventTest.php (1)
14-21: ⚡ Quick winUse a realistic
GUILD_MEMBER_ADDpayload shape in this fixture.For this event type, asserting via nested
d.user.idprevents false confidence arounduser_idextraction.Suggested test update
- 'd' => (object) [ - 'guild_id' => '123456789', - 'user_id' => '987654321', - 'channel_id' => null, - 'roles' => [], - ], + 'd' => (object) [ + 'guild_id' => '123456789', + 'user' => (object) ['id' => '987654321'], + 'channel_id' => null, + 'roles' => [], + ],🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app-modules/bot-discord/tests/Feature/Events/RawGatewayEventTest.php` around lines 14 - 21, Update the GUILD_MEMBER_ADD test fixture in RawGatewayEventTest to use a realistic payload shape: replace the top-level/flat 'user_id' field under 'd' with a nested 'user' object (e.g., 'd' => (object) ['guild_id' => ..., 'user' => (object) ['id' => '987654321', 'username' => 'botuser', ...], 'roles' => [], ...]) and remove the legacy 'user_id' key; then update any assertions in the test to assert against the nested d.user.id (or equivalent extraction logic) instead of d.user_id so the test validates the real Discord payload shape for GUILD_MEMBER_ADD.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@app-modules/bot-discord/src/Actions/VoiceChannel/PersistVoiceStateAction.php`:
- Around line 23-26: The ExternalIdentity lookup in PersistVoiceStateAction is
missing a provider filter, so add a condition to the ExternalIdentity::query()
that also matches the Discord provider (e.g., ->where('provider', 'discord') or
use the project constant like ExternalIdentity::PROVIDER_DISCORD) alongside the
existing where('model_type', (new Tenant)->getMorphClass()) and
where('external_account_id', (string) $state->guild_id) to ensure only Discord
identities resolve to the tenant.
In `@app-modules/bot-discord/src/Events/RawGatewayEvent.php`:
- Line 20: The 'user_id' extraction currently only checks $payload->d->user_id
and $payload->d->author->id, so events where the user is nested at
$payload->d->user->id (e.g., GUILD_MEMBER_ADD) are missed; update the 'user_id'
mapping expression (the array key 'user_id' that reads from $payload) to also
check for $payload->d->user->id—use isset() or null-coalescing checks in the
same order (d->user_id, d->user->id, d->author->id) so all Discord event shapes
produce a user_id.
---
Nitpick comments:
In `@app-modules/bot-discord/src/Events/RawGatewayEvent.php`:
- Line 22: The raw payload is being persisted as-is ('payload' =>
json_decode(json_encode($payload->d), true) in RawGatewayEvent.php) so you must
add a retention/pruning strategy before production: add a created_at timestamp
to the discord_event_logs records (via migration), make created_at indexed,
introduce a configurable retention_days setting, and implement either a periodic
cleanup task (cron/queue job) that deletes records older than retention_days or
use DB partitioning by date with automated partition drops; update the
RawGatewayEvent insert flow to populate created_at and ensure the cleanup job or
partitioning is documented and enabled in deployment config.
In
`@app-modules/bot-discord/tests/Feature/Actions/VoiceChannel/PersistVoiceStateActionTest.php`:
- Around line 26-48: Add a regression test in PersistVoiceStateActionTest
(inside the existing beforeEach/setup or a new it/test case) that ensures tenant
lookup uses the provider: create an ExternalIdentity with the same guildId value
but with a different provider (e.g., IdentityProvider::Slack) for the tenant,
then trigger the PersistVoiceStateAction (or the test flow that persists Voice)
using the Discord provider and assert that no Voice record is persisted for that
mismatched provider (or that the persisted Voice is only created when an
ExternalIdentity for IdentityProvider::Discord exists). Reference the test setup
variables ($this->tenant, $this->guildId, ExternalIdentity factory,
IdentityProvider constants, and the PersistVoiceStateAction) so the new case
mirrors the existing setup but swaps the provider to verify provider-specific
tenant resolution.
In `@app-modules/bot-discord/tests/Feature/Events/RawGatewayEventTest.php`:
- Around line 14-21: Update the GUILD_MEMBER_ADD test fixture in
RawGatewayEventTest to use a realistic payload shape: replace the top-level/flat
'user_id' field under 'd' with a nested 'user' object (e.g., 'd' => (object)
['guild_id' => ..., 'user' => (object) ['id' => '987654321', 'username' =>
'botuser', ...], 'roles' => [], ...]) and remove the legacy 'user_id' key; then
update any assertions in the test to assert against the nested d.user.id (or
equivalent extraction logic) instead of d.user_id so the test validates the real
Discord payload shape for GUILD_MEMBER_ADD.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro
Run ID: 7182c5b7-4807-4b18-85e7-de848ad1efeb
📒 Files selected for processing (9)
app-modules/bot-discord/src/Actions/VoiceChannel/PersistVoiceStateAction.phpapp-modules/bot-discord/src/BotDiscordServiceProvider.phpapp-modules/bot-discord/src/Events/DynamicVoiceEvent.phpapp-modules/bot-discord/src/Events/RawGatewayEvent.phpapp-modules/bot-discord/tests/Feature/Actions/VoiceChannel/PersistVoiceStateActionTest.phpapp-modules/bot-discord/tests/Feature/Events/RawGatewayEventTest.phpapp-modules/integration-discord/database/migrations/2026_05_19_155732_create_discord_event_logs_table.phpapp-modules/integration-discord/src/Models/DiscordEventLog.phpapp-modules/integration-discord/tests/Feature/Models/DiscordEventLogTest.php
Clintonrocha98
previously approved these changes
May 19, 2026
The created_at timestamp already captures when the event was recorded, making occurred_at redundant.
Add fallback chain for user_id (user_id → author.id → user.id) and channel_id (channel_id → id) to cover events like GUILD_MEMBER_UPDATE and VOICE_CHANNEL_STATUS_UPDATE that use non-standard field names.
…eStateAction Ensures only Discord identities are matched when resolving the tenant from guild_id, preventing false matches from other providers.
gvieira18
approved these changes
May 19, 2026
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app-modules/bot-discord/src/Events/RawGatewayEvent.php`:
- Line 22: The current line 'payload' => (array) $payload in RawGatewayEvent.php
persists the entire gateway payload and risks storing PII; modify the code that
builds the payload (e.g., the RawGatewayEvent constructor or its
toArray/serialize method) to apply a sanitize/redaction step before persisting:
implement and call a helper like sanitizePayload/redactSensitiveFields that
strips or hashes sensitive keys (e.g., message.content, author, member, token,
email, username, avatar, IPs, webhook_id) or applies an explicit allowlist of
safe fields, and add metadata for retention (e.g., retention_ttl or
redaction_version) so downstream storage can enforce a TTL rather than keeping
raw data indefinitely.
- Around line 17-23: Wrap the DiscordEventLog::query()->create([...]) call in a
try/catch so a transient DB failure doesn't abort gateway processing; catch the
exception thrown when creating the record (use the same surrounding
scope/variable names: $payload, $payload->t, $payload->d->guild_id), log an
error containing the exception message and contextual fields event_type
($payload->t) and guild_id ($payload->d->guild_id ?? null), and do not
rethrow—fail silently for persistence while allowing the gateway listener to
continue.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro
Run ID: 9eb63bf8-134a-492b-8ba3-dfb517925724
📒 Files selected for processing (5)
app-modules/bot-discord/src/Events/RawGatewayEvent.phpapp-modules/bot-discord/tests/Feature/Events/RawGatewayEventTest.phpapp-modules/integration-discord/database/migrations/2026_05_19_155732_create_discord_event_logs_table.phpapp-modules/integration-discord/src/Models/DiscordEventLog.phpapp-modules/integration-discord/tests/Feature/Models/DiscordEventLogTest.php
🚧 Files skipped from review as they are similar to previous changes (2)
- app-modules/integration-discord/database/migrations/2026_05_19_155732_create_discord_event_logs_table.php
- app-modules/integration-discord/tests/Feature/Models/DiscordEventLogTest.php
1pride
approved these changes
May 19, 2026
danielhe4rt
added a commit
that referenced
this pull request
May 19, 2026
## Summary - Add **Marketing cluster** in admin panel sidebar with dedicated sub-navigation - Add **Discord Dashboard** page with real-time community analytics (messages, voice, users) - Move **Meeting Showcase** page into Marketing cluster - Add **6 query classes** (`ActivityPerDay`, `VoicePerDay`, `MessageHeatmap`, `VoiceHeatmap`, `TopChannels`, `PeriodStats`) with proper UTC→BRT timezone handling - Extend `x-he4rt::dashboard.bar-chart` component with backward-compatible stacked mode - Add `CONTEXT.md` and `ADR-0001` for panel-admin module ### Dashboard widgets - **Stats overview** — Filament StatsOverviewWidget with sparklines and rolling comparison - **Period breakdown** — 5 switchable views (summary with line chart + narrative, table, cards, stacked bars, donut) - **Activity timeline** — Chart.js line chart (messages + voice + users) - **Heatmap** — SVG day-of-week × hour matrix - **Activity by DOW** — horizontal bar chart with All/Msgs/Voice toggle (stacked mode) - **Top Channels** — ranked horizontal bar chart ### Performance - PeriodStats uses `CASE WHEN` conditional aggregates to fetch all sub-periods in 2 queries instead of N×3 - `once()` memoization prevents duplicate query execution within a request - Heatmap data queried once and shared between heatmap + DOW widgets ## Test plan - [ ] Navigate to `/admin/marketing/discord` and verify all widgets render with real data - [ ] Switch range selector (7d, 14d, 30d) and confirm data updates - [ ] Verify heatmap shows correct day/hour assignments (Mon 22h SP activity should show on Mon, not Tue) - [ ] Toggle Activity by DOW between All/Msgs/Voice modes - [ ] Switch period breakdown between all 5 views (summary, table, cards, bars, donut) - [ ] Verify Meeting Showcase still works at `/admin/marketing/meeting-showcase` - [ ] Check sidebar navigation shows Marketing cluster with back button when inside cluster <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Description This PR adds a Marketing cluster to the admin panel with a comprehensive Discord Dashboard providing real-time community analytics across messages, voice activity, and user engagement. Six query classes handle activity metrics with UTC↔BRT timezone conversion, the bar-chart component is extended with backward-compatible stacked mode support, and two new pages deliver Discord Dashboard analytics and Meeting Showcase participant visualization. Includes documentation, English/Portuguese-Brazil translations, and performance optimizations using conditional aggregates and memoization. ## References - Related: feat: voice event persistence and Discord gateway data lake (`#259`) - Related commit: edda785 (wire Discord Dashboard to real queries and fix PHPStan) ## Dependencies & Requirements - **Timezone**: America/Sao_Paulo (BRT/UTC) for all dashboard queries - **JavaScript**: Chart.js via CDN for timeline and chart visualization - **Frontend**: Alpine.js for reactive dashboard state management - **Export**: html2canvas library for PNG export functionality - **Database**: Added `occurred_at` timestamp (indexed) to discord_event_logs table - **Framework**: Filament admin panel components and pages ## Contributor Summary | Contributor | Lines Added | Lines Removed | Files Changed | |---|---|---|---| | PR Author | 2,250 | 49 | 25 | ## Changes Summary | File Path | Change Description | |---|---| | `.gitignore` | Added `/storage/private/dumps` directory to ignore patterns | | `CONTEXT-MAP.md` | Registered Panel Admin bounded context entry | | `app-modules/bot-discord/src/Events/RawGatewayEvent.php` | Refactored user_id extraction and payload JSON encoding to flatten structure | | `app-modules/bot-discord/tests/Feature/Events/RawGatewayEventTest.php` | Updated payload assertions for flattened structure; removed obsolete fallback tests | | `app-modules/he4rt/resources/views/components/dashboard/bar-chart.blade.php` | Extended with stacked bar support: added `stacked`, `legend`, `segments` props with conditional rendering | | `app-modules/integration-discord/database/migrations/2026_05_19_155732_create_discord_event_logs_table.php` | Added indexed `occurred_at` timestamp column | | `app-modules/integration-discord/src/Models/DiscordEventLog.php` | Added `occurred_at` to fillable and datetime casting | | `app-modules/integration-discord/tests/Feature/Models/DiscordEventLogTest.php` | Updated tests to populate `occurred_at` during record creation | | `app-modules/panel-admin/CONTEXT.md` | Added 64-line context document with module boundaries, glossary, and navigation rules | | `app-modules/panel-admin/docs/adr/0001-discord-dashboard-architecture.md` | Added ADR documenting dashboard design, query layer, widget architecture, and alternatives | | `app-modules/panel-admin/lang/en/marketing.php` | Added English translations for Marketing cluster navigation | | `app-modules/panel-admin/lang/pt_BR/marketing.php` | Added Portuguese-Brazil translations for Marketing cluster navigation | | `app-modules/panel-admin/resources/views/marketing/discord-dashboard.blade.php` | Created 666-line dashboard template with stats, timeline chart, heatmap, DOW activity, and top channels | | `app-modules/panel-admin/resources/views/pages/meeting-showcase.blade.php` | Created 471-line showcase page with customizable grid, presets, and PNG export functionality | | `app-modules/panel-admin/src/Marketing/MarketingCluster.php` | Created Filament cluster class with navigation configuration | | `app-modules/panel-admin/src/Marketing/Pages/Discord/Dashboard/DiscordDashboard.php` | Created page controller managing dashboard state, data loading, and formatting logic | | `app-modules/panel-admin/src/Marketing/Pages/Discord/Dashboard/Queries/ActivityPerDay.php` | Created query class for daily message count and unique user aggregation | | `app-modules/panel-admin/src/Marketing/Pages/Discord/Dashboard/Queries/MessageHeatmap.php` | Created query class for 7-day × 24-hour message heatmap with DOW/hour grouping | | `app-modules/panel-admin/src/Marketing/Pages/Discord/Dashboard/Queries/PeriodStats.php` | Created query class with conditional aggregates for multi-period statistics and sparklines | | `app-modules/panel-admin/src/Marketing/Pages/Discord/Dashboard/Queries/TopChannels.php` | Created query class for top 10 channels by message volume and distinct users | | `app-modules/panel-admin/src/Marketing/Pages/Discord/Dashboard/Queries/VoiceHeatmap.php` | Created query class for voice join activity heatmap with DOW/hour breakdown | | `app-modules/panel-admin/src/Marketing/Pages/Discord/Dashboard/Queries/VoicePerDay.php` | Created query class for daily voice join counts converted to hours (0.75 multiplier) | | `app-modules/panel-admin/src/Marketing/Pages/MeetingShowcasePage.php` | Created page with participant loader, Discord metadata extraction, and avatar URL fallback logic | | `app-modules/panel-admin/src/Marketing/Widgets/DiscordStatsWidget.php` | Created stats widget with metrics, trend indicators, and sparkline charts | | `app-modules/panel-admin/src/PanelAdminServiceProvider.php` | Updated service provider to register Marketing cluster pages and navigation builder | <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/he4rt/heartdevs.com/pull/260?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai -->
](https://app.coderabbit.ai/change-stack/he4rt/heartdevs.com/pull/260?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack){kind=link}
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VOICE_STATE_UPDATEevents but only updating Redis cache — voice activity stopped being recorded in the database since March 2026. AddedPersistVoiceStateActionthat resolves tenant/identity and writesjoined/leftrecords tovoice_messages, following the same pattern as the ETL importer.discord_event_logstable (jsonb payload) that captures every raw Discord gateway event via therawlistener, registered through Laracord'sAFTER_BOOThook. Acts as a generic data lake for future analytics.Files changed
bot-discord/Actions/VoiceChannel/PersistVoiceStateAction.phpbot-discord/Events/DynamicVoiceEvent.phpbot-discord/Events/RawGatewayEvent.phpbot-discord/BotDiscordServiceProvider.phpintegration-discord/Models/DiscordEventLog.phpintegration-discord/migrations/create_discord_event_logs_table.phpTest plan
PersistVoiceStateAction(join, leave, channel switch, mute skip, missing tenant/identity)RawGatewayEvent(dispatch persist, author fallback, non-dispatch skip, heartbeat skip)DiscordEventLogmodel (persist, nullable fields)Description
Adds persistence for Discord voice join/leave events via PersistVoiceStateAction (fixes regression that stopped DB writes) and introduces a Discord gateway data lake by logging raw gateway events to a new discord_event_logs table via a RawGatewayEvent listener registered in Laracord's AFTER_BOOT hook.
References
#259: feat: voice event persistence and Discord gateway data lake #259#145,#149,#171Dependencies & Requirements
Contributor Summary
Changes Summary
joined/leftevents to Voice model