Skip to content

Feature: Freemius-to-Kit User Sync Engine #1

@ajaydsouza

Description

@ajaydsouza

Summary

Introduce a manual + automated sync engine that lets site owners pull existing Freemius users (free, paid, trial, etc.) for any configured plugin and push them into Kit forms/tags in bulk. This complements the current real-time webhook flow, which only catches new users after activation.

Motivation

Right now FreemKit only subscribes users who trigger a Freemius webhook after the integration is live. Every developer already has a history of installs, purchases, and trials sitting in Freemius that never reaches Kit. A sync feature closes this gap and gives users full control over which user cohorts get migrated into email marketing.

Proposed Architecture

1. New Core Class: Freemius_API_Client

File: includes/class-freemius-api-client.php

A lightweight wrapper around Freemius REST endpoints (re-using Freemius::build_fs_headers() for HMAC-SHA256 auth).

Method Purpose
get_users(string $plugin_id, array $filters, int $page) GET /v1/plugins/{id}/users.json with optional is_paying, is_trial, is_free, has_license query params.
get_licenses(string $plugin_id, array $filters, int $page) GET /v1/plugins/{id}/licenses.json (used to infer paid status if user endpoint is ambiguous).
get_user_count(string $plugin_id, array $filters) GET …/users.json?count=true for progress bars.
  • Pagination: Freemius returns users + pagination (total, page, pages). We iterate with WP_Cron or background AJAX batches.
  • Rate-limiting: Cache X-RateLimit-Remaining in a transient; pause/respect limits.
  • Error handling: Uses the same exponential-backoff pattern as Webhook_Handler (max 3 retries).

2. New Core Class: Sync_Engine

File: includes/class-sync-engine.php

Orchestrates the entire sync pipeline.

class Sync_Engine {
    public function start_sync( string $plugin_id, array $user_types, array $target_forms, array $target_tags );
    public function process_batch( string $job_id, int $page );
    public function get_job_status( string $job_id );
    public function cancel_job( string $job_id );
}

Job state is stored in a new custom table {prefix}freemkit_sync_jobs (see schema below) or as an option array if we want to avoid a schema change in v1.1. Given the existing Database pattern, a new table is preferred for observability.

Sync Job Lifecycle:

  1. Admin initiates sync → Sync_Engine::start_sync() creates a pending job.
  2. A WP-Cron hook (freemkit_sync_batch) fires every 2 minutes while jobs are active.
  3. Each run fetches one Freemius page (max 25–50 users), maps each user to the local Subscriber model, upserts via Database::upsert_subscriber_by_email(), then calls Kit_API::subscribe_to_form() + tag application.
  4. Job transitions: pendingrunningcompleted | failed.
  5. On completion, an admin notice + email (optional) is sent.

3. Database Schema Addition

Add a sync_jobs table in Database::create_table() (bump db_version to 1.1.0).

CREATE TABLE {$wpdb->prefix}freemkit_sync_jobs (
    id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
    plugin_id varchar(50) NOT NULL DEFAULT '',
    user_types varchar(100) NOT NULL DEFAULT '',      -- JSON array: [\"free\",\"paid\",\"trial\"]
    target_forms text DEFAULT NULL,                    -- JSON array of Kit form IDs
    target_tags text DEFAULT NULL,                     -- JSON array of Kit tag IDs
    status varchar(20) NOT NULL DEFAULT 'pending',   -- pending|running|completed|failed|cancelled
    total_users int(11) NOT NULL DEFAULT 0,
    processed_users int(11) NOT NULL DEFAULT 0,
    failed_users int(11) NOT NULL DEFAULT 0,
    current_page int(11) NOT NULL DEFAULT 0,
    total_pages int(11) NOT NULL DEFAULT 0,
    error_message text DEFAULT NULL,
    created datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
    modified datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (id),
    KEY plugin_id (plugin_id),
    KEY status (status)
) {$charset_collate};

4. Admin UI: New "Sync" Sub-Screen

File: includes/admin/class-sync-admin.php

Add a new tab under Settings → FreemKit called Sync (or a dedicated admin page under the existing FreemKit menu).

Interface elements:

  • Plugin selector: Dropdown of already-configured Freemius plugins (read from Options_API::get_option('plugins')).
  • User type filter: Multi-check — Free, Paid, Trial, All. Defaults to the same free/paid split already configured for webhooks.
  • Kit Form/Tag overrides: Autocomplete fields for forms and tags (same UI pattern as the existing per-plugin repeater). If left empty, fall back to the plugin’s existing free/paid or global defaults.
  • Sync button: Triggers an AJAX call to Sync_Engine::start_sync() and returns a job_id.
  • Progress panel: Polls wp_ajax_freemkit_sync_status every 5 s. Shows a progress bar (processed / total), live count of successes/failures, and a cancel button.
  • History table: Lists past sync jobs (plugin, user types, status, processed/total, date). Click a row to expand error details.

5. AJAX + REST Endpoints

Endpoint Type Handler Purpose
wp_ajax_freemkit_start_sync AJAX Sync_Admin::ajax_start_sync() Validate nonce, kick off Sync_Engine.
wp_ajax_freemkit_sync_status AJAX Sync_Admin::ajax_sync_status() Return JSON job status for progress bar.
wp_ajax_freemkit_cancel_sync AJAX Sync_Admin::ajax_cancel_sync() Set job status to cancelled.
wp_ajax_freemkit_sync_history AJAX Sync_Admin::ajax_sync_history() Paginated job history for the admin table.

6. Integration with Existing Components

  • Database: Add add_sync_job(), update_sync_job(), get_sync_job(), get_sync_jobs() methods. Reuse upsert_subscriber_by_email() so synced users and webhook-created users live in the same table.
  • Kit_API: Reuse subscribe_to_form() and tag_subscriber(). No new Kit methods needed.
  • Webhook_Handler: After a successful sync, deduplication is already handled by Database::upsert_subscriber_by_email() (ON DUPLICATE KEY UPDATE).
  • Audit_Log: Log every sync batch start/end and per-user Kit API result under the existing audit-log table/transient mechanism.
  • Options_API: Store a sync_settings key (global defaults: batch size, max retries, auto-schedule interval) without breaking the existing single-array pattern.

7. Batch Processing & Performance

  • Batch size: 25 users per Freemius page (configurable via freemkit_sync_batch_size filter).
  • Cron cadence: freemkit_sync_batch every 2 minutes while an active job exists. If the job is running, it processes the next page; if cancelled or completed, the schedule is allowed to expire.
  • Time-limit safety: Each batch checks time() < start_time + 20s to avoid shared-host timeouts.
  • Memory safety: After every user, check memory_get_usage(). If > 80 % of WP_MEMORY_LIMIT, pause and reschedule.

8. User Type Mapping (Freemius → FreemKit)

Freemius user objects contain is_paying, is_trial, is_free booleans. The sync engine maps them to the same user_type values already stored in {prefix}freemkit_subscriber_events:

Freemius Field Sync Filter Option user_type value
is_paying == true Paid paid
is_trial == true Trial trial
is_free == true (and not paying/trial) Free free

When syncing "All", each user is processed once and tagged with the highest-priority type found (paid > trial > free).

9. Security & Permissions

  • All AJAX endpoints require manage_options (same as existing settings).
  • Nonce verification on every admin action.
  • Freemius API credentials are pulled from the encrypted settings store (same secret_key decryption used in Settings).

10. Filters & Hooks for Extensibility

// Before a user is synced to Kit.
apply_filters( 'freemkit_sync_user_before_subscribe', bool $should_sync, object $freemius_user, array $job );

// Modify the mapped subscriber object before DB upsert.
apply_filters( 'freemkit_sync_subscriber_data', array $data, object $freemius_user );

// After a single user is successfully synced.
do_action( 'freemkit_sync_user_complete', int $subscriber_id, object $freemius_user, array $kit_result );

// After the entire job finishes.
do_action( 'freemkit_sync_job_complete', int $job_id, array $job );

Open Questions

  1. Should we support scheduled recurring syncs (e.g., nightly delta sync) or keep it strictly manual for v1?
  2. Should the sync UI live on a new admin page (e.g., FreemKit → Sync) instead of as a settings tab, given the interactive progress-bar UX?
  3. Freemius API returns users in chunks; do we need a separate plan filter (e.g., only Pro-plan purchasers) or is is_paying granular enough?

Acceptance Criteria

  • Admin can select any configured plugin and one or more user types.
  • Admin can override the target Kit form/tag for the sync job.
  • Clicking "Start Sync" creates a background job with a live progress bar.
  • Each synced user is upserted into {prefix}freemkit_subscribers and subscribed to Kit.
  • Duplicate users (already in DB from webhooks) are updated, not duplicated.
  • Failed individual users are logged and do not abort the entire job.
  • Completed/failed jobs are viewable in a history table.
  • composer test passes before merge.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions