Skip to content
TilloTech Docs

ADR-2: EOM partner report job dispatch

Date: 17/06/2026 Author(s): Jamie Batabyal (grill session) Amended: 21/07/2026 (FOUND-1542)

Status

Accepted

Context

On 1 June 2026, several partners did not receive their scheduled end-of-month reports (Partner Funds Reconciliation and Partner Monthly Order History). Investigation pointed to slow runs and out-of-memory failures in the monolithic scheduler commands.

Both EOM commands (GeneratePartnerFundsReconciliationReport and GeneratePartnerMonthlyOrderHistoryReport) iterate all eligible partners in a single long-running PHP process. Each partner may require multiple currency-specific reports. The reconciliation command already raises memory_limit to 500M in production and still risks failure when processing high-volume partners or when memory accumulates across the partner loop. When the process dies mid-run, every subsequent partner in that iteration misses their report.

A similar problem was solved for brand reconciliation reports: GenerateBrandReconciliationReport dispatches SendBrandReconciliationReport jobs (one per brand/brand-company) onto the reports queue, with idempotency locks and isolated memory per job.

Order History has a second generation path (Reporting API) that is not memory-bound; only local generation needs queuing. The Reporting API path uses a different delivery contract (per-user HTTP request + Report Centre ready mail, soft failures) and must not share the local generate/send job shape.

Decision

We will refactor scheduled EOM partner report generation to a dispatch-and-queue model:

  1. The scheduler command becomes a thin dispatcher: timezone gating, partner/currency eligibility, FileExportCommand creation, and job dispatch only.
  2. Each partner + currency report is handled by a dedicated queued job on Queue::REPORTS, with ShouldBeUnique, WithoutOverlapping, and tries = 3 (following SendBrandReconciliationReport).
  3. Each job runs in a fresh worker process with a 500M memory limit in production.
  4. FileExportCommand.status_code = 0 means dispatch completed successfully; per-report delivery is tracked via file_exports rows. For Order History, file_count counts local jobs dispatched only (Reporting API requests are uncounted).
  5. Permanent job failures (after retries) trigger a Slack notification via a dedicated webhook (rc.slack_webhooks.report_failure); channel is configured at deploy time via environment variable.
  6. Query/memory optimisation inside reconciliation generation (e.g. replacing ->get() on full-month sales) is deferred until Slack alerts prove a single partner+currency job still OOMs.

Idempotency lock TTL (shared):

Both partner EOM send jobs read uniqueFor from config('rc.partner_reports.idempotency_lock_ttl'), backed by env PARTNER_REPORT_LOCK_TTL (default 1200). This replaced the funds-only PARTNER_FUNDS_RECONCILIATION_REPORT_LOCK_TTL / rc.partner_funds_reconciliation.idempotency_lock_ttl path (hard cut).

Prefixed uniqueId convention:

Each partner EOM job uses a stable report-type prefix so uniqueness keys never collide across report types:

JobPrefix constantExample key
SendPartnerFundsReconciliationReportpartner-funds-reconciliationpartner-funds-reconciliation:{partnerSlug}:{currency}:{Y-m-d}:{Y-m-d}
SendPartnerMonthlyOrderHistoryReportpartner-monthly-order-historypartner-monthly-order-history:{partnerSlug}:{currency}:{Y-m-d}:{Y-m-d}

WithoutOverlapping is keyed on the same uniqueId().

Failure Slack payload:

Jobs call SendReportFailureSlackNotification with ReportFailureData (partner + currency scoped). Generalising that DTO for non-partner consumers is out of scope here — tracked as FOUND-1574.

Delivery split:

  • FOUND-1382 implements this for Partner Funds Reconciliation plus shared infrastructure (Slack notifier, job patterns).
  • FOUND-1542 applies the same pattern to Partner Monthly Order History local generation only: Request DTO + Generator + RecipientResolver, SendPartnerMonthlyOrderHistoryReport, and a dual-path dispatcher. The Reporting API path remains inline in GeneratePartnerMonthlyOrderHistoryReport (not queued through the local send job).

Consequences

Positive

  • OOM blast radius is limited to one partner+currency job instead of the entire EOM run
  • Failed jobs are visible via systemFailedJobs and Slack without waiting for partner complaints
  • Consistent with the established brand reconciliation and denomination breakdown dispatch patterns
  • Workers can process report jobs in parallel (subject to queue capacity)
  • Order History reuses shared notifier, lock TTL, and prefixed uniqueness conventions from FOUND-1382 / FOUND-1542

Negative

  • FileExportCommand success no longer implies all partners received reports; ops must use file_exports and Slack for delivery confirmation
  • More jobs on the reports queue on the 1st of each month; queue worker capacity must be adequate
  • Reconciliation / Order History command logic must be extracted from the monolithic command into a job (or shared generator), which is a non-trivial refactor
  • Dispatcher finishes before jobs complete, so end-to-end run duration is harder to observe from a single command log entry
  • Single-partner OOM within a 500M job remains possible until a follow-up query optimisation if alerts prove it necessary
  • Order History retains two in-command paths (queued local vs inline Reporting API), so dual-path complexity remains until a dedicated API-path design is warranted