Architecture

System Overview

Maverick is a lead-generation and client-management platform for insurance marketing. It processes contacts through a multi-stage pipeline: scraping, filtering, verification, batching, and upload to Email Bison campaigns.

graph TB
    Internet["Internet"]-->|HTTPS :443|Nginx["Nginx<br/>SSL Termination"]
    Nginx-->|:3000|Dashboard["Dashboard<br/>Next.js 14"]
    Nginx-->|:8000|API["API<br/>FastAPI + Uvicorn"]
    Nginx-->|:5555|Flower["Flower<br/>Task Monitor"]
    Nginx-->|:3003|Docs["Docs Site<br/>Fumadocs"]
 
    API-->Redis[("Redis<br/>Message Broker")]
    API-->Supabase[("Supabase<br/>PostgreSQL + Auth")]
 
    Redis-->W1["Scraping Worker<br/>Playwright Browser"]
    Redis-->W2["CSV Processing Worker"]
    Redis-->W3["Filtering Worker"]
    Redis-->W4["Verification Worker<br/>Debounce API"]
    Redis-->W5["Batching + Bison Worker"]
 
    W1-->|CSV files|W2
    W5-->Bison["Email Bison<br/>Campaign Platform"]
 
    API-->Pushgateway["Pushgateway"]-->Prometheus["Prometheus"]-->Grafana["Grafana"]

Docker Services

Production (`docker-compose.production.yml`)

Service	Container	Ports	Depends On	Health Check
redis	`maverick-redis-prod`	127.0.0.1:6379:6379	---	Yes
api	`maverick-api-prod`	127.0.0.1:8000:8000	redis	Yes
celery-scraping	`maverick-celery-scraping-prod`	---	redis, api	Yes
celery-csv	`maverick-celery-csv-prod`	---	redis, api	Yes
celery-filtering	`maverick-celery-filtering-prod`	---	redis, api	Yes
celery-verification	`maverick-celery-verification-prod`	---	redis, api	Yes
celery-webhooks	`maverick-celery-webhooks-prod`	---	redis, api	Yes
celery-batching	`maverick-celery-batching-prod`	---	redis, api	Yes
celery-sync	`maverick-celery-sync-prod`	---	redis, api	Yes
celery-beat	`maverick-celery-beat-prod`	---	redis, api	Yes
flower	`maverick-flower-prod`	127.0.0.1:5555:5555	redis, celery-filtering, celery-verification, celery-batching	Yes
pushgateway	`maverick-pushgateway-prod`	127.0.0.1:9091:9091	---	---
alertmanager	`maverick-alertmanager-prod`	127.0.0.1:9093:9093	---	Yes
prometheus	`maverick-prometheus-prod`	127.0.0.1:9090:9090	api, pushgateway, alertmanager	Yes
grafana	`maverick-grafana-prod`	127.0.0.1:4000:3000	prometheus	Yes
node-exporter	`maverick-node-exporter-prod`	---	---	---
frontend	`maverick-frontend-prod`	127.0.0.1:3000:3000	api	Yes
docs	`maverick-docs-prod`	127.0.0.1:3003:3003	---	Yes
client-portal	`maverick-client-portal-prod`	127.0.0.1:3004:3004	---	Yes
status	`maverick-status-prod`	127.0.0.1:3005:3005	---	Yes
nginx	`maverick-nginx-prod`	80:80, 443:443	api, frontend	---
certbot	`maverick-certbot`	---	---	---

Worker Configuration (Production):

Worker	Queue(s)	Concurrency	Max Tasks/Child
celery-scraping	`scraping`	2	25
celery-csv	`csv_processing`	4	100
celery-filtering	`filtering`	2	100
celery-verification	`verification`	1	500
celery-webhooks	`webhooks,email_bison,email_events`	4	200
celery-batching	`batching,bison_uploads`	3	50
celery-sync	`sync_periodic`	2	50
celery-beat	`default`	1	---
flower	`default`	1	---

Local (`docker-compose.local.yml`)

Service	Container	Ports	Depends On	Health Check
redis	`maverick-redis-local`	6379:6379	---	Yes
api	`maverick-api-local`	8000:8000	redis	Yes
celery-scraping	`maverick-celery-scraping-local`	---	redis, api	---
celery-csv	`maverick-celery-csv-local`	---	redis, api	---
celery-filtering	`maverick-celery-filtering-local`	---	redis, api	---
celery-verification	`maverick-celery-verification-local`	---	redis, api	---
celery-batching	`maverick-celery-batching-local`	---	redis, api	---
celery-beat	`maverick-celery-beat-local`	---	redis, api	---
flower	`maverick-flower-local`	5555:5555	redis	---
pushgateway	`maverick-pushgateway-local`	9091:9091	---	---
prometheus	`maverick-prometheus-local`	9090:9090	api, pushgateway	Yes
grafana	`maverick-grafana-local`	4000:3000	prometheus	Yes
landing	`maverick-landing-local`	3000:3000	---	Yes
dashboard	`maverick-dashboard-local`	3001:3001	api	Yes
client-portal	`maverick-client-portal-local`	3004:3004	api	Yes
docs	`maverick-docs-local`	3003:3003	---	Yes
status	`maverick-status-local`	3005:3005	---	---
node-exporter	`maverick-node-exporter-local`	---	---	---

Worker Configuration (Local):

Worker	Queue(s)	Concurrency	Max Tasks/Child
celery-scraping	`scraping`	1	10
celery-csv	`csv_processing`	2	50
celery-filtering	`filtering`	1	50
celery-verification	`verification`	2	25
celery-batching	`batching,email_bison`	1	50
celery-beat	`default`	1	---
flower	`default`	1	---

Celery Queue Architecture

Task Routing

Task Pattern	Queue
`app.workers.tasks.scraping.*`	`scraping`
`app.workers.tasks.processing.*`	`csv_processing`
`app.workers.tasks.filtering.*`	`filtering`
`app.workers.tasks.verification.*`	`verification`
`app.workers.tasks.batching.*`	`batching`
`app.workers.tasks.webhook_processing.*`	`webhooks`
`webhook.*`	`webhooks`
`sentiment.*`	`webhooks`
`auto_reply.*`	`webhooks`
`app.workers.tasks.email_bison_integration.*`	`bison_uploads`
`email_bison.upload_batch`	`bison_uploads`
`email_bison.generate_and_upload_batches`	`bison_uploads`
`email_bison.*`	`sync_periodic`
`daily_metrics.*`	`sync_periodic`
`infra_orders.*`	`sync_periodic`
`daily_infra_reports.*`	`sync_periodic`
`verification_maintenance.*`	`sync_periodic`
`options`	`sync_periodic`

Autodiscovered Task Modules

app.workers.tasks.scraping
app.workers.tasks.processing
app.workers.tasks.filtering
app.workers.tasks.verification
app.workers.tasks.batching
app.workers.tasks.email_bison_integration
app.workers.tasks.webhook_processing
app.workers.tasks.campaign_sync
app.workers.tasks.tag_burnt_accounts
app.workers.tasks.quarantine_burnt_accounts
app.workers.tasks.sentiment
app.workers.tasks.auto_reply
app.workers.tasks.daily_metrics
app.workers.tasks.infrastructure_orders
app.workers.tasks.hall_of_fame_backfill
app.workers.tasks.master_db_fill
app.workers.tasks.daily_infra_reports
app.workers.tasks.verification_maintenance
app.workers.tasks.task_notifications

Beat Schedule (Periodic Tasks)

Name	Task	Schedule	Queue
sync-bison-full-every-2h	`email_bison.full_sync`	`crontab(minute=0, hour='*/2')`	`sync_periodic`
sync-campaigns-hourly	`email_bison.sync_campaigns_all`	`crontab(minute=15)`	`sync_periodic`
classify-sentiment-hourly	`sentiment.classify_batch`	`crontab(minute=0)`	`default`
daily-client-metrics	`daily_metrics.generate_daily_client_metrics`	`crontab(hour=0, minute=5)`	`sync_periodic`
reset-api-counters-daily	`daily_metrics.reset_daily_api_counters`	`crontab(hour=0, minute=1)`	`sync_periodic`
daily-revenue-snapshot	`daily_metrics.generate_daily_revenue_snapshot`	`crontab(hour=0, minute=15)`	`sync_periodic`
reconcile-daily-revenue-snapshots-hourly	`daily_metrics.reconcile_daily_revenue_snapshots`	`crontab(minute=30)`	`default`
cleanup-slack-notifications-weekly	`daily_metrics.cleanup_old_slack_notifications`	`crontab(hour=3, minute=0, day_of_week=0)`	`sync_periodic`
refresh-pipeline-stats-cache	`daily_metrics.refresh_pipeline_stats_cache`	`crontab(minute='*/5')`	`sync_periodic`
sync-daily-stats-every-30m	`email_bison.sync_daily_stats`	`crontab(minute='*/30')`	`sync_periodic`
check-mx-records-daily	`daily_metrics.check_mx_records_all`	`crontab(hour=3, minute=0)`	`sync_periodic`
reconcile-interested-leads	`email_bison.reconcile_interested_leads`	`crontab(minute=45, hour='*/4')`	`sync_periodic`
check-order-fulfillment-every-2h	`infra_orders.check_fulfillment_all`	`crontab(minute=45, hour='*/2')`	`sync_periodic`
verify-order-warmup-every-4h	`infra_orders.verify_warmup_all`	`crontab(minute=0, hour='*/4')`	`sync_periodic`
check-warmup-completion-daily	`infra_orders.check_warmup_completion`	`crontab(hour=6, minute=0)`	`sync_periodic`
track-ramp-progress-daily	`infra_orders.track_ramp_progress`	`crontab(hour=8, minute=0)`	`sync_periodic`
daily-infra-disconnected-report	`daily_infra_reports.send_disconnected_csvs`	`crontab(hour=8, minute=5)`	`sync_periodic`
infra-launch-ready-alert-daily	`daily_infra_reports.launch_ready_alert`	`crontab(hour=8, minute=10)`	`sync_periodic`
report-queue-depths	`daily_metrics.report_queue_depths`	`crontab(unknown)`	`sync_periodic`

Worker Defaults

Setting	Value
`task_soft_time_limit`	`21600`
`task_time_limit`	`28800`
`worker_prefetch_multiplier`	`1`
`worker_max_tasks_per_child`	`50`

Contact Pipeline

The core business process: acquire contacts, clean them, and upload to campaigns.

graph LR
    XD["Xpressdocs<br/>X-Dates Platform"]-->|"Playwright<br/>scraping queue"|Raw[("raw_contacts")]
    Raw-->|"csv_processing queue"|Parsed["Parsed + Inserted"]
    Parsed-->|"filtering queue"|Filtered[("filtered_contacts")]
    Filtered-->|"verification queue<br/>Debounce API"|Verified[("verified_contacts")]
    Verified-->|"batching queue"|Batches[("contact_batches")]
    Batches-->|"email_bison queue"|Bison["Email Bison<br/>Campaigns"]

Each stage is a separate Celery queue with dedicated workers. Stages communicate via database tables (not direct task chaining), so any stage can be re-run independently.

Port Map

Service	Container Port	Host Port	Environment
API	8000	127.0.0.1:8000	production
Dashboard	3000	127.0.0.1:3000	production
Beta API	8000	127.0.0.1:8001	beta
Beta Dashboard	3001	127.0.0.1:3002	beta
Docs Site	3003	127.0.0.1:3003	production
Redis	6379	127.0.0.1:6379	shared
Flower	5555	127.0.0.1:5555	production
Grafana	3000	127.0.0.1:4000	production
Prometheus	9090	127.0.0.1:9090	production
Pushgateway	9091	127.0.0.1:9091	production

Architecture

On this page