Architecture

System Overview

Maverick is a lead-generation and client-management platform for insurance marketing. It processes contacts through a multi-stage pipeline: scraping, filtering, verification, batching, and upload to Email Bison campaigns.

graph TB
    Internet["Internet"]-->|HTTPS :443|Nginx["Nginx<br/>SSL Termination"]
    Nginx-->|:3000|Dashboard["Dashboard<br/>Next.js 14"]
    Nginx-->|:8000|API["API<br/>FastAPI + Uvicorn"]
    Nginx-->|:5555|Flower["Flower<br/>Task Monitor"]
    Nginx-->|:3003|Docs["Docs Site<br/>Fumadocs"]
 
    API-->Redis[("Redis<br/>Message Broker")]
    API-->Supabase[("Supabase<br/>PostgreSQL + Auth")]
 
    Redis-->W1["Scraping Worker<br/>Playwright Browser"]
    Redis-->W2["CSV Processing Worker"]
    Redis-->W3["Filtering Worker"]
    Redis-->W4["Verification Worker<br/>Debounce API"]
    Redis-->W5["Batching + Bison Worker"]
 
    W1-->|CSV files|W2
    W5-->Bison["Email Bison<br/>Campaign Platform"]
 
    API-->Pushgateway["Pushgateway"]-->Prometheus["Prometheus"]-->Grafana["Grafana"]

Docker Services

Production (`docker-compose.production.yml`)

Service	Container	Ports	Depends On	Health Check
redis	`maverick-redis-prod`	127.0.0.1:6379:6379	---	Yes
api	`maverick-api-prod`	127.0.0.1:8000:8000	redis	Yes
celery-scraping	`maverick-celery-scraping-prod`	---	redis, api	---
celery-csv	`maverick-celery-csv-prod`	---	redis, api	---
celery-filtering	`maverick-celery-filtering-prod`	---	redis, api	---
celery-verification	`maverick-celery-verification-prod`	---	redis, api	---
celery-batching	`maverick-celery-batching-prod`	---	redis, api	---
flower	`maverick-flower-prod`	127.0.0.1:5555:5555	redis, celery-filtering, celery-verification, celery-batching	---
pushgateway	`maverick-pushgateway-prod`	127.0.0.1:9091:9091	---	---
prometheus	`maverick-prometheus-prod`	127.0.0.1:9090:9090	api, pushgateway	Yes
grafana	`maverick-grafana-prod`	127.0.0.1:4000:3000	prometheus	Yes
frontend	`maverick-frontend-prod`	127.0.0.1:3000:3000	api	Yes
docs	`maverick-docs-prod`	127.0.0.1:3003:3003	---	Yes
nginx	`maverick-nginx-prod`	80:80, 443:443	api, frontend	---
certbot	`maverick-certbot`	---	---	---

Worker Configuration (Production):

Worker	Queue(s)	Concurrency	Max Tasks/Child
celery-scraping	`scraping`	2	25
celery-csv	`csv_processing`	4	100
celery-filtering	`filtering`	2	100
celery-verification	`verification`	4	50
celery-batching	`batching,email_bison`	2	100
flower	`default`	1	---

Local (`docker-compose.local.yml`)

Service	Container	Ports	Depends On	Health Check
redis	`maverick-redis-local`	6379:6379	---	Yes
api	`maverick-api-local`	8000:8000	redis	Yes
celery-scraping	`maverick-celery-scraping-local`	---	redis, api	---
celery-csv	`maverick-celery-csv-local`	---	redis, api	---
celery-filtering	`maverick-celery-filtering-local`	---	redis, api	---
celery-verification	`maverick-celery-verification-local`	---	redis, api	---
celery-batching	`maverick-celery-batching-local`	---	redis, api	---
flower	`maverick-flower-local`	5555:5555	redis	---
pushgateway	`maverick-pushgateway-local`	9091:9091	---	---
prometheus	`maverick-prometheus-local`	9090:9090	api, pushgateway	Yes
grafana	`maverick-grafana-local`	4000:3000	prometheus	Yes
landing	`maverick-landing-local`	3000:3000	---	Yes
dashboard	`maverick-dashboard-local`	3001:3001	api	Yes
client-portal	`maverick-client-portal-local`	3004:3004	api	Yes
docs	`maverick-docs-local`	3003:3003	---	Yes

Worker Configuration (Local):

Worker	Queue(s)	Concurrency	Max Tasks/Child
celery-scraping	`scraping`	1	10
celery-csv	`csv_processing`	2	50
celery-filtering	`filtering`	1	50
celery-verification	`verification`	2	25
celery-batching	`batching,email_bison`	1	50
flower	`default`	1	---

Celery Queue Architecture

Task Routing

Task Pattern	Queue
`app.workers.tasks.scraping.*`	`scraping`
`app.workers.tasks.processing.*`	`csv_processing`
`app.workers.tasks.filtering.*`	`filtering`
`app.workers.tasks.verification.*`	`verification`
`app.workers.tasks.batching.*`	`batching`
`app.workers.tasks.email_bison_integration.*`	`email_bison`
`app.workers.tasks.webhook_processing.*`	`email_bison`
`email_bison.*`	`email_bison`
`webhook.*`	`email_bison`
`sentiment.*`	`email_bison`

Autodiscovered Task Modules

app.workers.tasks.scraping
app.workers.tasks.processing
app.workers.tasks.filtering
app.workers.tasks.verification
app.workers.tasks.batching
app.workers.tasks.email_bison_integration
app.workers.tasks.webhook_processing
app.workers.tasks.campaign_sync
app.workers.tasks.sentiment

Beat Schedule (Periodic Tasks)

Name	Task	Schedule	Queue
sync-bison-full-every-15-min	`email_bison.full_sync`	`crontab(minute='*/15')`	`email_bison`
sync-campaigns-every-30-min	`email_bison.sync_campaigns_all`	`crontab(minute='*/30')`	`email_bison`
classify-sentiment-hourly	`sentiment.classify_batch`	`crontab(minute=0)`	`default`

Worker Defaults

Setting	Value
`task_soft_time_limit`	`21600`
`task_time_limit`	`28800`
`worker_prefetch_multiplier`	`1`
`worker_max_tasks_per_child`	`50`

Contact Pipeline

The core business process: acquire contacts, clean them, and upload to campaigns.

graph LR
    XD["Xpressdocs<br/>X-Dates Platform"]-->|"Playwright<br/>scraping queue"|Raw[("raw_contacts")]
    Raw-->|"csv_processing queue"|Parsed["Parsed + Inserted"]
    Parsed-->|"filtering queue"|Filtered[("filtered_contacts")]
    Filtered-->|"verification queue<br/>Debounce API"|Verified[("verified_contacts")]
    Verified-->|"batching queue"|Batches[("contact_batches")]
    Batches-->|"email_bison queue"|Bison["Email Bison<br/>Campaigns"]

Each stage is a separate Celery queue with dedicated workers. Stages communicate via database tables (not direct task chaining), so any stage can be re-run independently.

Port Map

Service	Container Port	Host Port	Environment
API	8000	127.0.0.1:8000	production
Dashboard	3000	127.0.0.1:3000	production
Beta API	8000	127.0.0.1:8001	beta
Beta Dashboard	3001	127.0.0.1:3002	beta
Docs Site	3003	127.0.0.1:3003	production
Redis	6379	127.0.0.1:6379	shared
Flower	5555	127.0.0.1:5555	production
Grafana	3000	127.0.0.1:4000	production
Prometheus	9090	127.0.0.1:9090	production
Pushgateway	9091	127.0.0.1:9091	production

Architecture

On this page