Skip to main content

BizOSaaS Post-Deployment Operational Checklist & Workflows

Document Status: Active
Target System: BizOSaaS Platform (Multi-tenant AI Agentic Marketing & E-Commerce Platform)
Last Updated: July 25, 2026

This operational document outlines the mandatory post-deployment safety loops, knowledge base hygiene, observability standards, disaster recovery procedures, and automated testing routines required to ensure system stability, multi-tenant security, and self-optimizing performance.

1. Guardrails & Autonomous Safety Loops​

When 31+ agents operate autonomously and continuously learn across client assets, robust boundaries prevent run-away AI behavior or invalid business operations.

1.1 Human-In-The-Loop (HITL) & Confidence Thresholds​

  • Confidence Matrix:
    • High Confidence (Score at least 0.90): Direct execution allowed (e.g., standard blog post scheduling, social media drafts).
    • Medium Confidence (Score between 0.75 and 0.89): Enqueue for HITL review in the client portal.
    • Low Confidence (Score under 0.75): Escalate to fallback routine or cancel execution with detailed log.
  • Critical Asset Gates:
    • Pricing adjustments, ad spend allocation (above $100), external domain DNS edits, and live store catalog updates MUST ALWAYS require explicit HITL approval regardless of score.
  • Circuit Breakers:
    • Automatically pause an agent if it generates more than 3 consecutive errors or receives evaluation scores under 0.70 within a 1-hour window.

1.2 Hallucination & Quality Drift Monitoring (LLM-as-a-Judge)​

  • Evaluation Sampling:
    • Sample 5% of all agent-generated content and execution plans daily using an asynchronous BullMQ evaluation job (eval-judge.worker.ts).
  • Metrics Evaluated:
    • Factuality & Grounding: Context recall against client RAG/KAG vector stores.
    • Brand Tone Compliance: Adherence to tenant tone-of-voice directives.
    • Safety & Moderation: Screening against restricted categories (weapons, adult, fraud, drugs, harassment).
  • Drift Alerts:
    • Trigger automatic admin alerts when a tenant's agent quality score drops below 85% over a 7-day rolling window.

1.3 Programmatic Fallbacks & Rate Limits​

  • Model Fallbacks:
    • Primary Model (e.g., GPT-4o / Claude 3.5 Sonnet) timeout or 5xx $\rightarrow$ Fallback to Secondary Model (e.g., Gemini 1.5 Flash / Groq / Hermes-3 70B).
  • Per-Tenant Quotas & Token Caps:
    • Enforce Redis-backed token and API request rate limits per tenant plan tier (SMALL / PARTNER / ENTERPRISE).
    • Block runaway loops before API budgets are breached.

2. Knowledge Base & Vector Store Hygiene​

RAG (Retrieval-Augmented Generation) and KAG (Knowledge-Augmented Generation) demand pristine vector and graph data boundaries.

2.1 Vector DB Maintenance & Pruning​

  • Chunking & Invalidation:
    • Automatically re-index vector embeddings upon source updates (e.g., Shopify product updates, site re-crawls).
    • Purge stale embeddings (documents deleted or replaced over 30 days ago).
  • Deduplication:
    • Run weekly cosine similarity scans across tenant vector namespaces to remove duplicate embeddings (similarity over 0.98).

2.2 Knowledge-Graph (KAG) Constraint Enforcement​

  • Business Rule Validation:
    • Enforce verified business rules in PostgreSQL graph representations (e.g., brand guidelines, maximum discount percentages, restricted keyphrases).
  • Context Grounding:
    • Ensure agents cross-reference graph nodes before generating strategy outputs to avoid hallucinating policy or product specs.

2.3 Strict Multi-Tenant Isolation Audits​

  • PostgreSQL RLS:
    • Enforce FORCE ROW LEVEL SECURITY on all tenant-isolated tables.
    • Test quarterly using automated tenant cross-query attempts (1.1-tenant-isolation.test.ts).
  • Vector Store Namespacing:
    • Embed tenant_id metadata into every pgvector embedding payload. Ensure queries rigidly filter on tenant_id = current_tenant.

3. Observability, Logging & Agent Tracing​

Autonomous multi-agent platforms require deep distributed tracing across workers, BullMQ queues, and LLM APIs.

3.1 Distributed Agent Tracing (SigNoz / OpenTelemetry)​

  • Trace Propagation:
    • Propagate trace IDs from HTTP requests -> Next.js API -> BullMQ job payloads -> Python AI service workers.
  • Span Granularity:
    • Track individual agent reasoning steps, vector searches, tool executions, and LLM calls.

3.2 Token & Cost Allocation Tracking​

  • Tenant Attributable Spend:
    • Instrument LLMCostTracker to record every token used (tenant_id, agent_name, model, input_tokens, output_tokens, cost_usd).
  • Stripe Metering Synchronization:
    • Flush usage metrics to Stripe Metering API every 60 minutes for accurate usage-based billing.

3.3 Queue & Dead-Letter Management​

  • BullMQ Monitoring:
    • Set alerts for queue depth over 500 or job wait times over 60 seconds.
  • Dead-Letter Queue (DLQ) Handling:
    • Move jobs failing after 3 retries to DLQ. Provide a one-click "Replay Task" option in the Admin Activity UI.

4. Operational Backup & Recovery​

Protecting system state and maintaining data integrity during continuous execution is critical.

4.1 Database & Redis Backups​

  • PostgreSQL (Payload CMS + Platform Data):
    • Daily automated full dumps to S3-compatible offsite backup storage.
    • Point-in-Time Recovery (PITR) enabled with 7-day WAL retention.
  • Redis Persistence:
    • RDB snapshots every 6 hours + AOF enabled for job state recovery.

4.2 Vector Store & Asset Backups​

  • pgvector & Media:
    • Daily snapshot of vector tables and Payload CMS uploads directory (/public/media).

4.3 Disaster Recovery Playbook​

  • RTO (Recovery Time Objective): under 1 hour.
  • RPO (Recovery Point Objective): under 15 minutes.
  • Recovery Steps:
    1. Spin up target KVM / Dokploy host.
    2. Restore PostgreSQL WAL / SQL dump and media storage.
    3. Pull Docker images from GHCR registry.
    4. Run infrastructure/scripts/deploy.sh and execute startup.mjs verification.

5. Automated E2E Regression & Operational Testing​

Ensuring continuous delivery does not degrade core platform workflows or tenant boundaries.

5.1 Synthetic E2E Workflows​

  • Target Suite: apps/e2e/tests/production/run-all.sh
  • Execution Phases:
    1. 1.1-tenant-isolation: Verifies CRM and AI data boundary enforcement under FORCE RLS.
    2. 1.2-load-test: Simulates concurrent API and agent task traffic.
    3. 1.3-chaos-worker: Simulates worker container restarts and verifies job recovery without duplication.
    4. 1.4-webhook-billing: Validates multi-provider webhook handling (Stripe, LemonSqueezy, Razorpay, Paddle, Dodo, TransactBridge).
    5. 1.5-operational-safety: Tests confidence scoring, HITL triggers, LLM cost tracking, and fallback execution.
    6. 2.x-manual-qa: Browser-based Playwright verification of dashboard UX and activity feeds.

6. Maintenance Routines Checklist​

FrequencyRoutine TaskResponsible Subsystem / Script
HourlyFlush Redis AI token usage to Stripe Meter APIapps/workers/src/billing.worker.ts
DailyPostgres DB full dump & PITR WAL syncBackup Cron / S3 sync script
DailySample & judge 5% agent outputs (LLM-as-a-Judge)apps/workers/src/eval.worker.ts
WeeklyVector DB deduplication & stale embedding pruneapps/workers/src/vector-hygiene.worker.ts
WeeklyStale page freshness audit & AEO share-of-voice scoreapps/workers/src/seo.worker.ts
MonthlyMulti-tenant RLS data leak penetration simulationapps/e2e/tests/production/1.1-tenant-isolation.test.ts