Skip to content

Monitoring & Dashboard — Technical Documentation

Purpose: Documents the marketplace's usage tracking, cost calculation, and dashboard aggregation system across all platform features.

1. Overview

The monitoring system tracks per-user resource consumption across 5 feature categories and aggregates the data into a single dashboard endpoint. All tracking is marketplace-side (MongoDB) and designed to never break the API response on failure.

┌─ Tracking Layer (usage_tracker.py) ──────────────────────────────┐
│                                                                   │
│  track_inference_usage()    → llm_dashboard_collection            │
│  track_vdb_operation()      → vdb_logs_collection                 │
│  (direct insert)            → agentic_dashboard_collection        │
│  record_data_pipeline_usage()  → data_pipeline_usage_collection   │
│  record_notebook_usage()       → data_pipeline_usage_collection   │
│  record_streaming_usage()      → data_pipeline_usage_collection   │
│                                                                   │
└───────────────────────────────────────────────────────────────────┘
          ▲
          │ called from route handlers after each operation
          │
┌─ Aggregation Layer (dashboard.py) ───────────────────────────────┐
│                                                                   │
│  GET /api/dashboard/general                                       │
│  ├── LLM: tokens, cost, daily breakdown per model                 │
│  ├── VDB: operation count, cost                                   │
│  ├── Agentic: execution count × rate                              │
│  ├── Pipeline: total cost, tokens, GPU-hours, per-feature split   │
│  └── Pie chart + daily line chart from real tracked data          │
│                                                                   │
└───────────────────────────────────────────────────────────────────┘
          ▲
          │
┌─ Frontend (Dashboard.jsx) ───────────────────────────────────────┐
│  Renders charts, usage tables, and cost summaries                 │
└───────────────────────────────────────────────────────────────────┘

2. MongoDB Collections

Collection Database Tracks Key Fields
llm_dashboard_data Marketplace DB Per-user LLM token usage from direct inference API calls user_id, api_name, model_name, tokens_used, prompt_tokens_used, completion_tokens_used, cost, daily_usage.{date}.{model}
logs VDB DB Per-user vector DB read/write operations user_id, operations_type, vector_db_type, cost_usd, executed_at
agentic_dashboard Marketplace DB Per-user agentic workflow executions user_id, execution metadata
data_pipeline_usage Marketplace DB ETL, cleaning, labeling, notebook, and streaming metering user_id, feature, resource_id, operation, event, status, prompt_tokens, completion_tokens, total_tokens, duration_seconds, gpu_count, cpu_cores, memory_gb, storage_gb, cost_usd, started_at, ended_at

data_pipeline_usage Document Schema

{
  "_id": ObjectId,
  "user_id": "string (marketplace user ObjectId string)",
  "feature": "etl | cleaning | labeling | notebook | streaming",
  "resource_id": "string (job_id or session_id)",
  "operation": "extract | transform | load | generate_schema | perform_operations | ai_assist_bulk | notebook | video_stream",
  "event": "start | stop | completed",
  "status": "running | completed | failed | cancelled | stopped",
  "prompt_tokens": 0,
  "completion_tokens": 0,
  "total_tokens": 0,
  "duration_seconds": 0,
  "gpu_count": 0,
  "gpu_profile": "string or empty",
  "cpu_cores": 0,
  "memory_gb": 0.0,
  "storage_gb": 0.0,
  "cost_usd": 0.0,
  "started_at": ISODate,
  "ended_at": ISODate,
  "recorded_at": ISODate
}

Idempotency: For async jobs (ETL, cleaning, labeling), resource_id holds the PNU job_id. record_data_pipeline_usage() checks for an existing document before inserting. For notebooks and streaming, the "start" event inserts a document and the "stop" event updates it by resource_id.

3. Pricing Rates

All rates are defined as constants in backend/utils/load_helper.py.

Constant Default Unit
INFERENCE_TOKEN_RATE 0.000002 USD per token
VDB_QUERY_RATE 0.00075 USD per VDB read
VDB_HOURLY_RATE 0.05 USD per VDB-hour
AGENTIC_RUN_RATE 0.03 USD per agentic execution
DATA_PIPELINE_GPU_HOURLY_RATE 1.50 USD per GPU-hour
DATA_PIPELINE_CPU_HOURLY_RATE 0.05 USD per core-hour
NOTEBOOK_GPU_HOURLY_RATE 1.50 USD per GPU-hour
NOTEBOOK_CPU_HOURLY_RATE 0.05 USD per core-hour
NOTEBOOK_MEMORY_HOURLY_RATE 0.002 USD per GB-hour
NOTEBOOK_STORAGE_HOURLY_RATE 0.0001 USD per GB-hour
STREAMING_GPU_HOURLY_RATE 2.00 USD per GPU-hour
STREAMING_CPU_HOURLY_RATE 0.05 USD per core-hour

4. Cost Formulas

4.1 LLM Inference (direct API calls)

cost = total_tokens × INFERENCE_TOKEN_RATE

Tracked via track_inference_usage() — atomic $inc on llm_dashboard_collection. Includes daily per-model breakdown for charting.

4.2 Vector DB Operations

cost = VDB_QUERY_RATE    (for read operations)
cost = 0                  (for write operations)

Tracked via track_vdb_operation() — inserts into vdb_logs_collection.

4.3 Agentic Workflows

cost = execution_count × AGENTIC_RUN_RATE

Tracked via direct insert in agentic routes.

4.4 Async Jobs (ETL, Data Cleaning, Data Labeling)

duration_hours = (completed_at - started_at) / 3600
token_cost     = total_tokens × INFERENCE_TOKEN_RATE
resource_cost  = gpu_count > 0
                   ? gpu_count × duration_hours × DATA_PIPELINE_GPU_HOURLY_RATE
                   : cpu_cores × duration_hours × DATA_PIPELINE_CPU_HOURLY_RATE
total_cost     = token_cost + resource_cost

Failed/cancelled jobs: cost_usd = 0.0. Tracked via record_data_pipeline_usage().

4.5 JupyterHub Notebooks

duration_hours = (stopped_at - created_at) / 3600
total_cost     = (gpu_count    × NOTEBOOK_GPU_HOURLY_RATE
                + cpu_cores    × NOTEBOOK_CPU_HOURLY_RATE
                + memory_gb    × NOTEBOOK_MEMORY_HOURLY_RATE
                + storage_gb   × NOTEBOOK_STORAGE_HOURLY_RATE) × duration_hours

Start/stop/delete event pattern. Tracked via record_notebook_usage(). The delete event (hard-delete) uses the same cost formula but marks the doc with status: "deleted".

4.6 Video Streaming

duration_hours = (completed_at - started_at) / 3600
total_cost     = (gpu_count × STREAMING_GPU_HOURLY_RATE
                + cpu_cores × STREAMING_CPU_HOURLY_RATE) × duration_hours

Start/stop event pattern. Tracked via record_streaming_usage().

5. Dashboard Endpoint

GET /api/dashboard/general

Auth: JWT cookie

Response (DashboardResponse model):

Field Type Description
pipeline_total_cost float Total cost across all data pipeline features
pipeline_total_tokens int Total LLM tokens consumed by async jobs
pipeline_total_gpu_hours float Total GPU-hours used (jobs with gpu_count > 0)
feature_costs dict[str, float] Per-feature cost breakdown (etl, cleaning, labeling, notebook, streaming)
(plus existing fields) LLM subscriptions, VDB ops, agentic executions, pie chart, daily line chart

Aggregation logic: 1. Queries data_pipeline_usage_collection by user_id (with ObjectId/str variant matching) 2. Sums cost_usd for pipeline_total_cost 3. Sums total_tokens for pipeline_total_tokens 4. Sums duration_seconds / 3600 for docs with gpu_count > 0 for pipeline_total_gpu_hours 5. Groups by feature field for feature_costs dict

6. Tracking Functions

All functions in backend/services/usage_tracker.py:

Function Collection Trigger Pattern
track_inference_usage() llm_dashboard_collection After every run_inference() call Atomic $inc on tokens, cost, requests; $set last_used
track_vdb_operation() vdb_logs_collection After VDB read/write Insert one document per operation
record_data_pipeline_usage() data_pipeline_usage_collection On job completion (ETL/cleaning/labeling) Idempotent insert (checks resource_id == job_id)
record_notebook_usage() data_pipeline_usage_collection On notebook start + stop + delete Insert on start, update on stop (status → stopped), update on delete (status → deleted)
record_streaming_usage() data_pipeline_usage_collection On stream start + stop Insert on start, update on stop
get_user_daily_token_breakdown() llm_dashboard_collection Dashboard chart Aggregates daily_usage map per model per day

All functions wrap in try/except and never raise — metering failures are logged but do not break the API response.

7. Key Files

File Role
backend/services/usage_tracker.py 5 tracking functions + daily token breakdown aggregator
backend/routes/dashboard.py GET /api/dashboard/general — aggregates all collections
backend/models/dashboard.py DashboardResponse Pydantic model with metering fields
backend/utils/load_helper.py All pricing rate constants (env var + defaults)
backend/database.py MongoDB collection definitions (data_pipeline_usage_collection, etc.)
backend/routes/data_pipeline/etl_routes.py Metering hook in get_job_status() for async jobs
backend/routes/data_pipeline/streaming_routes.py Start/stop metering hooks for streaming
backend/routes/tools/tools_routes.py Start/stop metering hooks for notebooks