Skip to content

CDC Webhooks

Status: Production-Ready (85% Complete) Last Updated: January 4, 2026

Overview

HeliosDB's CDC-to-Webhook integration enables real-time event streaming from database changes (INSERT, UPDATE, DELETE) to external systems via HTTP webhooks. The system provides exactly-once delivery, circuit breaker protection, and comprehensive monitoring.

Key Capabilities

Capability Description Status
Event Capture WAL-based Change Data Capture Production
Event Delivery HTTP webhooks with retry Production
Delivery Guarantee Exactly-once semantics Production
Event Filtering Table/operation/column filters Production
Event Transformation JSONPath, Jinja2 templates Planned
Circuit Breaker Automatic failure detection Production
Monitoring Prometheus metrics, dashboards Production

Performance Characteristics

Metric Value
Event Capture Rate 100K events/sec
Webhook Delivery Latency (P50) 5-10ms
Webhook Delivery Latency (P99) 30-50ms
Delivery Success Rate 99.9%
Exactly-Once Guarantee 100% (within 24hr window)
Max Concurrent Webhooks 10,000
Queue Capacity Unlimited (disk-backed)

Architecture

                    CDC-to-Webhook Pipeline

  ┌─────────────┐     ┌─────────────────┐     ┌──────────────┐
  │  Database   │ --> │   WAL/CDC       │ --> │   Event      │
  │  Changes    │     │   Processor     │     │   Queue      │
  └─────────────┘     └─────────────────┘     └──────────────┘
                                                     │
                                                     v
  ┌─────────────┐     ┌─────────────────┐     ┌──────────────┐
  │  External   │ <-- │   Webhook       │ <-- │   Worker     │
  │  Endpoints  │     │   Delivery      │     │   Pool       │
  └─────────────┘     └─────────────────┘     └──────────────┘
        │                    │                      │
        v                    v                      v
  ┌─────────────────────────────────────────────────────────┐
  │              Monitoring & Observability                  │
  │  (Prometheus metrics, Grafana dashboards, Audit logs)   │
  └─────────────────────────────────────────────────────────┘

Feature Highlights

1. WAL-Based Event Capture

Capture database changes directly from the Write-Ahead Log for maximum performance and minimal overhead.

2. Exactly-Once Delivery

Idempotency keys ensure each event is delivered exactly once, even in the presence of retries and failures.

3. Circuit Breaker Protection

Automatic failure detection prevents cascading failures when webhook endpoints are unhealthy.

4. Flexible Filtering

Filter events by: - Table name - Operation type (INSERT, UPDATE, DELETE) - Column changes - Custom predicates

5. High Throughput

Process 100K+ events per second with low latency and high reliability.

Quick Example

SQL Interface

-- Create a CDC stream to Kafka
CREATE CHANGE DATA CAPTURE ON users
  TO KAFKA 'localhost:9092' TOPIC 'users-changes'
  FORMAT JSON;

-- Monitor the stream
SHOW CDC STREAMS;
SHOW CDC STREAM STATUS users_cdc_stream;

-- Pause during maintenance
ALTER CDC STREAM users_cdc_stream PAUSE;

-- Resume
ALTER CDC STREAM users_cdc_stream RESUME;

Rust API

use heliosdb_cdc::{CdcConfig, EventProcessor};
use heliosdb_webhooks::{WebhookServer, WebhookConfig};

#[tokio::main]
async fn main() -> Result<()> {
    // Create CDC processor
    let cdc_config = CdcConfig::default();
    let processor = EventProcessor::new(cdc_config).await?;

    // Create webhook server
    let webhook_config = WebhookConfig {
        bind_address: "0.0.0.0:8081".to_string(),
        max_retries: 3,
        timeout_secs: 30,
        ..Default::default()
    };
    let server = WebhookServer::new(webhook_config).await?;

    // Register webhook endpoints
    server.register("https://api.example.com/webhooks/db-changes").await?;

    // Start processing
    processor.start().await?;

    Ok(())
}

Use Cases

Use Case Description
Real-time Analytics Stream changes to analytics systems
Event-Driven Architecture Trigger downstream services on data changes
Data Sync Keep external systems in sync with database
Audit Logging Capture all changes for compliance
Cache Invalidation Invalidate caches when data changes
Search Indexing Update search indexes in real-time

API Modules

Module Description
heliosdb_cdc::EventProcessor CDC event processor
heliosdb_webhooks::WebhookServer Webhook delivery server
heliosdb_webhooks::CircuitBreaker Circuit breaker manager
heliosdb_webhooks::ExactlyOnce Exactly-once delivery

See Also: HeliosDB Feature Index