Couchbase High Availability in Production: XDCR, Kubernetes Operator, and Multi-Region Deployment

Backup and Restore with cbbackupmgr

Couchbase provides cbbackupmgr, an enterprise backup tool that supports full, incremental, and differential backups with optional compression and encryption. For production HA deployments, a robust backup strategy combines Couchbase-level backups with cloud snapshot capabilities.

Backup Configuration

# Initialize a backup repository
/opt/couchbase/bin/cbbackupmgr config \
  --archive /backup/couchbase \
  --repo production-backup \
  --include-data production-data \
  --include-data user-profiles \
  --exclude-data _system

# Run a full backup
/opt/couchbase/bin/cbbackupmgr backup \
  --archive /backup/couchbase \
  --repo production-backup \
  --cluster couchbase://localhost \
  --username Administrator \
  --password "$CB_PASSWORD" \
  --threads 4 \
  --no-progress-bar

# Run an incremental backup (only mutations since last backup)
/opt/couchbase/bin/cbbackupmgr backup \
  --archive /backup/couchbase \
  --repo production-backup \
  --cluster couchbase://localhost \
  --username Administrator \
  --password "$CB_PASSWORD" \
  --threads 4

# List available backups
/opt/couchbase/bin/cbbackupmgr list \
  --archive /backup/couchbase \
  --repo production-backup

# Restore from a specific backup
/opt/couchbase/bin/cbbackupmgr restore \
  --archive /backup/couchbase \
  --repo production-backup \
  --cluster couchbase://target-cluster:8091 \
  --username Administrator \
  --password "$CB_PASSWORD" \
  --start 2026-04-12T00_00_00 \
  --end 2026-04-12T14_30_00 \
  --threads 4

Automated Backup Script for Kubernetes

#!/bin/bash
# couchbase-backup.sh — Automated backup to S3-compatible storage

set -euo pipefail

CLUSTER_HOST="cb-production-srv.couchbase.svc.cluster.local"
BACKUP_DIR="/backup/couchbase"
REPO_NAME="prod-$(date +%Y%m%d)"
S3_BUCKET="s3://couchbase-backups/production"
RETENTION_DAYS=14

log() { echo "[$(date +'%Y-%m-%d %H:%M:%S')] $*"; }

log "Starting Couchbase backup for cluster: $CLUSTER_HOST"

if [ ! -d "$BACKUP_DIR/$REPO_NAME" ]; then
  log "Configuring new backup repository: $REPO_NAME"
  cbbackupmgr config \
    --archive "$BACKUP_DIR" \
    --repo "$REPO_NAME" \
    --include-data production-data \
    --include-data user-profiles
fi

log "Running incremental backup..."
cbbackupmgr backup \
  --archive "$BACKUP_DIR" \
  --repo "$REPO_NAME" \
  --cluster "couchbase://$CLUSTER_HOST" \
  --username "$CB_USERNAME" \
  --password "$CB_PASSWORD" \
  --threads 4 \
  --no-progress-bar

BACKUP_SIZE=$(du -sh "$BACKUP_DIR/$REPO_NAME" | cut -f1)
log "Backup complete. Size: $BACKUP_SIZE"

log "Syncing to S3: $S3_BUCKET/$REPO_NAME"
aws s3 sync "$BACKUP_DIR/$REPO_NAME" "$S3_BUCKET/$REPO_NAME" \
  --storage-class STANDARD_IA \
  --sse aws:kms

log "Cleaning up backups older than $RETENTION_DAYS days..."
find "$BACKUP_DIR" -maxdepth 1 -name "prod-*" -mtime +"$RETENTION_DAYS" -exec rm -rf {} \;

log "Backup pipeline complete."

CouchbaseBackup CRD (Operator-Managed)

The Autonomous Operator provides CRDs for automated backup management:

apiVersion: couchbase.com/v2
kind: CouchbaseBackup
metadata:
  name: cb-daily-backup
  namespace: couchbase
spec:
  strategy: full_incremental
  full:
    schedule: "0 2 * * 0"   # Full backup every Sunday at 2 AM
  incremental:
    schedule: "0 2 * * 1-6" # Incremental Mon-Sat at 2 AM
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 3
  backOffLimit: 3
  logRetention: 168h
  size: 100Gi
  s3bucket: s3://couchbase-backups/production
---
apiVersion: couchbase.com/v2
kind: CouchbaseBackupRestore
metadata:
  name: cb-restore-pitr
  namespace: couchbase
spec:
  backup: cb-daily-backup
  repo: "20260412"
  start:
    int: 1
  end:
    int: 5
  backOffLimit: 3

N1QL Query Performance Tuning

N1QL (SQL++ for JSON) is Couchbase's query language. Tuning N1QL performance requires understanding the query planner, index design, and server-side optimizations.

Index Strategies: GSI and FTS

-- Global Secondary Index (GSI) for common query patterns

-- Composite index for user lookups
CREATE INDEX idx_users_email_status
  ON `user-profiles`(email, status)
  WHERE type = 'user'
  WITH {"num_replica": 1, "defer_build": false};

-- Covering index (includes all queried fields to avoid fetch)
CREATE INDEX idx_orders_covering
  ON `production-data`(customer_id, order_date, total_amount, status)
  WHERE type = 'order'
  WITH {"num_replica": 1};

-- Array index for nested documents
CREATE INDEX idx_order_items
  ON `production-data`(DISTINCT ARRAY item.product_id FOR item IN items END)
  WHERE type = 'order'
  WITH {"num_replica": 1};

-- Partial index for active records only
CREATE INDEX idx_active_sessions
  ON `production-data`(user_id, created_at)
  WHERE type = 'session' AND status = 'active'
  WITH {"num_replica": 1};

-- Adaptive index for dynamic query patterns
CREATE INDEX idx_adaptive_products
  ON `production-data`(DISTINCT PAIRS(self))
  WHERE type = 'product'
  WITH {"num_replica": 1};

-- Check index status
SELECT name, state, num_docs_indexed, num_docs_pending
FROM system:indexes
WHERE keyspace_id = 'production-data';

-- Analyze query execution plan
EXPLAIN SELECT u.name, u.email, COUNT(o.id) AS order_count
FROM `user-profiles` u
JOIN `production-data` o ON o.customer_id = u.id
WHERE u.status = 'active' AND o.type = 'order'
GROUP BY u.name, u.email
ORDER BY order_count DESC
LIMIT 100;

-- Use ADVISE to get index recommendations
ADVISE SELECT * FROM `production-data`
WHERE type = 'order'
  AND customer_id = 'cust-12345'
  AND order_date BETWEEN '2026-01-01' AND '2026-04-12'
ORDER BY order_date DESC;

Query Optimization Tips

-- Use PREPARE for frequently executed queries (cached plan)
PREPARE get_user_orders AS
SELECT o.id, o.order_date, o.total_amount, o.status
FROM `production-data` o
WHERE o.type = 'order'
  AND o.customer_id = $customer_id
ORDER BY o.order_date DESC
LIMIT $page_size OFFSET $page_offset;

-- Execute prepared statement
EXECUTE get_user_orders
USING {"customer_id": "cust-12345", "page_size": 20, "page_offset": 0};

-- Use META().id for direct key-value lookups (fastest path)
SELECT META().id, *
FROM `production-data`
USE KEYS ["order::2026-001", "order::2026-002", "order::2026-003"];

-- Correlated subquery with USE KEYS for joins
SELECT u.name,
  (SELECT o.id, o.total_amount
   FROM `production-data` o
   USE KEYS u.order_ids
   WHERE o.status = 'completed') AS completed_orders
FROM `user-profiles` u
WHERE META(u).id = 'user::12345';

-- Use INFER to understand document schema
INFER `production-data` WITH {"sample_size": 10000, "similarity_metric": 0.6};

Memory Management and Bucket Configuration

Couchbase's memory-first architecture means RAM allocation directly impacts performance. Each service has its own memory quota, and buckets share the Data Service quota. Proper sizing prevents cache evictions that degrade latency.

# Configure cluster-level memory quotas
/opt/couchbase/bin/couchbase-cli setting-cluster \
  --cluster localhost:8091 \
  --username Administrator \
  --password password \
  --cluster-ramsize 8192 \
  --cluster-index-ramsize 4096 \
  --cluster-fts-ramsize 2048 \
  --cluster-eventing-ramsize 2048 \
  --cluster-analytics-ramsize 4096

# Memory allocation guidelines:
# Data Service:      60% of available node RAM
# Index Service:     20% of available node RAM
# Search Service:    10% of available node RAM
# OS/overhead:       10% reserved

# Bucket memory sizing formula:
# Required RAM = (avg_doc_size * num_docs * 2.5) / num_data_nodes
# The 2.5 multiplier accounts for:
#   - Metadata overhead (~56 bytes per document)
#   - Internal fragmentation
#   - Replica copies in memory

# Create an optimized production bucket
curl -X POST http://localhost:8091/pools/default/buckets \
  -u Administrator:password \
  -d name=production-data \
  -d ramQuota=4096 \
  -d bucketType=couchbase \
  -d replicaNumber=2 \
  -d threadsNumber=8 \
  -d evictionPolicy=valueOnly \
  -d compressionMode=active \
  -d maxTTL=0 \
  -d conflictResolutionType=lww \
  -d flushEnabled=0 \
  -d durabilityMinLevel=majorityAndPersistActive

# Eviction policies:
# valueOnly  - Evicts document values but keeps metadata in RAM
#              Best for workloads where key access patterns are predictable
# fullEviction - Evicts both values and metadata
#                Best for very large datasets that exceed available RAM
# noEviction - (Ephemeral buckets only) Rejects writes when RAM is full
#              Best for caching use cases

TLS Encryption and RBAC

Securing Couchbase in production requires encryption of data in transit (TLS), fine-grained role-based access control (RBAC), and audit logging.

# Enable TLS for all Couchbase services
/opt/couchbase/bin/couchbase-cli ssl-manage \
  --cluster localhost:8091 \
  --username Administrator \
  --password password \
  --set-node-certificate

# Enforce minimum TLS version
/opt/couchbase/bin/couchbase-cli setting-security \
  --cluster localhost:8091 \
  --username Administrator \
  --password password \
  --set \
  --tls-min-version tlsv1.2 \
  --tls-honor-cipher-order 1 \
  --hsts-max-age 31536000 \
  --hsts-preload-enabled 1

# Create application-specific RBAC users
/opt/couchbase/bin/couchbase-cli user-manage \
  --cluster localhost:8091 \
  --username Administrator \
  --password password \
  --set \
  --rbac-username app-service \
  --rbac-password "$(openssl rand -base64 32)" \
  --rbac-name "Application Service Account" \
  --roles 'data_reader[production-data],data_writer[production-data],query_select[production-data],query_insert[production-data],query_update[production-data],query_delete[production-data]' \
  --auth-domain local

# Create a read-only analytics user
/opt/couchbase/bin/couchbase-cli user-manage \
  --cluster localhost:8091 \
  --username Administrator \
  --password password \
  --set \
  --rbac-username analytics-reader \
  --rbac-password "$(openssl rand -base64 32)" \
  --roles 'data_reader[production-data],query_select[production-data],analytics_reader[production-data]' \
  --auth-domain local

# Enable audit logging
/opt/couchbase/bin/couchbase-cli setting-audit \
  --cluster localhost:8091 \
  --username Administrator \
  --password password \
  --set \
  --audit-enabled 1 \
  --audit-log-path /opt/couchbase/var/lib/couchbase/logs \
  --audit-log-rotate-interval 86400 \
  --audit-log-rotate-size 20971520

Kubernetes TLS with cert-manager

# Certificate for Couchbase server TLS
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: couchbase-server-tls
  namespace: couchbase
spec:
  secretName: couchbase-server-tls
  duration: 8760h   # 1 year
  renewBefore: 720h  # 30 days before expiry
  privateKey:
    algorithm: RSA
    size: 4096
  usages:
  - server auth
  - client auth
  dnsNames:
  - "*.cb-production.couchbase.svc.cluster.local"
  - "*.cb-production.couchbase.svc"
  - "cb-production-srv.couchbase.svc.cluster.local"
  - "localhost"
  issuerRef:
    name: couchbase-ca-issuer
    kind: ClusterIssuer

Monitoring with Prometheus Exporter

Couchbase exposes rich metrics through its REST API. The couchbase-exporter translates these into Prometheus format for comprehensive monitoring.

# Deploy Couchbase Prometheus Exporter
apiVersion: apps/v1
kind: Deployment
metadata:
  name: couchbase-exporter
  namespace: couchbase
spec:
  replicas: 1
  selector:
    matchLabels:
      app: couchbase-exporter
  template:
    metadata:
      labels:
        app: couchbase-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9091"
    spec:
      containers:
      - name: exporter
        image: couchbase/exporter:1.0.9
        args:
        - --couchbase-address=cb-production-srv.couchbase.svc.cluster.local
        - --couchbase-port=8091
        - --couchbase-username=$(CB_USERNAME)
        - --couchbase-password=$(CB_PASSWORD)
        - --server-address=0.0.0.0:9091
        - --per-node-refresh=5
        env:
        - name: CB_USERNAME
          valueFrom:
            secretKeyRef:
              name: cb-admin-credentials
              key: username
        - name: CB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cb-admin-credentials
              key: password
        ports:
        - containerPort: 9091
          name: metrics
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: couchbase-monitor
  namespace: couchbase
spec:
  selector:
    matchLabels:
      app: couchbase-exporter
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

Key Couchbase Metrics to Monitor

• cb_bucket_ops_per_sec — Total operations per second per bucket. Baseline your normal throughput and alert on anomalies.
• cb_bucket_mem_used_bytes — Bucket memory usage. Alert when approaching the RAM quota to prevent evictions.
• cb_bucket_cache_miss_ratio — Ratio of requests that miss the cache and require disk fetch. Should stay below 2% for optimal performance.
• cb_bucket_disk_queue_items — Disk write queue depth. A growing queue indicates disk I/O cannot keep up with write throughput.
• cb_xdcr_changes_left — Number of mutations pending XDCR replication. Indicates cross-region replication lag.
• cb_xdcr_docs_written — Documents replicated per second through XDCR.
• cb_node_cpu_utilization_percent — Per-node CPU usage. Couchbase is CPU-intensive for compaction and indexing.
• cb_bucket_vbucket_active_num — Number of active vBuckets per node. Should be roughly even across data nodes.
• cb_index_num_docs_pending — Documents pending index update. Indicates index build lag.
• cb_n1ql_requests_per_sec — N1QL query throughput. Combined with average latency, identifies query performance issues.

Prometheus Alerting Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: couchbase-alerts
  namespace: couchbase
spec:
  groups:
  - name: couchbase.rules
    rules:
    - alert: CouchbaseNodeDown
      expr: cb_node_healthy == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "Couchbase node {{ $labels.node }} is unhealthy"
    - alert: CouchbaseHighCacheMissRate
      expr: cb_bucket_cache_miss_ratio > 0.05
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Cache miss rate {{ $value | humanizePercentage }} on bucket {{ $labels.bucket }}"
    - alert: CouchbaseXDCRLag
      expr: cb_xdcr_changes_left > 10000
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "XDCR replication lag: {{ $value }} pending mutations"
    - alert: CouchbaseDiskQueueGrowing
      expr: rate(cb_bucket_disk_queue_items[5m]) > 100
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Disk queue growing on bucket {{ $labels.bucket }}"
    - alert: CouchbaseMemoryPressure
      expr: cb_bucket_mem_used_bytes / cb_bucket_mem_quota_bytes > 0.9
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Memory usage at {{ $value | humanizePercentage }} for bucket {{ $labels.bucket }}"

SDK Connection String Configuration for HA

Couchbase SDKs are topology-aware — they maintain an internal cluster map and route operations directly to the correct node. Proper SDK configuration is critical for HA, ensuring fast failover detection and automatic retry on transient errors.

// Node.js SDK configuration for HA
const couchbase = require('couchbase');

const clusterConnStr = 'couchbases://cb-node1.example.com,cb-node2.example.com,cb-node3.example.com';

const cluster = await couchbase.connect(clusterConnStr, {
  username: process.env.CB_USERNAME,
  password: process.env.CB_PASSWORD,
  timeouts: {
    kvTimeout: 2500,           // Key-value operation timeout (ms)
    kvDurableTimeout: 10000,   // Durable write timeout
    queryTimeout: 75000,       // N1QL query timeout
    searchTimeout: 75000,      // FTS search timeout
    analyticsTimeout: 75000,   // Analytics query timeout
    connectTimeout: 10000,     // Initial connection timeout
    managementTimeout: 75000   // Management API timeout
  },
  security: {
    trustStorePath: '/etc/couchbase/ca.pem'
  },
  transactions: {
    durabilityLevel: couchbase.DurabilityLevel.MajorityAndPersistToActive,
    timeout: 15000
  }
});

const bucket = cluster.bucket('production-data');
const collection = bucket.defaultCollection();

// Durable write with observe-based durability
await collection.upsert('order::2026-001', orderDocument, {
  durabilityLevel: couchbase.DurabilityLevel.MajorityAndPersistToActive,
  timeout: 10000
});

// Read with replica fallback for HA
try {
  const result = await collection.get('user::12345');
} catch (err) {
  if (err instanceof couchbase.errors.TimeoutError) {
    const replicaResult = await collection.getAnyReplica('user::12345');
  }
}

# Java SDK configuration for HA
import com.couchbase.client.java.*;
import com.couchbase.client.java.env.*;
import java.time.Duration;

ClusterEnvironment env = ClusterEnvironment.builder()
    .timeoutConfig(TimeoutConfig.builder()
        .kvTimeout(Duration.ofMillis(2500))
        .kvDurableTimeout(Duration.ofSeconds(10))
        .queryTimeout(Duration.ofSeconds(75))
        .connectTimeout(Duration.ofSeconds(10))
        .build())
    .ioConfig(IoConfig.builder()
        .numKvConnections(4)
        .enableMutationTokens(true)
        .enableDnsSrv(true)
        .build())
    .securityConfig(SecurityConfig.builder()
        .enableTls(true)
        .trustCertificate(Paths.get("/etc/couchbase/ca.pem"))
        .build())
    .build();

Cluster cluster = Cluster.connect(
    "couchbases://cb-node1.example.com,cb-node2.example.com",
    ClusterOptions.clusterOptions("username", "password")
        .environment(env)
);

Kubernetes Service DNS for SDK Connection

# When using the Autonomous Operator, connect via the headless service:
# couchbase://cb-production-srv.couchbase.svc.cluster.local
#
# The operator creates these services:
# cb-production-srv      - Headless service for SDK auto-discovery
# cb-production-ui       - Web Console (port 8091/18091)
# cb-production-cloud    - External connectivity (NodePort/LoadBalancer)
#
# For external SDK access (outside Kubernetes), use:
# - NodePort with explicit node addresses
# - LoadBalancer with MetalLB (bare metal)
# - Ingress with TCP passthrough for port 11210 (SDK) and 11207 (SDK TLS)

Couchbase Mobile and Sync Gateway for Edge Deployments

Couchbase Mobile extends the Couchbase ecosystem to edge devices and mobile applications. Couchbase Lite runs embedded on iOS, Android, and IoT devices, while Sync Gateway acts as the synchronization middleware between Couchbase Lite and Couchbase Server.

// Sync Gateway configuration for production
{
  "interface": ":4984",
  "adminInterface": "127.0.0.1:4985",
  "logging": {
    "console": {
      "log_level": "info",
      "log_keys": ["HTTP", "Sync", "Auth", "Changes"]
    }
  },
  "databases": {
    "mobile-app": {
      "server": "couchbases://cb-production-srv.couchbase.svc.cluster.local",
      "bucket": "production-data",
      "username": "sync-gateway",
      "password": "${SG_PASSWORD}",
      "enable_shared_bucket_access": true,
      "import_docs": true,
      "num_index_replicas": 1,
      "delta_sync": {
        "enabled": true,
        "rev_max_age_seconds": 86400
      },
      "cache": {
        "channel_cache": {
          "max_number": 50000,
          "compact_high_watermark_pct": 80,
          "compact_low_watermark_pct": 60
        },
        "rev_cache": {
          "size": 5000,
          "shard_count": 16
        }
      },
      "users": {
        "GUEST": {"disabled": true}
      },
      "sync": "function(doc, oldDoc) { if (doc.type === 'user-data') { channel(doc.channels); requireAccess(doc.channels); } else { channel('public'); } }"
    }
  }
}

# Deploy Sync Gateway on Kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sync-gateway
  namespace: couchbase
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sync-gateway
  template:
    metadata:
      labels:
        app: sync-gateway
    spec:
      containers:
      - name: sync-gateway
        image: couchbase/sync-gateway:3.1.4-enterprise
        args: ["/etc/sync-gateway/config.json"]
        ports:
        - containerPort: 4984
          name: public
        - containerPort: 4985
          name: admin
        resources:
          requests:
            cpu: "2"
            memory: 4Gi
          limits:
            cpu: "4"
            memory: 8Gi
        volumeMounts:
        - name: config
          mountPath: /etc/sync-gateway
      volumes:
      - name: config
        configMap:
          name: sync-gateway-config

Capacity Planning and Sizing

Proper capacity planning is essential for Couchbase performance and cost optimization. The following table provides sizing guidelines based on workload tier:

Sizing Formula

# Data Service RAM sizing
# Required RAM per node = (num_documents * (doc_metadata_size + avg_value_size)) / num_data_nodes * (1 + num_replicas)
# doc_metadata_size = 56 bytes (fixed overhead per document)
# Include 25% headroom for fragmentation and growth

# Example: 100M documents, 1KB avg size, 3 data nodes, 1 replica
# RAM = (100,000,000 * (56 + 1024)) / 3 * 2 = ~72 GB per node
# With 25% headroom: ~90 GB per node

# Index Service RAM sizing (memory-optimized)
# RAM = total_index_size * 3 (for build/merge overhead)
# Use system:indexes to check current index sizes

# Disk sizing
# Disk = (num_documents * avg_doc_size * (1 + num_replicas)) * 3 (compaction headroom)
# Use SSD/NVMe with provisioned IOPS for predictable performance

Disaster Recovery and Failover Procedures

A comprehensive disaster recovery plan ensures business continuity when infrastructure failures exceed the scope of automatic failover.

Single Node Failure (Automatic)

# Auto-failover handles single node failures automatically.
# Verify failover occurred:
/opt/couchbase/bin/couchbase-cli server-list \
  --cluster localhost:8091 \
  --username Administrator \
  --password password

# After replacing the failed node, add and rebalance:
/opt/couchbase/bin/couchbase-cli server-add \
  --cluster localhost:8091 \
  --username Administrator \
  --password password \
  --server-add new-node.example.com:8091 \
  --server-add-username Administrator \
  --server-add-password password \
  --services data,index

/opt/couchbase/bin/couchbase-cli rebalance \
  --cluster localhost:8091 \
  --username Administrator \
  --password password

Complete Cluster Failure (Manual)

# Scenario: Primary region (US-EAST) completely lost

# Step 1: Verify XDCR target cluster (EU-WEST) has latest data
# Check XDCR replication status before failure
curl -s http://eu-west-node:8091/pools/default/remoteClusters \
  -u Administrator:password | jq .

# Step 2: Pause XDCR replications pointing to the failed cluster
/opt/couchbase/bin/couchbase-cli xdcr-replicate \
  --cluster cb-eu-west.example.com:8091 \
  --username Administrator \
  --password password \
  --pause \
  --xdcr-replicator <replication-id>

# Step 3: Update application connection strings to EU-WEST cluster
# (via DNS update, service mesh, or environment variable change)

# Step 4: Scale up EU-WEST cluster if needed to handle full production load
# In Kubernetes, update the CouchbaseCluster CRD:
kubectl patch couchbasecluster cb-eu-west -n couchbase --type merge \
  -p '{"spec":{"servers":[{"name":"data-zone-a","size":4}]}}'

# Step 5: After US-EAST cluster is restored, re-establish XDCR
# and perform a full resync from EU-WEST back to US-EAST

Graceful Failover and Recovery

# Graceful failover (for maintenance, drains data before removal)
/opt/couchbase/bin/couchbase-cli failover \
  --cluster localhost:8091 \
  --username Administrator \
  --password password \
  --server-failover node-to-remove.example.com:8091

# Recovery (re-add the node after maintenance)
/opt/couchbase/bin/couchbase-cli recovery \
  --cluster localhost:8091 \
  --username Administrator \
  --password password \
  --server-recovery node-to-recover.example.com:8091 \
  --recovery-type delta

# Delta recovery re-synchronizes only the changed data,
# which is much faster than full recovery.
# Full recovery rebuilds the node from scratch.

# Rebalance to complete the recovery
/opt/couchbase/bin/couchbase-cli rebalance \
  --cluster localhost:8091 \
  --username Administrator \
  --password password

Helm Values for Complete Production Deployment

Below is a comprehensive Helm values file for deploying Couchbase with the Autonomous Operator in a production environment:

# helm-values-production.yaml
couchbase-operator:
  operator:
    image:
      repository: couchbase/operator
      tag: 2.7.1
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: "1"
        memory: 1Gi
  admissionController:
    enabled: true
    resources:
      requests:
        cpu: 100m
        memory: 128Mi

cluster:
  image: couchbase/server:7.6.1-enterprise
  antiAffinity: true
  autoFailoverTimeout: 30s
  autoFailoverMaxCount: 3
  autoFailoverOnDataDiskIssues: true
  autoFailoverServerGroup: true
  security:
    adminSecret: cb-admin-credentials
  networking:
    tls:
      static:
        serverSecret: couchbase-server-tls
        operatorSecret: couchbase-operator-tls
    exposeAdminConsole: true
    adminConsoleServiceType: NodePort
  buckets:
    managed: true
  servers:
    data:
      size: 4
      services:
      - data
      - index
      serverGroups:
      - zone-a
      - zone-b
      resources:
        requests:
          cpu: "4"
          memory: 16Gi
        limits:
          cpu: "8"
          memory: 20Gi
      volumeMounts:
        default: couchbase-data
        data: couchbase-data
        index: couchbase-index
    query:
      size: 2
      services:
      - query
      - search
      resources:
        requests:
          cpu: "4"
          memory: 8Gi
        limits:
          cpu: "8"
          memory: 12Gi
      volumeMounts:
        default: couchbase-default
    analytics:
      size: 2
      services:
      - analytics
      - eventing
      serverGroups:
      - zone-c
      resources:
        requests:
          cpu: "8"
          memory: 32Gi
        limits:
          cpu: "16"
          memory: 40Gi
      volumeMounts:
        default: couchbase-analytics
        analytics:
        - couchbase-analytics
  serverGroups:
  - zone-a
  - zone-b
  - zone-c
  volumeClaimTemplates:
  - metadata:
      name: couchbase-data
    spec:
      storageClassName: ebs-gp3-couchbase
      resources:
        requests:
          storage: 100Gi
  - metadata:
      name: couchbase-index
    spec:
      storageClassName: ebs-gp3-couchbase
      resources:
        requests:
          storage: 50Gi
  - metadata:
      name: couchbase-default
    spec:
      storageClassName: ebs-gp3-couchbase
      resources:
        requests:
          storage: 20Gi
  - metadata:
      name: couchbase-analytics
    spec:
      storageClassName: ebs-gp3-couchbase
      resources:
        requests:
          storage: 200Gi

Conclusion

Couchbase Server's architecture—built around vBucket-based sharding, memory-first data access, and integrated multi-model services—provides a uniquely powerful foundation for high availability production deployments. The combination of intra-cluster replication with automatic failover ensures that single-node failures are handled transparently, while XDCR extends this resilience across geographic regions for global applications.

The Couchbase Autonomous Operator for Kubernetes transforms what would be complex manual operations into declarative, self-healing deployments. Server groups provide rack/zone awareness, the operator manages rebalance operations during scaling events, and integrated backup CRDs automate disaster recovery preparation.

Key takeaways from this guide:

• Leverage Multi-Dimensional Scaling — Separate Data, Index, Query, Search, Analytics, and Eventing services onto dedicated node pools for independent scaling and resource isolation.
• Configure XDCR for multi-region resilience — Bidirectional XDCR with timestamp-based conflict resolution enables active-active deployments across AWS, Azure, and GCP. Always ensure NTP synchronization.
• Use server groups for zone awareness — Map server groups to availability zones or racks to guarantee that active and replica vBuckets are in different failure domains.
• Size memory carefully — Couchbase's performance is directly tied to how much of the working set fits in RAM. Use the sizing formulas and monitor cache miss ratios.
• Implement comprehensive monitoring — Deploy the Prometheus exporter from day one. XDCR replication lag, cache miss ratio, disk queue depth, and node health are your critical signals.
• Automate backups with cbbackupmgr — Combine full and incremental backups with cloud snapshots. Test restore procedures regularly.
• Secure with TLS and RBAC — Enable node-to-node and client-to-node TLS encryption. Use fine-grained RBAC roles for every application service account.
• Configure SDKs for HA — Use multiple bootstrap nodes, configure appropriate timeouts, implement replica reads as fallback, and leverage durable writes for critical data.
• Plan for disaster recovery — Document and rehearse failover procedures for single-node, multi-node, and complete cluster failure scenarios. XDCR standby clusters should be ready for promotion at all times.

With this comprehensive foundation, you are equipped to deploy and operate Couchbase Server in high availability production environments across any infrastructure—from managed Kubernetes on AWS, Azure, and GCP to bare metal k3s clusters managed by Rancher. The combination of Couchbase's native distributed architecture with Kubernetes orchestration delivers a database platform that meets the demands of modern, globally distributed applications.