Skip to content

Deployment Architecture

This document illustrates the deployment topology of the Scality CSI Driver for S3, showing how components are distributed across a Kubernetes cluster. The architecture differs between static and dynamic provisioning modes.

Press "Ctrl" to enable Pan & Zoom
graph TB
    subgraph cluster["Kubernetes Cluster"]

        subgraph controlplane["Kubernetes Control Plane"]
            APIServer["Kubernetes API Server"]
        end

        subgraph node1["Kubernetes Node 1"]
            K1[Kubelet]
            subgraph ds1["CSI Driver Pod (DaemonSet)"]
                N1[CSI Driver Node Service]
                R1[CSI Driver Registrar Sidecar]
                L1[CSI Driver Liveness Probe]
            end
            subgraph init1["CSI Driver Init Container"]
                I1[mount-s3 Installer: Copies binary to host]
            end
            S1[Host systemd]
            M1[mount-s3 FUSE processes: One per mounted volume]
            A1[Application Pods]
        end

        subgraph node2["Kubernetes Node 2"]
            subgraph controller["CSI Controller Deployment"]
                subgraph controllerPod["Controller Pod (1 replica)"]
                    CSIController["CSI Controller Service"]
                    CSIProvisioner["CSI Provisioner Sidecar(Watch PVC)"]
                end
            end
            K2[Kubelet]
            subgraph ds2["CSI Driver Pod (DaemonSet)"]
                N2[CSI Driver Node Service]
                R2[CSI Driver Registrar Sidecar]
                L2[CSI Driver Liveness Probe]
            end
            subgraph init2["CSI Driver Init Container"]
                I2[mount-s3 Installer: Copies binary to host]
            end
            S2[Host systemd]
            M2[mount-s3 FUSE processes: One per mounted volume]
            A2[Application Pods]
        end

        subgraph node3["Kubernetes Node N..."]
            K3[Kubelet]
            subgraph ds3["CSI Driver Pod (DaemonSet)"]
                N3[CSI Driver Node Service]
                R3[CSI Driver Registrar Sidecar]
                L3[CSI Driver Liveness Probe]
            end
            subgraph init3["CSI Driver Init Container"]
                I3[mount-s3 Installer: Copies binary to host]
            end
            S3[Host systemd]
            M3[mount-s3 FUSE processes: One per mounted volume]
            A3[Application Pods]
        end


    end

    S3Storage[S3 Storage Endpoint]

    %% Controller operations (Dynamic Provisioning)
    APIServer <-->|"Watch PVC/StorageClass, Resolve PV/PVC Templates, Create PV, Update PVC Status"| CSIProvisioner
    CSIProvisioner -->|"CreateVolume/DeleteVolume RPC | Unix socket /csi/csi.sock"| CSIController
    CSIController -->|"Bucket Create/Delete via S3 API"| S3Storage

    %% Init container flow
    I1 -.->|Install binary to /opt/mountpoint-s3-csi/bin/| S1
    I2 -.->|Install binary to /opt/mountpoint-s3-csi/bin/| S2
    I3 -.->|Install binary to /opt/mountpoint-s3-csi/bin/| S3

    %% CSI Driver Registration
    R1 -->|Register via /registration/ entry| K1
    R2 -->|Register via /registration/ entry| K2
    R3 -->|Register via /registration/ entry| K3

    %% Health monitoring
    L1 -->|Monitor Unix socket /csi/csi.sock| N1
    L2 -->|Monitor Unix socket /csi/csi.sock| N2
    L3 -->|Monitor Unix socket /csi/csi.sock| N3

    %% Node operations
    K1 -->|Volume requests via gRPC on host Unix socket| N1
    K2 -->|Volume requests via gRPC on host Unix socket| N2
    K3 -->|Volume requests via gRPC on host Unix socket| N3

    N1 -->|Create/stop services via D-Bus| S1
    N2 -->|Create/stop services via D-Bus| S2
    N3 -->|Create/stop services via D-Bus| S3

    S1 -->|Start/stop/monitor processes| M1
    S2 -->|Start/stop/monitor processes| M2
    S3 -->|Start/stop/monitor processes| M3

    %% Application access
    A1 -->|File I/O| M1
    A2 -->|File I/O| M2
    A3 -->|File I/O| M3

    %% S3 connections
    M1 -->|S3 API| S3Storage
    M2 -->|S3 API| S3Storage
    M3 -->|S3 API| S3Storage

    %% Styling for clarity without colors
    classDef optional stroke-dasharray: 5 5

Deployment Components

Controller Components (Dynamic Provisioning Only)

Component Type Purpose Details
CSI Controller Service Main Container Volume lifecycle management • Binary: scality-s3-csi-driver with CSI_CONTROLLER_ONLY=true
• Handles CreateVolume/DeleteVolume RPCs for dynamic provisioning
• Creates and deletes S3 buckets based on StorageClass parameters
• Manages provisioner and node-publish secrets from StorageClass
• Single replica Deployment (not DaemonSet)
• Runs on exactly one Kubernetes node in the cluster at any time
CSI Provisioner Sidecar Sidecar Container Kubernetes integration • Standard csi-provisioner from Kubernetes
• Watches for PVCs that need dynamic provisioning
• Reads StorageClass parameters and templates
• Resolves template variables in StorageClass parameters (${pvc.name}, ${pvc.namespace}, ${pv.name}, etc.)
• Calls CSI Controller's CreateVolume/DeleteVolume
• Creates PV objects after successful bucket creation

Node Components

Component Type Purpose Details
mount-s3 Installer Init Container Binary deployment • Copies mount-s3 binary from container to host at /opt/mountpoint-s3-csi/bin/
• Runs first and must complete successfully before main containers start
• Required because systemd executes processes on host filesystem
• Sets appropriate file permissions for systemd execution
CSI Driver Node Service Main Container Core CSI functionality • Binary: scality-s3-csi-driver
• Creates gRPC server on /csi/csi.sock Unix socket file
• Exposes HTTP /healthz endpoint for Kubernetes liveness probe
• Pod restart triggered if HTTP health check fails
• Handles volume mount/unmount operations by launching mount-s3 binary installed by init container
• Manages systemd services via D-Bus that execute the mount-s3 binary installed by init container
CSI Driver Registrar Sidecar Kubelet registration • Creates registration entry in /registration/ directory watched by kubelet
• Registration entry announces CSI driver name s3.csi.scality.com and Unix socket location /var/lib/kubelet/plugins/s3.csi.scality.com/csi.sock
• Maintains registration while driver is deployed on node
• Has own liveness probe for registration health
• Uses standard Kubernetes CSI node-driver-registrar sidecar
CSI Driver Liveness Probe Sidecar CSI socket health logging • Checks CSI Driver Node Service via /csi/csi.sock Unix socket file
• Logs health status to container logs for troubleshooting
• Does NOT trigger pod restarts (logging only)

Host-Level Components

Scope Component Purpose Details
Per Kubernetes Node Host systemd Service management • Host's service manager receiving D-Bus commands from CSI Driver Node Service
• Creates transient systemd services that execute mount-s3 binary installed by init container
• Manages service lifecycle: start, stop, monitor mount processes
• Provides process supervision and cleanup on service failures
• Runs on host filesystem context, not in container
Per Volume mount-s3 FUSE processes S3 filesystem mounting • One process per mounted volume using mount-s3 binary installed by init container
• Executed by systemd services created via D-Bus by CSI Driver Node Service
• Creates FUSE mount presenting S3 bucket as POSIX filesystem
• Handles S3 API communication, caching, and file system semantics

Key Deployment Characteristics

Resource Distribution

Resource Scope What Gets Deployed Deployment Method When Required
Cluster-wide One CSI Controller pod Deployment (1 replica) Dynamic provisioning only
Per Kubernetes Node One CSI Driver pod DaemonSet Always
Per Volume One mount-s3 process systemd service Always

Communication Paths

Path From To Protocol Purpose Provisioning Mode
PVC Monitoring CSI Provisioner Sidecar Kubernetes API HTTPS Watch PVC/StorageClass events Dynamic only
Volume Provisioning CSI Provisioner Sidecar CSI Controller Service gRPC on Unix socket /csi/csi.sock CreateVolume/DeleteVolume Dynamic only
Bucket Operations CSI Controller Service S3 endpoint HTTPS Create/delete S3 buckets Dynamic only
CSI Driver Registration CSI Driver Registrar Kubelet Unix socket /registration/ Register driver per Kubernetes node Both
Volume Operations Kubelet CSI Driver Node Service gRPC on Unix socket /var/lib/kubelet/plugins/s3.csi.scality.com/csi.sock Mount/unmount requests Both
Health Monitoring CSI Driver Liveness Probe CSI Driver Node Service gRPC on Unix socket /csi/csi.sock Health status checks Both
Service Management CSI Driver Node Service systemd D-Bus on /run/systemd/ Create/stop services Both
File I/O Application pods mount-s3 processes FUSE File system operations Both
Storage Access mount-s3 processes S3 endpoint HTTPS S3 API calls Both

Host Mounts Required

Host Path Purpose Used By
/var/lib/kubelet/plugins/s3.csi.scality.com/ CSI driver Unix socket creation and registration info storage CSI Driver Node Service (creates gRPC socket), kubelet (volume operations), CSI Driver Registrar (driver registration), CSI Driver Liveness Probe (health checks)
/var/lib/kubelet/pods/<pod-id>/volumes/kubernetes.io~csi/<volume-id>/mount/ S3 bucket content mount point for application access kubelet (creates mount point directory), mount-s3 processes (FUSE filesystem), Application pods (file I/O)
/run/systemd/ D-Bus socket for systemd service lifecycle management systemd (owns D-Bus sockets), CSI Driver Node Service (D-Bus client for service management)
/opt/mountpoint-s3-csi/bin/mount-s3 FUSE mount binary executable storage mount-s3 Installer (creates binary file), systemd transient services (execute FUSE mounts)

Scaling Behavior

Resource Scaling Behavior Mechanism Notes
CSI Controller Single instance Deployment with 1 replica Only one controller needed cluster-wide (dynamic provisioning)
Kubernetes Nodes Automatic deployment to new nodes DaemonSet controller One CSI node pod per Kubernetes node
Volumes One process per volume systemd service creation Each mounted volume gets its own mount-s3 process

Static vs Dynamic Provisioning

Static Provisioning

  • No controller deployment needed
  • Only DaemonSet for node pods
  • Administrator pre-creates S3 buckets
  • PersistentVolumes reference existing buckets

Dynamic Provisioning

  • Requires controller deployment (controller.enable: true in Helm values)
  • Controller creates/deletes S3 buckets automatically
  • StorageClass defines bucket creation parameters
  • Supports credential templating for multi-tenancy

Credential Flow Differences

Aspect Static Provisioning Dynamic Provisioning
Bucket Creation Manual by admin Automatic by CSI Controller
Credential Sources • Driver-level (global)
• PV-level (nodePublishSecretRef)
• Driver-level (global)
• StorageClass provisioner secrets
• StorageClass node-publish secrets
• Template-based secrets
Secret Resolution At mount time by CSI Node • Provisioner secrets at CreateVolume
• Node-publish secrets at mount time
Multi-tenancy Per-PV secrets Per-StorageClass or per-PVC templated secrets