Technical Deep Dive

This page unpacks the technical specifications of the On-Premise Kubernetes Platform. If you’re looking for the design rationale and tradeoffs, start there.

Production context: This cluster runs a multi-tenant SIEM platform that accepts logs from over 1,500 remote agents across WAN boundaries, sustaining 6,000–10,000 logs per second around the clock. The cluster has maintained 100% uptime over the past 90 days (as of this writing).

Physical Infrastructure: Proxmox Hyperconverged Cluster

The Kubernetes cluster runs on virtual machines provisioned on a 5-node Proxmox hyperconverged cluster. Understanding the physical layer matters because it sets the constraints on what the Kubernetes cluster can do—and explains why I distribute virtual nodes the way I do.

The hyperconverged design means adding a Proxmox node automatically expands both compute capacity and Ceph storage. This is intentional—I wanted infrastructure that scales linearly without separate storage procurement.

Physical Servers

High-Performance Tier (2 servers):

General-Purpose Tier (3 servers):

Aggregate Physical Resources

Ceph Distributed Storage

Virtualization Strategy: Blast Radius Containment

With 5 physical servers and 21 Kubernetes nodes, I had choices about how to distribute virtual machines. The distribution is designed around blast radius containment.

Kubernetes provides resilience regardless of whether a virtual node or a physical host goes down—pods get rescheduled to surviving nodes either way. But splitting each physical server into multiple virtual nodes reduces the blast radius when a single Kubernetes node fails. If I ran one massive VM per physical host, losing that VM (kernel panic, misconfiguration, failed upgrade) would take a large chunk of cluster capacity offline. By running multiple smaller VMs per host, a single VM failure only loses a fraction of that host’s resources.

The tradeoff is overhead: more VMs means more operating system instances, more memory reserved for each VM’s kernel, and more coordination. I sized the VMs to balance blast radius against that overhead—large enough to run meaningful workloads efficiently, small enough that losing one doesn’t cascade.

• Control plane nodes are distributed one-per-host across the three general-purpose servers. A single host failure loses one of three control plane nodes, leaving quorum intact.
• Pool A workers are spread 3-per-host across the same three servers. A host failure loses 3 of 9 workers; a single VM failure loses 1 of 9.
• Pool B and Pool C run on dedicated high-performance hosts. These pools run workloads with application-level redundancy (database replicas, distributed caches) that can tolerate node-level failures.

Kubernetes Cluster Specifications

Aggregate Resources

Control Plane Design

I keep the control plane minimal and dedicated—these nodes run etcd and the Kubernetes API server, not workloads. The sizing is intentionally modest because the control plane isn’t where compute-intensive work happens.

The three-node control plane provides quorum for etcd—two nodes can fail before the cluster loses the ability to make scheduling decisions. I distribute them across different Proxmox hosts to ensure a single host failure doesn’t take out the control plane.

Worker Pool Design

I run three worker pools with distinct resource profiles and scheduling characteristics. The reasoning: not all workloads have the same shape, and trying to run everything on a homogeneous pool means either over-provisioning everywhere or starving some workloads.

Pool A – General Purpose (9 nodes)

This pool handles the bulk of workloads—web services, background jobs, platform services. The nodes are sized to run multiple medium-sized pods without contention, and spreading across three hosts provides resilience against single-host failures.

Pool B – High Performance (4 nodes)

This pool runs latency-sensitive and memory-intensive workloads—databases, caches, search indices. NUMA awareness helps here because these workloads benefit from memory locality. The larger per-node sizing means fewer pods per node, which reduces noisy-neighbor effects.

Pool C – High Performance + GPU (5 nodes)

The GPU taint ensures only workloads that explicitly request GPU resources get scheduled to the GPU node. Without this, the scheduler might place general workloads there and starve GPU workloads of the node’s CPU and memory.

Storage Classes

I use multiple storage backends because different workloads have different storage requirements. A database wants block storage with strong consistency; a shared config directory wants a filesystem that multiple pods can mount simultaneously.

Ceph Integration (Rook)

Ceph provides the primary storage tier, integrated via Rook’s CSI driver. The underlying Ceph cluster runs on the same Proxmox hosts, which means storage performance scales with compute—adding a node improves both.

• RBD (Block): ReadWriteOnce volumes for databases, stateful workloads
• CephFS (File): ReadWriteMany volumes for shared data across pods

TrueNAS Integration (Democratic CSI)

TrueNAS provides an alternative storage tier via the Democratic CSI driver. I use this for workloads that benefit from ZFS features (snapshots, clones) or need a different performance profile than Ceph provides.

Block Storage at the Hypervisor Layer

The Kubernetes storage classes above sit on top of another tier — the virtual machines themselves need block devices for their root and data disks. This layer is invisible to K8s, but it matters operationally because VM-level disk performance bounds what container-level storage can ever deliver.

I run three block storage backends at the Proxmox layer:

• Ceph RBD (default for most VMs) — the same Ceph cluster that backs the K8s ceph-rbd storage class, consumed directly by Proxmox via its Ceph RBD plugin. I default to this because it gives me thin provisioning, snapshots, and live migration without provisioning separate storage infrastructure for the hypervisor.
• iSCSI from TrueNAS — Proxmox connects directly to TrueNAS iSCSI targets for VMs where I want ZFS-level features at the disk-image layer (compression, replication, ZFS snapshots independent of Ceph), or want to keep specific VM disks on a physically separate pool for blast-radius isolation. The K8s truenas-iscsi storage class consumes the same TrueNAS appliance through Democratic CSI, but the two paths are independent — Proxmox’s iSCSI initiator is talking to TrueNAS without K8s in the loop.
• NVMe over Fabrics (NVMeoF) from TrueNAS — for the small set of latency-sensitive VMs where iSCSI’s SCSI-emulation overhead is measurable. NVMeoF preserves the NVMe command set across the fabric instead of translating to SCSI, which trims protocol overhead for workloads that care about it.

The choice is per-VM, not per-pool. Most VMs default to Ceph RBD; specific VMs move to TrueNAS-backed storage only when there’s a concrete reason — a different performance profile, an isolation requirement, or a ZFS-specific feature need. I don’t want to maintain two parallel storage infrastructures unless a workload actually benefits from it.

The result is block storage at two distinct layers — Proxmox VM disks at the hypervisor and PVs at the Kubernetes layer — both served by the same two underlying systems (Ceph and TrueNAS). Keeping the same vendors and protocols across layers keeps the operational mental model consistent: one set of monitoring dashboards, one set of capacity-planning conversations, two consumption interfaces.

Related

• On-Premise Kubernetes Platform — Design rationale and tradeoffs
• Observability Platform — How I monitor this cluster