A $0 Kubernetes lab on Oracle Cloud

Most Kubernetes demos end when kubectl get nodes returns two green rows. I wanted a cluster that would stay online long enough to expose the harder problems: upgrades, storage, secret rotation, DNS, certificate renewal, monitoring, access control and recovery.

The result is a two-node Oracle Kubernetes Engine cluster in Santiago. It runs public sites and the platform services that support them. OpenTofu creates the OCI boundary. Argo CD manages the cluster. OpenBao holds workload secrets. Envoy Gateway provides one shared public edge. VictoriaMetrics and VictoriaLogs record what the platform is doing.

The OCI bill remains at $0. The domain does not.

This is not a claim that two small workers provide a commercial production service. The cluster has no paid reserve, no third worker and no financially backed service-level agreement. It is a permanent engineering lab built with the same controls I want to test elsewhere.

What the cluster proves

The cluster is useful because it joins several systems that are often tested alone:

• a private managed Kubernetes API;
• two private ARM64 workers in separate fault domains;
• one public load balancer shared by every hostname;
• declarative infrastructure and GitOps reconciliation;
• automatic public and private certificates;
• central secret storage with automatic unseal;
• namespace-scoped secret delivery;
• default-deny workload networking;
• metrics, logs, dashboards and network telemetry;
• immutable application releases from private container images.

The important part is not the list. It is the boundary between each item: who creates it, how traffic reaches it, where its state lives and what happens when a node disappears.

Architecture

OpenTofu owns the OCI resources: the compartment, network, OKE cluster, workers, Bastion, IAM, KMS key, backup bucket and budget alerts. Kubernetes creates the public load balancer from an Envoy Service. Argo CD owns the controllers and workloads inside the cluster.

flowchart TB
    user[Public client] --> cf[Cloudflare DNS and proxy]
    operator[Administrator] --> bastion[OCI Bastion session]

    subgraph OCI[OCI home region: Santiago]
        budget[Budget and forecast alerts]
        kms[OCI KMS<br/>software-protected key]
        objects[Versioned Object Storage<br/>state and backups]
        lb[Flexible Load Balancer<br/>10 Mbps]
        api[Private OKE Basic API<br/>Kubernetes 1.35.2]

        subgraph OKE[Private OKE worker pool]
            w1[Worker 01<br/>2 OCPU · 12 GB · 50 GB]
            w2[Worker 02<br/>2 OCPU · 12 GB · 50 GB]
            envoy[Envoy data plane<br/>2 replicas]
            argocd[Argo CD]
            bao[OpenBao<br/>Raft + 50 GB block volume]
            eso[External Secrets]
            obs[VictoriaMetrics · VictoriaLogs<br/>Grafana · Retina]
            apps[Public applications]
        end
    end

    cf -->|HTTPS only| lb
    lb --> envoy
    envoy --> apps
    envoy --> argocd
    envoy --> bao
    envoy --> obs
    bastion -->|TCP 6443| api
    api --> w1
    api --> w2
    kms -->|auto-unseal| bao
    bao --> eso
    eso --> apps
    argocd --> envoy
    argocd --> bao
    argocd --> obs
    objects -. snapshots .-> bao
    budget -. watches spend .-> lb
    budget -. watches spend .-> w1

The cluster uses OKE Basic, so Oracle operates the control plane while I operate the worker pool and everything scheduled on it. The API endpoint has no public address. Both workers use VM.Standard.A1.Flex, Oracle Linux 8 and Kubernetes 1.35.2.

The worker pool is deliberately fixed:

OKE node cycling is disabled. The account has no free compute for a temporary third worker, so a replacement must delete an old worker before it creates the new one. That makes upgrades less convenient, but it prevents an automated rollout from crossing the cost limit.

The $0 envelope

The design uses a Pay As You Go tenancy. That detail matters.

OCI allowance change — June 2026. Oracle now documents 1,500 OCPU-hours and 9,000 GB-hours for Ampere A1 on Always Free tenancies, equal to 2 OCPUs and 12 GB. The current OCI price list separately states that each paid tenancy receives 3,000 OCPU-hours and 18,000 GB-hours per month. This cluster uses that paid-tenancy allowance, which covers 4 OCPUs and 24 GB over a 720-hour month. Oracle can change either offer. Check the current Free Tier documentation, tenancy limits and cost forecast before you copy this layout.

The remaining OCI resources fit inside documented free allocations:

• OKE Basic control plane: free;
• 200 GB of combined boot and block volume storage;
• one Flexible Load Balancer fixed at 10 Mbps;
• software-protected KMS keys;
• Object Storage for Terraform state and small backups;
• OCI Bastion, VCN, NAT Gateway and Service Gateway;
• monitoring, notifications and budget alerts within their free quotas.

OpenTofu turns the cost model into validation. The root module rejects any worker count other than two, any worker shape other than VM.Standard.A1.Flex, and any total other than 4 OCPUs and 24 GB. A second check requires exactly 150 GB of provisioned disk and a 50 GB replacement reserve.

OCI Budgets adds two more signals: one for actual spend and one for forecast spend. A budget is an alert, not a spending lock, so the Terraform checks and plan review still matter.

Network design

The VCN uses separate subnets for the API, workers, VCN-native pod addresses and the public load balancer. The Kubernetes Service CIDR does not overlap the VCN.

flowchart LR
    internet((Internet)) --> cf[Cloudflare proxy]

    subgraph VCN[VCN 10.20.0.0/16]
        igw[Internet Gateway]
        nat[NAT Gateway]
        sgw[Service Gateway]

        subgraph LBS[Public LB subnet<br/>10.20.3.0/24]
            lb[OCI Flexible LB<br/>TCP 443]
        end

        subgraph API[Private API subnet<br/>10.20.0.0/28]
            bastion[OCI Bastion endpoint]
            okeapi[OKE API<br/>TCP 6443]
        end

        subgraph WORKERS[Private worker subnet<br/>10.20.1.0/24]
            node1[Worker 01]
            node2[Worker 02]
        end

        subgraph PODS[VCN-native pod subnet<br/>10.20.2.0/24]
            envoy[Envoy proxy pods]
            services[ClusterIP services]
            workloads[Platform and app pods]
        end
    end

    cf --> igw --> lb --> envoy --> services --> workloads
    admin[Allowed administrator /32] --> bastion --> okeapi
    okeapi --> node1
    okeapi --> node2
    node1 --> nat --> internet
    node2 --> nat
    workloads --> nat
    node1 --> sgw --> oci[OCI KMS and Object Storage]
    node2 --> sgw

OCI VCN-native pod networking gives each pod an address from 10.20.2.0/24. OCI remains the CNI. Calico runs in policy-only mode, so it enforces Kubernetes NetworkPolicy without installing another pod network or changing OCI’s CNI files.

There are three important paths.

Public traffic

Cloudflare publishes the DNS record and proxies the request. The OCI load balancer accepts TCP/443 only from Cloudflare’s published IPv4 ranges. Envoy terminates the public certificate, selects an HTTPRoute, and starts a new connection to the target ClusterIP service.

Administrative services use TLS again between Envoy and the backend. Envoy verifies the private certificate issued by the cluster CA before it connects to Argo CD, Grafana or OpenBao.

Kubernetes administration

The OKE API is private. An administrator creates a short-lived OCI Bastion port-forwarding session from an allowed public /32 address. The Bastion policy can reach only the API endpoint on TCP/6443. The session lasts at most one hour.

OCI IAM decides who may cross the cloud boundary. Kubernetes RBAC decides what that identity may do after authentication. The two controls solve different problems and remain separate.

Private egress

Workers and pods use the NAT Gateway for public registries and package endpoints. They use the OCI Service Gateway for regional OCI services such as KMS and Object Storage. Worker nodes have no public addresses.

Bootstrap and GitOps ownership

tofu apply cannot finish the whole platform by itself because the Kubernetes API is private and OpenBao initialization is intentionally interactive. The build therefore has three stages.

1. OpenTofu builds OCI. It creates the network, OKE, IAM, Bastion, KMS, backup bucket and cost guardrails.
2. Bootstrap installs Argo CD. A local script opens the private API path, renders the pinned Argo CD Helm chart, applies it and creates the App of Apps.
3. Argo CD takes ownership. It adopts its own chart and reconciles every application from Git.

After the handoff, OpenTofu never manages Helm releases and Argo CD never manages OCI foundations. This avoids two controllers fighting over the same resource.

flowchart LR
    tofu[OpenTofu] --> cloud[OCI resources<br/>VCN · OKE · IAM · Bastion · KMS · bucket · budgets]
    bootstrap[Local bootstrap] --> first[First Argo CD install<br/>App of Apps]
    first --> argo[Argo CD]
    argo --> charts[Helm releases and Kubernetes manifests]
    charts --> service[Envoy LoadBalancer Service]
    service --> ccm[OCI cloud controller]
    ccm --> lb[OCI Flexible Load Balancer and frontend NSG]
    argo --> self[Argo CD chart and configuration]
    self --> argo

Each application keeps its Argo CD Application, Helm values and any extra manifests in one directory. Sync waves establish dependencies: storage and namespaces first, then networking, certificates, OpenBao, secret delivery, DNS, monitoring and workloads.

Automated sync, pruning and self-healing handle ordinary drift. Durable resources use explicit prune confirmation. OpenBao volumes, stateful namespaces and similar objects cannot disappear because one file was removed by mistake.

What runs in the cluster

The following table describes the current platform, not a planned diagram.

Most controllers use one replica because the cluster has only two workers and little spare memory. The public Envoy data plane is the exception: it runs one pod on each worker with a disruption budget that keeps one available during a voluntary drain.

Secrets and identity

OpenBao runs one server with integrated Raft storage. OCI KMS decrypts its seal key through the worker instance principal, so the server can unseal itself after a restart. No OCI API key is stored in the pod.

Initialization still requires an operator. The initial recovery keys and root token are created once and stored offline. A named administrator account replaces the root token for normal work, and the initial root token is revoked after that login is tested.

External Secrets uses Kubernetes authentication. Each workload gets three matching objects:

1. an OpenBao policy for one exact KV v2 path;
2. an OpenBao Kubernetes role bound to one service account and namespace;
3. a namespace-scoped SecretStore used by an ExternalSecret.

sequenceDiagram
    participant App as Workload service account
    participant ESO as External Secrets Operator
    participant Auth as OpenBao Kubernetes auth
    participant KV as OpenBao KV v2
    participant K8s as Kubernetes Secret

    ESO->>App: Request projected service-account token
    ESO->>Auth: Login with token and bound role
    Auth-->>ESO: Short-lived OpenBao token
    ESO->>KV: Read one policy-approved path
    KV-->>ESO: Secret fields
    ESO->>K8s: Create or update namespace Secret

The controller requests a fresh five-minute OpenBao token for each reconciliation. It does not keep a broad cluster token in its cache. Cluster-wide secret stores and push APIs are disabled because this cluster does not use them.

Google OIDC protects Argo CD, Grafana and OpenBao. Each service has its own OAuth client. Authorization remains local to the service: a successful Google login does not grant access unless the email also matches the configured policy.

Storage and failure behavior

The workers already require boot volumes, so the cluster uses them for disposable observability data. Local Path Provisioner creates directories on those disks and exposes them as PersistentVolumes. A pod restart on the same node keeps the data. Loss or replacement of that node can lose it.

That failure mode is acceptable for seven days of metrics and three days of logs. It is not acceptable for OpenBao.

flowchart LR
    allowance[200 GB combined<br/>boot and block allowance]
    allowance --> boot1[50 GB<br/>worker 01 boot]
    allowance --> boot2[50 GB<br/>worker 02 boot]
    allowance --> bao[50 GB<br/>OpenBao block volume]
    allowance --> reserve[50 GB<br/>replacement reserve]

    boot1 --> local1[Local PVC directories]
    boot2 --> local2[Local PVC directories]
    local1 --> metrics[VictoriaMetrics<br/>4 GiB · 7 days]
    local2 --> logs[VictoriaLogs<br/>4 GiB · 3 days]
    bao --> raft[OpenBao Raft]

OpenBao uses one 50 GB OCI Block Volume through the OCI CSI driver. Its storage class has Retain reclaim policy, delayed binding and no expansion. The volume can detach from one worker and attach to the other. The data survives a pod restart and a worker replacement, though the single OpenBao pod still causes a short outage while it moves.

The current disk plan is:

A PersistentVolume is not a backup. Raft snapshots belong in the private, versioned Object Storage bucket created by OpenTofu.

CloudNativePG is also a later milestone. PostgreSQL replication can protect against one pod or worker failure. Object Storage backups are still required for deletion, corruption and bad migrations.

Observability within eight gigabytes

The monitoring stack has to diagnose the cluster without becoming its largest workload.

VictoriaMetrics stores seven days of metrics in a 4 GiB local PVC. VictoriaLogs stores three days of logs in another 4 GiB local PVC and also enforces a 4 GiB application limit. Both stop ingestion while at least 10 GiB remains free on the worker filesystem, which leaves room for the operating system, images and container layers.

vmagent scrapes every 30 seconds. vmalert evaluates rules every 30 seconds. Grafana uses the Prometheus-compatible VictoriaMetrics endpoint and a pinned VictoriaLogs data-source plugin. The log collector runs on each worker and reads pod logs. Retina supplies pod, DNS and network metrics that OKE’s default node counters do not label well enough for useful dashboards.

Application release path

The public website is built outside the cluster on GitHub-hosted Actions runners. A successful workflow:

1. builds a small ARM64 image;
2. pushes it to private GitHub Container Registry with an immutable short-SHA tag;
3. records the image digest in GitOps;
4. commits the new reference through a narrowly scoped GitHub App;
5. lets Argo CD reconcile the commit.

The cluster pulls the private image with a namespace-scoped registry secret. External Secrets builds that secret from a read-only package token stored in OpenBao. The GitHub App used to write GitOps cannot pull images, and the package credential cannot change Git.

The tag tells me which commit produced the image. The digest tells Kubernetes which bytes to run.

Security controls

The cluster uses several small controls rather than one broad perimeter:

• private API, workers and pod subnets;
• time-limited Bastion sessions restricted to administrator /32 addresses;
• least-privilege OCI IAM for operators, workers and the OKE cluster principal;
• Cloudflare-only source ranges on the public load balancer;
• separate TLS sessions at the edge and administrative backends;
• Calico policy-only enforcement and namespace-level default-deny policies;
• restricted Pod Security labels, non-root containers, read-only root filesystems and dropped capabilities;
• one OpenBao policy and role per secret consumer;
• Google OIDC with explicit local authorization;
• immutable chart versions, image tags and application digests;
• OCI cost forecasts and Terraform capacity checks.

These controls do not make the lab immune to failure. They make access and ownership narrow enough to inspect.

Known limits

The cluster is not highly available as a whole. It can survive some single-pod and single-worker failures, but several platform services will pause while Kubernetes reschedules them. That is the correct description of a two-node lab.

Resource use

The cluster dashboard shows total pressure across both workers. The node dashboard adds the 24-hour shape for one worker, where the CPU and memory series stayed stable under the current workload.

Cluster utilization

At capture time, the cluster used 15.1% of its CPU and 40.1% of its memory. CPU limits exceed physical capacity because limits are ceilings, not reserved cores.

View full image

Worker history

The selected worker reported 15.2% CPU busy and 34.5% RAM used at capture time. Its 24-hour CPU and memory traces stayed stable.

View full image

Captured from Grafana with VictoriaMetrics as the data source. Open either image to inspect the full-resolution dashboard.

Grafana screenshot

Full-size image

Close

Repository

The public repository contains the OpenTofu stack, GitOps applications, bootstrap scripts and component notes.

The source is available in the free-k8s-oracle-cloud repository.