Overview

This post documents our experimentation running 1,000 CronJobs on a single-node k3s cluster with embedded etcd, stress-testing the Kubernetes CronJob controller and etcd under high scheduling load.

Setup

Platform: k3s (single node) with embedded etcd
Workload: 1,000 CronJobs (* * * * * — every minute)
Job spec: sleep 10 via busybox, concurrencyPolicy: Forbid
Namespace: cronjob-stress
Node capacity: max-pods=1024

Initially I tried with 1_000 but it becomes disaster. Next i though let’s do with 500.

Key Findings

CronJob Scheduling Lag

At 500 CronJobs (mid-test), 287–341 CronJobs (57–68%) were consistently falling behind their 1-minute schedule by an average of ~165 seconds (missing 2–3 cycles).

Metric	Value
Total CronJobs	500
On schedule	~190
Late (>2 min)	~310
Max lag	252s
Avg lag	~165s

The lag was not transient — it grew steadily over time, confirming steady-state controller backpressure.

Root Causes

1. Pod Startup Latency > Schedule Interval

Pods took ~90 seconds to start (image pull + cgroup setup), while the schedule interval is 60 seconds. With concurrencyPolicy: Forbid, the next schedule is blocked until the active job completes — causing every CronJob to miss at least one cycle.

podStartE2EDuration="1m32s"