Cloud Run in the middle

Setup

The experiment: six pipelines, two rounds

Both rounds share one shape: a load generator with known send timestamps, Cloud Run in the middle, and a sink whose contents can be counted and timed afterward. Round 1 varied the source and the language; round 2 fixed the source (Kafka, 12 partitions, 1 KB Avro) and varied the sink, after an external review added binding methodology amendments (same-clock primary metrics, per-run consumer-group health gates, drain timeouts with backlog accounting, and scripted teardown).

Numbers quoted below are p50 unless stated; all raw JSON, orchestration code, and the two per-round HTML reports live in the repo (round-1-cloudrun-pipelines/, round-2-kafka-sinks/). Round 2's primary latency metric is same-clock: consume→sink-ack measured entirely on the instance, immune to cross-machine clock skew. Cross-clock numbers (produce→consume) are labeled as such wherever they appear.

Finding 1

Your sink is your latency floor

Nothing you do to the service — language, concurrency, CPU, rate — moves the median until you change what the request path waits for.

Round 1 made this point by exhaustion. The HTTP→Cloud Storage API held p50 ≈ 51–63 ms across a 50× rate sweep, concurrency 1→80, and a doubling of CPU and memory — in all three languages at once. The GCS write (~35–50 ms server-side) simply dominates everything else in the request.

Round 2 then priced the floors themselves, with the skew-free instrument: the same Kafka consumer body, swapping only the write call. The spread is not percent — it is four orders of magnitude.

If the caller's response time matters more than write durability, the floor can be dodged rather than lowered: round 1's async mode acknowledged first and wrote to GCS in the background, taking the caller-visible p50 from ~54 ms to 7–10 ms — at the price of an at-least-once-after-ack durability window and mandatory backpressure (bounded queue, 503 shedding). Fine for telemetry; wrong for payments.

Finding 2

Language is mostly a configuration problem

Across both rounds there is exactly one scenario family where a language collapsed — and two cheap configuration changes that fully repaired it.

Python's synchronous BigQuery path fell apart under Pub/Sub push at concurrency 80: eighty concurrent requests multiplexed onto a single uvicorn event loop, each blocking on an insertAll. Go and Rust, with real per-request parallelism, never noticed.

Everywhere else, the languages converged — or inverted expectations:

• Streaming consumer (round 1, Kafka→Pub/Sub): Python won. 28 ms e2e at 500 msg/s vs Go's 35 ms and Rust's 47 ms, at identical throughput ceilings. Its hot loop is librdkafka, fastavro, and gRPC — C all the way down; the interpreter only shuttles buffers. Rust lost its margin on a slower REST-batching Pub/Sub client: the sink client, again, not the language.
• Round 2 (Go vs Python across six sinks): parity within noise, except the Storage Write API append gap (16 vs 50 ms) — a client-library maturity difference, pre-registered as an expected asymmetry before the run.
• Cold starts are the one place compiled languages win outright — but with a twist covered in Finding 5.

Finding 3

Throughput is bought with batching, not hardware

The measured ladder spans 0.9 to 50,000 rows per second — and every rung is a batching decision, not a CPU decision.

Three things in that chart deserve emphasis:

• The bottom rung is catastrophic, not slow. Per-message commits into external Iceberg delivered 163 of 9,000 expected rows — 1.8% — while accumulating conflicts. A sync write per message is how a streaming pipeline silently becomes a backlog machine.
• The middle rungs are free. Moving from per-message to 500-row/200 ms batches requires no new infrastructure and multiplies one instance's ceiling ~300×. CPU doubling bought only 1.7× (15k→25k, and even that was broker-capped).
• The top rung held under audit. 50k rows/s ran for 120 s with zero undrained backlog and per-run loss accounting — 5,991,840 rows counted in BigQuery against 2,995,920 sent × 2 consumers.

Round 1 adds the autoscaling caveat: none of this scales itself for pull-based consumers. Cloud Run's autoscaler watches HTTP, not consumer lag — the Kafka services only performed when pinned (min-instances = max-instances, CPU always allocated), scaling near-linearly with pinned count (one instance ≈ 2,000 msg/s in round 1's consumer; four ≈ 5,000/s per language, then broker-capped).

Finding 4

The lakehouse has a toll booth

“Iceberg” is one word for two very different write paths. Managed Iceberg behaves like BigQuery. External Iceberg behaves like a distributed systems exam.

BigQuery managed Iceberg tables are a free choice. Writes go through the same insertAll and land at the same cost (41–50 ms per batch vs 34–38 ms native); read benchmarks came back in the same class (sub-1.5 s aggregates on tens of millions of rows, with occasional cold-scan outliers). Pick managed Iceberg for format openness — you pay approximately nothing in performance.

External Iceberg — Parquet on GCS behind a BigLake REST catalog, written by pyiceberg / iceberg-go — is a different sport. Every commit is a catalog round-trip plus object writes: a ~2–3 s floor per commit regardless of size. And commits from concurrent writers contend optimistically:

The working recipes that fall out of the data:

• Streaming into external Iceberg: funnel to a single writer (or ≤2), commit 5k+ rows every 5–10 s, accept ~10–15 s sink-ack latency, and schedule compaction. At one writer, conflicts ≈ 0.
• Volume into external Iceberg: don't stream at all. The Dataproc Serverless Spark batch wrote 9.79M rows in 258.6 s (~37.9k rows/s) with 79 s of provisioning — scheduled micro-batches every 5–15 minutes beat a fleet of conflicted streaming writers.
• If “Iceberg” is a compliance checkbox, not an engine requirement: managed Iceberg gives you the format at native-BigQuery speed, and the whole section above stops applying.

Finding 5

Operational physics: cold starts, rebalances, failure & the bill for pinning

Cold starts rank Go ≈ Rust ≪ Python — with an asterisk on Rust

A rolling restart pauses a consumer group for ~24–28 s

Round 2 measured what a redeploy does to a live Kafka consumer group: the longest gap with zero consumed messages was 27.7 s with the default eager assignor and 23.7 s with cooperative-sticky — a pause, not a slowdown (between pauses, consume spacing was unchanged). The assignor choice is worth ~15%; graceful shutdown that commits offsets on SIGTERM is worth more. And the nastiest failure mode found in either round remains: after one round-1 redeploy, librdkafka wedged mid-rebalance holding zero partitions, logging no error, at half throughput — which is why round 2 gated every run on “12/12 partitions assigned + canary rows flowing.” Monitor group assignment yourself; Cloud Run won't.

Failure semantics: measured, and gentle

Killing 1 of 2 consumers mid-run at 2,000 msg/s (at-least-once inserts, 500-row batches, 500 ms offset commits) produced 9 duplicate rows and 0 lost messages out of 240,018 — 0.0019% duplication, visible only because every run counts distinct_ids against sent. That is the at-least-once contract working as designed: plan for dedup keys downstream, not for loss.

What pinning costs

The price of “Cloud Run as a Kafka consumer” is that the request-driven billing story inverts: pinned, always-allocated instances bill ~$24.50/month each (1 vCPU / 1 GiB) whether or not traffic flows. Four pinned instances ≈ $98/month — still cheap in absolute terms, but it is GKE-shaped economics. Round 2's Worker Pools runs showed the cleaner way to buy the same thing: identical pipeline results with no HTTP shim, purpose-built for pull workloads (and, since this study ran, Google ships a lag-based Kafka autoscaler for worker pools — the pin-everything workaround now has an official alternative worth testing).

Gotchas that cost real debugging time

• --vpc-egress=all-traffic without Cloud NAT silently breaks Google API access — round 2 ran private-ranges-only from the start.
• The single-broker Kafka VM hit disk-full mid-benchmark (fixed with 4 GB/1 h retention); an S4 100 KB producer run crashed on a full local queue and was rerun. Both incidents are disclosed in the round-2 report's limitations.
• Pub/Sub redelivery means sinks see slightly more than sent (240,077 received per 240,018 sent was typical) — design counts as at-least-once everywhere.

Cost

Cloud Run is rarely the expensive line item

Both rounds' cost models agree: at high concurrency the compute in the middle costs fractions of a dollar per million operations — the sink and the messaging fabric dominate.

Two structural rules: concurrency is a discount (conc 80 vs conc 10 cut the per-request compute ~30% at identical latency, since I/O-bound requests share instances), and pinning is a subscription (~$24.50/inst/month, traffic or not). The studies themselves cost ≈ $12–16 (round 1) and ≈ $9–13 (round 2, under its $22 hard stop).

Synthesis

Decision guide

Pick the requirement; every recommendation below is backed by a measured configuration from these runs, with its caveat attached.

Appendix

Method & limitations

How latency was measured. Round 1: client-side (vegeta) for HTTP; publish→row-visible or produce→pulled for pipelines, stamped by the load-generator VM. Round 2's primary metric is consume→sink-ack, both timestamps from the same instance clock (skew-free); cross-machine numbers are reported separately. Per-service clock probes ran before and after every round-2 run — though the probe's offset estimate (~65 ms ≈ its own RTT) is an artifact of an uncorrected formula, which is precisely why the primary metric avoids cross-clock arithmetic.

Run hygiene (round 2, per binding amendments). Every measured run required 12/12 Kafka partitions assigned plus flowing canary messages before the window; carried a 300 s drain timeout with undrained_backlog, conflict, retry, and duplicate counts as mandatory result fields; and the matrix ran under a tested teardown script with a $22 spend stop. Aggregates regenerate byte-identically from the raw JSON in both rounds.

Know the edges before transplanting numbers:

• Single region (europe-west3), single month (July 2026), all resources co-located — cross-region or cross-cloud latencies will differ substantially.
• Kafka was one e2-medium broker: a protocol-realistic harness, not a capacity statement. The 25k msg/s consumer ceiling and the 5k/s round-1 figures are broker-capped lower bounds.
• Python ran a single uvicorn worker throughout round 1 — its collapse and async shedding are floors that multi-worker configs would likely raise.
• Cold-start samples are tiny (n = 2 per cell) and round 2 deliberately skipped Rust and cold starts to spend budget on the sink dimension.
• Batch-mode acks precede durability in round 1's combo B (ack→sink window); round 2's sink-ack metric closes exactly this gap.
• S4's managed BigQuery-subscription delivery is visibility-polled every 5 s, so it is reported as lag curves, not milliseconds; its Kafka→Pub/Sub ingress (25–34 ms p50) and the S4b pull path (7–9 ms to BQ) are directly measured.