How this scales

The pipeline as parallel jobs

The public map is intentionally small, but the technical pattern is the part meant to travel. BirdNET inference is embarrassingly parallel: every clip is analysed independently (clips → BirdNET → detections), so a clip assigned to one worker never has to wait on any other. That shape maps onto a matrix execution model trivially — split the clip list into N disjoint shards, hand each shard to its own job, and merge the per-shard detection tables at the end. No coordination, no shared state, no inter-worker communication: the only thing a worker needs to know is its own shard index and the total shard count.

Benchmark

Wall-clock, 1 shard vs. 8-shard matrix

Benchmark data is not yet available.

No benchmark has been recorded yet. The wall-clock comparison and speedup figure will appear here once a real 8-shard run completes via the GitHub Actions matrix (or the equivalent C3SE Alvis SLURM array job below) and manifest.benchmark is populated with measured timings. Until then we show nothing rather than present zero-filled timings as evidence.

Same code on C3SE Alvis

The exact shard command that runs in the GitHub Actions matrix also runs unchanged as a SLURM array job on C3SE Alvis, the NAISS GPU cluster in Göteborg. The only difference is where the shard index comes from: SLURM provides $SLURM_ARRAY_TASK_ID where GitHub Actions provides ${{ matrix.shard-index }}— the same integer, sourced differently. This is not a toy translation: it is what a research software engineer at a Swedish HPC center (C3SE / NAISS) would actually submit with sbatch, allocation flag and all. Bumping --cpus-per-taskbuys each shard more cores for BirdNET's internal threading, shortening per-shard wall-clock without changing the partitioning at all.

#!/usr/bin/env bash
#SBATCH --job-name=birdnet-shard
#SBATCH --array=0-7
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH -A NAISS-XXXX-YYY        # your project allocation
module load Python/3.11
cd "$SLURM_SUBMIT_DIR/ingest"
uv sync
uv run python -m ingest.birdnet_shard \
  --clips ../clips.parquet \
  --shard-index "$SLURM_ARRAY_TASK_ID" \
  --shard-count 8 \
  --out "../out/shard-${SLURM_ARRAY_TASK_ID}.parquet"

Honest caveats

GitHub Actions runners aren't dedicated cores; these numbers are wall-clock, not CPU-time, and are subject to noisy-neighbour variance.
Real HPC scaling needs data staging (a parallel filesystem, not an artifact download), fault tolerance for preempted jobs, and shared scratch space.
This benchmark is symbolic — it demonstrates the parallel pattern, not raw scientific throughput.