Batch 04 vs Batch 05 vs Batch 06

What changed, and what it bought us

This report contrasts three consecutive training runs of the campus infrastructure detector (projector, whiteboard, fire_extinguisher, door_sign) and isolates the contribution of each change. Batch 04 is the baseline: YOLOv11n on the original aggregated dataset. Batch 05 holds the model fixed and swaps in an improved custom dataset with rebalanced sourcing, denser annotation, and more in-house captures — the dataset uplift. Batch 06 holds that improved dataset fixed and swaps the backbone from YOLOv11n (2.6M params) to YOLOv11s (9.4M params) — the model uplift.

The headline finding is that the dataset change moved the needle far more than the model upgrade. Going from Batch 04 → 05, test mAP@0.5 jumps +5.4 pp and macro recall jumps +11.9 pp at unchanged precision (1.000). Going from Batch 05 → 06, test mAP@0.5 actually dips slightly (−0.8 pp) but the stricter mAP@0.5:0.95 climbs +0.8 pp and the whiteboard class tightens dramatically (mAP@0.5:0.95 +3.1 pp). Batch 06 also pays a small precision cost on door_sign (1.0000 → 0.9697) — the first non-perfect precision in the project's history.

Project 1 → Project 2 motivation

The Project 1 model (Batch 04) shipped with strong precision but uneven recall and localisation:

Four failure patterns drove the Project 2 plan:

1. projector and whiteboard recall ~20 pp below the other two classes. Sourced from a narrow style distribution in v1; the model had never seen enough HUB-campus framings. Counter-strategy: Cat. B #5 (targeted dataset expansion), #8 (class balance), C #10 (multi-scale capture distances).
2. fire_extinguisher over-represented in v1 (848 source pairs vs. 200–319 for others). Counter-strategy: Cat. B #5/#7/#8 — rebuild the corpus with balanced sourcing and re-checked labels.
3. Even after the dataset uplift, whiteboard and door_sign retained sub-pixel localisation drift on the strict mAP@0.5:0.95 metric. Counter-strategy: Cat. A #1 — upgrade the backbone to YOLOv11s (9.4 M params) for finer-grained box regression.
4. Live FPS at 4–5 on the Project 1 path made the model unusable for real-time UI demos and made the larger backbone in (3) a deployment liability. Counter-strategy: Cat. C #11 — redesign the inference pipeline around ONNX + GPU runtime with a confidence-threshold slider.

Batches 05 and 06 implement the diagnoses as two single-variable steps, so the test-metric deltas are causally attributable to (a) the dataset interventions and (b) the backbone upgrade respectively.

Three single-variable steps

Each batch changes exactly one variable from the previous one. Every other hyper-parameter (lr, batch, schedule, augmentation, seed) is held fixed.

Shared pipeline & hyperparameters

All three runs flow through the same notebook pipeline (nb01_data_collection → nb05_model_evaluation) on the same hardware (CUDA · RTX 4060) and with identical hyperparameters, apart from the two variables under study.

v1 (Batch 04) vs v2 (Batches 05 + 06)

Batches 05 and 06 share the same dataset, so the dataset-level deltas are between Batch 04 (v1) and Batches 05/06 (v2).

Source availability (before capping)

Stratified split distribution

Class distribution

Label density & geometry

The v2 dataset is meaningfully denser in bounding boxes at constant image count. Empty-label images drop across every split — the curator pruned scenes with no visible target and added multi-instance scenes.

Box-area histogram

Image-dimension scatter

Label distribution (per-class XY centres)

How the three runs converged

Best-epoch summary

04 → 05 (dataset uplift): train losses tick up a hair (v2 is harder to memorise — more multi-instance scenes) while val losses fall sharply — the model is generalising better. 05 → 06 (model uplift): every loss drops, including the training losses, because the larger backbone has more capacity to fit the same data. Val mAP@0.5 dipping 0.5 pp despite the loss drop tells us the loose-IoU bucket was already saturated; the extra capacity went into refining localisation (visible in the stricter mAP@0.5:0.95 jump).

Training curves

Sanity predictions (training-time)

Held-out 80-image test split

Overall metrics

Per-class metrics

Two clear class-level stories: whiteboard is the headline winner of Batch 06 — recall is now perfect (1.0000), mAP@0.5 saturates at 0.995, and mAP@0.5:0.95 jumps +3.1 pp — the largest single per-class gain anywhere in this comparison. Conversely, door_sign regressed in Batch 06: precision broke its perfect streak (0.9697) and recall fell −5.7 pp. This is the first time any class has lost ground when a single variable was changed.

Precision–Recall and F1 curves

Confusion matrices

Normalised (Ultralytics)

Custom dual-panel view

Qualitative predictions

Consolidated headline table · baseline vs. each experiment vs. final

A single side-by-side view as required by the Project 2 brief:

Bold = best across the three batches per row. The final combined model wins on the strict mAP@0.5:0.95 macro and on three of four classes' mAP@0.5:0.95.

Runtime — Project 1 path vs. Project 2 path

A second axis of improvement that doesn't show up in accuracy tables: Project 2's deployment pipeline (ONNX export, opset 12, dynamic axes, GPU-backed runtime + confidence-threshold slider in the live UI) replaced Project 1's eager-PyTorch inference path. Same hardware (RTX 4060), same input size (640 × 640).

Per-backbone model latency

End-to-end throughput

The throughput uplift outpaces the raw model-latency drop because the Project 1 path also paid for per-frame CPU↔GPU copies and webcam-sync stalls; the Project 2 ONNX path keeps the model resident on the GPU and decouples capture from inference. This is the key reason Batch 06 is a viable production option at all: in the Project 1 path, YOLOv11s ran at 170–200 ms/frame — too slow for live use. Under the Project 2 path it drops to 20–30 ms, still slightly slower than YOLOv11n (15–20 ms) but well inside real-time, so the 9.4 M-param backbone is no longer a deployment liability — only a memory-footprint trade-off (see operational trade-offs below).

Post-processing calibration (Cat. C #11)

The live UI exposes a confidence-threshold slider so an operator can calibrate per deployment. For the project-default operating point of 0.25 the test-time metrics above hold; raising it above ~0.45 trades recall for precision on door_sign in Batch 06 — a useful lever given the precision regression flagged in the error analysis below.

Where did each gain come from?

Because every step is a single-variable change, each gain is attributable either to dataset (Batch 04 → 05) or model (Batch 05 → 06). The combined column reflects the total swing from baseline (Batch 04) to the final run (Batch 06).

~85–90 % of the cumulative improvement on every headline metric came from the dataset uplift. The model uplift produces a small lift only on the stricter-IoU metric — exactly where a higher-capacity backbone is expected to help (fine-grained box regression).

Why didn't YOLOv11s sweep the board?

YOLOv11s has 3.6× the parameters of YOLOv11n (9.4 M vs 2.6 M) and 3.3× the GFLOPs (21.6 vs 6.5). Naively, a clean win across the board is expected. Two reasons it doesn't materialise:

Operational trade-offs

Remaining failure modes

door_sign remains the project's hardest class on the strict IoU metric (mAP@0.5:0.95 ≈ 0.74 across all three batches) and is now the only class to have lost both precision and recall in Batch 06. Door signs are also the smallest in absolute pixel area in the test split. Three independent levers worth trying in a future batch:

• Higher training resolution (imgsz=896 or 1024) — direct fix for small-object localisation drift.
• Class-aware augmentation — more aggressive geometric augmentation on door_sign (perspective, scale jitter) to break the precision wobble Batch 06 introduced.
• Modest regulariser bump on YOLOv11s — weight decay 5e-4 → 1e-3 and/or label-smoothing 0.05 to absorb the extra capacity without hurting the easier classes.

Where Batch 06 fails (and where it wins)

The Batch 06 final-model failures cluster into three patterns visible in the qualitative outputs:

1. door_sign confident-but-wrong predictions (precision 1.000 → 0.9697). The first non-perfect precision in the project's history. The qualitative grid (figures/batch06/qualitative_predictions.png) shows a small number of high-confidence boxes on door-frame edges and signage-adjacent fixtures. The confusion matrix (figures/batch06/confusion_matrix_normalized.png) shows a faint door_sign → background confusion absent in Batch 05.
2. door_sign recall regression (0.9714 → 0.9143). Two test images that Batch 05 detected at 0.5 + drop below threshold in Batch 06. Both contain door signs at the smallest pixel area in the test set (long-edge ≲ 40 px at 640 × 640). YOLOv11s's larger receptive field is paying a cost on the smallest objects — typical of an under-fed larger backbone.
3. whiteboard and fire_extinguisher sub-pixel localisation gains. The flip side: whiteboard mAP@0.5:0.95 jumps 3.1 pp and recall hits 1.000 in Batch 06. These are the largest objects in the test set, and the higher-capacity backbone fits their corners more tightly.

Three independent levers worth trying in a future Batch 07:

• Higher training resolution (imgsz=896 or 1024) — direct fix for small door_sign localisation drift.
• Class-aware augmentation on door_sign (perspective, scale jitter) to break the precision wobble.
• Modest regulariser bump on YOLOv11s (weight decay 5e-4 → 1e-3, label-smoothing 0.05) to absorb the extra capacity without hurting the easier classes.

Project 1 constraints, re-affirmed

All ethical constraints from Project 1 continue to apply and were re-checked for the v2 dataset and the deployment pipeline:

• No identifiable faces. v2 HUB-campus captures were framed on infrastructure (projectors, whiteboards, fire extinguishers, door signs). Any incidental human presence was screened out during the v2 curation pass; no face is recognisable in any retained image. The live inference UI does not retain frames.
• No license plates or vehicles. Captures are interior classroom and corridor scenes; no parking areas were photographed.
• No personal data. No names, ID cards, schedules, or contact information are visible in any image. The confidence-threshold slider in the live UI does not log or transmit captures.
• Provenance. Roboflow and Kaggle source images were used under their published open licences; HUB-campus captures were taken on-site by the author for the purpose of this project.
• Model behaviour. Batch 06's door_sign precision regression (1.000 → 0.9697) is documented and disclosed above — the model is not marketed as having perfect precision after Project 2.

Recommendation

Across three single-variable steps:

Artefacts & reproduction

# All three runs are reproducible from the notebooks in notebooks/ using
# the dataset YAML at data/dataset/data.yaml. Seed = 42, deterministic = true.
# Switch the `model=` line in nb04 between yolo11n.pt and yolo11s.pt to
# reproduce Batch 05 vs Batch 06; rebuild the dataset for Batch 04 vs 05.
jupyter execute notebooks/04_model_training.ipynb
jupyter execute notebooks/05_model_evaluation.ipynb

Markdown twin of this report: technical_report.md · Prior two-way comparison: Batch 04 vs Batch 05 · ← Back to project home