Batch 04vsBatch 05

00Abstract

This report contrasts two consecutive training runs of the campus infrastructure detector (projector, whiteboard, fire_extinguisher, door_sign). Both runs share the same backbone (YOLOv11n, 2.6M params), input size (640 × 640), optimiser (SGD), schedule (100 epochs, patience 15), and seed (42). The only material variable is the training corpus: Batch 05 was retrained on an updated custom dataset with more balanced per-class sourcing, a larger share of in-house door-sign captures, and richer multi-instance scenes.

00bBaseline diagnosis — Project 1 → Project 2 motivation

The Project 1 model (Batch 04) shipped with strong precision but uneven recall:

Two failure patterns drove the Project 2 plan:

1. projector and whiteboard recall ~20 pp below the other two classes. Sourced from a narrow style distribution in v1 (Roboflow / Kaggle); the model had never seen enough HUB-campus framings of these two classes. Counter-strategy: Cat. B #5 (targeted dataset expansion on projector/whiteboard), #8 (class balance), and C #10 (multi-scale capture distances).
2. fire_extinguisher over-represented in v1 (848 source pairs vs. 200–319 for others); cap-to-200 left a narrow style slice. Counter-strategy: Cat. B #5/#7/#8 — rebuild the corpus with balanced sourcing and re-checked labels.

Batch 05 therefore holds every model / training knob fixed and changes only the dataset, so any test-metric delta is causally attributable to the Project 2 dataset-side interventions listed in §1.2.

01Experimental setup

Both runs were executed end-to-end through the same notebook pipeline (nb01_data_collection → nb05_model_evaluation) on the same hardware and with identical hyperparameters. Because every controllable variable is held fixed, every metric delta below is attributable to the dataset change.

Raw exports (Roboflow / Kaggle / custom HUB captures)
        │
    [NB01]  Aggregate ~200 (image, label) pairs/class
        │
    [NB02]  Remap class IDs · stratified 70/20/10 split
        │
    [NB03]  Health check: balance, box geometry, leakage
        │
    [NB04]  Train YOLOv11n · 100 epochs · SGD · patience 15
        │
    [NB05]  Evaluate on held-out test split (80 images)
        │
    [NB07]  Export to ONNX (opset 12, dynamic axes)

Identical hyperparameters

1.2 Improvement strategies applied (Project 2 brief)

This report maps to the Project 2 improvement-strategy catalogue as follows. The Project 1 model (Batch 04) is the official baseline; every strategy below is realised in Batch 05.

A backbone upgrade (Category A #1) is the subject of the companion three-way report (Batch 06 — YOLOv11s). Together the two reports exercise strategies in all three categories.

1.3 Environment specification

02Dataset comparison

2.1 Source availability (before capping)

The new dataset narrows the gap between over- and under-sourced classes — trimming the previously dominant fire_extinguisher pool while broadening whiteboard and adding more in-house door_sign captures.

2.2 Stratified split distribution

Batch 04 produced a perfectly balanced split; Batch 05 has a mild shortfall on projector after deduplication and filtering.

Class distributionBatch 04

Class distributionBatch 05

2.3 Label density and box geometry

The updated dataset is meaningfully denser in bounding boxes — Batch 05 carries 1,151 total boxes vs Batch 04's 997, despite the identical image budget. Empty-label images dropped across every split.

Box-area histogramBatch 04

Box-area histogramBatch 05

Image dimensionsBatch 04

Image dimensionsBatch 05

Label distribution · per-class XYBatch 04

Label distribution · per-class XYBatch 05

03Training dynamics

3.1 Best-epoch summary

3.2 Training curves

Training curvesBatch 04

Training curvesBatch 05

3.3 Sanity predictions (training-time)

Sanity predictionsBatch 04

Sanity predictionsBatch 05

04Test evaluation — held-out · 80 images

4.1 Overall metrics

4.2 Per-class metrics

The two weakest classes in Batch 04 — projector and whiteboard — were the largest beneficiaries. Both now sit above 0.975 mAP@0.5 and break 0.94 mAP@0.5:0.95. The minor regressions on the stricter mAP@0.5:0.95 metric for fire_extinguisher and door_sign are localised tightness losses; both classes still record higher recall and higher mAP@0.5.

4.3 Precision–Recall curves

PR curveBatch 04

PR curveBatch 05

4.4 F1 curves

F1 curveBatch 04

F1 curveBatch 05

4.5 Confusion matrices · normalised

Confusion · normalisedBatch 04

Confusion · normalisedBatch 05

4.6 Confusion matrices · custom view

Confusion · customBatch 04

Confusion · customBatch 05

4.7 Qualitative predictions

Qualitative predictionsBatch 04

Qualitative predictionsBatch 05

4.8 Consolidated headline table (baseline vs. final)

A single side-by-side view of the official Project 1 baseline (Batch 04) against the Project 2 final model (Batch 05) on the held-out test split.

04bInference performance — runtime

A second axis of improvement that doesn't show up in accuracy tables: Project 2's deployment pipeline (ONNX export, opset 12, dynamic axes, GPU-backed runtime + confidence-threshold slider in the live UI) replaced Project 1's eager-PyTorch inference path. Same hardware (RTX 4060), same input size (640 × 640), YOLOv11n backbone.

05Discussion

5.1 Why did Batch 05 win?

The improvement is not a hyperparameter win — every knob was identical. Three dataset-level changes explain the lift:

1. Better source balance. Batch 04's fire_extinguisher pool (848 candidates) and projector pool (319) dominated the cap-to-200 step, so surviving training images were drawn from a narrow style distribution. Batch 05's source pools sit between 238–249 across all four classes — the cap is a much shallower filter and per-class style coverage is broader.
2. More custom in-house data. Batch 05 adds more HUB-campus captures to the door_sign set with materially different framing variety, and the new captures broaden the projector and whiteboard distributions toward real classroom conditions — which the test split also samples.
3. Multi-instance scenes. Total annotated boxes climbed from 997 → 1,151 (+15.4%) at constant image count. The model sees more boxes per gradient step and learns to handle co-occurring objects, lifting recall and the stricter IoU buckets that dominate mAP@0.5:0.95.

5.2 Why early-stop at 87 epochs?

The patience-15 trigger fired because val mAP plateaued shortly after the epoch-70 best. Combined with the lower final val losses, Batch 05 reached a flatter optimum earlier — more data of higher quality reduced the gradient steps needed to saturate.

5.3 Train-loss vs val-loss split

A higher train loss with a lower val loss is the canonical "more representative training data" outcome. Batch 05's training set is harder (more boxes per image, more scene variety) but the model that learns to fit it transfers cleanly to val and test.

5.4 Remaining failure modes

door_sign and fire_extinguisher retained the same recall ceiling on the strict 0.5:0.95 IoU band (−2.8 to −3.7 pp). These two classes are the smallest in absolute pixel area in the test split, so sub-pixel localisation drift is penalised. A follow-up batch should consider:

• Higher-resolution training (imgsz=896 or 1024) for small-object refinement.
• Stronger geometric augmentation (scale=0.75, degrees=10) to teach tighter localisation under perspective.
• A YOLOv11s upgrade (9.4M params) — the variant_comparison.csv slot is reserved for this experiment.

05bError analysis — with example images

The remaining errors in Batch 05 are concentrated in two test-time patterns:

1. Small-pixel-area door_sign localisation drift. Recall climbs to 0.9714 on the loose-IoU bucket but mAP@0.5:0.95 actually drops 3.7 pp vs. Batch 04. Inspection of the qualitative grid (figures/batch05/qualitative_predictions.png) shows correctly classified door signs whose predicted box is ~5–10 pixels off on the long edge. This is a localisation issue, not a recognition issue — visible in the confusion matrix (figures/batch05/confusion_matrix_normalized.png) where the off-diagonal mass is essentially zero.
2. fire_extinguisher tightness loss. A −2.8 pp regression on mAP@0.5:0.95 with identical mAP@0.5. Same signature as door_sign: correct detection, slightly loose box. The v2 dataset has fewer fire_extinguisher captures than v1 (248 vs. 848), so the model sees a narrower range of poses than before.

Representative figures (already embedded in §3.3 and §4.7):

• figures/batch04/qualitative_predictions.png vs. figures/batch05/qualitative_predictions.png — same scenes, tighter boxes in Batch 05 except on door signs.
• figures/batch04/confusion_matrix_normalized.png vs. figures/batch05/confusion_matrix_normalized.png — Batch 05's diagonal is brighter for projector and whiteboard; off-diagonal mass is unchanged.

A follow-up batch should consider higher-resolution training (imgsz=896) for small-object refinement (covered in §5.4).

05cEthics

All ethical constraints from Project 1 continue to apply and were re-checked for the v2 dataset:

• No identifiable faces. All HUB-campus captures used for the v2 corpus were framed on infrastructure (projectors, whiteboards, fire extinguishers, door signs). Any incidental human presence was screened out during the v2 curation pass; no face is recognisable in any retained image.
• No license plates or vehicles. Captures are interior classroom and corridor scenes; no parking areas were photographed.
• No personal data. No names, ID cards, schedules, or contact information are visible in any image.
• Provenance. Roboflow and Kaggle source images were used under their published open licences; HUB-campus captures were taken on-site by the author for the purpose of this project.

06Conclusion

Re-training on the updated custom dataset — same model, same hyperparameters, same seed — moved every meaningful headline metric in the right direction:

• +5.4 pp test mAP@0.5 → 0.9876
• +7.7 pp test mAP@0.5:0.95 → 0.8728
• +11.9 pp macro recall at unchanged precision (1.0) → 0.9804
• Three of four classes now exceed mAP@0.5 = 0.985; the fourth sits at 0.975