AutoDL Image Competition Tuning Log

I’ve been spending this period on AutoDL’s image-classification track (Auto_Image), with all offline test data drawn from the Pedro dataset. The overall idea is to explore, within a limited time budget, how different backbones, normalization layers, activation functions, precision formats, and pretrained weights affect the final score. What follows is a systematic write-up in the order the experiments were run.

Backbone Choice: ResNet18 vs. ResNet34

The first question is the choice of backbone. Looking at how the online scores trend, the score rises quickly early on, and ResNet18 is a reasonable fit for this phase; later the rise flattens out and the increments get small, so in theory it makes sense to try a larger backbone or switch to a different pipeline.

But the offline test results show that ResNet34 has no clear advantage over ResNet18. In the figure below, ResNet18 is on the left and ResNet34 is on the right:

Learning-curve comparison of ResNet18 and ResNet34 on the Pedro task
Figure 1: Learning curves on the Pedro task — left ResNet18 (ALC=0.6163), right ResNet34 (ALC=0.6169); the gap is insignificant

The gap is not significant, so for now we keep ResNet18 as the baseline.

Activation Function: Replacing ReLU with CELU

I tried replacing all the ReLUs in the network with CELU. On the offline data the two are roughly comparable, with CELU slightly worse; on the online data the gap widens noticeably, and the CELU variant actually scores considerably lower.

Offline CELU result:

Offline learning curve of CELU replacing ReLU
Figure 2: Offline learning curve of CELU replacing ReLU (Pedro, ALC=0.5834)

Online comparison (the new baseline on the left, CELU on the right):

Online learning curve of the new baseline on the Apollon task
Figure 3: Online learning curve of the new baseline (Apollon, ALC=0.7059)
Online learning curve of the CELU variant on the Apollon task
Figure 4: Online learning curve of the CELU variant (Apollon, ALC=0.6438), clearly below the baseline

CELU brings no benefit in this setting, so it’s dropped.

Normalization-Layer Experiments

Resetting All BN Layers

I tried resetting all the Batch Normalization layers of the pretrained model, hoping to let the statistics adapt to the new data faster. The offline results show that after resetting BN, the early stage (the first checkpoint) does perform better and converges faster, but the final accuracy isn’t high. The reset-BN result is on the right:

Offline learning curves with and without resetting BN
Figure 5: Offline learning curves with and without resetting BN — left no reset (ALC=0.6086), right BN reset (ALC=0.5972); resetting converges faster early but reaches slightly lower final accuracy

The online result is also slightly below the no-reset ResNet18 baseline:

Online learning curve with BN reset
Figure 6: Online learning curve with BN reset (Apollon, ALC=0.6936), slightly below the non-reset ResNet18 baseline

ResNet34 + Resetting All BN

Combining the two, hoping to get both a stronger backbone and faster early convergence. The offline first checkpoint slowed down a lot, which I suspect was caused by unstable system IO, so I don’t treat it as a reference. The online result is that the early stage was indeed much faster, but the later accuracy still isn’t high enough—judging by appearances, this is caused by overfitting.

Online test result:

Online learning curve of ResNet34 + BN reset
Figure 7: Online learning curve of ResNet34 + BN reset (Apollon, ALC=0.6844), fast early but lacking final accuracy, likely overfitting

Precise Norm (Group Norm) Replacing Batch Norm

Another direction is to fully replace BN with Group Norm (Precise Norm):

Error curves illustrating Precise BatchNorm and Group Norm under large-batch training
Figure 8: Motivation for Precise BatchNorm — left, regular BN (train/val error curves under large batch); right, Group Norm (IJCV version) validation error curves across different batch sizes

The final result is poor, so it’s not worth considering.

Mixed-Precision Training (FP16)

I tested FP16 locally. The result was unexpected: the version without FP16 (left) is actually faster than the one with FP16 enabled (right), and FP16 also has a fairly large impact on accuracy.

Local learning curve without FP16
Figure 9: Local learning curve without FP16 (Pedro, ALC=0.5570)
Local learning curve with FP16 enabled
Figure 10: Local learning curve with FP16 enabled (Pedro, ALC=0.5202), actually slower and with degraded accuracy

FP16 isn’t worth it under the current configuration, so I’m not introducing it for now.

Miscellaneous Other Experiments

Scan loop unrolling (unroll): I tested the time cost of unrolling the loop in scan (unroll=2). It turned out to be of no use—the original test scan time is about 43s, so the change makes little difference.

Increasing batch size: Moderately increasing the bs adds some time overhead but does little for stability—the root cause of online instability is still overfitting, so tuning bs treats the symptom rather than the disease.

Learning-rate adjustment: I’m still trying the idea of using different pipelines for the early and later stages, and haven’t reached a firm conclusion yet.

Pretrained Models: ReID and Pedestrian-Attribute Datasets

I tried replacing ImageNet pretraining with person re-identification (ReID) pretrained weights, testing versions pretrained on the MSMT dataset and the Market dataset respectively.

Offline result for MSMT-pretrained:

Offline learning curve of MSMT ReID pretraining
Figure 11: Offline learning curve of MSMT-dataset ReID pretraining (Pedro, ALC=0.3906), clearly worse than ImageNet pretraining

I also tested a PA-100K pretrained model (pedestrian-attribute recognition), which worked poorly both online and offline; I tried pretraining from several datasets and none were satisfactory.

Applicability of ImageNet Pretraining Across Scenarios

I tested the effect of ImageNet pretraining on three categories of scenarios: pedestrian, medical, and satellite imagery.

Satellite imagery:

Learning curve of ImageNet pretraining in a satellite-image scenario
Figure 12: Learning curve of ImageNet pretraining in the satellite-image scenario (Decal), ALC=0.6412

Medical imagery:

Learning curve of ImageNet pretraining in a medical-image scenario
Figure 13: Learning curve of ImageNet pretraining in the medical-image scenario (Hammer), ALC=0.6977

The differences across domains are pronounced; ImageNet pretraining cannot maintain a consistent advantage across all scenarios.

Predicting the CV Final Datasets

Based on what has shown up in the current feedback phase, here is a prediction of the datasets that might appear in the final phase:

  • Loukoum (handwriting): appeared in both the cv and cv2 finals; AutoCLINT does well on it, so no worries.

    Learning curve of AutoCLINT on the Loukoum handwriting task
    Figure 14: Learning curve of AutoCLINT on the Loukoum (handwriting) task, ALC=0.9256, doing very well

  • Tim (general objects): ImageNet pretraining is already sufficient.

  • Apollon (pedestrian): already appeared in the feedback phase.

  • Ideal (aerial/satellite): already appeared in the feedback phase.

  • Ray (medical): only appeared in the cv final, worth keeping an eye on.

Online-Data Reshuffle Experiment

Testing the impact of reshuffling the online data or not:

Learning curve with online data not reshuffled
Figure 15: Learning curve with online data not reshuffled (Apollon, ALC=0.6888)
Learning curve with online data reshuffled
Figure 16: Learning curve with online data reshuffled (Apollon, ALC=0.6577)

Finally, I tried using self-attention to handle the data-distribution problem, but it didn’t work.

  • Hyperparameter Tuning
  • Image Classification
  • Experiment Log