Skip to content

Training a neural network without gradients: 7 metaheuristics vs. Adam

Status
Completed
Date
2026
Venue
MS AI coursework, Bahria University
Tags
Optimization · Metaheuristics · Neural networks

  • 93.33%

    PSO accuracy

    +3.33 points over Adam

  • 0.0393

    Adam loss

    lowest, in 200 epochs

  • 0.0745

    PSO loss

    in 0.312 s

  • 44

    trainable parameters

Abstract

Gradient descent is the default way to train a network, but not the only one. This experiment trains the 44 weights and biases of a small PyTorch network (4-5-3-3 neurons) with seven derivative-free metaheuristics, comparing each with Adam on accuracy, loss, convergence and run time over 1,000 evaluation cycles.

01Setup

  • Network: a 4-layer feedforward PyTorch model (4-5-3-3) with ReLU and cross-entropy loss, 44 trainable parameters.
  • Algorithms: Genetic Algorithm, Particle Swarm, Ant Colony, Bat, Artificial Bee Colony, Monkey and Firefly, against Adam.
  • PSO written from scratch, with personal-best and global-best updates balancing exploration and exploitation.

02What it says

Adam still finds the lowest loss, but PSO classifies better on this problem and converges quickly. That trade-off between loss and accuracy is exactly the kind of model-internals question I want to keep pulling on.