last_modified: 2026-01-09

Note: This interactive playground requires JavaScript to function. Click on the graph to set a new starting point for the optimization.

Optimization Playground#

Understanding how different optimization algorithms navigate complex loss landscapes is crucial for deep learning practitioners. While the previous article focused on the theoretical derivation of Adam, this page provides an interactive playground to test various algorithms on different topological structures.

Key Features#

• Multiple Algorithms: Compare standard SGD against adaptive methods like Adam, RMSProp, and Nadam.
• Diverse Landscapes: Test on functions with specific pathological features (e.g., Rosenbrock’s narrow valley, Himmelblau’s multiple minima, Ackley’s many local traps).
• Interactive Start: Click anywhere on the contour plot to set the initial parameter values $\theta_0$. This allows you to inspect how initialization affects convergence.
• Contour Visualization: The background heatmap includes simulated contour lines to help visualize the steepness and shape of the function.

Supported Algorithms#

1. Classical Methods#

• SGD (Stochastic Gradient Descent): The baseline method. It often struggles in ravines and requires careful tuning of the learning rate.
• Momentum: Adds a “velocity” term to damp oscillations and accelerate through shallow regions.

2. Adaptive Learning Rate Methods#

These methods adjust the learning rate for each parameter individually, making them robust for sparse data and unscaled features.

• Adagrad: Accumulates squared gradients. Good for sparse data but learning rate decays too aggressively.
• RMSProp: Uses an exponential moving average of squared gradients. Solves Adagrad’s decay issue.
• Adadelta: An extension of Adadelta that seeks to reduce sensitivity to hyperparameters. Note that the “Learning Rate” slider is largely ignored for this method (it uses a decay rate $\rho$ instead).
• Adam: Combines Momentum and RMSProp. The de facto standard for many deep learning tasks.
• Nadam: Adam with Nesterov momentum for potentially faster convergence.

Objective Functions#

• Quadratic (Bowl): A simple convex function. Ideal for sanity checks.
• Rosenbrock (Banana): Non-convex with a global minimum inside a long, narrow, parabolic valley. Hard for SGD to navigate efficiently.
• Himmelblau: A multi-modal function with four identical global minima. Useful for testing if an algorithm can escape local attractions or simply finds the nearest one.
• Beale: A function with sharp peaks and flat regions, challenging for gradient-based methods to converge precisely.
• Ackley: Characterized by a nearly flat outer region and a large hole at the center, modulated by many small local minima. Optimizers can easily get trapped in the local minima.

Feel free to experiment with different combinations of algorithms, functions, and starting points to build an intuition for their behaviors!