Hierarchical multi-objective optimization for deep learning
A drop-in gradient optimizer that follows your primary objective and bends away from it only as much as needed to keep every secondary constraint progressing — governed by a single tolerance τ ∈ [0,1], with no per-loss weights to tune.
TL;DR
Most training pipelines mix several losses, but one is usually the real deliverable (task accuracy) while the rest only constrain it (sparsity, low-rank, fidelity). Symmetric multi-objective methods average all gradients as equal peers and can stall before the primary has converged. PCD anchors to the primary gradient and admits only the minimal distortion needed to keep every secondary making guaranteed first-order progress — one bounded knob τ, no weight search, and an update direction that vanishes only when every objective is individually stationary.
Video
An eight-minute tour: why symmetric methods stall at conflict equilibria, how PCD's projection works and what τ controls, and the results on pruning, sparsity and low rank.
Narration voice generated with ElevenLabs.
The mechanism, interactively
Every gradient-based multi-objective method takes the primary gradient g1 and the secondary gradient g2 and composes a single update d̃*. Drag the gradient handles to change the geometry, and slide τ to set how hard the secondary pushes. Watch how the symmetric methods treat the two objectives as interchangeable, while PCD stays pinned to the primary and projects onto the shaded feasible region.
τ = 0 only forbids hurting the secondary (the GEM/A-GEM projection, whose fixed points are Pareto-stationary); larger τ forces stronger secondary progress.
Gradient configuration
Drag g₂ in the figure, or use these. When ‖g₂‖ = 1 the symmetric methods (WS, MGDA, CAGrad) collapse onto a single line — only PCD stays distinct.
Methods (click to toggle)
FAMO and AuxiNash are history- / bargaining-based; their update can't be drawn from a single gradient snapshot, so they're shown only in the results below.
An interactive remake of Figure 3 in the paper. MGDA, PCGrad and CAGrad are label-symmetric — swapping g1 and g2 leaves their update unchanged — and weighted sum is too at the equal weights drawn here; its weights come from the user, not from the objectives' roles. PCD is the only one whose primary coefficient is structurally pinned to one.
The idea
Let g̃1, …, g̃K be the scale-normalized gradients of the primary and secondary objectives. PCD takes the direction closest to the primary gradient that still guarantees a τ-fraction of progress on every secondary:
This small convex quadratic program is exactly the Euclidean projection of the primary gradient onto the polyhedron of admissible directions. Its KKT solution has one revealing form:
The primary's coefficient is pinned to one; each secondary enters only through a non-negative multiplier that is zero unless its constraint is active. The "anchor" and the "corrections" are built into the construction — not recovered from it. That is the structural break from symmetric methods, which mix g1 and g2 as peers and cannot tell the deliverable from its constraints.
For K = 2 the update is fully closed-form — add exactly the component along g̃2 needed to meet the constraint at equality, and no more. K = 3 is closed-form via Cramer's rule; larger K enumerates active sets on the K×K Gram matrix. That solve is negligible next to the K backpropagations: under 2% of wall-clock time at K = 2.
Scale invariance. Gradients are normalized by an EMA of their squared norm, so ‖g̃j‖ ≈ 1 in steady state. That makes τ interpretable on [0,1] regardless of loss magnitudes, and the update asymptotically invariant to per-objective rescaling — a property weighted sum provably lacks.
What PCD guarantees
With τ > 0 and a feasible QP, every step satisfies g̃j⊤d̃* ≥ τ‖g̃j‖² — tolerance-controlled first-order descent on each non-stationary secondary (exactly for smooth secondaries; for nonsmooth ones up to a "kink tax" the paper quantifies). Stronger than PCGrad (no per-objective guarantee), MGDA (only an aggregate), and weighted sum (only the scalarized loss).
When the primary direction already gives enough progress on every secondary, the projection is the identity and d̃* = g̃1 exactly. The secondaries never perturb the primary when they don't have to.
EMA normalization makes the update asymptotically invariant (exactly as ε → 0) under per-objective rescaling ℒi → ciℒi, so τ keeps a consistent, scale-free meaning even when raw loss magnitudes differ by orders of magnitude.
For τ > 0, d̃* = 0 if and only if every objective is individually stationary — composite multi-stationarity. It is strictly stronger than Pareto stationarity, so the gradient-cancellation conflict equilibria where symmetric methods halt are never fixed points of the PCD direction. (This requires a feasible QP and is a property of the normalized direction d̃*, not a convergence result for the deployed optimizer.)
Sweeping τ ∈ [0,1] varies the feasible set monotonically and traces a continuous trade-off curve between the Pareto endpoint (τ=0) and the CMS endpoint, on problems where a jointly stationary point exists. No weight ratios, conflict coefficients, or per-loss schedules to search.
Farkas feasibility, the projection / KKT theorem, scale invariance, and the CMS and τ=0 endpoint characterizations — with a clickable theory map.
See the appendix →Results
Across four architectures (DenseNet-121, ResNet-34, Inception, MobileNetV2) on CIFAR-10 and CIFAR-100 — eight configurations — PCD's purple curve sits above every multi-objective baseline at every compression target up to 95%. (The one exception is MobileNetV2 / CIFAR-100 at 99%, where every method has collapsed.) On ResNet-34 / CIFAR-100 the symmetric direction methods (MGDA, PCGrad, CAGrad, FAMO) finish at 1–7% accuracy while PCD holds 70.9% at 90%. The cause here is objective scale, not gradient conflict: the raw regularizer gradients are one to two orders of magnitude larger than the task gradient, while the cosine between them never exceeds 1.2×10−3. PCD's per-objective normalization absorbs that mismatch.
Against tuned, pruning-specific and constrained baselines at matched sparsity, PCD is better in 28 of 47 comparisons in the compression regime (≥ 85% sparsity), worse in 1, and within seed noise in the rest.
Sweeping a single τ trades the unpruned network's 81.5 MB, 2.33 G FLOPs and 24.5 ms CPU latency against accuracy on ResNet-34 / CIFAR-100. The curve is smooth and nearly monotone — most of the benefit arrives within τ ∈ (0, 0.1] — so a practitioner dials to a resource budget instead of searching weight ratios.
Every baseline is physically pruned at each compression target and its latency, file size and FLOPs measured directly. On ResNet-34 / CIFAR-100, on every one of those axes and at every target, the baseline networks sit above PCD's τ-swept curve: at equal cost, PCD's network is more accurate. The same single knob that controls compression traces the whole accuracy–efficiency front.
Where it could apply
The paper evaluates pruning and compression (structured sparsity, ℓ1 sparsity, low rank). The other three settings share the same structure but are untested.
Pair the task loss with a group-lasso, ℓ1, or nuclear-norm penalty; τ becomes a single compression dial that traces a smooth accuracy–efficiency curve.
Keep the task loss primary and a fidelity term penalizing drift from quantized parameters as the secondary, anchoring the model against discrete-value drift without hand-tuned weights.
Hold local fitness as the deliverable against communication-budget or personalization terms, with τ modulating how hard those constraints push.
For generative models, keep generative fidelity primary while structural regularizers act as constraints rather than competing peers.
Beyond two objectives — a K = 3 hierarchy (task loss + ℓ1 sparsity + nuclear-norm low-rank):
Scope
Citation
@article{varam2026pcd,
title = {Not All Objectives Are Born Equal: Priority-Constrained
Descent for Hierarchical Multi-Objective Optimization},
author = {Varam, Dara and AlHajri, Mohamed I.},
journal = {Transactions on Machine Learning Research},
issn = {2835-8856},
year = {2026},
url = {https://openreview.net/forum?id=HT01yGHLEt},
eprint = {2606.29521},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}