Aurora: A Leverage-Aware Optimizer for Rectangular Matrices
Aurora introduces leverage-aware optimizer for rectangular matrices
Tilde Research proposes Aurora, an optimizer addressing training inefficiencies in rectangular matrices. It adapts gradient updates based on row/column leverage scores, improving convergence for vision transformers and other models with non-square weight matrices. The method generalizes existing optimizers while reducing computation overhead.