Approximate Equivariance via Projection-based Regularisation

Abstract

Equivariance is a powerful inductive bias in neural networks, improving generalisation and physical consistency. Recently, however, non-equivariant models have regained attention, due to their better runtime performance and imperfect symmetries that might arise in real-world applications. This has motivated the development of approximately equivariant models that strike a middle ground between respecting symmetries and fitting the data distribution. Existing approaches in this field either rely on sampling from a group, incurring a high sample complexity, or explicitly parameterise a model as a sum of an equivariant and non-equivariant network. This work instead approaches approximate equivariance via a projection-based regulariser which leverages a layer-wise orthogonal decomposition of a network’s layers into equivariant and non-equivariant components. In contrast to existing methods, this penalises non-equivariance at an operator level across the full group orbit, rather than point-wise as in sample-based approaches. We present a mathematical framework for computing the non-equivariance penalty exactly and efficiently in both the spatial and spectral domains. In our experiments, our method is competitive with prior approximate-equivariance approaches in task performance, while achieving substantial runtime gains over sample-based regularisers.

@article{berndt2026approximate,
  title = {Approximate equivariance via projection-based regularisation},
  author = {Berndt, Torben and St{\"u}hmer, Jan},
  booktitle = {Forty-third International Conference on Machine Learning},
  year = {2026},
  url={https://arxiv.org/pdf/2601.05028}
}