Mode combinability: Exploring convex combinations of permutation aligned models

Csiszárik, Adrián and F. Kiss, Melinda and Kőrösi-Szabó, Péter and Muntag, Márton and Papp, Gergely and Varga, Dániel (2024) Mode combinability: Exploring convex combinations of permutation aligned models. NEURAL NETWORKS, 173. ISSN 0893-6080

Preview

Text 2308.11511v1.pdf - Accepted Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (1MB) | Preview

Abstract

We explore element-wise convex combinations of two permutation-aligned neural network parameter vectors ΘA and ΘB of size d. We conduct extensive experiments by examining various distributions of such model combinations parametrized by elements of the hypercube [0,1]d and its vicinity. Our findings reveal that broad regions of the hypercube form surfaces of low loss values, indicating that the notion of linear mode connectivity extends to a more general phenomenon which we call mode combinability. We also make several novel observations regarding linear mode connectivity and model re-basin. We demonstrate a transitivity property: two models re-based to a common third model are also linear mode connected, and a robustness property: even with significant perturbations of the neuron matchings the resulting combinations continue to form a working model. Moreover, we analyze the functional and weight similarity of model combinations and show that such combinations are non-vacuous in the sense that there are significant functional differences between the resulting models.

Actions (login required)

Edit Item