Stochastic Weight Matrix-based Regularization Methods for Deep Neural Networks

Reizinger, Patrik and Gyires-Tóth, Bálint (2019) Stochastic Weight Matrix-based Regularization Methods for Deep Neural Networks. In: Fifth International Conference on Machine Learning, Optimization, and Data Science, September 10-13, 2019, Siena, Italy. (In Press)

Preview

Text lod_wmm_camera_ready_Reizinger_Gyires-Tóth.pdf - Accepted Version Download (856kB) | Preview

Abstract

The aim of this paper is to introduce two widely applicable regularization methods based on the direct modification of weight matrices. The first method, Weight Reinitialization, utilizes a simplified Bayesian assumption with partially resetting a sparse subset of the parameters. The second one, Weight Shuffling, introduces an entropy- and weight distribution-invariant non-white noise to the parameters. The latter can also be interpreted as an ensemble approach. The proposed methods are evaluated on benchmark datasets, such as MNIST, CIFAR-10 or the JSB Chorales database, and also on time series modeling masks. We report gains both regarding performance and entropy of the analyzed networks. We also made our code available as a GitHub repository.

Actions (login required)

Edit Item