Reizinger, Patrik and Gyires-Tóth, Bálint (2019) Stochastic Weight Matrix-based Regularization Methods for Deep Neural Networks. In: Fifth International Conference on Machine Learning, Optimization, and Data Science, September 10-13, 2019, Siena, Italy. (In Press)
|
Text
lod_wmm_camera_ready_Reizinger_Gyires-Tóth.pdf - Accepted Version Download (856kB) | Preview |
Abstract
The aim of this paper is to introduce two widely applicable regularization methods based on the direct modification of weight matrices. The first method, Weight Reinitialization, utilizes a simplified Bayesian assumption with partially resetting a sparse subset of the parameters. The second one, Weight Shuffling, introduces an entropy- and weight distribution-invariant non-white noise to the parameters. The latter can also be interpreted as an ensemble approach. The proposed methods are evaluated on benchmark datasets, such as MNIST, CIFAR-10 or the JSB Chorales database, and also on time series modeling masks. We report gains both regarding performance and entropy of the analyzed networks. We also made our code available as a GitHub repository.
Item Type: | Conference or Workshop Item (Paper) |
---|---|
Subjects: | T Technology / alkalmazott, műszaki tudományok > T2 Technology (General) / műszaki tudományok általában |
Depositing User: | Dr. Gyires-Tóth Bálint Pál |
Date Deposited: | 26 Sep 2019 06:37 |
Last Modified: | 03 Apr 2023 06:36 |
URI: | http://real.mtak.hu/id/eprint/101545 |
Actions (login required)
Edit Item |