City Research Online ***TEST***

Abstract

In this work, we introduce a novel approach to regularization in multivariable regression problems. Our regularizer, called DLoss, penalizes differences between the model’s derivatives and derivatives of the data generating function as estimated from the training data. We call these estimated derivatives data derivatives. The goal of our method is to align the model to the data, not only in terms of target values but also in terms of the derivatives involved. To estimate data derivatives, we select (from the training data) 2-tuples of input-value pairs, using either nearest neighbor or random selection. We evaluate the effectiveness of DLoss on synthetic and real datasets with different weights, to the standard mean squared error loss. The experimental results show that with DLoss (using nearest neighbor selection) we obtain, on average, the best rank with respect to MSE on validation data sets, compared to no regularization, L2 regularization, and Dropout. Our implementation code is available on Github.

Publication Type:	Conference or Workshop Item (Paper)
Additional Information:	For the purpose of open access, the author(s) has applied a Creative Commons Attribution (CC BY) license to any Accepted Manuscript version arising
Publisher Keywords:	Machine Learning, Neural Networks, Regularization, Regression, DLoss, Data Derivative
Subjects:	H Social Sciences > HF Commerce Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	School of Science & Technology School of Science & Technology > Department of Computer Science
SWORD Depositor:	Symplectic Administrator

Export

Downloads