Abstract
We are addressing two fundamental problems in authorship verification (AV): Topic variability and miscalibration. Variations in the topic of two disputed texts are a major cause of error for most AV systems. In addition, it is observed that the underlying probability estimates produced by deep learning AV mechanisms oftentimes do not match the actual case counts in the respective training data. As such, probability estimates are poorly calibrated. We are expanding our framework from PAN 2020 to include Bayes factor scoring (BFS) and an uncertainty adaptation layer (UAL) to address both problems. Experiments with the 2020/21 PAN AV shared task data show that the proposed method significantly reduces sensitivities to topical variations and significantly improves the system’s calibration.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The source code is accessible online: https://github.com/boenninghoff/pan_2020_2021_authorship_verification.
- 2.
SA = same author, DA = different authors, SF = same fandom, DF = different fandoms.
- 3.
\(\alpha ,\gamma \) are learned, first four rows for tanh activation, last two rows for Swish activation.
References
Bevendorff, J., Hagen, M., Stein, B., Potthast, M.: Bias analysis and mitigation in the evaluation of authorship verification. In: 57th Annual Meeting of the ACL, pp. 6301–6306 (2019)
Boenninghoff, B., Hessler, S., Kolossa, D., Nickel, R.M.: Explainable authorship verification in social media via attention-based similarity learning. In: IEEE International Conference on Big Data, pp. 36–45 (2019)
Boenninghoff, B., Rupp, J., Nickel, R., Kolossa, D.: Deep bayes factor scoring for authorship verification. In: PAN@CLEF 2020, Notebook Papers (2020)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. ACL 5, 135–146 (2017)
Cumani, S., Brümmer, N., Burget, L., Laface, P., Plchot, O., Vasilakakis, V.: Pairwise discriminative speaker verification in the I-vector space. IEEE Trans. Audio, Speech Lang. Process. 21, 1217–1227 (2013)
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: 34th ICML, vol. 70, pp. 1321–1330. PMLR (2017)
Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? In: NeurIPS, vol. 30. Curran Associates, Inc. (2017)
Kestemont, M., et al.: Overview of the cross-domain authorship verification task at PAN 2020. In: CLEF 2020, Notebook Papers (2020)
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: NeurIPS, pp. 6405–6416 (2017)
Luo, B., et al.: Learning with noise: enhance distantly supervised relation extraction with dynamic transition matrix. In: 55th Annual Meeting of the ACL, pp. 430–439 (2017)
Pampari, A., Ermon, S.: Unsupervised calibration under covariate shift (2020)
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., Hinton, G.: Regularizing neural networks by penalizing confident output distributions (2017)
Ramachandran, P., Zoph, B., Le, Q.V.: Searching for activation functions (2017)
Stamatatos, E.: Authorship attribution using text distortion. In: Proceedings of EACL (2017)
Acknowledgment
This work was in significant parts performed on a HPC cluster at Bucknell University through the support of the National Science Foundation, Grant Number 1659397. Project funding was provided by the state of North Rhine-Westphalia within the Research Training Group “SecHuman - Security for Humans in Cyberspace” and by the Deutsche Forschungsgemeinschaft (DFG) under Germany’s Excellence Strategy - EXC2092CaSa- 390781972.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Boenninghoff, B., Kolossa, D., Nickel, R.M. (2021). Self-calibrating Neural-Probabilistic Model for Authorship Verification Under Covariate Shift. In: Candan, K.S., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2021. Lecture Notes in Computer Science(), vol 12880. Springer, Cham. https://doi.org/10.1007/978-3-030-85251-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-85251-1_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85250-4
Online ISBN: 978-3-030-85251-1
eBook Packages: Computer ScienceComputer Science (R0)