Journal of High Energy Physics

, 2017:174 | Cite as

Classification without labels: learning from mixed samples in high energy physics

  • Eric M. MetodievEmail author
  • Benjamin Nachman
  • Jesse Thaler
Open Access
Regular Article - Theoretical Physics


Modern machine learning techniques can be used to construct powerful models for difficult collider physics problems. In many applications, however, these models are trained on imperfect simulations due to a lack of truth-level information in the data, which risks the model learning artifacts of the simulation. In this paper, we introduce the paradigm of classification without labels (CWoLa) in which a classifier is trained to distinguish statistical mixtures of classes, which are common in collider physics. Crucially, neither individual labels nor class proportions are required, yet we prove that the optimal classifier in the CWoLa paradigm is also the optimal classifier in the traditional fully-supervised case where all label information is available. After demonstrating the power of this method in an analytical toy example, we consider a realistic benchmark for collider physics: distinguishing quark- versus gluon-initiated jets using mixed quark/gluon training samples. More generally, CWoLa can be applied to any classification problem where labels or class proportions are unknown or simulations are unreliable, but statistical mixtures of the classes are available.




Open Access

This article is distributed under the terms of the Creative Commons Attribution License (CC-BY 4.0), which permits any use, distribution and reproduction in any medium, provided the original author(s) and source are credited.


  1. [1]
    J. Cogan, M. Kagan, E. Strauss and A. Schwarztman, Jet-Images: Computer Vision Inspired Techniques for Jet Tagging, JHEP 02 (2015) 118 [arXiv:1407.5675] [INSPIRE].ADSCrossRefGoogle Scholar
  2. [2]
    L.G. Almeida, M. Backović, M. Cliche, S.J. Lee and M. Perelstein, Playing Tag with ANN: Boosted Top Identification with Pattern Recognition, JHEP 07 (2015) 086 [arXiv:1501.05968] [INSPIRE].ADSCrossRefGoogle Scholar
  3. [3]
    L. de Oliveira, M. Kagan, L. Mackey, B. Nachman and A. Schwartzman, Jet-images — deep learning edition, JHEP 07 (2016) 069 [arXiv:1511.05190] [INSPIRE].CrossRefGoogle Scholar
  4. [4]
    P. Baldi, K. Bauer, C. Eng, P. Sadowski and D. Whiteson, Jet Substructure Classification in High-Energy Physics with Deep Neural Networks, Phys. Rev. D 93 (2016) 094034 [arXiv:1603.09349] [INSPIRE].ADSGoogle Scholar
  5. [5]
    J. Barnard, E.N. Dawe, M.J. Dolan and N. Rajcic, Parton Shower Uncertainties in Jet Substructure Analyses with Deep Neural Networks, Phys. Rev. D 95 (2017) 014018 [arXiv:1609.00607] [INSPIRE].ADSGoogle Scholar
  6. [6]
    G. Kasieczka, T. Plehn, M. Russell and T. Schell, Deep-learning Top Taggers or The End of QCD?, JHEP 05 (2017) 006 [arXiv:1701.08784] [INSPIRE].ADSCrossRefGoogle Scholar
  7. [7]
    P.T. Komiske, E.M. Metodiev and M.D. Schwartz, Deep learning in color: towards automated quark/gluon jet discrimination, JHEP 01 (2017) 110 [arXiv:1612.01551] [INSPIRE].ADSCrossRefzbMATHGoogle Scholar
  8. [8]
    L. de Oliveira, M. Paganini and B. Nachman, Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis, arXiv:1701.05927 [INSPIRE].
  9. [9]
    P.T. Komiske, E.M. Metodiev, B. Nachman and M.D. Schwartz, Pileup Mitigation with Machine Learning (PUMML), arXiv:1707.08600 [INSPIRE].
  10. [10]
    G. Louppe, K. Cho, C. Becot and K. Cranmer, QCD-Aware Recursive Neural Networks for Jet Physics, arXiv:1702.00748 [INSPIRE].
  11. [11]
    ATLAS collaboration, Performance of Top Quark and W Boson Tagging in Run 2 with ATLAS, ATLAS-CONF-2017-064 (2017).
  12. [12]
    ATLAS collaboration, Optimisation and performance studies of the ATLAS b-tagging algorithms for the 2017-18 LHC run, ATL-PHYS-PUB-2017-013 (2017).
  13. [13]
    ATLAS collaboration, Identification of Hadronically-Decaying W Bosons and Top Quarks Using High-Level Features as Input to Boosted Decision Trees and Deep Neural Networks in ATLAS at \( \sqrt{s}= 13 \) TeV, ATL-PHYS-PUB-2017-004 (2017).
  14. [14]
    CMS collaboration, Heavy flavor identification at CMS with deep neural networks, CMS-DP-2017-005 (2017).
  15. [15]
    CMS collaboration, CMS Phase 1 heavy flavour identification performance and developments, CMS-DP-2017-013 (2017).
  16. [16]
    ATLAS collaboration, Identification of Jets Containing b-Hadrons with Recurrent Neural Networks at the ATLAS Experiment, ATL-PHYS-PUB-2017-003 (2017).
  17. [17]
    A. Butter, G. Kasieczka, T. Plehn and M. Russell, Deep-learned Top Tagging using Lorentz Invariance and Nothing Else, arXiv:1707.08966 [INSPIRE].
  18. [18]
    J. Pearkes, W. Fedorko, A. Lister and C. Gay, Jet Constituents for Deep Neural Network Based Top Quark Tagging, arXiv:1704.02124 [INSPIRE].
  19. [19]
    K. Datta and A. Larkoski, How Much Information is in a Jet?, JHEP 06 (2017) 073 [arXiv:1704.08249] [INSPIRE].ADSCrossRefGoogle Scholar
  20. [20]
    ATLAS collaboration, Quark versus Gluon Jet Tagging Using Jet Images with the ATLAS Detector, ATL-PHYS-PUB-2017-017 (2017).
  21. [21]
    CMS collaboration, New Developments for Jet Substructure Reconstruction in CMS, CMS-DP-2017-027 (2017).
  22. [22]
    ATLAS collaboration, Performance of b-Jet Identification in the ATLAS Experiment, 2016 JINST 11 P04008 [arXiv:1512.01094] [INSPIRE].
  23. [23]
    CMS collaboration, Identification of b-quark jets with the CMS experiment, 2013 JINST 8 P04013 [arXiv:1211.4462] [INSPIRE].
  24. [24]
    ATLAS collaboration, Light-quark and gluon jet discrimination in pp collisions at \( \sqrt{s}=7 \) TeV with the ATLAS detector, Eur. Phys. J. C 74(2014) 3023 [arXiv:1405.6583] [INSPIRE].
  25. [25]
    CMS collaboration, Performance of quark/gluon discrimination in 8 TeV pp data, CMS-PAS-JME-13-002 (2013).
  26. [26]
    CMS collaboration, Performance of quark/gluon discrimination in 13 TeV data, CMS-DP-2016-070 (2016).
  27. [27]
    ATLAS collaboration, Identification of boosted, hadronically decaying W bosons and comparisons with ATLAS data taken at s = 8 TeV, Eur. Phys. J. C 76 (2016) 154 [arXiv:1510.05821] [INSPIRE].
  28. [28]
    CMS collaboration, Identification techniques for highly boosted W bosons that decay into hadrons, JHEP 12 (2014) 017 [arXiv:1410.4227] [INSPIRE].
  29. [29]
    ATLAS collaboration, Identification of high transverse momentum top quarks in pp collisions at \( \sqrt{s}= 8 \) TeV with the ATLAS detector, JHEP 06 (2016) 093 [arXiv:1603.03127] [INSPIRE].
  30. [30]
    CMS collaboration, Boosted Top Jet Tagging at CMS, CMS-PAS-JME-13-007 (2014).
  31. [31]
    G. Louppe, M. Kagan and K. Cranmer, Learning to Pivot with Adversarial Networks, arXiv:1611.01046 [INSPIRE].
  32. [32]
    G. Blanchard, M. Flaska, G. Handy, S. Pozzi, and C. Scott, Classification with asymmetric label noise: consistency and maximal denoising, Electron. J. Stat. 10 (2016) 2780.CrossRefzbMATHMathSciNetGoogle Scholar
  33. [33]
    J. Hernández-González, I. Inza, and J.A. Lozano, Weak supervision and other non-standard classification problems: a taxonomy, Pattern Recogn. Lett. 69 (2016) 49.CrossRefGoogle Scholar
  34. [34]
    L.M. Dery, B. Nachman, F. Rubbo and A. Schwartzman, Weakly Supervised Classification in High Energy Physics, JHEP 05 (2017) 145 [arXiv:1702.00414] [INSPIRE].ADSCrossRefGoogle Scholar
  35. [35]
    N. Quadrianto, A.J. Smola, T.S. Caetano and Q.V. Le, Estimating labels from label proportions, J. Mach. Learn. Res. 10 (2009) 2349.zbMATHMathSciNetGoogle Scholar
  36. [36]
    G. Patrini, R. Nock, P. Rivera and T. Caetano, (Almost) no label no cry, in Advances in Neural Information Processing Systems, (2014), pp. 190-198.Google Scholar
  37. [37]
    P. Gras, S. Höche, D. Kar, A. Larkoski, L. Lönnblad, S. Plätzer et al., Systematics of quark/gluon tagging, JHEP 07 (2017) 091 [arXiv:1704.03878] [INSPIRE].ADSCrossRefGoogle Scholar
  38. [38]
    T. Cohen, M. Freytsis and B. Ostdiek, (Machine) Learning to Do More with Less, arXiv:1706.09451 [INSPIRE].
  39. [39]
    J. Neyman and E.S. Pearson, On the problem of the most efficient tests of statistical hypotheses, in Breakthroughs in statistics, Springer (1992), pp. 73-108.Google Scholar
  40. [40]
    N. Natarajan, I.S. Dhillon, P.K. Ravikumar and A. Tewari, Learning with noisy labels, in Advances in neural information processing systems 26 (NIPS 2013), (2013), pp. 1196-1204.Google Scholar
  41. [41]
    C. Scott, A rate of convergence for mixture proportion estimation, with application to learning from noisy labels, in Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, (2015), pp. 838-846.Google Scholar
  42. [42]
    K. Cranmer, J. Pavez and G. Louppe, Approximating Likelihood Ratios with Calibrated Discriminative Classifiers, arXiv:1506.02169 [INSPIRE].
  43. [43]
    H.P. Nilles and K.H. Streng, quark-gluon Separation in Three Jet Events, Phys. Rev. D 23 (1981) 1944 [INSPIRE].
  44. [44]
    L.M. Jones, Tests for Determining the Parton Ancestor of a Hadron Jet, Phys. Rev. D 39 (1989) 2550 [INSPIRE].ADSGoogle Scholar
  45. [45]
    Z. Fodor, How to See the Differences Between Quark and Gluon Jets, Phys. Rev. D 41 (1990) 1726 [INSPIRE].ADSGoogle Scholar
  46. [46]
    L. Jones, Towards a systematic jet classification, Phys. Rev. D 42 (1990) 811 [INSPIRE].ADSGoogle Scholar
  47. [47]
    L. Lönnblad, C. Peterson and T. Rognvaldsson, Using neural networks to identify jets, Nucl. Phys. B 349 (1991) 675 [INSPIRE].ADSCrossRefGoogle Scholar
  48. [48]
    J. Pumplin, How to tell quark jets from gluon jets, Phys. Rev. D 44 (1991) 2025 [INSPIRE].ADSGoogle Scholar
  49. [49]
    J. Gallicchio and M.D. Schwartz, Quark and Gluon Tagging at the LHC, Phys. Rev. Lett. 107 (2011) 172001 [arXiv:1106.3076] [INSPIRE].ADSCrossRefGoogle Scholar
  50. [50]
    J. Gallicchio and M.D. Schwartz, Quark and Gluon Jet Substructure, JHEP 04 (2013) 090 [arXiv:1211.7038] [INSPIRE].ADSCrossRefGoogle Scholar
  51. [51]
    A.J. Larkoski, J. Thaler and W.J. Waalewijn, Gaining (Mutual) Information about Quark/Gluon Discrimination, JHEP 11 (2014) 129 [arXiv:1408.3122] [INSPIRE].ADSCrossRefGoogle Scholar
  52. [52]
    ATLAS collaboration, Measurement of the charged-particle multiplicity inside jets from \( \sqrt{s}=8 \) TeV pp collisions with the ATLAS detector, Eur. Phys. J. C 76(2016) 322 [arXiv:1602.00988] [INSPIRE].
  53. [53]
    J. Gallicchio and M.D. Schwartz, Pure Samples of Quark and Gluon Jets at the LHC, JHEP 10 (2011) 103 [arXiv:1104.1175] [INSPIRE].ADSCrossRefGoogle Scholar
  54. [54]
    J.M. Butterworth, A.R. Davison, M. Rubin and G.P. Salam, Jet substructure as a new Higgs search channel at the LHC, Phys. Rev. Lett. 100 (2008) 242001 [arXiv:0802.2470] [INSPIRE].ADSCrossRefGoogle Scholar
  55. [55]
    S.D. Ellis, C.K. Vermilion and J.R. Walsh, Techniques for improved heavy particle searches with jet substructure, Phys. Rev. D 80 (2009) 051501 [arXiv:0903.5081] [INSPIRE].ADSGoogle Scholar
  56. [56]
    S.D. Ellis, C.K. Vermilion and J.R. Walsh, Recombination Algorithms and Jet Substructure: Pruning as a Tool for Heavy Particle Searches, Phys. Rev. D 81 (2010) 094023 [arXiv:0912.0033] [INSPIRE].ADSGoogle Scholar
  57. [57]
    D. Krohn, J. Thaler and L.-T. Wang, Jet Trimming, JHEP 02 (2010) 084 [arXiv:0912.1342] [INSPIRE].ADSCrossRefGoogle Scholar
  58. [58]
    M. Dasgupta, A. Fregoso, S. Marzani and G.P. Salam, Towards an understanding of jet substructure, JHEP 09 (2013) 029 [arXiv:1307.0007] [INSPIRE].ADSCrossRefGoogle Scholar
  59. [59]
    A.J. Larkoski, S. Marzani, G. Soyez and J. Thaler, Soft Drop, JHEP 05 (2014) 146 [arXiv:1402.2657] [INSPIRE].ADSCrossRefGoogle Scholar
  60. [60]
    C. Frye, A.J. Larkoski, M.D. Schwartz and K. Yan, Precision physics with pile-up insensitive observables, arXiv:1603.06375 [INSPIRE].
  61. [61]
    C. Frye, A.J. Larkoski, M.D. Schwartz and K. Yan, Factorization for groomed jet substructure beyond the next-to-leading logarithm, JHEP 07 (2016) 064 [arXiv:1603.09338] [INSPIRE].ADSCrossRefGoogle Scholar
  62. [62]
    C.F. Berger, T. Kucs and G.F. Sterman, Event shape/energy flow correlations, Phys. Rev. D 68 (2003) 014012 [hep-ph/0303051] [INSPIRE].
  63. [63]
    L.G. Almeida, S.J. Lee, G. Perez, G.F. Sterman, I. Sung and J. Virzi, Substructure of high-p T Jets at the LHC, Phys. Rev. D 79 (2009) 074017 [arXiv:0807.0234] [INSPIRE].
  64. [64]
    S.D. Ellis, C.K. Vermilion, J.R. Walsh, A. Hornig and C. Lee, Jet Shapes and Jet Algorithms in SCET, JHEP 11 (2010) 101 [arXiv:1001.0014] [INSPIRE].ADSCrossRefGoogle Scholar
  65. [65]
    D. Bertolini, T. Chan and J. Thaler, Jet Observables Without Jet Algorithms, JHEP 04 (2014) 013 [arXiv:1310.7584] [INSPIRE].ADSCrossRefGoogle Scholar
  66. [66]
    A.J. Larkoski, D. Neill and J. Thaler, Jet Shapes with the Broadening Axis, JHEP 04 (2014) 017 [arXiv:1401.2158] [INSPIRE].ADSCrossRefGoogle Scholar
  67. [67]
    G. Salam, E t Scheme, unpublished.Google Scholar
  68. [68]
    CMS collaboration, Search for a Higgs boson in the decay channel \( H\to Z{Z^{\Big(}}^{\ast \Big)}\to q\overline{q}{\ell}^{-}{\ell}^{+} \) in pp collisions at \( \sqrt{s}=7 \) TeV, JHEP 04 (2012) 036 [arXiv:1202.1416] [INSPIRE].
  69. [69]
    J.R. Andersen et al., Les Houches 2015: Physics at TeV Colliders Standard Model Working Group Report, in 9th Les Houches Workshop on Physics at TeV Colliders (PhysTeV 2015), Les Houches, France, 1-19 June 2015, (2016) [arXiv:1605.04692] [INSPIRE].
  70. [70]
    T. Sjöstrand, S. Mrenna and P.Z. Skands, A Brief Introduction to PYTHIA 8.1, Comput. Phys. Commun. 178 (2008) 852 [arXiv:0710.3820] [INSPIRE].
  71. [71]
    M. Cacciari, G.P. Salam and G. Soyez, The Anti-k t jet clustering algorithm, JHEP 04 (2008) 063 [arXiv:0802.1189] [INSPIRE].ADSCrossRefzbMATHGoogle Scholar
  72. [72]
    M. Cacciari, G.P. Salam and G. Soyez, FastJet User Manual, Eur. Phys. J. C 72 (2012) 1896 [arXiv:1111.6097] [INSPIRE].ADSCrossRefGoogle Scholar
  73. [73]
    F. Chollet, Keras, (2017)
  74. [74]
    M. Abadi et al., Tensorflow: a system for large-scale machine learning, proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16), volume 16, (2016), pp. 265-283.Google Scholar
  75. [75]
    K. He, X. Zhang, S. Ren, and J. Sun, Delving deep into rectifiers: surpassing human-level performance on imagenet classification, in proceedings of the IEEE international conference on computer vision, (2015), pp. 1026-1034.Google Scholar
  76. [76]
    D. Kingma and J. Ba, Adam: a method for stochastic optimization, arXiv:1412.6980.

Copyright information

© The Author(s) 2017

Authors and Affiliations

  • Eric M. Metodiev
    • 1
    Email author
  • Benjamin Nachman
    • 2
  • Jesse Thaler
    • 1
  1. 1.Center for Theoretical PhysicsMassachusetts Institute of TechnologyCambridgeU.S.A.
  2. 2.Physics Division, Lawrence Berkeley National LaboratoryBerkeleyU.S.A.

Personalised recommendations