Skip to main content
SpringerLink
Log in

Testing Precision Limits of Neural Network-Based Quality Control Metrics in High-Throughput Digital Microscopy

  • Research Paper
  • Published:
Pharmaceutical Research Aims and scope Submit manuscript

Abstract

Objective

Digital microscopy is used to monitor particulates such as protein aggregates within biopharmaceutical products. The images that result encode a wealth of information that is underutilized in pharmaceutical process monitoring. For example, images of particles in protein drug products typically are analyzed only to obtain particle counts and size distributions, even though the images also reflect particle characteristics such as shape and refractive index. Multiple groups have demonstrated that convolutional neural networks (CNNs) can extract information from images of protein aggregates allowing assignment of the likely stress at the “root-cause” of aggregation. A practical limitation of previous CNN-based approaches is that the potential aggregation-inducing stresses must be known a priori, disallowing identification of particles produced by unknown stresses.

Methods

We demonstrate an expanded CNN analysis of flow imaging microscopy (FIM) images incorporating judiciously chosen particle standards within a recently proposed “fingerprinting algorithm” (Biotechnol. & Bioeng. (2020) 117:3322) that allows detection of particles formed by unknown root-causes. We focus on ethylene tetrafluoroethylene (ETFE) microparticles as standard surrogates for protein aggregates. We quantify the sensitivity of the new algorithm to experimental parameters such as microscope focus and solution refractive index changes, and explore how FIM sample noise affects statistical testing procedures.

Results & Conclusions

Applied to real-world microscopy images of protein aggregates, the algorithm reproducibly detects complex, distinguishing “textural features” of particles that are not easily described by standard morphological measurements. This offers promise for quality control applications and for detecting shifts in protein aggregate populations due to stresses resulting from unknown process upsets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Agresti, A.: Statistics : The Art and Science of Learning from Data. Pearson Prentice Hall, Upper Saddle River, NJ (2006).

  2. Calderon, C.P., Daniels, A.L., Randolph, T.W.: Deep Convolutional Neural Network Analysis of Flow Imaging Microscopy Data to Classify Subvisible Particles in Protein Formulations. Journal of Pharmaceutical Sciences 107(4), 999–1008 (2018). https://doi.org/10.1016/j.xphs.2017.12.008.http://linkinghub.elsevier.com/retrieve/pii/S002235491730878X.

  3. Cavicchi, R.E., Ripple, D.C.: Improving Diameter Accuracy for Dynamic Imaging Microscopy for Different Particle Types. Journal of Pharmaceutical Sciences 109(1), 488–495 (2020). https://doi.org/10.1016/j.xphs.2019.10.017.https://doi.org/10.1016/j.xphs.2019.10.017.

  4. D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M.D., Hormozdiari, F., Houlsby, N., Hou, S., Jerfel, G., Karthikesalingam, A., Lucic, M., Ma, Y., McLean, C., Mincu, D., Mitani, A., Montanari, A., Nado, Z., Natarajan, V., Nielson, C., Osborne, T.F., Raman, R., Ramasamy, K., Sayres, R., Schrouff, J., Seneviratne, M., Sequeira, S., Suresh, H., Veitch, V., Vladymyrov, M., Wang, X., Webster, K., Yadlowsky, S., Yun, T., Zhai, X., Sculley, D.: Underspecification presents challenges for credibility in modern machine learning (2020)

  5. Daniels, A.L., Calderon, C.P., Randolph, T.W.: Machine learning and statistical analyses for extracting and fingerprints of antibody aggregation at container interfaces from flow microscopy images. Biotechnology and Bioengineering 117(11), 3322–3335 (2020). https://doi.org/10.1002/bit.27501.https://onlinelibrary.wiley.com/doi/abs/10.1002/bit.27501.

  6. Duong, T.: ks: Kernel Density Estimation and Kernel Discriminant Analysis for Multivariate Data in R. Journal of Statistical Software 21(7) (2007).

  7. Duong, T.: ks: Kernel Smoothing (2021). https://CRAN.R-project.org/package=ks. R package version 1.12.0

  8. Fradkin, A.H., Carpenter, J.F., Randolph, T.W.: Immunogenicity of Aggregates of Recombinant Human Growth Hormone in Mouse Models. J. Pharm. Sci. 98(9), 3247–3264 (2009). https://doi.org/10.1002/jps.21834.to.

  9. Gambe-Gilbuena, A., Shibano, Y., Krayukhina, E., Torisu, T., Uchiyama, S.: Automatic Identification of the Stress Sources of Protein Aggregates Using Flow Imaging Microscopy Images. Journal of Pharmaceutical Sciences 109(1), 614–623 (2019). https://doi.org/10.1016/j.xphs.2019.10.034.http://www.ncbi.nlm.nih.gov/pubmed/31669608.

  10. Grabarek, A.D., Senel, E., Menzen, T., Hoogendoorn, K.H., Pike-Overzet, K., Hawe, A., Jiskoot, W.: Particulate impurities in cell-based medicinal products traced by flow imaging microscopy combined with deep learning for image analysis. Cytotherapy 000, 1–9 (2020). https://doi.org/10.1016/j.jcyt.2020.04.093.https://doi.org/10.1016/j.jcyt.2020.04.093.

  11. Helbig, C., Ammann, G., Menzen, T., Friess, W., Wuchner, K., Hawe, A.: Backgrounded Membrane Imaging (BMI) for High-Throughput Characterization of Subvisible Particles During Biopharmaceutical Drug Product Development. Journal of Pharmaceutical Sciences 109(1), 264–276 (2020). https://doi.org/10.1016/j.xphs.2019.03.024.https://doi.org/10.1016/j.xphs.2019.03.024.

  12. Hermans, A., Beyer, L., Leibe, B.: In Defense of the Triplet Loss for Person Re-Identification. arXiv: 1703.07737 (2017). http://arxiv.org/abs/1703.07737

  13. Hong, Y., Li, H.: Nonparametric Specification Testing for Continuous-Time Models with Applications to Term Structure of Interest Rates. Review of Financial Studies 18(1), 37–84 (2005). https://doi.org/10.1093/rfs/hhh006.http://rfs.oupjournals.org/cgi/doi/10.1093/rfs/hhh006.

  14. Kiese, S., Pappenberger, A., Friess, W., Mahler, H.C.: Shaken, Not Stirred: Mechanical Stress Testing of An IgG1 Antibody. Journal of Pharmaceutical Sciences 97(10), 4347 (2008). DOI: . https://doi.org/10.1002/jps.

  15. Kiyoshi, M., Shibata, H., Harazono, A., Torisu, T., Maruno, T., Akimaru, M., Asano, Y., Hirokawa, M., Ikemoto, K., Itakura, Y., Iwura, T., Kikitsu, A., Kumagai, T., Mori, N., Murase, H., Nishimura, H., Oda, A., Ogawa, T., Ojima, T., Okabe, S., Saito, S., Saitoh, S., Suetomo, H., Takegami, K., Takeuchi, M., Yasukawa, H., Uchiyama, S., Ishii-Watabe, A.: Collaborative Study for Analysis of Subvisible Particles Using Flow Imaging and Light Obscuration: Experiences in Japanese Biopharmaceutical Consortium. Journal of Pharmaceutical Sciences 108(2), 832–841 (2019). https://doi.org/10.1016/j.xphs.2018.08.006.https://linkinghub.elsevier.com/retrieve/pii/S0022354918305057.

  16. Kotarek, J., Stuart, C., De Paoli, S.H., Simak, J., Lin, T.L., Gao, Y., Ovanesov, M., Liang, Y., Scott, D., Brown, J., Bai, Y., Metcalfe, D.D., Marszal, E., Ragheb, J.A.: Subvisible Particle Content, Formulation, and Dose of an Erythropoietin Peptide Mimetic Product Are Associated with Severe Adverse Postmarketing Events. Journal of Pharmaceutical Sciences 105(3), 1023–1027 (2016). https://doi.org/10.1016/S0022-3549(15)00180-X.

    Article  CAS  PubMed  Google Scholar 

  17. Krause, N., Kuhn, S., Frotscher, E., Nikels, F., Hawe, A., Garidel, P., Menzen, T.: Oil-Immersion Flow Imaging Microscopy for Quantification and Morphological Characterization of Submicron Particles in Biopharmaceuticals. AAPS Journal 23(1), 1–11 (2021).https://doi.org/10.1208/s12248-020-00547-9.

    Article  CAS  Google Scholar 

  18. Lehmann, E., Romano, J.: Testing statistical hypotheses. Springer-Verlag, New York (2005).

    Google Scholar 

  19. Ma, Y., Ashraf, M., Srinivasan, C.: Microscopic evaluation of pharmaceutical glass container-formulation interactions under stressed conditions. International Journal of Pharmaceutics 596(December 2020), 120248 (2021). 10.1016/j.ijpharm.2021.120248. https://doi.org/10.1016/j.ijpharm.2021.120248

  20. Maddux, N.R., Daniels, A.L., Randolph, T.W.: Microflow Imaging Analyses Reflect Mechanisms of Aggregate Formation: Comparing Protein Particle Data Sets Using the Kullback-Leibler Divergence. Journal of Pharmaceutical Sciences 106(5), 1239–1248 (2017). DOI: https://doi.org/10.1016/j.xphs.2017.01.030.http://dx.doi.org/10.1016/j.xphs.2017.01.030.

  21. Narhi, L.O., Corvari, V., Ripple, D.C., Afonina, N., Cecchini, I., Defelippis, M.R., Garidel, P., Herre, A., Koulov, A.V., Lubiniecki, T., Mahler, H.C., Mangiagalli, P., Nesta, D., Perez-Ramirez, B., Polozova, A., Rossi, M., Schmidt, R., Simler, R., Singh, S., Spitznagel, T.M., Weiskopf, A., Wuchner, K.: Subvisible (2–100 μm) particle analysis during biotherapeutic drug product development: Part 1, considerations and strategy. Journal of Pharmaceutical Sciences 104(6), 1899–1908 (2015). DOI: https://doi.org/10.1002/jps.24437.http://dx.doi.org/10.1002/jps.24437.

  22. Probst, C.: Characterization of Protein Aggregates, Silicone Oil Droplets, and Protein-Silicone Interactions Using Imaging Flow Cytometry. Journal of Pharmaceutical Sciences 109(1), 364–374 (2020). https://doi.org/10.1016/j.xphs.2019.05.018.https://doi.org/10.1016/j.xphs.2019.05.018.

  23. Probst, C., Zayats, A., Venkatachalam, V., Davidson, B.: Advanced Characterization of Silicone Oil Droplets in Protein Therapeutics Using Artificial Intelligence Analysis of Imaging Flow Cytometry Data. Journal of Pharmaceutical Sciences 109(10), 2996–3005 (2020). DOI: https://doi.org/10.1016/j.xphs.2020.07.008.https://doi.org/10.1016/j.xphs.2020.07.008.

  24. Probst, C., Zeng, Y., Zhu, R.R.: Characterization of Protein Particles in Therapeutic Formulations Using Imaging Flow Cytometry. Journal of Pharmaceutical Sciences 106(8), 1952–1960 (2017). https://doi.org/10.1016/j.xphs.2017.04.034.http://dx.doi.org/10.1016/j.xphs.2017.04.034.

  25. Ripple, D., Telikepalli, S., Steffens, K., Carrier, M., Montgomery, C., Ritchie, N., Lu, J.: Reference Material 8634: Ethylene Tetrafluoroethylene for Particle Size Distribution and Morphology (2019). https://doi.org/10.6028/NIST.SP.260-193. NIST Special Publication 260-193

  26. Ripple, D.C., Montgomery, C.B., Hu, Z.: An interlaboratory comparison of sizing and counting of subvisible particles mimicking protein aggregates. Journal of Pharmaceutical Sciences 104(2), 666–677 (2015). https://doi.org/10.1002/jps.24287.http://dx.doi.org/10.1002/jps.24287.

  27. Rosenblatt, M.: Remarks on a Multivariate Transformation. Ann. Math. Statist. 23(3), 470 (1952).

    Article  Google Scholar 

  28. Saggu, M., Patel, A.R., Koulis, T.: A Random Forest Approach for Counting Silicone Oil Droplets and Protein Particles in Antibody Formulations Using Flow Microscopy. Pharmaceutical Research 34(2), 479–491 (2017). https://doi.org/10.1007/s11095-016-2079-x.http://dx.doi.org/10.1007/s11095-016-2079-x.

  29. Scott, D.: Multivariate Density Estimation: Theory, Practice, and Visualization. John Wiley & Sons, New York (1992).

    Book  Google Scholar 

  30. Strehl, R., Rombach-Riegraf, V., Diez, M., Egodage, K., Bluemel, M., Jeschke, M., Koulov, A.V.: Discrimination between silicone oil droplets and protein aggregates in biopharmaceuticals: A novel multiparametric image filter for sub-visible particles in microflow imaging analysis. Pharmaceutical Research 29(2), 594–602 (2012).https://doi.org/10.1007/s11095-011-0590-7.

    Article  CAS  PubMed  Google Scholar 

  31. Vargas, S.K., Eskafi, A., Carter, E., Ciaccio, N.: A comparison of background membrane imaging versus flow technologies for subvisible particle analysis of biologics. International Journal of Pharmaceutics 578(January), 1–9 (2020). DOI: 10.1016/j.ijpharm.2020.119072.

    Article  CAS  Google Scholar 

  32. Wang, H., Wu, X., Huang, Z., Xing, E.P.: High-frequency component helps explain the generalization of convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

  33. Werk, T., Volkin, D.B., Mahler, H.C.: Effect of solution properties on the counting and sizing of subvisible particle standards as measured by light obscuration and digital imaging methods. European Journal of Pharmaceutical Sciences 53(1), 95–108 (2014). DOI: https://doi.org/10.1016/j.ejps.2013.12.014.http://dx.doi.org/10.1016/j.ejps.2013.12.014.

  34. Witeof, A.E., Daniels, A.L., Rea, L.T., Movafaghi, S., Kurtz, K., Davis, M., Eveland, R.W., Calderon, C.P., Randolph, T.W.: Machine Learning and Accelerated Stress Approaches to Differentiate Potential Causes of Aggregation in Polyclonal Antibody Formulations During Shipping. Journal of Pharmaceutical Sciences 110, 2743–2752 (2021).https://doi.org/10.1016/j.xphs.2021.02.029.https://doi.org/10.1016/j.xphs.2021.02.029.

Download references

Acknowledgements

The authors would like to thank Drs. Lewis Geer and Sumona Sarkar at NIST for helpful comments on this manuscript. Research performed in part at the NIST Center for Nanoscale Science and Technology. We would like to acknowledge Oak Ridge Associated Universities (ORAU) for funding the ORISE fellowship supporting C.S. and Y.M.’s work.

Funding

CPC's work was supported by NIH Award # R41GM130513, FDA PO 75F40120P00203, and private funding through SentrySciences, LLC.

Author information

Authors and Affiliations

  1. Ursa Analytics, Inc., Denver, CO, 80212, USA

    Christopher P. Calderon

  2. Department of Chemical and Biological Engineering, University of Colorado Boulder, CO, 80303, Boulder, USA

    Christopher P. Calderon & Theodore W. Randolph

  3. Biomolecular Measurement Division, National Institute of Standards and Technology, Gaithersburg, MD, 20899, USA

    Dean C. Ripple & Michael J. Carrier

  4. Division of Product Quality Research, Office of Testing and Research, OPQ, CDER, FDA, MD, 20993, USA

    Charudharshini Srinivasan, Youlong Ma & Thomas F. O’Connor

Authors
  1. Christopher P. Calderon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dean C. Ripple
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charudharshini Srinivasan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Youlong Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael J. Carrier
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Theodore W. Randolph
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Thomas F. O’Connor
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christopher P. Calderon.

Ethics declarations

Conflict of interest

The authors (C.P.C., and T.W.R.) are inventors of intellectual property related to this manuscript that is owned by Ursa Analytics, Inc. and the Regents of the University of Colorado. Aspects of this technology are licensed to SentrySciences who partially funded C.P.C.’s efforts. C.S. and T.F.O. are employed by the US Food and Drug Administration. The authors do not declare any other conflict of interest. Commercial equipment and materials are identified in order to adequately specify certain procedures. In no case does such identification imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose. This article reflects the views of the authors and should not be construed to represent FDAs views or policies.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Calderon, C.P., Ripple, D.C., Srinivasan, C. et al. Testing Precision Limits of Neural Network-Based Quality Control Metrics in High-Throughput Digital Microscopy. Pharm Res 39, 263–279 (2022). https://doi.org/10.1007/s11095-021-03130-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11095-021-03130-9

Keywords

Search

Navigation