Abstract
With the rapid development of artificial intelligence and computing chips approaching the bottleneck of power consumption and computing power, the research on intelligent computing hardware with high speed and high energy efficiency is an important trend. Recently, neuromorphic computing represented by photonic circuit neural networks and all-optical diffraction neural networks has attracted widespread attention due to their ultra-fast and ultra-efficient computing architectures. In this perspective, we first review some representative works and introduce them through two main lines of planar photonic circuit neural networks and three-dimensional diffraction neural networks to compare their characteristics and performance. We further discuss programmable designs for neuromorphic computing hardware, which bring it closer to general-purpose computing devices. Besides intelligent neural networks in the optical band, we also review the development and application of the diffractive neural networks in the microwave band, showing their programmable capabilities. Finally, we present the future directions and development trends of intelligent neuromorphic computing and its potential applications in wireless communications, information processing, and sensing.
Graphical abstract
Similar content being viewed by others
Data availability
The data that support the findings of this study are available on request from the corresponding authors.
References
D. Silver et al., Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016)
E.A. Van Dis, J. Bollen, W. Zuidema, R. van Rooij, C.L. Bockting, ChatGPT: five priorities for research. Nature 614, 224–226 (2023)
T. Brown et al., Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)
X. Xu et al., Self-calibrating programmable photonic integrated circuits. Nat. Photon. 16, 595–602 (2022). https://doi.org/10.1038/s41566-022-01020-z
J. Feldmann et al., Parallel convolutional processing using an integrated photonic tensor core. Nature 589, 52–58 (2021). https://doi.org/10.1038/s41586-020-03070-1
J. Feldmann, N. Youngblood, C.D. Wright, H. Bhaskaran, W.H.P. Pernice, All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019). https://doi.org/10.1038/s41586-019-1157-8
G. Wetzstein et al., Inference in artificial intelligence with deep optics and photonics. Nature 588, 39–47 (2020). https://doi.org/10.1038/s41586-020-2973-6
Y. Li, Y. Luo, D. Mengu, B. Bai, A. Ozcan, Quantitative phase imaging (QPI) through random diffusers using a diffractive optical network. Light Adv. Manuf. (2023). https://doi.org/10.37188/lam.2023.017
D. Mengu, Y. Luo, Y. Rivenson, A. Ozcan, Analysis of diffractive optical neural networks and their integration with electronic neural networks. IEEE J. Sel. Top. Quantum Electron. (2020). https://doi.org/10.1109/JSTQE.2019.2921376
T. Wang et al., Image sensing with multilayer nonlinear optical neural networks. Nat. Photonics 17, 408–415 (2023). https://doi.org/10.1038/s41566-023-01170-8
T.W. Hughes, M. Minkov, Y. Shi, S. Fan, Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica (2018). https://doi.org/10.1364/optica.5.000864
Y. Chen, T. Zhou, J. Wu, H. Qiao, X. Lin, L. Fang, Q. Dai, Photonic unsupervised learning variational autoencoder for high-throughput and low-latency image transmission. Sci. Adv. 9, eadf8437 (2023). https://doi.org/10.1126/sciadv.adf8437
Y. Luo et al., Design of task-specific optical systems using broadband diffractive neural networks. Light Sci. Appl. 8, 112 (2019). https://doi.org/10.1038/s41377-019-0223-1
I.A.D. Williamson et al., Reprogrammable electro-optic nonlinear activation functions for optical neural networks. IEEE J. Sel. Top. Quantum Electron. 26, 1–12 (2020). https://doi.org/10.1109/jstqe.2019.2930455
Y. Shen et al., Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–446 (2017). https://doi.org/10.1038/nphoton.2017.93
X. Lin et al., All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018). https://doi.org/10.1126/science.aat8084
F. Shokraneh, S. Geoffroy-Gagnon, M.S. Nezami, O. Liboiron-Ladouceur, A single layer neural network implemented by a $4\times 4$ MZI-based optical processor. IEEE Photon. J. 11, 1–12 (2019). https://doi.org/10.1109/jphot.2019.2952562
M.M. Pour Fard et al., Experimental realization of arbitrary activation functions for optical neural networks. Opt. Express 28, 12138–12148 (2020). https://doi.org/10.1364/OE.391473
T.W. Hughes, M. Minkov, Y. Shi, S.H. Fan, Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018). https://doi.org/10.1364/Optica.5.000864
K. Liao et al., All-optical computing based on convolutional neural networks. Opto-Electron. Adv. 4, 200060–200060 (2021). https://doi.org/10.29026/oea.2021.200060
R. Hamerly, L. Bernstein, A. Sludds, M. Soljačić, D. Englund, Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X (2019). https://doi.org/10.1103/PhysRevX.9.021032
T. Park, Y. Jeong, K. Yu, Cascaded optical resonator-based programmable photonic integrated circuits. Opt. Express 29, 4645–4660 (2021). https://doi.org/10.1364/OE.415545
H. Chen et al., Diffractive deep neural networks at visible wavelengths. Engineering 7, 1483–1491 (2021). https://doi.org/10.1016/j.eng.2020.07.032
Yan, T. et al. in optoelectronic imaging and multimedia technology VI (2019)
J. Li et al., Unidirectional imaging using deep learning–designed materials. Sci. Adv. 9, eadg1505 (2023)
T. Yan et al., Fourier-space diffractive deep neural network. Phys. Rev. Lett. 123, 023901 (2019). https://doi.org/10.1103/PhysRevLett.123.023901
M.S.S. Rahman, J. Li, D. Mengu, Y. Rivenson, A. Ozcan, Ensemble learning of diffractive optical networks. Light Sci. Appl. 10, 14 (2021). https://doi.org/10.1038/s41377-020-00446-w
L. Zhang et al., Optical machine learning using time-lens deep neural networks. Photonics (2021). https://doi.org/10.3390/photonics8030078
J. Chang, V. Sitzmann, X. Dun, W. Heidrich, G. Wetzstein, Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018). https://doi.org/10.1038/s41598-018-30619-y
G. Qu et al., All-dielectric metasurface empowered optical-electronic hybrid neural networks. Laser Photon. Rev. 16, 2100732 (2022). https://doi.org/10.1002/lpor.202100732
Y.-L. Xiao, S. Li, G. Situ, Z. You, Unitary learning for diffractive deep neural network. Opt. Lasers Eng. (2021). https://doi.org/10.1016/j.optlaseng.2020.106499
J. Li et al., Spectrally encoded single-pixel machine vision using diffractive networks. Sci. Adv. 7, eabd7690 (2021)
E. Goi, S. Schoenhardt, M. Gu, Direct retrieval of Zernike-based pupil functions using integrated diffractive deep neural networks. Nat. Commun. 13, 7531 (2022). https://doi.org/10.1038/s41467-022-35349-4
X. Luo et al., Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible. Light Sci. Appl. 11, 158 (2022). https://doi.org/10.1038/s41377-022-00844-2
D. Mengu et al., Misalignment resilient diffractive optical networks. Nanophotonics 9, 4207–4219 (2020). https://doi.org/10.1515/nanoph-2020-0291
J. Li, D. Mengu, Y. Luo, Y. Rivenson, A. Ozcan, Class-specific differential detection in diffractive optical neural networks improves inference accuracy. Adv. Photon. (2019). https://doi.org/10.1117/1.Ap.1.4.046001
T. Zhou et al., Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photon. 15, 367–373 (2021). https://doi.org/10.1038/s41566-021-00796-w
T. Fu et al., On-chip photonic diffractive optical neural network based on a spatial domain electromagnetic propagation model. Opt. Express 29, 31924–31940 (2021). https://doi.org/10.1364/OE.435183
S. Zarei, M.R. Marzban, A. Khavasi, Integrated photonic neural network based on silicon metalines. Opt. Express 28, 36668–36684 (2020). https://doi.org/10.1364/OE.404386
T. Fu et al., Photonic machine learning with on-chip diffractive optics. Nat. Commun. 14, 70 (2023). https://doi.org/10.1038/s41467-022-35772-7
Z. Wang, L. Chang, F. Wang, T. Li, T. Gu, Integrated photonic metasystem for image classifications at telecommunication wavelength. Nat. Commun. 13, 2131 (2022). https://doi.org/10.1038/s41467-022-29856-7
E. Khoram et al., Nanophotonic media for artificial neural inference. Photon. Res. (2019). https://doi.org/10.1364/prj.7.000823
V.J.U.F.N. Veselago, Electrodynamics of substances with simultaneously negative and. Usp. Fiz. Nauk 92, 517 (1967)
J.B. Pendry, A.J. Holden, W.J. Stewart, I. Youngs, Extremely low frequency plasmons in metallic mesostructures. Phys. Rev. Lett. 76, 4773–4776 (1996). https://doi.org/10.1103/PhysRevLett.76.4773
R.A. Shelby, D.R. Smith, S. Schultz, Experimental verification of a negative index of refraction. Science 292, 77–79 (2001). https://doi.org/10.1126/science.1058847
T.J. Cui, M.Q. Qi, X. Wan, J. Zhao, Q. Cheng, Coding metamaterials, digital metamaterials and programmable metamaterials. Light Sci. Appl. 3, e218–e218 (2014). https://doi.org/10.1038/lsa.2014.99
T.J. Cui et al., Information metamaterial systems. Iscience (2020). https://doi.org/10.1016/j.isci.2020.101403
Q. Ma, T.J. Cui, Information metamaterials: bridging the physical world and digital world. PhotoniX 1, 1–32 (2020). https://doi.org/10.1186/s43074-020-00006-w
Q. Ma et al., Digital coding metasurfaces: from theory to applications. IEEE Antennas Propag. Mag. 64, 96–109 (2022). https://doi.org/10.1109/MAP.2022.3169397
T.J. Cui, S. Liu, G.D. Bai, Q. Ma, Direct transmission of digital message via programmable coding metasurface. Research 2019, 2584509 (2019). https://doi.org/10.34133/2019/2584509
J.Y. Dai et al., Wireless communication based on information metasurfaces. IEEE Trans. Microw. Theory Tech. 69, 1493–1510 (2021). https://doi.org/10.1109/tmtt.2021.3054662
L. Zhang et al., A wireless communication scheme based on space- and frequency-division multiplexing using digital metasurfaces. Nat. Electron. 4, 218–227 (2021). https://doi.org/10.1038/s41928-021-00554-4
L. Chen et al., Touch-programmable metasurface for various electromagnetic manipulations and encryptions. Small (2022). https://doi.org/10.1002/smll.202203871
L. Li et al., Intelligent metasurface imager and recognizer. Light Sci. Appl. 8, 1–9 (2019). https://doi.org/10.1038/s41377-019-0209-z
Q. Ma et al., Smart metasurface with self-adaptively reprogrammable functions. Light-Sci. Appl. 8, 98 (2019). https://doi.org/10.1038/s41377-019-0205-3
Q. Ma et al., Directly wireless communication of human minds via non-invasive brain-computer-metasurface platform. eLight (2022). https://doi.org/10.1186/s43593-022-00019-x
Z. Gu et al., High-resolution programmable metasurface imager based on multilayer perceptron network. Adv. Opt. Mater. (2022). https://doi.org/10.1002/adom.202200619
Q. Xiao et al., Electromagnetic brain–computer–metasurface holography. ACS Photon. (2023). https://doi.org/10.1021/acsphotonics.2c01349
L. Li et al., Machine-learning reprogrammable metasurface imager. Nat. Commun. 10, 1082 (2019). https://doi.org/10.1038/s41467-019-09103-2
X. Gao et al., Programmable surface plasmonic neural networks for microwave detection and processing. Nat. Electron. 6, 319–328 (2023). https://doi.org/10.1038/s41928-023-00951-x
C. Liu et al., A programmable diffractive deep neural network based on a digital-coding metasurface array. Nat. Electron. 5, 113–122 (2022). https://doi.org/10.1038/s41928-022-00719-9
C. Qian et al., Performing optical logic operations by a diffractive neural network. Light-Sci. Appl. (2020). https://doi.org/10.1038/s41377-020-0303-2
C. Qian et al., Dynamic recognition and mirage using neuro-metamaterials. Nat. Commun. 13, 2694 (2022). https://doi.org/10.1038/s41467-022-30377-6
G. Xinxin, M.Q. Guze, C.W. Yi, C.T. Jun, C.C. Hou, Reconfigurable spoof localized surface plasmonic for frequency detections. Laser Photon. Rev. (2023). https://doi.org/10.1002/lpor.202300267
X. Gao et al., Multimode and reconfigurable phase shifter of spoof surface plasmons. IEEE Trans. Antennas Propag. 71, 5361–5369 (2023). https://doi.org/10.1109/tap.2023.3262348
X. Gao et al., Multifunctional terahertz spoof plasmonic devices. Adv. Mater. Technol. (2023). https://doi.org/10.1002/admt.202202050
X. Gao et al., Reprogrammable spoof plasmonic modulator. Adv. Func. Mater. (2023). https://doi.org/10.1002/adfm.202212328
X. Gao et al., Programmable multifunctional device based on spoof surface plasmon polaritons. IEEE Trans. Antennas Propag. 68, 3770–3779 (2020). https://doi.org/10.1109/tap.2020.2969745
X. Gao et al., Nonmagnetic spoof plasmonic isolator based on parametric amplification. Laser Photon. Rev. (2022). https://doi.org/10.1002/lpor.202100578
X. Gao et al., Reconfigurable parametric amplifications of spoof surface plasmons. Adv. Sci. 8, e2100795 (2021). https://doi.org/10.1002/advs.202100795
X. Gao et al., Dynamic controls of second-harmonic generations in both forward and backward modes using reconfigurable plasmonic metawaveguide. Adv. Opt. Mater. (2020). https://doi.org/10.1002/adom.201902058
X. Gao et al., Crosstalk suppression based on mode mismatch between spoof SPP transmission line and microstrip. IEEE Trans. Compon. Packag. Manuf. Technol. 9, 2267–2275 (2019). https://doi.org/10.1109/tcpmt.2019.2931373
H.C. Zhang et al., A plasmonic route for the integrated wireless communication of subdiffraction-limited signals. Light Sci Appl 9, 113 (2020). https://doi.org/10.1038/s41377-020-00355-y
S.R. Joy, M. Erementchouk, H. Yu, P. Mazumder, Spoof plasmon interconnects—communications beyond RC limit. IEEE Trans. Commun. 67, 599–610 (2019). https://doi.org/10.1109/tcomm.2018.2874242
Y.R. Xing Lin, N.T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, A. Ozcan, All-optical machine learning using diffractive deep neural networks. Science 7, 361 (2018)
G. Qu et al., All-dielectric metasurface empowered optical-electronic hybrid neural networks. Laser Photon. Rev. (2022). https://doi.org/10.1002/lpor.202100732
H. Zhang et al., An optical neural chip for implementing complex-valued neural network. Nat. Commun. 12, 457 (2021). https://doi.org/10.1038/s41467-020-20719-7
H.H. Zhu et al., Space-efficient optical computing with an integrated chip diffractive neural network. Nat. Commun. 13, 1044 (2022). https://doi.org/10.1038/s41467-022-28702-0
Y. Shi et al., Nonlinear germanium-silicon photodiode for activation and monitoring in photonic neuromorphic networks. Nat. Commun. 13, 6048 (2022). https://doi.org/10.1038/s41467-022-33877-7
W.W. Tiankuang Zhou, J. Zhang, Yu. Shaoliang, Lu. Fang, Ultrafast dynamic machine vision with spatiotemporal photonic computing. Sci. Adv. 9, eadg4391 (2023)
H. Dou et al., Residual D2NN: training diffractive deep neural networks via learnable light shortcuts. Opt. Lett. 45, 2688–2691 (2020). https://doi.org/10.1364/OL.389696
Y. Chen et al., Photonic unsupervised learning variational autoencoder for high-throughput and low-latency image transmission. Sci. Adv. 9, eadf8437 (2023). https://doi.org/10.1126/sciadv.adf8437
Szegedy, C. et al. in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1–9
P. Yao et al., Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020). https://doi.org/10.1038/s41586-020-1942-4
T. Zhou, W. Wu, J. Zhang, S. Yu, L. Fang, Ultrafast dynamic machine vision with spatiotemporal photonic computing. Sci. Adv. 9, eadg4391 (2023). https://doi.org/10.1126/sciadv.adg4391
Y. Chen et al., All-analog photoelectronic chip for high-speed vision tasks. Nature 623, 48–57 (2023). https://doi.org/10.1038/s41586-023-06558-8
C. Qian et al., Performing optical logic operations by a diffractive neural network. Light Sci. Appl. (2020). https://doi.org/10.1038/s41377-020-0303-2
T. Cui, B. Bai, H.-B. Sun, Tunable metasurfaces based on active materials. Adv. Func. Mater. 29, 1806692 (2019). https://doi.org/10.1002/adfm.201806692
Z. Zheng et al., Dual adaptive training of photonic neural networks. Nat. Mach. Intell. 5, 1119–1129 (2023). https://doi.org/10.1038/s42256-023-00723-4
S. Venkatesh, X.Y. Lu, H. Saeidi, K. Sengupta, A high-speed programmable and scalable terahertz holographic metasurface based on tiled CMOS chips. Nat. Electron. 3, 785–793 (2020). https://doi.org/10.1038/s41928-020-00497-2
J. Park et al., All-solid-state spatial light modulator with independent phase and amplitude control for three-dimensional LiDAR applications. Nat. Nanotechnol. 16, 69–76 (2021). https://doi.org/10.1038/s41565-020-00787-y
H. Kwon, T. Zheng, A. Faraon, Nano-electromechanical spatial light modulator enabled by asymmetric resonant dielectric metasurfaces. Nat. Commun. 13, 5811 (2022). https://doi.org/10.1038/s41467-022-33449-9
Hermans, A. et al. in 2022 Conference on Lasers and Electro-Optics (CLEO). 1–2.
T. Zhou et al., In situ optical backpropagation training of diffractive optical neural networks. Photon. Res. 8, 940–953 (2020). https://doi.org/10.1364/PRJ.389553
Y.M. Ning, Q. Ma, Q. Xiao, Z. Gu, T.J. Cui, Reprogrammable nonlinear transmission controls using an information metasurface. Adv. Opt. Mater. (2023). https://doi.org/10.1002/adom.202301525
C.T.J. Gao Xinxin, Using surface plasmons to create programmable neural networks. Nat. Electron. 6, 266–267 (2023). https://doi.org/10.1038/s41928-023-00952-w
M. Gu, Y. Dong, H. Yu, H. Luan, Q. Zhang, Perspective on 3D vertically-integrated photonic neural networks based on VCSEL arrays. Nanophotonics 12, 827–832 (2023). https://doi.org/10.1515/nanoph-2022-0437
Funding
The work is supported by the National Natural Science Foundation of China (62288101 and 92167202), the National Key Research and Development Program of China (2022YFA1404903, 2017YFA0700201, 2017YFA0700202, and 2017YFA0700203), the Major Project of Natural Science Foundation of Jiangsu Province (BK20212002), the State Key Laboratory of Millimeter Waves, Southeast University, China (K201924), the Fundamental Research Funds for the Central Universities (2242023K5002, 2242018R30001, 2242022R20017), the 111 Project (111-2-05), and the China Postdoctoral Science Foundation (2242023K5002, 2021M700761, 2022T150112), Young Elite Scientists Sponsorship Program by CAST (2022QNRC001).
Author information
Authors and Affiliations
Contributions
T. J. Cui and Q. Ma initiated the plan and supervised the entire study. Q. Ma, X. Gao, and Z. Gu write the main part of the manuscript. L. Li, J. W. You, C. Liu, and T. J. Cui carried out the modifications and inspections. Q. Ma, X. Gao, Z. Gu, and T. J. Cui prepared the final manuscript with input from all authors. All authors discussed the research.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflict of interest to declare.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ma, Q., Gao, X., Gu, Z. et al. Intelligent neuromorphic computing based on nanophotonics and metamaterials. MRS Communications (2024). https://doi.org/10.1557/s43579-024-00520-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1557/s43579-024-00520-z