Biologically Inspired and Energy-Efficient Neurons

Abstract

Silicon neurons represent different levels of biological details and accuracies as a trade-off between complexity and power consumption. With respect to this trade-off and high similarity to neuron behaviour models, relaxation-type oscillator circuits often yield a good compromise to emulate neurons. In this chapter, two exemplified relaxation-type silicon neurons are presented that emulate neural behaviour with energy consumption under the scale of nJ/spike. The first proposed fully CMOS relaxation SiN is based on mathematical Izhikevich model and can mimic a broad range of physiologically observable spike patterns. The results of kinds of biologically plausible output patterns and coupling process of two SiNs are presented in 0.35 $\upmu$m CMOS technology. The second type is a novel ultra-low-frequency hybrid CMOS-memristive SiN based on relaxation oscillators and analog memristive devices. The hybrid SiN directly emulates neuron behaviour in the range of physiological spiking frequencies (less than 100 Hz). The relaxation oscillator is implemented and fabricated in 0.13 $\upmu$m CMOS technology. An autonomous neuronal synchronization process is demonstrated with two relaxation oscillators coupled by an analog memristive device in the measurement to emulate the synchronous behaviour between spiking neurons.

1 Introduction

Living species are well adapted to their environments, a result of a hundred million years of evolution on earth. Due to constrains in space, time and energy, biological information processing in nervous systems of creatures are shaped during evolution towards an optimum between capabilities and resource consumption [1,2,3]. It provides a benchmark for technical systems particularly when it comes to elaborate brain functions such as conscious awareness or decision-making which consume an incredible small amount of energy within a limited volume (i.e. a space of a brain) [4]. Although, there are continuous remarkable progresses in very large scale integration (VLSI) technology, the gap still remains between digital processors and biological computing systems [5, 6].

Neuromorphic engineering emerges and endeavors to develop intelligent machines with comparable biological computation and energy efficiency [6,7,8,9,10,11,12]. More recently, bio-inspired silicon neurons (SiNs) and artificial neural networks have been vastly investigated and developed to imitate the biological computing scheme [13,14,15]. The nervous system is a fine-grained parallel processing and highly linked neural network [5, 6, 16, 17]. Therefore, the large superiority of biological computing systems for certain tasks like sensory processing or pattern recognition comes from its real-time analog computation in which data processing (e.g., computing and learning) and data storage (i.e., memory) are inseparably linked. However, digital processors based on von Neumann architecture execute binary computing with a strict separation of data processing and storing [6, 18, 19]. Hence, a large part of power consumption is distributed on the data transfer between processing units and memory. As a consequence, the artificial neurons capable of real-time analog computing have recently received increasing interest in silicon neuron design [20]. Furthermore, due to high similarity to neuron behaviour models, the relaxation-type oscillators draw more attentions in recent researches [21, 22].

In this chapter, two exemplified high energy efficient relaxation-type silicon neurons are presented. These two analog silicon neurons represent two promising development strategies in artificial neuron design. One strategy is fully CMOS technology spiking neurons based on mathematical neuron models (here: the Izhikevich-model) and generating a broad range of physiologically observable spike patterns. Spiking frequency, i.e. operating speed, of these artificial neurons is considerably higher than that of biological neurons (e.g. typical mean firing frequencies of biological neurons are in the range of up to 100 Hz). Another strategy is hybrid silicon neurons, which use CMOS analog oscillators and analog memristive devices to implement neuron and synapse circuits. These hybrid SiNs directly emulate the behaviour of neurons at the biological frequency. Both SiNs introduced here allow real-time analog computation rather than binary data processing and enable the realization of large scale monolithic analog neural network in the future.

Before introducing details of two exemplified SiNs, the analog memristive device used in the hybrid SiNs is introduced below firstly. Then Izhikevich-model based low-power SiN is introduced in Sect. 2 including the theories, topology, simulation and measurement results. In Sect. 3 the hybrid SiN based on low frequency relaxation oscillator is described. In order to understand the synchronization process of memristive coupling, the design and measurement results of single oscillator is introduced at first. Subsequently, the autonomous synchronization processes of two pulse-coupled oscillators via a RC network and an analog memristive device are experimentally verified, respectively. Finally, a brief conclusion in Sect. 4 completes the chapter.

1.1 Memristive Devices

The mutual memristive coupling of two self-sustained relaxation oscillators has been successfully realized experimentally [22], through the use of an Ag-doped-$\textrm{TiO}_{2-x}$-Al memristive device with digital switching behavior. Compared with Ag-doped-$\textrm{TiO}_{2-x}$-Al memristive devices, interface-based memristive devices like the double barrier memristive devices (DBMDs) [23] with analog switching behavior (i.e., a continuous change in resistance) behave more similarly to synapses in nervous systems. Consequently, to come closer to emulate the process of synchronization in neurons, in Sect. 3.3.3 an interface-based device will be adopted as an artificial synapse in our experiments. A brief overview of DBMDs is provided in the following. Interested readers are referred to the literature for more details [23,24,25,26,27,28,29,30].

Fig. 1

A schematic structure a and current density versus voltage characteristics of a double barrier memristive device plotted as absolute value on a semi-logarithmic scale (b). b is reproduced from [23] (licensed under CC BY 4.0—https://creativecommons.org/licenses/by/4.0/)

The schematic of a double barrier memristive device is shown in Fig. 1a. It consists of a Nb/NbO$_{\textrm{z}}$/$\textrm{Al}_2\textrm{O}_3$/$\textrm{Nb}_x\textrm{O}_y$/Au material stack [23, 28], in which Nb and Au are the bottom and top electrodes, respectively, while $\textrm{Al}_2\textrm{O}_3$ acts as tunnel barrier and the $\textrm{Nb}_x\textrm{O}_y$/Au interface forms a Schottky-like contact [23, 24, 29]. The devices are produced by DC magnetron sputtering of all materials on 100 mm wafer without breaking the vacuum and subsequent structuring by standard photolithography, lift-off and etching [23, 24]. Figure 1b shows a typical absolute value of the current density versus voltage $(|\textit{J}|-\textit{V})$ hysteresis curve of DBMDs [23]. A gradual resistive switching behavior is present [23] rather than abrupt resistance jumps observed by digital memristive devices [22]. In the $|\textit{J}|-\textit{V}$ measurement, the voltage has been ramped from 0 to 3 V to set the device from its initial high resistance state (HRS) to a low resistance state (LRS). Afterwards, the voltage has been ramped down to -2 V and back to 0 V to reset the device. Voltage was applied to the top electrode while the bottom electrode was grounded. Using low voltages (e.g., 0.5 V) allows for non-destructive read-out of the device state. The diode-like characteristic with a high $\textit{J-V}$ non-linearity obtained by the Schottky barrier facilitates integration into passive crossbar arrays [27], since the current at negative bias voltage is negligible compared to positive bias. The devices can further be gradually switched by using voltage pulses with different amplitudes and widths [26]. Switching occurs for pulses with a width in the millisecond regime or beyond. The non-linear switching process is further crucial dependent on the voltage amplitude. Moreover, the memristive state relaxes towards HRS with time [23]. These effects are considered in the coupling experiments shown in Sect. 3.3.3. Due to the diode-like characteristics, the main interaction between two oscillators happens when a positive voltage is applied to the memristive device, i.e. higher output voltage from an oscillator connected to the top electrode and lower output voltage from an oscillator connected to the bottom electrode, as it is shown in Sect. 3.3.3.

2 Izhikevich-Model Based Low-Power Neuron

In 2003, Izhikevich presented a mathematical model [31] that describes the spiking and bursting behavior of cortical neurons. This model combines the biological plausibility of the Hodgkin-Huxley type [32] and the computational efficiency of the integrate-and-fire model [33]. It has been described, in [31], as a two-dimensional system:

$$\begin{aligned} v'=0.04v^2+5v+140-u+I \end{aligned}$$

(1)

$$\begin{aligned} u'=a(bv-u) \end{aligned}$$

(2)

where $v'$ and $u'$ are the derivatives of v and u with respect to time, respectively: v represents the membrane voltage; and u is a membrane recovery variable that provides a negative feedback to the membrane voltage [31]. The variable I adds incoming synaptic currents to the system. All variables in (1) and (2) and also the parameters a and b are dimensionless.

Fig. 2

Block diagram of the Izhikevich-model based low-power neuron

In [13], the Izhikevich model was implemented in a 0.35 $\upmu$m CMOS technology, consisting of 14 transistors for a single neuron. This neuron’s output, which represents the membrane potential measured at the axon, is tunable by the five different inputs. Figure 2 shows the different inputs and one output of the neuron. The output $V_{mem}$ stays at a resting potential, as long as no currents at the postsynaptic input $I_{syn}$ exceed the threshold. By changing the inputs $V_c$, $V_d$ and $V_{th}$, different biologically plausible spike patterns can be generated [13]. The neuron uses a silicon area of $70\times \,40\, \upmu \textrm{m}^2$ and has a power consumption of 8 pJ/spike.

2.1 Topology

The schematic of this neuron is shown in Fig. 3. We adapted the circuit presented in [13] for the usage in a low-power analog neural network. As shown in Fig. 3, the current input $I_{syn}$ is directly connected to the membrane voltage output. In a neural network built without additional current output, the input current of the first neuron will couple directly to the second neuron. In the biological neuron as in Fig. 2, however, the current input $I_{syn}$ is separated from the axon voltage $V_{mem}$. Therefore, a dedicated current output node delivering the current $I_{axon}$ is added to the circuit. This output can be taken to represent an action potential propagating towards the synapse. Furthermore, we modified the biasing circuit of the original circuit. Instead of a voltage bias via $V_{bias}$, we employ current-biasing using a mirrored input current $I_{bias}$. This enables more sensitive bias control in an integrated circuit.

The neuron circuit can be structured into four parts: a circuit representing the cell membrane and, specifically, the membrane voltage (Fig. 3, blue), the slow variable (Fig. 3, green), a comparator (Fig. 3, yellow), and the axon output (Fig. 3, orange). The membrane circuit integrates the $I_{syn}$ current on the capacitor $C_{mem}$ . The comparator controls a transistor which discharges the membrane capacitance to the resting voltage between spikes. This voltage level is defined by the applied input voltage $V_c$. The slow variable circuit represents Eq. (2), and slows the depolarization of the membrane voltage. The input voltage $V_d$ controls the amount of charge stored on the capacitor $C_u$.

Fig. 3

Schematic of the neuron circuit, divided in four parts: the membrane voltage (blue), the slow variable (green), the comparator (yellow) and the axon output (orange)

A possible configuration for the neural network is presented in Fig. 4. The first layer neuron output current is connected to the synaptic inputs of the second layer. The coupling strength is set via the axon output current by programmable weighted current mirrors. Although the neurons are interconnected, the individual membrane voltages of each neuron can be monitored through the output $V_{mem}$. In doing so, the output of the neural network can be combined using the information of the different membrane voltages.

Fig. 4

Configuration of multiple neurons in a neural network

2.2 Simulation Results

The circuit shown in Fig. 3 is implemented in a 0.35 $\upmu$m CMOS technology and simulated in the Cadence design environment. It is able to generate multiple biologically plausible spike patterns when a stimulation current is applied to the input. By changing the control parameters presented in the previous section, the output of the neuron varies.

2.2.1 Individual Izhikevich-Model Based Low-Power Neuron

The circuit can either be supplied with the nominal operating voltage for this technology of 3 V or with reduced power consumption with 1.5 V. The different operating modes are capable of generating different output patterns. The simplest pattern–single spikes–is shown in Fig. 5. The input current is integrated on the membrane capacitance until the membrane voltage reaches the threshold voltage and a spike is generated. After the spike, the slow variable keeps the circuit reset for a short time. This pattern is generated with a supply voltage of $V_{DD} =$ 3 V, $V_{d} = V_{c} = 0$, $V_{\textrm{th}} =$ 700 mV, $I_{bias} = 1\,\upmu$A, and $I_{syn} =$ 200 nA. All voltages refer to $V_{ss}$ (–1.5 V/–750 mV). Even though the spike frequency is considerably higher than that of a biological neuron, the spike shape is realistic.

Fig. 5

Simulated transient waveform of the circuit’s spiking membrane voltage

Fig. 6

a Transient waveform of Membrane Spiking Voltage. b State trajectory of (a)

With $V_c = 100$ mV and $I_{syn} = 1\, \upmu$A, fast spikes with a reset offset of 100 mV (shown in Fig. 6a) are generated. The larger the value of $I_{syn}$, the faster $C_{mem}$ is charged; thus, the membrane voltage reaches the threshold voltage of the comparator faster. The state trajectory presented in Fig. 6b shows that the circuit reaches a limit cycle. This happens very quickly compared to the circuit in [13]. Additionally, there is only very small jitter in this state.

In the low-power configuration the supply voltage is reduced to 1.5 V, $I_{bias}$ is set to 20 nA, and $I_{syn}$ is set between 100 nA and 1 $\upmu$A. For a fair comparison of different circuit implementations and spike patterns, the energy consumption per spike is frequently reported, referring to the spike as the smallest computational unit. Thus, the total current consumed by the circuit is integrated over the time of a single spike event and multiplied by the supply voltage. Under low-power operation, an energy consumption of 8–10 pJ per spike is simulated, which is in good agreement with what was achieved in [13].

Fig. 7

a $V_{mem}$ of the coupled neurons during the settling process. b Change in amplitude during the settling process. c Change in frequency during the settling process

2.2.2 Coupled Izhikevich-Model Based Low-Power Neuron

Two neurons are connected as shown in Fig. 4 to form a simple network. The current output $I_{axon}$ of neuron 1 connects to the current input $I_{syn}$ of neuron 2. The neurons are both operated in their low-power configurations, and the variables are set to $V_c =$130 mV, $V_d = 0$ V, $V_{th} =$540 mV, $I_{syn1} =$300 nA, and $I_{syn2} =$200 nA, where $I_{syn1}$ is applied to neuron 1 and $I_{syn2}$ to neuron 2. Thereby, both neurons generate the same spike pattern with different spike frequencies. The state of each neuron can be seen by assessing the different $V_{mem}$ outputs. The membrane voltages of neurons 1 and 2 are shown in Fig. 7a. After about 30 $\upmu$s, both neurons become locked and oscillate in synchrony. Figure 7b shows the change in amplitude, while Fig. 7c shows the change in the frequencies during the settling process. As expected, the initially faster neuron 1 remains unchanged, while neuron 2 changes its output to the same frequency as neuron 1. Thereby, the frequency first rises close to that of neuron 1, but then drops to 830 Hz before beginning to rise again. Additionally, the amplitude changes during the settling process. While the frequency of neuron 2 is slow—between 15 and 20 $\upmu$s—its amplitude is high, as more energy can be stored in $C_{mem}$. After both neurons settle to the same frequency, the amplitude of neuron 1 remains larger than that of neuron 2, as $I_{syn}$ differs between the neurons.

2.3 Measurement Results

The circuit was implemented and fabricated in 0.35 $\mu$m CMOS technology. Figure 8a shows a microphotograph of the die, where the rectangular shape indicates the location of the circuit. Figure 8b presents the related layout of this neuron, with an area of 33.4 $\times$ 55.9 $\upmu \textrm{m}^2$

Fig. 8

a Photograph of the fabricated die with marked dimensions of $33.4\,\times \, 55.9\, \upmu {\textrm{m}}^2$. b Layout of the presented neuron

Fig. 9

Measurement of different output pattern during a constant current input: a Slow simple spikes with a frequency of 900 Hz. b Fast simple spikes with a frequency of 30 kHz. c Periodical bursting behavior. d Aperiodic bursting behavior

For all measurements, the neuron was operated in low-power mode (supply voltage 1.5 V). All variable inputs were generated externally, using a laptop-controlled multifunctional input/output card. The output waveforms measured during a constant $I_{syn}$ input are presented in Fig. 9a–d. Figure 9a shows simple slow spikes ($V_c = V_d =$ 0 V, $V_{th} =$ 0.9 V, $I_{bias} =$ 20 nA, $I_{syn} =$ 200 nA). The frequency was about 900 Hz and, therefore, close to biological spike frequencies. Additionally, the spike shape was in good agreement with that of biological spikes. By changing the input voltage to $V_c =$ 100 mV and setting $V_{th} =$ 0.96 V, the neuron generated fast spikes, as presented in Fig. 9b. The spikes were fired with a frequency of 30 kHz. Due to the high frequency, the neuron was not able to fully discharge to $V_c$. Figure 9c shows periodic bursting ($V_c =$ 0.1 V, $V_d =$ 0.1 V, $V_{th} =$ 0.85 V, and $I_{syn} = 1\,\upmu$A). The neuron generated three very fast spikes and then discharged to $V_c$ periodically. The opposite -aperiodic bursting- is presented in Fig. 9d. The setup was the same as that for periodic bursting, except that $V_d$ was set to $V_{dd}$.

3 Ultra-Low-Frequency Hybrid CMOS-Memristive Silicon Neuron

Synchronization and memory of spiking neurons are vastly accepted and explained as underlying mechanisms of neuronal signal processing, such as recognition, perception and awareness, in brains of living creatures [17, 34,35,36,37,38,39,40]. In neuromorphic engineering, to emulate neuron activities, a variety of ways to transfer these biological mechanisms to electronic circuits are realized [10, 41,42,43]. Researches in recent years demonstrate that the neuromorphic system built by hybrid CMOS-memristive silicon neurons consisting of relaxation oscillators and memristive devices is a promising candidate for neuromorphic computing, since they allow one to emulate neuronal synchronization and synaptic functionalities in a detailed way with energy efficiency and a high packing density [11, 19, 22, 44,45,46,47,48,49]. One example is given in Fig. 10. It shows an experiment based on two discrete self-sustained relaxation oscillators coupled with an Ag-doped-$TiO_{2-x}$-Al memristive device to emulate basal coupling and an autonomous synchronization scheme for neuronal ensembles [22]. More specifically, two relaxation oscillators with intrinsic oscillation frequencies of 543 Hz and 414 Hz, respectively, are pulse-coupled through a resistor–capacitor network comprising a single memristive device. A synchronization process is observed as anticipated, i.e. the left ‘slow’ oscillator eventually follows the right ‘fast’ oscillator.

Fig. 10

Block diagram of discrete setup of two mutually coupled self-sustained relaxation oscillators with intrinsic frequencies of $f_{1N}=$ 414 Hz and $f_{2N}=$ 543 Hz, respectively. M(x,t) (middle) is a memristive resistor network, where x is the state variable of the memory process

The experiment successfully demonstrates that, with the memristive pulse-coupling of two relaxation oscillators, two relevant dynamic aspects of biology (memory and synchronization) can be transferred to electronic circuits in a small-scale compact system. In the future, the construction of a large scale pulse-coupled memristive oscillator network is expected to facilitate the emulation of higher cognitive functions and perceptual processes. Very large-scale integrated (VLSI) circuits have been shown as a feasible solution owing to their variety of advantages: highly complex neural network connections, highly efficient parallel computing, and real-time processing [7, 12, 19, 20, 50]. However, to directly transfer the design in the experiment given in Fig. 10 to neuromorphic VLSI network, the structure of the oscillator from the experiment has two constraints: at first, the oscillator is based on a programmable unijunction transistor (PUT; 2N2067), discrete resistors and capacitors. It is necessary to find a new circuit structure to replace PUT by using basic elements (i.e. CMOS transistors) in VLSI; In addition, the frequency of oscillation and static power consumption of the oscillator from the above experiment are limited by discrete resistors and capacitors. Especially for the biological frequency applications—at which the frequency should be less than one hundred hertz and below—it is infeasible to realize ultra-large resistors or capacitors in the custom on-chip design to satisfy the ultra-low frequency requirements. As a result, a new integrated relaxation-type oscillator is highly demanded.

In this section, we present a study of a hybrid CMOS-memristive silicon neuron consisting of a monolithic ultra low frequency relaxation oscillator which is designed and realized in CMOS technology and the analog memristive device introduced before. The circuit design, simulation and measurement results of the proposed relaxation oscillator are introduced at first. Then, the principle of pulse coupled oscillators is explained and examined by experiments of resistive coupling system. At the end, an autonomous synchronization process is demonstrated by experiments of memristive coupling system. This hybrid SiN is strongly biologically oriented and paves the way for large neuromorphic VLSI system.

3.1 Ultra-Low-Frequency Relaxation Oscillator

This section introduces the design, the oscillation mechanism, the simulation and measurement results of the relaxation-type oscillator which is fabricated with 130 nm IHP technology. The supply voltage 3.3 V is adopted to meet the state transition requirements of the analog memristive devices used to realize a coupling between on-chip oscillators. To make oscillator self sustained, negative differential resistance (NDR) circuit is adopted here.

3.1.1 Circuit Design and Oscillation Mechanism

The architecture of the proposed integrated self-sustained relaxation oscillator [9] is shown in Fig. 11. To overcome the obstacle of integration of ultra-large resistors or capacitors induced from the structure of oscillator in previous experiment, it adopts a pA-scale charging current to realize ms-scale charging time and biological oscillation frequency. The output current of a MOS-only current reference $I_{REF}$ (1 nA) [51] is mirrored through two pairs of NMOS current mirrors (N3 and N4, N3 and N5) and two pairs of PMOS current mirrors (P1 and P3, P2 and P4), in order to supply the charging current $I_{c}$ (250 pA) and output stage current $I_{o}$ (3 nA), respectively. To improve the accuracies of the current mirrors, cascode current mirror structures are adopted (but not shown here). Furthermore, the output stage current $I_{o}$ flows into configurable diode-connected stacked NMOS transistors (N6, N7, N8, and N9) to generate the output voltage $V_g$, which adjusts the threshold voltage $V_{thG}$ of negative differential resistance (NDR) circuit [52]. It avoids the difficult tradeoff of size of resistors or power consumption in the structure designed by the experiment shown in Fig. 10. The NDR circuit consists of PMOS transistor P5, a pair of NMOS transistors N1 and N2, and the configurable diode-connected stacked NMOS transistors.

The oscillation frequency is tunable by applying a digital configuration to the programmable capacitor $C_{c}$. It consists of binary weighted capacitance with the size of 1, 2, 4, 8 and 16 $C_0$. By considering chip area and reducing parasitic effects, the unit capacitance $C_0$ is designed to be 0.94 pF with the size of 25 $\upmu$m $\times$ 25$\upmu$m. This enables a programming range of $C_{c}$ from 0.94 pF to 30 pF. In addition, the tunable transistor N2 can adjust the NDR region. Another configurable module is the diode-connected stacked NMOS transistors which control the output voltage $V_g$ in the range of 2.6–2.8 V in order to ensure the state transition of the memristive device.

Fig. 11

Schematic of proposed oscillator, which is designed to represent the discrete oscillator in Fig. 10. The output voltage port $V_g$ is an interface to outside

Fig. 12

Timing diagram of oscillator output voltage $V_g$ and internal voltage potential $V_a$ and $V_c$; $t_c$ indicates the time required for charging $C_{c}$, while $t_d$ is its discharge time ($C_{c}= 4.7$ pF, f $=$ 19.7 Hz)

A timing diagram for the voltages $V_a$, $V_g$, and $V_c$ under the conditions of f= 19.7 Hz and $C_{c}$= 4.7 pF is illustrated in Fig. 12. When the circuit is powered on, the potential of $V_a$ and $V_g$ is zero and P5 is turned off. The constant charging current $I_{c}$ from P3 begins to flow into the programmable capacitor $C_{c}$ and potential $V_a$ rises linearly to $V_{aH}$. At the same time, the output stage current $I_{o}$ from P4 flows into the diode-connected stacked NMOS transistors and generates output voltage $V_g$. To make sure P5 keeps turned off before $V_a$ arrives at $V_{aH}$, output voltage $V_g$ should reach $V_{gH}$ earlier than $V_a$ reaches $V_{aH}$. The time of $V_g$ from 0 to $V_{gH}$ depends on the output stage current $I_{o}$ and $C_{L}$ which is parasitic capacitance at $V_g$ port. Considering parasitic capacitance from I/O Pad and additional test point on PCB, to make sure $V_g$ arriving $V_{gH}$ earlier, $I_{o}$ (3nA) is chosen 12 times bigger than $I_{c}$ (250pA) here. When the voltage $V_a$ reaches the turn-on voltage $V_{thG}$ (i.e., $V_{aH}$ in Fig. 12) of the NDR, the transistor P5 turns on, $C_{c}$ begins discharging and $I_{c}$ current flows into N1. It generates potential $V_c$ and turns on N2. In the design, the size of N2 is large enough to swallow all the current from $I_{o}$ when its $V_{gs}$ equals $V_c$ with small drain-source voltage $V_{ds}$ (i.e. $V_g$). The drain-source voltage of N2 is then immediately pulled down (i.e., output voltage $V_{g}$ is pulled down to be $V_{gL}$). This speeds up the discharging of $C_{c}$ and generates spike signal $V_c$.

The capacitor $C_{c}$ discharges until the voltage $V_a$ reaches $V_{aL}$ and the gate voltage $V_c$ of N1 and N2 also decreases, which reduces the drain current of N2. $I_{o}$ flows back to the diode-connected stacked NMOS transistors. Thus, $V_{g}$ becomes high and switches off P5. Then, the next charging phase starts. From simulated waveforms in Fig. 12, we can see that $V_a$ presents a saw-tooth-type signal, $V_g$ is a rectangular pulse-type signal, and $V_c$ is a neuron spiking-type signal.

Fig. 13

$V_a$–$I_{c}$ DC characteristic for different values of N2, showing the negative differential resistance area. For increasing size (W/L) of N2, the NDR area decreases

3.1.2 Negative Differential Resistance Regime

The $V_a$–$I_{c}$ DC characteristics of the proposed oscillator, with different sizes of N2, are shown in Fig. 13. The width of the NDR region changes with the size of N2; for example, with an aspect ratio of 240 for N2, when $V_a$ is less than $V_{thG}$, P5 is cut off. When $V_a=V_{thG}$, then P5 turns on, and the oscillator enters an NDR region in which the voltage $V_a$ decreases as $I_{c}$ increases.

The negative-resistance region extends until the valley point is reached, where the current is defined by $I_{sat}$. Beyond the valley point, further increases in $V_a$ produce increases in $I_{c}$. This region is the so-called saturation region, which should be avoided in the circuit designed for oscillation-based applications. An appropriate size for N2 is chosen, in order to make sure that the charging current $I_{c}$ should be greater than $I_{thG}$, thus guaranteeing the turning on of the oscillator and also not exceeding $I_{sat}$. For this purpose, to make sure $I_{c}$ (i.e. 250pA) is in the NDR region, the W/L of N2 should be less than 240. In the design, the W/L of N2 is decided 220 and a 4-bit transistor bank is added to the oscillator to calibrate the size of N2, considering the deviation of the process. The unit size of the transistor bank is $W/L = 1 \upmu$m/1 $\upmu$m.

Fig. 14

The photograph of die with core area of 0.05546 ${\textrm{mm}}^2$. The three main parts of this work, i.e. current reference (top), oscillator core (middle) and programmable capacitor bank (bottom) are shown in it

Fig. 15

Post-layout simulation and measurements of oscillation frequency and calculated energy per spike versus programmable capacitance

Fig. 16

Measured current versus IC number

3.2 Simulation and Measurement Results of Relaxation Oscillator

As shown in Fig. 14, the total area of chip layout is 0.05546 mm$^2$, including the current reference, the programmable capacitor bank, and the core of the oscillator. The simulated static power consumption of a single oscillator is 24 nW excluding the power consumption of the current reference, as the current reference will be shared with other oscillators in future neuromorphic networks. The power consumption of the current reference is 255 $\upmu$W. Figure 15 summarizes the performance of the proposed oscillator for different configurations, based on post-layout simulations and measurement results. The energy per spike is defined as the power consumption of a single oscillator core (active power) integrated over the period of spiking. Post-layout simulations indicate that it varies from 0.8 nJ/spike to 7.12 nJ/spike, and the oscillation frequency is tunable from 3.15 to 81.30 Hz by programming the on-chip capacitors. A total of 10 chips were bonded and tested. The output of the current reference was tested, as shown in Fig. 16. Seven chips were within the acceptable limits for low-frequency neuromorphic applications. Compared with the simulation results, the oscillation frequency of chip 9 in the test results was programmable from 3.58 to 13.01 Hz. The oscillation frequency was relatively lower when the charging capacitor was small (from 1 to 15 $C_0$), due to the parasitic capacitor at the input of oscillators from PCB and chip package being comparably large, with respect to the small charging capacitor $C_{c}$. The parasitic effects can be reduced in the integrated on-chip design for neuromorphic network applications.

3.3 Experiments of Coupling Systems

In a previous work the coupling of two relaxation-type oscillators built by discrete Programmable Unijunction Transistors (PUTs) and by a digital type of memrisitive device was investigated by Ignatov et al. In this section two types of coupling experiments between both on-chip oscillators were discussed. The first experiment uses a RC network as intermedia, which aims to check the functionality of synchronization of the two oscillators. In the second experiment, the two oscillators are coupled by an analog memristive device. Due to its memristive characteristic, the test allows us to further observe the autonomous transition process from unsynchronization to synchronization. This transition phase mimic the synchronization process of neuron signals [45, 46].

3.3.1 Coupling System with RC Network

The coupling system of two oscillators via a RC network is shown in Fig. 17. Two capacitors $C_1 = C_2$ within the coupling network forms the DC potential decoupling between both oscillators. The coupling network which consists of $R_C$ serials connected with $C_1$ and $C_2$ is a passive high-pass filter with a cut-off frequency $f_c$ determined by the following equation [22]:

$$\begin{aligned} f_{c}=\frac{1}{\pi {R_c}{C_1}} \end{aligned}$$

(3)

In the uncoupled state (i.e. without RC network), the intrinsic frequency $f_{1N}$ of oscillator 1 differs from the intrinsic frequency $f_{2N}$ of oscillator 2. Hereby, $f_{1N}<f_{2N}$ and the frequency difference is $\Delta f_N = f_{2N}- f_{1N}$.

Fig. 17

Coupling test with RC network

Fig. 18

Waveform of coupled process in synchronous phase

After the coupling system with RC network is powered on and at $t_0$, the gate voltage $V_{g1}$ and gate voltage $V_{g2}$ are charged to be $V_{gH}$ (the blue and green curve in the lower graph in Fig. 18). At this moment, there is no current flowing through RC network due to $V_{g1}= V_{g2}$. The voltage $V_{a1}$ from oscillator 1 (green dotted curve in the upper graph in Fig. 18) and $V_{a2}$ from oscillator 2 (blue dotted curve in the upper graph in Fig. 18) rise linearly towards $V_{aH}$ as the capacitors $C_{c1}$ and $C_{c2}$ are charged with constant charging currents $I_{c1}$ and $I_{c2}$, respectively, as shown in Fig. 18. Induced by the difference in the charging capacitor selected for $C_{c1}$ and $C_{c2}$, $V_{a2}$ reaches the turn-on voltage $V_{aH}$ ahead of $V_{a1}$, as $C_{c2}<C_{c1}$, under the condition of the same charging current (i.e., $I_{c1} = I_{c2} = I_{c}$,). At the same time, $V_{g2}$ falls at the moment $t_{1}$ ahead of $V_{g1}$. Therefore, there is a low-resistance signal path existing from $V_{g2}$ to ground and the decreasing of $V_{g2}$ cause voltage difference between $V_{g1}$ and $V_{g2}$. There is current flowing from $V_{g1}$ to $V_{g2}$ through RC network. When $V_{g2}$ falls down to be $V_{gL}$ ($V_{gL}\approx 0$) at $t_{1d}$, so $V_{g1}$ is

$$\begin{aligned} V_{gp1}={R_c} \cdot {I_{RC}} \end{aligned}$$

(4)

where, $I_{RC}$ is the current flowing through the coupled resistor at the moment of $t_{1d}$. $I_{c}$ comes mainly from the $I_{O1}$ and the discharging current from parasitic capacitance at the node of $V_{g1}$. To ensure that oscillator 1 follows the faster oscillator 2 (i.e., that the two oscillators synchronize), $V_{aH1}$ (the value of $\text {V}_\textrm{a1}$ at the moment of $t_{1d}$) should be a threshold voltage $V_{on}$ bigger than $V_{gp1}$ in order to turn on pmos transistor $P5\_1$ in oscillator 1 and lead $C_{c1}$ into the self discharging state. Then, $\text {V}_\textrm{g1}$ is pulled down to be $V_{gL}$ due to the turn-on of $P5\_1$ oscillator 1. After that, the two oscillators independently go into self-sustained charging state again, and the synchronization process is repeated. As a result, the two self-sustained oscillators synchronize with the same frequency of $f_{2N}$.

Therefore, $R_c$ needed for synchronization of RC coupled two oscillators can be calculated as

$$\begin{aligned} V_{gp1}={R_c} \cdot {I_{RC}} \le (V_{aH1}-V_{on}) \end{aligned}$$

(5)

Therefore,

$$\begin{aligned} Rc\le \frac{(V_{aH1}-V_{on})}{I_{RC}} , \end{aligned}$$

(6)

where $V_{aH1}$ is determined by

$$I_c\cdot t_{1d}=V_{aH1}\cdot {C_{c1}}$$

hence,

$$\begin{aligned} V_{aH1}=\frac{I_{c}\cdot {t_{1d}}}{C_{c1}}\approx \frac{I_c}{f_{2N}\cdot {C_{c1}}} \end{aligned}$$

(7)

where, $t_{1d}\approx T_2$. From $I_{c}\cdot T_{1N}=I_{c}/f_{1N}=V_{aH}\cdot {C_{c1}}$, where, $T_{1N}$ is the intrinsic oscillation period of oscillator 1, we can have

$$\begin{aligned} C_{c1}=\frac{I_c}{f_{1N}\cdot V_{aH}}. \end{aligned}$$

(8)

Substituting Eq. 8 into Eq. 7, we obtain

$$\begin{aligned} V_{aH1}=V_{aH}\cdot f_{1N}/f_{2N}. \end{aligned}$$

(9)

Therefore,

$$\begin{aligned} R_c\le \frac{(V_{aH}\cdot {\frac{f_{1N}}{f_{2N}}}-V_{on})}{I_{RC}} =\frac{[V_{aH}\cdot {(1-\frac{\Delta f_N}{f_{2N}})}-V_{on}]}{I_{RC}} \end{aligned}$$

(10)

From Eq. 10, it can be easily concluded that, when the frequency ratio of the two oscillators $f_{1N}/f_{2N}$ gets smaller (i.e., $\Delta f_N/f_{2N}$ is larger), $R_c$ should be smaller, such that the two oscillators can synchronize. Therefore, we can obtain the following conclusions: The coupling resistance is proportional to the frequency ratio and inversely proportional to the frequency difference under the same value of $f_{2N}$.

3.3.2 Measurement Results of Resistive Coupling

To verify the above conclusions, five experiments have been executed and the corresponding results are summarized in Table 1. When the frequency ratio ($f_{1N}/f_{2N}$) varies from 0.207 to 0.889, relative frequency difference ($\Delta f_N/f_{2N}$) identically decreasing from 0.793 to 0.111, the maximal coupling resistance $R_c$ required for synchronization increases from 564 k$\Omega$ to 251 M$\Omega$. The variations of $R_c$ matches our expectation, since Eq. 10 is derived by ignoring parasitic factors.

Table 1 Measurement results of maximal coupled resistor for synchronization

3.3.3 Coupling System with Analog Memristive Device

In this section, an analog memristive device was chosen as coupling element in the coupling system. Here, devices showing similar electrical characteristics as the DBMDs presented above are used. These devices incorporate HfO$_2$ instead of Nb$_x$O$_y$. While a typical |J|-V curve is shown below, a detailed analysis of the device performance will be published elsewhere. The circuit demonstrates an autonomous phase-locking and frequency synchronization process due to resistance changes in an analog memristive device.

To test the performance of the memristive coupling system, two oscillators were assembled with an analog memristive device in a single test PCB as shown in Fig. 19a. The structure of test PCB is depicted in Fig. 19b. Oscillator 1 with lower oscillation frequency $f_{1N}$ and oscillator 2 with higher oscillation frequency $f_{2N}$ are connected to the top and bottom electrode (BE) of the memristive device, respectively, through series connected DC decoupled capacitor $C_1$ and $C_2$. The gate voltages ($V_{g1}$ and $V_{g2}$) of both oscillators were recorded using a PicoScope 3000 Series mixed-signal oscilloscope after buffered AD820 amplifiers with high input impedance low input bias current to reduce the loss of signal current.

Fig. 19

Test system for the memristive coupling

The layout of a $5\,\times \,5$ mm$^2$ chip containing analog memristive device is shown in Fig. 20a [48]. It contains 25 sub-cells arranged in a 5x5 array. Except for two sub-cells (Tcells, gray boxes) containing test structures, the other 23 sub-cells can be used in the experiments. Six devices with area sizes increasing from 100 $\upmu$m$^2$ to 2500 $\upmu$m$^2$ [numbered with “1” to “6” in Fig. 20b] are located in a single sub-cell [23, 48]. To connect the memristive devices with the two oscillators, the chip was glued onto a JLCC 44 chip carrier shown in Fig. 20c and some devices with an area of 400 $\upmu$m$^2$ (size 3) are wire-bonded for experiments. A typical $|\textit{J}|-\textit{V}$ hysteresis of the HfO$_2$-based devices is given in Fig. 20d. The JLCC 44 chip carrier is then placed into a PLCC 44 socket on the test PCB to connect two oscillators, as shown in Fig. 19 for experiments.

Fig. 20

a The layout of a $5\,\times \,5$ mm$^2$ chip containing analog memristive device. b Layout of a sub-cell. c Chip glued on JLCC 44 chip carrier and wire-bonded. d Typical $|\textit{J}|-\textit{V}$ hysteresis of the HfO$_2$-based devices

3.3.4 Measurement Results of Memristive Coupling

As discussed above, the bigger the frequency ratio ($f_{1N}$/$f_{2N}$) is, the larger coupling resistance is needed for synchronization. Hence, the two oscillators can immediately synchronize without a small coupling resistance, when the frequency ratio ($f_{1N}$/$f_{2N}$) is big. The initial resistance value of the memristive device is small enough for their mutual coupling and synchronization. However, for small frequency ratios ($f_{1N}$/$f_{2N}$), the two oscillators require a small coupling resistance for synchronization. When the initial resistance value is not small enough, as long as the resistance of the memristive device decreases gradually to $R_c$ defined by the Eq. 10, the two oscillators can synchronize.

Fig. 21

Synchronization process of memristive coupling: Upper graph: $f_1$ and $f_2$ were the transient frequency variations of both self-sustained oscillators. The continuous resistance change of the memristive device caused a desynchronous phase (DS phase), an intermediate phase (I phase), and a synchronous phase (S phase). Middle graph: Transient waveform of $V_{g1}$ and $V_{g2}$ signal. Lower graph: Zoomed in transient waveforms of $V_{g1}$ and $V_{g2}$ signal in desynchronous phase (in yellow background) and synchronous state (in green background), respectively, corresponding time slots in the whole synchronization process are marked by the yellow bar and the green bar in the middle graph

A representative synchronization process of two oscillators coupled with a memristive device is shown in Fig. 21. A device with area size of 400 $\upmu$m$^2$ numbered with “3” in Fig. 20b was used in this experiment. By considering the switch dynamic and retention characteristics of the devices, the intrinsic frequencies of the two oscillators 1 and 2 were $f_{1N}$ = 2.24 Hz and $f_{2N} =$ 5.90 Hz, respectively. Initially, due to the high resistance of the memristive device, the two self-sustained oscillators can not synchronize (i.e. they are in the desynchronous state (DS)). In each discharging period of oscillator 2, the $V_{g1}$ was pulled down to be $V_{gp1}=R_M\cdot I_{M}$ when $V_{g2}$ reaches $V_{gL}$ (about 0 V), where $R_M$ was the resistance of memristive device, $I_{M}$ was the current flowing through the memristive device during each discharging from oscillator 2. As a consequence, every discharge of the faster oscillator (in this case, oscillator 2) would trigger a discharge of $V_{g1}$ to be $V_{gp1}$. During this period, voltage $V_{gp1}$ exerted on the top electrode of the memristive device and the bottom electrode of the memristive was $V_{gL}$. The voltage difference happened at each discharge period of the faster oscillator and gradually changed the resistance value of the memristive device. As long as the resistance of coupled memristive device decreased less than $R_c$ calculated by Eq. 10 and then $V_{gp1}\le V_{aH1}-V_{on}$, oscillator 1 can follow the rhythm of oscillator 2. In this experiment, after about 125 s, the two oscillators synchronized due to the state transition of the memristive device from a high to low resistance state.

In upper graph of Fig. 21, the transient frequency variations of oscillator 1 and 2 are given. In the desynchronous state phase (DS phase), the frequency of oscillator 1 was equal to its intrinsic frequency ($f_1=f_{1N}=$ 2.24 Hz which is calculated by 0.3 V threshold voltage). For the first 66 s, the oscillator network remained desynchronous until the intermediate phase (I phase) was reached. The intermediate state is characterized by the fact that the frequency of oscillator 1 jumps between the frequency $f_{2N}$ and $f_{1N}$ at irregular time intervals. The intermediate phase ended till synchronous phase (S phase) was reached at 125 s when phase and frequency of both oscillators synchronized. In the synchronous state phase, each $V_{g2}$ pulse of oscillator 2 triggered oscillator 1 to discharging fully and generated a full gate pulse (i.e. amplitude of $V_{g1}$ is from $V_{gH}$ to ground). Oscillator 2 oscillated at its intrinsic frequency $f_{2N}$ over the entire period of time.

4 Conclusion

In this work, two types of biologically plausible silicon neurons have been realised in integrated circuit technology. The first low-power spiking silicon neuron is inspired by mathematical Izhikevich model and built by relaxation oscillator implemented in 0.35 $\upmu$m CMOS technology. It achieves an energy consumption of 8$\sim$10 pJ per spike under low-power operation mode and synchronous processes of two coupled neurons is demonstrated in the simulation results. The measurement results show that it can mimic a broad range of physiologically observable spike patterns. This area and energy efficient fully CMOS silicon neuron could be used as a universal neuron circuit integrated in large scale analogue VLSI systems. The second ultra-low-frequency hybrid SiN achieves the biological spiking frequencies below 100 Hz with energy consumption in the range of 0.8–7.12 nJ/spike. To emulate a basic neural network, two integrated ultra-low-frequency relaxation oscillators coupled by an external analog memristive device were proposed. The autonomous neuronal synchronization processes of this basic neural network are presented and analysed. The realisation of biologically plausible oscillators in CMOS technology and an analog memristive device fabricated on silicon wafer pave the way towards large scale hybrid analog integrated neural network realisation in future.