A Bit-Encoding Based New Data Structure for Time and Memory Efficient Handling of Spike Times in an Electrophysiological Setup

Brain Machine Interfaces (BMI) and Brain Computer Interfaces (BCI) have developed substantially during the past decades. Within the field of neural prosthetics, for instance, the general feasibility of real time control of robotic arms using multi-electrode-array recordings of cortical neural activity has been demonstrated (Wessberg et al. 2000) and, more recently, a robotic device allowing advanced arm and hand movements has been successfully implemented in tetraplegic subjects (Velliste et al. 2008; Collinger et al. 2013; Hochberg et al. 2012; Gilja et al. 2012). This development has depended partly on the identification of important principles of motor control, revealed by neurophysiological investigations of neural activity in awake, behaving animals (Monfils et al. 2005), and partly on advances within the field of robotics (Velliste et al. 2008). However, a functional interaction between the brain and robotic devices or computers also requires sophisticated neuroinformatics to ensure an efficient organization and analysis of neural data.

The demands on hardware and software used in the context of BMI/BCI are already high, as recent studies have used recordings of up to 1792 channels for a single subject (Schwarz et al. 2014). However, demands can be foreseen to further increase in the near future, as BMI and BCI reach new levels of sophistication, for example by including real time feedback protocols. Furthermore, in January 2016 DARPA (U.S. Defense Advanced Research Project Agency) announced the Neural Engineering System Design (NESD) program (Miranda et al. 2015), in which it aims for fully implantable devices able to record from up to one million neurons and stimulate 100 000 neurons, supporting a bidirectional BMI. Although the hardware part to perform this feat is a challenge in itself, the neuroinformatic requirements are immense, surpassing those previously implemented in currently available neuroinformatics systems. Unless adequately addressed, this may potentially become the rate limiting factor for the development of this field of neurotechnology in the near future.

In a previous paper (Ljungquist et al. 2011), we showed how a traditional Relational DataBase Managements System (RDBMS), used for storing neural data, might be combined with a data transfer mechanism. RDBMSs have also been used by other groups to structure and store electrophysiological data (Rautenberg et al. 2011). However, as RDBMSs are not built for large data sets, they scale poorly with binary data and multidimensional time series data such as electrophysiological and neuroimaging data. Hierarchical Data Format (HDF5) https://www.hdfgroup.org, by contrast, is a data format that potentially fulfills the requirements outlined above and has received increasing attention within the biomedical community for storing data (Dougherty et al. 2009; Mouček et al. 2014; Chintaluri et al. 2014). It supports a large variety of data types, and is designed for flexible, efficient and parallel I/O also for complex data, including high dimensional data. Here we take advantage of the HDF5 file format support for high parallel throughput and introduce bit-encoding to allow compression of large amounts of data. Although progress has been made recently (Rossant et al. 2016), spike sorting still remains a challenge. In this work, we assume that a correct online spike sorting and tracking is performed by each recording system. We also assume that the spike sorting results in single units being identified.

Current initiatives to build open source hardware as well as software for electrophysiological recording and analysis, are using both HDF5 formats as well as custom formats to store data, for example Open Ephys http://www.open-ephys.org. However, in order to store data for transfer of data between data acquisition system and computer, Open Ephys transfers the raw signal over USB (Universal serial bus) link, although work is under development to transfer data over PCI (Peripheral Component Interconnect) express. PCI Express has the advantages of lower latency and impressive transfer speeds. Compared to Ethernet (Metcalfe and Boggs 1976), which we are using for transferring data in the present study, PCI Express has slightly lower latencies, but is less developed and proven as a technique; therefore we have chosen Ethernet. Future development may yield PCI Express as a viable option also for our architecture, especially since it could be used to communicate directly with GPUs (Graphics Processing Unit) and FPGAs (Field Programmable Gate Array) (Bittner and Ruf 2013). Commercial acquisition systems such as Neuralynx and Plexon also have online transfer of data, but the data format for the online transfer is not open, although software exists for accessing and interacting with the data stream. For all current initiatives the data stream is limited to the number of channels in a single system, typically 512 or 1024 channels.

The aim of this study was to develop a system architecture and data encoding that enables handling of electrophysiological data from very large numbers of neurons (1 million or more) in real time (< 25 ms) over long periods of time.

As intracranial electrode technology develops, an increasing number of neurons may be recorded from. The ability to analyze and process the data in real time from this larger set of neurons must be considered a fundamental feature of future Brain Machine Interfaces. In this study, we develop and evaluate a scalable software architecture, with a defined interchange data format for mainly spike data which enables storage and analysis from at least a million of neurons simultaneously in real time.

The system architecture, including a bit-encoding data format and multi-threading storage software, has been tested for a maximum of 10 concurrent acquisition systems (acquisition systems recording from a specific set of neurons in the subject), each handling a subset of the recorded neurons, and the data rate is well over the required theoretical limit of 3.2 million neurons spiking each with 1000 Hz; the mean was 6 times higher when streaming over LAN and 4 higher when streaming from a local simulator program on the storage server. The system however has to be supported by databases which periodically record waveform data and verifies cell identity over time, as well as recordings of the narrow band signal as shown in Fig. 1. Regarding time to feedback, a feedback within 25 ms could be achieved. Increasing the buffer will lead to higher latencies, but having a shorter buffer will lead to inefficient transfer of data, as many small packets take longer time than a corresponding larger packet. This performance was reached on quite modest hardware and could probably be increased further using for example RAIDed hard drives and/or cluster of servers with each having a separate network connection. Furthermore, the proposed architecture allows for both online feedback back to the subject as well as online analysis of data. The present solution has been shown to be highly scalable, seemingly limited by only the transfer rate across the network, both when scaling up number of neurons and acquisition systems. The bit-encoding data format limits firing rate to 1 kHz maximal firing rate with a bin size of 1 ms. However, the bin size may be chosen smaller if desired to capture finer resolution, but the number of neurons that are able to be recorded will be lesser. For spikes from a single neuron, this is well below the minimum refractory period for most neurons. However, if data originates from unsorted MUA (Multi Unit Activity), our data format will not be able to capture multiple spikes within the bintime. We expect, however, that development of electrodes will result in improved signal to noise ratio and more precise and stable identification of single units, and some promising electrode designs already exist (Agorelius et al. 2015).

In databases of scientific data there has previously typically been a trade off between organization of data and performance in terms of data write and read rates for larger data sets. Relational data bases, at the very end of the spectrum with regard to organizational capabilities with extensive support for complex relationships between data, have been used by many research groups for storing scientific data, also recently for storing larger files such as images, movies and also electrophysiological data (Ljungquist et al. 2011; Ljungquist et al. 2016). However, if storing data values as single data entries in the data base, performance drops quickly, as INSERT statements in relational databases are costly (Schwartz et al. 2012). One way to go around this is to store item as so called BLOBs, binary large objects. However, performance drops with data sizes over a couple of MegaBytes in a BLOB (Stancu-Mara et al. 2014). It is also possible to DROP indexes during insertion, in order to increase performance slightly, but this would also result in problems during simultaneous INSERT and readout of data and during repeated INSERT statements. Relational databases in data intense real time applications is thus not feasible, although it might be fine for offline applications. In contrast, acquisition system developers have recently developed data formats which are prioritizing performance. For example the recent PL2 format by Plexon (http://www.plexon.com/pl2-file-system-overview) is capable of impressive data read and write speed. In Memory Data Grids (IMDG) is another technique which recently has received increasing attention in some real time intensive applications in other fields, such as finance (e.g. Hazelcast https://hazelcast.com and Grid-gain https://www.gridgain.com). These databases, while accepting impressive throughput in terms of transactions per second, however do not handle persistence and the requirement of being able to correlate the data to previously recorded data. Our suggested system architecture allows for multi-trial and multi-subject integration or cross-system integration within the same data source, combining the real time performance ability of instantaneous persistence together with capabilities of correlating the recently recorded data to a previously recorded pattern.

In previous studies of sensorimotor feedback loop learning (O’Doherty et al. 2011), bidirectional neural interfaces have been used in setups in which external world states decide which intracranial feedback to provide, as well as for studying and inducing spike-timing-dependent plasticity in corticospinal connections using an implanted chip for providing feedback (Nishimura et al. 2013). In the latter study, feedback within 25 ms was required, which is within the capabilities of our proposed system architecture, although with a small margin. If set too low, however, it affects data transfer performance. It furthermore enables trigging response trains also due to internal states, tracked and followed in real time by the database.

Our proposed architecture also allows for integration of distributed recording of many groups of channels, each handled by a separated processor or computer. In a neural interface with a large number of channels, this is required. Todays acquisition systems are in the maximum range of around 500 channels, which is around the limit of what a single computer can cope with. But distributing the computing power to possibly thousands of acquisition systems, which all are writing to the same data structure, allows for integration of these data sources in real time, which is a necessity for performing calculations concerning all of the channels of the neural interface. This allows for handling the massive amounts of data that may come from future hardware technology, which would allow for recording of many more neurons than today. When recording from these larger number of neurons, for performance reasons, it should be avoided to put preprocessing such as spike sorting and thresholding on the same processing device as the process handling the storage; thus we predict a distributed setup of smaller devices collaborating in a local network to store data into an integrated repository to be the future of neural interfaces.

In addition to integration, the suggested bit encoding results in reduced data size, for transfer and storage of data. If, for example, spikes would be encoded with timestamp and unit ID, which would occupy 32 bits each, they would need to be encoded with 64 bits per spike. Our suggested sampling rate of 1000 bit/s would then correspond to a maximum spiking rate per neuron of 1000/64 = 16 Hz. However, during a simultaneous discharge, e. g. so called neuronal avalanches, as seen in vitro, Beggs (2007) the neuronal firing rate may increase to many times this value, thus a system based on timestamp and unit ID for spike data would not be able to cope with the information. On the other hand, the bit format is currently more memory intensive as compared to representing spikes with unit ID and timestamp for firing rates below 16 Hz.

Another important benefit of the new bitencoded data format proposed in the present work is the capability for analyzing complex data patters during short periods of time. In closed loop experiments, latency must be kept minimal in real time interaction with data, as mentioned previously. This also includes analysis; in our test setup, we have implemented a basic pattern matching, which will return a similarity measure of the recorded pattern toward a desired pattern or previously recorded patterns. As the patterns, as well as the signals, are bit-encoded, these operations are very fast since whole arrays may be compared bit-wise simultaneously. In particular, it will enable binary neural network classifiers, which are deep neural networks constrained to operate on data and have weights that are binary (Courbariaux et al. 2016) to operate on the recorded data. This type of classifiers have recently attracted interest due to their significant lower computational and power cost in implementations, and have been shown to have both very high throughput (in the range of teraoperations per second) as well as very low latency (for most datasets in the microsecond range) in classifier implementations on specialized computational hardware such as FPGAs (Umuroglu et al. 2016; Fraser et al. 2017) in benchmarks of pattern recognition on established test data sets. In addition to impressive computational results, power consumption needed for calculations is comparably low (Umuroglu et al. 2016), which may result in future development of specialized analysis hardware in field conditions, e.g. carried or implanted electronics in the recording subject.

Further work involves implementation of the suggested architecture in a more extensive electrophysiological data analysis project, fully implementing also the suggested parts of the proposed architecture, for example the waveform storage. Also, the integration of multiple modalities, for example imaging, with electrophysiological data remains to be done. The work could be used together with existing methods for analysis of binned representations of massive parallel spike train data such as detection of synfire chain activity and sequences of synchronous events (Schrader et al. 2008; Torre et al. 2016).

Lastly, the suggested architecture enables storage of data from various subjects in real time in the same database, which opens up entirely new possibilities for analysis. Cross subject phenomena, for example electrophysiological recordings of social interactions between two subjects, could be investigated. This opens up for a range of new research that may be undertaken.

All source code is freely available in a public repository at GitHub: https://www.github.com/NRC-Lund/spikebit.git.