Multichannel Data Acquisition System for GEM Detectors

This paper presents the data acquisition system for GEM-detector (Sauli in Nucl Instrum Methods Phys Res Sect A Accel Spectrom Detect Assoc Equip 386(2–3): 531–534, 1997. https://doi.org/10.1016/S0168-9002(96)01172-2) based cameras and spectrometers (Chernyshova et al. in Fusion Eng Des, 2017. https://doi.org/10.1016/j.fusengdes.2017.03.107). The system is modular (Czarski et al. in Rev Sci Instrum 87(11): 11E336, 2016. https://doi.org/10.1063/1.4961559) and supports 1D and 2D GEM arrays with software-defined readout modes. Channel count can vary from 8 up to hundreds. The readout electronics consists of two units—radiation tolerant Analog Front End and rack-mount data processing unit. Data processing is split into two parts—real-time hardware, based on FPGA and software, based on embedded multicore CPUs and hardware accelerators. The FPGA subsystem together with PCIe interface and multi-core Xeon processors forms low latency, high-performance processing chain suitable for applications requiring feedback. The detectors in tokamaks operate in the high magnetic field, so the system was equipped with multipoint magnetic field measurement synchronized with detector readout. The system also includes custom HV supply, developed for triple GEM detectors operating in the high-rate mode. Dedicated protection and diagnostic subsystems were developed as well to ensure safe and reliable operation of the detector in harsh conditions. To support the operation of the detector in the high-temperature surrounding, the liquid cooling subsystem was developed (Wojenski et al. in J Instrum 11(11): C11035, 2016).


Introduction
Precise estimation of soft X-ray distribution of plasma in future research tokamak reactors will be a key factor in plasma parameter optimization. Two-dimensional tomography with single photon energy estimation will deliver new quality diagnostics of the tokamak plasma. Large plasma structure dimensions (i.e., ITER) require complex imaging system that must fulfill contrary requirements: large detection area, good spatial resolution, good energy resolution for X-ray photons and resistance to strong neutron and ionised particle flux. Another issue is the high level of electromagnetic noise. Proper processing and discrimination of such events caused by radiation and EMI from other diagnostics and reactor itself, is one of the most important challenges. Several data acquisition systems and GEM detectors were constructed to cope with such requirements [7]. This article describes recent achievement in this field-complex data acquisitions system that was installed at WEST [5]. In the last chapter, the next generation system based on MTCA [1] platform is presented.

Data Acquisition System Architecture
The architecture of the readout system is based on full modularity. Such approach gives great flexibility in system construction where both small and large, 1D and 2D imaging systems can be relatively quickly constructed using standardized building blocks. The only difference between 64 channel 1D linear array and 2048 channel 2D/ 3D readout is detector pixel board with detector backplane. These are relatively simple designs that can be designed and manufactured quickly. Further processing is realized using repeatable blocks that consist of Analog Front End (AFE) channels, ADC modules, FPGA backplane, Data Concentrator Board and computer subsystem. Figure 1 presents the block diagram of the system for WEST linear camera. The aim of detector's signals processing is an estimation of energy distribution for each channel. The system consists of:

Journal of Fusion Energy
All modules except for AFE and detector fit inside single 15U 19'' computer rack called main processing unit. The main processing unit is not affected by radiation. Temperature is not an issue neither because it operates stand-alone with the independent cooling system.

GEM Detector Practical Realisation
GEM detector assembly for WEST vertical camera is installed inside of the vertical port. Mechanical construction (Figs. 2, 3) was optimised to fit within the constrained area. Due to the lack of the airflow, the liquid cooling was utilised.
The detector stripboard is connected with the AFE backplane using FPC jumpers and shield (Figs. 4,5). That eliminates issues with the previous construction where both boards were connected using rigid Samtec connectors. It caused unwanted tension on the connectors caused by fixing of the detector assembly to the AFE cassette frame. Thanks to flex-rigid connectivity solution, such tension is also avoided during plugging and unplugging of the AFE boards. Moreover, it allows adjustment of the detector window relatively to the AFE cassette and pinhole. AFE boards are designed as Micro-TCA form-factor. Eight such boards can be installed in the backplane connectors (Fig. 4).

AFE Modules for GEM Signals
MicroTCA form factor has several benefits: • EMC compliance, fully shielded panel with EMC seal.
• The module can be plugged and unplugged using single hand. It enables replacement of the AFE module when the detector is installed deeply inside the port. • The reliable locking mechanism that is released when the handle is pulled. • The signalization of correct locking in the slot using miniature switch.   Each AFE module (Figs. 6, 7) consists of 16 charge amplifiers followed by zero-deadtime switched integrator. Optionally, the signal can be routed directly to the AFE output. The charge amplifiers convert current signals from the strips and also shape the output voltage signal. High speed low noise current feedback amplifier is used. Analogue bandwidth is limited to 20 MHz. Each channel is equipped with dedicated calibration and protection circuit. The calibration circuit allows injecting known charge selectively to any combination of the AFE board inputs. This enables possibility of rapid diagnostic and calibration of the whole detector chain. Expected charge from the detector is 100 fC (it corresponds to 10 ns current pulse with 10 lA amplitude). Each channel has dedicated offset trimming circuit and fast discriminator with adjustable threshold. The AFE can work in one of 4 modes: • Direct sampling of the detector signal. Bandwidth is limited to 20 MHz, ADC sampling is 125 MHz, 12 bits. • Fast, switched hybrid integrator with zero dead time.
• Fast discriminator with adjustable threshold.
• Direct sampling and fast discrimination running in parallel.
Mode selection can be done during DAQ system runtime. To enable advanced diagnostics, each AFE board has dedicated temperature sensor and EEPROM memory. To limit the number of wires, outputs of the discriminators are routed to the serializer chip that informs FPGA that charge compensation took place or that there were multiple hits in a single sample, depending on selected AFE mode. All the signals to the ADC board are routed using single Very-high-density cable interconnect (VHDCI) connector and cable. Since more than 10 W of power is dissipated on every AFE board, the liquid cooling cold plate is attached to it with the bottom metal plate that receives the thermal energy from all integrated circuits. Due to radiation limitation, the p-n junction was used as temperature sensor. Active cooling is ensured using liquid cold plate. The prototype cold plates were made of copper, but due to radiation limitations, they were made of aluminium in the final version of the system. The detector and AFE are main components of the system that are affected by hazard conditions. In case of AFE, all the components were carefully selected to mitigate these threads. All amplifiers, multiplexers, serializers, diodes, transistors were selected from the list of components that are known to be radiation tolerant to the minimum of 50 kRad. That is why it is so important to mount correct components from specified vendors. The same type of, i.e., amplifier from two different vendors may have completely different behavior during irradiation.
GEM detector is very sensitive to the atmospheric pressure and temperature. Due to this reason, air-flow cooling cannot be used because it would create static  pressure on the detector window foil. Liquid cooling does not cause such problems, and each AFE board is equipped with liquid cold plate connected to common distribution box (Detector & AFE Cooling section). External magnetic field seriously affects the operation of the GEM detector. It is possible to shield the detector against it using sheets of mu-metal. From the electronic system point of view, monitoring of magnetic field is assured using several magnetic field sensors distributed around the detector cassette volume.

AFE Modules for B Sensors
Due to the strong magnetic field which affects the GEM detector operation, it is important to know the distribution of magnetic field along the detector's active volume. For this purpose, each detector is equipped with 16 linear magnetic field sensors. Each sensor consist of 3 Hall ICs arranged perpendicularly to each other (Fig. 8). This setup enables 3D magnetic mapping of magnetic field distribution around the detector.
Eight sensors were installed at the GEM stripboard on the side of the GEM foils ( Fig. 9), another eight sensors were installed on the backplane board ( Fig. 10). That enables magnetic field estimation in the Z axis (along the AFE modules).
All Hall sensors installed around the GEM detector require 48-channel conditioning and digitization system. It is implemented as an additional board (Fig. 11) inserted to the backplane close to other AFE boards. Each channel consists of instrumentation amplifier with an offset correction potentiometer and 12 bit ADC. Measurement range is from -0.5 to ? 0.5 T. All the components were selected from ones known to be radiation tolerant. Each ADC has SPI output which after conversion to LVDS is transferred over VHDCI connector, identical as in case of AFE boards.
The LVDS signal is then transmitted to simple 3U interface board, installed in the same slots as ADC boards. Its role is LVDS signal buffering. Digital data are processed directly by FPGA circuit, in parallel with GEM detector data.

ADC Boards (ADB)
The role of ADC boards (Fig. 12) is: • Conversion of the analogue differential signal to digital form    The ADC boards are designed as 3U Euro module with the standard 3HP front panel (Fig. 13). The rear side is equipped with PCIe-type edge connector where ADCs have their serial outputs routed directly to the FPGA processing board slots. Linear Technology, 12 Bits, 125 MHz quad ADCs were used.

FPGA Processing Boards: ADC Backplane
Each FPGA backplane (Figs. 14, 15) supports up to 4 ADC boards using edge connectors. The FPGA backplane collects digital signals from 64 ADC channels and also provides control of 64 hybrid integrators. The FPGA also controls offset and calibration DACs and generates DAC level for the integrator. The reference clock signal is generated on the data concentrator board (Data Concentrator section) and distributed to the ADC boards via low jitter clock buffer. FPGA backplane is also equipped with 2 GB of fast DDR-3 SDRAM, four fast 25 Gbit/s links which utilize mini-SAS connectors. These links are used to transfer edge channels data to neighboring processing modules and provide PCIe x4 connectivity with the Data Concentrator Board.

Data Concentrator
The main role of the data concentrator (Fig. 16) is the collection of data through PCI Express Gen1 x4 links. DCB consists of following resources: • 8 port, PCIe Gen3, switch with 16x upstream port and eight x4 downstream ports. • Low jitter ADC and PCIe clock distribution circuits.    Thanks to the USB to JTAG bridge, all the FPGA bit streams are located in the file system of the server. With such approach, every single piece of code or configuration is loaded during booting process which ensures reliable reconfiguration and the same state of the system after every power up process.

Data Processing Xeon CPU Platform
Recent trends in High Energy Physics and other fields of science move towards software-defined data processing. Software Defined Radio is a popular example. Hardware components are used as front-end interfaces where amount of data is too high for computer systems to handle. The previous generation of systems [7] used FPGA-based data handling and processing which was characterized by low latency and high throughput. However, FPGA-driven approach has several disadvantages: the complexity of the algorithm, issues with maintenance, is inflexible, implementation of the new algorithm requires a highly skilled engineer who knows details of FPGA configuration.
Recent progress in multi-core computers caused a shift towards software-defined signal processing, and such approach was utilized in case of this architecture. The data processing subsystem was equipped with the embedded x86 platform. Server CPU Board (Fig. 17) provides GbE connectivity and 4 PCIe, Gen 3, x16 slots that are used to simultaneously transfer data from the Data Concentrator Boards to up to 24 Xeon CPU cores. Standard ETX formfactor main board is used. Up to two Xeon processors can be installed and optional 56-core Xeon PHI numerical accelerator can be used when additional processing power is needed.

Protection and Interlock System
GEM detectors are very fragile devices-they can be physically destroyed by overvoltage conditions, arching, too high temperature, wrong gas mixture or lack of gas flow. Moreover, the GEM detector is accompanied by power-hungry Analog Front End electronics that need effective cooling to evacuate the heat that otherwise would threaten the detector. The WEST Protection and Interlock System (WPIS) is designed to mitigate all hazard conditions.
The system is based on non-volatile FPGA technology to make it independent from the circuit that is being protected. Figures 18 and 19 present the block schematic and photo of the protection board. It performs following functions:

Gas Interlock
High voltage is enabled only when the gas is flowing and has correct parameters.

ADC Cooling Interlock
Power supply for ADCs is delivered only when cooling fans are rotating, and temperature is below the threshold

AFE Cooling Interlock
Power supply for AFE modules is delivered only when the cooling liquid pumps are rotating, and the temperature is below the threshold. The power supply cannot be switched on without complete initialization of all the sensors.

Power Supply Interlock
AFE supply is enabled only when ADC board is supplied.

Temperature Alarm
All temperature sensors on AFE boards, ADC boards, in the detector proximity, in processing unit, on the FPGA backplane have programmable alarm output. All sensors outputs are wired together using the WIRED-AND scheme to make sure that power supply will be disabled when any of the sensors exceeds permitted temperature.

Latch-Up Detection
When high energy particle passes through the semiconductor device, it may cause the latch-up effect. It opens parasitic thyristor inside of the integrated circuit structure and causes the flow of excessive current from the positive to the negative terminal of the device. Such current, when not limited, causes quick overheating of the device and its breakdown. It may also cause an explosion of the device.
So monitoring of supply current in such operating conditions is critical. The WPIS is connected to the sensitive current sensors installed on every ADC board, that directly measure the supply current on every AFE board. The current sensors outputs are sampled by specialised ADC with alarm function. Each channel of the ADC has dedicated threshold register and comparator that monitors measured value and when exceeded, in a few microseconds sets the alarm. The alarm causes immediate low voltage supply disconnect which protects the AFE. Afer programmable delay, the supply can be enabled.

He Buffer Humidity Monitor
Humidity of the helium buffer is critical because it indicates serious leaks of the cooling system or lack of helium in the buffer. SHT25 sensor is placed in the outlet of the helium buffer and continuously monitored by the embedded software.

Temperature Monitoring
The GEM detector is very sensitive to the temperature variations since it causes frame and window foil deflections and wrinkling. Moreover, the temperature exceeding 150 C cause Mylar foil melting which damages the detector. The Analogue Front End Electronics (AFE) generates a large amount of heat it is over 100 W per detector. In case of AFE cooling failure it presents threat to the detector foils and the electronics itself due to lack of air circulation and limited heat exchange with walls. So the protection against over temperature has the highest priority. Following means to mitigate temperature hazards were foreseen: • Each AFE has the independent temperature sensor. It is connected to the main processing unit and monitored by software. Moreover, each sensor has programmable threshold comparator which causes hardware action when the limit is exceeded. This can be programmed to physically disconnect the power supply from the detector. • Since the solution above is based on some software and needs initialization to work, another redundant solution was foreseen. It is based on the external, independent Power Strip with temperature watchdog which physically disconnects the whole detector when the temperature exceeds the limit. • The cooling unit has independent temperature and flow sensors that are connected directly to the main processing unit. In case of pump failure or heat exchanger problems, the detector will be shut down and errors will be reported.

Neutron Sensor
It is based on SRAM conception. 8 large area static memory chips are used. The supply of the RAM cells is programmable and can be lowered below value recommended by the manufacturer to increase device sensitivity to the neutrons. The software periodically writes memory with fixed pattern and then reads them in order to determine how many errors occurred in specified time.

Fan Speed Control
To limit the amount of noise generated by the processing unit, active fan speed control is implemented. The speed of fans in the ADC compartment is proportional to the temperature measured by two sensors.

Pump Wear Alarm
The rotation of the pump is constantly monitored. When significant loss of RPM is detected, warning condition is asserted to inform operator that pumps need replacement or something wrong happens in the installation

Fan Wear Alarm
The rotation of the fans is constantly monitored. When significant loss of RPM is detected, warning condition is asserted to inform operator that fans need replacement or something blocks the air flow

Redundant Supply for Fans and Pumps
The supply delivered to fans and pump can be sourced by HV & LV unit or processing unit. In case of failure of one of them, protection system will continue operation as well as pumps and cooling.

Watchdog
The software running on processing unit CPU periodically performs monitoring operations. In case of communication failure or over-temperature event, the external power strip (Fig. 20) will disable power supplies. The WPIS is configured by data concentration block that bridges USB to several I2C interfaces. It is connected using several ruggedized M12 connectors with HV & LV unit, cooling distributor, and heat exchanger. Physically WPIS is installed inside the processing unit.

Detector and AFE Cooling
The liquid cooling system is a critical part of the data acquisition system. The cooling is equipped with 2 circuits primary and secondary. The primary is connected to the reactor cooling system that uses water. The secondary uses a special, not conductive, not flammable liquid. Special care was taken designing cooling system since it is crucial for detector operation. The decision was taken not to use water to cool the AFE boards because, in case of a leak, the electronics would be destroyed due to corrosion and electro-migration. To separate these two circuits, the heat exchanger was used. Additional pump, the fluid reservoir, and the flow sensor were foreseen to provide reliable cooling. The liquid cooling system is critical for the safety of the electronics. For this reason, long-term tests were conducted in simplified setup. The tests took 7 months and were performed in order to qualify the following components: pumps, flow meters, reservoir, couplings, heat exchanger, tubing, cold plates. The temperature of the electronics is not critical and does not need stabilisation. Calibration procedure performs periodic offset cancellation to mitigate temperature variation. One need to make sure that the circuits do not exceed 50 degrees at which point GEM detector window may get deformed.
During the test period we discovered several issues: • One of the pumps failed due to leak that caused the inverter malfunction. The pump must be installed vertically to prevent such events. • Both tested flow meters failed after a few weeks. It was caused by copper debris remaining of the cold plate machining process. Installation of particle filter fixed this issue. • Plastic reservoir caused leaks. It was replaced with the solid aluminium one. • Couplings were working reliably, no single leak was observed. • Heat exchanger had some leak which was immediately removed. • Tubing caused issues with the vapour collection. We had to replace them with cooling-agent compatible types. • Cold plates didn not have any issues.
The final version of the cooling system was built (Fig. 21). The main difference between the test bench is the stronger pump and bigger reservoir with the transparent lid that protrudes through the front panel giving visual information about liquid level (Fig. 22).

High Voltage Power Supply
To fulfill detector requirements (high rate and high linearity), dedicated High Voltage Power Supply was developed. It has a form of PCB modules within plastic enclosures that are mounted inside of dedicated 2U aluminium casing (Fig. 23). Unlike most of existing HV supplies it provides following features: • Voltage range 0-1500 V, 0-10 lA.
• Modular construction (Fig. 24)-the number of channels can be customised. Resonant converter topology was used to provide high efficiency and low EMI level. • Fully isolated inputs and outputs-the outputs can be connected in any manner in series or in a parallel way.
Each module has 10 kV galvanic isolation from common control and power supply. • A very low stored charge that is equivalent to 200 nF of output capacitance. It is 10x less than in the previous generation of HV supplies. It is critical in case of GEM foil breakdown. • Very fast current limiter-critical to the operation of GEM detectors. It enables protection of detector in case of arching. It is critical in case of high gain detectors because can reduce the value of series protection resistor and significantly improve the linearity of the detector. • Transient recorder on each channel-currents and voltages can be recorded thanks to embedded scope function. When arcing occurs, one can observe signals before and after the event. • Voltage and current measurement in each channel. One can selectively measure current consumed by each GEM foil. • Ethernet and USB interfaces.
• Local keyboard and LCD display.

Development of Next-Generation Systems
Continuous development of commercially available platforms (COTS) together with constantly changing design requirements inspired us to do the feasibility study on the application of standard measurement and control platforms for plasma physics diagnostic system. There are at least two approaches to the construction of equipment-build it yourself or adopt existing standards. Table 1 summarizes such approaches. In plasma diagnostics number of channels   is usually low, rarely exceeding a few hundreds while experiments last several years. In such cases, R&D costs are often higher than the purchase of COTS equipment. The recent development of MictoTCA standard towards needs of High Energy Physics, especially detector readout and low-level RF application [3], makes this platform potentially suitable for plasma diagnostics as well.

MicroTCA for Detector Applications
MTCA is a modular system for data acquisition, control, management, and data processing [1]. It is derived from established telecommunication ATCA standard. Architecture is based on the high-speed backplane with the point to point connections up to 40 Gbit/s (Fig. 25). Recently developed version 4 of the standard provides extended synchronization, timing distribution and support for RF and analogue front-end electronics. Embedded computer, controller, and PCIe Gen 3 or 40G Ethernet hub make such platform compact and powerful solution. The standard defines several extensions Rear Transition Modules, ruggedized, convection cooled for military and aviation applications. The platform is managed using IPMI including cooling, power distribution, hot-plug and remote control of the chassis. The recent upgrade brings 128 Gbit link (Fig. 26) to external processing system so existing bottleneck between the hub, and the embedded computer was eliminated.
Feasibility of mTCA.4 platform for detector systems: • Industrial standard with scientific community support.
• High-speed links between the neighboring boardsperfectly suited for GEM detector data processing. • FPGA oriented architecture (i.e., JTAG support on the backplane [10], PCIe clock distribution, remote firmware update provided by the crate controller). • Centralized clock and trigger distribution.
• AMC board and RTM board plenty of space for components.
• Additional RF backplane [3] developed at WUT and DESY with low jitter clock, low noise analog connectivity, and dedicated low noise power supply. • N.A.T. MCH with 128 Gbit/s uplink (by AIES).
Enables SW processing in the powerful external server or inside of the MTCA crate [2].
MTCA.4 with recent developments (PCIe uplink, RTM, RF backplane) is the very attractive platform for detector processing systems. Intensive work with N.A.T on new Open Source White Rabbit timing module [8] takes place, so another limitation of the platform is eliminated. Next generation of DAQ system for GEM and other detectors will be based on the MicroTCA platform. Currently, the large scale processing system for GEM detectors based on MTCA for CBM experiment is under development [9]. Another implementation of GEM processing chain for tokamaks in MTCA.4 standard is under development.

Initial results
First laboratory experiments were performed with 1D GEM detector and Fe55 soft X-ray source. The measurement system used 64 analog input channels. One of the first results are presented in Figs. 29 and 30.

Conclusion
Complex real-time SXR measurement systems was built and installed at the WEST Tokamak. The system follows trends set by leading High Energy Physics laboratories where majority of the real time processing is performed in software, using high performance multi-core computer. Due to several threads and harsh conditions at the site, several measures had to be taken to provide safe and reliable operation of the system. Important role plays Protection and Interlock subsystem which after recent upgrade is able to physically remove whole detector subsystem from the tokamak port using custom made elevator mechanism.
The system is currently tested with radioactive source and software integration with WEST control system is performed. Second system with curved GEM detector for horizontal plasma monitoring is in assembly phase. Both detection systems will form SXR tomography diagnostics.