Data Acquisition in Scientific Applications

by Rob Halsall

Large scale scientific instrumentation systems for particle physics research at CERN are currently being deployed with many millions of sensor channels and Data Acquisition (DAQ) rates of the order of 100s of GByte/s. These systems, consisting of many racks of large format electronics modules, are typically implemented using custom digital logic in Field Programmable Gate Array (FPGA) devices. Installed in under ground caverns, closely coupled to massive detectors and networked to PC processing farms, these systems could be regarded as very large examples of embedded systems.

In other science areas (eg Synchrotron Radiation (SR), Neutron beams) there are growing requirements to move to similar detector technology. Although on a smaller scale this is resulting in high sample rates and channel counts with a dramatic increase in the data rates over previous systems from around 100 MByte/s to 10 GByte/s.

Recent advances in FPGA’s have seen these devices evolve into programmable System On Chip (SOC) ASICs making these requirements feasible at reasonable cost and size. Typical FPGA components now contain multiple microprocessors, multiple multi-gigabit transceivers capable of driving common network standards, as well as multi million gates of programmable logic with DSP enhancements allowing performance rates up to 250 GMACS/s for the largest devices.

This provides the opportunity for shrinking our systems and producing embedded high performance DAQ with standard networked readout such as GBit Ethernet – Large Scale Science DAQ systems shrink wrapped into DAQ Sensor appliances.
In CCLRC Technology we have demonstrated the technical feasibility of embedded hardware based on high-speed serial optical links, standard network protocols, and programmable System On Chip (SOC) FPGA devices to deliver across a wide range of requirements and projects.

Figure 1: One of 500 digitiser cards for a Large Scale Science Project.

Figure 2: High performance Embedded DAQ for Custom CMOS Sensor (not shown).

Our designs have implemented embedded hardware support features such as electronic fuses, supply monitoring, thermal monitoring, shutdown on fault, Wake On LAN type recovery from shutdown and low stand-by power. Remote reconfiguration of FPGA FLASH memory is possible together with encryption of the bit stream and automatic reversion to a fallback FLASH memory in the event of failure during re-programming. A typical small system might consist of a pixel sensor chip readout by an FPGA which is in turn readout by a PC server over a GBit Ethernet connection. A commodity SDRAM memory module connected to the FPGA provides data buffering and optical transceivers allow transmission of the GBit Ethernet over longer distances.

Figure 3 shows an embedded DAQ module planned for use on SR applications reading out 16 CMOS Pixel image sensors capable of a multi KHz frame rate. The module can log up to 800 MByte/s into SDRAM for a few seconds and then is readout to a PC server on two GBit Ethernet cables each sustaining 80 MByte/s.

Figure 3: Multiple Custom CMOS Pixel Sensor with high performance Embedded DAQ.

One problem for these more widely deployed smaller projects based on high performance embedded modules is the design cost and turnaround time. Unfortunately there is no all-encompassing design tool for DAQ systems. The design process relies on an increasingly large and diverse portfolio of complex tools that require a high level of expertise to operate. Thus there is a growing cost of keeping up with the tool flows required to access more powerful hardware.

Currently we have an R&D program funded internally by our Centre for Instrumentation (CFI) which will investigate the embedded processor design tools. This project aims to reduce the cost of the firmware and software design for DAQ projects by developing a unified approach based around FPGA vendor embedded development tools.

The strategy we propose is to exploit the hardware IP modules and facilities existing within the embedded design tools by adding instrumentation-specific DAQ IP for integration of the sensor data stream. Customising the vendor’s framework minimises the cost by reducing the amount of work that needs to be done, and minimises the risk by using standard interfaces.
A similar approach is taken for the software libraries – by using standard network protocols (Ethernet) and operating system APIs the software side of the DAQ library is more like 'DAQ middleware'.

The addition of embedded networked microprocessors within our designs gives us the ability to add higher levels of intelligence to our systems which might otherwise be difficult to include. Even allowing for the availability of C to hardware tools and an abundant gate count in the FPGA using embedded microprocessors would still be an efficient method.
Examples of the type of features which could be added include real-time tuning of the data flow and processing algorithms, logging of environmental information, safety critical monitoring of sensors (eg to prevent radiation damage), interaction with higher level control systems using standard protocols and provision of information to the e-science systems to aid in the management of the integrity of the data set.

Please contact:
Rob Halsall, CCLRC Technology
Rutherford Appleton Laboratory UK
Tel: +44 1235 445140
E-mail: R.Halsallrl.ac.uk