Experimental Physics and
Industrial Control System

Mark Rivers <[email protected]> · Sat, 28 Jul 2012 14:29:47 +0000

This is slightly off-topic but I thought there might be some interest in a project I am working on which uses libCom for something completely unrelated to EPICS itself.  Here is a summary of a paper I'm writing for the upcoming SPIE conference on computed tomography.

*******************************************
Computers have changed remarkably in just the last 2-3 years.  Memory is now cheap, about $25/GB, or less than $2,000 for 64GB).  Systems with 8 or 12 cores are now inexpensive, starting at less than $3,000.

This means that single workstations are in principle now capable of processing large tomography datasets. But for the most part tomography reconstruction software has not changed to take advantage of these new capabilities.  Most installations use clusters of Linux machines, spreading the work over multiple computers running multiple processes.  It would be simpler and cheaper to run a single process that spreads the job over multiple threads running on multiple cores.

tomoRecon is a multi-threaded library for such tomography data reconstruction.  It consists of only 500 lines of C++ code, on top of the 800 lines of C in the Gridrec reconstruction code.

In order to implement a multithreaded application it is necessary to have a library that provides the following:
   - Support for creating and operating on threads
   - Support for mutexes to prevent conflicts when threads need to access shared data
   - Support for a message passing system for inter-thread communication

To make such an application portable multiple operating systems it is very helpful if this library is written to operate identically on several operating systems.  For this project I have used the libCom library from the EPICS control system, which provides an OS-independent implementation of all of the tools described above. libCom operates transparently on the most commonly used workstation operating systems (Linux, Windows and Mac OSX) as well a number of real-time operating systems (vxWorks, RTEMS).

Preliminary performance tests were done on the following:
- Windows 7 machine with dual quad-core (8 cores total) Xeon X5647 processors running at 2.93 GHz, 48 GB of RAM.
  Two 500 GB 15K RPM SAS disks arranged in Raid 0 configuration.

- Fedora Core 15 Linux system with 16 cores, Xeon E5630 running at 2.53GHz, 12 GB of RAM.
  I am not sure about the disk speed or Raid configuration on this machine.

Here is a summary of the results.

Dataset size          Computer   Time to reconstruct (in memory)
[X, Y, Projections]                 (seconds)

[696, 520, 720]        Windows       4.68
[696, 520, 720]        Linux         4.62

[1392, 1040, 900]      Windows       27.83
[1392, 1040, 900]      Linux         28.0 (14.0 for 520 slices, insufficient memory for all slices)

[2048, 2048, 1500]     Windows       90.4 (45.2 for 1024 slices, insufficient memory for all slices)
[2048, 2048, 1500]     Windows       90.4 (11.3 for 256 slices, insufficient memory for all slices)

There is another test that reconstructs all of the [2048, 2048, 1500] dataset, and writes the reconstruction to disk. It does this by processing the data in chunks.

[2048, 2048, 1500]     Windows       190.2 (00:3:10.20)
[2048, 2048, 1500]     Linux         433.8 (00:7:13.80)

So on single Windows workstation I can do a complete reconstruction, including the time to read the input data, reconstruct and write the output data in just over 3 minutes.

This is 6 to 10 times faster than the equivalent job on the current Linux clusters at the APS.

The input data to the reconstruction is 24 GB, and the output data is 32GB.  So the total data rate is 294 MB/sec, which means that the data is being processed and saved to disk about 3 times faster than GBit Ethernet can transfer it.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System