This is slightly off-topic but I thought there might be some interest in a project I am working on which uses libCom for something completely unrelated to EPICS itself. Here is a summary of a paper I'm writing for the upcoming SPIE conference on computed tomography.
*******************************************
Computers have changed remarkably in just the last 2-3 years. Memory is now cheap, about $25/GB, or less than $2,000 for 64GB). Systems with 8 or 12 cores are now inexpensive, starting at less than $3,000.
This means that single workstations are in principle now capable of processing large tomography datasets. But for the most part tomography reconstruction software has not changed to take advantage of these new capabilities. Most installations use clusters of Linux machines, spreading the work over multiple computers running multiple processes. It would be simpler and cheaper to run a single process that spreads the job over multiple threads running on multiple cores.
tomoRecon is a multi-threaded library for such tomography data reconstruction. It consists of only 500 lines of C++ code, on top of the 800 lines of C in the Gridrec reconstruction code.
In order to implement a multithreaded application it is necessary to have a library that provides the following:
- Support for creating and operating on threads
- Support for mutexes to prevent conflicts when threads need to access shared data
- Support for a message passing system for inter-thread communication
To make such an application portable multiple operating systems it is very helpful if this library is written to operate identically on several operating systems. For this project I have used the libCom library from the EPICS control system, which provides an OS-independent implementation of all of the tools described above. libCom operates transparently on the most commonly used workstation operating systems (Linux, Windows and Mac OSX) as well a number of real-time operating systems (vxWorks, RTEMS).
Preliminary performance tests were done on the following:
- Windows 7 machine with dual quad-core (8 cores total) Xeon X5647 processors running at 2.93 GHz, 48 GB of RAM.
Two 500 GB 15K RPM SAS disks arranged in Raid 0 configuration.
- Fedora Core 15 Linux system with 16 cores, Xeon E5630 running at 2.53GHz, 12 GB of RAM.
I am not sure about the disk speed or Raid configuration on this machine.
Here is a summary of the results.
Dataset size Computer Time to reconstruct (in memory)
[X, Y, Projections] (seconds)
[696, 520, 720] Windows 4.68
[696, 520, 720] Linux 4.62
[1392, 1040, 900] Windows 27.83
[1392, 1040, 900] Linux 28.0 (14.0 for 520 slices, insufficient memory for all slices)
[2048, 2048, 1500] Windows 90.4 (45.2 for 1024 slices, insufficient memory for all slices)
[2048, 2048, 1500] Windows 90.4 (11.3 for 256 slices, insufficient memory for all slices)
There is another test that reconstructs all of the [2048, 2048, 1500] dataset, and writes the reconstruction to disk. It does this by processing the data in chunks.
[2048, 2048, 1500] Windows 190.2 (00:3:10.20)
[2048, 2048, 1500] Linux 433.8 (00:7:13.80)
So on single Windows workstation I can do a complete reconstruction, including the time to read the input data, reconstruct and write the output data in just over 3 minutes.
This is 6 to 10 times faster than the equivalent job on the current Linux clusters at the APS.
The input data to the reconstruction is 24 GB, and the output data is 32GB. So the total data rate is 294 MB/sec, which means that the data is being processed and saved to disk about 3 times faster than GBit Ethernet can transfer it.
- Navigate by Date:
- Prev:
RE: Waveform read problems Mark Rivers
- Next:
Re: Waveform read problems Eric Norum
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
<2012>
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: Waveform read problems Eric Norum
- Next:
seq_pvPut() Bradley Pietrzak
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
<2012>
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
|