Re: Data migration

["Pearson, Matthew R. via Tech-talk" <tech-talk@aps.anl.gov>]

Hi Steve,

We use rsync to do a similar task, where we have a high speed Andor camera writing to a local disk, and then we copy it to central storage. Some problems we encountered were:

1) Running rsync every few minutes via a cronjob only works if there’s not much data to copy. If there’s too much data, then you’ll end up with multiple rsync processes copying the same data. So you need to ensure only one rsync can run at once.

2) Reading data off the disk at the same time as the IOC is writing data could impact IOC write performance. So we have to ensure that the rsync can’t run when we expect to write a lot of data. 

3) If you have written a lot of data, then you may need to wait for the data to be read off the disk again before resuming data collection. This means planning some dead time between data collection runs.

4) Consider using the rsync auto delete option. Otherwise people will need to manually delete data once the SSD fills up.

5) Make sure you don’t have a 100Mbit/s link between the SSD and the NAS. A 1Gbit/s link will still be slower than the local SSD write speed, so 10Gbit/s links may be required if you don’t want the network to be a limiting factor.

In our case, we haven’t solved all the issues I mentioned. One thing I’m considering is wrapping the rsync process in an IOC so I can control it via PVs and monitor the data transfer process. Then it could be better integrated into the data collection scans.

Cheers,
Matt


Data Acquisition and Controls Engineer
Spallation Neutron Source
Oak Ridge National Lab


> On Feb 13, 2019, at 12:36 PM, Steve Kinder - UKRI STFC via Tech-talk <tech-talk@aps.anl.gov> wrote:
> 
> Apologies for being a bit of subject but thought people might have some useful experience or suggestions. I have an IOC with an SSD disk which I would like to use for short burst rapid saving of camera images; that's fine. I would then like to migrate the data from the SSD to a NAS server in the background somehow. Any suggestions on the latter bit would be appreciated.
> 
> 
> Thanks
> 
> 
> Steve
>