Re: Epics Archiver Appliance and Network Storage slowdown

[Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov>]

On 2/10/22 15:11, Manoussakis, Adamandios via Tech-talk wrote:
> Hello All,
>
> I am running into a strange slowdown when using our NAS (red drives) with the archiver for MTS STS LTS folders.  We have softlinks for each of the folders in /arch to the NAS MTS STS LTS folders, writing data seems to work perfectly fine but it is taking 2-3mins to pull a 40MB file from the archiver instead of <30seconds previously when it was local storage.  Sometimes this is even causing our application that is pulling the data to timeout due to the long wait time to grab data.  I switched back to local and seems to run at less than 30 seconds a file for the 40MB of data JSON using the same front end application to do the request.  We are using the getDataAtTime endpoint to get the JSON with the post including the list of records we want in JSON.  I checked with our IT to see if there was any reason why there would be slowdown over the network but didn’t get any reason why it should.  I wrote my own python script just to see if it matched what our application request time 
> was seeing and it matched about 2min-2.5min per 40MB file.  Just wondering if there might be something I am missing to speed this up or is this expected from using NAS with the archiver?
Sorry if I'm a little confused by the above.  Is your python script testing retrieval via AA?

It might be interesting to start with some storage access benchmarks.  AA retrieval
is never going to be faster than this.  At simplest, time how long it takes to read
some files from the NAS vs. local.  Something like:

> dd bs=1M of=/dev/null if=/arch/lts/whatever.pb
When doing this keep the Linux disk cache in mind.  Either clear the disk cache (cf.
/proc/sys/vm/drop_caches) or read different .pb files each time.  (the second might
be friendlier on a production system)

Also keep in mind that because AA doesn't index .pb files, the retrieval process involves
a binary search within each file.  This can involve reading larger parts of the file than
you might expect.  So eg. retrieval time will not be linear with the length of time requested.