Experimental Physics and Industrial Control System
|
Hi Murali,
I had some data back from testing, the http post request is asking for about 3000 PVs from the AA which stores about 4500 PVs Total in the LTS.
I did the benchmark test to just transfer a single pb file and was getting around 100MB/s, so I do not think it has to do much with the network since the file size for the 3000 PVs is about 40MB only. I then reduced the request down to
about 1200 PVs and it definitely sped up quite a bit around 15 seconds from the 2.5mins it was taking. Someone mentioned the lookup time for each PV was log(n) or nlog(n) for the AA but then that doesn’t seem like it would impact going from 1200 to 3000 in
the POST.
I am curious how many PVs you usually request in a single http when requesting from the AA, is there an upper limit I should stick to? You mentioned possibly breaking up the requests into smaller chunks.
Thanks
Adam
From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of
Manoussakis, Adamandios via Tech-talk
Sent: Sunday, February 20, 2022 10:44 PM
To: Shankar, Murali <mshankar at slac.stanford.edu>; tech-talk at aps.anl.gov
Subject: RE: Epics Archiver Appliance and Network Storage slowdown
Thanks Murali,
I will try to run some more tests including the one that was mentioned to make sure the transfer rates look correct. I cant imagine the local vs NAS on this small of a setup shouldnt be vastly different.
>> I think for this experiment its only 6000 PVs
I think that should not take this long. Will look into this a bit here as well.
>> We are using the getDataAtTime endpoint
>> 40MB file from the archive
By this I'd guess that you are getting data for several 100,000's of PVs? The getDataAtTime API call has to look at all these 100,000 files ( with perhaps cache-misses
for most of them ) and then do binary search to determine the data point. Your NAS needs to support quite a high rate of IOPS to for this to come back quickly. And this is a usecase where even the smallest latency tends to accumulate quickly. Perhaps you can
consider breaking down your request into smaller chunks when using the NAS?
|
- Replies:
- Re: Epics Archiver Appliance and Network Storage slowdown Shankar, Murali via Tech-talk
- References:
- Re: Epics Archiver Appliance and Network Storage slowdown Shankar, Murali via Tech-talk
- Re: Epics Archiver Appliance and Network Storage slowdown Shankar, Murali via Tech-talk
- RE: Epics Archiver Appliance and Network Storage slowdown Manoussakis, Adamandios via Tech-talk
- Navigate by Date:
- Prev:
Re: Linking PVs to EGU field Ralph Lange via Tech-talk
- Next:
ORCA Flash4.0 integration Tomasz Brys via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
<2022>
2023
2024
- Navigate by Thread:
- Prev:
RE: Epics Archiver Appliance and Network Storage slowdown Manoussakis, Adamandios via Tech-talk
- Next:
Re: Epics Archiver Appliance and Network Storage slowdown Shankar, Murali via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
<2022>
2023
2024
|
ANJ, 14 Sep 2022 |
·
Home
·
News
·
About
·
Base
·
Modules
·
Extensions
·
Distributions
·
Download
·
·
Search
·
EPICS V4
·
IRMIS
·
Talk
·
Bugs
·
Documents
·
Links
·
Licensing
·
|