> pvlist = []
> for pvName in pvNames:
> pvlist.append( epics.PV(pvName) )
>
> for pvs in pvlist:
> val = pv.get()
> <store value>
>
> With this sort of approach, I typically see on the order of 10ms per
> PV connection on startup. That is, if I create and connect to 5K
> PVs, it takes ~50 seconds (meaning 40 to 80 seconds) to get initial
> values. I believe that is mostly the CA library, not the python part,
> and I believe it would scale, suggesting that any save/restore process
> that runs once and then quit would take 10 minutes for 50K PVs.
Are you saying that 10 min for 50K PVs is also what is observed with C utilities?
10 min for 50K is rather long ...
If that is the case, then one whould be careful to backup only PV which are of interest
and not the 'entire' machine.
How is this being managed at large installation?
> My understanding from a little playing with Cython is that it is very
> good at turning the slow parts of Python to C-like performance. I
> think the "slow Python parts" for this code are
> a) creating a list of 50k names / PVs -- not that big a cost
>
> b) creating the 50k PV object -- shouldn't be that big, as it creates
> a CHID and asks for a connection callback -- which will not prevent
> you from going to the next python statement.
>
> c) the python parts of the actual PV.get() , which shouldn't be that
> big, especially for scalar values, and especially assuming that the
> connection has been established.
That's still 10 min for 50K PVs...
>
> I think these are not very slow, though there might be some python GIL
> issues with having all those different bits of code run from the CA
> connection callbacks -- I haven't noticed a problem, but I haven't
> pushed this hard on it either. I think most of the real time would
> be spent in the initial network connection part of the CA library --
> which is to say that I'd guess the speed of this loop would not be
> that much slower than the C equivalent.
I understand your points, but somehow I am having a hard time believing this.
Did anyone benchmark their backup platform?
(I remember seeing a few powerpoints on this issue a while back...)
Thank you for your input.
PS:
Is there a difference between a connection to 50K Pvs versus, 50K connections to the same PVs?
That would make the generation of my test request file quite simple...
Just wondering
Thanks again.
--
Emmanuel
- Replies:
- Re: How to write a python-based backpup tool? Matt Newville
- References:
- How to write a python-based backpup tool? emmanuel_mayssat
- Re: How to write a python-based backpup tool? Matt Newville
- Navigate by Date:
- Prev:
Re: How to write a python-based backpup tool? Matt Newville
- Next:
Re: How to write a python-based backpup tool? Matt Newville
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
<2011>
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: How to write a python-based backpup tool? Matt Newville
- Next:
Re: How to write a python-based backpup tool? Matt Newville
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
<2011>
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
|