Hi Xiaoqiang, Emmanuel
Yes, thanks Xiaoqiang. Actually, I played with this some over the
weekend, and found that exact point. Here are a few more details --
I'll try a more complete writeup and post code for this over the next
couple days.
Connecting to 20K PVs (all connected, on several VME IOCs on the same
subnet), using a "normal" epics.PV approach:
import epics
<read pvnames>
pvs = []
for name in pvnames:
pvs.append(epics.PV(name))
vals = []
for pv in pvs:
vals.append(pv.get())
<write values to disk>
takes about 11 seconds for me. PVs use connection and event callbacks
internally.
Asserting that python was "the slow part", I went "straight to libca",
which I believe to be effectively equivalent to using C (the
libca.ca_*** calls go straight to the dll). Here, a more complicated
example:
import ctypes
from epics import ca, dbr # used only for constants
libca = ca.initialize_libca()
<read pvnames>
chidlist = []
for name in pvnames:
chid = ctypes.c_long()
# note: no connection callback
ret = libca.ca_create_channel(name, 0, 0, 0, ctypes.byref(chid))
chidlist.append(chid)
ca.pend_event(1.e-3)
ca.pend_io(1.0)
for chid in chidlist:
name = libca.ca_name(chid)
count = libca.ca_element_count(chid)
ftype = libca.ca_field_type(chid)
pdat = (count*dbr.Map[ftype])() # ok, a python lookup for type
ret = libca.ca_array_get(ftype, count, chid, pdat)
ca.pend_event(1.e-3)
ca.pend_io(0.01)
val = < code to convert pdat to python data type>
This *also* takes around 11 seconds, even though it uses a very
different model -- no connection callbacks, no subscription to
changes. I checked the "convert pointer to python data" ... not the
culprit.
But, like Xiaoqiang pointed out, the pend_event() and pend_io() after
ca_array_get() are the culprit. Replacing the "get loop" with two
loops (and adding more python code!) with:
data = {}
for chid in chidlist:
name = libca.ca_name(chid)
count = libca.ca_element_count(chid)
ftype = libca.ca_field_type(chid)
pdat = (count*dbr.Map[ftype])() # ok, a python lookup for type
ret = libca.ca_array_get(ftype, count, chid, pdat)
data[name] = ftype, count, pdat
ca.pend_event(1.e-3)
ca.pend_io(1.0)
for names in pvnames:
ftype, count, pdat = data[name]
val = < code to convert pdat to python data type>
cuts the run time to 2 seconds. That is, replacing 20K pend_event();
pend_io() with 1 makes a huge difference. I guess that's probably not
surprising to many people here.
FWIW, putting a connection callback back in to libca.create_channel()
seems to slow the code down to about 4 seconds.
OK, so here are my conclusions so far:
a) Python is not the slow part, at least not yet. For the
original question of whether it is suitable for a backup tool, I'd say
it is. I don't see any reason why it wouldn't be within a factor of
2 of the equivalent C code for 100K PVs.
b) You need to be careful about every "pend()" call when trying to
get the best performance when connecting to many PVs.
I'm OK with changing pyepics ca layer so that connection callbacks are
optional and so that get() can optionally return the
"pointer-to-dat", with polling for the data to show up. That would
make it much easier to write code to "quickly connect to thousands of
PVs".
I'm not sure how to get this performance boost using higher-level PV
objects. These use both connection callback and event callbacks,
which I consider an important feature for ease-of-use. It's a
reasonable goal to want to create thousands of PVs as quickly as
possible. I think that might imply an extra step was needed to setup
the callbacks. I'll have to think about how to do that -- suggestions
welcome!
--Matt Newville <newville at cars.uchicago.edu> 630-252-0431
- References:
- How to write a python-based backpup tool? emmanuel_mayssat
- Re: How to write a python-based backpup tool? Matt Newville
- Re: How to write a python-based backpup tool? emmanuel_mayssat
- Re: How to write a python-based backpup tool? Matt Newville
- Re: How to write a python-based backpup tool? emmanuel_mayssat
- Re: How to write a python-based backpup tool? Matt Newville
- RE: How to write a python-based backpup tool? Wang Xiaoqiang
- Navigate by Date:
- Prev:
2 software engineering associate positions open at Advanced Photon Source Claude Saunders
- Next:
RE: Issues on image displaying in CSS [SEC=UNCLASSIFIED] Chen, Xihui
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
<2011>
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
RE: How to write a python-based backpup tool? Wang Xiaoqiang
- Next:
Re: How to write a python-based backpup tool? Ralph Lange
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
<2011>
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
|