Argonne National Laboratory

Experimental Physics and
Industrial Control System

2002  2003  2004  <20052006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  Index 2002  2003  2004  <20052006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 
<== Date ==> <== Thread ==>

Subject: Re: EPICS base V4
From: Andrew Johnson <anj@aps.anl.gov>
To: Ralph Lange <Ralph.Lange@bessy.de>
Cc: Marty Kraimer <mrk@aps.anl.gov>, Benjamin Franksen <benjamin.franksen@bessy.de>, Eric Norum <norume@aps.anl.gov>, Jeff Hill <johill@lanl.gov>, Bob Dalesio <bdalesio1@comcast.net>, Ned Arnold <nda@aps.anl.gov>, Matej Sekoranja <matej.sekoranja@ijs.si>
Date: Fri, 25 Feb 2005 11:25:39 -0600
Responding to these comments in the order they were made:

Ralph Lange wrote:

Does this imply that IOC load is evenly distributed? I.e. that records with the same scanPeriod period are processed in a "spread over time" manner rather than "en bloc" as in V3?

I suspect that's how it would behave, although I'm not 100% sure how we should be measuring the delay period - presumably measured from the start time of the previous record processing (i.e. we'd add another timestamp field in dbCommon to record the start of processing time, although we could just start the processing delay time from the *end* of the previous processing). If we do it that way then records of the same period would get spread out over time by the delays between their timer going off and actually getting processed.

I would prefer such a spread behaviour, as it really should result in a more "elastic" IOC behavior. But: How would overload situations be handled (i.e. processing all records with a certain scanPeriod takes longer than that period). While the V3 method was inherently stable, it might be that this new way requires additional care. (When spread over different threads, overload situations may be hard to detect.)

I think we could detect and report this problem relatively easily - a record can only appear in the processing queue once, and an overload just means that at the end of processing when it's re-arming the record's timer, the 'time when I'm next due to be processed' has already passed.

The V3 mechanism of the PHAS field determining the order of processing within a scan period does not make sense anymore with your approach, right?

Right (at least that's what I was thinking, but see below).

Will we have to have some kind of replacement for that mechanism, or can we urge the user to go to hard links and events (yes: named events!) instead?

Hmm, named events... Making the scanEvent field into a string might not be a good idea, and I'm not sure about the idea of making it a menu either as there's no way to add a new event type at runtime (unless we allow new event menu choices to be added at runtime; removing or renumbering menu choices should be impossible, but adding new ones isn't beyond the realm of possibility...)

Ralph Lange wrote:
Bob Dalesio wrote:

Marty pointed out then that the use of forward links and passive records
made the load leveling a very difficult problem. Since then, I would add
that there are times that you want to have records of a slower period still all process at the same time.

But most probably not *all* records of that slower period, only those who belong to a certain subsystem, right? Hmm ... let's assume that every of these user named and created scanner threads can be configured to process its records either "synchronised en bloc" or "balanced". In that case the desired behaviour could be achieved by having your set of records all processed by a certain thread, which is configured to do it "en bloc", right?

I don't know how we would actually do the load balancing, and I'm not sure it's necessary to really be pro-active about it. Load balancing happens naturally if we queue a record for processing when its timer runs out, but implement it such that other records already on the 'process now' queue for this thread will cause the record's actual processing to be delayed, and have the timer for the next processing tick doesn't start running until the record processing actually begins. Thus a periodic scan is only a promise to initiate record processing "no more often than every x seconds".

I don't think this is significantly different to what we actually end up doing in V3, we're just being a bit more up front about how it actually works in an overload situation, and the natural load balancing that I think would result just falls out of the algorithm. I may be wrong though...

As to how you tell the database to 'process these records all at once', I'd rather that this requirement be stated explicitly in the database design, either through record linking or by declaring a soft event. With the soft event approach you can even use multiple threads to run the records processed as a result of the event, according to their scanThread field setting.

Hmm, here's another suggestion that makes 'process these records all at once' easier to achieve - how about we make the FLNK field into an array of links? That way you can kick off large numbers of other records if you want to without using a fanout record; it might mean an extra set of braces in the field setting for every forward link, but maybe not.

Perhaps an alternative would be to specify the base rate for the periodic
scanner.
Use PHAS to set which of the sub-periods to use to process the record
So - if the base scan period is 60 Hz, and you define a 1 second scan
record, PHAS of 30 would be at the halfway point of the 60 Hz scanning.

I can it being useful to be able to say "process this record 0.5 seconds after that other 1 second scanned record gets processed" but it's not that common a requirement. To do that with my approach we would need some kind of delay - maybe we write a delay record that is asynchronous and kicks off its forward link some fixed time interval after it first gets processed. You can do this at the moment with a sequence record, so it's not really that new a concept.

To bring this back to the functional specifications - the question seems to be:
Do we need to be able to support "load balancing"

I'm not sure I want to call my suggestion "load balancing", which to me would imply a different approach that I don't want to have to implement myself.

I think that it does not buy us much - but it is nice to give the database engineer more control.

I have seen some cases where in case of a serial-type connection underneath a large number of IO-connected records in a slow task (e.g. the 10 second scan taking more than 1.5 seconds to finish) was filling up low level queues so that fast processing (going through the same queue) was heavily affected. With almost no way to change this behaviour.
Load-balancing would make the IOC a lot more "elastic" in this regard.

You could manually spread the serial I/O over the 10 second scan period in V3 by using a sequence record to delay different parts of it. I'm not sure whether my suggestion will do so naturally or not, that really depends on your configuration.

- Andrew
--
Dear God, I didn't think orange went with purple until I saw
the sunset you made last night.  That was really cool. - Caro




Replies:
Re: EPICS base V4 Eric Norum
References:
Re: EPICS base V4: iocCore database Marty Kraimer
Re: EPICS base V4: iocCore database Andrew Johnson
Re: EPICS base V4: iocCore database: Booleans Benjamin Franksen
EPICS base V4 Marty Kraimer
Re: EPICS base V4 Andrew Johnson
Re: EPICS base V4 Ralph Lange

Navigate by Date:
Prev: Re: EPICS base V4: iocCore database Marty Kraimer
Next: Re: EPICS base V4 Eric Norum
Index: 2002  2003  2004  <20052006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 
Navigate by Thread:
Prev: Re: EPICS base V4 Ralph Lange
Next: Re: EPICS base V4 Eric Norum
Index: 2002  2003  2004  <20052006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 
ANJ, 02 Feb 2012 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·