Re: Weird behaviour in wait=True when using epics.Motor.get(something, something, wait=True)

[Torsten Bögershausen via Tech-talk <tech-talk at aps.anl.gov>]

Hej Marco,

some (late) and hopefully useful response.

 

To my knowledge and understanding, writing to a motorRecord field with wait=True

does only make sense for the .VAL (or .DVAL, .RVAL) fields.

Because that what is what the motorRecord code is written to do.

 

I have been able to dig into the stuff a little bit more.

0) The motor is not moving: .DMOV is 1

1) we write 0 to the .CNEN field through channel access.

     The special() function in motorRecord is called with after=0

     The special() function in motorRecord is called with after=1

      A " WRITE_MSG(DISABL_TORQUE)" is send to the driver

    The record processes, process() is called

      Inside do_work() a GET_INFO is send to the driver

      The first callback is fired via recGblFwdLink()

2) Now things happen in parallel:

    2a) The driver polls the motorController (at least a model 3 driver does),

           which may take some milliseconds, when we have an asyn motor

           connected via ethernet.

          The poll() completes, and the record is processed again.

          The second callback is fired via recGblFwdLink()

     2b) The firmware inside the motion controller handles the

            TORQUE message. This can take µsec, milliseconds.

            Depending on the driver/controller, the TORQUE message

            may enable TORQUE control, closed loop, or enables the amplier.

            This variable/code is used for different things:

            It is heavily driver dependent, for what CNEN is used.

3)  The poller continues to poll in the background.

      If e.g. an encoder readback changes, the record will be processed again,

      and a  third, forth... callback is fired via recGblFwdLink()

      Another reason for a callback is when the driver detects that the amplifier is

      now on in the controller.

      This takes some hundred milliseconds in our system.

      A model 3 driver may write the parameter "motorStatusPowerOn_".

      The motorRecord reflects this bit in the .MSTA  field:

        if (msta.Bits.GAIN_SUPPORT)

        {

            unsigned short pos_maint = (msta.Bits.EA_POSITION) ? 1 : 0;

            if (pos_maint != pmr->cnen)

            {

                pmr->cnen = pos_maint;

                db_post_events(pmr, &pmr->cnen, local_mask);

            }

        }
        //

       So there is a possible readback of CNEN:
       When the value is changed inside the controller, by e.g. using an engineering tool,

       the driver is able to update the .CNEN field!
===================

I did some testing here, but couldn't make sense out of the number of callbacks either
(for the .CNEN field)

     

Depending on the timing, it seems as if callbacks can be "bundled" in some way.

If this is done on the IOC, or within python/pyepics, I don't know.

Probably a wireshark run would reveal more details.

 

As a kind of summary, integrating the answer from Mark, it looks as if

the "wait=True" feature make only sense when writing to the ".VAL" field,

and probably (not tested by me) to DVAL and/or RVAL.

That is what the motorRecord is designed for.

BR
/Torsten

 

 

 

 

 

From: "Marco A. Barra Montevechi Filho" <marco.filho at lnls.br>
Date: Thursday, 26 January 2023 at 16:20
To: Torsten Bögershausen <Torsten.Bogershausen at ess.eu>, Matt Newville <newville at cars.uchicago.edu>
Cc: SWC <swc at lnls.br>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Subject: Re: Weird behaviour in wait=True when using epics.Motor.get(something, something, wait=True)

 

Hi, Torsten!

>The Record is processed twice, first because the update of the DLLM field.

And another time when the motor had been polled caused by a "GET_INFO"

transaction.

Understood.
It would be easier for me to understand if it was a constant behavior. It is still a little harder when it is expected to happen twice but happens only once (rarely, but occurs as reported in previous emails).
Anyway, the point here is just to understand how the record works so that i can plan my scans in a way that makes sense, so thanks for the explanation 🙂

> Writing to CNEN with wait=True seems to wait for the Record to have processed

the command, it seems to me.

However, when the record has processed the command, and send it to the controller,

it doesn't mean that the controller has digested it fully.

Yes, thats what i would intuitively expect but after Mark's explanation i dont think its the case:

From Mark> I think there may be a complication with the motor record, and other complex records like mca, scaler, etc.  Using Python put(wait=True) means you are using ca_put_callback() in the underlying Channel Access C code.  ca_put_callback() completes when the record calls recGblFwdLink().  For those complex records recGblFwdLink() is called when the record is "done".  For the motor record that means that a move is complete(...)

Which kinda makes sense or at least would explain the assertion errors i get when doing in python:

epics.caput("MOTORPV.SOME_FIELD",value,wait=True)
assert epics.caget("MOTORPV.SOME_FIELD")==value

If the record has processed the command, wouldnt that mean that caget("MOTORPV.SOME_FIELD") would give me my desired value? Or is the database updated onlye after controller responds to query?
As i said before: i couldnt find anything polling CNEN value with SVO command, so whatever value is in it is defined only when the put to the CNEN field is processed, i think.

> What are you using CNEN for?

It sends the GCS comand SVO to the controller, which according to the manual sets the servo state on or off. I couldnt find in the manual how much time this should take, but makes sense to me to think that it takes some seconds.

As i said, im not interested particularly in this field, though. It was one of many whith which i got the assertion errors that didnt make sense at the time but seem to make sense to me after Mark's explanation.

Best regards,

Marco

---

From: Torsten Bögershausen <Torsten.Bogershausen at ess.eu>
Sent: 23 January 2023 14:43
To: Matt Newville <newville at cars.uchicago.edu>; Marco A. Barra Montevechi Filho <marco.filho at lnls.br>
Cc: SWC <swc at lnls.br>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Weird behaviour in wait=True when using epics.Motor.get(something, something, wait=True)

 

Hej again,

are we still wondering about callbacks and wait=True ?

 

Writing to VELO give one callback and one camonitor event here.

Writing to DLLM give 2 callbacks and 2 monitors. This feels like a bug to me,

but I don't think it worth to fix it.

The Record is processed twice, first because the update of the DLLM field.

And another time when the motor had been polled caused by a "GET_INFO"

transaction.

 

Back to the wait=True question, if anybody is interested.

Writing to CNEN with wait=True seems to wait for the Record to have processed

the command, it seems to me.

However, when the record has processed the command, and send it to the controller,

it doesn't mean that the controller has digested it fully.

What are you using CNEN for ?

Often it is used it for "enabling the amplifier".

Other definitions that I have seen are "closed loop" or "torque control".

 

We use it for "enabling the amplifier". Once the controller has received

the "enable", it make take 1 second until the amplifier is ready, and the motor

can be moved.

 

HTH
/Torsten

 

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Matt Newville via Tech-talk <tech-talk at aps.anl.gov>
Reply-To: Matt Newville <newville at cars.uchicago.edu>
Date: Thursday, 19 January 2023 at 20:25
To: "Marco A. Barra Montevechi Filho" <marco.filho at lnls.br>
Cc: SWC <swc at lnls.br>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Subject: Re: Weird behaviour in wait=True when using epics.Motor.get(something, something, wait=True)

 

Hi Marco, 

 

 

On Thu, Jan 19, 2023 at 9:43 AM Marco A. Barra Montevechi Filho <marco.filho at lnls.br> wrote:

Updates on testing:

So, i did the suggested test alternating CNEN (between 0 and 1), VELO (1 and 2), DHLM (4 and 5) and DLLM (-4 and -5) fields. Not because they have any particular physical meaning, but to understand if the behavior is similar in several motor record fields. I tested at least twice for each field.

I append the code for testing below and attach the logs as files. What i found interesting:

1 - In some tests, the first print from callback function is the first value in which the field was before changing (i.e., before the first caput in the code is called). In some other tests, the first print is the value only after being changed via caput. This suggests to me that the "add_callback" method by itself sometimes triggers a callback and sometimes doesnt. I remember having observed this behavior before with other PVs.

2 - In several parts of the log, callback prints the same value twice. This is more or less predictable: DLLM field for example seems to almost always trigger two callbacks at once, but there are cases like the end of Loop 6 in CNEN_LOG1 where it seems out of the pattern: CNEN almost always triggers only one callback, but in this loop i got two at once.

What i understand:

1 - some fields typically (but not always) trigger callbacks twice and some dont. 
2 - the add_callback method may or may not immediately call the callback function, even if no value has been altered in the PV.

What i dont understand:

1 - What are the causes of intermittence in behavior 1, such as observed in end of loop 6? 

2 - What are the causes of intermitence in behavior 2?

 

 

I suspect that the "sometimes get an initial callback" is because creating an `epics.Motor()` will make an initial connection for PVs to some fields, including

  'VAL', 'DESC', 'RTYP', 'RBV', 'PREC', 'TWV', 'FOFF', 'VELO', 'STAT', 'SET', 'LLM', 'HLM', 'SPMG', 'LVIO', 'HLS', 'LLS'

but will not create PVs on all of the other ~80 fields of the Motor record until you ask for them.  If you add a callback (to be clear, on "monitor events") when creating a PV, that will be called when it gets the initial value.  So for an `epics.Motor()`, PVs other than those above should get an event with the initial value on PV creation.

 

I also see that writing to the DHLM and DLLM of the Motor Record causes two nearly simultaneous events with the same new value.  I don't see that for all other fields, but the motor record is complicated, and setting some values will cause a cascade of events.  I suspect that setting DHLM will also set HLM (or maybe LLM if the direction is negative), which might then also update the DHLM value. 

 

Generic question: can these things be controlled or is it just something i have to live with?

 

General answer: Some things are easier to control than other things. 

 

If you first fetch the value of "CNEN" or "DHLM", and then add the callback, you should only see changes from that initial value (all assuming that connections happen in time).   

 

I don't know how to get only 1 callback if DHLM is changed.

 

--Matt

 

 

Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.