RE: galil softIOC lose the galil controller

[Mark Clift <Mark.Clift@synchrotron.org.au>]

Hi Heinz,

Thanks for the update.  I agree, I think Matt will win his bet.

Please let me know how it's going in a few days.  I will test the RIO here, and report back also.

I also have worked with Galil on RIO ethernet issues in the past.  I agree, it sounds like the RIO firmware could be at fault.  I can't see any obvious problems in the driver.  I also dream of using asyn as the communication library to avoid some of the Qt problems you mention.  In the same time, I've found the Galil gcl is usable, and it's performance is good.

Stay in touch,

Mark.

Dr. Mark Clift
Senior Controls Engineer
Australian Synchrotron
800 Blackburn Road
Clayton 3168
Ph: +613 8540 4264
Fax: +613 8540 4200
Mail: Mark.Clift@Synchrotron.org.au
________________________________________
From: Heinz Junkes [junkes@fhi-berlin.mpg.de]
Sent: Sunday, 2 November 2014 07:05
To: Mark Clift
Cc: Pearson, Matthew R.; tech-talk@aps.anl.gov (tech-talk@aps.anl.gov)
Subject: Re: galil softIOC lose the galil controller

Hi Mark,

I had the same experiences as Matt with a DMC-40x0 for several years. The galil came with an undulator. We used a telnet connection (asyn, StremaDevice) to
communicate with the controller. The controller was unreachabe at irregular intervals. We always had to reset it to get it working again on the ethernet. We had
many discussions and attempts to repair by Galil.

But it did not help. Our solution was to connect the galil to a dedictaed  ethernet-interface (with a own IP-Net)  on our server running the ioc. And since it works.

I then purchased three rio controllers. I had the hope that these are more modern and connected them to the standard production net with the other ioc’s and devices
(except the GigE cameras) connected to. But then I lost them every few days or couple of weeks (I share the same assessment as Matt, it will be the broadcast
messages killing the RIO’s ethernet stack).

Then I wanted to use the galil-software-lib to communicate with the RIOs. So I startet with galil-tool-lib. I was from the beginning not happy with this library as it uses
qtconnect which needs a X-server running on the IOC … But even so it did not work, the RIO’s ar no longer accessible after some time.

I think Matt will win his bet. It has nothing to do with the communication but with the incorrect implementation in the RIO firmware.

I have connected the RIO's via the serial port ( connected to a Moxa terminal server, 115 kBaud, CTS/RTS) on Friday, they are still working ;-)

Cheers,
Heinz


> On 31 Oct 2014, at 13:05, Mark Clift <Mark.Clift@synchrotron.org.au> wrote:
>
> Hi,
>
> The advice to use a private network is a good idea.  Also, when exiting the softIOC type "exit" so that the exit handler turns off the RIO data record at IOC shutdown.  If the softIOC is killed with ^C or Kill, the RIO will continue to send data records with the IOC down, which may cause issues?
>
> Also is the RIO stable when connected only to GalilTools?
>
> I'll investigate further with one of our spare RIO's.
>
> Best wishes,
>
> Mark
>
> Senior Controls Engineer
> Australian Synchrotron
> 800 Blackburn Road
> Clayton 3168
> Ph: +613 8540 4264
> Fax: +613 8540 4200
> Mail: Mark.Clift@Synchrotron.org.au
> ________________________________________
> From: Pearson, Matthew R. [pearsonmr@ornl.gov]
> Sent: Friday, 31 October 2014 00:44
> To: Mark Clift
> Cc: Heinz Junkes; tech-talk@aps.anl.gov (tech-talk@aps.anl.gov)
> Subject: Re: galil softIOC lose the galil controller
>
> Hi,
>
> This may or may not be related, but we failed to have any kind reliability with our two Gaili DMC-40x0 series controllers as long as they were on an open network with broadcast traffic.
>
> On our general purpose office network, the controllers would lock up their network interface after a few minutes. The error LED comes on, and only a power cycle fixes it. On the single beamline where we have the two controllers this happened every few days or couple of weeks. Eventually we placed the two controllers on a private network interface connected directly to a Linux server, and we haven't had any issues since.
>
> We're using Marks original motor driver, but that wasn't related to the controller network problems. The controllers locked up even without any EPICS software talking to the controllers, which points to problem broadcast traffic.
>
> We did notify Galil about the issue, but once we resolved it we stopped investigating it here. I know others have had the same issue with these controllers. I bet it's the same issue if the RIO-47xx is using the same ethernet controller. We found that it was listening by default on 1000s of port numbers, and any one of them accepted a TCP connection (up to 8 at a time I think).
>
>
> Cheers,
> Matt
>
> Data Acquisition and Control Engineer
> Spallation Neutron Source
> Oak Ridge National Lab
>
>
> On Oct 30, 2014, at 4:47 AM, Mark Clift <Mark.Clift@synchrotron.org.au> wrote:
>
>> Hi Heinz,
>>
>> Please follow this setup procedure for the RIO PLC.
>>
>> 1.  Use GalilTools to set the IP address
>> 2.  Record the MAC address of the RIO
>> 3.  Edit /etc/rc.d/rc.local and add the following:
>>
>> # Add multicast route for Galil hardware
>> route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
>>
>> # Add arp entry for Galil RIO device
>> arp -s <IPADDRESS> <MACADDRESS>
>>
>> 4.  Reboot softIOC or type above commands as root, then exit root login
>> 5.  Telnet into RIO from softIOC, and issue the following commands
>>
>> :IB1   (Possibly IK1 on some firmware)
>> :BN   (Burn the changes)
>> :^M
>> exit
>>
>> 6.  From now on only hosts with the arp entry can telnet into RIO
>>
>> Then try the IOC again.  I recommend using a single IOC and connection to the RIO.
>>
>> Also try the email again.
>>
>> Best wishes,
>>
>> Mark.
>>
>> Dr. Mark Clift
>> Senior Controls Engineer
>> Australian Synchrotron
>> 800 Blackburn Road
>> Clayton 3168
>> Ph: +613 8540 4264
>> Fax: +613 8540 4200
>> Mail: Mark.Clift@Synchrotron.org.au
>> ________________________________________
>> From: Heinz Junkes [junkes@fhi-berlin.mpg.de]
>> Sent: Thursday, 30 October 2014 19:16
>> To: Mark Clift
>> Cc: tech-talk@aps.anl.gov (tech-talk@aps.anl.gov)
>> Subject: Re: galil softIOC lose the galil controller
>>
>> Hi techtalk,
>>
>> we are running two galil RIO-47120 controllers. We are using the newest versions of galilsuite and galiltools on debian linux.
>>
>> ii  galilsuite                          1.0.2                               amd64        Test GalilSuite build
>> ii  galiltools                          1.6.3                               amd64        GalilTools
>>
>> and the Galil-3-0 motorapp.
>>
>> We could'nt get them running stable. Until the day before yesterday I connected to both controllers with one softioc. About every second week they stopped working and the controllers could'nt be ping-ed anymore. We have to reset the controllers. Both stopped at the same time.
>> Now I run two softioc's on the same Unix-System. Each of them opens a connection to a single controller ... now they sopped working after about 5-6 hours.
>> And again both of them at the same time. I have no info's in any log-file.
>>
>> The program on the IOC is very simple:
>>
>> dmc-file, rioDIag.dmc :
>>
>> 'Galil Motion Control epics@galilmc.com
>> 'http://www.galilmc.com/
>> #epics
>> 'Insert code to run on
>> ' GalilStartController() here...
>> MA 141,14,141,214;
>> MD junkes@fhi-berlin.mpg.de;
>> MS rioDiag@fhi-berlin.mpg.de;
>> WT 100
>> XQ#update,1;'spawn update thread
>> EN
>> '
>> #update;'Example update loop
>> AT0
>> epics[0]=TIME;'time tick
>> epics[1]=_OP0
>> epics[2]=_TI0
>> OPepics[9]
>> IF (epics[10])
>> SB8
>> WT50
>> CB8
>> epics[10]=0
>> ENDIF
>> AT200;'see SCAN field in
>> ' galil_array_access.substitutions
>> JP#update
>> '
>> #CMDERR;'runs if an error occurs
>> MG "An error occured at line",_ED
>> TC 1;'print type of error info
>> ZS;'Remove returns from callbck
>> EN;'End execution
>> '
>> #TCPERR
>> MG {M} "TCPERR.  Dropped handle", _IA4
>> RE
>>
>> started with this cmd-file, rioDiag.cmd:
>>
>> #Galil Motion Control epics@galilmc.com http://www.galilmc.com/
>>
>> #Load Galil Data Record Support
>> #dbLoadTemplate("$(TOP)/GalilDiagApp/Db/rioDiag_motor.substitutions")
>>
>> #Provide access to Galil Array Table
>> dbLoadTemplate("$(TOP)/GalilDiagApp/Db/rioDiag_array_access.substitutions")
>>
>> # GalilCreateController command parameters are:
>> # 1. Const char *portName - The name of the asyn port to be created for this controller
>> # 2. Const char *address  - The address of the controller
>> # 3. int numAxes          - The number of axis that will be used on the controller
>> # 4. double updatePeriod  - The time in ms between datarecords
>> #   (See controller's DR command for minimum).
>> #   Async if controller + bus supports it, otherwise is polled/synchronous.
>> GalilCreateController("rioDiag", "10.0.0.210", 1, 32)
>> #GalilCreateController("rioDiag", "141.14.128.41", 2, 8)
>>
>> # GalilCreateAxis command parameters are:
>> # 1. char *portName Asyn port for controller
>> # 2. char axis A-H,
>> GalilCreateAxis("rioDiag","A")
>> #GalilCreateAxis("rioDiag","B")
>> #GalilCreateAxis("Galil","C")
>> #GalilCreateAxis("Galil","D")
>> #GalilCreateAxis("Galil","E")
>> #GalilCreateAxis("Galil","F")
>> #GalilCreateAxis("Galil","G")
>> #GalilCreateAxis("Galil","H")
>>
>> # GalilStartController command parameters are:
>> # 1. char *portName Asyn port for controller
>> # 2. char *code file to deliver to the controller, "" for none.
>> # 3. char *label to execute on file, "" to not execute.
>> GalilStartController("rioDiag", "rioDiag.dmc", "#epics")
>>
>> and this st.cmd:
>>
>> #!../../bin/linux-x86_64/GalilDiagApp
>> #Galil Motion Control epics@galilmc.com http://www.galilmc.com/
>>
>> epicsEnvSet("ARCH","linux-x86_64")
>> epicsEnvSet("IOC","IOCRIODIAG")
>> epicsEnvSet("TOP","/srv/ioc/sys/$(IOC)")
>> #epicsEnvSet("SUPPORT","/home/epics/EPICS/support")
>> #epicsEnvSet("EPICS_BASE","/home/epics/EPICS/base")
>> #epicsEnvSet("ASYN","/home/epics/EPICS/support/asyn/asyn4-22")
>> #epicsEnvSet("MOTOR","/home/epics/EPICS/support/motor/motorR6-8")
>> epicsEnvSet("DISPLAY",":0")
>> epicsEnvSet("LOG_DIR", "/srv/ioc/log")
>>
>>
>> ## Register all support components
>> dbLoadDatabase("$(TOP)/dbd/GalilDiagApp.dbd",0,0)
>> GalilDiagApp_registerRecordDeviceDriver(pdbbase)
>>
>> # Configure an example controller
>> #< rioMezz.cmd
>> < rioDiag.cmd
>>
>> #dbLoadRecords ("$(TOP)/db/rioMezz.db","P=FHIFEL:,R=HELPER:,S=GUNRELAY:")
>>
>> dbLoadRecords("$(TOP)/db/attenuator.db","P=FHIFEL:,R=DIAG:,S=ATTENUATOR:")
>>
>> dbLoadRecords("$(TOP)/db/flipper.db","P=FHIFEL:,R=DIAG:,S=FLIP:")
>>
>> #dbLoadRecords("$(TOP)/db/checkAlive.db","RIO=rioMezz:")
>> dbLoadRecords("$(TOP)/db/checkAlive.db","RIO=rioDiag:")
>>
>>
>> ## autoSaveRestore setup
>> save_restoreSet_Debug(0)
>>
>> # status-PV prefix, so save_restore can find its status PV's.
>> save_restoreSet_status_prefix("${IOC}:")
>>
>> # ok to restore a save set that had missing values (no CA connection to PV)?
>> # ok to save a file if some CA connections are bad?
>> save_restoreSet_IncompleteSetsOk(1)
>>
>> #In the restore operation, a copy of the save file will be written.  The
>> # file name can look like "auto_settings.sav.bu", and be overwritten every
>> # reboot, or it can look like "auto_settings.sav_020306-083522" (this is what
>> # is meant by a dated backup file) and every reboot will write a new copy.
>> save_restoreSet_DatedBackupFiles(1)
>>
>> # specify where save files should go
>> set_savefile_path("/srv/ioc/log/as","/save")
>>
>> # specify where request files can be found
>> set_requestfile_path("/srv/ioc/log/as","/req")
>>
>> # Specify what save files should be restored when.
>> # Up to eight files can be specified for each pass.
>> set_pass0_restoreFile("${IOC}_0.sav")
>> set_pass1_restoreFile("${IOC}_1.sav")
>>
>> # Number of sequenced backup files (e.g., 'auto_settings.sav0') to write
>> save_restoreSet_NumSeqFiles(3)
>>
>> save_restoreSet_SeqPeriodInSeconds(600)
>>
>> # Time between failed .sav-file write and the retry.
>> save_restoreSet_RetrySeconds(60)
>>
>> iocInit()
>>
>> makeAutosaveFileFromDbInfo("/srv/ioc/log/as/req/${IOC}_0.req", "autosaveFields_pass0")
>> makeAutosaveFileFromDbInfo("/srv/ioc/log/as/req/${IOC}_1.req", "autosaveFields_pass1")
>>
>> create_monitor_set("${IOC}_0.req", 5, "P=${IOC}:")
>> create_monitor_set("${IOC}_1.req", 30, "P=${IOC}:")
>>
>> ## Report Installed and Configured I/O-Hardware Information
>> dbior 0 1 > ${LOG_DIR}/Database/${IOC}.dbior
>> dbhcr     > ${LOG_DIR}/Database/${IOC}.dbhcr
>> dbl       > ${LOG_DIR}/Database/${IOC}.dbl
>>
>>
>>
>> By the way, the mail-sending on RIO works only once or twice .. then it stops sending any mails
>>
>> Best regards,
>> Heinz
>>
>>
>> On 30.10.2014, at 01:17, Mark Clift <Mark.Clift@synchrotron.org.au> wrote:
>>
>>> Hi Qingru,
>>>
>>> I would guess that the multi-cast route is missing, which is required for the Galil driver you're using.
>>>
>>> We add the following line into /etc/rc.local
>>>
>>> route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
>>>
>>> The error simply says it could not poll the first 8 digital inputs with Galil TI (Tell input) command.
>>>
>>> Also there is a newer Galil driver designed to replace the one your using located here:
>>>
>>> http://motorapp.github.io/Galil-3-0/
>>>
>>> Best wishes,
>>>
>>> Mark
>>>
>>> Dr. Mark Clift
>>> Senior Controls Engineer
>>> Australian Synchrotron
>>> 800 Blackburn Road
>>> Clayton 3168
>>> Ph: +613 8540 4264
>>> Fax: +613 8540 4200
>>> Mail: Mark.Clift@Synchrotron.org.au
>>
>>
>