RE: galil softIOC lose the galil controller

[Mark Clift <Mark.Clift@synchrotron.org.au>]

Hi, 

The advice to use a private network is a good idea.  Also, when exiting the softIOC type "exit" so that the exit handler turns off the RIO data record at IOC shutdown.  If the softIOC is killed with ^C or Kill, the RIO will continue to send data records with the IOC down, which may cause issues?

Also is the RIO stable when connected only to GalilTools?

I'll investigate further with one of our spare RIO's.

Best wishes,

Mark

Senior Controls Engineer
Australian Synchrotron
800 Blackburn Road
Clayton 3168
Ph: +613 8540 4264
Fax: +613 8540 4200
Mail: Mark.Clift@Synchrotron.org.au
________________________________________
From: Pearson, Matthew R. [pearsonmr@ornl.gov]
Sent: Friday, 31 October 2014 00:44
To: Mark Clift
Cc: Heinz Junkes; tech-talk@aps.anl.gov (tech-talk@aps.anl.gov)
Subject: Re: galil softIOC lose the galil controller

Hi,

This may or may not be related, but we failed to have any kind reliability with our two Gaili DMC-40x0 series controllers as long as they were on an open network with broadcast traffic.

On our general purpose office network, the controllers would lock up their network interface after a few minutes. The error LED comes on, and only a power cycle fixes it. On the single beamline where we have the two controllers this happened every few days or couple of weeks. Eventually we placed the two controllers on a private network interface connected directly to a Linux server, and we haven't had any issues since.

We're using Marks original motor driver, but that wasn't related to the controller network problems. The controllers locked up even without any EPICS software talking to the controllers, which points to problem broadcast traffic.

We did notify Galil about the issue, but once we resolved it we stopped investigating it here. I know others have had the same issue with these controllers. I bet it's the same issue if the RIO-47xx is using the same ethernet controller. We found that it was listening by default on 1000s of port numbers, and any one of them accepted a TCP connection (up to 8 at a time I think).


Cheers,
Matt

Data Acquisition and Control Engineer
Spallation Neutron Source
Oak Ridge National Lab


On Oct 30, 2014, at 4:47 AM, Mark Clift <Mark.Clift@synchrotron.org.au> wrote:

> Hi Heinz,
>
> Please follow this setup procedure for the RIO PLC.
>
> 1.  Use GalilTools to set the IP address
> 2.  Record the MAC address of the RIO
> 3.  Edit /etc/rc.d/rc.local and add the following:
>
> # Add multicast route for Galil hardware
> route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
>
> # Add arp entry for Galil RIO device
> arp -s <IPADDRESS> <MACADDRESS>
>
> 4.  Reboot softIOC or type above commands as root, then exit root login
> 5.  Telnet into RIO from softIOC, and issue the following commands
>
> :IB1   (Possibly IK1 on some firmware)
> :BN   (Burn the changes)
> :^M
> exit
>
> 6.  From now on only hosts with the arp entry can telnet into RIO
>
> Then try the IOC again.  I recommend using a single IOC and connection to the RIO.
>
> Also try the email again.
>
> Best wishes,
>
> Mark.
>
> Dr. Mark Clift
> Senior Controls Engineer
> Australian Synchrotron
> 800 Blackburn Road
> Clayton 3168
> Ph: +613 8540 4264
> Fax: +613 8540 4200
> Mail: Mark.Clift@Synchrotron.org.au
> ________________________________________
> From: Heinz Junkes [junkes@fhi-berlin.mpg.de]
> Sent: Thursday, 30 October 2014 19:16
> To: Mark Clift
> Cc: tech-talk@aps.anl.gov (tech-talk@aps.anl.gov)
> Subject: Re: galil softIOC lose the galil controller
>
> Hi techtalk,
>
> we are running two galil RIO-47120 controllers. We are using the newest versions of galilsuite and galiltools on debian linux.
>
> ii  galilsuite                          1.0.2                               amd64        Test GalilSuite build
> ii  galiltools                          1.6.3                               amd64        GalilTools
>
> and the Galil-3-0 motorapp.
>
> We could'nt get them running stable. Until the day before yesterday I connected to both controllers with one softioc. About every second week they stopped working and the controllers could'nt be ping-ed anymore. We have to reset the controllers. Both stopped at the same time.
> Now I run two softioc's on the same Unix-System. Each of them opens a connection to a single controller ... now they sopped working after about 5-6 hours.
> And again both of them at the same time. I have no info's in any log-file.
>
> The program on the IOC is very simple:
>
> dmc-file, rioDIag.dmc :
>
> 'Galil Motion Control epics@galilmc.com
> 'http://www.galilmc.com/
> #epics
> 'Insert code to run on
> ' GalilStartController() here...
> MA 141,14,141,214;
> MD junkes@fhi-berlin.mpg.de;
> MS rioDiag@fhi-berlin.mpg.de;
> WT 100
> XQ#update,1;'spawn update thread
> EN
> '
> #update;'Example update loop
> AT0
> epics[0]=TIME;'time tick
> epics[1]=_OP0
> epics[2]=_TI0
> OPepics[9]
> IF (epics[10])
> SB8
> WT50
> CB8
> epics[10]=0
> ENDIF
> AT200;'see SCAN field in
> ' galil_array_access.substitutions
> JP#update
> '
> #CMDERR;'runs if an error occurs
> MG "An error occured at line",_ED
> TC 1;'print type of error info
> ZS;'Remove returns from callbck
> EN;'End execution
> '
> #TCPERR
> MG {M} "TCPERR.  Dropped handle", _IA4
> RE
>
> started with this cmd-file, rioDiag.cmd:
>
> #Galil Motion Control epics@galilmc.com http://www.galilmc.com/
>
> #Load Galil Data Record Support
> #dbLoadTemplate("$(TOP)/GalilDiagApp/Db/rioDiag_motor.substitutions")
>
> #Provide access to Galil Array Table
> dbLoadTemplate("$(TOP)/GalilDiagApp/Db/rioDiag_array_access.substitutions")
>
> # GalilCreateController command parameters are:
> # 1. Const char *portName - The name of the asyn port to be created for this controller
> # 2. Const char *address  - The address of the controller
> # 3. int numAxes          - The number of axis that will be used on the controller
> # 4. double updatePeriod  - The time in ms between datarecords
> #   (See controller's DR command for minimum).
> #   Async if controller + bus supports it, otherwise is polled/synchronous.
> GalilCreateController("rioDiag", "10.0.0.210", 1, 32)
> #GalilCreateController("rioDiag", "141.14.128.41", 2, 8)
>
> # GalilCreateAxis command parameters are:
> # 1. char *portName Asyn port for controller
> # 2. char axis A-H,
> GalilCreateAxis("rioDiag","A")
> #GalilCreateAxis("rioDiag","B")
> #GalilCreateAxis("Galil","C")
> #GalilCreateAxis("Galil","D")
> #GalilCreateAxis("Galil","E")
> #GalilCreateAxis("Galil","F")
> #GalilCreateAxis("Galil","G")
> #GalilCreateAxis("Galil","H")
>
> # GalilStartController command parameters are:
> # 1. char *portName Asyn port for controller
> # 2. char *code file to deliver to the controller, "" for none.
> # 3. char *label to execute on file, "" to not execute.
> GalilStartController("rioDiag", "rioDiag.dmc", "#epics")
>
> and this st.cmd:
>
> #!../../bin/linux-x86_64/GalilDiagApp
> #Galil Motion Control epics@galilmc.com http://www.galilmc.com/
>
> epicsEnvSet("ARCH","linux-x86_64")
> epicsEnvSet("IOC","IOCRIODIAG")
> epicsEnvSet("TOP","/srv/ioc/sys/$(IOC)")
> #epicsEnvSet("SUPPORT","/home/epics/EPICS/support")
> #epicsEnvSet("EPICS_BASE","/home/epics/EPICS/base")
> #epicsEnvSet("ASYN","/home/epics/EPICS/support/asyn/asyn4-22")
> #epicsEnvSet("MOTOR","/home/epics/EPICS/support/motor/motorR6-8")
> epicsEnvSet("DISPLAY",":0")
> epicsEnvSet("LOG_DIR", "/srv/ioc/log")
>
>
> ## Register all support components
> dbLoadDatabase("$(TOP)/dbd/GalilDiagApp.dbd",0,0)
> GalilDiagApp_registerRecordDeviceDriver(pdbbase)
>
> # Configure an example controller
> #< rioMezz.cmd
> < rioDiag.cmd
>
> #dbLoadRecords ("$(TOP)/db/rioMezz.db","P=FHIFEL:,R=HELPER:,S=GUNRELAY:")
>
> dbLoadRecords("$(TOP)/db/attenuator.db","P=FHIFEL:,R=DIAG:,S=ATTENUATOR:")
>
> dbLoadRecords("$(TOP)/db/flipper.db","P=FHIFEL:,R=DIAG:,S=FLIP:")
>
> #dbLoadRecords("$(TOP)/db/checkAlive.db","RIO=rioMezz:")
> dbLoadRecords("$(TOP)/db/checkAlive.db","RIO=rioDiag:")
>
>
> ## autoSaveRestore setup
> save_restoreSet_Debug(0)
>
> # status-PV prefix, so save_restore can find its status PV's.
> save_restoreSet_status_prefix("${IOC}:")
>
> # ok to restore a save set that had missing values (no CA connection to PV)?
> # ok to save a file if some CA connections are bad?
> save_restoreSet_IncompleteSetsOk(1)
>
> #In the restore operation, a copy of the save file will be written.  The
> # file name can look like "auto_settings.sav.bu", and be overwritten every
> # reboot, or it can look like "auto_settings.sav_020306-083522" (this is what
> # is meant by a dated backup file) and every reboot will write a new copy.
> save_restoreSet_DatedBackupFiles(1)
>
> # specify where save files should go
> set_savefile_path("/srv/ioc/log/as","/save")
>
> # specify where request files can be found
> set_requestfile_path("/srv/ioc/log/as","/req")
>
> # Specify what save files should be restored when.
> # Up to eight files can be specified for each pass.
> set_pass0_restoreFile("${IOC}_0.sav")
> set_pass1_restoreFile("${IOC}_1.sav")
>
> # Number of sequenced backup files (e.g., 'auto_settings.sav0') to write
> save_restoreSet_NumSeqFiles(3)
>
> save_restoreSet_SeqPeriodInSeconds(600)
>
> # Time between failed .sav-file write and the retry.
> save_restoreSet_RetrySeconds(60)
>
> iocInit()
>
> makeAutosaveFileFromDbInfo("/srv/ioc/log/as/req/${IOC}_0.req", "autosaveFields_pass0")
> makeAutosaveFileFromDbInfo("/srv/ioc/log/as/req/${IOC}_1.req", "autosaveFields_pass1")
>
> create_monitor_set("${IOC}_0.req", 5, "P=${IOC}:")
> create_monitor_set("${IOC}_1.req", 30, "P=${IOC}:")
>
> ## Report Installed and Configured I/O-Hardware Information
> dbior 0 1 > ${LOG_DIR}/Database/${IOC}.dbior
> dbhcr     > ${LOG_DIR}/Database/${IOC}.dbhcr
> dbl       > ${LOG_DIR}/Database/${IOC}.dbl
>
>
>
> By the way, the mail-sending on RIO works only once or twice .. then it stops sending any mails
>
> Best regards,
> Heinz
>
>
> On 30.10.2014, at 01:17, Mark Clift <Mark.Clift@synchrotron.org.au> wrote:
>
>> Hi Qingru,
>>
>> I would guess that the multi-cast route is missing, which is required for the Galil driver you're using.
>>
>> We add the following line into /etc/rc.local
>>
>> route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
>>
>> The error simply says it could not poll the first 8 digital inputs with Galil TI (Tell input) command.
>>
>> Also there is a newer Galil driver designed to replace the one your using located here:
>>
>> http://motorapp.github.io/Galil-3-0/
>>
>> Best wishes,
>>
>> Mark
>>
>> Dr. Mark Clift
>> Senior Controls Engineer
>> Australian Synchrotron
>> 800 Blackburn Road
>> Clayton 3168
>> Ph: +613 8540 4264
>> Fax: +613 8540 4200
>> Mail: Mark.Clift@Synchrotron.org.au
>
>