| 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 <2025> | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 <2025> |
| <== Date ==> | <== Thread ==> |
|---|
| Subject: | Re: Channel Access performance with RTEMS on MVME5500 |
| From: | "Wells, Alex \(DLSLtd,RAL,LSCI\) via Tech-talk" <tech-talk at aps.anl.gov> |
| To: | Heinz Junkes <junkes at fhi-berlin.mpg.de> |
| Cc: | "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov> |
| Date: | Thu, 20 Mar 2025 11:50:00 +0000 |
|
Hi Heinz,
Thank you for the information. There's definitely something fishy going on with the thread priorities somewhere; it seems that everything is being started with the same priority.
I will also note that we are building RTEMS with "USE_POSIX_THREAD_PRIORITY_SCHEDULING
= NO". This appears to be required, as we use EPICS Mutexes in an interrupt context, and if we leave this set to "YES" we see a crashes. Could this be related to the oddities below?
SR25A-PC-IOC-01> rt pthread
ID NAME SHED PRI STATE MODES EVENTS WAITINFO
------------------------------------------------------------------------------
0b010001 _main_ UPD 253 READY P:T:nA NONE
0b010002 NTPTimeSync UPD 253 SEM P:T:nA NONE
0b010003 taskwd UPD 253 SEM P:T:nA NONE
0b010004 errlog UPD 253 SEM P:T:nA NONE
0b010005 timerQueue UPD 253 SEM P:T:nA NONE
0b010006 cbLow UPD 253 SEM P:T:nA NONE
0b010007 cbMedium UPD 253 SEM P:T:nA NONE
0b010008 cbHigh UPD 253 SEM P:T:nA NONE
0b010009 dbCaLink UPD 253 SEM P:T:nA NONE
0b01000a Scan8002 UPD 253 TIME:IS P:T:nA NONE Nanosleep
0b01000b timerQueue UPD 253 SEM P:T:nA NONE
0b01000c psC0S0L0rp UPD 253 TIME:IS P:T:nA NONE Nanosleep
0b01000d psC0S0L0wo UPD 253 TIME:IS P:T:nA NONE Nanosleep
0b01000e psC0S0L0sw UPD 253 SEM P:T:nA NONE
0b01000f scanOnce UPD 253 SEM P:T:nA NONE
0b010010 scan-10 UPD 253 SEM P:T:nA NONE
0b010011 scan-5 UPD 253 SEM P:T:nA NONE
0b010012 scan-2 UPD 253 SEM P:T:nA NONE
0b010013 scan-1 UPD 253 SEM P:T:nA NONE
0b010014 scan-0.5 UPD 253 SEM P:T:nA NONE
0b010015 scan-0.2 UPD 253 SEM P:T:nA NONE
0b010016 scan-0.1 UPD 253 SEM P:T:nA NONE
0b010017 CAS-TCP UPD 253 SYSEV P:T:nA NONE
0b010018 CAS-UDP UPD 253 SYSEV P:T:nA NONE
0b010019 CAS-beacon UPD 253 TIME:IS P:T:nA NONE Nanosleep
0b01001a ipToAsciiProxy UPD 253 SEM P:T:nA NONE
0b01001b timerQueue UPD 253 SEM P:T:nA NONE
0b01001c CAC-UDP UPD 253 SYSEV P:T:nA NONE
0b01001d CAC-repeater UPD 253 SYSEV P:T:nA NONE
0b01001e CAS-event UPD 253 SEM P:T:nA NONE
0b01001f CAS-client UPD 253 SYSEV P:T:nA NONE
However, EPICS seems to be trying to say something else, but I must admit I don't understand what. I can see the EPICS priorities for each task matching with the "OSIPRI" column, but I don't know what the "OSSPRI" column means. Is
this the priority "2" that you mention?
SR25A-PC-IOC-01> epicsThreadShowAll
NAME EPICS ID PTHREAD ID OSIPRI OSSPRI STATE
_main_ 0x34c218 0 0 0 OK
NTPTimeSync 0x34d828 184614914 90 2 OK
taskwd 0x3907e0 184614915 10 2 OK
errlog 0x3d26e0 184614916 10 2 OK
timerQueue 0xbcbce0 184614917 70 2 OK
cbLow 0xbcdd90 184614918 59 2 OK
cbMedium 0xbcfe58 184614919 64 2 OK
cbHigh 0xbcffd0 184614920 71 2 OK
dbCaLink 0xbd0220 184614921 50 2 OK
Scan8002 0xbd06b8 184614922 10 2 OK
timerQueue 0xbd1170 184614923 58 2 OK
psC0S0L0rp 0x1236400 184614924 10 2 OK
psC0S0L0wo 0x12ba768 184614925 10 2 OK
psC0S0L0sw 0x133cae0 184614926 10 2 OK
scanOnce 0x13ca800 184614927 67 2 OK
scan-10 0x14cbe78 184614928 60 2 OK
scan-5 0x15cc0f8 184614929 61 2 OK
scan-2 0x16cc378 184614930 62 2 OK
scan-1 0x17cc5f8 184614931 63 2 OK
scan-0.5 0x18cc878 184614932 64 2 OK
scan-0.2 0x19ccf80 184614933 65 2 OK
scan-0.1 0x1acd200 184614934 66 2 OK
CAS-TCP 0x1bd64b0 184614935 18 2 OK
CAS-UDP 0x1bd65d0 184614936 16 2 OK
CAS-beacon 0x1ce7540 184614937 17 2 OK
ipToAsciiProxy 0x1d30278 184614938 10 2 OK
timerQueue 0x1e30798 184614939 51 2 OK
CAC-UDP 0x1ec0fd8 184614940 52 2 OK
CAC-repeater 0x1f41e10 184614941 10 2 OK
CAS-event 0x2033770 184614942 19 2 OK
CAS-client 0x20b3968 184614943 20 2 OK
Is there a setting somewhere in EPICS or RTEMS that may be causing RTEMS to schedule everything with the same priority?
Thanks,
Alex Wells From: Heinz Junkes Sent: Thursday, March 20, 2025 11:11 AM To: Wells, Alex (DLSLtd,RAL,LSCI) Cc: Michael Davidsaver; tech-talk at aps.anl.gov Subject: Re: Channel Access performance with RTEMS on MVME5500 Hello Alex,
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt
by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.This reminds me a little of a priority problem I noticed when we switched to POSIX_API. There is still a confusion in RTEMS with the assignment of POSIX-Prios and RTEMS-Prios. POSIX_Init is started with prio 2. This means the second highest priority after RTEMS, but with Posix it is a low priority. That's why it looks like this in rtems_init.c at the moment: ``` void * POSIX_Init ( void *argument __attribute__((unused))) { int result; char *argv[3] = { NULL, NULL, NULL }; rtems_status_code sc; struct timespec now; char timeBuff[100]; initConsole (); /* * Explain why we're here */ logReset(); /* * If RTEMS is used with the POSIX API, the init task * 'POSIX_Init()' is unfortunately given the priority '2'. This * corresponds to the second lowest POSIX prio (RTEMS pthread * prio 253). This task shoud have IOCsh prio. */ pthread_attr_t attr; struct sched_param param; int policy; sc = pthread_attr_init(&attr); assert(sc == RTEMS_SUCCESSFUL); sc = pthread_attr_getschedpolicy(&attr, &policy); assert(sc == RTEMS_SUCCESSFUL); param.sched_priority = (sched_get_priority_max(policy) - sched_get_priority_min(policy)) * epicsThreadPriorityIocsh / 100; sc = pthread_setschedparam(pthread_self(), policy, ¶m); assert(sc == RTEMS_SUCCESSFUL); … ``` Can you check with which priority the threads are really running? On the RTEMS shell with the command ‘pthread’, ’task’ (and ‘irq’): I use an ‘RTEMS-shell-wrapper’ from Michael on EPICS MVME6100, RTEMS6 wiht legacy-stack: IOC1> rt pthread ID NAME SHED PRI STATE MODES EVENTS WAITINFO ------------------------------------------------------------------------------ 0b010001 _main_ UPD 24 READY P:T:nA NONE 0b010002 scan-0.5 UPD 93 SEM P:T:nA NONE 0b010003 osdNTP_Runner UPD 27 SYSEV P:T:nA NONE 0b010004 taskwd UPD 229 SEM P:T:nA NONE 0b010005 osdNTP_Monitor UPD 128 SEM P:T:nA NONE 0b010006 NTPTimeSync UPD 27 SEM P:T:nA NONE 0b010007 errlog UPD 229 SEM P:T:nA NONE 0b010008 UPD 27 SEM P:T:nA NONE 0b010009 timerQueue UPD 103 SEM P:T:nA NONE 0b01000a timerQueue UPD 103 SEM P:T:nA NONE 0b01000b CAS-event UPD 206 SEM P:T:nA NONE 0b01000c awgport1 update UPD 100 SEM P:T:nA NONE 0b01000d CAS-client UPD 204 SYSEV P:T:nA NONE 0b01000e awgport2 update UPD 100 SEM P:T:nA NONE 0b01000f timerQueue UPD 77 SEM P:T:nA NONE 0b010010 cbLow UPD 105 SEM P:T:nA NONE 0b010011 cbMedium UPD 93 SEM P:T:nA NONE 0b010012 cbHigh UPD 75 SEM P:T:nA NONE 0b010013 dbCaLink UPD 128 SEM P:T:nA NONE 0b010014 PVAL UPD 128 SEM P:T:nA NONE 0b010015 timerQueue UPD 108 SEM P:T:nA NONE 0b010016 PDB-event UPD 206 SEM P:T:nA NONE 0b010017 pvAccess-client UPD 166 SEM P:T:nA NONE 0b010018 UDP-rx 0.0.0.0: UPD 128 SYSEV P:T:nA NONE 0b010019 UDP-rx 10.0.0.2 UPD 128 SYSEV P:T:nA NONE 0b01001a UDP-rx 10.0.1.2 UPD 128 SYSEV P:T:nA NONE 0b01001b UDP-rx 224.0.0. UPD 128 SYSEV P:T:nA NONE 0b01001c scanOnce UPD 85 SEM P:T:nA NONE 0b01001d scan-10 UPD 103 SEM P:T:nA NONE 0b01001e scan-5 UPD 100 SEM P:T:nA NONE 0b01001f scan-2 UPD 98 SEM P:T:nA NONE 0b010020 scan-1 UPD 95 SEM P:T:nA NONE 0b010021 scan-0.2 UPD 90 SEM P:T:nA NONE 0b010022 scan-0.1 UPD 88 SEM P:T:nA NONE 0b010023 CAS-TCP UPD 209 SYSEV P:T:nA NONE 0b010024 CAS-UDP UPD 214 SYSEV P:T:nA NONE 0b010025 CAS-beacon UPD 211 TIME:IS P:T:nA NONE Nanosleep 0b010026 CAC-event UPD 125 SEM P:T:nA NONE 0b010027 reccaster UPD 128 SYSEV P:T:nA NONE 0b010028 PVAS timers UPD 191 SEM P:T:nA NONE 0b010029 TCP-acceptor UPD 128 SYSEV P:T:nA NONE 0b01002a UDP-rx 0.0.0.0: UPD 128 SYSEV P:T:nA NONE 0b01002b UDP-rx 10.0.0.2 UPD 128 SYSEV P:T:nA NONE 0b01002c UDP-rx 10.0.1.2 UPD 128 SYSEV P:T:nA NONE 0b01002d UDP-rx 224.0.0. UPD 128 SYSEV P:T:nA NONE 0b01002e CAS-event UPD 206 SEM P:T:nA NONE 0b01002f CAS-client UPD 204 SYSEV P:T:nA NONE 0b010032 save_restore UPD 204 MSG P:T:nA NONE 1b010001 0b010033 CAC-event UPD 201 SEM P:T:nA NONE 0b010034 CAS-event UPD 206 SEM P:T:nA NONE 0b010035 CAS-client UPD 204 SYSEV P:T:nA NONE 0b010036 CAS-event UPD 206 SEM P:T:nA NONE 0b010037 CAS-client UPD 204 SYSEV P:T:nA NONE 0b010038 CAS-event UPD 206 SEM P:T:nA NONE 0b010039 CAS-client UPD 204 SYSEV P:T:nA NONE 0b01003a CAS-event UPD 206 SEM P:T:nA NONE 0b01003b CAS-client UPD 204 SYSEV P:T:nA NONE 0b01003d CAS-event UPD 206 SEM P:T:nA NONE 0b01003e CAS-event UPD 206 SEM P:T:nA NONE 0b01003f CAS-client UPD 204 SYSEV P:T:nA NONE 0b010040 CAS-event UPD 206 SEM P:T:nA NONE 0b010041 CAS-client UPD 204 SYSEV P:T:nA NONE 0b010042 CAS-event UPD 206 SEM P:T:nA NONE 0b010043 CAS-client UPD 204 SYSEV P:T:nA NONE 0b010044 CAS-event UPD 206 SEM P:T:nA NONE 0b010045 CAS-client UPD 204 SYSEV P:T:nA NONE 0b010046 CAS-event UPD 206 SEM P:T:nA NONE 0b010047 CAS-client UPD 204 SYSEV P:T:nA NONE 0b010048 CAS-event UPD 166 SEM P:T:nA NONE 0b010049 CAS-client UPD 163 SYSEV P:T:nA NONE 0b01004a CAS-event UPD 166 SEM P:T:nA NONE 0b01004b CAS-client UPD 163 SYSEV P:T:nA NONE 0b010050 CAS-client UPD 204 SYSEV P:T:nA NONE 0b010051 CAS-event UPD 206 SEM P:T:nA NONE 0b010052 CAS-client UPD 204 SYSEV P:T:nA NONE 0b010053 CAS-event UPD 206 SEM P:T:nA NONE 0b010054 CAS-client UPD 204 SYSEV P:T:nA NONE IOC1> IOC1> rt task ID NAME SHED PRI STATE MODES EVENTS WAITINFO ------------------------------------------------------------------------------ 0a010001 _BSD UPD 10 SYSEV P:T:nA NONE 0a010002 _BSD UPD 10 SYSEV P:T:nA NONE 0a010003 RPCd UPD 10 EV P:T:nA NONE 0a010004 TNTa UPD 100 SYSEV P:T:nA NONE 0a010005 TNTb UPD 100 SYSEV P:T:nA NONE 0a010006 TNTc UPD 100 SYSEV P:T:nA NONE 0a010007 TNTd UPD 100 SYSEV P:T:nA NONE 0a010008 TNTe UPD 100 SYSEV P:T:nA NONE 0a010009 TNTD UPD 100 SYSEV P:T:nA NONE IOC1> rt irq ------------------------------------------------------------------------------- INTERRUPT INFORMATION --------+----------------------------------+---------+------------+------------ VECTOR | INFO | OPTIONS | HANDLER | ARGUMENT --------+----------------------------------+---------+------------+------------ --------+----------------------------------+---------+------------+—————— ( Example on a beaglebone black with libbsd-stack ) beaglebone> rt pthread ID NAME SHED PRI STATE MODES EVENTS WAITINFO ------------------------------------------------------------------------------ 0b010001 _main_ UPD 25 READY P:T:nA 00002000 0b010003 errlog UPD 229 SEM P:T:nA NONE 0b010004 osdNTP_Runner UPD 27 READY P:T:nA NONE 0b010005 osdNTP_Monitor UPD 128 SEM P:T:nA NONE 0b010006 NTPTimeSync UPD 27 SEM P:T:nA NONE 0b010007 taskwd UPD 229 SEM P:T:nA NONE 0b010008 UPD 27 SEM P:T:nA NONE 0b010009 timerQueue UPD 77 SEM P:T:nA NONE 0b01000a cbLow UPD 105 SEM P:T:nA NONE 0b01000b cbMedium UPD 93 SEM P:T:nA NONE 0b01000c cbHigh UPD 75 SEM P:T:nA NONE 0b01000d dbCaLink UPD 128 SEM P:T:nA NONE 0b01000e scanOnce UPD 85 SEM P:T:nA NONE 0b01000f scan-10 UPD 103 SEM P:T:nA NONE 0b010010 scan-5 UPD 100 SEM P:T:nA NONE 0b010011 scan-2 UPD 98 SEM P:T:nA NONE 0b010012 scan-1 UPD 95 READY P:T:nA NONE 0b010013 scan-0.5 UPD 93 SEM P:T:nA NONE 0b010014 scan-0.2 UPD 90 READY P:T:nA NONE 0b010015 scan-0.1 UPD 88 READY P:T:nA NONE 0b010016 CAS-TCP UPD 209 WK P:T:nA NONE accept 0b010017 CAS-UDP UPD 214 WK P:T:nA NONE sbwait 0b010018 CAS-beacon UPD 211 TIME:IS P:T:nA NONE Nanosleep beaglebone> rt task ID NAME SHED PRI STATE MODES EVENTS WAITINFO ------------------------------------------------------------------------------ 0a010001 TIME UPD 98 READY P:T:nA NONE 0a010002 IRQS UPD 96 READY P:T:nA NONE 0a010003 _BSD swi6: Giant tas UPD 100 EV P:T:nA NONE 0a010004 _BSD config_0 UPD 100 WK P:T:nA NONE - 0a010005 _BSD kqueue_ctx task UPD 100 WK P:T:nA NONE - 0a010006 _BSD swi6: task queu UPD 100 EV P:T:nA NONE 0a010007 _BSD swi5: fast task UPD 100 EV P:T:nA NONE 0a010008 _BSD thread taskq UPD 100 WK P:T:nA NONE - 0a010009 _BSD swi1: netisr 0 UPD 100 EV P:T:nA NONE 0a01000a _BSD usbus0 UPD 100 WK P:T:nA NONE - 0a01000b _BSD usbus0 UPD 100 WK P:T:nA NONE - 0a01000c _BSD usbus0 UPD 100 WK P:T:nA NONE - 0a01000d _BSD usbus0 UPD 100 WK P:T:nA NONE - 0a01000e _BSD usbus0 UPD 100 WK P:T:nA NONE - 0a01000f _BSD usbus1 UPD 100 WK P:T:nA NONE - 0a010010 _BSD usbus1 UPD 100 WK P:T:nA NONE - 0a010011 _BSD usbus1 UPD 100 WK P:T:nA NONE - 0a010012 _BSD usbus1 UPD 100 WK P:T:nA NONE - 0a010013 _BSD usbus1 UPD 100 WK P:T:nA NONE - 0a010014 _BSD bufdaemon UPD 100 READY P:T:nA NONE 0a010015 _BSD vnlru UPD 100 WK P:T:nA NONE vlruwt 0a010016 _BSD syncer UPD 100 WK P:T:nA NONE syncer 0a010017 _BSD softirq_0 UPD 100 WK P:T:nA NONE - 0a010018 DHCP UPD 254 WK P:T:nA NONE select 0a010019 _BSD bufspacedaemon- UPD 100 READY P:T:nA NONE 0a01001a _BSD nfscl UPD 100 WK P:T:nA NONE nfscl 0a01001b TNTa UPD 100 SYSEV P:T:nA NONE 0a01001c TNTb UPD 100 SYSEV P:T:nA NONE 0a01001d TNTc UPD 100 SYSEV P:T:nA NONE 0a01001e TNTd UPD 100 SYSEV P:T:nA NONE 0a01001f TNTe UPD 100 SYSEV P:T:nA NONE 0a010020 TNTD UPD 100 WK P:T:nA NONE accept beaglebone> rt irq ------------------------------------------------------------------------------- INTERRUPT INFORMATION --------+----------------------------------+---------+------------+------------ VECTOR | INFO | OPTIONS | HANDLER | ARGUMENT --------+----------------------------------+---------+------------+------------ 18 | IRQS | SHARED | 0x803d6100 | 0x8072b4a8 19 | IRQS | SHARED | 0x803d6100 | 0x80757410 28 | IRQS | SHARED | 0x803d6100 | 0x807287f8 30 | BBB_I2C | UNIQUE | 0x803d6b6c | 0x8065e5c8 40 | IRQS | SHARED | 0x803d6100 | 0x8078a4d0 41 | IRQS | SHARED | 0x803d6100 | 0x8078a538 42 | IRQS | SHARED | 0x803d6100 | 0x8078a5a0 43 | IRQS | SHARED | 0x803d6100 | 0x8078a608 64 | IRQS | SHARED | 0x803d6100 | 0x80727a90 67 | Clock | UNIQUE | 0x803d381c | 0x0 70 | BBB_I2C | UNIQUE | 0x803d6b6c | 0x8065e4f0 72 | NS16550 | SHARED | 0x803d4db8 | 0x0 --------+----------------------------------+---------+------------+—————— Heinz ------------------------------------------------------------------------------ Fritz-Haber-Institut | Phone: (+49 30) 8413-4270 Heinz Junkes | Fax (G3+G4): (+49 30) 8413-5900 Faradayweg 4-6 | VC: https://zoom.fhi.berlin/junkes D - 14195 Berlin | E-Mail: junkes at fhi-berlin.mpg.de ------------------------------------------------------------------------------ “Sorry I’m a bit late, had a terrible time… All sort of things cropping up at the last moment. Uh, how are we for time?” —Zarquon's address to Milliways > On 18. Mar 2025, at 17:41, Wells, Alex (DLSLtd,RAL,LSCI) via Tech-talk <tech-talk at aps.anl.gov> wrote: > > Hi Michael, > > I can confirm the CPU use goes to zero when the network cable is unplugged. We do image and file loading over the network, but once all that has completed I can pull the cable out and observe no CPU use by that thread. The system also becomes fully responsive on the console, which correlates the significant drop in CPU use. > > I have only tested RTEMS 5.1, which I believe is using the "legacy" IP stack. In a few months, during the next shutdown, I may be able to test RTEMS6 but we have no particular reason to expect the newer stack to work better, as the MVME5500 is 30 years old now so the new stack is unlikely to be optimized for its use case. > > Thanks, > Alex WellsFrom: Michael Davidsaver <mdavidsaver at gmail.com> > Sent: Friday, March 14, 2025 4:39 PM > To: Wells, Alex (DLSLtd,RAL,LSCI) <alex.wells at diamond.ac.uk> > Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov> > Subject: Re: Channel Access performance with RTEMS on MVME5500 > Hello Alex, > > As a quick test, have you tried unplugging the ethernet cable to ensure that > the reported CPU usage for CAS-UDP goes to zero? > > Also, which RTEMS version(s) have you tested? > > Is this the RTEMS "legacy" IP stack, or the newer libbsd stack? > > > On 3/14/25 06:30, Wells, Alex (DLSLtd,RAL,LSCI) via Tech-talk wrote: > > Hello Tech Talk, > > > > I'm in the process of moving some VME crates from VxWorks to RTEMS. This also moves from EPICS 3.14.12.7 to EPICS 7.0.7. I'm seeing significant amounts of CPU activity in the "CAS-UDP" thread, and was hoping someone may be able to explain it. > > > > Averaged over time, RTEMS reports that the CAS-UDP thread is using ~40% of the CPU. This persists even when removing all the processing from all records (thus removing all other possible CPU use). During initialization of the IOC, and for several minutes after initialization finished, I see 80%+ CPU usage on the IOC overall, mostly on this one thread, and it makes the overall IOC very unresponsive. This same behaviour also happens intermittently during operation, where even record processing seems to become delayed due to Channel Access processing - for example the devIocStats Heartbeat record that should just increase by 1 per second sometimes pauses for multiple seconds. > > > > The network the IOC is on is requesting a fair number of PVs from this IOC. CaSnooper shows between 20 and 40 requests per second to PV(s) on this IOC. Approximately 500 PVs are being requested from the IOC. These requests are spread across approximately 30 separate clients. > > > > When I compare this to the VxWorks version, running on the same hardware and network, I see less almost 0% CPU usage on the CAS-UDP thread, and no record processing hitches. The CPU also idles significantly lower than for RTEMS. > > > > Our current theory is that the RTEMS network stack for UDP processing is much less efficient, and is struggling with the volume of requests. Is anyone able to confirm/deny this, and does anyone have any potential workarounds? > > > > Thanks, > > Alex Wells > > Diamond Light Source > > > > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. > > Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom. > > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom. |