NFS performance problems on vxWorks

[Dirk Zimoch <dirk.zimoch@psi.ch>]

Hi all,

our vxWorks IOCs mount their boot directory, auto-save-and-restore 
directory and data aquisition directory via NFS.

Now, I noticed a very poor performance on several IOCs. Test 
performance, I have a 10MB file in a mounted directory and copy it to 
the null device:

timex copy,"/work/bench/big10","/null"

or

sp timex,copy,"/work/bench/big10","/null"

In the best case, it takes about 4 to 6 seconds, but it can also take up 
to 16, 25, 60, even 300 seconds. On some IOCs I can't even read the file 
at all (copy: error reading file after copying 5619712 bytes.)

It does not matter if the copy runs in the background or in the foreground.

With tcpdump, we found that from time to time while reading the file, 
the IOC repeats an NFS read request for a block of data after 5 seconds. 
This leads to the assumption that the IOC had not received the data 
block. Every read problem adds another 5 seconds to the execution time.

I have also tried to write 10MB to an NFS directory. There are the same 
problems. From time to time, the file stops growing for a few seconds.

I found that on newer IOCs (MVME5110) running vxWorks 5.5, the problem 
is less frequent than on MVME2306 running vxWorks 5.3, but I also found 
fast MVME2306s and slow MVME5110s.

Different IOCs on the same switch mounting from the same PC behave 
differently.

When only vxWorks is running but not EPICS, it seems to be ok. If an 
empty EPICS (no records) is running, there few errors. The problem does 
not depend on the EPICS version (I tried 3.13.2 and 3.13.9). The idle 
time of the ioc is about 80%.

Unforunately, I don't know for how long the performance is that bad. 
Maybe it was always like now.


Does anyone have any idea?

What is your experience with NFS?


Dirk

--
Dr. Dirk Zimoch
Swiss Light Source
Paul Scherrer Institut
Computing and Controls
phone +41 56 310 5182
fax   +41 56 310 4413