EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <20222023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <20222023  2024 
<== Date ==> <== Thread ==>

Subject: Re: mrfioc2-PCI device Configuration
From: Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov>
To: gjzhai at mail.ustc.edu.cn
Cc: "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Date: Mon, 28 Feb 2022 08:56:08 -0800
On 2/28/22 05:27, gjzhai at mail.ustc.edu.cn wrote:
Some file permissions need to be changed.  The best way to handle this is with a udev rule.

cat << EOF > /etc/udev/rules.d/99-mrfioc2.rules
KERNEL=="uio*", ATTR{name}=="mrf-pci", GROUP="softioc", MODE="0660"
EOF
udevadm control --reload

It is also be worth noting that DKMS packaging is now include (added by Michael Abbot @DLS)
to make OS kernel updates more automatic.

In any case, you will need to choose an appropriate GROUP name for your site.

After I followed the above, the following appeared:
mrmEvrSetupPCI("MTCAEVR","5:0.0")
Notice: devPCIFindSpec() expect B:D.F in hex
Device MTCAEVR  5:0.0 slot=10
Using IRQ 162
Failed to open resource file for PCI device 0000:05:00.0: Permission denied
FWVersion 0x18100207
Found version 519
Found SFP EEPROM
Sequencer capability detected
mTCA: Out FP:4 FPUNIV:0 RB:16 IFP:32 GPIO:2
EVR FIFO task start
Enabling interrupts

Why does the error message of Permission denied still appear?

The underlying devlib2 PCI access library knows two ways to access a PCI
BAR under Linux.  With /dev/uio* and with another special file deep down
in /sys.  The are equivalent.  The second is (currently) tried first.

The /sys method doesn't require an OS driver.  So it can be sufficient for
simple devices, or for troubleshooting (or some other strange situations).

In addition, the geographic address of the EVM is 08:00.0, but the following error message appears:
mrmEvgSetupPCI("EVG", 8, 0, 0)
Error: invalid spec '8'
PCI Device not found


The four argument form of mrmEvgSetupPCI(NAME, BUS, DEV, FUNC) is legacy.
Back when I thought PCI addressing was always as simple as BUS:DEV.FUNC .
I tried to keep it working, but you have encountered a pathological case
with DEV==FUNC==0.

https://github.com/epics-modules/mrfioc2/blob/a8cb48d549b7aae13206cf2d168c6f1cfec2e1c4/evgMrmApp/src/evgInit.cpp#L371-L382

I recommend switching to the newer two argument form.

mrmEvgSetupPCI("EVG", "8:0.0")



Although seeing "slot=" indicates that geographical addressing is available.
I tend to prefer putting slot numbers in config files when possible.

...
Device MTCAEVR  5:0.0 slot=10
...

So

mrmEvrSetupPCI("MTCAEVR","5:0.0")

could be replaced with

mrmEvrSetupPCI("MTCAEVR","slot=10")


The devPCIShow() IOCsh function should show any slot numbers.

http://epics.sourceforge.net/devlib2/iocsh.html#iocshpci


In my experience.  While PCI slot numbers don't always map to physical slots in
an intuitive way, that mapping is at least as stable as B:D.F addressing.
Typically only backplane bus segments provide slot numbering.  So slot numbers
aren't effected by the addition or removal of a bus segment with a lower BUS
number.

Slot number still isn't 100%.  Things get weird when two bus segments provide
overlapping slot number ranges.  eg. a cPCI CPU with on board PMC sockets as
well as the backplane.


-----原始邮件-----
发件人: "Michael Davidsaver" <mdavidsaver at gmail.com>
发送时间: 2022-02-28 01:49:27 (星期一)
收件人: gjzhai at mail.ustc.edu.cn
抄送: "GAGET Alexis" <alexis.gaget at cea.fr>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
主题: Re: mrfioc2-PCI device Configuration

On 2/26/22 23:50, gjzhai at mail.ustc.edu.cn wrote:
...
Failed to open resource file for PCI device 0000:09:00.0: Permission denied
Can neither mmap resource file nor uio file of PCI device 0000:09:00.0 BAR 0
PCI error: Failed to map BAR 0

Some file permissions need to be changed.  The best way to handle this is with a udev rule.
There is an example in the README

https://github.com/epics-modules/mrfioc2/tree/master/mrmShared/linux

cat << EOF > /etc/udev/rules.d/99-mrfioc2.rules
KERNEL=="uio*", ATTR{name}=="mrf-pci", GROUP="softioc", MODE="0660"
EOF
udevadm control --reload

It is also be worth noting that DKMS packaging is now include (added by Michael Abbot @DLS)
to make OS kernel updates more automatic.

In any case, you will need to choose an appropriate GROUP name for your site.


mrmEvgSetupPCI("EVG", 8, 0, 0)
Error: invalid spec '8'
PCI Device not found

More information:
[opi@localhost dev]$ ll
crw-rw----.  1 root root    241,     0 Feb 27 10:39 uio0
crw-rw----.  1 root root    241,     1 Feb 27 10:39 uio1
crw-rw----.  1 root root    241,     2 Feb 27 10:39 uio2
[opi@localhost]$ sudo udevadm info -q path -n /dev/uio0
/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/0000:04:00.0/0000:05:00.0/uio/uio0
[opi@localhost]$ sudo udevadm info -q path -n /dev/uio1
/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/0000:04:03.0/0000:08:00.0/uio/uio1
[opi@localhost]$ sudo udevadm info -q path -n /dev/uio2
/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/0000:04:04.0/0000:09:00.0/uio/uio2

-----原始邮件-----
发件人: "Michael Davidsaver" <mdavidsaver at gmail.com>
发送时间: 2022-01-24 01:03:06 (星期一)
收件人: gjzhai at mail.ustc.edu.cn
抄送: "GAGET Alexis" <alexis.gaget at cea.fr>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
主题: Re: mrfioc2-PCI device Configuration

On 1/23/22 01:15, gjzhai at mail.ustc.edu.cn wrote:
...
Can't read private key
     DEPMOD  3.10.0-1160.42.2.el7.x86_64
...
[root@localhost mrmShared]# dmesg | grep mrf
[    1.338766] pci_mrfevr: loading out-of-tree module taints kernel.
[    1.339091] pci_mrfevr: module verification failed: signature and/or required key missing - tainting kernel

I guess your (RHEL7) system has kernel module signing at least partially enabled?
Personally, I have no experience with this feature.  So I can't advise you on how
to sign locally built kernel modules.

[    1.340025] mrfevr:BAR0 start df200000 end df23ffff, mmap c3700000
[    1.340296] mrfevr:BAR0 start df000000 end df03ffff, mmap c3780000
[    1.340432] mrfevg:BAR0 start df100000 end df17ffff mmaped c3800000

[root@localhost mrmShared]# lsmod |grep mrf
mrf                    18421  0
uio                    19338  1 mrf
parport                46395  1 mrf
pci_mrfevg             53731  0
pci_mrfevr             53731  0

It looks like you have two different drivers for MRF cards loaded.
The Linux driver which is part of mrfioc2 is independent of, and
conflicts with, all other Linux drivers for MRF cards.

You should remove the pci_mrfevg and pci_mrfevr.  It would be best
to reboot, then load only mrf.ko build from mrfioc2.

If all goes well you should see a line like the following in the kernel log.

[   10.000000] mrf-pci 0000:02:01.0: MRF Setup complete

https://github.com/epics-modules/mrfioc2/blob/a8cb48d549b7aae13206cf2d168c6f1cfec2e1c4/mrmShared/linux/uio_mrf.c#L675


Thanks

-----原始邮件-----
发件人: "Michael Davidsaver" <mdavidsaver at gmail.com>
发送时间: 2022-01-21 04:26:28 (星期五)
收件人: "gjzhai at mail.ustc.edu.cn" <gjzhai at mail.ustc.edu.cn>
抄送: "GAGET Alexis" <alexis.gaget at cea.fr>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>[root@localhost
主题: Re: mrfioc2-PCI device Configuration

On 1/20/22 7:31 AM, GAGET Alexis via Tech-talk wrote:
Hi,

I got this error a lot.

Many points :

- is the kernel module well loaded ? lsmod | grep mrf

If the kernel module is being loaded, but the /dev/uio* devices
are not being created, check the kernel log ("dmesg") for lines
including "uio_mrf" and/or the specific PCI B:D.F.

- Depending on the CPU you use, you need to configure your PCIe virtual switch correctly.
    for me with NAT COMEx and NAT MCH it's like that (in the MCH webserver) :


With my configuration that is MTCA EVM300 andMTCA EVR300U I also had this problem because the CPU (NAT COMEX) hosting my EPICS IOC booted after the EVR and EVM. So the PCI configuration was not well done and produced this exact message you write below.

A manual solution is to reboot manually just your CPU to see if it's the problem. If it is, for me in the MCH (NAT MCH PHYS80) I have an argument "upstream slot power delay" that I put to 40 sec, meaning that my COMEx in RTM boot 40sec after the crate boot. (this work only with the latest version of the NAT MCH firmware)


ps : I prefer to use "lspci -nmmn" to list my PCIe device

fyi. I keep a collection of lspci output from cards I've encountered.
These were captured when correctly configured and running.  Maybe
helpful as a point of comparison.

https://github.com/epics-modules/mrfioc2/tree/master/documentation/lspci


Cheers


Alexis GAGET

CEA Saclay DRF/IRFU/DIS/LDISC

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*De :* Tech-talk <tech-talk-bounces at aps.anl.gov> de la part de gjzhai--- via Tech-talk <tech-talk at aps.anl.gov>
*Envoyé :* jeudi 20 janvier 2022 14:52:33
*À :* tech-talk at aps.anl.gov
*Objet :* mrfioc2-PCI device Configuration
Hello all I am trying to get MRF MTCA-EVG-300 and MTCA-EVR-300U configured and working on Linux using mrfioc2. I found geographic address(bus:device.function) in /proc/bus/pci/devices. The information is
    as follows:

0800 10ee7011 10 df100000 0 0 0 0 0 0 80000 0 0 0 0 0 0 pci_mrfevg

0900 10ee7011 11 df000000 0 0 0 0 0 0 40000 0 0 0 0 0 0 pci_mrfevr



According to the above information, my st.cmd in the iocevrmrm folder is defined as follows:

mrmEvrSetupPCI("MTCAEVR","09:00.0")

mrmEvgSetupPCI("EVG",08,00,0)



But the following error occurs:

mrmEvrSetupPCI("MTCAEVR","09:00.0")

Notice: devPCIFindSpec() expect B:D.F in hex

Device MTCAEVR 9:0.0 slot=8

Using IRQ 17

Failed to open uio device for PCI device 0000:09:00.0: No such filleor directory

Can neither open resource file nor uio file of PCI device 0000:09:00.0 BAR 0

PCI error: Failed to map BAR 0

mrmEvgSetupPCI("EVG",08,00,0)

Error: invalid spec '08'

PCI Device not found


I use mrfioc2 2.3.0 . How deal with this problem?

Thank you for your time. I am looking forward to your reply.






Replies:
Re: mrfioc2-PCI device Configuration gjzhai--- via Tech-talk
References:
mrfioc2-PCI device Configuration gjzhai--- via Tech-talk
RE: mrfioc2-PCI device Configuration GAGET Alexis via Tech-talk
Re: mrfioc2-PCI device Configuration Michael Davidsaver via Tech-talk
Re: mrfioc2-PCI device Configuration gjzhai--- via Tech-talk
Re: mrfioc2-PCI device Configuration Michael Davidsaver via Tech-talk
Re: mrfioc2-PCI device Configuration gjzhai--- via Tech-talk
Re: mrfioc2-PCI device Configuration Michael Davidsaver via Tech-talk
Re: mrfioc2-PCI device Configuration gjzhai--- via Tech-talk

Navigate by Date:
Prev: Re: mrfioc2-PCI device Configuration gjzhai--- via Tech-talk
Next: Re: mrfioc2-PCI device Configuration gjzhai--- via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <20222023  2024 
Navigate by Thread:
Prev: Re: mrfioc2-PCI device Configuration gjzhai--- via Tech-talk
Next: Re: mrfioc2-PCI device Configuration gjzhai--- via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <20222023  2024 
ANJ, 14 Sep 2022 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·