Re: mrfioc2-PCI device Configuration

[Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov>]

On 2/28/22 05:27, gjzhai at mail.ustc.edu.cn wrote:
> > Some file permissions need to be changed.  The best way to handle this is with a udev rule.
> >
> > > cat << EOF > /etc/udev/rules.d/99-mrfioc2.rules
> > > KERNEL=="uio*", ATTR{name}=="mrf-pci", GROUP="softioc", MODE="0660"
> > > EOF
> > > udevadm control --reload
> > It is also be worth noting that DKMS packaging is now include (added by Michael Abbot @DLS)
> > to make OS kernel updates more automatic.
> >
> > In any case, you will need to choose an appropriate GROUP name for your site.
> After I followed the above, the following appeared：
> mrmEvrSetupPCI("MTCAEVR","5:0.0")
> Notice: devPCIFindSpec() expect B:D.F in hex
> Device MTCAEVR  5:0.0 slot=10
> Using IRQ 162
> Failed to open resource file for PCI device 0000:05:00.0: Permission denied
> FWVersion 0x18100207
> Found version 519
> Found SFP EEPROM
> Sequencer capability detected
> mTCA: Out FP:4 FPUNIV:0 RB:16 IFP:32 GPIO:2
> EVR FIFO task start
> Enabling interrupts
>
> Why does the error message of Permission denied still appear?
The underlying devlib2 PCI access library knows two ways to access a PCI
BAR under Linux.  With /dev/uio* and with another special file deep down
in /sys.  The are equivalent.  The second is (currently) tried first.

The /sys method doesn't require an OS driver.  So it can be sufficient for
simple devices, or for troubleshooting (or some other strange situations).

> In addition, the geographic address of the EVM is 08:00.0, but the following error message appears:
> mrmEvgSetupPCI("EVG", 8, 0, 0)
> Error: invalid spec '8'
> PCI Device not found

The four argument form of mrmEvgSetupPCI(NAME, BUS, DEV, FUNC) is legacy.
Back when I thought PCI addressing was always as simple as BUS:DEV.FUNC .
I tried to keep it working, but you have encountered a pathological case
with DEV==FUNC==0.

https://github.com/epics-modules/mrfioc2/blob/a8cb48d549b7aae13206cf2d168c6f1cfec2e1c4/evgMrmApp/src/evgInit.cpp#L371-L382

I recommend switching to the newer two argument form.

> mrmEvgSetupPCI("EVG", "8:0.0")


Although seeing "slot=" indicates that geographical addressing is available.
I tend to prefer putting slot numbers in config files when possible.

...
> Device MTCAEVR  5:0.0 slot=10
...

So

> mrmEvrSetupPCI("MTCAEVR","5:0.0")
could be replaced with

> mrmEvrSetupPCI("MTCAEVR","slot=10")

The devPCIShow() IOCsh function should show any slot numbers.

http://epics.sourceforge.net/devlib2/iocsh.html#iocshpci


In my experience.  While PCI slot numbers don't always map to physical slots in
an intuitive way, that mapping is at least as stable as B:D.F addressing.
Typically only backplane bus segments provide slot numbering.  So slot numbers
aren't effected by the addition or removal of a bus segment with a lower BUS
number.

Slot number still isn't 100%.  Things get weird when two bus segments provide
overlapping slot number ranges.  eg. a cPCI CPU with on board PMC sockets as
well as the backplane.


> > -----原始邮件-----
> > 发件人: "Michael Davidsaver" <mdavidsaver at gmail.com>
> > 发送时间: 2022-02-28 01:49:27 (星期一)
> > 收件人: gjzhai at mail.ustc.edu.cn
> > 抄送: "GAGET Alexis" <alexis.gaget at cea.fr>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
> > 主题: Re: mrfioc2-PCI device Configuration
> >
> > On 2/26/22 23:50, gjzhai at mail.ustc.edu.cn wrote:
> > > ...
> > > Failed to open resource file for PCI device 0000:09:00.0: Permission denied
> > > Can neither mmap resource file nor uio file of PCI device 0000:09:00.0 BAR 0
> > > PCI error: Failed to map BAR 0
> > Some file permissions need to be changed.  The best way to handle this is with a udev rule.
> > There is an example in the README
> >
> > https://github.com/epics-modules/mrfioc2/tree/master/mrmShared/linux
> >
> > > cat << EOF > /etc/udev/rules.d/99-mrfioc2.rules
> > > KERNEL=="uio*", ATTR{name}=="mrf-pci", GROUP="softioc", MODE="0660"
> > > EOF
> > > udevadm control --reload
> > It is also be worth noting that DKMS packaging is now include (added by Michael Abbot @DLS)
> > to make OS kernel updates more automatic.
> >
> > In any case, you will need to choose an appropriate GROUP name for your site.
> >
> >
> > > mrmEvgSetupPCI("EVG", 8, 0, 0)
> > > Error: invalid spec '8'
> > > PCI Device not found
> > >
> > > More information：
> > > [opi@localhost dev]$ ll
> > > crw-rw----.  1 root root    241,     0 Feb 27 10:39 uio0
> > > crw-rw----.  1 root root    241,     1 Feb 27 10:39 uio1
> > > crw-rw----.  1 root root    241,     2 Feb 27 10:39 uio2
> > > [opi@localhost]$ sudo udevadm info -q path -n /dev/uio0
> > > /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/0000:04:00.0/0000:05:00.0/uio/uio0
> > > [opi@localhost]$ sudo udevadm info -q path -n /dev/uio1
> > > /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/0000:04:03.0/0000:08:00.0/uio/uio1
> > > [opi@localhost]$ sudo udevadm info -q path -n /dev/uio2
> > > /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/0000:04:04.0/0000:09:00.0/uio/uio2
> > >
> > > > -----原始邮件-----
> > > > 发件人: "Michael Davidsaver" <mdavidsaver at gmail.com>
> > > > 发送时间: 2022-01-24 01:03:06 (星期一)
> > > > 收件人: gjzhai at mail.ustc.edu.cn
> > > > 抄送: "GAGET Alexis" <alexis.gaget at cea.fr>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
> > > > 主题: Re: mrfioc2-PCI device Configuration
> > > >
> > > > On 1/23/22 01:15, gjzhai at mail.ustc.edu.cn wrote:
> > > > ...
> > > > > Can't read private key
> > > > >      DEPMOD  3.10.0-1160.42.2.el7.x86_64
> > > > ...
> > > > > [root@localhost mrmShared]# dmesg | grep mrf
> > > > > [    1.338766] pci_mrfevr: loading out-of-tree module taints kernel.
> > > > > [    1.339091] pci_mrfevr: module verification failed: signature and/or required key missing - tainting kernel
> > > > I guess your (RHEL7) system has kernel module signing at least partially enabled?
> > > > Personally, I have no experience with this feature.  So I can't advise you on how
> > > > to sign locally built kernel modules.
> > > >
> > > > > [    1.340025] mrfevr:BAR0 start df200000 end df23ffff, mmap c3700000
> > > > > [    1.340296] mrfevr:BAR0 start df000000 end df03ffff, mmap c3780000
> > > > > [    1.340432] mrfevg:BAR0 start df100000 end df17ffff mmaped c3800000
> > > > >
> > > > > [root@localhost mrmShared]# lsmod |grep mrf
> > > > > mrf                    18421  0
> > > > > uio                    19338  1 mrf
> > > > > parport                46395  1 mrf
> > > > > pci_mrfevg             53731  0
> > > > > pci_mrfevr             53731  0
> > > > It looks like you have two different drivers for MRF cards loaded.
> > > > The Linux driver which is part of mrfioc2 is independent of, and
> > > > conflicts with, all other Linux drivers for MRF cards.
> > > >
> > > > You should remove the pci_mrfevg and pci_mrfevr.  It would be best
> > > > to reboot, then load only mrf.ko build from mrfioc2.
> > > >
> > > > If all goes well you should see a line like the following in the kernel log.
> > > >
> > > > > [   10.000000] mrf-pci 0000:02:01.0: MRF Setup complete
> > > > https://github.com/epics-modules/mrfioc2/blob/a8cb48d549b7aae13206cf2d168c6f1cfec2e1c4/mrmShared/linux/uio_mrf.c#L675
> > > >
> > > >
> > > > > Thanks
> > > > >
> > > > > > -----原始邮件-----
> > > > > > 发件人: "Michael Davidsaver" <mdavidsaver at gmail.com>
> > > > > > 发送时间: 2022-01-21 04:26:28 (星期五)
> > > > > > 收件人: "gjzhai at mail.ustc.edu.cn" <gjzhai at mail.ustc.edu.cn>
> > > > > > 抄送: "GAGET Alexis" <alexis.gaget at cea.fr>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>[root@localhost
> > > > > > 主题: Re: mrfioc2-PCI device Configuration
> > > > > >
> > > > > > On 1/20/22 7:31 AM, GAGET Alexis via Tech-talk wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I got this error a lot.
> > > > > > >
> > > > > > > Many points :
> > > > > > >
> > > > > > > - is the kernel module well loaded ? lsmod | grep mrf
> > > > > > If the kernel module is being loaded, but the /dev/uio* devices
> > > > > > are not being created, check the kernel log ("dmesg") for lines
> > > > > > including "uio_mrf" and/or the specific PCI B:D.F.
> > > > > >
> > > > > > > - Depending on the CPU you use, you need to configure your PCIe virtual switch correctly.
> > > > > > >     for me with NAT COMEx and NAT MCH it's like that (in the MCH webserver) :
> > > > > > >
> > > > > > >
> > > > > > > With my configuration that is MTCA EVM300 andMTCA EVR300U I also had this problem because the CPU (NAT COMEX) hosting my EPICS IOC booted after the EVR and EVM. So the PCI configuration was not well done and produced this exact message you write below.
> > > > > > >
> > > > > > > A manual solution is to reboot manually just your CPU to see if it's the problem. If it is, for me in the MCH (NAT MCH PHYS80) I have an argument "upstream slot power delay" that I put to 40 sec, meaning that my COMEx in RTM boot 40sec after the crate boot. (this work only with the latest version of the NAT MCH firmware)
> > > > > > >
> > > > > > >
> > > > > > > ps : I prefer to use "lspci -nmmn" to list my PCIe device
> > > > > > fyi. I keep a collection of lspci output from cards I've encountered.
> > > > > > These were captured when correctly configured and running.  Maybe
> > > > > > helpful as a point of comparison.
> > > > > >
> > > > > > https://github.com/epics-modules/mrfioc2/tree/master/documentation/lspci
> > > > > >
> > > > > >
> > > > > > > Cheers
> > > > > > >
> > > > > > >
> > > > > > > Alexis GAGET
> > > > > > >
> > > > > > > CEA Saclay DRF/IRFU/DIS/LDISC
> > > > > > >
> > > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > > > > > *De :* Tech-talk <tech-talk-bounces at aps.anl.gov> de la part de gjzhai--- via Tech-talk <tech-talk at aps.anl.gov>
> > > > > > > *Envoyé :* jeudi 20 janvier 2022 14:52:33
> > > > > > > *À :* tech-talk at aps.anl.gov
> > > > > > > *Objet :* mrfioc2-PCI device Configuration
> > > > > > >     
> > > > > > >
> > > > > > > Hello all I am trying to get MRF MTCA-EVG-300 and MTCA-EVR-300U configured and working on Linux using mrfioc2. I found geographic address(bus:device.function) in /proc/bus/pci/devices. The information is
> > > > > > >     as follows:
> > > > > > >
> > > > > > > 0800 10ee7011 10 df100000 0 0 0 0 0 0 80000 0 0 0 0 0 0 pci_mrfevg
> > > > > > >
> > > > > > > 0900 10ee7011 11 df000000 0 0 0 0 0 0 40000 0 0 0 0 0 0 pci_mrfevr
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > According to the above information, my st.cmd in the iocevrmrm folder is defined as follows:
> > > > > > >
> > > > > > > mrmEvrSetupPCI("MTCAEVR","09:00.0")
> > > > > > >
> > > > > > > mrmEvgSetupPCI("EVG",08,00,0)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > But the following error occurs:
> > > > > > >
> > > > > > > mrmEvrSetupPCI("MTCAEVR","09:00.0")
> > > > > > >
> > > > > > > Notice: devPCIFindSpec() expect B:D.F in hex
> > > > > > >
> > > > > > > Device MTCAEVR 9:0.0 slot=8
> > > > > > >
> > > > > > > Using IRQ 17
> > > > > > >
> > > > > > > Failed to open uio device for PCI device 0000:09:00.0: No such filleor directory
> > > > > > >
> > > > > > > Can neither open resource file nor uio file of PCI device 0000:09:00.0 BAR 0
> > > > > > >
> > > > > > > PCI error: Failed to map BAR 0
> > > > > > >
> > > > > > > mrmEvgSetupPCI("EVG",08,00,0)
> > > > > > >
> > > > > > > Error: invalid spec '08'
> > > > > > >
> > > > > > > PCI Device not found
> > > > > > >
> > > > > > >
> > > > > > > I use mrfioc2 2.3.0 . How deal with this problem?
> > > > > > >
> > > > > > > Thank you for your time. I am looking forward to your reply.