RE: linux kernel module for mrfioc2 on MTCA

[문건영(선형가속기팀) via Tech-talk <tech-talk at aps.anl.gov>]

Dear All,

 

With the help of Dr. Han, I was able to run mrfioc2.

However, I have one question.

 

I have one EVG and two EVRs as shown below.

(mTCA-EVM-300, mTCA-EVR-300, mTCA-EVR-300RF)

 

nat> show_fru

 

FRU Information:

----------------

FRU  Device   State  Name

==========================================

  0   MCH       M4    NAT-MCH-CM

  3   mcmc1     M4    NAT-MCH-MCMC

  5   AMC1      M4    CCT AM G64/471

  6   AMC2      M4    mTCA-EVM-300

  7   AMC3      M4    mTCA-EVR-300

  8   AMC4      M4    mTCA-EVR-300RF

 40   CU1       M4    Schroff uTCA CU

50   PM1       M4    NAT-PM-AC600D

60   Clock1    M4    MCH-Clock

61   HubMod1   M4    MCH-PCIe

92   AMC3-RTM  M4    mTCA-EVRTM-300

==========================================

 

I checked it by referring to lspci of mrfioc2 to distinguish 3 cards.

 

# dmesg | grep mrf

mrf: loading out-of-tree module taints kernel.

mrf-pci 0000:06:00.0: Attaching MRF device w/o PLX bridge (00007011)

mrf-pci 0000:06:00.0: MRF Setup complete

mrf-pci 0000:09:00.0: Attaching MRF device w/o PLX bridge (00007011)

mrf-pci 0000:09:00.0: MRF Setup complete

mrf-pci 0000:0a:00.0: Attaching MRF device w/o PLX bridge (00007011)

mrf-pci 0000:0a:00.0: MRF Setup complete

#

 

# lspci -v -s 6:00.0

06:00.0 Signal processing controller: Xilinx Corporation Device 7011

           Subsystem: Device 1a3e:132c

           Flags: bus master, fast devsel, latency 0, IRQ 16

           Memory at df200000 (32-bit, non-prefetchable) [size=256K]

           Capabilities: [40] Power Management version 3

           Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+

           Capabilities: [60] Express Endpoint, MSI 00

           Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00

           Kernel driver in use: mrf-pci

           Kernel modules: mrf

 

# lspci -v -s 9:00.0

09:00.0 Signal processing controller: Xilinx Corporation Device 7011

           Subsystem: Device 1a3e:132c

           Flags: bus master, fast devsel, latency 0, IRQ 18

           Memory at df100000 (32-bit, non-prefetchable) [size=256K]

           Capabilities: [40] Power Management version 3

           Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+

           Capabilities: [60] Express Endpoint, MSI 00

           Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00

           Kernel driver in use: mrf-pci

           Kernel modules: mrf

 

# lspci -v -s a:00.0

0a:00.0 Signal processing controller: Xilinx Corporation Device 7011

           Subsystem: Device 1a3e:232c

           Flags: bus master, fast devsel, latency 0, IRQ 16

           Memory at df000000 (32-bit, non-prefetchable) [size=512K]

           Capabilities: [40] Power Management version 3

           Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+

           Capabilities: [60] Express Endpoint, MSI 00

           Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00

           Kernel driver in use: mrf-pci

           Kernel modules: mrf

 

https://github.com/epics-modules/mrfioc2/blob/master/documentation/lspci/mtca-evm-300.txt

In the document above, the device name is specified in the subsystem item as shown below, but I do not.

 

06:00.0 Signal processing controller: Xilinx Corporation XILINX PCI DEVICE

        Subsystem: Micro-Research Finland Oy MTCA Event Master 300

        Physical Slot: 4

 

I couldn't distinguish between EVG and EVR, so I had to run them one by one.

 

epics> mrmEvgSetupPCI("EVG","6:00.0")

Notice: devPCIFindSpec() expect B:D.F in hex

PCI Device not found

epics>

epics> mrmEvgSetupPCI("EVG","9:00.0")

PCI Device not found

epics>                      

epics>

epics> mrmEvgSetupPCI("EVG","a:00.0")

Device EVG  a:0.0

Using IRQ 16

FPGA version: 280e0207

mTCA-EVM-300 #Inputs FP:3 UV:16 RB:0

EVM automatically creating 'EVG:FCT', 'EVG:EVRD', and 'EVG:EVRU'

Found SFP EEPROM

Found SFP EEPROM

Found SFP EEPROM

Found SFP EEPROM

Found SFP EEPROM

Found SFP EEPROM

Found SFP EEPROM

Found SFP EEPROM

Warning: Recommended minimum firmware 2 version is 207.6, found 207.0

Found SFP Strangeness 00000000

Sequencer capability detected

VME 64: Out FP:8 FPUNIV:0 RB:0 IFP:8 GPIO:0

EVR FIFO task start

Warning: Recommended minimum firmware 2 version is 207.6, found 207.0

Found SFP Strangeness 00000000

Sequencer capability detected

VME 64: Out FP:8 FPUNIV:0 RB:0 IFP:8 GPIO:0

EVR FIFO task start

PCI interrupt connected!

epics>

epics> mrmEvrSetup'

Unbalanced quote.

epics> mrmEvrSetupPCI("EVR1","slot=4")

PCI Device not found on slot=4

epics> mrmEvrSetupPCI("EVR1","6:00.0")

Device EVR1  6:0.0 slot=(null)

Using IRQ 16

FWVersion 0x18120207

Found version 519

Found SFP EEPROM

Sequencer capability detected

mTCA: Out FP:4 FPUNIV:0 RB:16 IFP:2 GPIO:0

EVR FIFO task start

Enabling interrupts

epics> mrmEvrSetupPCI("EVR2","9:00.0")

Device EVR2  9:0.0 slot=(null)

Using IRQ 18

FWVersion 0x18100207

Found version 519

Found SFP EEPROM

Sequencer capability detected

mTCA: Out FP:4 FPUNIV:0 RB:16 IFP:2 GPIO:0

EVR FIFO task start

Enabling interrupts

epics>

 

Subsystem: Device 1a3e:232c

Subsystem: Device 1a3e:132c

Is there no problem if I separate it into the above information?

1a3e:232c is mTCA-EVM-300.

1a3e:132c is mTCA-EVR-300RF or mTCA-EVR-300.

I can distinguish the two EVRs after the IOC runs.

Is there any way to know before the IOC is implemented?

 

Best regards,

Geonyeong Mun

_________________________________________

Geonyeong Mun

Staff Engineer, Accelerator Division

 

Pohang Accelerator Laboratory (PAL)
37673 80 Jigokro-127-beongil, Nam-gu,
Pohang, Gyeongbuk, Korea

 

phone: +82 10 4164 - 2260

office: +82 54 279 - 1098

email: gymun at postech.ac.kr

web: pal.postech.ac.kr

 

From: Han Lee <jeonglee at lbl.gov>
Sent: Thursday, January 5, 2023 3:00 PM
To: 문건영(선형가속기팀) <gymun at postech.ac.kr>
Cc: tech-talk at aps.anl.gov
Subject: Re: linux kernel module for mrfioc2 on MTCA

 

The actual kernel driver is located in

https://github.com/epics-modules/mrfioc2/tree/master/mrmShared/linux

 

There are several methods you may try, here is also very old repo, but you can find "dkms" information there.

 

https://github.com/icshwi/e3-mrfioc2/

 

Best,

Han

 

On Wed, Jan 4, 2023 at 9:57 PM Han Lee <jeonglee at lbl.gov> wrote:

Hi Geonyeong,

 

Please use the kernel driver for mrfiioc2 instead of what you used.

 

https://github.com/epics-modules/mrfioc2

 

PCIID is not up-to-date, if you want to see through lspci, please check my own archived repo.

 

https://github.com/jeonghanlee/pciids

 

HTH,

Han

 

 

On Wed, Jan 4, 2023 at 9:41 PM 문건영(선형가속기팀) via Tech-talk <tech-talk at aps.anl.gov> wrote:

Dear All,

I am trying to MTCA-EVG-300 and MTCA-EVR-300U are working on Linux using mrfioc2.
But, the following error is displayed:

epics> mrmEvgSetupPCI("EVG",08,00,0)
Can't open /sys/module/mrf/parameters/interfaceversion in order to read kernel module interface version. Kernel module not loaded or too old.

I used the latest version driver provided by MRF. https://github.com/jpietari/mrf-linux-driver
Do I have to use a driver other than this one?

The mrf directory does not exist In the /sys/module directory.
Instead, pci_mrfevg, pci_mrfevr exists.

# ls /sys/module/pci_mrfev*
/sys/module/pci_mrfevg:
coresize   holders    initstate  refcnt     taint
drivers    initsize   notes      sections   uevent

/sys/module/pci_mrfevr:
coresize   holders    initstate  refcnt     taint
drivers    initsize   notes      sections   uevent

# lsmod
Module                  Size  Used by    Tainted: G 
pci_mrfevr             57344  0
pci_mrfevg             53248  0
nfsv3                  28672  1
nfs                   135168  2 nfsv3
lockd                  77824  2 nfsv3,nfs
grace                  16384  1 lockd
sunrpc                196608  6 nfsv3,nfs,lockd
x86_pkg_temp_thermal    16384  0
igb                   184320  0

# ls /dev/e*
/dev/ega0  /dev/ega2  /dev/era0  /dev/era2  /dev/erb0  /dev/erb2
/dev/ega1  /dev/ega3  /dev/era1  /dev/era3  /dev/erb1  /dev/erb3

# dmesg | tail
Event Generator PCI/PCIe driver exiting.
Event Receiver PCI/PCIe driver exiting.
Event Generator PCI/PCIe driver init.
mrfevg:BAR0 start df000000 end df07ffff mmaped 41180000
udevd[165]: specified group 'mrf' unknown
Event Receiver PCI/PCIe driver init.
mrfevr:BAR0 start df200000 end df23ffff, mmap 40780000
mrfevr: Device Type: MTCAEVR300
mrfevr:BAR0 start df100000 end df13ffff, mmap 41040000
mrfevr: Device Type: MTCAEVR300

Best regards,
Geonyeong Mun

--

Jeong Han Lee, Dr.rer.nat

Staff Scientist and Engineer

Lawrence Berkeley National Laboratory

1 Cyclotron Road Mailstop 46R0161

Berkeley, CA 94720, United States

Tel :+1-510-486-6163

Cell:+1-510-384-3868

--

Jeong Han Lee, Dr.rer.nat

Staff Scientist and Engineer

Lawrence Berkeley National Laboratory

1 Cyclotron Road Mailstop 46R0161

Berkeley, CA 94720, United States

Tel :+1-510-486-6163

Cell:+1-510-384-3868