Multiple PCI-PCI Bridges

[Ned Arnold <nda@aps.anl.gov>]

Hello world -

I was wondering if anyone has experience with configuring PCI-PCI bridges for optimum performance (pre-fetching, wait states, etc)?  I have an MVME6100 (running vxWorks 5.5.2) with a PMC carrier board (specifically an Emerson/Motorola PMCspan) and a very old VMIC reflective memory PMC board.  This configuration has two PCI-PCI bridges between the CPU and the actual PMC board.  I am getting *extremely slow* data transfers between the processor and PMC card (reading 320 floats takes 460 microseconds!) .

So far I have tried ... with zero success ...
  - enabling "Fast back-to-back transfers"
  - ensuring the Pre-fetch memory range on both bridges encompasses the PMC board's address

Appended below is the output of pciConfigTopoShow, the PMC card is device [2,2,0].

Any help/hints/references would be appreciated ...

   Ned



iocrtfbdaq> pciConfigTopoShow                                
[0,0,0] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
        bar0 in prefetchable 32-bit mem space @ 0x00000000
        bar2 in prefetchable 32-bit mem space @ 0x40000000
        bar4 in prefetchable 32-bit mem space @ 0xe0800000
[0,0,0] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
        bar0 in prefetchable 32-bit mem space @ 0x00000000
        bar2 in prefetchable 32-bit mem space @ 0x40000000
        bar4 in prefetchable 32-bit mem space @ 0xe0800000
[0,0,1] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
[0,0,2] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
[0,0,3] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
        bar0 in 64-bit mem space @ 0xff000000
[0,0,4] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
        bar4 in I/O space @ 0x00000000
        bar5 in I/O space @ 0x00000000
[0,0,5] type=MEM_CNTLR
        status=0x0000 ( DEVSEL=0 )
        command=0x0000 ( )
[0,0,6] type=MEM_CNTLR
        status=0x0000 ( DEVSEL=0 )
        command=0x0000 ( )
[0,0,7] type=MEM_CNTLR
        status=0x0000 ( DEVSEL=0 )
        command=0x0000 ( )
[0,5,0] type=UNKNOWN (0x80) BRIDGE
        status=0x0230 ( CAP 66MHZ DEVSEL=1 )
        command=0x0006 ( MEM_ENABLE MASTER_ENABLE )
        bar0 in 64-bit mem space @ 0xe0000000
[0,6,0] type=P2P BRIDGE to [1,0,0]
        base/limit:
          mem=   0xe0100000/0iocrtfbdaq> pciConfigTopoShow                                
[0,0,0] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
        bar0 in prefetchable 32-bit mem space @ 0x00000000
        bar2 in prefetchable 32-bit mem space @ 0x40000000
        bar4 in prefetchable 32-bit mem space @ 0xe0800000
[0,0,0] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
        bar0 in prefetchable 32-bit mem space @ 0x00000000
        bar2 in prefetchable 32-bit mem space @ 0x40000000
        bar4 in prefetchable 32-bit mem space @ 0xe0800000
[0,0,1] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
[0,0,2] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
[0,0,3] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
        bar0 in 64-bit mem space @ 0xff000000
[0,0,4] type=MEM_CNTLR
        status=0x22b0 ( CAP 66MHZ FBTB DEVSEL=1 MSTR_ABORT_RCV )
        command=0x0217 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE WI_ENABLE FBTB_ENABLE )
        bar4 in I/O space @ 0x00000000
        bar5 in I/O space @ 0x00000000
[0,0,5] type=MEM_CNTLR
        status=0x0000 ( DEVSEL=0 )
        command=0x0000 ( )
[0,0,6] type=MEM_CNTLR
        status=0x0000 ( DEVSEL=0 )
        command=0x0000 ( )
[0,0,7] type=MEM_CNTLR
        status=0x0000 ( DEVSEL=0 )
        command=0x0000 ( )
[0,5,0] type=UNKNOWN (0x80) BRIDGE
        status=0x0230 ( CAP 66MHZ DEVSEL=1 )
        command=0x0006 ( MEM_ENABLE MASTER_ENABLE )
        bar0 in 64-bit mem space @ 0xe0000000
[0,6,0] type=P2P BRIDGE to [1,0,0]
        base/limit:
          mem=   0xe0100000/0xe01fffff
          preMem=0x00000000e0100000/0x00000000e01fffff
          I/O=   0xe0900000/0xe00fffff
        status=0x02b0 ( CAP 66MHZ FBTB DEVSEL=1 )
        command=0x0007 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE )
[1,4,0] type=P2P BRIDGE to [2,0,0]
        base/limit:
          mem=   0xe0100000/0xe01fffff
          preMem=0x00000000e0100000/0x00000000e01fffff
          I/O=   0xe0900000/0xe00fffff
        status=0x0290 ( CAP FBTB DEVSEL=1 )
        command=0x0207 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE FBTB_ENABLE )
[2,2,0] type=NET_CNTLR
        status=0x0000 ( DEVSEL=0 )
        command=0x0082 ( MEM_ENABLE WC_ENABLE )
        bar0 in 32-bit mem space @ 0xe0100000
xe01fffff
          preMem=0x00000000e0100000/0x00000000e01fffff
          I/O=   0xe0900000/0xe00fffff
        status=0x02b0 ( CAP 66MHZ FBTB DEVSEL=1 )
        command=0x0007 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE )
[1,4,0] type=P2P BRIDGE to [2,0,0]
        base/limit:
          mem=   0xe0100000/0xe01fffff
          preMem=0x00000000e0100000/0x00000000e01fffff
          I/O=   0xe0900000/0xe00fffff
        status=0x0290 ( CAP FBTB DEVSEL=1 )
        command=0x0207 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE FBTB_ENABLE )
[2,2,0] type=NET_CNTLR
        status=0x0000 ( DEVSEL=0 )
        command=0x0082 ( MEM_ENABLE WC_ENABLE )
        bar0 in 32-bit mem space @ 0xe0100000