Experimental Physics and
Industrial Control System

Andrew Johnson <[email protected]> · Wed, 15 May 2013 11:11:25 -0500

Hi Till,

[I don't normally top-post, but I want to move this thread to tech-talk and 
let everyone else read what has been written so far.  *Please* don't have 
discussions of this nature in private cliques.  Not your fault I know.]

Regarding your last paragraph, I think that's exactly what everyone is asking 
for: A standard DMA API in devLib2 which accepts registration from a DMA 
engine driver.  Is it is really necessary to take a DMA engine ID?  My Tempe 
DMA driver has a single queue of jobs which it passes to either engine as they 
become free.  If there are multiple DMA engines that have restrictions as to 
where they can read or write from, the appropriate engine could be selected 
automatically when the job is queued (this would require that the engine 
registration provide a routine that can look at a job request and tell the job 
management code "I can't do this, ask the next guy" engine).

The epicsDma.h API that Stephanie posted may need a few changes which should 
be discussed separately once we've worked out who is going to actually do the 
work.

- Andrew

On 2013-05-14 you wrote:
> There is no simple way to deal with linux.
> 
> VME systems obey a de-facto 'standard' since they all (well, since the
> 68k days are over anyways)
> use pretty much the same VME bus controller chip (older systems use the
> 'universe', newer ones
> the [very similar] 'TSI148', both by Tundra). This chip has a DMA engine
> built in. Hence, when you
> do DMA regardless to/from what VME device you can do it with the same
> engine and with
> the same driver. There are exceptions to this rule where a DMA engine on
> a VME card is used but
> mostly, epicsDma wraps around the driver for DMA with the tundra chip.
> 
> In the PCI/PC world there is no equivalent to the tundra chip. Most PCI
> CPU boards do not
> have a DMA engine but such an engine is integrated on peripherals. Hence
> DMA tends to
> involve whatever DMA engine a particular PCI board (i.e., *not* the CPU
> board) has.
> This means that there is no generic OSI service for DMA
> that could be harnessed (in VME this service would be the tundra driver
> for rtems/vxworks;
> well, a linux driver could be written but who runs linux on VME boards).
> 
> Besides that, there is the complication that DMA requires special
> privileges and fiddling
> with address mappings etc. which all are a bit more cumbersome to do
> across the
> user-kernel boundary which doesn't exist under rtems/vxworks.
> 
> OSI DMA for linux makes sense e.g., if there is a popular device which
> is widely
> used e.g., under linux and RTEMS and where most of the driver code can
> be shared
> (in a true OSI way), e.g., because it uses devlib2. If the device
> features a DMA
> engine then it may make sense to let the driver implement epicsDma -
> just so that
> the shared OSI pieces can use this API. But as you can see, epicsDma has
> become
> largely a driver feature rather than a generic service.
> 
> Thus, devlib2 could define the API. It should allow for drivers to
> register their DMA support.
> Drivers would need to be bound to DMA engines/devices.
> When creating a DMA ID the user would have to specify what DMA
> engine/device to use
> (it is theoretically possible to do DMA from device/card A to main
> memory on card B using
> a DMA engine on card C). Under VME we have not thought about this
> complexity
> because there was one DMA engine and the driver part of the BSP.
> 
> - T.
> 
> On 05/14/2013 04:55 PM, Allison, Stephanie wrote:
> > Hi -
> >
> >> Along the same lines, can we have a standard OSI interface for doing DMA
> >> in 3.15?
> >
> > Well, all I've done so far for DMA is to pull epicsDma.c and epicsDma.h
> > out of generic transient recorder (GTR) and made it its own module
> > (devEpicsDma).  And added the dependency on Till's drvRTEMSDma module. 
> > So epicsDma.c has a lot of ifdef rtems/vxWorks.  Needs work to do true
> > osi calls instead (meaning making one module with both epicsDma and
> > repackaged osd-ed drvRTEMSDma).  And somebody to do linuxRT.
> >
> > Has anybody else done anything?  Ernest says it may be worth proceeding
> > if there is interest.
> >
> > epicsDma.h:
> >
> > #ifndef _EPICSDMA_H_
> > #define _EPICSDMA_H_
> >
> > #include <epicsTypes.h>
> >
> > typedef void (*epicsDmaCallback_t)(void *);
> > typedef struct epicsDmaInfo *epicsDmaId;
> >
> > /*
> >   * EPICS wrappers/additions
> >   */
> > epicsDmaId epicsDmaCreate(epicsDmaCallback_t callback, void *context);
> > int epicsDmaStatus(epicsDmaId dmaId);
> > int epicsDmaToVme(epicsDmaId dmaId, epicsUInt32 vmeAddr, int adrsSpace,
> >                            void *pLocal, int length, int dataWidth);
> > int epicsDmaFromVme(epicsDmaId dmaId, void *pLocal, epicsUInt32 vmeAddr,
> >                            int adrsSpace, int length, int dataWidth);
> > int epicsDmaToVmeAndWait(epicsDmaId dmaId, epicsUInt32 vmeAddr, int
> > adrsSpace, void *pLocal, int length, int dataWidth); int
> > epicsDmaFromVmeAndWait(epicsDmaId dmaId, void *pLocal, epicsUInt32
> > vmeAddr, int adrsSpace, int length, int dataWidth);
> >
> > #endif /* _EPICSDMA_H_ */
> >
> > Thank you,
> > Stephanie
> >
> >> -----Original Message-----
> >> From: Mark Rivers [mailto:[email protected]]
> >> Sent: Friday, May 10, 2013 1:56 PM
> >> To: 'Michael Davidsaver'; Straumann, Till
> >> Cc: Williams Jr., Ernest L.; Andrew Johnson; Daron Chabot; Eric Norum;
> >> Boyes, Matthew; Beamline Controls and Data Acquisition; Dalesio, Leo;
> >> Allison, Stephanie Subject: RE: mmio in epics base
> >>
> >> This sounds good.
> >>
> >> Along the same lines, can we have a standard OSI interface for doing DMA
> >> in 3.15?
> >>
> >> Thanks,
> >> Mark
> >>
> >>
> >> -----Original Message-----
> >> From: Michael Davidsaver [mailto:[email protected]]
> >> Sent: Friday, May 10, 2013 3:51 PM
> >> To: Till Straumann
> >> Cc: Mark Rivers; Williams Jr., Ernest L.; Andrew Johnson; Daron Chabot;
> >> Eric Norum; Boyes, Matthew; Beamline Controls and Data Acquisition;
> >> Dalesio, Leo; Allison, Stephanie
> >> Subject: mmio in epics base
> >>
> >> On 05/10/2013 03:55 PM, Till Straumann wrote:
> >>> ...
> >>> Let's face it: Reordering of loads even to guarded and cache-inhibited
> >>> memory is
> >>> not prohibited and thus could happen. ...
> >>
> >> I would agree with Till in this case.  Existing hardware probably isn't
> >> taking advantage of all the reordering allowed by the PPC ISA.  So event
> >> if this isn't a problem with current hardware, future hardware could.
> >> So I think it is prudent to do the "right thing".
> >>
> >> In the interest of making this easier to do I'm proposing that the MMIO
> >> wrappers from devLib2 be included in EPICS Base 3.15.
> >>
> >> https://code.launchpad.net/~epics-core/epics-base/devlib2mmio/+merge/163
> >>365
> >>
> >> I would ask that those interested (Till and Eric especially) please look
> >> this over and see if it is sufficient.  Right now it has
> >> speciallizations only for vxWorks and RTEMS (PPC and x86 only). Other
> >> targets fall back to C volatile pointers, which are better than nothing.
> >>
> >>
> >> Michael

-- 
It is difficult to get a man to understand something, when his salary
depends upon his not understanding it. -- Upton Sinclair

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System