Experimental Physics and
Industrial Control System

Claude Saunders <[email protected]> · Wed, 14 Feb 2007 16:44:10 -0600

Hello All,

I thought it would be of value to publish a summary of results from some
prototyping and evaluation work I did. I was interested in exploring
alternatives to channel access in EPICS for several different reasons.
One, I wanted to learn more about details of the codebase of rsrv and by
extension, the behavior of channel access. Two, I wanted to learn more
about the OMG DDS specification, and in particular the underlying wire
protocol called RTPS. Three, I think it would be useful to have a more
broadly known and standards-based controls protocol in EPICS. I focused
on applying RTPS to EPICS, while a colleague here (Shifu Xu) has been
looking at ICE (still a work in progress, report forthcoming). Caveat: I
didn't have enough time to get really deep, so there are probably some
unexplored angles.
In summary, here's two bullet points to take away from this:
1. Implementations of OMG DDS Wire Protocol, as specified by OMG's "The
Real-time Publish-Subscribe Wire Protocol, DDS Interoperability Wire
Protocol (Final Adopted Specification ptc/08-02-2006)", are not well
suited as a substitute for EPICS channel access.
2. I have come to appreciate how precisely optimized the channel access
protocol is for what we do. Still, I think it is a valuable exercise to
explore the tradeoffs involved in alternatives.
What follows is some background information and details of what I did.
The Object Management Group (OMG) originally released a specification
called the Data Distribution Service (DDS). This was really just an API
and high-level behavior specification. Implementations could be
completely non-interoperable. Some time later, one commercial vendor of
DDS (RTI NDDS) submitted their implementation and standardized that as
the interoperable wire protocol, called the Real-Time Publish-Subscribe
(RTPS) protocol. To my knowledge, there are now 3 implementations available:

1. RTI NDDS 4.0 - a commercial implementation and generally
considered THE implementation

2. Thales Splice - a commercial implementation

3. Ocera Project ORTE - an open-source implementation called ORTE
from the Ocera project
The RTPS wire protocol specification consists of a PIM (Platform
Independent Model) part, and a PSM (Platform Specific Model) part. The
specification includes only the PSM for running RTPS on UDP/IP. I used
the ORTE open-source implementation to prototype with. This
implementation leaves much to be desired in that it is most certainly
not zero-copy, and uses malloc all over the place, but it is
interoperable (ie. "correct"), and works fine for basic prototyping. One
claimed benefit of RTPS on UDP is that the implementation becomes much
more portable, since we are now in full control and are not dependent on
the vagaries of various platform's TCP stacks. Reliable delivery is
built on top of UDP within RTPS. Disadvantage, possibly, is that much of
protocol is now in user-space (or, rather it is for ORTE).
The essence of RTPS is that one creates "topics" to which pre-defined
message structures may be published, and clients may subscribe. Think
channel access monitors. Secondly, all nodes on the network maintain a
cache that knows about all the available topics on the network. There is
no topic "nameserver". There is a peer-to-peer "discovery protocol"
which takes care of disseminating all topics to all caches (and cleaning
out topics that no longer exist). Topics are a one-way communication
(although through QoS settings you can specify guaranteed or best-effort
delivery). There are all kinds of neat bells-and-whistles to RTPS, but I
mainly had trouble using it at a fundamental level.
Messages to topics are fixed size. The protocol will batch messages in
network packets, but still, you have to enumerate every possible
structure you might send as a separate message type, and register that
with the topic. Not very convenient given the variety of message sizes
that are conveyed via channel access.
My prototype goal was to gut the EPICS rsrv of all channel access,
replace it with ORTE, and then code a simple caget and camonitor command
line client. I did not try to emulate the channel access API. Although I
managed to cobble this together, I ran into several dilemmas which
revealed how unsuited this was. Using rsrv as a basis...
1. I began with making a topic per process variable. Since topics are
one-way, I actually had to define 2 per process variable. One "writer"
topic was for clients to send commands to (like put, synchronous get,
monitor), and one "reader" topic was for the get response or monitor
events. This was clearly unscalable, since when an IOC comes up, the
RTPS discovery protocol has to communicate every single topic (PV) to
every single other node (including other IOCs). It does this regardless
of what PV's a given client on a given node would be interested in.
2. Ok, try again. I decided to view topics more like channel access
virtual circuits, and create them on demand. Basically, a pair of topics
for each desired host-host communication. This allows me to do some of
the nice things channel access does like clean up ioc resources when
clients go away. But publish-subscribe by its very nature is not good
for this, since fundamentally, publishers do not know who has subscribed
(or even IF anyone is currently subscribed). So I wound up having to
create special topics that are just there for the purpose of
establishing a set of "virtual circuit" topics. Then there is the
problem of naming these "virtual circuit" topics. The names have to be
unique network-wide. In short, this all turns out quite unwieldy as
well. It just gets ugly when you try to use fundamental abstractions
like topics to emulate other abstractions like TCP circuits.
Although I got something working along the lines of (2), I gave up at
this point.
I think for something like RTPS to work for us, we would wind up needing
a custom discovery protocol (allowable), and even then we would wind up
abusing the idea of topics anyways. Much of the advantage of using a
standards-based protocol would be lost, as we would not be interoperable.
  Hope this is of interest,
        Claude Saunders

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System