MPF - Message Passing Facility: A replacemnt for "Bare" Hideos

[Marty Kraimer <mrk@aps.anl.gov>]

I few months ago I sent a message proposing a replacement for "bare"
Hideos that is built on top of vxWorks. Attached is an html document
describing the current status. I have also made this document available
via the APS WWW Epics documents.

I am hoping to convert some of the APS/ASD systems to the new facility
during a scheduled shutdown this coming July. 

Marty Kraimer

MPF: Message Passing Facility

MPF is a facility that passes messages between tasks located on the same
or different processors. It requires vxWorks as the underlying operating
system. The primary purpose is to pass messages between device support
attacked to EPICS database records and tasks that access hardware. It is
a replacement for "bare" Hideos.

MPF will replace HIDEOS

Background:

Experience has shown that "bare" Hideos is too hard to use and extend.
It's debugging environment is not only different than vxWorks but 
more primitive. Porting Hideos to new architectures will be a major task
because it requires generating a "board support package" for each architecture.

Strategy:

The basic strategy is to abandon "bare" hideos. Instead vxWorks will be
required.

The most important goal is to provide a migration path for existing HIDEOS applications. An EPICS  Hideos application uses the following  hideos components:
 

EPICS Records

The DTYP and INP fields of records can specify a connection to HIDEOS device support.  An attempt will be made to keep these the same as now.

Hideos Device Support

These will be modified to interface to the new message passing system described below.

Hideos IP tasks

These will be modified to interface to the new message passing system described below.  Other changes necessary are:

• The idea of task groups will no longer be supported. When needed a special startup routine will have to be provided which just starts the individual tasks.
• It will no longer be possible to send messages from interrrupt level

Message Types

Message types are more general than the Hideos messages.

• Each message class must provide methods to transfer data between a message instance and a network buffer.
• A message class is allowed to define arbitrary methods

HIDEOS_SYS

Everything currently residing under the hideos_sys subdirectory will be abandoned. Instead the message passingfacilitywill be supported.

 

What hardware configurations do we want to support?

• Epics and IP support on a single processor
• Epics on one processor. vxWorks and IP support on another processor. Communication will be via a tcp connection. It will be possible to boot  a cpu via the VME backplane and also pass tcp messages over the backplane.
• Epics and IP support for a "dumb" VME IP module.

The first two options are currently supported by Hideos on 162s. The last
option is not currently supported by Hideos. The astronomy community supports
this configuration via support that does not use  Hideos. We should
figure out how their work can be integrated with the Hideos IP tasks.

Message Passing Overview

The new message passing system follows a client server model. A client
creates a connection to a server. Once a connection is established either
the client or server can send messages to the other.

Messages are sent through a router. Initially two routers will be provided: localRouter and tcpRouter. The localRouter passes messages between a clients and servers residing on the same processor. The tcpRouter sends messages between clients and servers on different processors.

Message Client Class

typedef void (*clientCallback)(Message *message,void *clientPvt);
class MessageClient
{
public:
    MessageClient(clientCallback,void *clientPvt);
 
    int bind(char *server, int location);
    int send(Message *message);
private:
    ...
}
Any code that wants to use the message passing system must:

1. Create an instance of MessageClient for each server with which it want to communicate.
2. Issue a bind call to establish communication.
3. Call send to send messages.
4. Must provide a callback which is called whenever a connection is made or broken and is also called whenever the server sends a message to the client.

Message Server Class

class MessageServer
{
public:
    MessageServer(const char* name);
    void waitForMessage();
    Message *receive();
    Message *allocReplyMessage(Message *clientMessage,messageType type);
    void  queueMessage(Message *);
    int reply(Message *);
private:
     ...
}
Ant code that wants to be a message server must

1. Create an instance of MessageServer.
2. Call receive to accept messages.
3. Call reply to send messages to a client.
4. Call allocReplyMessage if it wants to send additonal message to a client

Message Class

class Message
{
public:
    messageType  getType();
    virtual ~Message();
    virtual void init() = 0;
    virtual int  toBuffer(char **buffer);
    virtual int  fromBuffer(const char **buffer);
    virtual int  fromBufferSwitch(const char **buffer);
    virtual void print() const;
    static Message *allocate(messageType type);
protected:
    Message(messageType type);
    ....
}
Client and server code normally only call new, delete, getType, and message
specific methods. The other methods are called by MPF itself. A message
is allocated via a call to new and released via a call to delete. When
delete is called the message is placed on a free list rather than actually
being deleted.

Most methods are virtual and must be implement by any class that derives from Message. The init method is called whenever a messages in allocated. The toBuffer and fromBuffer methods are called to put and take messages from a buffer that passes over a communication link.

 Example

The following example is a client that send an Int32Message to a server.
The server receives the message, increments a private counter, sets the
long value equal to the previous counter value, and sends the message back
to the client.

Client

typedef struct privateData {
    connectStatus connectState;
    int32         value;
    SEM_ID        sem;
}privateData;
 
void myCallback(Message *message,void *clientPvt)
{
    privateData *pprivateData = (privateData *)clientPvt;
 
    switch(message->getType()) {
    case messageTypeConnect:
        {
            ConnectMessage *pConnectMessage = (ConnectMessage *)message;
            pprivateData->connectState = pConnectMessage->status;
            delete pConnectMessage;
            semGive(pprivateData->sem);
        }
        break;
    case messageTypeInt32:
        {
            Int32Message *pInt32Message = (Int32Message *)message;
            pprivateData->value = pInt32Message->value;
            delete pInt32Message;
            semGive(pprivateData->sem);
        }
        break;
    default:
        printf("got illegal response message\n");
        break;
    }
}
 
int clientExample(char *server,int location,int ntimes)
{
    MessageClient *pMessageClient;
    privateData *pprivateData = new privateData;
    int         status;
    Message     *message;
 
    pprivateData->connectState = connectNo;
    pprivateData->value = -1;
    pMessageClient = new MessageClient(myCallback,pprivateData);
    pprivateData->sem  = semBCreate(SEM_Q_PRIORITY,SEM_EMPTY);
    status = pMessageClient->bind(server,location);
    if(status) {
        printf("bind returned %d\n",status);
        return(status);
    }
    //wait for connect
    semTake(pprivateData->sem,WAIT_FOREVER);
    for(int i=0; i<ntimes; i++) {
        while(pprivateData->connectState!=connectYes) {
            printf("client connectState %d\n",pprivateData->connectState);
            taskDelay(60);
        }
        message = new Int32Message;
        status = pMessageClient->send(message);
        if(status) {
            printf("send returned %d\n",status);
            continue;
        }
        semTake(pprivateData->sem,WAIT_FOREVER);
        printf("new value is %ld message %p\n",pprivateData->value,message);
    }
    return(0);
}

Server

struct privateData {
    int32 counter;
};

int serverExample(char *name)
{
    privateData   *pprivateData = new privateData;
    Message       *message;
    Int32Message   *pInt32Message;
    MessageServer *pMessageServer;
 
    pMessageServer = new MessageServer(name);
    pprivateData->counter = 0;
    while(TRUE) {
        pMessageServer->waitForMessage();
        message = pMessageServer->receive();
        if(message->getType()!=messageTypeInt32) {
            printf("serverExample got illegal message type\n");
        } else {
                pInt32Message = (Int32Message *)message;
                pInt32Message->value = pprivateData->counter++;
                pMessageServer->reply(pInt32Message);
        }
    }
    return(0);
}

Configuration

Each vxWorks system using the message passing system needs the following
command in the startup file:

routerInit

If local communication is desired then the following command must appear:

localMessageRouterStart(0)

The parameter is the location of the local server. The above example sets
the location to 0.  Note that the client specifies a location when
it binds to a server.

If tcp communication is desired then for each remote system the client must have the following command:

tcpMessageRouterClientStart(location,port,"address")

• location is the location assigned to the server.
• port is the port the server will use to listen for a connection.
• address is the inet address of the  server.

Each system supporting remote servers must have the following command for
each client  machine:

tcpMessageRouterServerStart(location,port,"address")

• location is the location assigned to the server.
• port is the port the server will use to listen for a connection.
• address is the inet address of the  server. This argument can be null, in which case the server will accept connections via all internet interfaces attached to it's cpu.

Architecture Independent Message Passing

MPF supports message passing between different architectures, i.e. different
byte orders and/or different padding requirements. It does, however only
work on architectures with 32 bit addressing. It provides support for the
following data types:

• signed and unsigned 16 and 32 bit integers
• IEEE float and double.
• C style ascii strings
• octet arrays. Octets are uninterpreted 8-bit values whose contents are guaranteed not to undergo any conversions during transmission.

In order to support architecture independent message passing MPF provides
some help but also requires that each message type provide methods to transfer
data between a message instance and the network buffer.

MPF provides a header file "mpfType.h" which defines typedefs for int16, uint16, int32, uint32, float32, and float64. MPF provides static methods to transfer the following to/from network buffers:

• signed and unsigned 16 and 32 bit integers
• IEEE float and double
• C style ascii strings
• Octet arrays

Adding New Message Types

In order to provide a new message type the following must be done:

• messageType.h and messageType.cc must be modified.

These files contain instructions about what to add. This is a simple change.
• A header and source file for the new message must be created.

The following are a skeleton header and source file for a new message type
. A particular message type can provide additional methods and data items
as desired.

xxxMessage.h

// xxxMessage.h

#ifndef xxxMessageH
#define xxxMessageH

class Message;
class DLList;

class xxxMessage : public Message
{
public:
    xxxMessage();
    virtual ~xxxMessage() {}
    virtual void init();
    virtual int32  toBuffer(char **buffer) const;
    virtual int32  fromBuffer(const char **buffer);
    virtual int32  fromBufferSwitch(const char **buffer);
    virtual void print() const;
    static void *operator new(size_t size);
    void operator delete(void *p, size_t size);
private:
    static DLList *pFreeList;
    // Disallow copy constructor, assignment operator 
    xxxMessage(const xxxMessage&);
    xxxMessage& operator=(const xxxMessage&);
};

inline xxxMessage:: xxxMessage() : Message(messageTypexxx), value(0)
{ init(); }

#endif /*xxxMessageH*/

xxxMessage.cc

// xxxMessage.cc

extern "C" {
#include <stdlib.h>
#include <stddef.h>
#include <string.h>
#include <stdio.h>
}

#include "Message.h"
#include "xxxMessage.h"

DLList* xxxMessage::pFreeList = 0;

void xxxMessage::init() { value=0; }

int32 xxxMessage:: toBuffer(char **buffer) const
{
    int32 nbytes;

    nbytes = Message::toBuffer(buffer);
    //now transfer any values added by the message type
    // nbytes += MPF::toBuffer(buffer,???);
    return(nbytes);
}

int32 xxxMessage:: fromBuffer(const char **buffer)
{
    int32 nbytes=0;

    nbytes = Message::fromBuffer(buffer);
    //now transfer any values added by the message type
    //nbytes += MPF::fromBuffer(buffer,&???);
    return(nbytes);
}

int32 xxxMessage:: fromBufferSwitch(const char **buffer)
{
    int32 nbytes=0;

    nbytes = Message::fromBufferSwitch(buffer);
    //now transfer any values added by the message type
    //nbytes += MPF::fromBufferSwitch(buffer,&???);
    return(nbytes);
}

void xxxMessage:: print() const
{
    Message::print();
    //print any values added by the message type
    //printf("value %d\n",???);
}

void *xxxMessage:: operator new(size_t size)
{
    if(size==sizeof(xxxMessage)) return MPF:: allocate(&pFreeList,size);
    printf("xxxMessage new size !=sizeof(xxxMessage)\n");
    return(0);
}
 
void xxxMessage:: operator delete(void *p, size_t size)
{
    if(size==sizeof(xxxMessage)) {
        MPF::free(pFreeList,p,size);
        return;
    }
    printf("xxxMessage delete size !=sizeof(xxxMessage)\n");
}

Discussion

I claim that the proposed message passing system will provide all the features
needed to support existing Hideos IP tasks and the Hideos EPICS device
support.  In addition it provides the following features not supported
by Hideos.

1. Network Transparent message passing. This will allow messages to be passed between different machine architectures.  The initial version, however, only works on 32 bit architectures.
2. Connection Management - The initial release will allow either server or client to be started first. It will also allow either a client or server to be rebooted without requiring the partner to be rebooted.
3. Messages can have complicated methods. (HIDEOS required that all message methods be in-line)

Status

Simple messages have been sent between client and server via both a local
router and via a tcp router.  Connection managemt still needs work,
i.e. neither client or server can survive a reboot of the partner.

Joe Sullivan is currently developing mv167 and mv162 board support packages that allow a 162 to be booted via the VME backplane. this will be for vxWorks 5.3

The EPICS Hideos device support and the Hideas IP tasks have not been modified to  support the new Message Passing facility.