Channel Access Protocol Specification

This document is distributed under the terms of the GNU Free Documentation License, version 1.2.

This document is a specification of the EPICS Channel Access (CA) protocol. Structure of messages exchanged between communicating nodes is defined, as well as semantics of the exchange. The documentation is focused on version 4.11 of the CA protocol, which comes with EPICS 3.14, and later versions.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", "OPTIONAL" in this document are to be interpreted as described in RFC 2119: Key words for use in RFCs to Indicate Requirement Levels[3].

An implementation is not compliant if it fails to satisfy one or more of the MUST or REQUIRED level requirements for the protocols it implements. An implementation that satisfies all the MUST or REQUIRED level and all the SHOULD level requirements for its protocols is said to be "unconditionally compliant"; one that satisfies all the MUST level requirements but not all the SHOULD level requirements for its protocols is said to be "conditionally compliant."

This document is based on the original CA protocol implementation that accompanies EPICS. Wherever the MAY, OPTIONAL or RECOMMENDED is specified, it implies that experience has shown such behaviour to improve either performance, portability or memory consumption.

A protocol implementation that is not unconditionally compliant MUST NOT be used in any production environment or used with EPICS databases controlling physical devices.

A Process Variable (PV) is representation of a single value within an EPICS host.

A channel is created whenever a PV is connected using CA.

From implementation point of view, a channel is a connection, established over virtual circuit, between server and client through which a single PV is accessed. Both the client and the server will provide a way of uniquely identifying channels. These identifiers are explained in section Message Identifiers.

A monitor is created on a channel as a means of registering for asynchronous change notifications. CA protocol defines the subscription mechanism through which clients register for notifications. These changes will then be provided through monitor object or callback, depending on implementation and environment. Monitors may be filtered to receive only a subset of events, such as value or alarm changes. Several different monitors may be created for each channel.

A client library implements channel access protocol and exposes it through a programmer-friendly API. How the protocol and its mechanisms are implemented in the library depends on the programming language and no specific requirements are made with respect to this.

Repeater is a standalone process that allows several clients on one machine to share a predefined UDP port for recieving broadcast messages. Broadcast messages are sent without prior knowledge of which port the CA clients and hosts are listening at, so predefined port numbers are used. Since most network stack implementations do not allow multiple clients to listen on the same port, repeater is installed to listen on a single port, used for broadcast notifications. Repeater will fan-out unmodified incoming datagram messages to all the clients on the same host, that have previously registered with the repeater.

Typically, a client library will attempt to register with the repeater during startup. If repeater cannot be found or is not available, the library will attempt to spawn a new repeater process. Repeater and client communicate using a minimal set of messages over UDP.

Server beacons are simple protocol messages that allow servers to broadcast their availability. These messages are sent out periodically to announce their presence. Additionally, these beacons are used to restore virtual circuits that have lost connections.

Channel Access protocol is designed to minimize resources used on both client and server. Virtual circuits minimize number of TCP connections used between clients and servers. Each client will have exactly one active and open TCP connection to an individual server, regardless of how many channels it accesses over it. This helps to ensure that servers do not get overwhelmed by too many connections.

The life-cycle of a virtual circuit is defined in section Virtual Circuit Operation.

When sending packets over network, channel access allows huge savings to be made by grouping individual messages in a single TCP packet. This is performed by maintaining per virtual circuit buffers that collect all outgoing messages. These buffers are flushed upon application’s explicit request. This mechanism is independent of the TCP/IP stack, which maintains its own send queue and defines a maximum frame size. Messages sent over virtual circuit may be larger than this and will not always be aligned on frame boundary. Implementation should also be aware that messages received may not yet be available entirely or will be split over several TCP frames.

Certain aspects of Channel Access protocol have changed between releases. In this document, Channel Access versions are identified using CA_VXYY, where X represents single-digit major version number and YY represents a single- or double-digit minor version number. Stating that a feature is available in CA_VXYY implies that any client supporting version XYY must support the feature. Implementation must be backward compatible with all versions up to and including its declared supported minor version number.

Currently, all protocol definitions are assumed to have major version 4. Minor version ranges from 1 through 11.

This document uses the concept of exceptions to refer to the mechanism of reporting errors that occur during command request execution. An exception defines reporting of error condition on the server. This can occur either due to client problem or due to some unexpected condition on the server (such as running out of resources). Exceptions are reported either within the command’s response or using a specialized message. Actual form and method (response or asynchronously received message) depends on circumstances where the exception occurs. Individual commands and messages determine which method is used.

This section is intended as a quick overview of channel access functionality and behaviour. For more information, please refer to Channel Access Reference Manual[1].

The goal of channel access (CA) is to provide remote access to records and fields managed by IOC, including search and discovery of hosts and minimal flow contol. Protocol itself is designed to provide minimal overhead and maximize network throughput for transfer of large number of small data packets. Additionaly, implementation overhead of the protocol can be kept very small, to allow operation with limited resources.

All commands and events in CA are encapsulated in predefined messages, which can be sent in one of three forms:

Communication between server and client is performed by sending command messages over UDP and TCP. Client will use UDP to search for hosts and PVs, server will use them to notify its startup and shutdown. Once client requests a specific PV (by specifying its name), UDP message will be broadcast to either a subnet or a list of predefined addresses, and the server which hosts requested PV will respond.

Data exchange between client and server is performed over TCP. After locating the PV, the client will establish a TCP connection to the server. If more that one PV is found to be on the same server, client will use existing TCP connection. Reusable TCP connection between client and server is called Virtual circuit.

Once a virtual circuit is established or already available, channels can be created to PVs.

Let’s assume the following setup of PVs and hosts:

A typical scenario would be similar to this:

This section defines all primitive data types employed by the CA, as well as their C/C++ equivalents. These data types are referred to in the subsequent sections.

All values are transmitted over the network in big-endian (network) order. For example: UINT32 3145 (0x00000C49) would be sent over the network represented as 00 00 0C 49.

All channel access messages are composed of a header, followed by the payload.

Header is always present. Its structure is fixed, and contains predefined fields. At the very least, this will be command ID and payload size. Other header fields may carry command-specific meaning. If a field is not used within a certain message, its value must be 0 (0x00).

Payload is a sequence of bytes, which are command and version dependent. Implementation is required to provide proper payload with respect to individual command specification.

Total size of an individual message is limited. With CA versions older than CA_V49, the maximum message size is limited to 16384 (0x4000) bytes. Out of these, header has a fixed size of 16 (0x10) bytes, with the payload having a maximum size of 16368 (0x3ff0) bytes.

Versions CA_V49 and higher may use the extended message form, which allows for larger payloads. The extended message form is indicated by the header fields Payload Size and Data Count being set to 0xffff and 0, respectively. Real payload size and data count are then given as UINT32 type values immediately following the header. Maximum message size is limited by 32-bit unsigned integer representation, 4294967295 (0xffffffff). Maximum payload size is limited to 4294967255 (0xffffffe7).

Extended message form should only be used if payload size actualy exceeds the pre-CA_V49 message size limit of 16368 bytes.

Names of header fields are based on their most common use. Certain messages will use individual fields for purposes other than those described here. These variations are documented for each message individually. All of values in header are unsigned integers.

The common header present in all messages is structured as follows:

The extended message header (CA_V49 and newer) has the following structure:

The structure of the payload depends on the type of the message. The size of the payload matches the Payload Size header field.

Some fields in messages serve as identifiers, which must be properly handled by the implementation. These fields serve as identification tokens in asynchronous communication and must be unique within the context of the client library. Recommended scheme for allocating these values is to create them sequentially starting at 0. All IDs are represented with UINT32.

Overflow must be anticipated! Although it is unlikely that an application will need more than 2³² different channel IDs, such an overflow can occur with the IOID identifier (see below).

CID identifies individual channel within client library context. When a message requires the CID, this value must be passed. CID for a given channel may not change within the lifetime of the channel.

Recommended way of allocating CIDs is sequential 0-based indexing. First client library channel reference that is created has value of 0, second 1, etc. Duplicate CIDs within a single active instance of client library are not allowed.

CID value is passed to the server when the channel is actually connected.

Overflow should be potentially handled, since an application with expected long lifetime and frequent channel allocation and destruction might increase the index frequently. The best way to handle it is to ignore the problem until the first overflow. When that occurs, old CIDs should be no longer relevant, and CID value can be restarted from 0, but excluding any already used values.

Performs the same function as CID, but is provided by the server when the channel is connected. SID of channel will never change, unless the channel represented by particular SID is removed from the server during runtime. In that case, the channel will be no longer available.

To handle overflows, the same method could be used as for the Client Channel ID.

When a subscription is created on a channel (monitor), a unique subscription ID must be provided. Client library will use this ID to identify different subscriptions on the same channel. This value will allow the client library to dispatch monitors appropriately.

Any subscription ID is only valid during the subscription’s lifespan. After that, IDs may be reused.

Unique ID is given to all read and write messages sent by the client library.

ID passed with the request will be returned in the matching response.

Properly handling the wrapping of identifiers is vital to IOID, since an individual ID is used each time a request is made.

The following commands are sent as either UDP datagrams or TCP messages. Some of the messages are also used within the context of a Virtual Circuit.

The following commands are sent as UDP datagrams.

The following commands are used within the context of Virtual Circuit and are sent using TCP.

Payload

Deprecated since protocol version 3.13.

|[options="header"]

Channel access defines special structures to transferring data. Main reason is efficiency, since in many cases more than one value can be transferred. These types are organized in typed hierarchies with loose inheritance. There are seven basic data types: DBR_STRING, DBR_SHORT, DBR_INT, DBR_FLOAT, DBR_ENUM, DBR_CHAR, DBR_LONG and DBR_DOUBLE. Each of these types can be represented as an array, if the corresponding field in header indicates that.

Additional to basic data types, structured types allow access to more than one value within the same record. These structures are status (STS), time stamp (TIME), graphic (GR) and control (CTRL). All these structures contain value as the last field.

Status structure adds alarm severity. Time stamp structures extends the status structure by adding time stamp of the value. Graphic extends status structure by providing alarm limits, units and precision. Control structure extends graphic by adding control limits.

Following is the list of all structures as transmitted over network.

Although there is no requirement as to which port numbers are used by either servers or clients, there are some standard values which must be used as defaults, unless overriden by application.

Port numbers are dependant on protocol versions and are calculated using the folowing definitions:

CA_PORT_BASE = 5056

CA_SERVER_PORT = CA_PORT_BASE + MAJOR_PROTOCOL_VERSION * 2

CA_REPEATER_PORT = CA_PORT_BASE + MAJOR_PROTOCOL_VERSION * 2 + 1

Based on protocol version described in this document (4.11), port numbers used are CA_SERVER_PORT = 5064 and CA_REPEATER_PORT = 5065.

Since registration of port numbers with IANA and in the interest of compatibility, the version numbers are unlikely to change. Therefore, the port numbers described here (5064 and 5065) may be considered final.

This section lists various constants, their types and values used by protocol.

Some constants can be combined using logical OR operation. Example: Monitor mask of DBE_VALUE and DBE_ALARM are combined using (DBE_VALUE or DBE_ALARM) resulting in (1 or 4 == 5).

To query the whether certain value is present in such combined value, and operation is used. Example: to query whether DBE_ALARM of monitor mask is set, (DBE_VALUE and MASK > 0) will return 0 if DBE_VALUE is not present, otherwise DBE_ALARM is present.

Indicates which changes to the value should be reported back to client library. Different values can be combined using logical OR operation.

Type: not defined, depends on the field it is in (usually UINT16)

Indicates whether server should reply to failed search messages. If a server does not know about channel name, it has the option of replying to request or ignoring it. Usually, servers contacted through address list will receive request for reply.

Type: not defined, depends on the field it is in (usually UINT16).

Defines access rights for a given channel. Accss rights are defined as logicaly ORred value of allowed access.

Type: not defined, depends on the field it is in (usually UINT16).

As a reference, the following values are valid.

This example shows construction of messages. For details of individual structures, see message and data type reference (CA_PROTO_READ_NOTIFY and DBR_GR_INT16).

A client will send CA_PROTO_READ_NOTIFY message with the following contents.

The messsage would be represented as follows:

Server would respond with success and return requested value with individual DBR_GR_INT16 fields having the following values.

Virtual circuit is a TCP connection between a client and a server. Each virtual circuit has an associated priority, which can be used by server to prioritize requests depending on current load. An independent circuit for each priority level selected by the client might be used because it allows for preemptive prioritized dispatch scheduling in the server when the OS supports that, and also allows specifying the networks scheduling priority when the router and or LAN support that. Regardless of how many channels are handled by either client or server, each client-server pair will be connected with exactly one TCP connection for each priority level. This level is specified when creating a virtual circuit.

When establishing a virtual circuit, a simple handshake will be performed. Client will open a TCP connection to the server. After that, it sends CA_PROTO_VERSION, CA_PROTO_CLIENT_NAME and CA_PROTO_HOST_NAME messages. If server accepts all the supplied parameters and the client and host are permitted to connect, the server will respond with CA_PROTO_VERSION. After that, client may start sending requests.

Virtual circuit will remain active as long as there is at least one channel active. If the TCP connection is lost, all channels using this circuit must be notified.

In most basic role, the client will send requests to the server and await one or more responses, or in some special cases, no response will be expected. For each type of request, one of three events will occur:

When a matching response is received, the operation has completed successfully. Any result returned will report any applicable return information. Client library will then notify the client about completion.

If a request fails and server could handle this failure, CA_PROTO_ERROR response will be received or error code is received in response, depending on the request sent. This response contains original request header, error code and text description of error.

Although servers are designed to always return a response, in some cases this will not occur. An unlikely case is that a server has failed or stopped responding. A more common case is broadcast search, where no replies are sent to search messages. The later case is designed to not overload the network by sending failure notifications. This however does not allow client library to determine, whether a search has failed, or the server load is simply too high and the response will arrive later. In this case, client library may assume that search yielded no results and notify on search failure. Regardless of this, should any such response arrive late, the client must be notified.

In case of network failure, two situations should be anticipated. One is TCP connection loss for a virtual circuit. Virtual circuit has mechanisms for dealing with such case and should attempt to restore the connection itself. Repeater has no such mechanisms, since it depends on reliability of UDP protocol. There will be in most cases no way for repeater to determine, whether the packets sent actually arrive at destination. Repeater can however verify the clients registered with it, by attempting to bind to their port. If binding succeeds, the client no longer exists. Alternate way is using connected UDP sockets and checking ICMP destination unreachable error.

During virtual circuit life-time, the circuit may become unresponsive. Since this is tightly related to network availability and server load, determining actual cause for unresponsiveness is difficult. Basic criteria on which unresponsiveness should be determined are:

Behaviour of virtual circuit implementation under such conditions is undefined. Experience has shown however, that virtual circuit should not be disconnected when it becomes unresponsive, since this negatively impacts network performance under load.

A channel discussed here is based on two levels. One is channel as seen from client application, the other is channel implementation in client library. This difference will be pointed out where applicable.

When a channel is created by client application, it is initialized in NEVER_CONN state. This indicates that channel is currently being connected for the first time. If connection process fails, channel moves into FAILED state and is not usable. Connection proccess within the client library will attempt to connect the channel repeatedly. Intervals between consecutive attempts should be increased, before finally giving up and determining the channel cannot be connected and signaling FAILED state. A total number of attempts can be on the order of 100.

When connection is established, the channel will signal to client with CONN_UP event and changing its state to connected.

Should the virtual circuit connection be lost, CONN_DOWN event will change channel state to disconnected. Whenever virtual circuit is reconnected, the channel state is restored to connected. Note that reconnection process follows the same rules as initial process. It is also possible, that the PV will move between the servers, but client application should not be aware of such internal changes to channel.

Once client application is done using the channel, it will be closed and its state moved changed to CLOSED. After that, this channel can no longer be used.

Detailed description of channel operations is outlined below.

Connection of a channel is a two phase process. First, server that hosts channel with a given name must be located. Second, client and server must both register a connection.

Before connection, client library allocates CID identifier that will be used to reference new channel. Next it sends CA_PROTO_SEARCH with channel name request via broadcast and/or to address list. Each server that receives such message checks to see if it knows about this name. If it does, response to CA_PROTO_SEARCH is sent back to client. If not, the server checks if reply field of request is DO_REPLY and responds with CA_PROTO_NOT_FOUND, otherwise does nothing.

Client library may receive more than one response. In this case, first response should be used and others rejected. The cause of multiple responses is same PV hosted on multiple different hosts. Client or server has no method of knowing which PV is right. Multiple responses should be reported by the client to the application or user.

After extracting server address and port, client library checks to see if this server is already known and connected via virtual circuit. If not, virtual circuit is established. Any requests sent through this channel will use this circuit.

For this it sends CA_PROTO_CREATE_CHAN request with channels CID and channel name. Server will respond with CA_PROTO_CREATE_CHAN response which will provide channel type, data count and server identifier SID. In case the channel could not be created, error message will be returned.

Having initialized both CID and SID and registered the channel with the server, this channel is ready for use. Client library should store these values for further use.

Read and write operations require the channel to be properly initialized and connected and its virtual circuit to be active. These operations use CA_PROTO_READ_NOTIFY and CA_PROTO_WRITE_NOTIFY messages.

To read a value from a channel client library will create a CA_PROTO_READ_NOTIFY request. This request will contain desired data type and data count parameters which may differ from channels native type as obtained during connect. Additionally, IOID is stored in request to provide unique identification. Channel that is used will be identified by SID that was obtained during channel creation. No payload is sent with request.

After server processes the message reponse is either valid CA_PROTO_READ_NOTIFY response or CA_PROTO_ERROR to indicate error. In case of success, response contains same field values as request, but has additional payload. This payload is formatted as requested data type with desired number of elements.

Write operation is performed identically with payload roles reversed. Here the request will contain payload with DBR formatted value to write and response will have no payload.

Client creates monitors by registering a subscription on a channel. This causes server to notify subscribers of value changes.

Subscriptions are created by first allocating unique Subscription ID. This identifier is used to uniquely identify various subscriptions. Next, CA_PROTO_EVENT_ADD request is created using SID and Subscription ID. Data type and data count indicate desired value format in response. Additional information is provided in payload as specified in CA_PROTO_EVENT_ADD reference.

First CA_PROTO_EVENT_ADD response is received immediately to confirm successful subscription. If error occurs, CA_PROTO_ERROR response is received.

Responses arrive asynchronously until client cancels the subscription using CA_PROTO_EVENT_ADD response and payload containing DBR formatted value. If server is shutdown during this time, events will no longer arrive, but the subscription will not be cancelled. Once the server is restarted, client library will reestablish the subscription silently without notifying client application.

Subscription is cancelled by sending CA_PROTO_EVENT_CANCEL request with relevant parameters identical to those in original CA_PROTO_EVENT_ADD request that created the subscription. Successful cancelation is confirmed by CA_PROTO_EVENT_ADD response without payload.

Cancelling the subscription while the server is down should be handled by cancelling the subscription locally and not reestablishing it once the server is up again.

If a virtual circuit is disconnected (server goes down or stops sending beacons), any connected channels that use this circuit should be notified. There is no direct way to reconnect the channel from client side, but this will be done by the library. Once the circuit is reestablished, client application is notified of status change for all channels. Even if a channel is reported to be disconnect as a result of virtual circuit failure, the channel is not closed. The only way a channel connection is actually closed is by explicitly closing it using CA_PROTO_CLEAR_CHANNEL thereby invalidating its resources.

Channel is closed by sending CA_PROTO_CLEAR_CHANNEL request. Regardless of whether response confirmes closing or reports an error, CID and SID associated with the channel are no longer valid. Any resources still available to the client application are considered invalid. A channel that was cleared may no longer be reconnected.

Simple demonstration of repeater operation. A repeater will be used both on client and server side to forward incoming UDP requests to multiple clients and/or server on same host. Each host or client will then respond on its own.

Since role of the repeater is related to host, not the client, its implementation must be independent from individual client instance or process. Repeater must not stop operation until all clients depending on it registered with it have shut down.

Each client must check for presence of repeater on startup, before any access to EPICS hosts is made. This check is made by attempting to bind to CA_REPEATER_PORT. If binding fails, the client may assume the repeater is already running and may attempt to register. This is done by sending CA_REPEATER_REGISTER datagram to CA_REPEATER_PORT. If repeater is already active, it will respond with CA_REPEATER_CONFIRM datagram back to client, otherwise the client can assume, the repeater is invalid, the process bound to the port is not even a repeater or that some unknown error occured. In this case, client will spawn new repeater and attempt registration again. Attempt to detect the repeater and register should be made several times with some reasonable delay to avoid any transient unresponsiveness that might occur.

After the client has received registration confirmation from the repeater, it should keep listening for messages from the repeater. Any broadcasts sent by servers will be received by the repeater and forwarded to registered clients.

The repeater tests to see if its clients exist by periodically attempting to bind to their ports. If unsuccessful when attempting to bind to the client’s port, then the repeater concludes that the client no longer exists. A technique using connected UDP sockets and ICMP destination unreachable can also used. If a client is determined to no longer be present then the repeater un-registers that client and no longer sends messages to it.

Each message, the repeater receives, must be forwarded to local clients, to the address provided during registration. No assumption should be made about existence or state of clients.

Repeater should not shutdown on its own, if it does, there should be no active clients registered with it.

Each EPICS server will send beacons periodically to report it is still active. Beacon messages will contain server’s IP and port, as well as sequential beacon ID. Beacons will be broadcast and sent to servers address list.

When a server becomes active, it will immediately start sending beacons with an increasing delay. Time between beacons will start at 0.02 seconds. After each beacon is sent, this time is doubled. Maximum delay between beacons will be limited by server specified parameter, but is commonly 15 seconds.

If a beacon is not received within expexted time, virtual circuits connected to this server should be notified and virtual circuit must handle this situation.

This section covers return codes and exceptions that can occur during CA command processing. In general, exceptions will be used to report various events to the application. Return codes are predefined values for conditions that can occur, where as exceptions are actually reported. Apart from exceptions that occur on server or due to network transport, additional error conditions may be reported on the client side as local exceptions.

Implementation is required to handle all return codes. Local exceptions should be thrown whenever implied, but only if such exception is reasonable within the scope of implementation. Certain local exceptions that deal with state dependent exception must always be provided.

Naming convention: All return codes defined within the scope of CA are listed with all capitals, with underscore character (_) replacing spaces. Additionally, all exception names are prefixed with string ECA_.

Return codes are represented as UINT16. The lowest 3 least bits indicate severity, the higher 13 bits are a return code identifier.

Return codes are communicated in the protocol by the CA_PROTO_READ_NOTIFY, CA_PROTO_WRITE_NOTIFY, monitor subscription responses, and the CA_PROTO_ERROR responses.

This is example conversation between client and server. Client first establishes TCP connection to the server and immediately requests creation of a channel. After server aknowledges channel creation, client reads the value of the channel twice. First as a single string value and second as a DBR_GR_INT16 type. After the response to both queries has been received, the channel is destroyed.