TIPC

Transparent Inter-process Communication (TIPC) is a network communications protocol for inter-process communication (IPC) that was specially designed for intra-cluster communication.

Features

Some features of TIPC:
* Location transparency of services in a network.
* Auto-discovery mechanism.
* Reliable transport.
* Standard socket interface support.
* Connectionless, connection-oriented and multicast messaging.
* Subscription to network events.
* Dual BSD/GPL licensed code.

Implementations

TIPC protocol support is available in Linux kernel starting with version 2.6.16. It can also be built into earlier versions of the kernel as a supplementary module. Other operating systems are also supported, including Wind River's VxWorks and Sun Microsystems' Solaris.

Bearer Media

While designed to be able to use all kinds of bearer media, implementations only support Ethernet at the moment (October 2007). The VxWorks implementation also supports a shared memory which can be accessed by multiple instances of the operating system, running simultaneously on the very same hardware.

Neighbouring Node Links

Link setup

A TIPC node periodically broadcasts "Link Request" messages on all configured media interfaces to detect neighbouring cluster nodes. If such a message is received by a node with no prior established link to the sending node, it replies with a unicast "Link Response" message. This establishes a "Link" between those two nodes.

Link Continuity Check

A background timer is maintained for each link. When a certain amount of time is passed without regular incoming traffic, a message is sent over the link to indicate to the counterpart that the link is still up. This message also contains an acknowledge for the last received "Link Level Sequence Number", in order to allow the receiver to release sent packet buffers, and a "Last Sent Sequence Number", allowing the receiver to detect gaps in the packet sequence. The continuity check mechanism allows rapid detection of communication media failure, or node crashes. The "Link Tolerance" is a configurable parameter for each link endpoint, determining how long the other link endpoint may remain unresponsive before the link is declared faulty, and reset. For Ethernet the default value of this parameter is 1.5 s. After a reset, the remaining link endpoint will continue to probe the link until it is re-established.

Logical Network Topology

The logical Network Topology is not necessarily equal to the physical one.

TIPC nodes with point-to-point links to each other () are logically grouped and named a "Cluster". Fully meshed clusters can in turn be grouped in "Zones". Again, the zones of a TIPC network have to be able to reach each other directly.

Addressing scheme

Unlike in most other network protocols like IP, the address is not attached to an interface but to the whole physical node. Also a node can only possess one single address which identifies it throughout the whole network.

The addressing scheme is mapped to the logical Network Topology. The human readable notation is where "Z" stands for Zone, "C" for Cluster and N for Node. Internally the address is represented as a 32-bit integer. Here, the 8 most significant bits identify the Zone, the next 12 ones the Cluster and the 12 least significant bits the individual Node.

Communication semantics

In order to be able to adapt to the user's needs, TIPC allows to choose from four different communication semantics:
* Unreliable connectionless messages (SOCK_DGRAM). This is comparable to UDP.
* Reliable connectionless messages (SOCK_RDM).
* Reliable connection-oriented messages (SOCK_SEQPACKET).
* Reliable connection-oriented byte streams (SOCK_STREAM). This is comparable to TCP.

Protocol Operations

The protocol operations are optimized for networks matching the following assumptions:
* Most messages cross only one direct hop.
* Transfer time for most messages is short.
* Most messages are passed over intra-cluster connections.
* Packet loss rate is normally low; retransmission is infrequent.
* Available bandwidth and memory volume is normally high.
* For all relevant bearers packets are check-summed by hardware.
* The number of inter-communicating nodes is relatively static and limited at any moment in time.
* Security is a less crucial issue in closed clusters than on the Internet.While being able to work within environments not complying to this list, TIPC will not be able to show its advantages over other protocols which were designed to work within a wider range of parameters.

Packet loss detection

Packet loss detection is done on the node-to-node link level. A packet is given a 16 bit "Link Level Sequence Number" which is incremented modulo 2¹⁶-1 every time a packet is created and sorted into the link's send queue. At the time a packet is taken from this queue and actually sent, its "Link Level Sequence Acknowledge Number" field is filled with the latest received in-sequence "Link Level Sequence Number" of the respective link. When a link endpoint receives the acknowledge it purges the messages with the equal and lower sequence numbers from its memory. After 16 incoming messages with no traffic in the other direction which would have been used to acknowledge them, a receiving node transmits an extra message in order to achieve this.

In case the sequence number of an arriving packet indicates that there were one or more previous packets lost, the receiving node calculates the gap and reports it to the sender who instantly retransmits the missing packets. For both connectionless and -oriented message flows, packet or byte-stream order is maintained from the receiving application's point of view.

Flow control

User-to-user flow control is only supported for established connections. Here, flow control is achieved by keeping a counter SENT_CNT on the sending side which is increased with every sent package. The receiving node counts the packages which were read by the receiving application. When this counter reaches the configurable number N, it informs the sender of this, which subtracts N from SENT_CNT. In case SENT_CNT reaches a certain limit, the sender stops sending packages until SENT_CNT falls under the threshold again.

The recommended value for N is at least 200, while the threshold for SENT_CNT should be at least 2N.

When the communication mode is connectionless, flow control can only be performed at the application level, implemented according to the particular characteristics of the application.

Apart from this, there are various mechanisms for "congestion control" in TIPC, both at bearer, link and node level. These are transparent for the user programs.

Development

The TIPC Project Team is currently involved in the evolution of TIPC and is engaged in developing a freely available, portable implementation of the TIPC protocol.

History

This protocol was originally developed by Jon Paul Maloy at Ericsson, and has been used by that company in carrier-grade cluster applications for many years, before subsequently being released to the open source community.

Reference Links

* TIPC Project Team: http://tipc.sourceforge.net
* Linux Kernel: http://www.kernel.org