futebus - FUse TExt BUS
~~~~~~~~~~~~~~~~~~~~~~~

Futebus is a text based message bus implemented on top of fuse. A bus is
created by mounting it. The bus can have multiple channels, which are files
in the mounted directory. Channel files are created by user processes.

Multiple processes can open the same channel file for read and/or write.
Whenever a message is written into a channel file, it is relayed to other
processes reading the channel.

In this document the following terminology is used:
 - bus: a futebus mount (a directory that hosts channels)
 - channel: a (virtual) file in a bus
 - client: an open file handle on a channel
 - master: on channel types that supports, one of the clients that is
   treated as the central point of one-to-many communication
 - slave: on channel types that supports, all the non-master clients
 - message: a single line (\n terminated) with a certain size limit

There are a few different bus types depending on:
 - whether the bus treats all connected clients equally or there is one
   special (master) client serving the other (slave) clients
 - whether clients can be addressed or all messages are broadcast


I. Bus mechanism

Every client for every open file has three buffers:
 - wbuf: a byte organized write buffer, where the incoming message is collected
 - rlist: a list of messages waiting for delivery to a specific client/channel
 - rbuf: a single message being read by a client from a channel

The client writes wbuf in multiple write() calls. Whenever a \n is
written, a message is completed, removed from the wbuf and placed to the rlist
of any client reading the channel at that moment. If the client closes the
file with a partial message in wbuf, the partial message is lost. Empty
(zero-length) messages are discarded. Messages longer than a certain message
length limit result in write error.

The read-message-list, rlist, holds the messages that should be delivered to
a client on a channel, as an ordered list. It normally behaves like a FIFO,
but there is a certain limit on the number of messages per rlist. When the
limit is reached, flow control takes place:
 - for a non-master client a new message is added at the cost of removing
   the oldest waiting message.
 - for a master client, the write() from the slave blocks until the
   master can receive messages again (see blockwr.txt)

When a client starts to read() a channel, the oldest message from the rlist
is copied into the client's rbuf. Any read() is served from the rbuf(), to
guarantee the message is delivered even if the client is reading it in
multiple smaller chunks, without risking the message is removed during the
process (when rlist gets full).


II. Bus types

There are different types of channels depending on the first character of
their channel file name, each with their own way of message distribution
among clients.


1. Unaddressed one-to-many channels (file name starting with 'U')

The first process creating and opening the channel file becomes the master
process, any subsequent process opening the channel becomes a slave. When
a slave sends a message, it is delivered only to the master. When a master
sends a message, it is delivered to all slaves.

Other than the bus and channel names, there is no addressing.

If the master process closes the file:
 - the file is unlinked
 - subsequent writes to the file result in error
 - subsequent reads pop existing rlist/rbuf items
 - read() when rlist and rbuf are empty results in EOF

The master process must open the file name with an extra "." prefix to
the U prefixed file name. This way a casual slave process (e.g. a
shell echo > /mountpoint/U_foo) can't accidentally become the master even
if mount point directory permissions would let it. The file is created
without the "." prefix.

2. Addressed many-to-many channels (file name starting with 'A')

There is no master client.

Messages are prefixed with an address. The address is always in curly
brackets and holds the either a hexadecimal integer that identifies a
client or the single character "*" which means the message is broadcast.
Examples on valid addresses: {5}, {1a3F}, {*}. There is no whitespace
inserted after the closing bracket.

When a client writes a message, the message must be prefixed with an
address, else write() returns error. Once the message is read, the address
is decoded and the message is copied to the rlist of the client(s) addressed,
with the address modified in the message to the sender's address.

When a client gets a message using read(), it always has an address prefix
with the integer ID of the sender client.

Note: clients may close the file any time and client IDs may be reused. It
is possible that a client receiving a message from {6} is sending its reply
later to a different client identified as {6}. This problem should be
considered in the message format design.

Practical application for unicast messaging: a client can't learn its own ID;
when it join, it should broadcast a message announcing/introducing
itselves, this is how other clients can learn its ID. Alternatively it
could broadcast a discover request on which other clients may answer.

3. Service: addressed one-to-many channels (file name starting with 'S')

A combination of 1. and 2.: there is a master client, like in 1, which
uses address prefixed communication. Any other clients are slaves, sending
and receiving raw messages, without address prefixing.

A slave can send messages only to the master and can receive messages
only from the master. The master can send broadcast messages or unicast
messages to specific slaves and with any messages received it gets the
sender slave ID.

When a client opens or closes the channel, a special empty control
message is sent to the master, prefixed with {ID:o} or {ID:c}, where ID
is the hexadecimal client ID. These control messages bypass flow control
and are always inserted in the master's rlist.

The master process must open the file name with an extra "." prefix, see
point 1.

4. Broadcast, unaddressed many-to-many channels (file name starting with 'B')

There is no master client.

Any message sent is delivered to any process reading the channel, except to
the one that placed the message in the channel.

The file is not unlinked automatically, even if all clients closed the file.

Other than the buses and channels, there is no addressing.

III. Misc

There is a read-only file called _stat in every bus. When open, a human
readable snapshot of all channels and clients is copied into the rbuf of
the client. This snapshot is then served in subsequent read() calls until
it runs out and an EOF is delivered. _stat is intended for debugging; the
file format is not well defined or guaranteed, computer processing of the
file is discouraged.

Poll(): all channel types can be polled for POLLIN. Most channel types are
always writable for most clients (excess messages are discarded), so POLLOUT
is always true. There is a special case: when a slave writes to a
master on 'U' and 'S' channels; if the master's buffer is full, the slave's
write() is blocked (see blockwr.txt). However, there is a race condition in
checking for this, because between the time of returning POLLOUT and the time
the client actually attempts to write other clients may fill up the master's
rlist and the write could block. Thus clients should poll for POLLOUT on
futebus channels.

Non-blocking read: if a client sets O_NONBLOCK on its fd, read() will return
immediately with EAGAIN if there if the clietn's rbuf and rlist are empty.

Failed calls set errno, see errno.txt.

For invocation, see the manual page futebox(1).