Plumb2 is a program for starting and managing multiple processes and various pipes between them. Plumb2 itself can be controlled from programs running inside it, through pipes; so a program running under Plumb2 can add or kill processes or pipes during runtime.
For futher documentation, see the man page.
Plumb2 comes with an additional tool called plumbnet, for managing multiple TCP connections from a single program without having to do networking, threads, or select(2). See man page for more information.
Plumb2 and plumbnet are written in C99 (they currently depend on GCC to compile). The only other external dependency is libevent.
You can get Plumb2 from Subversion:
svn co svn://repo.hu/plumb/tags/2.0.0 plumb2
Debian packages are available from the repo.hu Debian repository.
Plumb2 may be useful together with other programs that communicate over their STDIN/STDOUT:
Copyright (C) 2009-2010 Máté Nagy <mnagy AT port70.net>
Plumb2 is licensed under the GNU General Public License Version 3 (available in the file COPYING) or any later version.
NAME
plumb2 - connect multiple programs with lots of pipes
SYNOPSIS
plumb2 [-v key value]+ [-V key values.. .]+ [-f filename] [-d] "com-
mands"
69 "command1" "command2"
DESCRIPTION
Plumb2 is a program for starting and managing multiple processes and
various pipes between them. Plumb2 itself can be controlled from pro-
grams running inside it, through pipes; so a program running under
Plumb2 can add or kill processes or pipes during runtime.
Multiple pipes may go from a single source to various targets, or from
various sources to a single target. Line buffering can be used to avoid
mixing up data; adding and filtering by line prefixes serve to distin-
guish sources and targets.
In the end a single program can communicate with and control a multi-
tude of other programs through just reading from STDIN and writing to
its STDOUT.
COMMAND LINE
The Plumb2 command line is a sequence of four different kinds of parame-
ters, in any order. Some of the parameters serve to set the values of
variables. These variables are used to pass in data (such as file-
names) to the Plumb2 control language.
-69 command1 command2
The two commands are started behind a shell (like with sys-
tem(3)) and connected together with two pipes (STDOUT to STDIN).
Invoking Plumb2 as "69 command1 command2" is equivalent to "plumb2
-69 command1 command2".
-v key value
Assigns a single value to a variable.
-V key value1 value2... .
Assigns a list of values to a variable (terminate list with a
single period).
-f filename
Read Plumb2 control commands from the specified file.
-d Increases the (debug) log verbosity. May be specified multiple
times. Log messages are written to Plumb2's STDERR. The initial
level is ERROR; further levels are WARNING, DEBUG, JUNK.
"commands"
Execute the given commands as Plumb2 control commands.
THE MODEL
Processes provide sources (such as their STDOUT and STDERR) and targets
(their STDIN). There are special sources and targets not associated
with processes (for example, Plumb2's own STDOUT is a target).
Pipes go between sources and targets. Each source has an ordered list
of pipes going from it. When data arrives at a source, the data will be
written into the pipes, in sequence. In certain conditions, a pipe can
decide to "break the chain" and stop data from getting to subsequent
pipes.
There is a special target (called :ctrl ) from which Plumb2 parses con-
trol instructions, and a source (called :ev ) to which Plumb2 writes
events (e.g. when a process dies). Not connecting them to anywhere is
not an error.
Process sources have a file descriptor and a buffering scheme. By
default, processes are started with two sources: STDOUT for FD 1,
STDERR for FD 2, both line-buffered. Fixed record buffering (always
waiting until N bytes are accumulated) and no buffering (sending any-
thing received right away) are the possible alternatives. Processes can
be started with more pre-opened sources on user-specified FDs.
Process targets have a file descriptor. By default, processes are
started with one target: STDIN for FD 0. Processes can be started with
more pre-opened targets on user-specified FDs.
Processes have an associated termination action: either "event" or
"kill". When a process terminates, either of two things can happen. If
"kill" is set, then Plumb2 exits completely (killing all started pro-
cesses). If "event" is set, only an event is generated, and everything
continues. The global default is "kill", adjustable per process or by
the "default proc" command.
Each source has an associated termination action: also either "event"
or "kill". This action is taken when the source gets an EOF (or an
error). The global default is "kill", adjustable per source or by the
"default eof" command.
Each target has an associated termination action: also either "event"
or "kill". This action is taken when the target gets a write error (not
when otherwise closed, by e.g. the termination of the associated
process). The global default is "kill", adjustable per target or by the
"default write" command.
Each source and target has an associated "cascade" flag. When this is
true, the source or target gets closed when all pipes going from/to it
are closed. The global default is true, adjustable per source/target or
by the "default source|target keep|cascade" commands.
CONTROL LANGUAGE
The Plumb2 control language is line-based. Each line is fed to the token
parser, which splits it into a sequence of strings called tokens; this
usually just means splitting the line into whitespace-separated words.
The generated lists of tokens are then interpreted as Plumb2 commands.
TOKEN PARSER
Lines are split into tokens along the whitespace between them. A hash
mark (#) begins a comment, the rest of the line will be ignored.
Multiple Plumb2 commands can be given in a single line, separated by
semicolons (;). No ending semicolon is necessary.
Any string of non-whitespace, non-special characters will be considered
a token in itself.
Tokens may be string literals, enclosed in quotation marks ("). All
characters until the closing quotation marks are considered part of the
same token, including spaces. Some special characters have to be
escaped using a backslash (\). Supported escape notation: \\ for a sin-
gle backslash, \n for a newline, \t for a tab, \b for a backspace, \xAB
for an arbitrary character given in hex.
Words beginning with the dollar sign ($) are considered variable names.
The variable name is the part after the $ (until the next whitespace or
special character). In place of the word, the tokens set as the value
of the variable will be substituted. Undefined variables will result in
no inserted tokens (and no error). A variable may have an arbitrary
number of tokens associated with it (including zero).
PLUMB COMMANDS
Some commands have IDs as parameters. IDs are ordinary tokens, but the
usable character set is restricted: [a-zA-Z_][a-zA-Z0-9_]*
set ID [values...]
Set the variable named by the ID to the given list of (zero or
more) tokens.
proc ID [kill|event] [-pause] [inputfd]* [outputfd]* = program [argu-
ments...]
This command creates a new process.
The ID identifies the process.
[kill|event] specifies the action to take when the process ter-
minates, overriding the default given with the "default proc"
command.
-pause disables reading from all sources (until an "unpause"
command is issued for the process). This may be useful to stop
data being lost if a process writes to a source before the
appropriate pipes going from the source are defined. It's guar-
anteed that no reading will take place during the interpretation
of commands given on the plumb2 command line, or between execut-
ing multiple commands given on a single line (using semicolon
separators).
Zero or more target (input FD) definitions may be given, in the
following form:
-i FD [kill|event] [keep|cascade]
[kill|event] specifies the EOF action, overriding the default
given with the "default eof" command.
[keep|cascade] overrides the cascade behavior, given with the
"default target keep|cascade" commands.
If -i 0 (STDIN) is not specified, it is added implicitly.
Zero or more source (output FD) definitions may be given, in
the following form:
-o FD [recordsize] [kill|event] [keep|cascade]
[recordsize] overrides the buffering scheme (by default line-
based). 0 means anything read is transmitted immediately; a pos-
itive value specifies a record size.
[kill|event] specifies the write error action, overriding the
default given with the "default write" command.
[keep|cascade] overrides the cascade behavior, given with the
"default source keep|cascade" commands.
If -o 1 (STDOUT) or -o 2 (STDERR) are missing, they're added
implicitly. If STDERR is implicit, an implicit pipe is also
generated, going to Plumb2's STDERR. You can inhibit this by giv-
ing -o 2 explicitly.
The part after the = is the program ARGV, as interpreted by
execvp(3). At least one token has to be given (the program
name).
pipe [as ID] [-front] SOURCE TARGET [-(filter|cut)[:hex] prefix] [-pre-
fix[:hex] prefix] [-endchain]
This command creates a new pipe, from the given source to the
given target.
Usually unique IDs are automatically generated for pipes. Fol-
lowing the "as" keyword, the automatic ID can be overridden.
The -front option inserts the pipe at the beginning of the
source's pipe sequence. Otherwise it's appended to the end.
A source definition looks like this: processid[/FD] - if the FD
is not given, /1 is used (STDOUT). Instead of the processid,
:plumb2 may be used to get plumb2's STDIN as a source, or :ev to
get events.
A target definition also looks like this: processid[/FD] - if
the FD is not given, /0 is used (STDIN). Instead of the proces-
sid, :plumb2 may be used to get plumb2's STDOUT as a target
(:plumb2/2 can be used to get plumb2's STDERR), or :ctrl to write
to the control parser.
An optional filter definition follows the target, with one of
four options: -filter -filter:hex -cut -cut:hex, followed by the
filter prefix. If such an option is given, only lines/records
with the given prefix are let through the pipe, the rest is
ignored. -cut will remove the prefix before transmitting the
data, -filter will leave it. Using :hex in the option indicates
that the prefix is given in hexadecimal (two characters per
byte), so any binary data can be specified (useful for record-
based binary transmission).
An optional prefix definition follows, with one of two options:
-prefix or -prefix:hex, followed by the prefix string. The
string will be prepended to each transmitted line/record.
The -endchain option indicates that processing the source's
pipe sequence should stop after this pipe - tipically used
together with a filter option, in which case the pipe sequence
continues if the filter doesn't match.
unpause PROCESSID
Start reading from the sources of the given process (used with
the proc -pause option).
endproc PROCESSID
Kill the given process.
endpipe PIPEID
Shut down the given pipe.
default proc kill|event
Set default process termination action. Only affects processes
started after using the command. The initial default is "kill".
default eof kill|event
Set default source EOF termination action. Only affects sources
started after using the command. The initial default is "kill".
default write kill|event
Set default target write error termination action. Only affects
targets started after using the command. The initial default is
"kill".
default source|target keep|cascade
Set default cascade flag for sources or targets. Only affects
sources and targets started after using the command. Both ini-
tial defaults are "cascade".
timer [as ID] [repeat] TIMEOUT OUTPUT
Create a timer. OUTPUT will be written to :ev after TIMEOUT sec-
onds (a floating point value). If "repeat" is given, the timer
will repeat until plumb2 shutdown, or until it's deleted with the
endtimer command.
endtimer ID
Delete the given timer. The timer will not trigger after this.
EVENTS
When reading from the :ev source, a process can get notification of
events happening in Plumb2. The following events are generated:
ENDPROC processid KILL|EXIT exitvalue
A process has ended (either KILLed or EXITed normally).
ENDPIPE pipeid CASCADE|KILL|EOF|WRITE
A pipe has shut down for some reason.
EXAMPLES
To simulate cat: plumb2 "pipe :plumb2 :plumb2"
For a simple example, see the test/count10 script.
For more complicated programs, you can set "default eof event", leave
"default proc kill" for the controlling process set, then set "default
proc event" for the temporary processes started after.
Need more examples...
BUGS
Lots. EOF termination action cannot be overridden for :plumb2/0, so it's
always KILL. Need better shutdown procedure logic so outputs from all
dying processes can be gathered.
SEE ALSO
plumbnet(1)
AUTHOR
Mate Nagy (mnagy (at) port70.net)
Home Page
http://repo.hu/projects/plumb2
NAME
plumbnet - line-based TCP relay server
SYNOPSIS
plumbnet [-i interface] -p port [-n] [-c hostname] [-s]
DESCRIPTION
Plumbnet waits for TCP connections on a specified port. It assigns each
new connection an unique ID, and prints this ID in the new connection
event to STDOUT. It will prints each subsequent line received from this
connection to STDOUT, prefixed with the unique ID.
Plumbnet also reads lines from its STDIN. It looks for a connection ID
prefix; the rest of the line will then be sent to the referenced con-
nection.
In short: a single controlling program can manage a multitude of TCP
connections by communicating with only plumbnet (possibly through
plumb2).
COMMAND LINE
-i interface
The interface address to bind to (by default 127.0.0.1).
-p port
This mandatory argument specifies the TCP port to listen on.
-n Enable TCP_NODELAY.
-c hostname
Netcat mode: instead of usual plumbnet operation, connect to the
hostname (to the port given with -p). Data read from STDIN is
sent to the host; data read from the host is written to STDOUT.
-s In netcat mode, shut down only after EOF both on STDIN and the
socket. If -s is not given, plumbnet will shut down after get-
ting EOF from the socket.
-s In plumbnet mode, report the EOF event instead of the END event
on EOF from a socket. The user is responsible to shut down the
socket explicitly using the "end" (or "kill") command.
OUTPUT
Plumbnet reports events on its STDOUT. The following events are possi-
ble:
NEW id addr
A new connection is opened, with the given ID, from the given TCP
address.
END id
A connection is closed.
EOF id
A connection received EOF. (Only reported when the -s flag is speci-
fied.)
IDs are integers, unique for each new connection.
Furthermore, all lines received from connections will be printed to
plumbnet's STDOUT, prefixed by their ID, followed by a single space.
INPUT
Plumbnet reads line by line from its STDIN. Each line has to be com-
posed of two parts, separated by a single space.
In the common case, the first part before the first space is the con-
nection ID; the rest of the line will be sent to the given connection.
Instead of the connection ID, the controlling program may give commands
(the parameter of the command follows the first space):
end ID
Close the connection referenced by the ID, but wait for the write buf-
fer to empty first.
kill ID
Close the connection referenced by the ID (the contents of the write
buffer are lost).
It's also possible to use an asterisk (*) instead of the connection ID.
In this case, the rest of the line will be sent to all open connec-
tions.
SEE ALSO
plumb2(1)
AUTHOR
Mate Nagy (mnagy (at) port70.net)
Home Page
http://repo.hu/projects/plumb