Plumb2 is a program for starting and managing multiple processes and various pipes between them. Plumb2 itself can be controlled from programs running inside it, through pipes; so a program running under Plumb2 can add or kill processes or pipes during runtime.
For futher documentation, see the man page.
Plumb2 comes with an additional tool called plumbnet, for managing multiple TCP connections from a single program without having to do networking, threads, or select(2). See man page for more information.
Plumb2 and plumbnet are written in C99 (they currently depend on GCC to compile). The only other external dependency is libevent.
You can get Plumb2 from Subversion:
svn co svn://repo.hu/plumb/tags/2.0.0 plumb2
Debian packages are available from the repo.hu Debian repository.
Plumb2 may be useful together with other programs that communicate over their STDIN/STDOUT:
Copyright (C) 2009-2010 Máté Nagy <mnagy AT port70.net>
Plumb2 is licensed under the GNU General Public License Version 3 (available in the file COPYING) or any later version.
IRC: IRCNet #awk
NAME plumb2 - connect multiple programs with lots of pipes SYNOPSIS plumb2 [-v key value]+ [-V key values.. .]+ [-f filename] [-d] "com- mands" 69 "command1" "command2" DESCRIPTION Plumb2 is a program for starting and managing multiple processes and various pipes between them. Plumb2 itself can be controlled from pro- grams running inside it, through pipes; so a program running under Plumb2 can add or kill processes or pipes during runtime. Multiple pipes may go from a single source to various targets, or from various sources to a single target. Line buffering can be used to avoid mixing up data; adding and filtering by line prefixes serve to distin- guish sources and targets. In the end a single program can communicate with and control a multi- tude of other programs through just reading from STDIN and writing to its STDOUT. COMMAND LINE The Plumb2 command line is a sequence of four different kinds of parame- ters, in any order. Some of the parameters serve to set the values of variables. These variables are used to pass in data (such as file- names) to the Plumb2 control language. -69 command1 command2 The two commands are started behind a shell (like with sys- tem(3)) and connected together with two pipes (STDOUT to STDIN). Invoking Plumb2 as "69 command1 command2" is equivalent to "plumb2 -69 command1 command2". -v key value Assigns a single value to a variable. -V key value1 value2... . Assigns a list of values to a variable (terminate list with a single period). -f filename Read Plumb2 control commands from the specified file. -d Increases the (debug) log verbosity. May be specified multiple times. Log messages are written to Plumb2's STDERR. The initial level is ERROR; further levels are WARNING, DEBUG, JUNK. "commands" Execute the given commands as Plumb2 control commands. THE MODEL Processes provide sources (such as their STDOUT and STDERR) and targets (their STDIN). There are special sources and targets not associated with processes (for example, Plumb2's own STDOUT is a target). Pipes go between sources and targets. Each source has an ordered list of pipes going from it. When data arrives at a source, the data will be written into the pipes, in sequence. In certain conditions, a pipe can decide to "break the chain" and stop data from getting to subsequent pipes. There is a special target (called :ctrl ) from which Plumb2 parses con- trol instructions, and a source (called :ev ) to which Plumb2 writes events (e.g. when a process dies). Not connecting them to anywhere is not an error. Process sources have a file descriptor and a buffering scheme. By default, processes are started with two sources: STDOUT for FD 1, STDERR for FD 2, both line-buffered. Fixed record buffering (always waiting until N bytes are accumulated) and no buffering (sending any- thing received right away) are the possible alternatives. Processes can be started with more pre-opened sources on user-specified FDs. Process targets have a file descriptor. By default, processes are started with one target: STDIN for FD 0. Processes can be started with more pre-opened targets on user-specified FDs. Processes have an associated termination action: either "event" or "kill". When a process terminates, either of two things can happen. If "kill" is set, then Plumb2 exits completely (killing all started pro- cesses). If "event" is set, only an event is generated, and everything continues. The global default is "kill", adjustable per process or by the "default proc" command. Each source has an associated termination action: also either "event" or "kill". This action is taken when the source gets an EOF (or an error). The global default is "kill", adjustable per source or by the "default eof" command. Each target has an associated termination action: also either "event" or "kill". This action is taken when the target gets a write error (not when otherwise closed, by e.g. the termination of the associated process). The global default is "kill", adjustable per target or by the "default write" command. Each source and target has an associated "cascade" flag. When this is true, the source or target gets closed when all pipes going from/to it are closed. The global default is true, adjustable per source/target or by the "default source|target keep|cascade" commands. CONTROL LANGUAGE The Plumb2 control language is line-based. Each line is fed to the token parser, which splits it into a sequence of strings called tokens; this usually just means splitting the line into whitespace-separated words. The generated lists of tokens are then interpreted as Plumb2 commands. TOKEN PARSER Lines are split into tokens along the whitespace between them. A hash mark (#) begins a comment, the rest of the line will be ignored. Multiple Plumb2 commands can be given in a single line, separated by semicolons (;). No ending semicolon is necessary. Any string of non-whitespace, non-special characters will be considered a token in itself. Tokens may be string literals, enclosed in quotation marks ("). All characters until the closing quotation marks are considered part of the same token, including spaces. Some special characters have to be escaped using a backslash (\). Supported escape notation: \\ for a sin- gle backslash, \n for a newline, \t for a tab, \b for a backspace, \xAB for an arbitrary character given in hex. Words beginning with the dollar sign ($) are considered variable names. The variable name is the part after the $ (until the next whitespace or special character). In place of the word, the tokens set as the value of the variable will be substituted. Undefined variables will result in no inserted tokens (and no error). A variable may have an arbitrary number of tokens associated with it (including zero). PLUMB COMMANDS Some commands have IDs as parameters. IDs are ordinary tokens, but the usable character set is restricted: [a-zA-Z_][a-zA-Z0-9_]* set ID [values...] Set the variable named by the ID to the given list of (zero or more) tokens. proc ID [kill|event] [-pause] [inputfd]* [outputfd]* = program [argu- ments...] This command creates a new process. The ID identifies the process. [kill|event] specifies the action to take when the process ter- minates, overriding the default given with the "default proc" command. -pause disables reading from all sources (until an "unpause" command is issued for the process). This may be useful to stop data being lost if a process writes to a source before the appropriate pipes going from the source are defined. It's guar- anteed that no reading will take place during the interpretation of commands given on the plumb2 command line, or between execut- ing multiple commands given on a single line (using semicolon separators). Zero or more target (input FD) definitions may be given, in the following form: -i FD [kill|event] [keep|cascade] [kill|event] specifies the EOF action, overriding the default given with the "default eof" command. [keep|cascade] overrides the cascade behavior, given with the "default target keep|cascade" commands. If -i 0 (STDIN) is not specified, it is added implicitly. Zero or more source (output FD) definitions may be given, in the following form: -o FD [recordsize] [kill|event] [keep|cascade] [recordsize] overrides the buffering scheme (by default line- based). 0 means anything read is transmitted immediately; a pos- itive value specifies a record size. [kill|event] specifies the write error action, overriding the default given with the "default write" command. [keep|cascade] overrides the cascade behavior, given with the "default source keep|cascade" commands. If -o 1 (STDOUT) or -o 2 (STDERR) are missing, they're added implicitly. If STDERR is implicit, an implicit pipe is also generated, going to Plumb2's STDERR. You can inhibit this by giv- ing -o 2 explicitly. The part after the = is the program ARGV, as interpreted by execvp(3). At least one token has to be given (the program name). pipe [as ID] [-front] SOURCE TARGET [-(filter|cut)[:hex] prefix] [-pre- fix[:hex] prefix] [-endchain] This command creates a new pipe, from the given source to the given target. Usually unique IDs are automatically generated for pipes. Fol- lowing the "as" keyword, the automatic ID can be overridden. The -front option inserts the pipe at the beginning of the source's pipe sequence. Otherwise it's appended to the end. A source definition looks like this: processid[/FD] - if the FD is not given, /1 is used (STDOUT). Instead of the processid, :plumb2 may be used to get plumb2's STDIN as a source, or :ev to get events. A target definition also looks like this: processid[/FD] - if the FD is not given, /0 is used (STDIN). Instead of the proces- sid, :plumb2 may be used to get plumb2's STDOUT as a target (:plumb2/2 can be used to get plumb2's STDERR), or :ctrl to write to the control parser. An optional filter definition follows the target, with one of four options: -filter -filter:hex -cut -cut:hex, followed by the filter prefix. If such an option is given, only lines/records with the given prefix are let through the pipe, the rest is ignored. -cut will remove the prefix before transmitting the data, -filter will leave it. Using :hex in the option indicates that the prefix is given in hexadecimal (two characters per byte), so any binary data can be specified (useful for record- based binary transmission). An optional prefix definition follows, with one of two options: -prefix or -prefix:hex, followed by the prefix string. The string will be prepended to each transmitted line/record. The -endchain option indicates that processing the source's pipe sequence should stop after this pipe - tipically used together with a filter option, in which case the pipe sequence continues if the filter doesn't match. unpause PROCESSID Start reading from the sources of the given process (used with the proc -pause option). endproc PROCESSID Kill the given process. endpipe PIPEID Shut down the given pipe. default proc kill|event Set default process termination action. Only affects processes started after using the command. The initial default is "kill". default eof kill|event Set default source EOF termination action. Only affects sources started after using the command. The initial default is "kill". default write kill|event Set default target write error termination action. Only affects targets started after using the command. The initial default is "kill". default source|target keep|cascade Set default cascade flag for sources or targets. Only affects sources and targets started after using the command. Both ini- tial defaults are "cascade". timer [as ID] [repeat] TIMEOUT OUTPUT Create a timer. OUTPUT will be written to :ev after TIMEOUT sec- onds (a floating point value). If "repeat" is given, the timer will repeat until plumb2 shutdown, or until it's deleted with the endtimer command. endtimer ID Delete the given timer. The timer will not trigger after this. EVENTS When reading from the :ev source, a process can get notification of events happening in Plumb2. The following events are generated: ENDPROC processid KILL|EXIT exitvalue A process has ended (either KILLed or EXITed normally). ENDPIPE pipeid CASCADE|KILL|EOF|WRITE A pipe has shut down for some reason. EXAMPLES To simulate cat: plumb2 "pipe :plumb2 :plumb2" For a simple example, see the test/count10 script. For more complicated programs, you can set "default eof event", leave "default proc kill" for the controlling process set, then set "default proc event" for the temporary processes started after. Need more examples... BUGS Lots. EOF termination action cannot be overridden for :plumb2/0, so it's always KILL. Need better shutdown procedure logic so outputs from all dying processes can be gathered. SEE ALSO plumbnet(1) AUTHOR Mate Nagy (mnagy (at) port70.net) Home Page http://repo.hu/projects/plumb2
NAME plumbnet - line-based TCP relay server SYNOPSIS plumbnet [-i interface] -p port [-n] [-c hostname] [-s] DESCRIPTION Plumbnet waits for TCP connections on a specified port. It assigns each new connection an unique ID, and prints this ID in the new connection event to STDOUT. It will prints each subsequent line received from this connection to STDOUT, prefixed with the unique ID. Plumbnet also reads lines from its STDIN. It looks for a connection ID prefix; the rest of the line will then be sent to the referenced con- nection. In short: a single controlling program can manage a multitude of TCP connections by communicating with only plumbnet (possibly through plumb2). COMMAND LINE -i interface The interface address to bind to (by default 127.0.0.1). -p port This mandatory argument specifies the TCP port to listen on. -n Enable TCP_NODELAY. -c hostname Netcat mode: instead of usual plumbnet operation, connect to the hostname (to the port given with -p). Data read from STDIN is sent to the host; data read from the host is written to STDOUT. -s In netcat mode, shut down only after EOF both on STDIN and the socket. If -s is not given, plumbnet will shut down after get- ting EOF from the socket. -s In plumbnet mode, report the EOF event instead of the END event on EOF from a socket. The user is responsible to shut down the socket explicitly using the "end" (or "kill") command. OUTPUT Plumbnet reports events on its STDOUT. The following events are possi- ble: NEW id addr A new connection is opened, with the given ID, from the given TCP address. END id A connection is closed. EOF id A connection received EOF. (Only reported when the -s flag is speci- fied.) IDs are integers, unique for each new connection. Furthermore, all lines received from connections will be printed to plumbnet's STDOUT, prefixed by their ID, followed by a single space. INPUT Plumbnet reads line by line from its STDIN. Each line has to be com- posed of two parts, separated by a single space. In the common case, the first part before the first space is the con- nection ID; the rest of the line will be sent to the given connection. Instead of the connection ID, the controlling program may give commands (the parameter of the command follows the first space): end ID Close the connection referenced by the ID, but wait for the write buf- fer to empty first. kill ID Close the connection referenced by the ID (the contents of the write buffer are lost). It's also possible to use an asterisk (*) instead of the connection ID. In this case, the rest of the line will be sent to all open connec- tions. SEE ALSO plumb2(1) AUTHOR Mate Nagy (mnagy (at) port70.net) Home Page http://repo.hu/projects/plumb