Plumb - connect multiple programs with lots of pipes </head> <body> <h1>Plumb</h1> <p> Plumb is a program for starting and managing multiple processes and various pipes between them. Plumb itself can be controlled from programs running inside it, through pipes; so a program running under Plumb can add or kill processes or pipes during runtime. <p> For futher documentation, see the <a href="#plumb">man page</a>. <p> Plumb comes with an additional tool called <b>plumbnet</b>, for managing multiple TCP connections from a single program without having to do networking, threads, or select(2). See <a href="#plumbnet">man page</a> for more information. <p> Plumb and plumbnet are written in C99 (they currently depend on GCC to compile). The only other external dependency is <a href="http://monkey.org/~provos/libevent/">libevent</a>. <h2>Get Plumb</h2> <p>You can get Plumb from Subversion: <pre>svn co svn://repo.hu/plumb/trunk plumb</pre> <p>Debian packages are available from the <a href="http://repo.hu/debian/">repo.hu Debian repository</a>. <h2>Other useful programs</h2> <p>Plumb may be useful together with other programs that communicate over their STDIN/STDOUT: <ul> <li><a href="http://repo.hu/projects/animator/">Animator</a></li> </ul> <h2>Author</h2> <p>Copyright (C) 2009-2010 Máté Nagy <mnagy AT port70.net> <p>Plumb is licensed under the GNU General Public License Version 3 (available in the file COPYING) or any later version. <p>IRC: IRCNet #awk <a name="plumb"></a><h2>Man page for Plumb</h2> <pre> NAME plumb - connect multiple programs with lots of pipes SYNOPSIS plumb [-v key value]+ [-V key values.. .]+ [-f filename] [-d] "com- mands" 69 "command1" "command2" DESCRIPTION Plumb is a program for starting and managing multiple processes and various pipes between them. Plumb itself can be controlled from pro- grams running inside it, through pipes; so a program running under Plumb can add or kill processes or pipes during runtime. Multiple pipes may go from a single source to various targets, or from various sources to a single target. Line buffering can be used to avoid mixing up data; adding and filtering by line prefixes serve to distin- guish sources and targets. In the end a single program can communicate with and control a multi- tude of other programs through just reading from STDIN and writing to its STDOUT. COMMAND LINE The Plumb command line is a sequence of four different kinds of parame- ters, in any order. Some of the parameters serve to set the values of variables. These variables are used to pass in data (such as file- names) to the Plumb control language. -69 command1 command2 The two commands are started behind a shell (like with sys- tem(3)) and connected together with two pipes (STDOUT to STDIN). Invoking Plumb as "69 command1 command2" is equivalent to "plumb -69 command1 command2". -v key value Assigns a single value to a variable. -V key value1 value2... . Assigns a list of values to a variable (terminate list with a single period). -f filename Read Plumb control commands from the specified file. -d Increases the (debug) log verbosity. May be specified multiple times. Log messages are written to Plumb's STDERR. The initial level is ERROR; further levels are WARNING, DEBUG, JUNK. "commands" Execute the given commands as Plumb control commands. THE MODEL Processes provide sources (such as their STDOUT and STDERR) and targets (their STDIN). There are special sources and targets not associated with processes (for example, Plumb's own STDOUT is a target). Pipes go between sources and targets. Each source has an ordered list of pipes going from it. When data arrives at a source, the data will be written into the pipes, in sequence. In certain conditions, a pipe can decide to "break the chain" and stop data from getting to subsequent pipes. There is a special target (called :ctrl ) from which Plumb parses con- trol instructions, and a source (called :ev ) to which Plumb writes events (e.g. when a process dies). Not connecting them to anywhere is not an error. Process sources have a file descriptor and a buffering scheme. By default, processes are started with two sources: STDOUT for FD 1, STDERR for FD 2, both line-buffered. Fixed record buffering (always waiting until N bytes are accumulated) and no buffering (sending any- thing received right away) are the possible alternatives. Processes can be started with more pre-opened sources on user-specified FDs. Process targets have a file descriptor. By default, processes are started with one target: STDIN for FD 0. Processes can be started with more pre-opened targets on user-specified FDs. Processes have an associated termination action: either "event" or "kill". When a process terminates, either of two things can happen. If "kill" is set, then Plumb exits completely (killing all started pro- cesses). If "event" is set, only an event is generated, and everything continues. The global default is "kill", adjustable per process or by the "default proc" command. Each source has an associated termination action: also either "event" or "kill". This action is taken when the source gets an EOF (or an error). The global default is "kill", adjustable per source or by the "default eof" command. Each target has an associated termination action: also either "event" or "kill". This action is taken when the target gets a write error (not when otherwise closed, by e.g. the termination of the associated process). The global default is "kill", adjustable per target or by the "default write" command. Each source and target has an associated "cascade" flag. When this is true, the source or target gets closed when all pipes going from/to it are closed. The global default is true, adjustable per source/target or by the "default source|target keep|cascade" commands. CONTROL LANGUAGE The Plumb control language is line-based. Each line is fed to the token parser, which splits it into a sequence of strings called tokens; this usually just means splitting the line into whitespace-separated words. The generated lists of tokens are then interpreted as Plumb commands. TOKEN PARSER Lines are split into tokens along the whitespace between them. A hash mark (#) begins a comment, the rest of the line will be ignored. Multiple Plumb commands can be given in a single line, separated by semicolons (;). No ending semicolon is necessary. Any string of non-whitespace, non-special characters will be considered a token in itself. Tokens may be string literals, enclosed in quotation marks ("). All characters until the closing quotation marks are considered part of the same token, including spaces. Some special characters have to be escaped using a backslash (\). Supported escape notation: \\ for a sin- gle backslash, \n for a newline, \t for a tab, \b for a backspace, \xAB for an arbitrary character given in hex. Words beginning with the dollar sign ($) are considered variable names. The variable name is the part after the $ (until the next whitespace or special character). In place of the word, the tokens set as the value of the variable will be substituted. Undefined variables will result in no inserted tokens (and no error). A variable may have an arbitrary number of tokens associated with it (including zero). PLUMB COMMANDS Some commands have IDs as parameters. IDs are ordinary tokens, but the usable character set is restricted: [a-zA-Z_][a-zA-Z0-9_]* set ID [values...] Set the variable named by the ID to the given list of (zero or more) tokens. proc ID [kill|event] [-pause] [inputfd]* [outputfd]* = program [argu- ments...] This command creates a new process. The ID identifies the process. [kill|event] specifies the action to take when the process ter- minates, overriding the default given with the "default proc" command. -pause disables reading from all sources (until an "unpause" command is issued for the process). This may be useful to stop data being lost if a process writes to a source before the appropriate pipes going from the source are defined. It's guar- anteed that no reading will take place during the interpretation of commands given on the plumb command line, or between execut- ing multiple commands given on a single line (using semicolon separators). Zero or more target (input FD) definitions may be given, in the following form: -i FD [kill|event] [keep|cascade] [kill|event] specifies the EOF action, overriding the default given with the "default eof" command. [keep|cascade] overrides the cascade behavior, given with the "default target keep|cascade" commands. If -i 0 (STDIN) is not specified, it is added implicitly. Zero or more source (output FD) definitions may be given, in the following form: -o FD [recordsize] [kill|event] [keep|cascade] [recordsize] overrides the buffering scheme (by default line- based). 0 means anything read is transmitted immediately; a pos- itive value specifies a record size. [kill|event] specifies the write error action, overriding the default given with the "default write" command. [keep|cascade] overrides the cascade behavior, given with the "default source keep|cascade" commands. If -o 1 (STDOUT) or -o 2 (STDERR) are missing, they're added implicitly. If STDERR is implicit, an implicit pipe is also generated, going to Plumb's STDERR. You can inhibit this by giv- ing -o 2 explicitly. The part after the = is the program ARGV, as interpreted by execvp(3). At least one token has to be given (the program name). pipe [as ID] [-front] SOURCE TARGET [-(filter|cut)[:hex] prefix] [-pre- fix[:hex] prefix] [-endchain] This command creates a new pipe, from the given source to the given target. Usually unique IDs are automatically generated for pipes. Fol- lowing the "as" keyword, the automatic ID can be overridden. The -front option inserts the pipe at the beginning of the source's pipe sequence. Otherwise it's appended to the end. A source definition looks like this: processid[/FD] - if the FD is not given, /1 is used (STDOUT). Instead of the processid, :plumb may be used to get plumb's STDIN as a source, or :ev to get events. A target definition also looks like this: processid[/FD] - if the FD is not given, /0 is used (STDIN). Instead of the proces- sid, :plumb may be used to get plumb's STDOUT as a target (:plumb/2 can be used to get plumb's STDERR), or :ctrl to write to the control parser. An optional filter definition follows the target, with one of four options: -filter -filter:hex -cut -cut:hex, followed by the filter prefix. If such an option is given, only lines/records with the given prefix are let through the pipe, the rest is ignored. -cut will remove the prefix before transmitting the data, -filter will leave it. Using :hex in the option indicates that the prefix is given in hexadecimal (two characters per byte), so any binary data can be specified (useful for record- based binary transmission). An optional prefix definition follows, with one of two options: -prefix or -prefix:hex, followed by the prefix string. The string will be prepended to each transmitted line/record. The -endchain option indicates that processing the source's pipe sequence should stop after this pipe - tipically used together with a filter option, in which case the pipe sequence continues if the filter doesn't match. unpause PROCESSID Start reading from the sources of the given process (used with the proc -pause option). endproc PROCESSID Kill the given process. endpipe PIPEID Shut down the given pipe. default proc kill|event Set default process termination action. Only affects processes started after using the command. The initial default is "kill". default eof kill|event Set default source EOF termination action. Only affects sources started after using the command. The initial default is "kill". default write kill|event Set default target write error termination action. Only affects targets started after using the command. The initial default is "kill". default source|target keep|cascade Set default cascade flag for sources or targets. Only affects sources and targets started after using the command. Both ini- tial defaults are "cascade". timer [as ID] [repeat] TIMEOUT OUTPUT Create a timer. OUTPUT will be written to :ev after TIMEOUT sec- onds (a floating point value). If "repeat" is given, the timer will repeat until plumb shutdown, or until it's deleted with the endtimer command. endtimer ID Delete the given timer. The timer will not trigger after this. EVENTS When reading from the :ev source, a process can get notification of events happening in Plumb. The following events are generated: ENDPROC processid KILL|EXIT exitvalue A process has ended (either KILLed or EXITed normally). ENDPIPE pipeid CASCADE|KILL|EOF|WRITE A pipe has shut down for some reason. EXAMPLES To simulate cat: plumb "pipe :plumb :plumb" For a simple example, see the test/count10 script. For more complicated programs, you can set "default eof event", leave "default proc kill" for the controlling process set, then set "default proc event" for the temporary processes started after. Need more examples... BUGS Lots. EOF termination action cannot be overridden for :plumb/0, so it's always KILL. Need better shutdown procedure logic so outputs from all dying processes can be gathered. SEE ALSO plumbnet(1) AUTHOR Mate Nagy (mnagy (at) port70.net) Home Page http://repo.hu/projects/plumb </pre> <a name="plumbnet"></a><h2>Man page for Plumbnet</h2> <pre> NAME plumbnet - line-based TCP relay server SYNOPSIS plumbnet [-i interface] -p port [-n] [-c hostname] [-s] DESCRIPTION Plumbnet waits for TCP connections on a specified port. It assigns each new connection an unique ID, and prints this ID in the new connection event to STDOUT. It will prints each subsequent line received from this connection to STDOUT, prefixed with the unique ID. Plumbnet also reads lines from its STDIN. It looks for a connection ID prefix; the rest of the line will then be sent to the referenced con- nection. In short: a single controlling program can manage a multitude of TCP connections by communicating with only plumbnet (possibly through plumb). COMMAND LINE -i interface The interface address to bind to (by default 127.0.0.1). -p port This mandatory argument specifies the TCP port to listen on. -n Enable TCP_NODELAY. -c hostname Netcat mode: instead of usual plumbnet operation, connect to the hostname (to the port given with -p). Data read from STDIN is sent to the host; data read from the host is written to STDOUT. -s In netcat mode, shut down only after EOF both on STDIN and the socket. If -s is not given, plumbnet will shut down after get- ting EOF from the socket. -s In plumbnet mode, report the EOF event instead of the END event on EOF from a socket. The user is responsible to shut down the socket explicitly using the "end" (or "kill") command. OUTPUT Plumbnet reports events on its STDOUT. The following events are possi- ble: NEW id addr A new connection is opened, with the given ID, from the given TCP address. END id A connection is closed. EOF id A connection received EOF. (Only reported when the -s flag is speci- fied.) IDs are integers, unique for each new connection. Furthermore, all lines received from connections will be printed to plumbnet's STDOUT, prefixed by their ID, followed by a single space. INPUT Plumbnet reads line by line from its STDIN. Each line has to be com- posed of two parts, separated by a single space. In the common case, the first part before the first space is the con- nection ID; the rest of the line will be sent to the given connection. Instead of the connection ID, the controlling program may give commands (the parameter of the command follows the first space): end ID Close the connection referenced by the ID, but wait for the write buf- fer to empty first. kill ID Close the connection referenced by the ID (the contents of the write buffer are lost). It's also possible to use an asterisk (*) instead of the connection ID. In this case, the rest of the line will be sent to all open connec- tions. SEE ALSO plumb(1) AUTHOR Mate Nagy (mnagy (at) port70.net) Home Page http://repo.hu/projects/plumb </pre> </html>