pcb-rnd knowledge pool
The original lihata announcement
| lht1 by Tibor 'Igor2' Palinkas on 2016-11-02 | Tags: announcement, lihata, format, native |
Abstract: Historical moment: first announcement of pcb-rnd's native format (lihata board v1).
Imported from the mailing list archives.
Since this old announcement, lihata collected a considerable amount of mileage and is the stable native format of pcb-rnd.
we've reached the biggest turning point in the history of the project. Please take your time to read this wall-of-text, and please take actions.
We are at the very end of a development cycle, with a release expected in a few hours. One of the focuses for this cycle was to get the lihata board format (our new file format for storing pcb designs) production quality. This has been achieved. Now we need YOUR help.
1. Current state: load & save
Our default file format is still .pcb, so "save" uses that and "save as" offers that as default. On load, pcb-rnd tries all available io plugins automatically, so the user doesn't need to specify the format. Depending on the bugs reported, this will potentially change within 1..2 months and lihata will be the primary/default file format.
The load auto-format mechanism also means running from the command line, pcb-rnd foo.lht or pcb-rnd foo.kicad_pcb will do the right thing. You may get errors from some of the parsers that are not for the given file - that's a known bug, please ignore them (will be fixed later).
I made extensive tests on io_lihata and I am fairly sure it doesn't mess up boards (see point 3). I believe it is safe to use lihata boards in production.
We have support for full boards now, lihata footprints and save/load buffer in lihata will happen later (these are calling parts of the full-board save code - nothing hard, but it's better to fully test out the full board support first).
2. What you should do
You should consider gradually switching your designs from .pcb files to .lht files. This can be done manually by using "save as" menu or automatically, using actions (please contact me if you need a script to mass-convert pcb files).
When using the lihata format, please look at the diffs after save (this is easy if you are using version control). Please report any diff that you think is unreasonable (see point 4).
3. Why do I think it won't ruin your files
- I have a round-trip test script (in svn: util/devhelpers/pcb-rtrip; it loads a .pcb, saves it in .lht, loads it from .lth and saves it in another .pcb. The first and last .pcb files need to match byte-to-byte, except for order of Attribute lines.
- I wrote 35 special test cases; each demonstrates a single feature of pcb-rnd, preferrably with some corner cases presented. They all pass with the round-trip script.
- I collected 244 random .pcb boards from my disk, a lot of designs accumulated in the past decade. All pass the round-trip test.
- There are about 6 of the 244 files that yield a minimal difference, but all those cases are bugs in the input file; the .pcb load/save code doesn't do much checking and keeps the bug as is, io_lihata fixes the bug (e.g. pin thermal connection to a non-existing layer; 2 layer groups claiming to be the top layer)
- If you still have doubts: I've implemented a configurable feature that first saves the design in the .pcb format and then proceeds saving it in lihata. This means if your board.lht is somehow ruined, you have a backup in the format you would have used originally. This feature is enabled by default for now. The .pcb file name comes from a configurable template so it's possible to collect these files in a separate directory, save multiple of them or keep only the last one, etc.
So at the moment, I simply don't have a .pcb file on which io_lihata doesn't work perfectly and we have a mechanism to restore from .pcb if it still fails somehow.
4. What io_lihata offers today
It has a feature that I believe is unique among all cad programs: it uses an easy-to-edit, non-strict text format, but on a load-save cycle it keeps formatting, so it does not enforce it's own canonical indentation or numeric formats or units. This is done by not simply overwriting an existing file from memory, but doing a smart merge, trying to keep byte-to-byte as much of the original file as possible:
- comments are kept everywhere in the file (both content and position)
- when copying existing, unchanged nodes: newlines, whitespaces, indentations are kept
- when copying existing, unchanged nodes: keep numeric format; this means if you or your script prefers to use "+5", "05", "5.00" instead of "5", as long as the numeric value matches, a load-save will not change the format.
- the same applies for units: if your input file for whatever reason specified a coordinate in centimeter, kilometer, decimil, or whatever unit, as long as the coord value did not change in memory (e.g. the object didn't move), the unit is not changed on save
- if new, simple nodes need to be inserted in a crowd of existing nodes, e.g. a few more points in a polygon, more attributes, etc., the code attempts to copy the local indentation and lihata style by looking at existing nodes of the same type on the same level
- when inserting new subtrees that did not exist in the original input, it uses a pcb-rnd-specific, compact form instead of the usual json/xml approach of "each node in a new line, indented". This makes the output more compact and readable; some subtrees even look similar to the ones in .pcb
- but unlike .pcb, almost all fields have explicit name, so there are no positional fields (except for the table of x;y polygon coords). This makes us able to extend the format in a backward-compatible way.
If you find diffs after a save that violates any of the above rules, please contact me.
5. what happens if you don't switch; what io_lihata will offer tomorrow
You can keep using the .pcb format indefinitely. We'll keep it to stay comaptible with mainline. It will have the same role as any other secondary IO format, e.g. the kicad formats. We'll try to do our best to keep up with mainline format changes/improvements, although it is not a high priority.
However, any new feature will be implemented only in io_lihata, and NOT in io_pcb. So in short: if you want to use the new features that will be added in the future, you can't do that without using the lihata board format.
There's already an example of this: attributes. For the "50 ohm impedance" wishlist item, I upgraded the core so that any object can have arbitrary, user defined key=value attributes. The property editor can create/edit these attributes. The advanced search can search for them. You can tag your 50 ohm objects and you can select them by tag. (You can even select any 50 ohm object that is "on the wrong layer" or is of the "wrong thickness", by the way). But if you save your board in .pcb, these attributes are simply lost, and there's nothing we can do about it. Saving in lihata should keep all attributes.
6. plans
The next development cycle will be about the big data struct cleanup I was considering for a year now. Even mainline was considering this multiple times in the past years. The ability to save new subtrees of information in lihata and the upcoming cleanup would unlock the path to at least the following features:
- pad stacks
- arbitrary polygon shaped pin/via rings and smd pads
- elements (footprints) containing any object (yes, this means polygons on copper or silk, text on copper, arc on copper, etc.)
- programmable drc - we already had most of the code for the query/advanced search, we only needed to save more of the "query expressions" as a "scripts" (done in 2.2.1)
- optionally multiple fonts on the board
- better 3d modeller, fem, etc. export facilities: with lihata we have more room to save extra, plugin-specific data in footprints; this may include references to external model files or even an embedded model file so that your board.lht can stay self-containing
Please note that about 2/3 of the work went into the external lihata project. This is good news for those who are waiting for cschem to happen: the current pcb-rnd lihata board can be considered as a pilot project for cschem's native (lihata) format. By the time we can start cschem we have the save/load code ready.
7. known bugs
When saving new subtrees (or saving the board initially), the indentation is not 100% perfect everywhere. This should never interfere with the validity of the files, it's purely an aestethical aesthetical issue; I am still working on this, but it won't block the release.
8. tools? how do I script with lihata?
The lihata format and the lihata lib is a small project of mine.
It is pretty well documented.
It has a parser written in C, that can be very easily bound to your favorite scripting language (I once made the binding to php in a few hours - without knowing anything about php).
The main repository offers a few extra tools. Most notably there's a tool that converts the hierarchical lihata tree into a flat text file: each line is a node with all the metadata included. This makes it very easy to process lihata files with line-based UNIX tools (e.g. shell, sed, grep, awk, perl, etc.). Actually it's easier to write a safe script for the flat version of lihata than for the original .pcb format.
I do not yet have converters between lihata and json or xml. There's no 1:1 mapping (like there's no 1:1 mapping between json and xml either). I am open to contribution on this.
I don't yet have editor syntax highlight files (but JSON's is usually a good approximation). I'm open to contribution.
9. ok, so what now?
In case you try lihata, please report back - both if it worked or if it failed. We need extensive testing of this, by multiple users, not just me.
In the same time we should probably start to collect boards successfully designed with pcb-rnd, in an online gallery. If you have such boards to share, please contact me.