Mailing list archives : pcb-rnd

ID:106
From:ge...@igor2.repo.hu
Date:Wed, 2 Nov 2016 11:57:12 +0100 (CET)
Subject:[pcb-rnd] lihata board format: please switch
Hi fellow pcb-rnd users,
 
we've reached the biggest turning point in the history of the project. 
Please take your time to read this wall-of-text, and please take actions.
 
We are at the very end of a development cycle, with a release expected in 
a few hours. One of the focuses for this cycle was to get the lihata board 
format (our new file format for storing pcb designs) production quality. 
This has been achieved. Now we need YOUR help.
 
1. Current state: load & save
 
Our default file format is still .pcb, so "save" uses that and "save as" 
offers that as default. On load, pcb-rnd tries all available io plugins 
automatically, so the user doesn't need to specify the format. Depending 
on the bugs reported, this will potentially change within 1..2 months and 
lihata will be the primary/default file format.
 
The load auto-format mechanism also means running from the command line, 
pcb-rnd foo.lht or pcb-rnd foo.kicad_pcb will do the right thing. You may 
get errors from some of the parsers that are not for the given file - 
that's a known bug, please ignore them (will be fixed later).
 
I made extensive tests on io_lihata and I am fairly sure it doesn't mess 
up boards (see point 3). I believe it is safe to use lihata boards in 
production.
 
We have support for full boards now, lihata footprints and save/load 
buffer in lihata will happen later (these are calling parts of the 
full-board save code - nothing hard, but it's better to fully test out the 
full board support first).
 
2. What you should do
 
You should consider gradually switching your designs from .pcb files to 
.lht files. This can be done manually by using "save as" menu or 
automatically, using actions (please contact me if you need a script to 
mass-convert pcb files).
 
When using the lihata format, please look at the diffs after save (this 
is easy if you are using version control). Please report any diff that you 
think is unreasonable (see point 4).
 
3. Why do I think it won't ruin your files
 
- I have a round-trip test script (in svn: util/devhelpers/pcb-rtrip; it 
loads a .pcb, saves it in .lht, loads it from .lth and saves it in another 
.pcb. The first and last .pcb files need to match byte-to-byte, except for 
order of Attribute lines.
 
- I wrote 35 special test cases; each demonstrates a single feature of 
pcb-rnd, preferrably with some corner cases presented. They all pass with 
the round-trip script.
 
- I collected 244 random .pcb boards from my disk, a lot of designs 
accumulated in the past decade. All pass the round-trip test.
 
- There are about 6 of the 244 files that yield a minimal difference, but 
all those cases are bugs in the input file; the .pcb load/save code 
doesn't do much checking and keeps the bug as is, io_lihata fixes the bug 
(e.g. pin thermal connection to a non-existing layer; 2 layer groups 
claiming to be the top layer)
 
- If you still have doubts: I've implemented a configurable feature that 
first saves the design in the .pcb format and then proceeds saving it in 
lihata. This means if your board.lht is somehow ruined, you have a backup 
in the format you would have used originally. This feature is enabled by 
default for now. The .pcb file name comes from a configurable template so 
it's possible to collect these files in a separate directory, save 
multiple of them or keep only the last one, etc.
 
So at the moment, I simply don't have a .pcb file on which io_lihata 
doesn't work perfectly and we have a mechanism to restore from .pcb if it 
still fails somehow.
 
4. What io_lihata offers today
 
It has a feature that I believe is unique among all cad programs: it uses 
an easy-to-edit, non-strict text format, but on a load-save cycle it keeps 
formatting, so it does not enforce it's own canonical indentation or 
numeric formats or units. This is done by not simply overwriting an 
existing file from memory, but doing a smart merge, trying to keep 
byte-to-byte as much of the original file as possible:
 
- comments are kept everywhere in the file (both content and position)
 
- when copying existing, unchanged nodes: newlines, whitespaces, 
indentations are kept
 
- when copying existing, unchanged nodes: keep numeric format; this means 
if you or your script prefers to use "+5", "05", "5.00" instead of "5", as 
long as the numeric value matches, a load-save will not change the format.
 
- the same applies for units: if your input file for whatever reason 
specified a coordinate in centimeter, kilometer, decimil, or whatever 
unit, as long as the coord value did not change in memory (e.g. the object 
didn't move), the unit is not changed on save
 
- if new, simple nodes need to be inserted in a crowd of existing nodes, 
e.g. a few more points in a polygon, more attributes, etc., the code 
attempts to copy the local indentation and lihata style by looking at 
existing nodes of the same type on the same level
 
- when inserting new subtrees that did not exist in the original input, it 
uses a pcb-rnd-specific, compact form instead of the usual json/xml 
approach of "each node in a new line, indented". This makes the output 
more compact and readable; some subtrees even look similar to the ones in 
.pcb
 
- but unlike .pcb, almost all fields have explicit name, so there are no 
positional fields (except for the table of x;y polygon coords). This makes 
us able to extend the format in a backward-compatible way.
 
 
If you find diffs after a save that violates any of the above rules, 
please contact me.
 
5. what happens if you don't switch; what io_lihata will offer tomorrow
 
You can keep using the .pcb format indefinitely. We'll keep it to stay 
comaptible with mainline. It will have the same role as any other 
secondary IO format, e.g. the kicad formats. We'll try to do our best to 
keep up with mainline format changes/improvements, although it is not a 
high priority.
 
However, any new feature will be implemented only in io_lihata, and NOT in 
io_pcb. So in short: if you want to use the new features that will be 
added in the future, you can't do that without using the lihata board 
format.
 
There's already an example of this: attributes. For the "50 ohm impedance" 
wishlist item, I upgraded the core so that any object can have 
arbitrary, user defined key=value attributes. The property editor can 
create/edit these attributes. The advanced search can search for 
them. You can tag your 50 ohm objects and you can select them by tag. (You 
can even select any 50 ohm object that is "on the wrong layer" or is of 
the "wrong thickness", by the way). But if you save your board in .pcb, 
these attributes are simply lost, and there's nothing we can do about it. 
Saving in lihata should keep all attributes.
 
6. plans
 
The next development cycle will be about the big data struct cleanup I was 
considering for a year now. Even mainline was considering this multiple 
times in the past years. The ability to save new subtrees of information 
in lihata and the upcoming cleanup would unlock the path to at least the 
following features:
 
  - pad stacks
 
  - arbitrary polygon shaped pin/via rings and smd pads
 
  - elements (footprints) containing any object (yes, this means polygons 
on copper or silk, text on copper, arc on copper, etc.)
 
  - programmable drc - we already have most of the code for the 
query/advanced search, we only need to save more of the "query 
expressions" as a "script"
 
  - optionally multiple fonts on the board
 
  - better 3d modeller, fem, etc. export facilities: with lihata we have 
more room to save extra, plugin-specific data in footprints; this may 
include references to external model files or even an embedded model file 
so that your board.lht can stay self-containing
 
Please note that about 2/3 of the work went into the external lihata 
project. This is good news for those who are waiting for cschem to happen: 
the current pcb-rnd lihata board can be considered as a pilot project for 
cschem's native (lihata) format. By the time we can start cschem we have 
the save/load code ready.
 
7. known bugs
 
When saving new subtrees (or saving the board initially), the indentation 
is not 100% perfect everywhere. This should never interfere with the 
validity of the files, it's purely an aestethical aesthetical issue; I am 
still working on this, but it won't block the release.
 
8. tools? how do I script with lihata?
 
The lihata format and the lihata lib is a small project of mine.
 
It is pretty well documented.
 
It has a parser written in C, that can be very easily bound to your 
favorite scripting language (I once made the binding to php in a few hours 
- without knowing anything about php).
 
The main repository offers a few extra tools. Most notably there's a tool 
that converts the hierarchical lihata tree into a flat text file: each 
line is a node with all the metadata included. This makes it very easy to 
process lihata files with line-based UNIX tools (e.g. shell, sed, grep, 
awk, perl, etc.). Actually it's easier to write a safe script for the 
flat version of lihata than for the original .pcb format.
 
I do not yet have converters between lihata and json or xml. There's no 
1:1 mapping (like there's no 1:1 mapping between json and xml either). I 
am open to contribution on this.
 
I don't yet have editor syntax highlight files (but JSON's is usually a 
good approximation). I'm open to contribution.
 
9. ok, so what now?
 
In case you try lihata, please report back - both if it worked or if it 
failed. We need extensive testing of this, by multiple users, not just me.
 
In the same time we should probably start to collect boards successfully 
designed with pcb-rnd, in an online gallery. If you have such boards to 
share, please contact me.
 
TIA,
 
Igor2
 

Reply subtree:
106 [pcb-rnd] lihata board format: please switch from ge...@igor2.repo.hu