pcb-rnd knowledge pool
Data rewrite: car analogy and long term plans
data4 by Tibor 'Igor2' Palinkas on 2017-11-06
Tags: insight, data, model, rewrite, car, analogy, strategy, plan, edakrill, cschem, suite
Abstract: Car analogy for the data model replacement in 2017; what's for 2018 and some words about the long term strategy.
Car analogy: replacing the data model on the road
2017 is about a huge rewrite in pcb-rnd: we are replacing the whole data structure, bit by bit. We are not doing it by exiling ourselves to a far away island and coming back with the "finished" version after ayear or two. We are doing it bit by bit, always trying to guarantee pcb-rnd keeps working and can be used in production.
It's like replacing your car on the highway. Without stopping, or even slowing down. We have already mounted the new electric engine that gives us extra power (subc) and have replaced the transmission while the driver was not looking (layer rewrite). The new wheels are also mounted (padstack), but they are barely touching the asphalt yet. (Once we finish these, we'll remove the old engine and the old wheels, again without stop).
But there were a lot of other part replacements, a bit less nosiy, because they were on parts that don't deliver torque. Naming a few, randomly:
- we had a major redesign on the windshield (HID API fixups and gtk modularization around February and March, HID rendering API redesing this summer for composite layers)
- we are replacing the dashboard from analog needles and pins to electronics (tool rewrite; you won't see the difference first because the new TFT screens on the dashboard emulate the analog instruments perfectly)
- we have upgraded the fuel system so that we can take all sort of liquid, gas, electric and solar fuel (I/O plugins and file formats) and even old stuff, solid coal (eagle binary and protel), literally emulating a steam engine (don't worry, it's all hidden under the trunk)
Because I commit a lot and write a lot of such announcements, it may seem I did this all - but that's far from the truth. There are big parts of the above where all I did was following how others did the actual work.
long term strategy - how to fight the bugs
So we are going at 200 km/h on a highway and are replacing everything but the chassis in our car. Isn't that dangerous?
Well, it is. While we are still finding bugs that are in our code base for 10 or even 20 years, we find bugs that got introduced in 2017.
In other words, replacing the engine, and especially the replace-the-engine-on-the-go does introduce bugs. I very much appreciate the effort of those users/testers/contributors who are so brave that they are not just using svn HEAD for production, but they are deliberately using the freshest, often totally untested features, on production boards. This is what makes us able to fix these bugs.
But isn't this a downward spiral? More big changes in the data model and the infrastructure wil introduce more bugs and we won't ever get a stable version?
To understand why this won't be the case, we need to zoom out even more, not focusing on 2017, but on a period between early 2016 and late 2018.
Up to a specific point in 2016 autumn, I did not consider doing changes this big. This had many reasons, partly that I didn't realize how overrated compatibility with mainline was, so I thought any fix that includes changing how things work would frighten users away because of the consequences in compatibiliy. And partly because I didn't see enough users to feel safe that the new features would be tested out.
But then I realized the way forward is not binding us to another, mostly dead project's bad design decisions. And we also started to have users. So the big adventure started around 2016 summer: different communication strategy, new native file format based on lihata, layer rewrite, subc, padstack.
pcb-rnd was very stable before 2017, before all this started. It's somewhat more buggy and less stable today.
The key to get it very stable again is how this period ends, and that's in 2018. A visual way to demonstrate what's going on is our data model rewrite plan:
The implemetation for everything we wanted to have for 4th quarter of 2017 is already mosty done, we are starting the testing/bufixing on them - we are even a bit ahead of the schedule because we also have bbvia in 2017. The "element replacement" task will be huge in volume, but not as dangerous or complex as the previous tasks. Once that's done, the element removal will be mostly mechanical hacking. Subc-in-subc will again touch some infra, but hopefully only to a limited extent. Finally, the hierarchic netlist option - that will probably cost much more in UI programming than in core.
After executing this plan, we are essentially done with the data model rewrite. When that happens, for a few cycles I plan to cut back on big infrastructural changes in core, and concentrate on bugfixes and plugin features. Unlike core features, plugin features are much safer: a plugin bug rarely affects anything else beyond the plugin.
So the key is that 2017 is the year of big rewrites, but 2018, especially the second half, will be the period of big bugfixes. At the end we will have a software with the stability of the early 2016 versions, but with the capabilities of the early 2018 versions. A lot of capabilities that are many years ahead of gEDA/pcb's and a few that are years ahead of kicad's.
what's beyond pcb-rnd - times after 2018
Multiple products are being crafted in our codesmithy. We don't have infinite time available, so we have to throttle them, putting only as much time in each as needed to keep them up and good enough for our current purposes.
They are: pcb-rnd, cschem, edakrill, genxproj. Our flagship is clearly pcb-rnd. Edakrill is ready but is not too popular yet. Genxproj is a working prototype, but as expected, not popular yet either. Cschem is still in early design phase.
Once cschem starts to be a thing, we will also start to be having a suite: a schematics editor, a pcb editor, a parts database "in the cloud" and a project manager for the GUI users. The optional plugins in pcb-rnd and cschem make the user feel edakrill is an integral part of all this. The optional project manager makes the user feel the whole suite is really one tool. All this without sacrificing modularity.
To make it clear: our long term plan is to build up a new EDA suite, using the best ideas from gEDA combined with the best traditions/experiences/methods and code from pcb-rnd. I don't expect it to happen before 2019 or maybe even 2020. Once it happens, I expect that the significance of edakrill and genxproj will raise a lot.
This doesn't mean pcb-rnd development will stop in any way. It only means our scope will be a bit wider.