KDE 4.2 KPilot coming! (how to both be excited and have realistic expectations)

Monday January 19, 2009 by Jason 'vanRijn' Kasper | 22 Comments

I just sent this to the KDE PIM mailing lists, but I know not everyone who uses KPilot subscribes, so I’ll re-post it here…

Howdy all,

I just wanted to get a note out to as wide a distribution list as possible to spread some important news about the upcoming KDE 4.2 release and KPilot’s (exciting!!) part in it. If you don’t care about KDE PIM, data syncing, Palm devices, or KPilot, you may stop reading now and I won’t be offended. =;P

For the last 2 years, the talented Bertjan Broeksema and I have spent our Google Summer of Code months doing some major rework and redesign for KPilot. You can see our sync algorithm redesign Use Case here: http://snurl.com/acftl and some UML sequence and class diagrams that we used here: http://snurl.com/acfwd. Previous to our redesign, each conduit in KPilot contained all of the logic necessary for the syncing of record-based data, as well as the transformation, comparison, and resolution of that data. This means that each conduit did things its own way. We didn’t share any code between the conduits for syncing, and we weren’t consistent about how the data was synced. In addition, NONE of our conduits had the ability to synchronize category information from PC to Palm. We had inconsistent smatterings of success going from Palm to PC, but again, we were inconsistent and buggy. And obviously with this duplication of work, whenever a bug popped up in one conduit, it likely existed and had to be fixed (and most probably did not get fixed) in the other conduits. In addition, we did a very poor job of ensuring that we didn’t lose our users’ data, and subsequently, we had a bunch of “KPilot ate my future” e-mails (http://snurl.com/acgf0) that nobody was happy about.

So one of our most important goals in our KPilot redesign was the safety of our users’ data. We now go to great lengths at the beginning and end of the sync process to make sure that our record counts match between Palm and PC, and that we’re able to save all of our data successfully. If anything breaks in the process, we do not commit or save our changes to the Palm and PC sides.

We also didn’t want to keep writing and maintaining “how to sync” logic in our conduits anymore. This was a major design flaw previously and led to a lot of rework and problems. So we designed a record-based (since all of our syncing is from record-based sources) syncing conduit set of base classes that all of our new conduits inherit from. This means that now all of our conduits use the same syncing logic and flow. If we got it wrong, we fix it and now all of our conduits are fixed. If we need to improve the performance and efficiency (which we do, still), we fix it in one place and now it’s fixed and improved in all of our conduits. This is a much cleaner design and it means that our implementing conduit classes are now MUCH simpler, have MUCH less code in them, are MUCH easier to understand, and all follow the same patterns, meaning the learning curve for debugging, writing and contributing conduit code just became darned near close to flat. The job of our conduits is reduced to data transformation, comparison, and record domain knowledge, as it should be. The syncing logic all happens in the base classes.

Another nice feature that we added, although it’s not completely finished, is a final “volatility” check in the base conduit classes. At the end of the sync, we look at how many records we have created/updated/deleted in comparison to starting and ending counts, and if the changes are too volatile, we’ll ask our users whether they want us to save the sync or undo it.

Anyone familiar with programming in professional environments should be familiar with the idea of commit and rollback. Well, previous to KDE 4.2’s KPilot, we had one mode: change live data on the fly and don’t bother with commit/rollback. This made things nice and fast, to be sure, but we also lost a lot of data that we shouldn’t have. In fact, we didn’t even have a way to not lose data, previously. So this was another important design decision we made in the new version of KPilot. We do all of our syncing work in temporary storage and only at the end of a successful sync process do we commit (or rollback) our changes to the real (PC and Handheld) data stores.

We also have a nice new Keyring (http://gnukeyring.sourceforge.net/) conduit for KPilot which uses our new base syncing classes. Unfortunately, we didn’t get a chance to finish polishing it off, and thus it will most likely be disabled for the final KDE 4.2 release. We’d love your help in getting it finished, though. Bertjan has even written a nice Qt4-based PC Keyring data file viewer/editor that we’d love to get finished, polished, and released. Please help!

So there’s a LOT of coolness in the upcoming KDE 4.2 version of KPilot. Not to mention the fact that it’s the first KDE4 version of KPilot. But I also want to set realistic expectations so that we’re all on the same level.

This will be the first release of a major rewrite of KPilot, and we have had very little user feedback thus far (although there have been a few very helpful folks and I really appreciate them!!!). This means that there are guaranteed to be as-yet-undiscovered bugs. I have been doing a bunch of testing and bug fixing, even since the last RC1 release of KDE 4.2, but there’s just no way I can hit all of the use cases we have. Things are looking extremely smooth and stable and I have not once lost data (yay!), but I’m sure bugs are in there, so make sure you make and keep a good backup of your Palm and PC data before trying new software.

Previous to KDE 4.2, KPilot relied on the external data sources to know about the condition of that data source’s records and PC->Palm mappings, etc. With KDE 4.2, KPilot now keeps this information itself. This is a good thing! We had a bunch of problems in KDE3 which were caused by the mapping being wrong and/or the state being wrong (X-PILOTID and X-PILOTSTAT). So we have our own XML-based mapping files now that are kept in $KDEDIR/apps/kpilot/conduits/<

username>/mapping/ and these help us to keep track of the mappings as well as per-conduit last-synced times, etc. We currently rely on this mapping to be valid at the start of a sync (which is also a good thing), but which has one unfortunate side effect that we’ll most probably not be able to get fixed before 4.2 is release: CopyHHtoPC and CopyPCtoHH don’t work as advertised right now. This also means that the first time you sync with the new KDE 4.2 KPilot, we will do a “first sync”, regardless of whether you’ve already synced with KPilot so that we can correctly establish this mapping file, and you’ll end up with a combination of PC and Palm data on both sides at the end.

We’re using Akonadi for syncing your data in KDE 4.2! This is also really good goodness! This will allow us (eventually, once the Akonadi back ends are written) to sync your Palm pilot with an Exchange data source, or a Google calendar, etc., etc. The realistic immediate view, though, is that KPilot is using and relying on a really new Akonadi solution. KDE PIM 4.2 is still not fully ported to using Akonadi purely, although we have bridge resources as an interim solution. This means that before you can sync with KDE’s 4.2 KPilot, you need to add an Akonadi resource for each data type (Calendar, Contacts, etc.), then change your KDE PIM applications to use that (korganizer, kaddressbook, etc.), and then reconfigure KPilot’s conduits to use those new Akonadi data sources. There are glitches to be found in all these layers, to be sure!! In my testing, it looks like you have to hit reload in korganizer to see kpilot changes to akonadi’s cache after a HotSync, for example. Also, it’s a manual process to convert your PIM data sources to Akonadi resources so KPilot can sync with them. This MUST be done, since KPilot now syncs only with Akonadi data sources. Together, we’ll be finding (hopefully just a few) bugs in KPilot, the new KDE PIM code, and Akonadi. The pay-offs for all of this work are immense, but we need to have realistic expectations about it too.

We have a serious developer shortage in KPilot–both real-person-resource as well as available-time-resource–and as a result, we were only able to bring so much forward from KDE3’s KPilot into KDE 4.2. We tried to focus on the most widely used functionality and, to be honest, that which we the KPilot developers actually use. This means that KDE 4.2’s KPilot will not have: the Knotes conduit (which was badly broken anyway), the DOC conduit (which I’d love to get working again but we’ll need someone to help code it to get it done), the Avantgo/MAL conduit (does _anyone_ actually use Avantgo anymore???), the popmail conduit, or the notepad conduit. So, if you absolutely have to have one of those, please continue to use the KDE3 KPilot, or better yet, help us port your favorite conduit to KDE4!!! What we do support in KDE 4.2’s KPilot is: Calendar conduit (syncs against Akonadi), Contacts conduit (syncs against Akonadi), Memofile conduit (syncs to a directory/files on your hard drive), the Time conduit, the Palm database install conduit, and the ToDo conduit (syncs against Akonadi). These are by far the biggest heavy hitters (the 4 main Palm apps), but again, if you absolutely must have one of the others (or a new conduit altogether), please help us make it a reality by contributing.

Our little First Time Setup Wizard’s device detection functionality is broken. Actually, it’s been broken for a while. But we didn’t have time to try to get it working, so we disabled that functionality. Really, this is ONLY a one-time setup issue, and there are far better ways to figure out what connection string you should be using, and they’re pretty standard now (“/dev/ttyUSB1” if you’re using the visor kernel module, “usb:” for libusb syncing, /dev/ttyS* for serial devices, etc.).

Previously, KPilot tried to be a “anything that has to do with a Palm device” solution, and we had built-in viewers and editors for Palm databases. These were buggy and error-prone, and confusing, since it looked like KPilot could sync with your Palm, KDE PIM apps, and any changes you made to the databases through the viewers/editors. Well, no more. We’ve removed these viewers and editors to fix a bunch of bugs, to cut down on the porting and maintenance efforts, and to provide a cleaner user interface. KPilot is now focused on being a syncing solution between your Palm and KDE’s PIM back ends. If you need a generic palm database viewer/editor solution, JPilot is much-better suited to that.

Unfortunately, using “usb:” (libusb) to sync your USB palm still has some weirdness to it. I have not had time to dig until I found the solution, but even though we do the exact same thing with “usb:” devices through pilot-link that we do with old-style USB devices (/dev/ttyUSB1, etc.), we periodically hit a crash when closing libusb sockets. Sometimes this shows up as a double-free bug down inside libusb, and sometimes there’s other weirdness. Bottom line is this: “usb:” should be fine for syncing your Palm with KPilot, but you may see crashes after your data has been synced or when KPilot closes the socket handles to libusb (pi_close, in paricular, causes this). This is an area I’d REALLY like some help with, since I don’t see anything that KPilot is doing wrong.

One of the challenges we’ve always had is in getting our users to help us understand what’s going wrong when something goes wrong. In the past, this has meant that we’ve had to tell them to get KPilot’s source code, reconfigure it to make sure debugging is turned on, recompile, re-install, and test again before we could get ANY useful information for as to what might be going wrong. This presented an insurmountable barrier to entry for 99% of our users and has resulted in a lot of bugs never getting fixed and a lot of frustration and wasted time, all around. Well, no more. In KDE 4.2, KPilot now should have debugging (with tunable output levels) turned on by default. The positive to this is that we should be able to ask our users to just run their already-installed version of “kpilotDaemon –debug=9” and send us the output for us to be able to get a much better look at what might be going wrong inside the wee beastie. The down-side is that we will be producing more console spew as a result. I’ve tried to keep this to a minimum, but I’m sure this will upset some people anyway. For those folks: workaround would be to run “kpilotDaemon >/dev/null 2>&1”. We can definitely improve in this area and prune down what gets printed by the default debug level, and I’d greatly appreciate any patches to help do so.

And lastly, as I mentioned previously, we made some intentional design decisions that favored the protection of our users’ data first and foremost. As a result, some things will take longer in KDE 4.2’s KPilot than they used to in KDE3. Most notably in this area is our “first sync”, where we take in both collections of data (Palm and PC) and match them up and result in a combined set. This area has not been performance-tuned (we’re currently at an operational efficiency of O(n^2) in our data matching code, I believe, which is obviously bad), and there are definitely things we need to improve on (and would REALLY appreciate the community’s help on!!!). If you have a database with thousands of records, then “first sync” (which happens obviously the first time, but also if the record mappings are invalid) will take a while and your Palm may time out. Less than optimal? Yes. Slower? Maybe, the first time. You lose data? Definitely not. Can you help make it better? YES! =:)

So, in closing, I hope this helps to get all you Palm users excited about our new version of KPilot. Never before has KPilot looked so good, been as careful with your data, or been easier to help debug, extend, and help out with. There will be some challenges, as there are with any new major version of software (and this is a doozie, all around!!!). But I’m really pleased with what we have and where we can go from here.

Thanks, enjoy, and please give us feedback!! As always, bugs should be reported to bugs.kde.org, e-mails should be sent to kdepim-users@kde.org, and more-or-less real-time help can be found in #kpilot on FreeNode IRC.=:)