The Omni Group Forums

The Omni Group Forums (
-   OmniOutliner for iPad (
-   -   The challenge of syncing OmniOutliner—solved! (

Ken Case 2012-04-16 03:21 PM

The challenge of syncing OmniOutliner—solved!
Hi, all!

It's been almost one year since we shipped OmniOutliner for iPad. Every day since we shipped we've received feedback along the lines of "This looks great! But when will it automatically sync with OmniOutliner for Mac?"—and each time we reply back "We're working on it!" But it's been almost a year now, and I know you all must be wondering: Why is it taking so long to add automatic syncing to OmniOutliner?

To make documents with large embedded attachments more efficient, OmniOutliner 3 stores its documents as "file wrappers", which behind the scenes are simply directories holding separate files which represent different parts of the document. (The outline itself is one file, and each attachment is its own separate file.) Unfortunately, most sync services don't support syncing a directory as an atomic operation.

Wait, hold on, what does "atomic operation" mean? Well, in computer speak, an atomic operation is something that gets done all at once. In this case, we want all of the parts of the document to get synchronized to or from the cloud at the same time, otherwise different parts of the document might not be in sync with itself: it might be missing some attachments, or perhaps some of them will be out of date, or maybe there will be some extra attachments that shouldn't be there.

In other words, when syncing OmniOutliner, it's not just important to have all the parts eventually arrive on the other end; we want all of the parts of the document to show up on the other end at the very same time (and we want to know when they've all finished arriving) so that we know it's safe to open and you won't end up with a corrupt document.

One way to solve this syncing challenge is to change our document format—to stop storing OmniOutliner documents as directories behind the scenes. This change would make it much less efficient for attachments, but would also make it much more compatible with many file servers, email, web forms, source control systems, and so on. We can do that—but such a big change to the OmniOutliner file format won't work with the currently shipping apps, and doesn't really make sense to do before shipping OmniOutliner 4. So taking this route to solving the problem means for us to try to ship OmniOutliner 4 as soon as possible (which we're certainly working very hard to do).

Another way to solve this problem is to help syncing solution providers (such as DropBox and iCloud) improve their support for atomic directory operations so we can sync using their solutions. The most promising option here by far is iCloud, since iCloud's engineers explicitly do want to support syncing of file wrappers. We've had some of our most experienced developers spend much of the past year working on this approach—but much of that time was spent blazing new territory, and most issues we encountered weren't within our control to fix because we don't control the technology and servers. Fortunately, we think we've pushed through most of the critical issues and we're hoping this approach will bear fruit soon.

Of course, if it takes too long to change our document format or to get atomic directory syncing working with other people's cloud servers, another approach would be for us to add syncing support for our own cloud servers. We recently brought our Omni Sync Server out of beta, and we're using it now to automatically sync OmniFocus and OmniPlan.

One way or another, we're working very hard to bring automatic syncing to OmniOutliner as soon as possible: we know it's absolutely critical for anyone who wants to use OmniOutliner on more than one device. We're very sorry it's taking so long!

As always, we welcome your feedback!


[I]Update: April 9, 2013: We've solved this problem with OmniPresence, which is [URL=""]now available for testing[/URL].[/I]

bowdengl 2012-04-17 01:47 AM

Interim solution to sync
Just a thought, but wouldn't a 2 stage sync process solve this where you would tar (or otherwise serialise) the whole directory structure, transfer it and finally untar it, thus completing the whole package move. The only difficulty is freezing it quiesced whilst performing the tar, but that could be done fairly easily with temporary directories either for transactions occurring during the sync or to move the files to prior to the sync process to make sure it happens quicker. Just a thought and interested to hear why this was discounted.


Ken Case 2012-04-17 06:53 AM

[QUOTE=bowdengl;109559]Just a thought, but wouldn't a 2 stage sync process solve this where you would tar (or otherwise serialise) the whole directory structure, transfer it and finally untar it, thus completing the whole package move.[/QUOTE]

Serializing the document wrapper is a perfectly reasonable approach: it's how we send OmniOutliner documents through email now (using zip). That's the approach we're looking at for OmniOutliner 4.

But that's a change to the file format, one which OmniOutliner 3 doesn't understand today. It works fine for email transfers because those are one-way and opening a zip file will automatically unpack it, but it doesn't work with syncing because we need every edit to end up zipped back up again automatically.

bowdengl 2012-04-17 07:00 AM

The challenge of syncing OmniOutliner
Fair enough. The serial approach works for scheduled sync's but I see what your saying for the dynamic updates of individual items. That would be a bit onerous to freeze, tar the lot and send it up. Thanks for the response.

samoconnor 2012-04-17 11:32 PM

Content Addressed - Immutable Objects?
Have you considered using a content-adressed scheme where attachment objects are referenced by e.g. a SHA1 hash of the binary object (e.g. like the git scheme [url][/url]).

In a content adressed scheme many of the issues of syncing directories go away (including n-level deep directories).

It might look something like this:
- The top level XML document refers to each attachment by the SHA1 hash of its content.
- The .oo3 package directory has a subdirectory called "attachments".
- Each attachment is stored in a file where the file name is the SHA1 hash of the file content.
- If a multi-level deep structure is required, some of the "attachments" could be XML files that refer to other attachments and so-on.

The following properties arise:
- It is always ok to blindly sync the contents of the "attachments" subdirectory in either direction. There is no risk of overwriting anything because two files with the same name are guaranteed to have the same content.
- The top level document can be synced by whatever single-file atomic syncing mechanism is available.
- If the top-level document references an attachment object that is not available on the local device, it can be simply requested from the cloud by name (SHA1 hash). The GUI can either show "busy" while the download happens, or can make sure all the attachments are downloaded before activating the newly synced top-level document (e.g. before copying it form a tmp filename to the "proper" file name).
- Every time an attachment is "changed" it gets a new filename.
- Garbage (unreferenced) objects will accumulate, but it is trivial to purge them from time to time.

A content-addressed scheme has other potential benefits in a cloud-storage system e.g.:
- A single cloud-based object repository could be shared across e.g. all omni apps for a particular user. If the user has the same files attached to multiple documents, no extra storage space is used and syncing is faster. Imagine a photo that is originally snapped in the ofocus iPhone app, then sits in an outline for a while, then finds its way into a graffle document, then the graffle document gets placed in some higher level outline etc. The .JPG would only be stored and synced once.
- Peer-peer syncing is trivial. If a device needs an object that it does not have in its local repository, it can retrieve it from any peer (even untrusted peers because if the SHA1 matches, the object is good).


StefanPapp 2012-04-17 11:57 PM

Thank you for sharing this information

Thank you for sharing this information. As a customer, who waits for new OmniOutliner features it is good to know what the problems are and that the dev team is still working hard to solve the technical problems.


santra 2012-05-05 12:50 PM

Ditch the embedded attachments. I never even knew that OO had that capability. It's an outlining program! If I need attachments, I'll use DevonThink or TheBrain (and even TheBrain allows attachments to be links instead of actual files).

This is ironic beyond belief. OO was always the runt of the litter, the OG app that no one cared about. I had always thought it had the smallest footprint in terms of file sizes as well. Everyone worked for years to get OF to its amazing current state of syncing, and OF stores 100s of projects with 100s of tasks.

Why can't OO be "lite"? It's an Outlining program.

Syncing is much more important (to me) than attachments. Please don't let a feature that doesn't even belong in such a simple program get in the way of getting OO to Dropbox (or OG) sync.

There have been SO many outlining data storage programs that knew how to do it right...I don't understand why OG, an incredibly brilliant company, can't do this.

Even Natara, a TINY TINY outfit, came out with version after version after version of Bonsai year after year after year. Yes, they were cranky. So what?!? They GOT THINGS DONE.

Everyone at OG is so kind, especially Ken, but when are things going to get done? Narrow the scope, ditch all these extra requirements, set a reasonable deadline (NOT THREE YEARS!!!), go past it maybe a week or two (I wouldn't care about that), but please please please get us sync. Preferably on Dropbox or your own network.

What with an MBP, MBA, iPhone, iPad, and going to London to speak at a conference, I'm dying here over the lack of sync in OO.

And in case no one at OG has noticed, the iPad 3 is The Bomb. Any app that wants to really make it should play nice with the iPad and have built-in, seamless, syncing connectivity.

Example: I change the cover art for an album in iTunes on my MacBook Pro—an album that's already playing from my iPad over AirPlay to my network stereo systems—and FOUR SECONDS LATER the album art instantly changes on the iPad display.

That's cloud syncing done right.

whpalmer4 2012-05-05 01:13 PM

Many of us do make use of the ability to attach files to our outlines. It isn't very difficult (inconvenient, I'll grant you) to move files to/from OmniOutliner with the tools available now; one merely has to pay attention. Removing the ability to embed files, recordings, photos, etc. in outlines would be a substantial step backward in functionality, and one that could not be worked around by the user who needs that functionality.

chestercheeto 2012-05-06 07:57 AM

Could not agree with Santra more. Get with it on dropbox sync or those of us who need it will move on to a different solution. It is an outlining program and you dont need attachments. For those that do, have two file types, ie with and without attachments. Without can sync, with cannot. You are putting yourself at an extreme competitive disadvantage by not offering this basic and expected functionality.

whpalmer4 2012-05-07 07:50 AM

Ever stop to consider that the ability to add attachments to the outline is also a competitive advantage, one which is potentially useful to [B]all[/B] OmniOutliner users, not just those who have multiple devices and can't be bothered to move files by hand?

If you just want some barebones checklist maker with Dropbox sync, no doubt you can obtain a number of them already. If that's all you need, please go use one of those instead of insisting that Omni dumb down OmniOutliner by removing a feature it has had for the better part of a decade.

All times are GMT -8. The time now is 03:43 AM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2022, vBulletin Solutions, Inc.