OMNI: Possible iDisk bug, we need your help.

[xmas]

As you are well aware, there have been many reports of syncing issues, and we think we might have tracked one of them down.

If you are using iDisk syncing with MobileMe, and you've turned on iDisk sync from your system preferences:

http://people.omnigroup.com/xmas/forums/iDisk.png

And you're having issues with syncing to your phone, let us know!

omnifocus@omnigroup.com

The subject line should be: [FORUM USERNAME]:iDisk


Here is what we think is happening:

You make a new task on the phone, and it uploads to iDisk. Seconds later, iDisk on your mac decides to upload, and it clobbers the new task your phone just made. If we can get reports of when people are seeing issues we can see if our hypothesis is correct, and if so, work on a fix.

Thanks for your help!

[whpalmer4]

Do you think this bug only happens if there is an iPhone in the mix?

[Lizard]

An iPhone may make it more likely, since it probably depends on having two different clients sync near the same time. Few people switch back and forth between a desktop and laptop as frequently as people might switch between their phone and mac.

[whpalmer4]

Okay. I didn't want to waste time trying to make it fail if it was an iPhone-only bug (to your knowledge).

[ScottE22]

I'm having sync issues and I do have iDisk syncing enabled. Heading to System Prefs to turn it off now!

I have no need for offline access to my iDisk anyway.

[Lizard]

Once you do so, ScottE22, you may also want to delete OmniFocus from your WebDAV and push up a new copy from your most current client just to make sure any scrambling that occurred while you had the syncing on gets cleared out.

[Quorcork]

Good chance you're right guys. My personal guess is that about 30-50% of my syncing issues in the last two weeks originated from iDisk performance or stability issues.
I switched off local iDisk syncing about a week ago which helped a lot in syncing stability...
Cheers

[kidtreo]

I've not activated iDisk sync (am using trial .me only) and having much success and stability.

[Dominicorourke]

I do not have iDisk syncing turned on but I do have the iDisk icon dragged into my Dock - apparantly this keeps a link to the iDisk open so you don't have to load it every time you open finder.

Will try unmounting the iDisk when I get home.

[santra]

I turned on iDisk syncing in System Preferences, it started syncing, and when it got to Item 11 of 13: "OmniFocus.ofocus," it took forever to sync. Now that it's done, the date doesn't make sense: 7/4/08. I sync both OF desktop and OF iphone every day.

[xmas]

I turned on iDisk syncing in System Preferences, it started syncing, and when it got to Item 11 of 13: "OmniFocus.ofocus," it took forever to sync. Now that it's done, the date doesn't make sense: 7/4/08. I sync both OF desktop and OF iphone every day.

Hmm... try turning it off.

Having it on is what causes the issues.

Sorry if that wasn't clear.

[Dominicorourke]

I do not have iDisk syncing turned on but I do have the iDisk icon dragged into my Dock - apparantly this keeps a link to the iDisk open so you don't have to load it every time you open finder.

Will try unmounting the iDisk when I get home.

Before attempting to umount iDisk, I updated the OF iPhone app to 1.0.2and the sync worked perfectly.

[brianogilvie]

Xmas: many thanks for this thread! I just started using OmniFocus on two machines and have encountered problems with iDisk syncing wiping out my most recent changes. I was racking my brains trying to figure out why.

[MacBerry]

I've sent the email as requested, but yes I have iDisk syncing on, and do have syncing issues.

I can't turn iDisk syncing off as I use it to keep the files on two Macs in sync, which is essential for my business, but of there's any help I can give in testing I'll be happy to.

The second Mac doesn't use the OF file, but could easily be syncing at the same time as either OF version.

Is it possible Apple don't expect the iDisk version of any file to be worked on directly (i.e. modified by OF when IT syncs)? Maybe they only expect the local copy to me worked on and then synced?

Mark

[santra]

For me it's the same drill every morning:

1. Check my desktop OF to verify it's correct; close and reopen to back it up.
2. Turn on iPhone OF and sync--only to discover the sync produces incorrect data.
3. Reset iPhone database and try again--still incorrect data.
4. Open iDisk on MacBookPro and blow away ofocus file(s).
5. Go to iDisk in cloud and blow away ofocus.
6. Open desktop OF and sync.
7. Open iPhone OF and sync.

So every morning I do this--and it takes 10 or 15 minutes--just so I can be certain I can use iPhone OF on my commute into Manhattan on the subway.

If I wanted to "hand off" my synced data between Desktop and iPhone OF again later in the day, I'd have to do some version of the above again.

It's getting frustrating, of course.

[whpalmer4]

For me it's the same drill every morning:

1. Check my desktop OF to verify it's correct; close and reopen to back it up.
2. Turn on iPhone OF and sync--only to discover the sync produces incorrect data.
3. Reset iPhone database and try again--still incorrect data.
4. Open iDisk on MacBookPro and blow away ofocus file(s).
5. Go to iDisk in cloud and blow away ofocus.
6. Open desktop OF and sync.
7. Open iPhone OF and sync.

From 4 & 5, I get the definition impression that you have iDisk syncing turned ON, correct? Have you tried turning it off, as suggested by Omni here (http://forums.omnigroup.com/showpost.php?p=43253&postcount=11)? No point in driving up your frustration level by continuing to try something that is known to be broken in the current version!

[wolfneuralnet]

I have to go through the exact same procedure as Santra above every morning to make sure I have a good sync, and I DO NOT have iDisk syncing turned on.

For a productivity app, this is really unacceptable, as it is eating a lot of my time. I may just give up on the iPhone app until this is straightened out.

At this point, I cannot even get the desktop versions to sync appropriately, and I have a tiny database.

I was really hoping that Omni was going to be the first company to get syncing done correctly. I know its one of the harding things to do, but I thought they were up to the task. I must say it hasn't gone any better for me here than with any other App that tried to sync data between machines and devices.

Heres hoping a fix is coming soon, and one that includes the fact that this is happening to people who ARE NOT using using iDisk syncing.

JAW

[whpalmer4]

Sorry if you took my message to mean that turning off iDisk syncing would guarantee success in all scenarios. I make no such claim. However, I think it is pointless to continue to do something that is known to be a problem hoping that "this time it will be different" and then continue to complain that it fails!

There are some sync fixes that haven't gotten into the general population for the iPhone yet, so it isn't particularly surprising to me that you might still have sync problems even without iDisk syncing turned on. I can report that my experiences syncing via MobileMe between desktop versions of OmniFocus have been very successful since the latest round of sync fixes came out, and with nearly 7,500 actions in my database and quite a bit of daily turnover, I think I give it a bit of a stress test.

I took a quick look at your postings and didn't find a description of the problems you are having syncing with just the desktop version. Care to elaborate?

[MacBerry]

Umm, just a little reminder - OF 1.1 for desktop is still an alpha app, and is completely free to use at the moment. IMHO it's one of the most stable alpha builds anywhere, ever, but never-the-less problems should be expected and tollerated.

There's less defence for the iPhone app I suppose, as it's not sold as alpha or beta, but given that the iTunes app store has only been around for what, a month (?), I think we should cut ALL third party iPhone developers a little slack.

Syncing, even from the iPhone, is alpha of course, because half of the equation (OF desktop), is.

I think we should also remember that Apple are not yet 100% squeaky clean with MobileMe - it still gives me trouble from time to time, even with the native apps, so I doubt all of the blame for syncing issues lies with OG anyway.

OG's rate of development has taken me by surprise I have to say, giving me the confidence to "hang in there" while they fix these issues.

Of course, I still want and expect it all to be fixed by this time tomorrow at the very latest please ;)

Mark

[MacBerry]

Well, every picture tells a story as they say:

http://homepage.ntlworld.com/mark.berry93/Picture%2016.png

This happened with OF desktop open but last synced about an hour ago. iPhone OF is closed and hasn't been synced since yesterday. How can the file on this computer (i.e. my local copy of my iDisk) possibly be newer than the one on MobileMe? It (theoretically) can only possibly have come from MobileMe.

The working OF desktop file is held in User/Documents of course - that's NOT the file the message is referring to.

I have absolutely no idea how to answer this message!

Mark

[Brian]

Mark, the troubling part here is that the older MobileMe version of your iDisk claims to be one byte of data. Don't pick that version.

It looks like you've reproduced the iDisk syncing problem; if you open up console.app, are there any messages that look related to the sync process?

We have a pretty good handle on *what* is happening here - iDisk gets confused and blows away your data; this makes it look like it's happening on the iDisk servers.

The next step is to figure out *why* this is happening so we can work around the problem. Any messages in your console log might help us do that.

(I also responded to the support ticket you sent in - if there is console info, please send it via email.)

[MacBerry]

Well, there are loads, but I don't know what to look for, and couldn't possibly send you everything.

I'm working at the moment, and will be for the next 2 hours at least, but after that will gladly give you VNC access to my machine if you want (it's all set up - you just need the url, the port number and password) to have a look around in Console for yourself? e-mail or PM me if you want to do that. I'm in the UK and it's 21:14 now - I'd need my machine back by 7 am!!

Remember I sync 2 macs to iDisk, so I guess that could compound the issue.

Mark

[Brian]

2. Turn on iPhone OF and sync--only to discover the sync produces incorrect data.


The issue that folks are running into is caused by a copy of OmniFocus starting up after several hours of inactivity and reviewing actions in the existing database for status changes that happened during that time.

When there are sync conflicts between the changes the app made on its own and the changes the sync server specifies, it decides in favor of the changes it made. This is what causes information from your sync to disappear. The bug has been fixed in current versions of the desktop app. Once 1.0.3 is released, the iPhone version will no longer make this mistake.

One thing it's important to make clear, though, is that sync conflicts in OmniFocus are record-based, not field based. We do this for performance reasons: field-based conflict resolution would make sync take much, much longer.

What does this mean? If you sync two machines, then edit an action on both machines - change the note on one machine and the start date on the other - and then sync both machines again, you're not going to see each machine get the change from the other machine. You're going to see one or the other change win out over the other.

I suggest that folks sync before they start editing on a device, and push those changes back up to the server once you're done. It makes your syncs quicker by reducing the number of change files we process during any given sync. It also means you're more likely to get the results you expect, since there are fewer sync conflicts to resolve.

[braver]

The issue that folks are running into is caused by a copy of OmniFocus starting up after several hours of inactivity and reviewing actions in the existing database for status changes that happened during that time.

When there are sync conflicts between the changes the app made on its own and the changes the sync server specifies, it decides in favor of the changes it made. This is what causes information from your sync to disappear. The bug has been fixed in current versions of the desktop app. Once 1.0.3 is released, the iPhone version will no longer make this mistake.

One thing it's important to make clear, though, is that sync conflicts in OmniFocus are record-based, not field based. We do this for performance reasons: field-based conflict resolution would make sync take much, much longer.

What does this mean? If you sync two machines, then edit an action on both machines - change the note on one machine and the start date on the other - and then sync both machines again, you're not going to see each machine get the change from the other machine. You're going to see one or the other change win out over the other.

I suggest that folks sync before they start editing on a device, and push those changes back up to the server once you're done. It makes your syncs quicker by reducing the number of change files we process during any given sync. It also means you're more likely to get the results you expect, since there are fewer sync conflicts to resolve.
Brian -- this is important, can you please specify exactly the granularity -- action? So everything about it -- project, context -- should be OK?

[santra]

What wolf said. And the amount of time it eats up is unbelievable.

I'm spoiled, I guess: OS X and Mac Apps are generally so stable and awesome.

But "sync is hard to get," as they say.

Unlike wolf, I am not willing to give up. I got through the entire OF alpha last winter, and I'll get through this iPhone crisis.

Now it's time to blow away ofocus files again! (2nd time today)

I have to go through the exact same procedure as Santra above every morning to make sure I have a good sync, and I DO NOT have iDisk syncing turned on.

For a productivity app, this is really unacceptable, as it is eating a lot of my time. I may just give up on the iPhone app until this is straightened out.

[santra]

Umm, just a little reminder - OF 1.1 for desktop is still an alpha app, and is completely free to use at the moment. IMHO it's one of the most stable alpha builds anywhere.

Agreed! It's been excellent!

There's less defence for the iPhone app I suppose, as it's not sold as alpha or beta

Exactly. That's why it's a little ironic: the app that's beta is working great, the app that's public and being sold on the iTunes store is trouble--well, at least if you're syncing.

[MacBerry]

Exactly. That's why it's a little ironic: the app that's beta is working great, the app that's public and being sold on the iTunes store is trouble--well, at least if you're syncing.

Well, except that the syncing bit really is declared as alpha, because it needs an alpha product on the desktop.

But more to the point, I think it's looking more and more as if the issue lies largely, if not completely, with MobileMe/iDisk, not OG. Given the fiasco that was the launch of MobileMe, that's hardly surprising.

I'm very very tempted to set up a WebDAV on one of my (always on) macs, and sync OF using that instead of MobileMe, which I'll leave to the native apps. The only things stopping me are the time involved, and the feeling that if I continue to sync via MobileMe I'll maybe be helping OG find and fix or work around the issues.

Mark

[Lizard]

As much troubles as you've had, Mark, I think setting up the WebDAV might save you time in the long-run.

http://manas.tungare.name/blog/2008/07/10/howto-setup-webdav-on-mac-os-x-leopard-for-syncing-omnifocus-to-iphone/

[Andrew]

Braver, granularity is at the level of an item. So changes to an action on one side and its project on the other should not conflict. Some cases can be a bit ambiguous, such as reordering actions - is that a change to the parent (project or parent action, as appropriate), or to the reordered actions? For our purposes, that is considered a change to the parent, so for instance if you update the notes for a project on one device and reorder that project's actions on another device and then sync, either the note change or the reordering will be lost, depending on which change was made most recently.

[Brian]

Brian -- this is important, can you please specify exactly the granularity -- action? So everything about it -- project, context -- should be OK?

I'm not a developer, so I'm not really qualified to answer at this level of detail. I know some other folks are working on posting more info about how sync works, though.

Here's what I do know: if you avoid sync conflicts by syncing changes to a device before you changing anything in OmniFocus, and then push those changes back to the server before switching to another copy, your syncs will be faster and will you will have a smaller chance of getting unexpected results.

Start up OmniFocus and pull all the changes down from the sync server before you start changing things. Change things; doesn't matter what. After you make changes, send them back to the server. Switching between 2, 10, or 100 machines will work better this way. Order doesn't matter.

Think of it like a sandwich - all your changes want to be a tasty layer of goodness cushioned between two delicious, fluffy syncs. Your server can hold an infinite number of sandwiches which come in any order from your machines.

If you start interleaving your sync sandwiches, though, things get really messy really fast.

[santra]

Got home from subway commute. Carefully synced changes from iPod OF to MobileMe cloud. Turned on MacBookPro, opened OF, synced. Everything matched. Perfect. Closed iPhone OF, turned OFF iPhone.

Worked on and off for the next hour on the MacBookPro OF. Then up pops a message: server copy of database out of sync with local copy. So I pick server--it's been syncing, right?--and lose all my changes from the last hour. Try reverting to backup, desperately hoping, but it's half an hour old.

I am ready to cry.

[santra]

Much happier this morning. Turned off Sync in OF Desktop. Deleted OF from iPhone. Am now using small paper notebook and pen during subway commute. Much more dependable. No down time. No time wasted syncing or troubleshooting syncing.

But I'm checking iTunes Store hourly for 1.0.3 update release. Don't know if it will fix these MobileMe sync problems, but I certainly hope so.

[Brian]

Then up pops a message: server copy of database out of sync with local copy.

I'm really sorry this happened - unless you've actually have reset the server or local databases, this dialog shouldn't appear.

If anyone sees this again on a desktop machine, it would be helpful if they could zip up the local and server databases, then send them to us so we can figure out why this dialog showed up.