Sync failure for Office-iPhone-Home scenario

[chidmf]

Omni Response:

This has been filed internally at Omni as:
<bug://bugs/48533> (Sync failure for Office-iPhone-Home scenario [overwrite])

If you are also experiencing this issue please
Email OmniFocus Support (mailto:omnifocus@omnigroup.com?subject=<bug://bugs/48533> (Sync failure for Office-iPhone-Home scenario [overwrite]))


Original User Question:

Just an alert that at least for me, the Office-iPhone-Home sync scenario just failed badly.

Setup a database at home over the weekend with sneakypeaks and iPhone 2.0.

Go to work, start up sneakypeaks with the iDisk overwriting so everything is in agreement. Did it first thing this morning... all looks good.

Made a bunch of modications to the Office database during the day. I sync'd to iDisk. My iPhone sync'd (eventually... that's another story) and was uptodate with the modifications at work. Looks good.

Came home... started up OF... reports a sync shortly after I logged in.. but NO! It's an old version of the database. My changes are NOT sync'd. I try syncing again.. no feedback, but it says the sync happened. No changes.

Worse! A short while later I ask my iPhone to sync, and now ALL the changes I made today at work are gone!

Disaster.

[gcrump]

Other than the slow syncing, I can say that I have not had a similar experience. I sync between my iMac, Macbook Air and iPhone with no problems what so ever. It has been flawless. I used to always be careful to make sure that the last sync has completed on each device before I moved to the next one, now I don't even worry about that.

Not saying that your experience isn't frustrating but I don't want people to think that it is the norm either.

On the iPhone slow part and application crashes I am hopeful, nah, confident that Omnigroup will get those issues resolved in short order.

[chidmf]

Fair enough. I guess I'll have to wait for OF staff to weigh in and explain how this might happen so I can try to figure it out. (And, are there any clues that would be left behind?) As far as I can tell all settings are correct (there's not much to 'set'). Also, of course, there were long lags between the sync at work, sync to iphone and then sync at home (i.e. all syncs were complete).

[Ken Case]

Have you contacted our support ninjas about this yet? They're at omnifocus@omnigroup.com, and can look into this in more depth. (In particular, they'll probably want to walk you through getting a list of the transaction files on your server and your out-of-sync client so they can compare them.)

[chidmf]

Yes, I've emailed about this to support.

Sync disaster continues at work this morning. First issue - looked almost ok after starting up - seemed that entries made overnight and this morning were updated at work. Good.

But I noticed that a couple of items that I had cleared yesterday and last night were still present on the list at work. Uh oh. What's that about - not a complete 'sync'? Cleared them (checked them as complete) again.

But later, got a request to reload the database... Disaster. Now I'm back with a bunch of OF entries that were cleared and/or rescheduled reverting to their old status.

Oh well. I'm shutting it down here and hoping I can rescue my iPhone copy before everything is corrupted.

[whpalmer4]

I've seen a couple of cases where it said it couldn't sync because I had reverted the local database (and I knew that no such thing had happened). I hit the cancel button, synced again and everything was happy (and my data intact). It seems like that should be safe, but perhaps someone with definitive knowledge about the sync code could comment...

[kevinkingmd]

I've had a similar, albeit not as dramatic, experience. Stuff is dropping off, not getting checked off, etc. I have an EDGE network and the software seems a bit pokey. I've already emailed the support ninjas.

KK

[marieboyer]

I am in the same "synching" boat.

Eager to hear of a fix.

[gadgetdoctor]

If one's screen goes dark (i.e. times out) during a sync on the iPhone (as mine sometimes does) what effect does this have. Does the sync continue in the background even when the phone is locked or does this cause issues?

[LawDaddy]

I've seen a couple of cases where it said it couldn't sync because I had reverted the local database (and I knew that no such thing had happened). I hit the cancel button, synced again and everything was happy (and my data intact). It seems like that should be safe, but perhaps someone with definitive knowledge about the sync code could comment...

I have had the same experience. I have got this error when I KNOW my data is current. I have dismissed this warning, re-synced, and all is well.

[Ken Case]

I've seen a couple of cases where it said it couldn't sync because I had reverted the local database (and I knew that no such thing had happened). I hit the cancel button, synced again and everything was happy (and my data intact). It seems like that should be safe, but perhaps someone with definitive knowledge about the sync code could comment...

Yes, it's safe to cancel and sync again. If it's a real issue, it will report the error when you re-sync.

We've finally managed to reproduce this in a development build, though we haven't yet figured out how to reproduce it reliably. But it's a start; hopefully we'll be able to track this down and fix it soon.

[chidmf]

man. disaster - v2.0.

so today i figure, maybe it's the third party (my office computer) that's screwing things up. i'll just setup my tasks at home, sync, let the iPhone pick them up, and do all modifications during the day on the iPhone. that'll be simple. and when i get home, my laptop OF will pick up the changes and everything will be groovy.

NOT.

get home, start up OF. doesn't look synced, but i figure what the hey, maybe it will take a little while. get a prompt to reload database to incorporate changes (something like that - sorry i wasn't taking notes). database reloads, and, while some additions seem to have shown up...

a number (maybe all) of the rescheduling changes that I made on the iphone today are lost! and, items i've checked off as completed during the day are unchecked!

this is why i'm getting what appears to be no sync at first glance. it's more complicated than that - added actions seem to be coming over but changes are not (?).

the bad part is: the screw ups are almost immediately propagated back to the iPhone.

ha! joke's on me. and here i was in a good mood, what with all my accomplishments...

----- how about some suggestions!
* sync needs transparency and record keeping on all platforms (so that user can see - sync'd version on home computer is xyz, sync'd version on iphone is uvw, sync'd version available on iDisk are xyz, uvw, pqr).
* sync needs recovery on all platforms (so that iphone data can be recovered)
* sync needs some kind of statement - 'this will be changed, this will be deleted, these changes were made on iPhone, these made on 'office''

[chidmf]

of course, for me this behavior makes the OF desktop/iPHone system completely unusable for anything beyond a providing a simple check-off list (desktop -> iphone).

[Ken Case]

Sorry, I've never seen anything like that, and I've been syncing between four systems for months now (work desktop, home desktop, laptop, and iPhone). Please continue your dialog with our support ninjas at omnifocus@omnigroup.com so they can help you track down the problem.

[rotorheadbob]

I bought OmniFocus for my iPhone last night. I've been waiting for it since I heard about it last June. Starting using today at work. I spent 2 to 3 hours, heck, maybe 4 hours creating all sorts of Projects and Actions. I got way more done than the desktop version as I find the interface there confusing (yeah, most of you don't but I sure do).

I'm loving this product, except for the crashes, which, by the way are 2 to 3 per hour.

I go home to sync and about 25% of the items transfer to the phone, where the hell are the other 75% of my items? What's up here?

BTW, I'm syncing to my Mobile Me account.

What's going to happen to the items on my iPhone? Can I turn the syncing off?

Rob Clark
I love the promise, but hate the execution of the iPhone OmniFocus

[rotorheadbob]

An update...

I looked at my iDisk copy and found the copy there matched the copy on my Mac, not what I posted on my iPhone. I don't understand the discrepancy except to say I have a Gen 1 iPhone with a slow data connection at work.

I use PCs at work, and the Mac at home.

Anyway, I deleted the copy on iDisk and re-synced the iPhone to my iDsik. Then I synced my Mac's OmniFocus with my iDisk. I got an expected 'which document version should I use' message to which I told it to use the copy from my iDisk.

All my data was intact.

While I currently have a good copy on my Mac, I'm worried that this syncing problem happen again and that all my work will be lost.

Rob Clark

[chidmf]

Sorry, I've never seen anything like that, and I've been syncing between four systems for months now (work desktop, home desktop, laptop, and iPhone). Please continue your dialog with our support ninjas at omnifocus@omnigroup.com so they can help you track down the problem.

I'll try. They seem to be busy.

[Brian]

chidmf - I see that one of the ninjas mailed you this afternoon, but I had a thought.

It sounds like the "Your database couldn't be synchronized because the local copy has been reverted since the last sync." dialog is coming up pretty frequently. Is that correct? If so, that may be a clue: it really should be a 'once in a blue moon' sort of situation. If we can figure out why, that might help us prevent this.

I'm wondering what happens in the case where you have changes both on the server *and* on the local device and that dialog comes up. In that case, it would seem like either choice is going to throw out some of those changes. (Ken may have a hundred reasons why this isn't going to happen.)

The one case that I know can provoke that dialog is when multiple devices try to sync at the same time. Any chance that's happening?

[chidmf]

Ninja's emailed about the 'duplicate Miscellaneous' problem, not this one. It's only been a couple of days since I emailed about this one - they must be busy.

I'm not getting the 'local database reverted' message in this case, that I can tell. I have seen it before, but not the last two days.

I shouldn't be doing simultaneous syncing any more than anybody else (only an iPhone and a laptop are talking to the database, and doing their thing). Maybe I'm unconsciously provoking the devices to conflict?

Some details - I'm using an Edge iPhone. I do notice some folders with names like 'OmniFocus.ofocus-copy-in-progress-hXjTKd0fcik' in the iDisk Documents folders, along with the Omnifocus.ofocus file.

[chidmf]

Day 3's disaster.

OK, so I figure I'll not use my iPhone, surely if I work with the laptop at home and the desktop at work everything will merge perfectly.

NOT.

Changes I made at work today are NOT incorporated into the database at home, even after the 'reload your database' message.

In this case, I did sync my iPhone OF but didn't do any work on it. Turns out that it's now reflecting the up-to-date version that's on my office computer. But then after 'sync'ing (such as it is) at home, OF on my home computer does not incorporate the changes made today.

WTF? Any advice on how to diagnose?

[chidmf]

How about this - is there any way to force my home version of OF to accept whatever the current version of the database is on iDisk, so that this madness can stop?

Right now, I'm sure it has 'sync'd back to iDisk and screwed everything up, but maybe tomorrow I can force it to accept the changes I've made???

I'd be happy to deal with that as a workaround. Possible?

[Ken Case]

OK, my first advice is that if you're worried about potential data loss, make a backup before you do something that might risk that data. That way if something happens, you can always restore from your backup. (In General Preferences, you can tell OmniFocus to back up twice a day, and to also back up when quitting.)

I'd love to figure out what went wrong so we can prevent it from happening in the future, but first let's just get you back into a good state. Connect to your iDisk, and move all of the OmniFocus-related stuff from the Document folder into an archive folder that we can examine later.

Moving that stuff out of your iDisk documents folder means that your synchronization should be back in a pristine, original state. Synchronize from your desktop to make sure that works. Maybe synchronize one more time to make sure it still works. If you look at your iDisk at this point, you should have a single OmniFocus.ofocus folder inside your Documents folder, you shouldn't have any "copy-in-progress" files lying around.

Now go to your laptop, and sync there. It may prompt for you to replace your local copy with the server copy; if so, say yes. Sync again, to make sure that's solid. Go back to your desktop and sync there.

Everything still working? Try changing something, and sync on both sides to see if the other side sees the change.

Now try adding your iPhone to the mix. Try changing something and see if it changes everywhere following a sync. If any of these steps fail, please let me know which one and how it failed.

Thanks!

[chidmf]

the disaster continues. more to come.

brief version - laptop OF is reporting a sync but is overwriting mostly, and not in fact incorporating modifications and updates made on the iPhone. happily i'm backing up manually on iDisk before even starting the program. but this is a complete disaster - I'm in position to lose days of work if I'm not careful.

[Lizard]

chidmf:
If you're copying the file manually to the same place on your iDisk that OmniFocus is syncing to, you could actually be *causing* some of the conflicts. If you want an extra backup of your OmniFocus database on your iDisk, put it in some separate place, such as Documents/Backup instead of the same folder that OmniFocus is syncing to.

[bmonzing]

This exact issue started happening to me this morning.

[Reported to Ninjas]

I believe my iPod and work versions were attempting to sync at the same time, but I've deleted my copy-in-progress .ofocus files and re-synced the work computer to no avail. I've turned off iPod syncing for now and will see what happens with the laptop when I get home.

{Subscribing to this thread}

Update: Laptop sync failed, too.

[jet14]

[This was just e-mailed in, before anybody reminds me about that]

I think I may have an idea as to why some of us keep running into multiple client ID numbers and exploding DB sizes: intermittent connections to our iDisks.

I know for a fact that the crappy DSL my work site has can cause no end of trouble with intermittent availability of the server. What if OF is frequently losing its WebDAV connection in the middle of a sync? Wouldn't that trigger some of these glitches?

[chidmf]

chidmf:
If you're copying the file manually to the same place on your iDisk that OmniFocus is syncing to, you could actually be *causing* some of the conflicts. If you want an extra backup of your OmniFocus database on your iDisk, put it in some separate place, such as Documents/Backup instead of the same folder that OmniFocus is syncing to.

I'll give it a try.

[chidmf]

[This was just e-mailed in, before anybody reminds me about that]

I think I may have an idea as to why some of us keep running into multiple client ID numbers and exploding DB sizes: intermittent connections to our iDisks.

I know for a fact that the crappy DSL my work site has can cause no end of trouble with intermittent availability of the server. What if OF is frequently losing its WebDAV connection in the middle of a sync? Wouldn't that trigger some of these glitches?

This is an interesting idea. I'll try to keep track of this too.

[chidmf]

Nope. Disaster after disaster. Happily I recovered from a backup of the changes made on my iPhone (i.e. from a duplicate on my iDisk), but something is still up. OF on my laptop is "syncing" and is updating the iDisk file, but it's not actually 'syncing' to incorporate the changes (particularly, changes to the dates of existing tasks). Since I hadn't planned to be a test bed for debugging this $$ software, I haven't set things up so that it's easy for me to figure out what's going on (i.e. I had planned to be a user, and have a huge project/task list).
Oh well.

[ScottE22]

I'm having an almost identical issue. I'm running on two MacBook Pros (MacBooks Pro?) and an iPhone.

For instance, I checked off some stuff over the weekend on my phone and when I got to work this morning, they were still unchecked even after a sync. Then I check off some stuff at work, fire up the home computer, sync, and the stuff I checked off at work is still showing as "overdue."

Plus, I get the "OmniFocus needs to re-open your local database" message with almost every sync.

It is very frustrating to say the least that I can't reliably look at my task list without first (a) making sure stuff I added somewhere else is showing up, and (b) verifying that no tasks that I completed are still in the list.

Hoping it gets better from here!

[cestkim]

of course, for me this behavior makes the OF desktop/iPHone system completely unusable for anything beyond a providing a simple check-off list (desktop -> iphone).

I've drawn the same conclusion. Syncing between my ipod touch and my two home computers is a mess. Not all the changes on the ipod are reflected on my computer(s). Especially bothersome are changes in dates and contexts made on the Touch. They don't sync. Usually stuff that I check off is eliminated unless it was an action that has a start date that's the same as the day I check it off the phone. Then it remains on the computer(s). In addition, I've got the doubling and tripling of repeated actions too.

I know the above is a bit vague but I just wanted to quickly add my voice before I take off for vacation. Basically, everything that's been described here has happened to me as well.

[dryver8]

I just installed 1password on my iphone, they have a "wireless" sync set up, works flawless and quick is there any plan for this in the future, I'm sure a lot of the problems were ALL having have more to do with Mobileme than OF, just a thought......

[chidmf]

I've updated to all the latest sneakypeeks & versions (7.29.08) and can report:

Syncing remains a disaster for my (real-world) usage scenario.

The usual - home laptop/iPhone sync ok, looks good, i'm feeling fine. Go to work and sync and see... work OF reports a sync but is synced incorrectly (e.g. date changes not incorporated and items reported overdue that are not).

Force a restore from an iDisk backup copy (stored in a different folder) - great, both iPhone and work OF are synced and up to date.

Make changes on iPhone, sync on work computer - uh oh. Items checked off are not checked off on the work computer. Sync again - get a result that is a 'sort of sync' in which changes made on the iPhone appear to be lost.

Not fun.

True, it's not a complete 'disaster', since stuff does happen. It's just a 'disaster' as a trusted system.

Hypothesis - there are real-world usage situations such that I and others are doing things in an unanticipated way? Only thing I can think of is making changes on the iPhone and shutting down before syncing. What else?

[Lizard]

chidmf: It almost sounds like your work machine is syncing to a different place than your iPhone and home machine. Could you double-check that they match?

[chidmf]

Yes... they are talking to the same iDisk file. These issues are more subtle than that!

[Lizard]

Do you have iDisk syncing turned on in System Preferences on your work computer?

[chidmf]

Do you have iDisk syncing turned on in System Preferences on your work computer?

No I don't have 'iDisk Sync' turned on. I'm using the cloud, only.

[ScottE22]

No I don't have 'iDisk Sync' turned on. I'm using the cloud, only.

That's an interesting question - should we or should we not have iDisk syncing turned on?

I can see how that might be causing all kinds of issues in trying to keep the local and online copies in sync...

It would seem that if iDisk syncing is turned on, MobileMe is trying to sync the online and local folders and OmniFocus is trying to do the same thing. That could be bad!

[Lizard]

We don't have concrete evidence yet, but we strongly suspect that some portion of the syncing misbehavior is caused by having iDisk Sync turned on in System Preferences. So we recommend turning it off if possible.

[ScottE22]

Yeah - as soon as I posted this I saw the thread on that issue and immediately turned off iDisk Sync on my home machine. I'll disable it at work tomorrow before launching OF and hope that fixes it!

[beharkey]

I too am experiencing very similar problems. Sync between my desktop and my iPhone are working fine. Sync between my desktop and my laptop - changes that I've made on the desktop are not syncing to the laptop.

I've detailed my problem in this thread (http://forums.omnigroup.com/showthread.php?t=9038).

I am syncing via webDAV. I don't use the iDisk all.

I've resorted to not using my laptop for OF at all until this problem is resolved.

[kerplunk]

I'm having an almost identical issue. I'm running on two MacBook Pros (MacBooks Pro?) and an iPhone.

For instance, I checked off some stuff over the weekend on my phone and when I got to work this morning, they were still unchecked even after a sync. Then I check off some stuff at work, fire up the home computer, sync, and the stuff I checked off at work is still showing as "overdue."

Plus, I get the "OmniFocus needs to re-open your local database" message with almost every sync.

It is very frustrating to say the least that I can't reliably look at my task list without first (a) making sure stuff I added somewhere else is showing up, and (b) verifying that no tasks that I completed are still in the list.

Hoping it gets better from here!
I'm experiencing these exact same issues and am not synching my iDisk in System Preferences.

However, MobileMe is asking me for my password when I open System Preferences and I have neither reset or changed it since last time I visited the MobileMe area of System Preferences. I suspect the problems reported in this thread are at least in part caused by Apple's continuing less than stellar implementation of this service.

[beharkey]

Not to be a fly in the ointment, but...

I'm not convinced that this is a problem with MobileMe/iDisk...

Because it has been happening to me and my wife and we use webDAV (bingodisk).

[xmas]

We'll be looking into this, if you do track anything down, do let us know.

[chidmf]

Disaster of the day - this time it's work on my office computer being destroyed.

This morning - all OK at home and on iPhone. Sync'd & up to date.
Get to work - get 'incompatible database' message, used 'iDisk' version. OK... looked good.

Then did a LOT of work rescheduling, completing, and rethinking some projects and actions.

Synced on my office computer. Seemed to go OK. (How could I tell - the software went through the motions.)

I've set iPhone sync to 'manual' thinking that I could keep a safe copy there. But I figured - looks good, let's sync.... Uh. oh. Nothing changes on the iPhone!!

Sync on my office computer - BAD MOVE! - lost everything.

Ha ha ha. Joke's on me.

[OK, I was able to restore from the backup...]

[xmas]

Are you using MobileMe in some cases, and the "Advanced/WebDAV" configuration in other cases?

[chidmf]

Are you using MobileMe in some cases, and the "Advanced/WebDAV" configuration in other cases?

Good question! But, no - my computers are all set to 'MobileMe' sync.

It was only a few days ago, however, that I noticed that my original iDisk sync settings were showing up under 'Advanced', and I changed them then.

iPhone is 'MobileMe' too.

[ScottE22]

Good question! But, no - my computers are all set to 'MobileMe' sync.

It was only a few days ago, however, that I noticed that my original iDisk sync settings were showing up under 'Advanced', and I changed them then.

iPhone is 'MobileMe' too.

I'm noticing something similar. I've set this up as MobileMe on the computer and the iPhone several times, but on more than one occasion the computer and the iPhone have "spontaneously" changed to "Advanced."

I am loving the power of OmniFocus, but the anxiety over losing track of a task every time it syncs is counter-productive.

[chidmf]

Sync does not work/ is not stable/ enters into some strange oscillation/ who knows!

And I'm in some strange oscillating sync universe now.

Changes made on my laptop (at home) are not showing up on my iPhone. (And I'm sure that if I sync'd my laptop after seeing the failure on my iPhone, I would lose my changes (e.g. as at work, yesterday)).

They did show up, earlier today!

I'll 'work around' as usual (i.e. delete various incarnations of .ofocus and re-sync to get a 'good' copy out there)... Ugh.

(Oh yeah, I'm manually syncing and waiting for the sync to finish after every set of changes.)

[ScottE22]

I have turned off iDisk syncing and cannot seem to keep things in sync to save my life. I've paid good money for the software and am sorely disppointed that I spend so much time every day hacking around and trying to make it work.

This is beyond frustrating.

[chidmf]

OF Sync is not reliable.

Disaster of the day - made changes (mainly rescheduling due dates) on my laptop OF, sync'd. Did this a couple of times. Looked good and I thought things were uptodate and stable.

I have an iPhone sitting on my desk. But didn't pay too much attention to it. Sync'd once to bring it uptodate.

Just now, checked off a couple of more items on my laptop OF. And sync'd.

LOST ALL CHANGES! The laptop OF database has reverted to something close to what it was before I started this morning. Except, now I have duplicate entries for periodic items that I had earlier checked off.

I'll have to make all the changes again!

[Lizard]

We're aware this is very frustrating, and are working on a solution. (OmniFocus 1.1 is still in beta.)

Meanwhile, following these best practices will eliminate some causes of these issues: http://forums.omnigroup.com/showthread.php?t=9164

[whpalmer4]

I follow all of those best practices, religiously, and I still get the problem. If you've got repeating actions, those best practices are not sufficient.

I just constructed a test with a trivial OF database containing one repeating action (every 2 or 3 minutes), had both copies of OF running through the test, no sync communication errors, and it took only about 10 minutes for the two to get confused and have duplicate actions. I only manipulated the database on one side, the other one I just watched.

It really smells like the code to recognize an incoming action change from the sync is failing to match that change correctly against a change on the local copy when that change involves creation of a new instance of a repeating action. When it doesn't match, it keeps both. Logging why such comparisons failed during the sync might shed some light.

chidmf, if you look carefully, you may see that you can simply delete the completed actions which have seemingly come back to life. That's how my database is behaving, at least; your mileage may vary! Probably much easier to examine and repair the damage on the desktop.

[chidmf]

OK last post - sync seems to be thoroughly broken - I can't even work with one pair anymore. Changes made on my office computer and reported to be sync'd to iDisk are not incorporated by iPhone OF at the end of the day.

This, even after reinstalling, at the start of the day, a backup database onto the office computer, syncing that (writing the local to iDisk), then installing that database on the iPhone (via the 'use local/remote version decision).

Oh well. I'm done. Put a fork in it.

--------------------------- good bye