The Omni Group
These forums are now read-only. Please visit our new forums to participate in discussion. A new account will be required to post in the new forums. For more info on the switch, see this post. Thank you!

Go Back   The Omni Group Forums > OmniFocus > OmniFocus 1 for Mac
FAQ Members List Calendar Search Today's Posts Mark Forums Read

 
BUG ANALYSIS: Corrupt database following sync Thread Tools Search this Thread Display Modes
Why hello there,

Although I intend to submit a bug report via e-mail to Omni as they've requested, I thought it would be helpful for people to read about my own experiences that took place this morning, and how I worked to identify (and fix) the problem.

I woke up to find that my OmniFocus across my iPhone, iMac and MacBook Air was horribly corrupted since I left it last night. You can imagine that this was very upsetting, especially after having completed some monumental project-planning. The problem manifested itself differently on each device:
  • iMac: Random projects and action groups were missing names
  • iMac: Random projects and action groups appeared in my inbox
  • iMac: Missing projects and action groups
  • MacBook Air: Missing projects and action groups
  • iPhone: Missing projects and action groups
After the initial shock of seeing all devices corrupted, I began to plan my attack on the problem. I have the following setup:
  • iMac MacOS Lion 10.7.3: Primary machine. OF vers 1.9.4
  • MacBook Air MacOS Lion 10.7.3: OF vers 1.9.4. Syncs from iMac
  • iPhone iOS 5.0.1: OF vers 1.13.3. Syncs from iMac
It was still before 10am PST, and therefore too early to call Omni. Besides, I preferred not to send anyone my OF database.

My first attempt was to revert back to my most recent version of the database in the Backups directory. I tried this on both my iMac and my MacBook Air, but I still saw the same problem.

It had occurred to me that a last-minute OF edit on my iPhone last night might have caused the problem, but I dismissed the idea as the manifestation was too profound. As a sanity check, I also confirmed that OF wasn't in "focus" mode.

At that point, the only step was to try revert to an earlier backup. I imported the previous backup, and it worked! Unfortunately, the previous backup took place four hours earlier, and in those four hours I had made many changes. In addition, I was still uncomfortable about the fact that I did not know what happened.

Nonetheless, I was very pleased that, at worst, I would need to re-do four hours of planning. However, I had lost a huge amount of trust in OmniFocus and felt that I needed to better understand the problem as a way to rebuild that trust. After all, I've been using it non-stop for six months without a single problem, and I am hugely dependent upon it.

After a little research online, I discovered that I can open up the database by changing the extension of the file from .ofocus-backup to .xml. In this way, the file changes into a folder containing a bunch of zip files, each of which themselves contain a xml file. After suddenly thinking that I could read each of the 500 xml files and simply re-structure my project from scratch, I quickly realised that each xml file represented a change-set, instead of a project node.

I did actually contemplate parsing through each xml and re-building my projects manually, effectively re-running my decisions last night in slow motion, but, thankfully, I decided to have a bowl of cereal first! Over this bowl, I was struck by a better idea. That is, to reconstruct a new, hybrid ofocus-backup file by gradually adding change-sets from the newer corrupt database to the older non-corrupt database.

If I was fairly easily able to convert between .ofocus-backup and .xml versions of the database, I could hone in on the change-set that caused the corruption in the first place through a form of binary search. To begin, of the 500 changes that took place between the two most recent backups, I copied across the first 250 changes. It worked! I saw a non-corrupt version of my OF with half of my changes. I was startled.

From that point on, it was easy. I continued to add half-size groups of changes until I had around 10 changes remaining. As I originally suspected, the problem change-set was among the last few; it had to have been, because I saw no problems last night. Finally, I found the problem-causing change-set three before the end. The xml is:

Code:
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<omnifocus xmlns="http://www.omnigroup.com/namespace/OmniFocus/v1"
app-id="com.omnigroup.OmniFocus" 
app-version="77.86.0.157059" 
os-name="NSMACHOperatingSystem" 
os-version="10.7.3" 
machine-model="MacBookAir3,1">
<folder id="ffm62XkZ7r8" op="delete"/>
    [MANY FOLDER DELETES]
<task id="opxWeVcHLHu" op="delete"/>
    [MANY TASK DELETES]
</omnifocus>
While it seems that this xml is not itself corrupt, a clue lies in the machine-model. Just before closing up last night, I decided to create a new sync connection between my MacBook Air and my iMac. And, for some reason, that caused the MacBook Air to send a whole bunch of deletes to the iMac, thereby giving the appearance that the database had been corrupted. Besides creating a new sync connection, I took no further action on the MacBook Air last night and so those random deletes should not have happened.

And that's it! I am back to being a (mostly) happy Omni user again. I have recovered all my changes from last night, and I have a better sense of what happened so that I can avoid it / look out for it in future.

I am happy to answer any further questions about it.

Regards,
Bekhor
 
What do you mean by "create a new sync connection"? You clicked on the sync button, you changed the sync configuration preferences, or what?

Good job on dissecting the database, btw!
 
Hello again, whpalmer4!

I appreciate your remark. I was rather pleased with the diagnosis myself! Although I must admit that I have a technology background.

By creating a new sync connection, I meant that I reset the configuration preferences. The Air had previously synced with the iMac in the past, but then I turned off syncing on the Air a few days ago.

Last night, under the Bonjour tab, I chose to re-establish the sync. The same (and only) iMac was already in the drop-down list of machines. Is it possible that it confused machine/database profiles (if such exist)?

Or, perhaps, the fact that I deleted all my tasks on the Air in order to clean OF before re-establishing the sync confused the sync operation (having previously been connected with the same iMac only a few days ago).

If this is the case, it is still a bad bug in the instance when a user re-establishes a sync between the same two machines.
 
To me, it sounds like OmniFocus did exactly what you told it to do but that was not what you intended! You had your machines syncing. You turned off sync on one of them, and deleted everything. You turned sync back on. It promptly synced those changes you had made.

An argument can be had about whether once you had changed the sync settings OmniFocus should have not let the sync proceed without a choice of one database or the other. I think the current behavior is deliberate, to prevent someone from having to manually merge two databases if the sync settings are disturbed while there are changes outstanding. Someone from Omni might be able to give a definitive answer.

I'm a bit unclear as to what you were attempting to do by deleting all the tasks on the MBA, then syncing. Apparently you weren't trying to erase your sync database, or you wouldn't have gone to all the trouble to restore it :-) If you had made numerous changes on the MBA (which hadn't been synced) that you wished to discard, with the MBA just picking up the database as it existed on the other machines, deleting your OmniFocus.ofocus file (with OmniFocus not running) on the MBA, then launching OmniFocus and configuring sync would be the way to go. Essentially what you want in that scenario is to treat the MBA as if it is a new addition to the family, and deleting the old database file accomplishes that. You would be asked during the first sync whether to keep the device database or the sync database (as they would not match), and the correct answer would be the sync database.

All in all, I can't really fault anyone for not divining the correct set of steps to take there, even a guy who manages to replay only part of the transaction history to fix his database. You've raised the bar, however; next time I expect you to figure out how to remove only the offending transaction from the history :-)
 
:)

I appreciate the compliment. That certainly would be a greater challenge! I was fortunate, I suppose, that I had reset the sync connection as a final step without any further changes to the OF project structure. In this way, the next day, I only needed to delete the final three change-sets (the other two were further deletes from the MBA, all in a span of a minute) and I was all set.

To analyse a little further (because I love to analyse), it is indeed plausible that OF was doing what it was supposed to do. And, normally I would have thought twice about re-syncing machines. (As it happens, I've carried out the same procedure a number of times in the past on both my iPhone and MBA without a glitch; though perhaps I'd been lucky)

However, assuming that the tree we are barking up is correct in the first place, I believe that I was misled somewhat by OmniFocus (amounting to a User Experience problem), and I also still believe that there is an underlying bug:
  • Bug: If OF was indeed doing the right thing on re-establishing the sync, then why did I see any records at all in the morning? Instead, I saw what can only be described as patches of tasks all over the place. In fact, had OF instead been completely empty, it would have been far less disconcerting and I would probably have known exactly what had happened. (In addition, on having synced the night before, I remember seeing the entire project structure on my MBA.)
  • UX: In answer to your question, I had cleared OF on my MBA simply to clean the slate. I wanted to identify a successful sync (as a matter of habit, for good or for bad), and starting with a blank OF was the easiest way (though I realise that deleting all items is not the same as resetting the database). I agree with you that adding the MBA as a new addition to the family was the best way to go. However, I thought I was doing exactly that. In the sync settings last week, I had changed the setting to "Nothing". To me, that means I've erased the current sync settings and the next time I choose Bonjour sync, it's a fresh start. Instead, perhaps there should be a "Disable sync" setting under "Bonjour" to distinguish between turning off the sync temporarily, and erasing it.

I submitted a report to Omni during the week; I will let you know if I hear anything back. Thanks again for all your assistance. It's been one of the more challenging problems to solve this week!
 
I received a response from Omni. And I stand corrected concerning the first bullet-point above. OF would not necessarily be empty. I was reminded that those tasks that I had added on the iMac after turning off the sync on my MBA would not be deleted as the MBA would be unaware of them on re-establishing the sync.

Whether there is a bug or not (there will be further investigation), it is becoming clearer to me that my workflow was the trigger. In the future, I will reset the database on the MBA on the (hopefully rare) occasion that I need to disable the sync and then re-establish it.

At the same time, I will also be recommending to Omni the UX improvements that I outlined above. As a great sign of customer service, they have asked for suggestions.
 
 


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes


Similar Threads
Thread Thread Starter Forum Replies Last Post
"database on your iPhone is incompatible with the sync database" error message kevinwest66 OmniFocus Syncing 36 2011-05-08 04:32 PM
sync-database duffman9000 OmniFocus for iPhone 3 2011-04-19 01:27 PM
Omni qualitative analysis software? JimG Omni Lounge 9 2011-03-10 12:55 PM
Business Analysis justin efriend OmniPlan General 1 2010-04-29 01:16 AM


All times are GMT -8. The time now is 02:22 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2017, vBulletin Solutions, Inc.