The Omni Group Forums

The Omni Group Forums (
-   OmniFocus Syncing (
-   -   Why are there so many zip files in my database? [Answered: see thread] (

canadagoose 2009-02-06 01:36 AM

Why are there so many zip files in my database? [Answered: see thread]
Forgive me if I've missed something here but I've not been able to find an answer to this problem on the forums:

No matter how many times I sync, run the "coalesce database" script or use the new "compact database" feature on Omnifocus for iPhone, I simply cannot stop my Omnifocus folder on iDisk from filling up with .zip files. Since my latest reset yesterday (when I erased the whole Omnifocus folder from iDisk and re-uploaded the latest data from my main machine), the number has grown to 98, covering 557 tasks.

NB: I run OF on three devices: my Macbook Air (running Leopard), my iMac at work (running Tiger) and my iPhone. All are synced regularly via MobileMe. I've read that databases of more than 500 tasks can be problematic, but have to say I don't think that's a particularly high number of tasks for a comprehensive GTD system. Also, I only have a handful of tasks with files attached, and those files are pretty small.

I'm not suffering any loss of functionality as a result of all this still loving the product, after a brief flirtation with Things but the syncing speed is obviously very slow. Any advice much appreciated.

SpiralOcean 2009-02-06 06:48 AM

When you say all are synced regularly... does that mean you are syncing and then shutting one down?

Example, syncing your home computer then shutting the home computer down when you go to work?

I found I need to leave all three OF applications up and running.
As long al all your clients have synced within an hour of each other, the compaction should happen.

You can check this by viewing the clients from the sync OF preference.

On a side note... it sounds like data compaction may be happening all the time in the beta version of OF and the new iPhone version.

If you haven't done so yet, I would send in a support request by selecting:
help -> send feedback from the OmniFocus desktop menu.

Ken Case 2009-02-06 07:03 AM

The number of zip files isn't directly related to the number of tasks you have: you could have a few tasks stored in 15,000 zip files, or 15,000 tasks stored in one zip file. (At the moment, I personally have 1,745 tasks in 171 zip files.)

Each zip file represents a change that is being synchronized to other copies of OmniFocus on other machines. (We often refer these changes to using database/finance terminology, calling them "transactions", and we often refer to the different copies of OmniFocus as your "sync clients.") Once every sync client has had an opportunity to see a change, it gets compacted into the "root" transaction (the one whose filename starts with 0), which records the common history which every sync client has already seen.

In other words, the number of zip files you have is directly related to how often you make changes and how long it's been since you've synced each client.


OmniFocus is aggressively paranoid about whether a sync client has seen a change or not, so when a client syncs (and writes a new status update of where it is) it leaves an old status update around for at least an hour just in case another client is also in the middle of syncing (possibly over a slow EDGE network) and might be trying to read the old status. What that means in practice is that a sync client doesn't only tie down the transactions it currently needs to get caught up, it also ties down the transactions it needed prior to its last sync. So if I only sync my laptop once every week, I might have two weeks of transactions tied down by those client records.

We're looking at trying to improve how often we're willing to compact old changes, but we have to err on the side of caution: if we don't compact frequently enough, syncing will be slow--but if we compact too frequently, syncing could lose data that a sync client still needs!


There's another problem we're trying to solve: right now, we're only able to compact segments of synced history when they converge to a single branch of changes. I'll explain this in more detail (with graphs!) in a future post.


More importantly, compacting down to fewer zip files isn't the only way to speed up sync processing. In fact, it's not the most important way! It does help, of course, but the real problem is that if you sync a remote change which is earlier than your latest local change, we don't have a good way to apply that to your current database (which has been modified from what the other client would have seen), so we end up rebuilding your database from scratch. Rebuilding the database means we have to reprocess every transaction, and that's where lots of transactions makes syncing slow—but it would be even better if we could just apply the minimal set of changes to your existing database to get it into the right state. (Having 15,000 zip files of history matters a lot less if you're only having to look at the last few of them!)

That's the problem I'm solving right now—and in my explorations so far, it makes a huge difference. (No more "Updating with synced data" screens on the iPhone!) But before I can feel comfortable unleashing this optimized sync code on the world, I have lots of testing to do to prove that the incremental changes we're making are getting it to the exact same state as rebuilding the database would.

SpiralOcean 2009-02-07 04:44 AM

Thanks for the explanation Ken. It helps to hear about what is going on in the background of the syncing. I also appreciate the care that Omni takes with placing data integrity high on the priorities.

canadagoose 2009-02-08 08:53 AM

Thanks Ken. Much appreciated.

paige789 2009-04-01 08:39 AM

iPhone app load time
Omnifocus takes forever to load on my iphone which has confused me for some time since I have few available/remaining tasks. With your latest update, I think I found the culprit: my database has 70 projects and 1744 actions stored in 106 zip files.

Is there any way to remove all the completed actions and projects from the sync database without removing them from the database on my computer? I like being able to see when I completed tasks and details on those tasks, but I really don't need all of this on my iphone bungling it all up.

Perhaps I'm wrong on this being the culprit.... any advice/help would be appreciated.


JKT 2009-04-02 11:51 PM

You are correct, that it is the culprit - 106 zip files is a [I]lot[/I].

There are two ways to make sure that this doesn't occur:

1. Synch both your iPhone and Mac with each other twice each within an hour to make sure the database is compacted.

2. Make use of the Archiving feature in the Desktop version to reduce the size of your active database - however, be warned that this means that you will not be able to see older (you get to define just how old) completed actions on your iPhone so don't do this if that is of high importance to your workflow*.

To do so:

1. Select the [B]Move Old Data to Archive...[/B] option from the [B]File[/B] menu:


2. Choose the date that you wish to archive back from:

(N.B. As I recently did this I don't have any old data to archive)

3. If and when you need to see any of that data, select the [B]Open[/B] archive option in the [B]File[/B] menu (below the selected option in the screenshot in 1. above).

Note, it is quite seamless to view archived data and to perform this.

Personally, I do this at the start of each month and have a repeating task in OmniFocus to remind me to do it.

* added for the benefit of anyone else reading, btw, as this isn't important to you!

Young Daniel 2009-04-04 05:46 AM

this is exactly was I was searching for. Good luck!

matthew.e 2009-04-08 09:31 AM

I just started using OFI and i have 647 zip files, sync is very slow, much slower than Remember The Milk.

And because i dont have old items, i have nothing to archive.

Brian 2009-04-08 11:06 AM

It sounds like one of the machines you're syncing with hasn't done so in a while. We don't actually remove transactions from the server until all the machines know about them, so if one of the machines has not synced in a while, the transactions build up until they do.

If you press the "Show Clients" button in Sync preferences on one of your Macs, what is the oldest date that appears in the Latest Sync column? If it's more than a week or so, that's what's going on here. Sync the device(s) in question and within a couple of hours, you should see the number of zip files drop.

All times are GMT -8. The time now is 08:38 AM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2022, vBulletin Solutions, Inc.