Average Sync Times

[scaleslea]

My wife and I have been having lots of problems with long sync times both on the iPhone and the desktop client when syncing to iDisk. It wasn't until today when I finally ran the Coalesce database script that we finally got sync times measured in seconds instead of minutes. All this time we've been wondering what was wrong with OmniFocus and why it was syncing so slowly.

So I've been wondering, what kind of average sync times is everyone else getting?

Today my sync time went down to 20 seconds from 2-5 minutes.

Doc

[ohammersmith]

I'm finding that I have to run the coalase database script anytime I do a lot of editing on my desktop OmniFocus. I've been tracking the size of the folder on the server, along with the types of files in it. Yesterday I was catching up on my weekly review (3 weeks behind. bleah.) and my file exploded up to 3 megabytes. No amount of syncing could it it to come down very much. This made my iPhone sync times a copule minutes, or more.

These are the graphs I have from my monitoring system.

http://img.skitch.com/20081012-d67sy9xgcc4dw6i3b6dnqawbpf.jpg
http://img.skitch.com/20081012-gwicpc2ed8s2a8p6i4fmbjbqgw.jpg


I think the solution is to sync less often. Very rarely is the every minute sync going to be useful in my use cases. I still have to be sure to sync before I close my laptop, or lock my iPhone. All every minute does is create a crazy number of transactions to catch up on... you can see from the graphs that I got up to 600 transaction files.

[whpalmer4]

If you don't have some stale client entries (check the Show Clients display in the Sync preferences in OmniFocus), two syncs on all of your devices separated by an hour should cause your database to automatically compact.

[ohammersmith]

two syncs on all of your devices separated by an hour should cause your database to automatically compact.

Why the hour? I guess there's something OF is waiting for that I don't realize?

I'm pretty sure I did that yesterday, but I can't guarantee it. A number of the transactions were at least 2 hours old, I do know that.

[whpalmer4]

That's the way the code makes sure there isn't a really slow client sync in progress, or some other condition that would make it unsafe to compact the database. If everything is synced up twice, separated by at least an hour, the compaction code should run.

[ohammersmith]

Cool. Next time it blows up to 3 meg, I'll be more patient and see if the database compcats itself.

[uof]

Regarding the hour gap between syncs: does anyone know if you sync on any client before the hour is up (just a sync, with no changes made to the file) stops it coalescing until you get a full hour gap?

[Lizard]

It shouldn't keep coalescing from happening, you just won't get quite as much coalesced.

For example:
Day 1
9:00 sync all 3 clients -- 15 .zip files on server
10:05 sync all 3 clients -- 18 .zip files on server, deleting all client files before 9:00
10:10 sync 1 Mac -- coalesce brings you down to 4 .zip files

Day 2
9:00 sync all 3 clients -- 15 .zip files on server
9:30 sync iPhone -- 16 .zip files on server
10:05 sync all 3 clients -- 19 .zip files on server, deleting all client files before 9:00
10:10 sync 1 Mac -- coalesce brings you down to 5 .zip files

[eecue]

Can you post the scripts for that cacti graph? Thanks!

[ohammersmith]

Can you post the scripts for that cacti graph? Thanks!

It's actually a munin (http://munin.projects.linpro.no/) plugin, but yes, here they are. They are very simple and not set up for being any old user. You'll have to edit in your username in the script.

If I was feeling ambitious (or needed it), I'd rewrite it so you could symlink to omnifocus_disk_ to omnifocus_disk_<username> but I don't really have a need for that, right now.

FWIW, I've seen that OmniFocus really does compact things correctly, I just wasn't being patient enough... or syncing my iPhone often enough. I find it makes things better if I happen to open my iPhone and sync in the middle of a major multi-hour edit session (review day, etc).

Anyway, scripts.

Disk space usage graph: http://pastie.org/303332

Number of files and types stacked graph: http://pastie.org/303335

You'll note from the comments that the latter was modified from the apache process graph. :)

BTW, Pastie seems to to be repeating the here doc boundaries. Make sure you download the text version or you won't have a working script.

[santra]

Now that 1.1 OF on iPhone is out and I can use Bonjour syncing, my sync times have fallen from about 50-90 seconds to about 4 seconds. And my database has 100s of projects. I'm a very happy camper now.

[Ken Case]

Now that 1.1 OF on iPhone is out and I can use Bonjour syncing, my sync times have fallen from about 50-90 seconds to about 4 seconds. And my database has 100s of projects. I'm a very happy camper now.

Glad to hear it!

We worked hard to improve sync times in v1.1 in general (reducing the number of network transactions we have to perform), so hopefully WebDAV syncing wouldn't take quite so long now either. But it's hard to beat a local Wi-Fi connection to your server!

[peteypolo]

I've made a point of using the archive function to reduce the size of my OF db. Still I'm finding that every week or two I have to run the CoalesceDatabase.scpt file posted on these forums some months ago.

I usually know it's time to coalesce because iTunes takes for-ever to backup the phone. Forever meaning over 10 minutes, where it usually takes two minutes.

I just did the operation a few minutes ago, shrinking my db from 1.7 MB to 282 kB (over MobileMe).

I kind of suspect that syncing over EDGE has a little to do with it. No proof, though.

I hate to beat on this particular horse, oh OmniGurus, but having to run a script outside of the native application to keep the database in working condition seems wrong. Could it be there's something corrupted in my database that is causing this disk hoggery? I hope so, because this problem kind of feels like owning a car where I have to get a new battery installed every few weeks--it works great for a while, but then serious manual intervention is required.

[Lizard]

peteypolo: It does sound like something's not going quite right for you. Next time it starts slowing down again, I'd recommend emailing our Support Ninjas (omnifocus@omnigroup.com) with a list of the file names inside your .ofocus file. They may also want copies of the .client files. This will give them information about which clients are syncing when, but won't reveal any info about your actual actions in OmniFocus.