Will sync time be improved soon?
 

I'm just asking because I have the iBank application both for the desktop and the iPhone and syncing the iPhone app never takes more than 5 seconds no matter how many transactions I have entered. Same goes foe the desktop application.

How long does your OmniFocus database take to sync? How many clients do you have? How recently have they been synced? Are you using WebDAV or Bonjour? Have you tired contacting support for help? It's free for licensed users and you can reach them by email or phone.

I ask all this because my OmniFocus database rarely takes more than 5 seconds to sync over MobileMe on a Wi-Fi connection. I have two clients (a Mac and an iPhone), both are synced a couple times a day, and my database has around 2200 actions and 330 projects with a file size of about 500 KB (no attachments).

I know there are further syncing optimizations/fixes planned, but in my experience, OmniFocus as it is today has been pretty solid/quick with syncing. I have no complaints whatsoever in that area.

-Dennis

I have 2 clients, MacBook and iPhone, syncing via MobileMe on WiFi. My database has 123 actions in 37 projects.

The iPhone client sync took around 25 seconds while the desktop client took only 5 or 6 seconds.

[QUOTE=hedges;69123]I have 2 clients, MacBook and iPhone, syncing via MobileMe on WiFi. My database has 123 actions in 37 projects.

The iPhone client sync took around 25 seconds while the desktop client took only 5 or 6 seconds.[/QUOTE]

That's a pretty small database and certainly shouldn't be taking that long to sync on your iPhone, at least not on a regular basis. Mine occasionally takes that long (i.e. when it's compacting a lot of transactions or I've got a glacial Edge connection), but most of the time it's only a 4-second sync or so.

How many zip files do you have? You can check this by going into the Settings screen in OmniFocus on your iPhone and scrolling down to the bottom. If it's more than maybe a couple hundred, you might have a stale client (an old client that's still registered but no longer syncing).

-Dennis

I currently have 7 zip files. I get no more than 20 zip files since I started syncing regularly.

I have the most recent version of both the iPhone and desktop apps, AFAIK (desktop app on 1.7.5, iPhone app on 1.5.3).

I'd like to emphasize that it's only the iPhone app that takes long to sync. The desktop app takes about 5 or 6 seconds only. It's set to auto-sync while the iPhone app is on manual.

A couple of variables that haven't been pinned down here:

1) how large is your database (size in bytes, not # of zip files) -- do you attach files to any of your actions?

2) you say you are syncing to MobileMe with WiFi, but that doesn't really tell us anything about your network connection's throughput. What does the "Speed test" app on the iPhone (free from the Apple app store) or [URL="http://speakeasy.net/speedtest"]http://speakeasy.net/speedtest[/URL] on your computer reveal?

How do I get the database size in bytes? Anyhow, none of my actions have any attachments.

I have a 6Mbps connection, and speedtest indicates I get better than 4mbps at any given time, up to 8Mbps on a good day.

If you look in your Documents folder on MobileMe, there should be a file called OmniFocus.ofocus (possibly just OmniFocus if the extension is hidden). Do a Get Info on that and you'll have the size.

You're certain that the iPhone is connecting via your WiFi link and not a slow cellular data connection? Speed for upload is also important because when the database is compacted, the whole file has to be rewritten. Perhaps you have a symmetrical connection, but most people don't, usually the upload speed is a fraction of the download speed.

The omnifocus,ofocus file is only 20kb.

Yes, I'm certain that the iPhone is connected to the WiFi network. I even do an occasional speedtest before I sync and I get 4 to 6 Mbps DL speeds, and .8 upload.

What's a symmetrical connection?

Well, it sounds like there's a decent possibility that you're running into a bug that causes some very inefficient sync processing (it happens on the desktop as well as the iPhone/iPod, but the greater processing power of the desktop tends to mask it). Ken posted a while back that he thinks he has a fix, but wanted to do some extensive testing internally before releasing it.

If you email the support ninjas (via Help->Send Feedback, or email to [email]omnifocus-iphone@omnigroup.com[/email]) or call them up during normal business hours in Seattle at 1-800-315-OMNI, they can walk you through the steps to confirm that this is the only issue.

A symmetrical connection is one where the delivered upload and download speeds are approximately equal. Most residential users download much more traffic than they upload, so the telco's infrastructure budget goes farther if they don't try to provision the links to be symmetrical (and it gives them something extra for which they can soak the customers who need it, such as a business with a need to send more data).

The problem described in this [URL="http://forums.omnigroup.com/showthread.php?p=69207#post69207"]thread[/URL] sounds similar. Looks like reseting your iPhone database gets sync times back to normal. It doesn't help much in getting to the bottom of this issue, but at least it should get you back on your feet again. Just be sure to have all your data synced and a good backup before beginning this procedure.

-Dennis

Dennis says:

[QUOTE=Toadling;69065]my OmniFocus database rarely takes more than 5 seconds to sync over MobileMe on a Wi-Fi connection. I have two clients (a Mac and an iPhone), both are synced a couple times a day, and my database has around 2200 actions and 330 projects with a file size of about 500 KB (no attachments).
[/QUOTE]

I would really like to know how you pull this off. I don't think I have nearly that many actions or projects, and today I'm already 10 minutes into a sync and wondering how soon it's going to finish so I can get out of the house and start my day. This is nerve-wracking.

Can someone please post that OG link to the thread or sticky that describes how to pare down the zip files so that syncing is sped up? I can't find the procedure in my notes. Thanks!

[QUOTE=santra;69701]I would really like to know how you pull this off. I don't think I have nearly that many actions or projects, and today I'm already 10 minutes into a sync and wondering how soon it's going to finish so I can get out of the house and start my day. This is nerve-wracking.[/QUOTE]

I haven't done anything special other than avoiding attachments and syncing both of my clients regularly. It's been largely maintenance free.

In fact, I've [I]never[/I] had particularly long sync times, and I'm wondering what it is that's giving some people so much trouble in this area.

As for compacting your database, you could just sync all clients and then unregister them. On one of your Mac clients, go to Edit -> Compact Database. When it's done, turn syncing back on for that Mac client. When it's done, turn syncing back on for your other clients and be sure to choose "remote copy" or "server copy" of the database (not sure how that dialog is worded now).

-Dennis

With all due respect, people here don't want talk to "Support Ninjas." They want a new release that fixes the problem. This issue has existed for the entire life of the iPhone Omnifocus, and it makes it a lot less useful. If you are like most people and have a few hundred actions, and use the desktop much more often than the iPhone, sync on the iPhone will take on the order of minutes. The most common use case for the iPhone app is to either quickly add an item to the inbox for later processing, or look at what's available in a particular context/project, and then close the app. What should take a matter of seconds takes minutes, because the user has to wait for Omnifocus to sync down, and then if an action was entered, sync back up to the server. The amount of data that changes from one sync to the next is always just a few kilobytes, and so connection speed and the iPhone's processor should be more than adequate to do this job in a matter of a couple of seconds at the absolute maximum.

[QUOTE=kammeyer;70802]If you are like most people and have a few hundred actions, and use the desktop much more often than the iPhone, sync on the iPhone will take on the order of minutes.[/QUOTE]

I've got a few [I]thousand[/I] actions and use the desktop much more often than the iPhone. My sync rarely takes more than 5-10 seconds on a Wi-Fi connection.

There may well be a common problem here, but I'm not convinced it's as widespread as one might think. I've had virtually no problems with OmniFocus syncing since day one.

-Dennis

[QUOTE=kammeyer;70802]With all due respect, people here don't want talk to "Support Ninjas." They want a new release that fixes the problem.[/QUOTE]

I'm really sorry for the frustration this has caused, and what you're looking for here is totally reasonable. Unfortunately, there isn't a single problem, so we can't make one release that fixes everyone's troubles.

So, what do we do? Every time we can narrow down a problem that makes sync slow, we fix any problem in the code and ship it as soon as we can. We also try to post here on the forums to explain the best ways for everyone to make their syncs as fast as possible, if it's not something that a code fix will address.

It sounds like your syncs have been taking a long time, and it's been that way for quite a while. Our intent is absolutely not to have that be the case. The tips [URL="http://forums.omnigroup.com/showthread.php?t=12071"]here[/URL] have helped a lot of customers; when they don't help, we're pretty successful at getting people back up and running via phone or email.

In the cases where the ninjas are truly stumped, it means that we're working on a problem we haven't seen before. Gathering information about the problem means the dev team can work on fixing it so everyone can benefit from that fix in our next release.

I can't emphasize enough how much we'd like to get this working well for you. I wish you didn't have to take the time to contact the [EMAIL="omnifocus-iphone@omnigroup.com"]support ninjas[/EMAIL] to make that happen, but it really is the quickest and best way to get this taken care of. I really hope this helps.

[QUOTE=Brian;70826]
I can't emphasize enough how much we'd like to get this working well for you. I wish you didn't have to take the time to contact the [EMAIL="omnifocus-iphone@omnigroup.com"]support ninjas[/EMAIL] to make that happen, but it really is the quickest and best way to get this taken care of. I really hope this helps.[/QUOTE]

First, let me say that I love Omnifocus. No other app compares in power, and using it has significantly improved my life. It is worth at least 50x what I paid for it. The availability of Omnifocus was a signifcant reason I switched to the Mac platform.

That said, I have followed the advice here, and it always does help temporarily, but the problem always returns. The problem is that the sync problems are not fundamentally a support issue, they are a software architecture issue. Telling users to contact support as if it is some problem with their configuration rather than a problem with the software itself is disingenuous.

As for the architecture itself, I'm not sure where the biggest problems come from, but some aspects are obvious from looking at what drops on the webdav server.

1. The data files are too large and too complicated relative to what they are actually holding. Just adding an item to the inbox requires creating one or two XML files on the server, which contain tons of extraneous information. Just the file that contains the item contains all sorts of information about my computer. Here's the file generated from me adding a single two word action to the inbox:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<omnifocus xmlns="http://www.omnigroup.com/namespace/OmniFocus/v1" app-id="com.omnigroup.OmniFocus" app-version="77.41.6.0.121031" os-name="NSMACHOperatingSystem" os-version="10.6.2" machine-model="MacBookPro4,1"><task id="apXSog3buY1"><inbox/><added>2009-12-17T07:37:31.077Z</added><name>Test Action</name><rank>2080374784</rank><order>parallel</order></task></omnifocus>

2. The app creates a new XML file every time something changes, and then because the XML files are so verbose, it ZIPs them up. Thus, for every new action, the phone has to make an HTTP connection, download a ZIP file, unzip it, parse the XML, and integrate the new action into its memory structure.

3. The main database is stored in an inefficient XML format, so in order to startup, the iPhone version has to parse a 1/4 MB XML file, even though I have only 443 actions. If this were stored in an efficient text file, the data would only take up a small fraction of the size, and would be much quicker to parse.

4. The latency of the network connection becomes a serious issue when retrieving dozens of files over WebDAV. If there were a base file and a single list of deltas from that base which was appended to, each client could simply append to that file when making changes, and only read the new portion when updating a sync. WebDAV is sophisticated enough to allow partial reads of a file. This would make the typical sync process take much less than one second.

I really hope you guys implement a new sync system that is designed for both reliability and speed. Speed is critical to making the iPhone version work well, because I'm not using it to do a weekly review -- I want to look at a list or check off an item, or add an item to the inbox most of the time, and then close the app. Unfortunately since there are no background apps, I have to sit and wait until the program is done doing its thing, which means looking at a list is a 1-2 minute process, when it should be a 5 second process.

[QUOTE=kammeyer;70863]As for the architecture itself, I'm not sure where the biggest problems come from, but some aspects are obvious from looking at what drops on the webdav server.[/QUOTE]

With that kind of insight, maybe [I]you[/I] ought to be developing this app rather than the Omni folks.

-Dennis

[QUOTE=kammeyer;70863]
Telling users to contact support as if it is some problem with their configuration rather than a problem with the software itself is disingenuous.
[/quote]
For some users, it is an issue with their usage. For some users, it is an issue with bugs. For some users, it is an issue with their network, or the server they've chosen to use. In all of those cases, the Support Ninjas can help isolate the problem and guide the users to better results and practices, collect data needed to track down bugs, etc.
[quote]

1. The data files are too large and too complicated relative to what they are actually holding. Just adding an item to the inbox requires creating one or two XML files on the server, which contain tons of extraneous information. Just the file that contains the item contains all sorts of information about my computer. Here's the file generated from me adding a single two word action to the inbox:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<omnifocus xmlns="http://www.omnigroup.com/namespace/OmniFocus/v1" app-id="com.omnigroup.OmniFocus" app-version="77.41.6.0.121031" os-name="NSMACHOperatingSystem" os-version="10.6.2" machine-model="MacBookPro4,1"><task id="apXSog3buY1"><inbox/><added>2009-12-17T07:37:31.077Z</added><name>Test Action</name><rank>2080374784</rank><order>parallel</order></task></omnifocus>
[/quote]
Yes, it appears there's a lot of stuff there. And if you added another thousand words, it would grow by a thousand words. Most of the data that you apparently object to is there to ensure compatibility between clients of different versions and possible future new platforms. The rest of it describes where in the database that new action is supposed to live.
[quote]
2. The app creates a new XML file every time something changes, and then because the XML files are so verbose, it ZIPs them up. Thus, for every new action, the phone has to make an HTTP connection, download a ZIP file, unzip it, parse the XML, and integrate the new action into its memory structure.
[/quote]
Actually, it's a new XML file for every group of changes, though in normal operation the groups are usually of size 1.
[quote]
3. The main database is stored in an inefficient XML format, so in order to startup, the iPhone version has to parse a 1/4 MB XML file, even though I have only 443 actions. If this were stored in an efficient text file, the data would only take up a small fraction of the size, and would be much quicker to parse.
[/quote]
I do not believe that OmniFocus parses the entire XML database representation on startup unless the sqlite cache is out of date. It will obviously have to parse any new XML files that come in via sync if one is done at startup. My database currently has 4,554 actions and occupies 11 million bytes on the disk, and on my MacBook (which is much more powerful than the processor in any iPhone) it takes 35 seconds to rebuild the cache from scratch, which involves parsing the entire XML representation. It normally takes about 5 seconds to start.
[quote]
4. The latency of the network connection becomes a serious issue when retrieving dozens of files over WebDAV. If there were a base file and a single list of deltas from that base which was appended to, each client could simply append to that file when making changes, and only read the new portion when updating a sync. WebDAV is sophisticated enough to allow partial reads of a file. This would make the typical sync process take much less than one second.
[/quote]
Are you aware that OmniFocus supports syncing more than two devices, and without all changes being synced before another device makes a change? And that it has to be reliable in the face of network drops and very slow write times? Hmm, this is starting to look a bit more complicated...
[quote]
I really hope you guys implement a new sync system that is designed for both reliability and speed. Speed is critical to making the iPhone version work well, because I'm not using it to do a weekly review -- I want to look at a list or check off an item, or add an item to the inbox most of the time, and then close the app. Unfortunately since there are no background apps, I have to sit and wait until the program is done doing its thing, which means looking at a list is a 1-2 minute process, when it should be a 5 second process.[/QUOTE]
The current scheme is designed for both reliability and speed (in that order). There's a bug in the currently shipping version that results in inefficient processing of new data, but my suspicion without having seen the code is that with any reasonable abstraction, the same mistake would have been made had your proposal been in place. The version being tested now really flies -- I've got literally thousands of zip files that have yet to be compacted into my database because I've got a sync client that has been offline for almost a month now, and I can still add new items or check existing ones in just a few seconds.

If you've got a problem with slow startup on your iPhone, try turning off auto-sync if your usage pattern tends to be use one device for a while, then use the other device for a while. You can do a manual sync when you switch devices. When the new bits are deemed ready for public consumption, you can turn auto-sync back on with little penalty.

[QUOTE=kammeyer;70863]First, let me say that I love Omnifocus. No other app compares in power, and using it has significantly improved my life. It is worth at least 50x what I paid for it. The availability of Omnifocus was a signifcant reason I switched to the Mac platform.[/QUOTE]

Absolutely understood, and no offense taken. Thank you.

[QUOTE=kammeyer;70863]That said, I have followed the advice here, and it always does help temporarily, but the problem always returns. The problem is that the sync problems are not fundamentally a support issue, they are a software architecture issue. Telling users to contact support as if it is some problem with their configuration rather than a problem with the software itself is disingenuous.[/QUOTE]

It sounds like my intent didn't come through as well as it should have; I should have been clearer, and I'm sorry. When we post, we have to try and write a response that works both as a response to the poster, and also for some other person with a similar problem that reads the thread a day, month, or year later.

In any case, our goal in recommending this approach isn't to assign blame - it's to try and solve the problem as quickly as possible. Support that takes the customer's particular combination of variables into account does that, and stands the best chance of getting the dev team the information they need to fix the problem for everyone else.

It's absolutely not the customer's fault if (for example) the WebDAV server they use handles things differently than the other ones we've worked with. It may be something we need to address in our code. Before we can do that, we need to find out more about what's going on.

I know you're frustrated, and you don't necessarily have time contact us, and I'm truly sorry I need to ask you to do so. Especially if the problem keeps coming back in the manner you describe, I want to figure out why and fix it, so it doesn't take more time from you in the future.

That's how we helped Bill and Dennis get the performance they're reporting now, and we can do it for you if you let us. Trust me, they didn't always have it so good. :-)

(I didn't forget about the technical suggestions you made. I'm not an engineer, but I will show this to them, and post again with more info.)

Follow-up after engineer conversation: changing the data format would provide some improvement, and is one of the things we're considering as we work on the app. We feel like there are bigger gains to be had, though, and we're looking into them as well. :-)

[QUOTE=Brian;70911]Follow-up after engineer conversation: changing the data format would provide some improvement, and is one of the things we're considering as we work on the app. We feel like there are bigger gains to be had, though, and we're looking into them as well. :-)[/QUOTE]

Brian,

Thanks for replying, and I'm glad you have it figured out. I can't wait for the new version!

Toadling,

I'm not sure if that was meant to be sarcastic, but I've certainly entertained the idea of writing a todo list app, but Omni has developed one that is almost exactly what I want. It's much easier to needle the developers of an already existing app to fix the one problem I have than to write my own :)