SOLVED: Mysteriously growing OmniFocus.ofocus file
 

My database file in my Application Support folder is mysteriously increasing in size.

It started at aprox 900k, but I wanted to get it a bit smaller for syncing with my iPhone. (Here in Australia we have small and expensive data plans so I need to ration my data usage.) I deleted some tasks and projects, and used the new archive feature to archive old completed tasks. But the file stayed the same size. Then, over the next few hours my file increased to 1.5MB, and this morning it's at 1.6MB.

Does anyone have any idea what is going on?

(P.S. I've also emailed this to support ninjas)

From my observations, it seems the database file (actually a bundle of multiple zipped XML files) grows as you conduct transactions (i.e. add/edit/delete items). I think OmniFocus has been using this, or a similar model, for some time now.

Those zipped transaction files seem to hang around in the package until you sync to your remote server from your Mac, then sync your iPhone (or another Mac, I suppose), and then sync the original Mac again. I suspect OmniFocus does things this way to help ensure data integrity. Consequently, it's probably best to let it sync frequently so you don't get too many transactions piled up waiting to be processed. Of course, this is all speculation on my part.

In the OmniFocus for iPhone FAQ, it's recommended to keep your total number of actions under about 1,000 items for best performance. It's not clear to me, though, if this includes items that are "on hold". In other words, can I put things on hold to reduce my number of synced items to around 1,000? Also, what's the best way to find out how many items I'm syncing? Do I have to count them manually?

-Dennis

Thanks for the suggestion Toadling. Unfortunately i've synched now several times and the file is still growing. It's up to 2.1MB now. I've even done a rebuild.

Oh, and I'm definetly under 1000 tasks now (aprox 300) although I was over previously.

[QUOTE=markbrown00;40832]Thanks for the suggestion Toadling. Unfortunately i've synched now several times and the file is still growing. It's up to 2.1MB now. I've even done a rebuild.[/QUOTE]

Have you been attaching things to actions, perhaps? A 50 kB file here, a 70 kB file there, pretty soon you've got a couple of megabytes more than when you started...

No, i'm pretty sure i've got absolutely no attachments...just links to a few files here and there.

I'm seeing the same thing. My OF file jumped in size. I suspect it's from some audio files I've added when testing. Trouble is, I can't find the action where I added the audio file. Is there a way to search for them somehow? It takes my phone about 5 minutes to synch over Edge, during which time, I can't receive any calls. Can't wait for the push SDK to come out in September. I certainly hope OF utilizes it.

markbrown00: Definitely send us an email (omnifocus@omnigroup.com). Particularly if you are willing to email us your file, we should be able to figure out why it's growing.

I can top that mine is 6.1MB's. I tried to do a archive and a rebuild, with no success. in reduction. In fact the 6.1 MB is what is on the iDisk. My local copy on this machine is 9MB.

I have well under 1,000 actions (about 600) and no attachments other than what gets linked in as part of the mail to omnifocus script. Clearly some of the HTML code is pulled in from the emails.

I'm seeing this as well.

I think it may be "normal" for the new Omnifocus file format. Specifically, it looks like, to make synching much easier, that each task in OF is now represented as a file in the OF "file." Each of them is zipped. Thing is, the minimum file size with HFS+ looks to be around 4 KB, so each item takes 4 KB (minimum) of space. I have 560 .zip files in my .ofocus package file, and simple math gives about 2.2 MB for that. That's about what I have.

What I don't know is if OmniFocus will always leave the tasks this way, or if cleanup will address it at some point.

I can see that my Omnifocus file went from 132 KB on July 8th, 2008 and is now 2.4 MB in size. So the explosion of these same files is definitely responsible for it, but I don't know if it's by design or because cleanup of the sync items isn't happening along the way.

That approach of having distinct transaction files has been used by OmniFocus since before v1.0 was released back in January. File -> Rebuild Database used to coalesce those transactions into a single file, improving performance somewhat and reducing the overall size of the database file bundle.

However, there's clearly been some changes in how things are handled in v1.1, presumably in support of syncing. The transaction files are now compressed, which seems to make sense for sending them over a limited-bandwidth network. Also, using File -> Rebuild Database doesn't seem to trigger the coalescing anymore, at least not that I can see. However, I believe it still rebuilds your SQLite index that's used for searching (~/Library/Caches/com.omnigroup.OmniFocus/OmniFocusDatabase2), and maybe it checks the integrity of your database (not sure).

On my system, I can see what looks like a transaction file for every change I make on my Mac inside the database bundle. When I sync with MobileMe, the transaction files remain. But when I then sync OmniFocus on my iPhone and then sync OmniFocus on my Mac again, all those transactions look like they get coalesced into the root file again. It seems logical that you wouldn't want them to be coalesced until OmniFocus is sure they've been synced to the remote system; hence the delay.

It's been working this way for several days now and my file size is still hovering around 320 KB, which is close to what it was before I switched to the 1.1 sneakypeak over two weeks ago.

Bottom line: if your file is consistently growing and continues to do so even after multiple successful syncs with your iPhone or remote Mac, I'd get in touch with the Support Ninjas just in case.

-Dennis

For me as well a lot of these trasaction files
 

For me, it's the same problem, I got a lot of these files (about 190) and I can sync whatever I want, they won't get less ... actually, every time I do something, there's one more file.

Actually this one more file appears on all machines ...

My setup is an iPhone 3G, an iMac in my (home-) office and a MacBook Air "on the road". Is it possible, that this issue is related to syncing with more than two peers?

btw: Great job, guys. OmniFocus was the main reason, I needed to get an "official" iPhone here in switzerland as soon as possible.

I don't know if it has to do with 3+ peers, but I only have 2 (MacBook Pro and iPhone) and I haven't seen any problems.

-Dennis

I'm syncing 2 macs and an iPhone through the MobileMess, no problems here...

my database also seems to be growing. I'm at around 2.6mb and before I went to 1.1 and started to use syncing I was under 500k.

One nasty side effect is that this causes my omnifocus database on my iPhone to take FOREVER to load and/or sync

tried archiving and rebuilding, deleting all of my completed items, no help =(

Looks like a deletion problem
 

I'm not getting the synchronization process ... for me, it looks like there is my starting-content (as of July, 13th, when I started using OF 1.1) and everything else seems to be there just as a transaction file. And that's the same on my local hard disk as "in the cloud" ... actually on MobileMe.

I would have thought, that some client should clean up in the synchronization process, so that all transactions already synched by all clients were integrated into the "main" content and then deleted ... for me, all three clients were last synched today. But the transaction files were from July, 13th on.

But just as I said ... I don't get it. But I can say from my personal experience that the problem here could just as good be me.

I think, starting up could take longer and longer and longer, because OmniFocus needs to apply all transaction files to the starting-content.

But the support guys got my database and I'm sure, they can find and fix the problem. We're in an alpha stadium and I'm very happy with data growing ... that's way better than losing data.

THIS HACK IS NO LONGER NECESSARY. THE UNDERLYING BUG HAS BEEN ADDRESSED.


*Disclaimer: This is a little hackish. Proceed with caution.*

It's possible that the database has acquired more than one client file for one or more of your devices. This confuses it into thinking there's a device that hasn't synced recently, so it can't compact changes yet. Here's how to find and remove duplicate client files:

0) Sync all your devices if possible.
1) Browse to your OmniFocus file on your server in Finder.
2) Right-click -> Show package contents
3) Along with an awful lot of .zip files, there should be a few .client files. If everything's behaving correctly, there should be one for each device (computer or phone) you sync with.
If you have two .client files for the same device, OmniFocus will be confused. Eliminating any obsolete ones may help.
4) Open each of these .client files in a text editor of some sort.
5) Look at the 'HardwareModel' and 'name' keys to figure out which device each .client file is for. If you only have one for a device, you can close and ignore it.
6) Now you can use 'lastSyncDate' to figure out which is the current .client file for a device. You should be able to safely delete any other .client files for that device.
7) Now sync all your devices again, and after a couple syncs, your database should slim down if this was the problem. (You will need to sync all the devices at least once and then a mac again to get the file to compact.)

Note: If something goes wrong and you delete a .client file that you shouldn't have, the next time that device syncs, it will ask which database you want to use. If you synced in step 0, you can safely choose the server database and resolve the problem. If you were unable to sync that particular device in step 0, it's probably still best to choose the server database, but realize you'll lose any changes on that device.

With some magic, it did work ...
 

Hi Lizard,

I did a lot of looking through directories (my girlfriend is not at home, I left work early today, I'm a geek and pretty interested in OmniFocus working quite perfect because my working quality depends at least a little on that).

I am sure, there were only three client files on the server, one for every device (I got a backup of the whole directory if you need it for further investigation). They were all synched today (that was what the client files said) and there were a lot of transactions there.

And just as I wanted to check the files again directly on the server, I saw, that there were only four left ... and on my local drive, there were still all of them, but after synching again, they were magically gone on my local drive as well ... and all my tasks were still there (at least the ones, I know of).

Anyway: Many thanks for that great piece of software, I really appreciate it.

Regards
Boris

P.S.: And my iPhone startup and synching is way faster ... there are only 34 KB of 820 KB left ... I'm so happy.

lizard's tweak worked for me. my db is now 24k!!!!

That did it for me... I just synced all my machines and deleted all the client files.

I wonder if it has to do with the iPhone... I can't use the iPhone version of OF right now anyway as it crashes 100% of the time on launch. Maybe I'll delete it off the phone and re-try.

The "Lizard Tweak" cut my file size by about 20% and SIGNIFICANTLY cut the number of files in the package (by maybe 75%). I had two mysterious devices lurking in there. One seemed to correspond to when I first synced the phone and had it stop mid-way through (the device file name had "in-progress" in the title) and had to pair it again, and the other corresponded to a db reset on the iPhone a couple of days later. Users that have experienced similar issues may want to check this out.

Oh man, thanks for this thread. My .ofocus file was up to 3.3MB and it was making it completely unusable. I reset everything, and now it's just 12kb!

YES! I too was having this growing database problem on my phone and on my wife's phone. My database was almost 8MB! Hers was around 6MB.

After applying this fix, my database is 54K! Hers is 28K!

I was getting very frustrated about how long it was taking to sync my iPhone and was starting to think that there was no way I could use OmniFocus for the iPhone. Now a sync on the iPhone takes seconds instead of minutes.

Lizard, you are awesome!

I concur: although my database only went from 330 kb to 55 kb, I noticed dramatic improvements in sync performance. What causes all of this extra bloat to accumulate in the first place? I'd like to avoid it if possible, but I'd be willing to automate this process if need be.

Got my dbase from 4.1 down to 13k but not overly sure how i did it, but started with lizards tips and then rebuilding and deleting a frew times, also it seems to be more functional now since I have switched off idisk syncing, mybe that added an extar level of confusion to the mix....cheers guys, i am now totally dependant on ofocus

I'm guessing if you don't see a .client file, and your file size is still below 1MB things are hunky dory?

A couple of questions. Does it seem like since the switch to mobileme, that the idisk seems to take FOREVER to respond to things like right click and show package contents? It took five minute for the right click show package contents to respond.

Also deleting the files never responded. I had to go in and delete via the mobileme idisk web interface.

This changes everything and makes using OmniFocus much more usuable. My file had swollen up too... Perhaps Omni needs to build some sort of tool into the apps to do a clean up of the clients since there's several of us having this problem.

Grump: It definitely feels really slow on my end too. I've seen my OmniFocus become fairly unresponsive while synching in a crucial situation on a day when iDisk was particularly bad. If it happens again, I'm moving to another WebDAV server I have access to.

[QUOTE=cmmuk;41521]I'm guessing if you don't see a .client file, and your file size is still below 1MB things are hunky dory?[/QUOTE]

If you don't see ANY .client files, then you're looking in the wrong place. This fix/hack needs to be applied to the OmniFocus file on your central sync location (iDisk or other WebDAV), not the OmniFocus file in your Library.

And yes, we are working on a user interface for managing this. But it'll take a little bit of time, and thought this hack might help you wait patiently.

[QUOTE=AquaMethod;41519]I concur: although my database only went from 330 kb to 55 kb, I noticed dramatic improvements in sync performance. What causes all of this extra bloat to accumulate in the first place? I'd like to avoid it if possible, but I'd be willing to automate this process if need be.[/QUOTE]

The extra .client files are generally going to happen when you change your sync settings. For example, you switch from MobileMe to some other WebDAV and back again. So once you've got a stable solution syncing happily, this shouldn't happen. (If you do see it happening even when you're not messing with your sync settings, please send an email to [email]omnifocus@omnigroup.com[/email], so we can figure out what's gone awry.)

I was able to finally finish the process when I returned home today.

Totally fixed it for me.

Lizard for President!

[QUOTE=Lizard;41581]The extra .client files are generally going to happen when you change your sync settings. For example, you switch from MobileMe to some other WebDAV and back again. So once you've got a stable solution syncing happily, this shouldn't happen. (If you do see it happening even when you're not messing with your sync settings, please send an email to [email]omnifocus@omnigroup.com[/email], so we can figure out what's gone awry.)[/QUOTE]

Ah, that makes sense: I did bounce around between a generic WebDAV and MobileMe initially. I'll keep an eye on it. Thanks again for the tip!

This thread is sounding more and more like an advertisement for a weight loss company.

This morning I received an email from the support ninjas; they suggested I try rebuilding my database. I'd already tried this, but I gave it another go and it reduced my file from 2.1MB to 500K. Now I can run on the beach and play with my children with ease. Thanks Omni Group!

I've since looked in my server OF file and there's only 2 .client files as their should be. Still, i'm very jealous of those people with files under 50k. Being on a limited data plan (and a small one at that) I'm reluctant to sync large files.

I've spent the last few hours wrestling with things. My iDisk OF file is about 1.1MB. When I opened it a few hours ago, it had files dating back to July 10th, when I did my first sync. Presumably each of the .zip files I see (approximately 550 of them) corresponds to every change I've made on my iPhone or one of my two computers. There was a linger .client "write-in-progress" file that I deleted, but otherwise, there were no additional .client files beyond my three devices.

I did a ton of synching after that, and nothing changed. I tried compacting my database on one of my mac clients, thinking it would trigger a merge of all those changes, but that didn't help either.

Finally, I deleted the entire iDisk file and started anew. Even when I reset my synch data from my Mac to the iDisk server, it still re-creates all those 550 zip files that went back to July 10th.

Looking at my historic backups, my OF file was 56KB on July 9th. It passed 1MB on the 14th and is now up to 2.5MB. When I open these local files, I see all the changes back to July 10th. So, it looks as if the local OF file isn't merging things back into the database. This is somewhat backed up by the fact that all the change files show back up on the server when I do a reset from my local OF file. I'll e-mail this and the file to the Support Ninjas so they can have a look-see.

For people who have files under 100K, how many tasks do you have approximately?

Not exactly an answer to your question, but a data point: my OmniFocus package is currently 263.7 K, and I have 186 projects and 1,674 actions in my OmniFocus database. My package (on the local disk) contains 121 items.

Hi Chrobb,
I ran into the same problem. After deleting and the first sync, I did a second sync, rebuild database and synched again.
Now I'm at about 35K.
Regards
Boris

An Observation
 

It looks like the sync process can handle distinct changes quite perfect.

(1) New action A on client X
(2) Changed action B on client Y
(3) New action C on client Z

It get's a little fragmented as my changes are depending on each other.

(1) New action A on client X
(2) Changed action B on client Y
(3) Changed action A on client C

In this case, it looks like the Transactions are depending on each other and it's hard to decide what can be deleted. Especially if the clients are not synched after every change.

Just an observation, that might be completely wron, but I did a lot of testing the past few hours.

Hope that helps in any way.

Kind regards
Boris

Hmmm. Well, i've trimmed down to 200 tasks, and I've still got a 500k file. So something is still not quite right, even after all the rebuilds, and the 'lizard' maneuver.

Sorry about the "me too" posting, but -- yeah, me too. My file is steadily growing, even though I only have one .client file per device (three macs -- no iPhones or the like). it's now at 9.0M with over 2,196 .zip-files in the bundle. That just ain't right... (And, because iDisk does per-file-sync stuff, I guess it's one of the reasons why my iDisk syncs have slowed down?)

This behavior seems like a bug to me, no matter what the reasons (e.g., transactional integrity) are... Even though I rarely use/sync with my laptop, for example, the size of my other files shouldn't have to increase by an order of magnitude or two.

And in the interest if efficient iDisk syncing, it seems that limiting the number of files would be important too. According to the Apple docs, a high number of files is an important slowdown reason for the iDisk.

So... Anything I can do beyond the .client hack (since I only have one per device)? Sync all devices like repeatedly and rebuild like crazy? :->

An addendum: I've found one seemingly watertight (but painfully hackish) way of reducing the file size, at least. (Still haven't tested its impact on syncing... Maybe I just broke everything ;-)

On one of the machines:

1. Turn off sync
2. Export to OmniFocus file
3. Manually replace ~/Library/Application\ Support/OmniFocus/OmniFocus.ofocus with the exported file (as it seems impossible to simply import it)
4. Delete OmniFocus.ofocus from your iDisk
5. Turn on sync again

After this, you should hopefully be able to "broadcast" this new file to the other machines (or you could simply delete their files before syncing, to make sure). Haven't tried this part yet, though, as I'm still waiting for my iDisk to finish a previous sync of the OmniFocus file (which is taking *quite* some time...).

Update: I've now synced the new file with the server(and realized that I probably didn't have to wait for the iDisk -- I erroneously thought OF worked with the local copy), with some transactions added, downloaded it with another OF, synced that and then synced the first -- and now everything looks like it should. I.e., the added transactions are now consolidated with the database, which is 148K. Yay!

It remains to be seen if it will still work when I involve my third machine (at work) tomorrow... Oh, well. If not, at least now I have a reset hack when it grows too large :)

Can someone describe to a luddite like myself where I would find OF on the server. I am on Bingo disk. How do I find this file.

Usually I don't feel this "in the dark". I understood the directions that Lizard gave earlier in this thread, but don't know where to look for the server.

Thanks

I feel less Luddite-ish. After mucking around a bit I found it and ended up getting the same results as everyone else. When I first finished my synch on the desktop still took a long time. I quite OF and relaunched. After that my syncs are back to the good old days.

In my case though, I don't recall messing with the settings once I was successfully synching. It will be interesting to watch and see if performance stays down in the seconds range.

Thanks Lizard. OF feels much more useful again.

My DB was only 1 MB in size but I only have about 20 line items to do. After totally deleting out the file, and reverting to the back up, deleting the .client files (even though I had the appropiate number), and sync'ing, it went down to 16 KB. It went up to 20 KB after one action was updated.

Does cleanup not reduce the file size? Personally, I want a button I can click to remove the old and compact the size. Delete forever would make for a nice option to manually perform every once in a while.

Is This Fix Obsolete?
 

I suspect that the Omni guys changed something fundamental in the way the sync process works. Here's why:

A few days ago, I was having problems with my data file growing larger and larger. I applied the [URL="http://forums.omnigroup.com/showpost.php?p=41378&postcount=17"]"Lizard fix"[/URL], and my data file went from over 8MB to 56K. Then within a day it went up to around 300K, but stayed steady at that size.

Then yesterday, I upgraded to the most recent sneaky peak, and noticed in the release notes this sentence "Fixed problems with syncing multiple computers or devices at the same time against the same iDisk or WebDAV server." Hmmm... I thought.

And then I noticed that all of my clients (2 computers, 1 iPhone) wanted to revert their databases to a server or local copy. Thankfully, I had a good data copy on the server and so I didn't lose anything.

Well, then I checked out the package contents on the server and noticed a whole bunch of .client files. At first, I was alarmed... but I trust the Omni guys, so I kept an eye on it. And my file IS NOT growing, in fact, it has shrunk to 65K.

Lizard, could you give any insight into this? Should I be concerned that there are now a whole bunch of .client files in my OF database? It seems like the size is just fine, so I just want to make sure.

[QUOTE=beharkey;42053]Lizard, could you give any insight into this? Should I be concerned that there are now a whole bunch of .client files in my OF database? It seems like the size is just fine, so I just want to make sure.[/QUOTE]

I'm seeing the same thing on my iDisk: multiple .client files for my Mac OmniFocus database. However, there appears to be only one .client file for my iPhone (those are the only two clients I have syncing).

Despite the multiple .client files, everything seems to be syncing properly without a hitch.

-Dennis

Hello

I have a temp ( hope based ) .mobileme account. Your instructions are I assume to browse the iDisk for OM file to examine package contents is that correct, if so I do not find anything, if assumption is wrong kindly point me towards the correct "server" for me to examine.

Terence

The OmniFocus file if you're using MobileMe is in your Documents folder on the iDisk.

I've verified that's where it is on mine... you need to look for it harder ;-)

[QUOTE=Terence Dodge;42065]Your instructions are I assume to browse the iDisk for OM file to examine package contents is that correct, if so I do not find anything, if assumption is wrong kindly point me towards the correct "server" for me to examine.[/QUOTE]

Yes, in the Finder, select iDisk in your sidebar and go to Documents/OmniFocus.ofocus, then right-click on the file and select the "Show Package Contents" option. This will open a new Finder window revealing the database file, all transaction files, and multiple .client files if you've been syncing.

I personally find it easier to view my database on my iDisk using Panic's Transmit, but Finder works fine too.

-Dennis

[QUOTE=Toadling;42064]I'm seeing the same thing on my iDisk: multiple .client files for my Mac OmniFocus database.[/QUOTE]

Watching this closely over the past hour or so, it seems like the .zip and .client files are cleaned up after a delay. I now have fewer .client files than I had 30 minutes ago.

I love hearing about the inner workings of OmniFocus and the sync process, so if any Omni Group people want to shed some light on this, I'm all ears. :)

-Dennis

Toadling - I am seeing exactly the same behavior.

Syncing is working just fine. The .client files seem to be "flexible".

[QUOTE=beharkey;42098]Toadling - I am seeing exactly the same behavior.[/QUOTE]

Thanks, beharkey. It's good to hear some confirmation. I just tried syncing some more items and everything seems to be running smoothly here as well.

-Dennis

Had same issue
 

I had the same issue.
I also solved it using the same method.

Hope there will be a automatic way later on...

[QUOTE=Toadling;42068]Yes, in the Finder, select iDisk in your sidebar and go to Documents/OmniFocus.ofocus, then right-click on the file and select the "Show Package Contents" option. This will open a new Finder window revealing the database file, all transaction files, and multiple .client files if you've been syncing.

I personally find it easier to view my database on my iDisk using Panic's Transmit, but Finder works fine too.

-Dennis[/QUOTE]

I no longer can find any .zip files or .client files on my iDisk.

After I go to the Finder and select iDisk in the sidebar and go to Documents/OmniFocus.ofocus I do not see an option for "Show Package Contents". My documents file on the idisk only contains the OmniFocus.ofocus file. How do I get to the "Show Package Contents" option? When I click on the OmniFocus.ofocus file, a Preview screen appears that has a button for "more info" but within "more info" there is not an option for "Show Package Contents". I am running 10.4.11 not 10.5 yet - does that make a difference?

What I have done twice so far to get my database size down to allow the app to open on my iPhone is delete the entire OmniFocus.ofocus from my iDisk and resynch it back from my Mac which brought it from 1.8MB down to 168k this time and then deleted the app from my phone and reloaded the app back to the phone from iTunes and copied the database from my Mac back onto my phone.

The first time I did this (about a week ago), I remember when I went into documents on my iDisk there were a bunch of .zip files (and maybe .client files too, I don't remember). I deleted both the OmniFocus.ofocus as well as all of the .zip and any other Omnifocus related files in documents. Since then, all I've seen in my iDisk is the OmniFocus.ofocus file. Thus today, all I deleted from my iDisk is the OmniFocus.ofocus file.

Although my delete and resynch of my iDisk and phone successfully reduced my database size and now enable me to open OF on my phone again, as they did the last time, I was hoping to find the .client files so I can delete any extra because my fix seems to make the database smaller but not keep it from growing unnecessarily again.

[QUOTE=jt32;42131]I no longer can find any .zip files or .client files on my iDisk.

After I go to the Finder and select iDisk in the sidebar and go to Documents/OmniFocus.ofocus I do not see an option for "Show Package Contents". My documents file on the idisk only contains the OmniFocus.ofocus file. How do I get to the "Show Package Contents" option? When I click on the OmniFocus.ofocus file, a Preview screen appears that has a button for "more info" but within "more info" there is not an option for "Show Package Contents". I am running 10.4.11 not 10.5 yet - does that make a difference?

What I have done twice so far to get my database size down to allow the app to open on my iPhone is delete the entire OmniFocus.ofocus from my iDisk and resynch it back from my Mac which brought it from 1.8MB down to 168k this time and then deleted the app from my phone and reloaded the app back to the phone from iTunes and copied the database from my Mac back onto my phone.

The first time I did this (about a week ago), I remember when I went into documents on my iDisk there were a bunch of .zip files (and maybe .client files too, I don't remember). I deleted both the OmniFocus.ofocus as well as all of the .zip and any other Omnifocus related files in documents. Since then, all I've seen in my iDisk is the OmniFocus.ofocus file. Thus today, all I deleted from my iDisk is the OmniFocus.ofocus file.

Although my delete and resynch of my iDisk and phone successfully reduced my database size and now enable me to open OF on my phone again, as they did the last time, I was hoping to find the .client files so I can delete any extra because my fix seems to make the database smaller but not keep it from growing unnecessarily again.[/QUOTE]

right click or ctrl+click the .ofocus file to get the show package contents.

[QUOTE=jt32;42131]After I go to the Finder and select iDisk in the sidebar and go to Documents/OmniFocus.ofocus I do not see an option for "Show Package Contents"... I am running 10.4.11 not 10.5 yet - does that make a difference?[/QUOTE]

No, it shouldn't matter if you're running 10.4.x or 10.5.x. What you need to do is right-click or Control-click your OmniFocus.ofocus file. In the resulting contextual menu you'll see an option to "Show Package Contents". In the new window that opens, you should see one zip file that starts with a bunch of zeros. That's your root document, the core of your database. In addition to that, you might see a variety of zip files for any recent transactions. You will also likely see .client files for each client that syncs with your database.

[QUOTE=jt32;42131]...I was hoping to find the .client files so I can delete any extra because my fix seems to make the database smaller but not keep it from growing unnecessarily again.[/QUOTE]

The first thing to do is make sure you're running the most recent sneakypeak build (Revision 103163 as of 7/22/08 3:18 PM Pacific). Once you're up-to-date, there should't be anymore reason to delete old .client files, at least until we hear more from the Omni support people.

I would strongly discourage anyone from going into the database package and deleting files willy-nilly unless you know [I]exactly[/I] what you are doing or you're willing to accept the consequences of possible annihilation of the entire human civilization, and maybe some data loss too.

I suspect Lizard's advise was accurate a few days ago, but may not be anymore. With the latest build, at least on my machine (and beharkey's too, from the sound of it), all old zip transaction files [I]and[/I] .client files are getting cleaned up by the app after a short delay of about an hour or so.

Summary:
[LIST=1][*]Update to the latest sneakypeak

[*]Wait at least an hour before you do anything! Maybe take a screenshot of the file listing so it's easier to compare later.

[*]If you start deleting things inside the database package, be sure to [URL="http://www.youtube.com/watch?v=C0K_LZDXp0I"]duck and cover[/URL].[/LIST]
Hope this helps. And everybody... let's be safe out there.

-Dennis

If all your devices have the latest sneaky peek, this .client hack should be unnecessary. But if any of your devices are iPhones, they won't have this fix yet. So it may still resolve some issues. And Toadling is right that you should be careful.

mmmmm I just deleted my .client files, again because my file was growing. They are not back. So definitely be careful because it has caused a MobileMe sync conflict every time my iDisk syncs. I am going to start over since my file was small anyway.

Duck and cover is good advice. ;)

[QUOTE=Lizard;42139]If all your devices have the latest sneaky peek, this .client hack should be unnecessary. But if any of your devices are iPhones, they won't have this fix yet. So it may still resolve some issues. And Toadling is right that you should be careful.[/QUOTE]

I hope the iPhone version is coming soon because my 30 action file is 280 KB after starting out at 20 KB this evening. My iPhone is blowing this thing up like a hot air ballon. :D

[QUOTE=Lizard;42139]If all your devices have the latest sneaky peek, this .client hack should be unnecessary. But if any of your devices are iPhones, they won't have this fix yet. So it may still resolve some issues. And Toadling is right that you should be careful.[/QUOTE]

My DB is up to 360 KB. Since I have an iPhone, do you have a suggestion to keep the size small until an update for the iPhone comes out?

If the problem is .client files, that should only be happening (as far as I know) when you change your sync settings. So leaving your sync settings alone should be enough to resolve it.

If the problem is not .client files, then 1.0.2 may not fix it either. If you're willing to let us peek at your data, emailing it to us will probably help us figure out why it's swelling.

dude, I'm syncing both a MacBook Pro and an iPhone, and I'm not seeing any problems syncing with my iPhone - no database bloat, no stranded .client files, no real problems at all.

Here are some specifics: ~320 KB database file, ~2500 actions, syncing through MobileMe, multiple client and zip files appear on server but get cleaned up (by OmniFocus on my Mac, I guess), only one client file appears from my iPhone, most of my changes are initiated on my Mac but some come from my iPhone.

So I don't really have any advice other than it is possible to get the system working. I didn't do anything special; it just all worked.

-Dennis

PS - My database size does fluctuate by about 100 KB when I make lots of changes, but it has been returning to around 300 KB after about an hour or so. It seems like things take about that long to get cleaned up, maybe from the automatic syncing, which happens hourly I believe.

When you say your Mac cleans everything up, what do you mean exactly.

I have 24 .client files and have only sync'd my iMac, MacBook, and iPhone. Both machines have the latest build along with my iPhone having the latest build for it. I am wondering what the freek is going on since I just started over tonight with a brand new DB and have less than 30 actions.

For instance, my Mac just sync'd. The DB went from 360 KB to 304 KB. Client files reduce by ~16 down to 8 or so. This will continue to improve as my iMac sync's? What if my MB doesn't sync for a few days? Will it hold or bloat again if my iPhone resyncs?

Just trying to understand how this works.

[QUOTE=Lizard;42209]If the problem is .client files, that should only be happening (as far as I know) when you change your sync settings. So leaving your sync settings alone should be enough to resolve it.

If the problem is not .client files, then 1.0.2 may not fix it either. If you're willing to let us peek at your data, emailing it to us will probably help us figure out why it's swelling.[/QUOTE]

You will have my file in the morning. Would manually syncing cause problems?

dude, make extra sure your iMac and MacBook are running at least OmniFocus 1.1 build 77.7.0.103163

Here's a rough sketch of why the extra .client files cause bloat:
One file containing 10 changes takes up less space on many file systems than 10 separate files. And if one of those changes cancels another out (change a due date to tomorrow and back to today), they can be eliminated entirely.

But those compaction strategies will only cause trouble if another OmniFocus client knows about some changes but not the others. So each client writes a .client file, which tells all the other clients what the last transaction it saw was. If one of the apps goofs up (or the sync process gets interrupted) and leaves an old .client file around, all the other clients will think there's still someone who hasn't gotten the recent changes. So they just keep adding new changes and no one can compact all the old transactions for fear of confusing this ghost client.

OmniFocus on the iPhone doesn't do any of these transaction compactions because they're a little more processor intensive and the iPhone's connection to the WebDAV server is more likely to get interrupted than a Mac client.

thanks for the explanation. I am running the latest builds for sure. I will recheck in the morning.

[QUOTE=dude;42211]When you say your Mac cleans everything up, what do you mean exactly... I have 24 .client files and have only sync'd my iMac, MacBook, and iPhone.[/QUOTE]

I think every time one of your Macs sync with the server, a new client file is created, uniquely-named with a time stamp. This seems to happen whether there are outstanding changes or not. Also, a collection of zipped files gets copied to the server, one for each transaction with your database. Finally, it looks like the Mac client removes any old client files (older than an hour?) and merges any zip files that have already been synced to all clients.

Please note that I really have no idea what I'm talking about. The above paragraph is pure speculation based on what I've observed on my system. The Omni Group could very easily step up and tell me I'm completely wrong. :)

If I'm right, though, there's no longer any need to worry about excess client or zip files on the server, at least not ones created by your Mac clients. It looks like the ones that get created now are being done on purpose and get cleaned up in time.

I gather from Lizard's post that the iPhone client hasn't yet adopted this new syncing approach. It still creates only a single client file to identify itself (rather than a new client file for each instance of a sync session like the Mac client does). But unless you've had some kind of a mix up in setting up syncing with your iPhone, this shouldn't really be problem either. One client file gets created for each iPhone and it stays that way, at least for now.

Hope this helps.

-Dennis

UPDATE - I started writing my post before I saw Lizard's response above. Now I'm not clear whether the Mac client is supposed to create a new client file on the server with every sync or not. It sure looks like mine is doing that now, but maybe that's not right. However, I can also see that the old client files are getting cleaned up; after waiting an hour or so, there are a lot fewer of them and my database size goes back down to around its original size.

My understanding of the situation may be slightly out of date too.

Thanks Dude & Toadling for the walkthrough on how to see what is in "Show Package Contents". Luckily I only saw one the two .client files I was supponsed to have, one for my Mac and one for my iPhone.
[QUOTE=Toadling;42135]
I would strongly discourage anyone from going into the database package and deleting files willy-nilly unless you know [I]exactly[/I] what you are doing or you're willing to accept the consequences of possible annihilation of the entire human civilization, and maybe some data loss
[/QUOTE]
I definitely never went into this database package before so I'm pretty sure I never deleted anything from there.

Should I be concerned if I, as a final step procedure when my iPhone app becomes unable to load and after backing up my database, delete my Omnifocus.ofocus document on the iDisk and then sync my Mac database again onto the iDisk, then delete the database from my iPhone, and then reload this new smaller database from the iPhone.

Otherwise, the database seems to get too big on the IDisk to open on the iPhone and this is the only fix I've found.

Hopefully the new versions of OF for both the Mac and the IPhone will fix the mysteriously growing database problem, however, as of yesterday no such luck yet.

[QUOTE=jt32;42343]Should I be concerned if I, as a final step procedure when my iPhone app becomes unable to load and after backing up my database, delete my Omnifocus.ofocus document on the iDisk and then sync my Mac database again onto the iDisk, then delete the database from my iPhone, and then reload this new smaller database from the iPhone.[/QUOTE]

It sounds like you're essentially starting syncing over from scratch, as if you had just gotten a new iPhone and were setting it up for the first time.

I don't think this is really a problem. But I wonder what happens in this scenario:
[LIST=1][*]You have multiple transaction files in the OmniFocus database on your Mac that have been merged with the Mac client but not with other clients.
[*]You turn off syncing and delete OmniFocus from your server and from your iPhone.
[*]You then turn syncing back on and everything gets copied to the server.
[*]Then reinstall OmniFocus on your iPhone and point it to your server to sync the database.[/LIST]
What happens to those leftover transaction files? Presumably, the server and the iPhone both have the complete database copied from your Mac, so none of the clients need those leftover transaction files. Do they get cleaned up automatically even if they don't need to be merged anywhere?

All of this discussion is really moot since it's based entirely on speculation. There simply isn't enough information to make a decision or give advice with any real confidence. Consequently, it's risky to make assumptions and start deleting files, especially inside the database package (unless someone from the Omni Group says it's OK).

This is Omni's domain, they're the only ones who really know what's going on, and they're clearly still making changes under the hood. I think the best advice is to simply sit tight and wait for some authoritative response from the Omni Group. That might not be what you want to hear, but that's the reality of beta testing. ;-)

-Dennis

Running latest SP 1.1 and iOF 1.01

I deleted extra .client files for both mac and iphone and reduced file size to 75k (from 335 k), and also rebuilt database.

Every time I sync my mac, I'm generating a new .client file. (Same name, with differing date/time stamp appended at beginning. )

On the other hand, syncing my iPhone is not generating a new .client file -- the existing one just gets a more recent modified time.

From the cumulative prior discussion, it sounds like the mac is supposed to (or then again maybe not) be generating multiple .client files, is that correct?

- Art

Art,

This is actually now the default behavior of OmniFocus. The desktop clients produce multiple client files. The iPhone only produces one. You shouldn't have to worry about the "mysteriously growing .ofocus file" now.

By the way, this is new since about yesterday. The Omni guys managed to work some mojo on the way that OF handles it's database.

The only exception I can think of is if you change where you are syncing to from one server to another. That might cause some weird junk to happen.

I don't know all the details yet, but the additional client files are part of a fix to avoid clashes (resulting in invalid / misleading warnings) when multiple devices (two Macs, a Mac and an iPhone, etc) are syncing at the same time.

Since the fix is new, it's not in 1.0.1, which is the most recent version for the iPhone, and thus the warnings, at least for the common case, should be gone if you're just syncing between Macs, but you'll still see the warnings (on both Mac an iPhone/iPod, although probably less frequently on the Mac) until 1.0.2 (sorry, no timeframe yet) is available for the phone.

I'm using 1.1 SP 77.7.0.103301. I'm not synching with anything (no iPhone yet!).

My OF file used to be very small. Now, with 1.1 SP (all versions) it's sized at 6.2 Mb. While this size is not a problem per se (nothing is running slow), it seems a lot larger than it should be. This may cause a problem when i do get my iPhone.

Is there anything I can do at this point? Would it be helpful to send in my file to someone at OG?

mitchellm:
One way to try and get your file size down if you're not syncing is to Back Up Database and then Revert to Database Backup (both are in the File menu). Make sure to pick the backup that you just created!

If that doesn't help, we'd be happy to take a look at your file. You can email it to [email]omnifocus@omnigroup.com[/email]

Lizard,

Nope that didn't seem to help. I have sent the file via the email address you provided. Hopefully there will be some clues as the what's going on.

Lizard: I followed your advice from another thread. It reduced my file from 6.7 Mb (and growing) to 56k! Yikes. That solves it. I guess this is a bug that will get fixed, but it's good to know the solution. This is what you advised on the other thread based on a known OF bug:

I checked in with the engineers. There's a bug that keeps the deleted files from getting removed from the database. Fortunately, you can force OmniFocus to clean up after itself.

1. In OmniFocus, choose Export from the File menu.
2. Choose a save location other than your usual backup location, so you can find this file again easily in a minute.
3. For file format, choose "OmniFocus Backup Document"
4. Save
5. Choose "Revert to Database Backup" from the File menu
6. Browse to the file you just saved in step 4 and click Open

still getting multiple .client files per machine
 

I tried deleting the extra .client files for each machine I sync, but they kept reappearing after syncs. I decided to take a more drastic step (although simpler than looking through the .client files to figure out which is which). I first backedup up my OF database on my desktop machine. I then deleted the OmniFocus.ofocus file on MobileMe and then resynced on each machine. I then had one .client file for each machine. After a few more syncs, I again had multiple .sync files per machine (except for myiPhone which has never had more than one .client file). Any suggestions?

I sent this via Feedback also.

Dave

Dave,

The fix in this thread doesn't apply anymore. OmniFocus (desktop) is now SUPPOSED to create multiple .client files. You don't need to worry about them.

Lizard - people are going to continue to read this thread and try to apply the "Lizard Fix", which is now obsolete. Should this thread be closed?

Thanks for the info beharkey. I'm glad to know there's not a problem with my setup. So every time I sync I'm supposed to get another .client file? I realized they're only about 1K a piece, but it seems strange to keep all of them. Do old ones ever go away automatically?

I just noticed my sync times getting longer on my iPhone so I was trying to reduce that.

Dave

It appears that OF desktop currently creates a new .client file each time you sync. After a while, the old ones are automatically deleted.

The iPhone should have only 1 .client file. If you have more than one iPhone .client file, then you could follow the instructions in the "Lizard Fix".

However, the Omni guys have indicated that a future version of OF iPhone will behave just like the desktop app - creating more than 1 .client file.

I think the real key is the total size of your database file. If it is continuing to grow or if it is in the MB size range, then you probably have a problem. For instance, I have in the neighborhood of 600-700 actions in OF and my file hovers around 172K. My wife has a similar number of actions and her file hovers around 191K. At times, both of these files are actually quite a bit smaller (like 40-60K).

Yes, with the current code on the desktop (and due on the iphone in 1.0.2), multiple client files can exist for a single client. This is in order to help us avoid thinking that the database has been reset when in actuality the issue is that another client is in the middle of updating the sync database.

These extra client files are removed in subsequent syncs (from the same client) if they are too old (I think they get removed if they are over an hour old, but that timeframe is subject to change).


As an aside, if your files are getting really large, there are a couple of things I can think of offhand to check. First, make sure that you don't have clients that are too far behind on syncing. We have to leave transaction info around for the least-recent client, so that it has all the information it needs to merge changes. If this is the issue, getting all of your clients in sync will result in all the old transaction files getting cleaned up. (Unless there is data corruption, in which case your best solution is probably to get a known-good desktop client, do a database backup via File -> Back Up Database, then restore from that database via File -> Revert to Database Backup, then sync and push your local database to the server, then have any other clients sync and reset to use the server data.)

Second, be aware of what kind of data you are putting into your database. If you are making voice notes, for instance, those are going to take up a good bit of space, especially if you are still using 1.0 on the phone as we weren't compressing the data as intended at that point. Also, while when you drag in files we default to making a link, you can defeat that by option-dragging, which will embed the file, likely bloating your database. One customer also reported that they were dragging in files which were getting embedded, but it turned out that they were dragging the files in from another application which wasn't providing us the (normal filesystem) link information, they were providing the entire file on the pasteboard, so it was all getting embedded.