OmniFocus Data Storage

[ueila]

I tried to backup the .ofocus files to a Thecus N4100 NAS server on my network but found it was not possible and kept getting an error message about file length.

""Can't copy the file package
"/Users/username/Library/Application Support/OmniFocus/OmniFocus.ofocus" to:
"/Volumes/username:password/d_macs/omnifocus/OmniFocus.ofocus"
Error: (-37) Bad item name.
Would you skip and continue?""

I thought perhaps if I changed the file extention to .ofo I could back them up and then rename them on the server. Renaming surprised me by showing the .ofocus data file as a folder containing 336 .xml files.

I seem to remember another GTD app recently switched to SQL for storing data and the explanation for the change was potential corruption problems with .xml.

This creates several questions in my mind. Is OmniFocus data storage .xml AND do Omni Group feel it is a reliable way to store data AND is this only a temporary system in place for the BETA?

ueila

PS: I'm no expert just a user so please keep any replies simple.

[SpiralOcean]

Isn't just about every preference, data file in OSX XML based? Seems pretty stable to me.

[Ken Case]

I thought perhaps if I changed the file extention to .ofo I could back them up and then rename them on the server. Renaming surprised me by showing the .ofocus data file as a folder containing 336 .xml files.

This is probably a little technical, but for the curious: OmniFocus uses compressed XML transaction files to store its data, with a SQL cache for efficient access. (Each time you update the application, we rebuild the SQL cache to ensure that it's consistent with the latest schema.)

I seem to remember another GTD app recently switched to SQL for storing data and the explanation for the change was potential corruption problems with .xml.

This creates several questions in my mind. Is OmniFocus data storage .xml AND do Omni Group feel it is a reliable way to store data AND is this only a temporary system in place for the BETA?

XML is an extensible text file format. The only way to corrupt XML is to write or read it incorrectly (the most common ways being to incorrectly format the values when writing or to incorrectly interpret those escaped values when reading) or to have the underlying disk storage go bad (which has nothing to do with the file format itself).

Structured text formats are the basis of most modern Internet protocols (such as web pages), and we use them as the primary document format for all our applications. Because they're text, technically oriented users can open them up and see what's changing and whether the values being written are correct (while XML can't be corrupt if you're writing and reading it correctly, the data you're trying to store in XML might not be correct), and they are designed to be very flexible, making them easy to extend as needs change. (They also have the very nice property of being easy to read on all systems, so your data is never held hostage by a particular application.)

Before we got the XML format in place, we were using a CoreData SQL database ourselves, but that's a very bad idea for primary data storage in an evolving application (and what application doesn't evolve?). Migrating from SQL to an XML file format was our biggest priority before releasing the application to the pre-beta audience of users (as we mentioned on our blog (http://blog.omnigroup.com/2007/04/30/ethan-schoonovers-omnifocus-overview-video/)), because we wanted to be sure we had a file format that would continue to work with future versions of OmniFocus without losing any data.

Apple's Mail application uses a CoreData SQL database exactly as we do: the important data is actually stored in extensible text files, the SQL database is just used for efficient access to the data in those files. This means that if something goes awry with the SQL database (or if you need to change your SQL structure), you can just throw out the SQL and rebuild it from the text files.

Why is this a big deal? I apologize if I'm getting too technical, but all SQL databases have a fixed schema structure (SQL stands for "Structured Query Language"). As long as you keep the same structure you're fine, but as soon as you need to change that structure (e.g. to add support for repeating tasks) you're faced with a data migration problem, where you need to read all of the data in the old format and write it out in the new format. This isn't so bad when you control the database and the software and can make sure you migrate the database to the new format immediately as you update the software, but is very awkward when you don't know how old a database might be. (You end up having to write code to migrate from every possible earlier version of the database to the latest one, and all of that code has the potential for bugs that corrupt the data—possibly permanently, unless you're very careful about backups.)

(Sorry if this got too long-winded. For most of our first decade, Omni's business was based on building custom SQL database systems for large corporations (William Morris (http://www.wma.com/), AT&T Wireless (http://en.wikipedia.org/wiki/AT&T_Wireless), Standard & Poor's, etc.). During those years we also did quite a bit of work with extensible text formats, building one of the first web browsers in 1994 (the web is based around an extensible text format, or we would have a lot of trouble viewing those 1994 web pages today) and updating Lighthouse Design (http://en.wikipedia.org/wiki/Lighthouse_Design)'s applications to use extensible text formats rather than proprietary binary formats—so we have quite a bit of expertise in this area.)

[jem]

""Can't copy the file package
"/Users/username/Library/Application Support/OmniFocus/OmniFocus.ofocus" to:
"/Volumes/username:password/d_macs/omnifocus/OmniFocus.ofocus"
Error: (-37) Bad item name.
Would you skip and continue?""


Could the problem be the ':' in the file name?

[BwanaZulia]

Ken... very cool reply and not too long all.

I would actually be interested in more information.

BZ

[Ken Case]

(Thanks, jem, for the reminder about the original problem.)

I tried to backup the .ofocus files to a Thecus N4100 NAS server on my network but found it was not possible and kept getting an error message about file length.

What software are you using to make the backup? That doesn't look like a message the Finder would produce (it used slashes for the path separator while the Finder generally uses colons), so I'm assuming some other tool is involved?

I would actually be interested in more information.

Anything in particular you're curious about?

One obvious curiosity that comes to mind is: why are we writing our XML as separate transactions rather than in one big file the way we usually do (in OmniGraffle, OmniOutliner, and OmniPlan)?

Well, I talked a bit about the advantages of XML, but I left out a big disadvantage: they're not very efficient to read or update. To read an XML document, you have to read the entire contents of the file (and there's no easy way to skip parts you aren't interested in). To update them, you have to rewrite the entire contents of the file. A big advantage to SQL is that you can read using a query which gives you just the results you're interested in, and you can do an update which touches only the records you've changed without rewriting everything else.

But another downside to SQL (besides the extensibility problem) is that even though you can write a single record quickly and efficiently, an incremental backup (such as the backups Time Machine will make in Leopard) has to make a complete copy of your entire database even if you only changed one record.

We wanted OmniFocus to be able to read and update your data quickly, so that we'd have fast launch times and fast save times—particularly because (unlike our other applications) we wanted to automatically and immediately save every edit. This suggests we should use SQL. But we also wanted our data format to be extensible and efficient to back up, which suggests we use XML.

So we built a hybrid system which uses a SQL database to handle the reads and updates efficiently, while recording each database operation as an XML transaction for extensibility and efficient incremental backups (the backup system only has to record the new transactions, not the entire database).

[jlindeman]

I definitely appreciate the in-depth technical info. It is nice to hear the thought that went into the selection of the data format. I like the fact that data store isn't a monolithic sqlite db.

I use (and love) Yojimbo, but uploading my 500MB db to my online backup is a bit of a pain. It looks like an rsync of the OF datastore would be much easier.

[ueila]

Ken thanks for taking time to post such a comprehensive reply.

What software are you using to make the backup? That doesn't look like a message the Finder would produce (it used slashes for the path separator while the Finder generally uses colons), so I'm assuming some other tool is involved?

I tried to copy the file from a Windows 2003 Server to a Thecus N4100 NAS with a Terrabyte RAID array using FoldersSynchronizer 3.6.1 and also ChronoSync 3.3.5. I'd be grateful for any suggestions.

ueila

[SpiralOcean]

an incremental backup (such as the backups Time Machine will make in Leopard) has to make a complete copy of your entire database even if you only changed one record.

We wanted OmniFocus to be able to read and update your data quickly, so that we'd have fast launch times and fast save times—particularly because (unlike our other applications) we wanted to automatically and immediately save every edit. This suggests we should use SQL. But we also wanted our data format to be extensible and efficient to back up, which suggests we use XML.


Now that's just freakin' cool! Nice job on doing it right from the ground up. I did notice that I never had to save anything. It makes me feel extremely safe that I won't loose a lot of data if something happens to the computer or application.

[Andrew]

uelia, I take it that other files copy correctly? I'm not very familiar with the Thecus NAS, but the "username:password" portion of the destination path looks suspicious to me. I don't know the mount options, but are you sure that the network mount is correct? If you're using Samba/CIFS, can you try AFP instead?

Alternatively, the symptom doesn't really seem to indicate this, but since the .ofocus file is actually a folder containing multiple files, one possibility is that the problem is with the name of one of the constituent files.

Once you renamed the .ofocus file, causing Finder to reveal it as a folder, did you try to copy that folder to your server? If the problem has to do with one (or more) of the constituent files, I would expect Finder to succeed with copying some files, then get stuck with the problem file, and give you an error message that includes the name of the problem file. That name might be enough to determine the problem.

[ueila]

I take it that other files copy correctly?
Yes this is the first time I have seen this problem.

I'm not very familiar with the Thecus NAS, but the "username:password" portion of the destination path looks suspicious to me.
Both source and destination drives were already mounted

Alternatively, the symptom doesn't really seem to indicate this, but since the .ofocus file is actually a folder containing multiple files, one possibility is that the problem is with the name of one of the constituent files.
That could be the case and I will go back now that Ken has commented and try to back up just the 336 separate files and see what happens.

Once you renamed the .ofocus file, causing Finder to reveal it as a folder, did you try to copy that folder to your server?
No. See above.

If the problem has to do with one (or more) of the constituent files, I would expect Finder to succeed with copying some files, then get stuck with the problem file, and give you an error message that includes the name of the problem file. That name might be enough to determine the problem.
I'll let you know how it goes when I have time to investigate further.

Thanks for your suggestions.

ueila

[jelmore]

"/Volumes/username:password/d_macs/omnifocus/OmniFocus.ofocus"
Error: (-37) Bad item name.
Would you skip and continue?"

To me, the "username:password" looks like you are trying to prepend login information for a server share; that typically takes the form of:

username:password@hostname

Maybe replacing the "/" between "password" and "d_macs" with an ampersand would work?

But then again, you're referencing it as a pathname from /Volumes, so that might not work. Can you access the server share using an SMB URL? Something like:

smb://username:password@d_macs/omnifocus/OmniFocus.ofocus

[ueila]

Just to finish this thread I have tried drag'n'drop between two finder windows and that does not work either.

The file that gives the problem is OmniFocus~.ofocus and the error message is "You cannot copy the item "root=CA6138A6-7E8A-4A43-B40F-7AFDFF25AC/1E7B9A01-0DA4-48E1-B510-018FD631AD7D.xml@ because the name is too long or includes characters that the disk cannot display." This is a drag'n'drop from an iMac HD to a Server.

Trying to drag an OmniFocus backup like OmniFocus 2007-05-23 065126.ofocus produces the same error message.

If other users are seeing the same problem if makes me wonder how we are going to backup to Windows 2003 servers. There is no problem backing up to external HDs connected by firewire.

ueila

[Ken Case]

The file that gives the problem is OmniFocus~.ofocus and the error message is "You cannot copy the item "root=CA6138A6-7E8A-4A43-B40F-7AFDFF25AC/1E7B9A01-0DA4-48E1-B510-018FD631AD7D.xml@ because the name is too long or includes characters that the disk cannot display."

Hmm, I wonder whether it's tripping over the long filename or the slash? I guess I could believe that either one might be tripping up the Windows server. (You could test this by editing the "root=..." filename to use a space instead of a slash. OmniFocus won't be able to read it that way, but it would at least let us know whether that's the problem.)

If other users are seeing the same problem if makes me wonder how we are going to backup to Windows 2003 servers. There is no problem backing up to external HDs connected by firewire.

One approach might be to make an archive of the file and back that up instead: in the Finder, select the OmniFocus.ofocus file and then select "Create Archive of ..." from either the File menu or context menu.

[kunicki]

Ken, where is the SQL cache stored? From a programming perspective would there be any problem in just reading the data into another application when OmniFocus is running?

[Ken Case]

Ken, where is the SQL cache stored?

It's in ~/Library/Caches/com.omnigroup.OmniFocus.

From a programming perspective would there be any problem in just reading the data into another application when OmniFocus is running?

It's not going to upset OmniFocus if you read that sqlite database, but your code will be more fragile if you're relying on the schema of our cache (which could change from one build to the next). I'd definitely recommend using our AppleScript interfaces instead. (Bear in mind that you don't actually have to code in AppleScript to use those interfaces: you can also get at them from ruby or python using their scripting bridges.)

[GeekLady]

I'm sorry if this is a terribly snoopy and/or stupid question, but was it difficult to create this type of hybrid database and did you do it with Xcode?

[sprugman]

FWIW, I ran into the same file name problem last night just trying to back-up my OF file from my MacBook to my PowerBook. Very similar error message to the "root=..." one above.