[ueila]

I tried to backup the .ofocus files to a Thecus N4100 NAS server on my network but found it was not possible and kept getting an error message about file length.

""Can't copy the file package
"/Users/username/Library/Application Support/OmniFocus/OmniFocus.ofocus" to:
"/Volumes/username:password/d_macs/omnifocus/OmniFocus.ofocus"
Error: (-37) Bad item name.
Would you skip and continue?""

I thought perhaps if I changed the file extention to .ofo I could back them up and then rename them on the server. Renaming surprised me by showing the .ofocus data file as a folder containing 336 .xml files.

I seem to remember another GTD app recently switched to SQL for storing data and the explanation for the change was potential corruption problems with .xml.

This creates several questions in my mind. Is OmniFocus data storage .xml AND do Omni Group feel it is a reliable way to store data AND is this only a temporary system in place for the BETA?

ueila

PS: I'm no expert just a user so please keep any replies simple.

[SpiralOcean]

Isn't just about every preference, data file in OSX XML based? Seems pretty stable to me.

[Ken Case]

Originally Posted by ueila

I thought perhaps if I changed the file extention to .ofo I could back them up and then rename them on the server. Renaming surprised me by showing the .ofocus data file as a folder containing 336 .xml files.

This is probably a little technical, but for the curious: OmniFocus uses compressed XML transaction files to store its data, with a SQL cache for efficient access. (Each time you update the application, we rebuild the SQL cache to ensure that it's consistent with the latest schema.)

Originally Posted by ueila

I seem to remember another GTD app recently switched to SQL for storing data and the explanation for the change was potential corruption problems with .xml.

This creates several questions in my mind. Is OmniFocus data storage .xml AND do Omni Group feel it is a reliable way to store data AND is this only a temporary system in place for the BETA?

XML is an extensible text file format. The only way to corrupt XML is to write or read it incorrectly (the most common ways being to incorrectly format the values when writing or to incorrectly interpret those escaped values when reading) or to have the underlying disk storage go bad (which has nothing to do with the file format itself).

Structured text formats are the basis of most modern Internet protocols (such as web pages), and we use them as the primary document format for all our applications. Because they're text, technically oriented users can open them up and see what's changing and whether the values being written are correct (while XML can't be corrupt if you're writing and reading it correctly, the data you're trying to store in XML might not be correct), and they are designed to be very flexible, making them easy to extend as needs change. (They also have the very nice property of being easy to read on all systems, so your data is never held hostage by a particular application.)

Before we got the XML format in place, we were using a CoreData SQL database ourselves, but that's a very bad idea for primary data storage in an evolving application (and what application doesn't evolve?). Migrating from SQL to an XML file format was our biggest priority before releasing the application to the pre-beta audience of users (as we mentioned on our blog), because we wanted to be sure we had a file format that would continue to work with future versions of OmniFocus without losing any data.

Apple's Mail application uses a CoreData SQL database exactly as we do: the important data is actually stored in extensible text files, the SQL database is just used for efficient access to the data in those files. This means that if something goes awry with the SQL database (or if you need to change your SQL structure), you can just throw out the SQL and rebuild it from the text files.

Why is this a big deal? I apologize if I'm getting too technical, but all SQL databases have a fixed schema structure (SQL stands for "Structured Query Language"). As long as you keep the same structure you're fine, but as soon as you need to change that structure (e.g. to add support for repeating tasks) you're faced with a data migration problem, where you need to read all of the data in the old format and write it out in the new format. This isn't so bad when you control the database and the software and can make sure you migrate the database to the new format immediately as you update the software, but is very awkward when you don't know how old a database might be. (You end up having to write code to migrate from every possible earlier version of the database to the latest one, and all of that code has the potential for bugs that corrupt the data—possibly permanently, unless you're very careful about backups.)

(Sorry if this got too long-winded. For most of our first decade, Omni's business was based on building custom SQL database systems for large corporations (William Morris, AT&T Wireless, Standard & Poor's, etc.). During those years we also did quite a bit of work with extensible text formats, building one of the first web browsers in 1994 (the web is based around an extensible text format, or we would have a lot of trouble viewing those 1994 web pages today) and updating Lighthouse Design's applications to use extensible text formats rather than proprietary binary formats—so we have quite a bit of expertise in this area.)

[jem]

Originally Posted by ueila

""Can't copy the file package
"/Users/username/Library/Application Support/OmniFocus/OmniFocus.ofocus" to:
"/Volumes/username:password/d_macs/omnifocus/OmniFocus.ofocus"
Error: (-37) Bad item name.
Would you skip and continue?""

Could the problem be the ':' in the file name?

[BwanaZulia]

Ken... very cool reply and not too long all.

I would actually be interested in more information.

BZ

[Ken Case]

(Thanks, jem, for the reminder about the original problem.)

Originally Posted by ueila

I tried to backup the .ofocus files to a Thecus N4100 NAS server on my network but found it was not possible and kept getting an error message about file length.

What software are you using to make the backup? That doesn't look like a message the Finder would produce (it used slashes for the path separator while the Finder generally uses colons), so I'm assuming some other tool is involved?

Originally Posted by BwanaZulia

I would actually be interested in more information.

Anything in particular you're curious about?

One obvious curiosity that comes to mind is: why are we writing our XML as separate transactions rather than in one big file the way we usually do (in OmniGraffle, OmniOutliner, and OmniPlan)?

Well, I talked a bit about the advantages of XML, but I left out a big disadvantage: they're not very efficient to read or update. To read an XML document, you have to read the entire contents of the file (and there's no easy way to skip parts you aren't interested in). To update them, you have to rewrite the entire contents of the file. A big advantage to SQL is that you can read using a query which gives you just the results you're interested in, and you can do an update which touches only the records you've changed without rewriting everything else.

But another downside to SQL (besides the extensibility problem) is that even though you can write a single record quickly and efficiently, an incremental backup (such as the backups Time Machine will make in Leopard) has to make a complete copy of your entire database even if you only changed one record.

We wanted OmniFocus to be able to read and update your data quickly, so that we'd have fast launch times and fast save times—particularly because (unlike our other applications) we wanted to automatically and immediately save every edit. This suggests we should use SQL. But we also wanted our data format to be extensible and efficient to back up, which suggests we use XML.

So we built a hybrid system which uses a SQL database to handle the reads and updates efficiently, while recording each database operation as an XML transaction for extensibility and efficient incremental backups (the backup system only has to record the new transactions, not the entire database).

[jlindeman]

I definitely appreciate the in-depth technical info. It is nice to hear the thought that went into the selection of the data format. I like the fact that data store isn't a monolithic sqlite db.

I use (and love) Yojimbo, but uploading my 500MB db to my online backup is a bit of a pain. It looks like an rsync of the OF datastore would be much easier.

[ueila]

Ken thanks for taking time to post such a comprehensive reply.

Originally Posted by Ken Case

What software are you using to make the backup? That doesn't look like a message the Finder would produce (it used slashes for the path separator while the Finder generally uses colons), so I'm assuming some other tool is involved?

I tried to copy the file from a Windows 2003 Server to a Thecus N4100 NAS with a Terrabyte RAID array using FoldersSynchronizer 3.6.1 and also ChronoSync 3.3.5. I'd be grateful for any suggestions.

ueila

[SpiralOcean]

Originally Posted by Ken Case

an incremental backup (such as the backups Time Machine will make in Leopard) has to make a complete copy of your entire database even if you only changed one record.

We wanted OmniFocus to be able to read and update your data quickly, so that we'd have fast launch times and fast save times—particularly because (unlike our other applications) we wanted to automatically and immediately save every edit. This suggests we should use SQL. But we also wanted our data format to be extensible and efficient to back up, which suggests we use XML.

Now that's just freakin' cool! Nice job on doing it right from the ground up. I did notice that I never had to save anything. It makes me feel extremely safe that I won't loose a lot of data if something happens to the computer or application.

[Andrew]

uelia, I take it that other files copy correctly? I'm not very familiar with the Thecus NAS, but the "username:password" portion of the destination path looks suspicious to me. I don't know the mount options, but are you sure that the network mount is correct? If you're using Samba/CIFS, can you try AFP instead?

Alternatively, the symptom doesn't really seem to indicate this, but since the .ofocus file is actually a folder containing multiple files, one possibility is that the problem is with the name of one of the constituent files.

Once you renamed the .ofocus file, causing Finder to reveal it as a folder, did you try to copy that folder to your server? If the problem has to do with one (or more) of the constituent files, I would expect Finder to succeed with copying some files, then get stuck with the problem file, and give you an error message that includes the name of the problem file. That name might be enough to determine the problem.