The Omni Group
These forums are now read-only. Please visit our new forums to participate in discussion. A new account will be required to post in the new forums. For more info on the switch, see this post. Thank you!

Go Back   The Omni Group Forums > OmniFocus > OmniFocus 1 for Mac
FAQ Members List Calendar Search Today's Posts Mark Forums Read

 
OmniFocus Data Storage Thread Tools Search this Thread Display Modes
I tried to backup the .ofocus files to a Thecus N4100 NAS server on my network but found it was not possible and kept getting an error message about file length.

""Can't copy the file package
"/Users/username/Library/Application Support/OmniFocus/OmniFocus.ofocus" to:
"/Volumes/username:password/d_macs/omnifocus/OmniFocus.ofocus"
Error: (-37) Bad item name.
Would you skip and continue?""

I thought perhaps if I changed the file extention to .ofo I could back them up and then rename them on the server. Renaming surprised me by showing the .ofocus data file as a folder containing 336 .xml files.

I seem to remember another GTD app recently switched to SQL for storing data and the explanation for the change was potential corruption problems with .xml.

This creates several questions in my mind. Is OmniFocus data storage .xml AND do Omni Group feel it is a reliable way to store data AND is this only a temporary system in place for the BETA?

ueila

PS: I'm no expert just a user so please keep any replies simple.
 
Isn't just about every preference, data file in OSX XML based? Seems pretty stable to me.
 
Quote:
Originally Posted by ueila
I thought perhaps if I changed the file extention to .ofo I could back them up and then rename them on the server. Renaming surprised me by showing the .ofocus data file as a folder containing 336 .xml files.
This is probably a little technical, but for the curious: OmniFocus uses compressed XML transaction files to store its data, with a SQL cache for efficient access. (Each time you update the application, we rebuild the SQL cache to ensure that it's consistent with the latest schema.)

Quote:
Originally Posted by ueila
I seem to remember another GTD app recently switched to SQL for storing data and the explanation for the change was potential corruption problems with .xml.

This creates several questions in my mind. Is OmniFocus data storage .xml AND do Omni Group feel it is a reliable way to store data AND is this only a temporary system in place for the BETA?
XML is an extensible text file format. The only way to corrupt XML is to write or read it incorrectly (the most common ways being to incorrectly format the values when writing or to incorrectly interpret those escaped values when reading) or to have the underlying disk storage go bad (which has nothing to do with the file format itself).

Structured text formats are the basis of most modern Internet protocols (such as web pages), and we use them as the primary document format for all our applications. Because they're text, technically oriented users can open them up and see what's changing and whether the values being written are correct (while XML can't be corrupt if you're writing and reading it correctly, the data you're trying to store in XML might not be correct), and they are designed to be very flexible, making them easy to extend as needs change. (They also have the very nice property of being easy to read on all systems, so your data is never held hostage by a particular application.)

Before we got the XML format in place, we were using a CoreData SQL database ourselves, but that's a very bad idea for primary data storage in an evolving application (and what application doesn't evolve?). Migrating from SQL to an XML file format was our biggest priority before releasing the application to the pre-beta audience of users (as we mentioned on our blog), because we wanted to be sure we had a file format that would continue to work with future versions of OmniFocus without losing any data.

Apple's Mail application uses a CoreData SQL database exactly as we do: the important data is actually stored in extensible text files, the SQL database is just used for efficient access to the data in those files. This means that if something goes awry with the SQL database (or if you need to change your SQL structure), you can just throw out the SQL and rebuild it from the text files.

Why is this a big deal? I apologize if I'm getting too technical, but all SQL databases have a fixed schema structure (SQL stands for "Structured Query Language"). As long as you keep the same structure you're fine, but as soon as you need to change that structure (e.g. to add support for repeating tasks) you're faced with a data migration problem, where you need to read all of the data in the old format and write it out in the new format. This isn't so bad when you control the database and the software and can make sure you migrate the database to the new format immediately as you update the software, but is very awkward when you don't know how old a database might be. (You end up having to write code to migrate from every possible earlier version of the database to the latest one, and all of that code has the potential for bugs that corrupt the data—possibly permanently, unless you're very careful about backups.)

(Sorry if this got too long-winded. For most of our first decade, Omni's business was based on building custom SQL database systems for large corporations (William Morris, AT&T Wireless, Standard & Poor's, etc.). During those years we also did quite a bit of work with extensible text formats, building one of the first web browsers in 1994 (the web is based around an extensible text format, or we would have a lot of trouble viewing those 1994 web pages today) and updating Lighthouse Design's applications to use extensible text formats rather than proprietary binary formats—so we have quite a bit of expertise in this area.)

Last edited by Ken Case; 2007-05-20 at 10:15 AM.. Reason: Fixed some punctuation, added some links
 
Ken, where is the SQL cache stored? From a programming perspective would there be any problem in just reading the data into another application when OmniFocus is running?
 
Quote:
Originally Posted by kunicki
Ken, where is the SQL cache stored?
It's in ~/Library/Caches/com.omnigroup.OmniFocus.

Quote:
Originally Posted by kunicki
From a programming perspective would there be any problem in just reading the data into another application when OmniFocus is running?
It's not going to upset OmniFocus if you read that sqlite database, but your code will be more fragile if you're relying on the schema of our cache (which could change from one build to the next). I'd definitely recommend using our AppleScript interfaces instead. (Bear in mind that you don't actually have to code in AppleScript to use those interfaces: you can also get at them from ruby or python using their scripting bridges.)
 
I'm sorry if this is a terribly snoopy and/or stupid question, but was it difficult to create this type of hybrid database and did you do it with Xcode?
 
FWIW, I ran into the same file name problem last night just trying to back-up my OF file from my MacBook to my PowerBook. Very similar error message to the "root=..." one above.
 
Quote:
Originally Posted by ueila
""Can't copy the file package
"/Users/username/Library/Application Support/OmniFocus/OmniFocus.ofocus" to:
"/Volumes/username:password/d_macs/omnifocus/OmniFocus.ofocus"
Error: (-37) Bad item name.
Would you skip and continue?""
Could the problem be the ':' in the file name?
 
Ken... very cool reply and not too long all.

I would actually be interested in more information.

BZ
 
(Thanks, jem, for the reminder about the original problem.)

Quote:
Originally Posted by ueila
I tried to backup the .ofocus files to a Thecus N4100 NAS server on my network but found it was not possible and kept getting an error message about file length.
What software are you using to make the backup? That doesn't look like a message the Finder would produce (it used slashes for the path separator while the Finder generally uses colons), so I'm assuming some other tool is involved?

Quote:
Originally Posted by BwanaZulia
I would actually be interested in more information.
Anything in particular you're curious about?

One obvious curiosity that comes to mind is: why are we writing our XML as separate transactions rather than in one big file the way we usually do (in OmniGraffle, OmniOutliner, and OmniPlan)?

Well, I talked a bit about the advantages of XML, but I left out a big disadvantage: they're not very efficient to read or update. To read an XML document, you have to read the entire contents of the file (and there's no easy way to skip parts you aren't interested in). To update them, you have to rewrite the entire contents of the file. A big advantage to SQL is that you can read using a query which gives you just the results you're interested in, and you can do an update which touches only the records you've changed without rewriting everything else.

But another downside to SQL (besides the extensibility problem) is that even though you can write a single record quickly and efficiently, an incremental backup (such as the backups Time Machine will make in Leopard) has to make a complete copy of your entire database even if you only changed one record.

We wanted OmniFocus to be able to read and update your data quickly, so that we'd have fast launch times and fast save times—particularly because (unlike our other applications) we wanted to automatically and immediately save every edit. This suggests we should use SQL. But we also wanted our data format to be extensible and efficient to back up, which suggests we use XML.

So we built a hybrid system which uses a SQL database to handle the reads and updates efficiently, while recording each database operation as an XML transaction for extensibility and efficient incremental backups (the backup system only has to record the new transactions, not the entire database).

Last edited by Ken Case; 2007-05-20 at 12:36 PM.. Reason: Removed some stray punctuation, improved some wording about SQL reads
 
 


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes


Similar Threads
Thread Thread Starter Forum Replies Last Post
Scripts for calendar and storage mdmdiv OmniFocus Extras 0 2013-03-24 09:48 AM
DOM storage bmastenbrook OmniWeb General 0 2010-12-16 04:52 PM
Templates storage ronbailey42 OmniOutliner 3 for Mac 11 2010-09-15 12:32 PM
Feature Request: Storage of Preferences phillychuck OmniFocus 1 for Mac 0 2007-07-25 04:13 PM
Information Storage samaparicio OmniFocus 1 for Mac 4 2006-11-29 10:52 AM


All times are GMT -8. The time now is 05:54 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2024, vBulletin Solutions, Inc.