[pvonk]

I've been using OF since the early betas and have found it quite reliable. But now I'm not sure.

I have found in recent weeks that (rarely) a certain task that I had listed in context view was no longer there. Having thought that I must have completed it (but not able to find it), I'd pull up a very recent backup and go from there.

Today it happened again. This time I know that the task in question was there a few minutes ago, but not now. BTW, I use perspectives and my "view" remains pretty static, so it's not a question of having the task hidden away. I do all editing in OF and sync to iCal (and then to my Palm) were I use these devices as "read only", I don't edit there. I also use Anxiety to get a quick view of tasks in iCal. Today, when a particular task I had focused on was displayed and now it's no longer showing, I noticed that it was still showing in Anxiety - thus I know what it was called and what project it was in. I checked OF for "completed" tasks, etc., any indication of its existence in the project and couldn't find it.

Has anyone been experiencing this behavior?

And then I also discovered that OF no longer synced with iCal. I went to the sticky thread (Trouble syncing with iCal) near the top of the forum and followed the instructions including using terminal to do magic on the syncServer. I've had to do that a few times in the past. This time, it didn't work. I tried rebooting, no good. I repeated the instructions in the post. iCal won't sync.

I did do some experiments with "Back to my Mac" yesterday, turning it on in the .mac preferences and later turning it off. That's about the only thing I did remotely connected with syncing. Anyone have ideas here?

[pvonk]

Well, this is bizarre! After two days of occasionally trying to sync with iCal (unsuccessfully), I happened to try again today, and the thing worked! This time I haven't done anything unusual with syncing other apps or using .mac. Go figure.

I'll chalk this up to some system peculiarity as opposed to blaming OF.

[brianogilvie]

Originally Posted by pvonk

I've been using OF since the early betas and have found it quite reliable. But now I'm not sure.

I have found in recent weeks that (rarely) a certain task that I had listed in context view was no longer there.

I had this happen once or twice during alpha testing (back in July or August, I think). My conclusion at the time was user error--that I had inadvertently hit delete when I had the action selected. For a while I religiously checked my action list at the end of every day against the previous day's action list; after a couple weeks, I shifted to checking only once a week. After 2 more months I decided that OF was reliable. I still think user error is the most likely explanation for my case.

[Toadling]

This is why I still wonder if some kind of Trash bin implementation might be a good idea. Yeah, we have undo, but it's hard to catch an accidental delete.

[apinstein]

I unfortunately experienced this myself today.

I lost an unspecified amount of data today. The app crashed so bad that it brought up a crash reporter (I submitted, [OG #301899]), and when it came back up, it brought back data from a WHILE ago. Many items were missing (or deleted items returned).

I fortunately had one of the automated backups to restore from that was made about 20 hours ago, so I didn't lose too much.

But this type of database corruption and loss is unacceptable...

I am a bit scared now...

[apinstein]

I decided maybe to try rebuilding the "broken" database to see if that helped. A few notes...

1. My database was 11MB, but my backup file was only 88K!
1a. I looked in the contents of the db file. 3000+ files! No wonder OF has been S L O W lately...
2. After rebuilding, the 11MB was down to 88 (expected since OF says rebuild is just a backup/restore), but my data was still missing...

[steve]

Originally Posted by Toadling

This is why I still wonder if some kind of Trash bin implementation might be a good idea.

Great idea!
Steve

[Brian]

Apinstein, I'm really sorry about this.

More info follows, but first, I'd like you to open up an email to the support ninjas from the help menu, mention that you're the person having the problem in this thread, and send the original database (pre-rebuild), as well as the most recent backup. If you remember the name of any of the missing tasks, please include that in the email.

If you send us those files and that info, we may be able to recover your data; we need to run some tests before we can tell for sure.

Just in case you're able/willing to run terminal commands on the database, we need to see the output of:

xmllint -format * | grep -i "Name of missing action in quotes"

If that makes no sense to you, just send in the file and we'll take care of it for you.

(See my posts in this thread for a bit of background, including why the backup is so much smaller than the database.)

More detail: it sounds like there are two separate things going on here - there's a bug in the system frameworks that *could* cause data loss if the app kept running; we've tried to code defensively around that bug to prevent it from corrupting data, but we don't have access to all of the involved code and can't tell whether we've gotten every possible condition or not.

Ultimately, we try to prevent data loss by actually forcing the app to crash when we get into the bad state. From looking at your crash report, you were heading towards that bad state, so we crashed you on purpose.

A crash is annoying, but it prevents you from entering any data while we're in a state that could end up not recording it to disk. We have some additional fixes coming in 1.1 that should make the crashes less frequent, but if we missed any, the crash will still prevent data loss.

So, why was the database in such a weird state when you came back? We need to look at those files I mentioned to be sure, but if the backup was only 20 hours old, the data when we came back up *should* also be 20 hours old. Since it's not, our theory is that one or more of those transaction files in your database is corrupt. If that's correct, it means that when we started back up after the crash, we got partway through putting everything back together, hit the bad transaction file, and had to stop at that point.

This could be our bug, or it could just be a bad luck, in the form of a disk error that hit one of the transaction files. We don't have enough info to say for sure. Catching errors like this is one of the things that rebuilding the database is meant to do; in this case, the database rebuild was triggered by a crash, but the two aren't directly related.

Again, I'm really sorry this happened - we'll do everything we can to get this straightened out, but we'll need those files before we'll know anything for sure.