Sorry! It seems the problem is that your main database file is over 10MB and the new XML library now handles that differently.
We're aware of this issue and working on a fix.
In the meantime, if you have another computer or iOS device syncing to the same database, you may be able to slim down your database enough to get it working on 10.9.
Attachments do not contribute to this size limit, so you'd need to look for actions with long notes, or perhaps an action that repeats frequently that you can delete.
If you'd like help with this process, our Support Humans can be reached at
omnifocus@omnigroup.com or
+1 206-523-4152 or +1 800-315-OMNI (until 5pm Pacific Time weekdays)