I know it's been a while since this question was posted, but I have managed to get this working.
The "no root transactions" is caused by a combination of request filtering by the IIS web server, and a lack of error checking by OmniFocus.
During an initial sync (or a "Replace Server Database"), OmniFocus is sending double-escaped names in WebDAV PUT requests. Those requests get rejected by IIS with HTTP response code 404.11 -- which means "double-escaped URL". Unfortunately, OmniFocus never notices the errors. It simply continues on merrily and appears to complete the sync. But the files being PUT onto the server never made it. That's why you get a "no root transactions" error on the next sync attempt.
To make double-escaped URLs work, open IIS Manager, select the site with your WebDAV service, double-click on "Request Filtering" (if you don't have that control, install it using the IIS Administration Pack, which you can find
here. ). In the Actions pane on the right-hand side, click "Edit Feature Settings..." and make sure the "Allow double escaping" checkbox is filled in.
While we're discussing WebDAV transactions, I'd like to know why OmniFocus insists on using a full URL instead of a relative location as the target of its MOVE requests. If a proxy server such as ISA Server is forwarding requests to another server in an internal web farm, sync'ing breaks. (The MOVE target is in a part of the HTTP request that is considered data, and thus not subject to URL re-writing.) A local "hosts" file on the internal web server addresses this, but that's an awful hack...