This adds more cache consistency issues, concurrency issues, network blocking issues, authn+z issues, daemon process supervision issues, ... for what? To store your SQL data in a remote data store? I would rather my app shell out to gsutil (or curl!) than deal with all of this.
Simple hack: mount a tmpfs filesystem and write your sqlite database there. Every 30 seconds, stop writing, make a copy of the old database to a new file, start writing to the new file, fork a process to copy the database to the object store and delete the old database file when it's done. Add a routine during every 30 second check to look for stale files/forked processes.
Why use that "crazy hack", versus the impeccably programmed Cloud Backed SQLite solution?
- Easier to troubleshoot. The components involved are all loosely-coupled, well tested, highly stable, simple operations. Every step has a well known set of operations and failure modes that can be easily established by a relatively unskilled technician.
- File contents are kept in memory, where copies are cheap and fast.
- No daemon outside of the program to maintain
- Simple global locking semantics for the file copy, independent of the application
- Thread-safe
- No network blocking of the application
- Authentication is... well, whatever you want, but your application doesn't have to handle it, an external application can.
- It's (mostly) independent of your application, requiring less custom coding, allowing you to focus more on your app and less on the bizarre semantics of directly dealing with writes to a networked block object store.
Simple hack: mount a tmpfs filesystem and write your sqlite database there. Every 30 seconds, stop writing, make a copy of the old database to a new file, start writing to the new file, fork a process to copy the database to the object store and delete the old database file when it's done. Add a routine during every 30 second check to look for stale files/forked processes.
Why use that "crazy hack", versus the impeccably programmed Cloud Backed SQLite solution?
- Easier to troubleshoot. The components involved are all loosely-coupled, well tested, highly stable, simple operations. Every step has a well known set of operations and failure modes that can be easily established by a relatively unskilled technician.
- File contents are kept in memory, where copies are cheap and fast.
- No daemon outside of the program to maintain
- Simple global locking semantics for the file copy, independent of the application
- Thread-safe
- No network blocking of the application
- Authentication is... well, whatever you want, but your application doesn't have to handle it, an external application can.
- It's (mostly) independent of your application, requiring less custom coding, allowing you to focus more on your app and less on the bizarre semantics of directly dealing with writes to a networked block object store.