add some design docs

diff --git a/docs/design/architectural-pillars.md b/docs/design/architectural-pillars.md
new file mode 100644 (file)
index 0000000..8c954da
--- /dev/null
+++ b/docs/design/architectural-pillars.md
@@ -0,0 +1,23 @@
+## Database
+
+MySQL? Postgres would also work. Redis might work but would make things like reporting/aggregating and other work more difficult. MySQL/MariaDB/Postgres all support `SELECT ... FOR UPDATE SKIP LOCKED`, allow aggregation (`SELECT AVG(completed_time-pending_time)`), etc.
+
+## Insertion
+
+Jobs are typically inserted into the database by a front-end like dlqueue.php (optionally via bookmarklet) or dlqueue.sh
+
+## Middleman
+
+The middleman retrieves jobs from the database, processes them, and offers them to workers.
+
+## Worker
+
+The workers are running on distributed hosts, and check in with the middleman for new jobs and to update the status of jobs.
+
+Workers can also process the job but typically hand it off directly to an external command like yt-dlp.
+
+Workers send the output of the job command back to be saved into the database.
+
+## TODO
+
+How do files get centralized from the worker? Should there be another set of queues for "files that need to be synced"? Should the worker sync it before marking it as completed?

diff --git a/docs/design/job-flow.md b/docs/design/job-flow.md
new file mode 100644 (file)
index 0000000..814ff1f
--- /dev/null
+++ b/docs/design/job-flow.md
@@ -0,0 +1,18 @@
+dlqueue job flow
+
+- A job is inserted, by dlqueue.php or another tool. This inserts a row into `pending_jobs` table. The job data can be a URL by itself, or a URL with some extra data (such as `tags`).
+- A worker finishes a job and requests a new job (or if idle, polls for new jobs after a timeout). The middleman selects and locks a row of the database.
+- The middleman processes transforms and/or rules on the job
+  - In my use case, this would include expanding galleries/playlists into a list of video URLs, and then checking that the resulting videos have not already been downloaded
+    - I'll `yt-dlp -s` locally and then add a `--match-filter "url!=... & url!=... & ..."` to the arguments for existing URLs
+  - This can also do things like check the tags against the worker's identity for geo-locked content, or add yt-dlp arguments for audio-only
+- The middleman provides the resulting data to the worker, and waits for the worker to accept or reject.
+- The worker can also process transforms and/or rules, and accept or reject the job
+  - This can also check the tags and everything else
+  - For example, a transform rule which precedes all arguments with `--` could be used to secure workers against an untrusted middleman/queue
+- If the worker rejects, the job is unlocked and the next job (`WHERE id > rejected_id`) is processed and offered
+- If the middleman (or worker) has an error during this step, the lock on the job will simply be released by the database.
+- Once the worker accepts, the job is marked as "being worked on" as of $time
+- The worker updates the job progress regularly
+- If the worker crashes, the job will eventually be re-queued again due to age
+- Once completed, the worker notifies to mark the job as complete