Am I building this pair of background jobs correctly?

💻 Development app dev
1

I’m currently working on an app to give users the ability to easily and automatically convert media they upload to their Nextcloud instance.

(See implement on-the-fly transcoding to stream more types of videos · Issue #24712 · nextcloud/server · GitHub for mockup)
(See GitHub - cwilby/nextcloud-automatic-media-encoder: Automatically re-encode photos/videos uploaded to Nextcloud so they can be viewed on all devices.)

In a nutshell, users create Rules which allow them to choose a folder, then choose what extension should trigger a media conversion into a format of their choice. (Additionally, they can choose what happens to the source file after conversion)

After creating rules, the NextCloud job engine does the rest (:crossed_fingers:)

First, I’ve built a FindMediaJob extends TimedJob that ideally runs once a minute. This job scans for any media not currently being converted and registers ConvertMediaJob jobs to IJobList.

Code

ConvertMediaJob extends QueuedJob and runs when queued by FindMediaJob. It calls ffmpeg and runs post-conversion rules (Move / Keep / Delete source).

Code

There is also a retry mechanism so that if a conversion fails for some reason, it’ll retry it before adding it to a queue where users can triage (Ignore or Manual Retry).

That’s where things are at now.

I’ve got some questions/limitations that I’m either wrong about or do actually need to address:

  1. FindNewMedia cannot run once a minute due to 5-minute NC configuration.
  2. ConvertMedia also can only run once every 5 minutes.
  3. Jobs cannot run in parallel. (Not worried about this for ffmpeg conversion, worried for running multiple FindNewMedia jobs for lots of user accounts)
  4. Jobs may actually only be run one at a time via Cron? (i.e. if JobList has 5 queued jobs, it would take 25 minutes for them all to complete?).

Additionally I’m struggling to get FindNewMedia to run. I’ve registered it in info.xml as follows:

    <background-jobs>
        <job>OCA\AutomaticMediaEncoder\BackgroundJob\FindNewMediaJob</job>
    </background-jobs>

Thanks for reading - I’m excited about the project and hoping to have it shipped in the next month or two barring any major issues with the queue system I have going on :slight_smile:

2

Maybe GitHub - nextcloud/workflow_pdf_converter: Let Nextcloud automatically convert documents to PDF as some inspiration for you?

3

Thanks for the pointer - need to learn more about the workflows API, my wish list is retries, ability to collect/display job statistics, and fail reconciliation.

Feels promising though!

4

Media conversion is very CPU heavy. Best to make sure it is set to extremely low priority processing and give the server admin control over when jobs like this are allowed to run.