Pictures Tagging - best ways to make use of AI-based description and tags in NextCloud?

đź’» Development
, ,
1

I’ve been working on a personal project to make my photo archive (~5 million images) fully searchable. After a bit of hacking, I now have a setup I’m really happy with - I’ve wired up my local GPU and am using mistral-small-3.1 via Ollama to generate rich descriptions and keyword-style tags for each photo.

So far, the results are great - I can describe any image and extract meaningful tags like:

"car", "a girl in a red dress", "green grass", "invoice", "kids at the beach"

And it’s freaking terrifying how accurate this thing is (I was blown away when I asked it to find a picture with a “VIN number” and it just listed it…).

The challenge?
I don’t know how to make use of these tags inside Nextcloud in a way that integrates with its native search, filtering, or media browsing experience.

What I’ve got working:

  • Full image tagging pipeline using GPU + LLM
  • Results logged to .txt and .json files per image
  • Optional EXIF embedding and/or .tags.txt creation
  • Images accessible via WebDAV and shown inside Photos/Memories

What I haven’t figured out yet:

  • How to associate these tags with the actual file in Nextcloud (via API or WebDAV)
  • How to make these tags searchable via the default Nextcloud interface
  • Whether I can inject metadata (e.g., system tags, file comments, etc.) that the platform respects

What I’ve tried:

  • OCS API endpoints for system tags (/ocs/v2.php/cloud/systemtags) — works in theory, but file ID-based tagging via API seems undocumented or broken
  • WebDAV PROPFIND and PROPPATCH for setting metadata - no success with tags
  • Looking into occ files:metadata:* commands - seems promising but not ideal for ongoing automation
  • Memories app / EXIF-based workflows - better for filtering but not for actual “tags” per se

My goal here is to be able to search in Nextcloud for “car” or “receipt” and have relevant photos show up, even if the tag came from an LLM pipeline and not from me manually assigning it.

  • Has anyone had success assigning searchable tags to files in Nextcloud programmatically?
  • Are there any best practices for embedding metadata that the core search picks up?
  • Would you recommend writing a small app to sync tags and expose a custom search experience?
  • Is it worth investing in a custom workflow engine rule or file metadata scanner?

Any input - workflows, plugins, hacks, future roadmap hints - is super appreciated!

If you’ve done something similar or know of a plugin that could help, I’d love to hear about it.

Cheers,

2 Likes
2

Hi,

I don’t think tags would be such a good idea. It seems that you will create a lot of tags. And the purpose of tags (in my opinion) is to help to categorize, not a keyword that should be searchable itself.

What i suggest could be to use the filesmetada (introduced with NC 28). So you could create a free text metadata “image_description”, that could be managed and edited manually or generated with AI. Then, you have to created in your app an entrypoint to the unified search that let you match these metadata content with the search. But if I consider your 5 millions entries, you should have a look on a powerful search engine like Solr or ElasticSearch that is handled by Fulltext Search app (if i remember well).

All this require an independant Nextcloud app you’ll have to create, in order to manage these capacities.

Yours,

1 Like
3

I have the same question for many years. Agree on using Elasticsearch to power the search so we can focus on tying all them together for quick win. I also have millions of photos that thanks to unsubscribing from google photos, quickly becomes unwieldy and less useful to find the photo I wanted. If there is a tutorial that can describe how the plugin systems in Nextcloud works, maybe I can offer deep dive tutorial on the steps to this direction, although can’t give time commitment yet for the full integration like the popular apps in nextcloud appstore.

4

I agree with @smarinier that I’m not sure if tags is really the right approach. What I can recommend is the Memories app: https://apps.nextcloud.com/apps/memories, which works on the same principle relatively well.

But I have another tip for you: I also used the plugin “Memories” for a long time, but was never completely satisfied with it. Since then, I’ve also been using the program “Immich”. The program really fulfills all my wishes: a world map with the pictures, very good face recognition, AI classification (where you even have the option of selecting different models),… - I’m really very happy with it. Maybe you could take a look at it.
Unfortunately, you cannot search for the keywords in Nextcloud. Perhaps this could be realized at some point via Immich’s API.