Should Context Chat, and perhaps AppAPI be refactored?

💻 Development
,
1

In the past couple months I’ve been testing nextcloud’s AI capabilities, with the latest being context chat.

I had a rough time hosting its backend on arm, but I made it work.

I’m using Nextcloud AIO, which provides Elasticsearch out of the box for full text search.

AFAIK Elasticsearch supports vector embeddings since version 8.7 (May 2023), while context chat started development around Sep 2023.

I understand the separation of responsibilities logic, but it’s a huge waste of resources double indexing files in 2 separate databases.

Moreover; context_chat_backend is hardcoded to use its own embeddings engine (and model), even though nextcloud might have an active OpenAI integration. In my fork I added support for remote embeddings API, sadly with environment variables as there’s no OCS endpoint for embeddings or reading integration_openai’s configuration.

Things like these give me the impression that Nextcloud dev teams work too separately, or even that some features sacrifice good practices over fast development (although if you used Elasticsearch and OpenAI from the get-go context_chat would take like 20% of the effort currently put into it).

Backwards compatible suggestions

  1. Create an OCS endpoint for embeddings *
  2. Remove the CPU|GPU option from the app_api daemon **, it’s high time ExApps got their own configuration options in the Nextcloud GUI via a /settings endpoint ***.
  3. Now that context_chat_backend can have configurable options ****, the user can:
    a. Select the embeddings engine (CPU / GPU / OpenAI)
    b. Select a custom embeddings model
    c. Select a vector database (Elasticsearch if it’s available, or pgvector)
  4. If Elasticsearch is used as a vector database context_chat should not send files to context_chat_backend for indexing, but it should enable vector embeddings in Elasticsearch and trigger an embeddings generation for already indexed files (by adding an ingest pipeline ?)

* This shall be used by Elasticsearch when using it as a vector database, or context_chat_backend when using pgvector.

** Really useful for multi-model backends.

*** Having easy configurable options will enable more ExApps getting created in general.

**** context_chat_backend (or any ExApp) must handle restarting itself if needed for the new configuration to apply

P.S. OpenAI => OpenAI compatible API, not necessarily the OpenAI API. I found Koboldcpp easier to host than LocalAI (especially on new Macs).

3 Likes
2

Hello there @rallifs1,

you started a topic in development category. This category is intended for active developers of the core or apps in the Nextcloud ecosystem.
From the description in your topic, it is not clear if you are seeking help and advice about a concrete problem you have or you want to actually develop the corresponding solution.

Please specify explicitly the required information to help you best. These are:

  1. What you want to achieve
  2. What you have done so far
  3. What is failing
  4. What you expect from the forum community

Without additional information the community members cannot help you in an efficient manner. Please keep in mind that the help here in the forum are mostly based on work of volunteers and thus it is just fair to reduce the burden on them.

If you accidentally posted in the category, just give a hint and a moderator can move the corresponding category.

Unfortunately, I do not get the main point of your post, sorry. Maybe you could elaborate a bit what you are after?

Regards,
christianlupus

3

I am not an active developer of the core or official apps. If my post doesn’t belong in this category, I am sorry. Feel free to move it.

I am merely making some structural suggestions about the AppApi and Context Chat in particular. I’d be happy to contribute code regarding these changes, but I am in no position to take over these projects.

The current solution is working, it’s just not efficient.

A fork of context_chat_backend for arm64 with support for external embeddings API

Just an open dialogue about the project roadmap. I might be wrong anyway.

3 Likes