Problems with Nextcloud and Local AI

[apfelcast]

I am trying to get Nextcloud to work with my Local AI installation.

I installed Local AI on my CPU only system with the following docker command:

docker run -p 8081:8080 --name local-ai -ti localai/localai:latest-aio-cpu

When I use the chat in the Local AI web interface the chat and everything else works well.

Then I tried to connect Local AI to my Nextcloud version 30.0. Here when I enter in the chat the same prompt as in the Local AI web interface nothing happens. Even if I wait for a long time. After some time I get this error in the nextcloud log.

{"reqId":"3GORAQ3AN1Dx6YespCbr","level":3,"time":"2024-09-18T09:10:21+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Unknown error while processing TaskProcessing task","userAgent":"--","version":"30.0.0.14","exception":{"Exception":"RuntimeException","Message":"OpenAI/LocalAI request failed: Fehler bei der API-Anfrage:could not load model - all backends returned error: [llama-cpp]: could not load model: rpc error: code = Canceled desc = \n[llama-cpp]: could not load model: rpc error: code = Canceled desc = \n[llama-ggml]: could not load model: rpc error: code = Unknown desc = failed loading model\n[llama-cpp-fallback]: could not load model: rpc error: code = Canceled desc = \n[piper]: could not load model: rpc error: code = Unknown desc = unsupported model type /build/models/gpt-3.5-turbo (should end with .onnx)\n[rwkv]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[stablediffusion]: could not load model: rpc error: code = Unknown desc = stat /build/models/gpt-3.5-turbo: no such file or directory\n[whisper]: could not load model: rpc error: code = Unknown desc = stat /build/models/gpt-3.5-turbo: no such file or directory\n[huggingface]: could not load model: rpc error: code = Unknown desc = no huggingface token provided\n[bert-embeddings]: could not load model: rpc error: code = Unknown desc = failed loading model\n[/build/backend/python/bark/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/bark/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/openvoice/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/openvoice/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/transformers/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/coqui/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/coqui/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/vall-e-x/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vall-e-x/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/vllm/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vllm/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/rerankers/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/rerankers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/exllama2/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama2/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/diffusers/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/diffusers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/exllama/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/autogptq/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/autogptq/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/parler-tts/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/parler-tts/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/transformers-musicgen/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers-musicgen/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/mamba/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/mamba/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/sentencetransformers/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n[/build/backend/python/sentencetransformers/run.sh]: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/TaskProcessing/Manager.php","line":810,"function":"process","class":"OCA\\OpenAi\\TaskProcessing\\TextToTextProvider","type":"->"},{"file":"/var/www/nextcloud/lib/private/TaskProcessing/SynchronousBackgroundJob.php","line":54,"function":"processTask","class":"OC\\TaskProcessing\\Manager","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/Job.php","line":61,"function":"run","class":"OC\\TaskProcessing\\SynchronousBackgroundJob","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php","line":43,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php","line":29,"function":"start","class":"OCP\\BackgroundJob\\QueuedJob","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":162,"function":"execute","class":"OCP\\BackgroundJob\\QueuedJob","type":"->"}],"File":"/var/www/nextcloud/apps/integration_openai/lib/TaskProcessing/TextToTextProvider.php","Line":142,"message":"Unknown error while processing TaskProcessing task","exception":[],"CustomMessage":"Unknown error while processing TaskProcessing task"},"id":"66ea9cf5cf092"}

Here is an overview of how I connected Local AI to Nextcloud:

It would be great of anyone could help me getting this running

[marcelklehr]

Hey @apfelcast
You’re trying to use gpt-4 from LocalAI? That seems off.
Also, do check the nextcloud logs for errors.

[apfelcast]

Thanks for your response. Which LLM should I use with LocalAI?
I posted the error from the nextcloud log above.

[kyteinsky]

Hello, try using “Chat completions” and select the model which you know you have downloaded in local-ai’s webui.

[naruto]

gpt-4 should be fine… Local-ai is designed to perfectly integrate with OpenAI prompts, by just replacing the endpoints (with the local-ai one instead of OpenAI)

You should use the AIO docker image for this

It’s explained here https://localai.io/basics/container/#all-in-one-images for the matching between the “model name” (called in the API) and the “real model” (the one that process the query)

On my setup it’s working, but really really slow if going through the Nextcloud assistant.

A chat Query can take 3 or 4 minutes to answer through the Assistant, whereas if direcly called from the local-ai server (http GUI) it’s immediate, but also immediate if using an API call through with CURL (from the Nextcould server SSH)

Could you try to open the Assistant > Chat, then ask a simple question like “How are you?” and then leave the page as is to see of you get an answer after 3-5 minutes ? (and maybe increase the request timeout if needed)

Really strange, that looks like a bug

[naruto]

I forgot to mention that I’m using CUDA with 2 Nvidia A5000 cards. The nvidia-smi shows high power usage when querying through the Local-AI http GUI or direct curl API call, which means there are handling the query / load … but there is no change (almost idle) when issuing a query through the assistant

My Local-AI server is on a separate VM (with GPU), called through a local IP address (http://10.x.y.z:8080). I checked my config.php which has the ‘allow_local_remote_servers’ => true in it (as recommended in the “Artificial Intelligence” admin settings

I can see the Nextcloud is initiating the API call on the Docker log, I will try to check if I can get the POST data to try to analyze further

[szaimen]

Hi @kyteinsky i debugged this yesterday and it seems to be an issue in the integration_openai app if only one model for chats is available. See Local AI community container not working in AIO v9.5.1 · Issue #5299 · nextcloud/all-in-one · GitHub

[kyteinsky]

Thanks Simon, indeed seems like a bug, will be fixed soon.

[kyteinsky]

A chat Query can take 3 or 4 minutes to answer through the Assistant, whereas if direcly called from the local-ai server (http GUI) it’s immediate, but also immediate if using an API call through with CURL (from the Nextcould server SSH)

hi, see this section for faster processing of tasks: Nextcloud Assistant — Nextcloud latest Administration Manual latest documentation
Use it with a timeout of around 60 secs like so

set -e; while true; do occ background-job:worker -v -t 60 "OC\TaskProcessing\SynchronousBackgroundJob"; done

[naruto]

Interesting, I also finally realized the “Chat with AI” queries (and also other AI queries) are processed once every 5 minutes (at 9:00 / 9:05 / 9:10 / …) with the CRON (depending on setup), that explains the time before getting an answer even in the chat.

Does your command line change this setting globally (all tasks), or only for AI tasks ?

That would be nice to have a real-time setting (especially for chat with AI, we cannot wait even for 60 seconds for every answer)

[kyteinsky]

I’m not sure what other tasks you refer to. This command applies to all the task processing tasks done by PHP apps like with “Chat with AI”, Summarize, Text generation, etc. with PHP apps providing the backend like integration_openai, context_chat, llm (deprecated) etc.
It is not required for ex-app providers like llm2, translate2, stt_whisper2, etc.

Sidenote: if you want to speed up any other background jobs, just replace the class name “OC\TaskProcessing\SynchronousBackgroundJob” with the target one.

[naruto]

I was thinking the CRON could process many other tasks than Assitant related ones… (like sync, maintenance tasks, indexes, …) And maybe having a very short timer here could be problematic (calling the script too often)

The main problem with the “Chat with AI” is that even if I set at 60 second timer for Cron, it’s not a chat anymore

There should maybe be a new setting in the “Artificial Intelligence” admin setting to choose:

• if AI tasks should be processed immediately (at least for Chat) or scheduled
• define the timer here if needed

But maybe I’m not understanding exactly how it works behind… right now it’s just frustrating to have a blazing hast LocalAI backend, and having to wait for the schedule to get the answer, the image generated, or anything else about AI Assistant

(I’m using Assistant v2.0.4 and OpenAI and LocalAI integration v3.1.0, on NC v30.0.0)

[kyteinsky]

Sorry I should’ve clarified the 60 seconds timeout. It is a timeout to shutdown the worker itself and then the while loop spawns a new one. This ensures the latest settings and the latest php code is used. The worker does not use the updated code/settings if it is left to live forever. If the settings are modified often, consider lowering the timeout.

This has no relation to how fast the AI tasks are done. All tasks are scheduled are picked up as soon as they are available and processed sequentially. To process multiple tasks at the same time, run this same command multiple times in the background using programs like screen or tmux. 4 of them should generally be enough for small-scale usage.

like sync, maintenance tasks, indexes, …) And maybe having a very short timer here could be problematic (calling the script too often)

Yes, it would be better to leave them be unless you know what you’re doing since they are made keeping the 5 minute cron interval in mind.
Do you have any use cases in mind where executing them sooner would be useful?

[naruto]

Ok, thank you for the clarification!

Right now AI tasks are only processed on a delayed scheduled way (by default once every 5 min, at fixed time)

It’s described here https://github.com/nextcloud/assistant/issues/129 (latest posts) and here https://github.com/nextcloud/integration_openai/issues/126

I could also reproduce this with a fresh v30.0.0 setup (AIO Docker) and a LocalAI setup (on a dedicated local host): if you ask for an assistant request (Chat, Generate Text, or any other tasks) at 05:01, it will be processed at 05:05, not immediately.

Right now I don’t really know if this timer is due to:

• a bug in the flow to process the Assistant requests (sending the request to a delayed scheduler instead of trying to processing them live)
• the way the scheduler is setup for the Assistant (not trying to process the queue immediately)
• other tasks that could be queued before in the scheduler and blocking the list behind (I’m thinking to reported bugs like the “column cr.id” that could maybe timeout)

[eneiluj]

Hey Naruto,

I’ll try to bring some more clarity.
When launching a task via the assistant, it is scheduled as a background job. Then there are 2 ways for this job to be executed:

• Periodic execution of cron.php (there is a delay between the task creation and its execution)
• Running occ background-job:worker like @kyteinsky mentioned so there is a constantly running Php process that picks up the AI jobs immediately after they have been scheduled and executes them. In this case there is no delay.

We can’t really execute AI tasks at the precise moment they are created (in the Php process created by the network request from the Assistant to the server) because it would mean there can be a lot of parallel task executions. Also, the network requests have a timeout that can be reached if the task is too long, Php also has a run time limit when launched by a web server. All those limits don’t apply when using the occ worker.

We want to execute tasks sequentially as fast as possible. That’s what the occ worker does.

[naruto]

Thank you for this explanation, it’s very useful even if I’m still a little confused about this “occ background-job:worker” setting.

I understood this occ command was for having a constantly running queue not only AI tasks, but also for ALL tasks (from other services / apps than AI…) Is this correct?

Maybe this setting could also be accessible from the GUI in “Settings > Artificial Intelligence” (with some explanation on impact)

[eneiluj]

occ background-job:worker is not a setting. It is a command that you run.
Read more on occ there: Using the occ command — Nextcloud latest Administration Manual latest documentation
You can do all sort of things with occ like creating a user, updating an app etc…
To be more precise: “background-job:worker” is one of the available occ commands.
It accepts an argument to target a specific job class if you want the worker to only execute one specific type of background jobs.

cron.php is executed by your server’s operating system every 5 min (or whatever you’ve set). It consumes/executes all the jobs it can and then terminates. You have to wait for the next cron.php execution for further jobs to be executed.

occ background-job:worker is different. Once you launch it it will stay alive and “watch” the job list and as soon as one job appears, it will execute it. There is the “–stop_after” parameter you can set to make it stop after some time.
You can use this background job worker occ command to run some background jobs without delay. This command was created specifically for the AI tasks but it can be used to run any type of background job that seems more urgent than others.

Any better?

[naruto]

Sorry for the delay to answer, I was having some further testings with AI, Context Chat, Local AI, … and more generally keeping evaluating Nextcloud for our use case.

Yes I have a better understanding now, thank you!

I currently use 2 background-job:worker threads to watch for my AI requests (1 for Context Chat + 1 for the LocalAI as I’m currently using the “nc_texttotext” setting to forward the Context queries to the default Nextcloud LocalAI server)

This works great on my servers (with 2 GPUs)

I don’t know if it’s possible, but that would be great to have a GUI / option section to enable / manage / watch those workers (I actually start them from a terminal)

[vawaver]

Hello everyone,

I am sorry for off-topic but I’m trying to set up the Local AI CPU version in a Nextcloud AIO environment, following the guide by @apfelcast (his YT channel). While the WebUI of Local AI starts successfully, the Chat function does not respond, and I’m seeing repeated errors in the logs when attempting to load models.

Here are the key errors from the logs:

Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF

It seems like the models cannot be loaded properly. I’m using models in the .gguf format, and they are stored under the models/ directory. I’ve checked the permissions, and although the files are owned by root:root, running Docker as root didn’t resolve the issue.

Additionally, the gRPC communication appears to be failing with errors like:

connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:<port>: connect: connection refused"

I’ve also tried increasing the gRPC connection attempts and delay, but the problem persists.

Does anyone know a specific place or community where I can ask for more help regarding this issue? I’m unsure where the best place to seek advice is for problems related to Local AI in this context.

Any guidance on where I should post this question or potential solutions to my issue would be greatly appreciated.

[naruto]

Did you setup the “Artificial Intelligence” options in the “Administration Settings” from your Nextcloud instance ? The direct path after your instance URL is /settings/admin/ai

You should first check here the “Service URL” in the “OpenAI and LocalAI integration” section is correctly setup with your LocalAI server URL as you can access it from a browser.

Then you should be able to pull the model list from here, and keep configuring the Assitant