Alternate LLM file. Dolphin-Mistral gguf file

ℹ️ Support
1

I’m having difficulty installing an alternate gguf file for the Nextcloud AI LLM ExApp. The file name is Neo-Dolphin-Mistral-7B-E4-0-1-6-Q8-June-24.gguf, and I’m getting this error:

Text task returned error: llm2:llm2:Meta-Llama-3-8B-Instruct.Q4_K_M:summary:summary. Error: ('Error: Failed to initialize the LLM model on both GPU and CPU.', '1 validation error for LlamaCpp\n__root__\n Could not load Llama model from path: /nc_app_llm2_data/Neo-Dolphin-Mistral-7B-E4-0-1-6-Q8-June-24.json. Received error Model path does not exist: /nc_app_llm2_data/Neo-Dolphin-Mistral-7B-E4-0-1-6-Q8-June-24.json (type=value_error)')

I’ve tried just the raw .gguf file with and without the .json file.

It WAS running before, but numerous attempts at an earlier install failed until the last ExApp update.

Maybe I have the data in the wrong place or it is the wrong .json file:

{
  "prompt": "<|im_start|> system\n{system_prompt}\n<|im_end|>\n<|[INST]|> user\n
{user_prompt}\n<|[/INST]|>\n<|im_start|> assistant\n",
  "loader_config": {
     "n_ctx": 4096,
     "max_tokens": 2048,
     "stop": ["<|im_end|>"]
  }
}

I’ll update as I progress on this effort.

thanks!

Latest error with this after a quick pass over Llama3.1 with gguf files:

{
  "prompt": "<|[INST]|> system\n{system_prompt}\n<|[/INST]|>\n<|im_start|> user\n{user_prompt}\n<|im_end|>\n<|im_start|> assistant\n",
  "loader_config": {
     "n_ctx": 32768,
     "max_tokens": 4096,
     "stop": ["<|im_end|>"]
  }
}


An exception occurred while executing a query: SQLSTATE[22001]: String data, right truncated: 7 ERROR: value too long for type character varying(64)