Nextcloud and Elasticsearch in Docker

Guys this has me tearing my hair out and I’d love some input.

I’m trying to create a server that has Nextcloud, OnlyOffice and Elasticsearch running together in separate docker containers, speaking to each other. The docker-compose.yml looks like this:

version: '3'
services:
  app:
    container_name: app-server
    image: nextcloud:fpm
    stdin_open: true
    tty: true
    restart: always
    expose:
      - '80'
      - '9000'
    networks:
      - onlyoffice
    volumes:
      - app_data:/var/www/html
  onlyoffice-document-server:
    container_name: onlyoffice-document-server
    image: onlyoffice/documentserver:latest
    stdin_open: true
    tty: true
    restart: always
    networks:
      - onlyoffice
    expose:
      - '80'
      - '443'
    volumes:
      - document_data:/var/www/onlyoffice/Data
      - document_log:/var/log/onlyoffice
  nginx:
    container_name: nginx-server
    image: nginx
    stdin_open: true
    tty: true
    restart: always
    ports:
      - 80:80
      - 443:443
    networks:
      - onlyoffice
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - app_data:/var/www/html
      - /etc/letsencrypt:/etc/letsencrypt

  elasticsearch:
    container_name: elasticsearch
    image: dsteinkopf/elasticsearch-ingest-attachment
    restart: always
    networks:
      - onlyoffice
    volumes:
      - app_data:/var/www/html
      - document_data:/var/www/onlyoffice/Data
      - ./elasticsearch.yml:/etc/elasticsearch/elasticsearch.yml

networks:
  onlyoffice:
    driver: 'bridge'
volumes:
  document_data:
  document_log:
  app_data:
  mysql_data:
  esdata1:
    driver: local

Everything gets up and running, and I can even query the Elasticsearch container from the Nextcloud container with:

sudo docker exec app-server curl -XGET 'elasticsearch:9200/?pretty'

… and it produces a sensible response.

The issue arises when I try to engage the fulltextsearch indexer.

With:

sudo docker exec --user www-data app-server php occ fulltextsearch:index --no-readline

I get the start of the indexing engine, but an error at the bottom:

user@server:~/docker-onlyoffice-nextcloud$ sudo docker exec --user www-data app-server php occ fulltextsearch:index --no-readline

Options: {"_no-readline":true}
Memory:
┌─ Indexing  ────
│ Action:
│ Provider:                      Account:
│ Document:
│ Info:
│ Title:
│ Content size:
│ Chunk:       /
│ Progress:       /
└──
┌─ Results ────
│ Result:      0/0
│ Index:
│ Status:
│ Message:
│
│
└──
┌─ Errors ────
│ Error:      0/0
│ Index:
│ Exception:
│ Message:
│
│
└──

In Connection.php line 620:

  {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"requ
  est [/nextcloud] contains unrecognized parameter: [include_type_name]"}],"t
  ype":"illegal_argument_exception","reason":"request [/nextcloud] contains u
  nrecognized parameter: [include_type_name]"},"status":400}

No indexing, no nothing. This is so irritating because I feel so close to having a genuinely useful server.

Would really appreciate some input. This is driving me nuts.

1 Like

Please post:

  • version of Nextcloud
  • version of Elasticsearch
  • result of ./occ fulltextsearch:check

Hi @Cult , thanks for taking a look at this.

Nextcloud Version: 16.0.1
Elasticsearch Version: 6.2.4, Build: ccec39f/2018-04-12T20:37:28.497551Z, JVM: 1.8.0_161

Output of fulltextsearch:check is:

user@server:~/docker-onlyoffice-nextcloud$ sudo docker exec --user www-data app-server php occ fulltextsearch:check

Full text search 1.3.4

- Search Platform:
Elasticsearch 1.3.2
{
    "elastic_host": [
        "http:\/\/elasticsearch:9200"
    ],
    "elastic_index": "nextcloud"
}

- Content Providers:
Files 1.3.2
{
    "files_local": "1",
    "files_external": "0",
    "files_group_folders": "0",
    "files_encrypted": "0",
    "files_federated": "0",
    "files_size": "100",
    "files_pdf": "1",
    "files_office": "1",
    "files_image": "0",
    "files_audio": "0"
}

There are at least 5 office files on the server, so this is a bit odd.

1 Like

Would it be possible to upgrade Elasticsearch to (at least) v6.6.x ?

I’ve given that a go, but any version later than Elasticsearch v6.6x creates another rather separate problem while compiling the docker image with the ingest attachment:

-> Downloading ingest-attachment from elastic
[=================================================] 100%??
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.sun.java2d.cmm.kcms
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.security.SecurityPermission createAccessControlContext
* java.security.SecurityPermission insertProvider
* java.security.SecurityPermission putProviderProperty.BC
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Exception in thread "main" java.lang.IllegalStateException: unable to read from standard input; is standard input open and a tty attached?
        at org.elasticsearch.cli.Terminal$SystemTerminal.readText(Terminal.java:173)
        at org.elasticsearch.plugins.PluginSecurity.prompt(PluginSecurity.java:74)
        at org.elasticsearch.plugins.PluginSecurity.confirmPolicyExceptions(PluginSecurity.java:67)
        at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:802)
        at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:776)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:231)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
The command '/bin/sh -c bin/elasticsearch-plugin install ingest-attachment' returned a non-zero code: 1

The error code causes the docker build to fail, so I can’t bring up a docker image that’s later than v6.6x and has the ingest attachment.

Yes, the second issue is more docker oriented.

We could try to make your ES v6.2 working, if you are willing to try this PR:

After patching your files, you will also need to run this occ command:

./occ config:app:set --value '1' fulltextsearch_elasticsearch es_ver_below66

Just a quick update because this comes up in top searches on the topic. I was able to get this working using the below. Note, my example uses an external nginx proxy in a separate docker-compose.

Nextcloud:latest or 17 at the time of this writing
elasticsearch:6.8.8

Had to run this inside the elasticsearch container (install ingest-attachment)…

docker exec -it nextcloud_esearch bash

/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

Indexes could then successfully occur using
docker exec --user www-data nextcloud_app_1 php occ fulltextsearch:index --no-readline.

Make sure you have also filled in and installed the needed search plugins in your nextcloud admin panel. Hint internal docker IP cant be found using docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' nextcloud_esearch

Important thing to note as have been mentioned here. Support for 6.6 or lower might not work (I used 6.8.8), support for 7+ of elasticsearch wouldn’t work for me.

Extend Elasticsearch container…

FROM elasticsearch:6.8.8

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment && \
    yum-config-manager --add-repo https://download.opensuse.org/repositories/home:/Alexander_Pozdnyakov/CentOS_7/ && \
    rpm --import https://build.opensuse.org/projects/home:Alexander_Pozdnyakov/public_key && \
    yum update -y && \
    yum install tesseract {% for language in tessreact_language_packs %}tesseract-langpack-{{language }} {% endfor %} -y && \
    yum clean all

docker build -t nc_elasticsearch

Docker Compose Yaml

version: '2'

networks:
  dmz:
    external:
      name: dmz

services:
  db:
    image: mariadb
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    volumes:
      - /srv/nextcloud/db:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: secure_password
    env_file:
      - db.env
    networks:
      - dmz

  app:
    image: nextcloud:latest
    ports:
      - 8080:80
    links:
      - db
    volumes:
      - /srv/nextcloud/app:/var/www/html
      - /srv/nextcloud_data:/var/www/html/data
    environment:
      MYSQL_HOST: db
      VIRTUAL_HOST: nextcloud.example.com
      LETSENCRYPT_HOST: nextcloud.example.com
      LETSENCRYPT_EMAIL: admin@example.com
    env_file:
      - db.env
    restart: always
    networks:
      - dmz

  esearch:
    # image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2 # 7 doesn't work with nextcloud
    # image: elasticsearch:6.8.8
    image: nc_elasticsearch:latest
    container_name: nextcloud_esearch
    environment:
      - node.name=esearch
      - discovery.type=single-node
    volumes:
      - /srv/esearch:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300
    restart: always
    links:
      - app
    networks:
      - dmz
1 Like

maybe better build your own image with the following docker file. better maintain. or?

FROM elasticsearch:{{ docker_elasticsearch_image }}

RUN bin/elasticsearch-plugin install --batch ingest-attachment && \
    yum-config-manager --add-repo https://download.opensuse.org/repositories/home:/Alexander_Pozdnyakov/CentOS_7/ && \
    rpm --import https://build.opensuse.org/projects/home:Alexander_Pozdnyakov/public_key && \
    yum update -y && \
    yum install tesseract {% for language in tessreact_language_packs %}tesseract-langpack-{{language }} {% endfor %} -y && \
    yum clean all

thats from a ansible jinja2 template. hope you can read it.

https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_c_customized_image

1 Like

Thanks!

I will play around with this. I assumed that the plugin will get wiped if I did a full rebuild and had not tackled that problem. I was assuming I would have to map a volume/dir to preserve the data, but this may be more ideal (build your own image).

My install does not account for PDF’s and tesseract. I was just concerned with plain text so good info, thanks!

all the time when you update the image and launch a new container. yes.

Final for my solution is…

FROM elasticsearch:6.8.8

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment && \
    yum-config-manager --add-repo https://download.opensuse.org/repositories/home:/Alexander_Pozdnyakov/CentOS_7/ && \
    rpm --import https://build.opensuse.org/projects/home:Alexander_Pozdnyakov/public_key && \
    yum update -y && \
    yum install tesseract {% for language in tessreact_language_packs %}tesseract-langpack-{{language }} {% endfor %} -y && \
    yum clean all

docker build -t nc_elasticsearch

Amending to the original post.

It looks like the “approved” way is to extend the image by building a new one like /u/Reiner_Nippes pointed out. I didn’t find a method to preserve plugins using volumes or persistent mount points.

Hi, I’m having issues with installing Elastic Search in Nextcloud 18.4. It seems to be a path issue with fulltextsearch looking for /nextcloud. My nextcloud docker doesnt use a subfolder.

I’ve installed the ingest module in Elasticsearch as recommended above. Elastic search fails with an error on test.

root@Nexus:/mnt/user/appdata/nextcloud/www/nextcloud/apps/fulltextsearch_elasticsearch/lib/Service# docker exec --user abc nextcloud php7 /config/www/nextcloud/occ fulltextsearch:check
Full text search 1.4.1

  • Search Platform:
    Elasticsearch 1.5.1
    {
    “elastic_host”: [
    http://192.168.1.4:9200
    ],
    “elastic_index”: “nextcloud”,
    “fields_limit”: “10000”,
    “es_ver_below66”: “0”,
    “analyzer_tokenizer”: “standard”
    }

  • Content Providers:
    Deck 1.0.1
    []
    Files 1.4.2
    {
    “files_local”: “1”,
    “files_external”: “1”,
    “files_group_folders”: “1”,
    “files_encrypted”: “0”,
    “files_federated”: “0”,
    “files_size”: “100”,
    “files_pdf”: “1”,
    “files_office”: “1”,
    “files_image”: “0”,
    “files_audio”: “0”
    }
    root@Nexus:/mnt/user/appdata/nextcloud/www/nextcloud/apps/fulltextsearch_elasticsearch/lib/Service# docker exec --user abc nextcloud php7 /config/www/nextcloud/occ fulltextsearch:test

.Testing your current setup:
Creating mocked content provider. ok
Testing mocked provider: get indexable documents. (2 items) ok
Loading search platform. (Elasticsearch) ok
Testing search platform. ok
Locking process ok
Removing test. ok
Pausing 3 seconds 1 2 3 ok
Initializing index mapping. fail
Error detected, unlocking process ok
In Connection.php line 632:

{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“requ
est [/nextcloud] contains unrecognized parameter: [include_type_name]”}],“t
ype”:“illegal_argument_exception”,“reason”:“request [/nextcloud] contains u
nrecognized parameter: [include_type_name]”},“status”:400}

fulltextsearch:test [–output [OUTPUT]] [-j|–json] [-d|–platform_delay PLATFORM_DELAY]

I would be grateful for any input on resolving this.

How/where is the tessreact_language_packs variable defined?