Nextcloud and Elasticsearch in Docker

🍱 Features & apps fulltextsearch
1

Guys this has me tearing my hair out and I’d love some input.

I’m trying to create a server that has Nextcloud, OnlyOffice and Elasticsearch running together in separate docker containers, speaking to each other. The docker-compose.yml looks like this:

version: '3'
services:
  app:
    container_name: app-server
    image: nextcloud:fpm
    stdin_open: true
    tty: true
    restart: always
    expose:
      - '80'
      - '9000'
    networks:
      - onlyoffice
    volumes:
      - app_data:/var/www/html
  onlyoffice-document-server:
    container_name: onlyoffice-document-server
    image: onlyoffice/documentserver:latest
    stdin_open: true
    tty: true
    restart: always
    networks:
      - onlyoffice
    expose:
      - '80'
      - '443'
    volumes:
      - document_data:/var/www/onlyoffice/Data
      - document_log:/var/log/onlyoffice
  nginx:
    container_name: nginx-server
    image: nginx
    stdin_open: true
    tty: true
    restart: always
    ports:
      - 80:80
      - 443:443
    networks:
      - onlyoffice
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - app_data:/var/www/html
      - /etc/letsencrypt:/etc/letsencrypt

  elasticsearch:
    container_name: elasticsearch
    image: dsteinkopf/elasticsearch-ingest-attachment
    restart: always
    networks:
      - onlyoffice
    volumes:
      - app_data:/var/www/html
      - document_data:/var/www/onlyoffice/Data
      - ./elasticsearch.yml:/etc/elasticsearch/elasticsearch.yml

networks:
  onlyoffice:
    driver: 'bridge'
volumes:
  document_data:
  document_log:
  app_data:
  mysql_data:
  esdata1:
    driver: local

Everything gets up and running, and I can even query the Elasticsearch container from the Nextcloud container with:

sudo docker exec app-server curl -XGET 'elasticsearch:9200/?pretty'

… and it produces a sensible response.

The issue arises when I try to engage the fulltextsearch indexer.

With:

sudo docker exec --user www-data app-server php occ fulltextsearch:index --no-readline

I get the start of the indexing engine, but an error at the bottom:

user@server:~/docker-onlyoffice-nextcloud$ sudo docker exec --user www-data app-server php occ fulltextsearch:index --no-readline

Options: {"_no-readline":true}
Memory:
┌─ Indexing  ────
│ Action:
│ Provider:                      Account:
│ Document:
│ Info:
│ Title:
│ Content size:
│ Chunk:       /
│ Progress:       /
└──
┌─ Results ────
│ Result:      0/0
│ Index:
│ Status:
│ Message:
│
│
└──
┌─ Errors ────
│ Error:      0/0
│ Index:
│ Exception:
│ Message:
│
│
└──

In Connection.php line 620:

  {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"requ
  est [/nextcloud] contains unrecognized parameter: [include_type_name]"}],"t
  ype":"illegal_argument_exception","reason":"request [/nextcloud] contains u
  nrecognized parameter: [include_type_name]"},"status":400}

No indexing, no nothing. This is so irritating because I feel so close to having a genuinely useful server.

Would really appreciate some input. This is driving me nuts.

1 Like
2

Please post:

  • version of Nextcloud
  • version of Elasticsearch
  • result of ./occ fulltextsearch:check
3

Hi @Cult , thanks for taking a look at this.

Nextcloud Version: 16.0.1
Elasticsearch Version: 6.2.4, Build: ccec39f/2018-04-12T20:37:28.497551Z, JVM: 1.8.0_161

Output of fulltextsearch:check is:

user@server:~/docker-onlyoffice-nextcloud$ sudo docker exec --user www-data app-server php occ fulltextsearch:check

Full text search 1.3.4

- Search Platform:
Elasticsearch 1.3.2
{
    "elastic_host": [
        "http:\/\/elasticsearch:9200"
    ],
    "elastic_index": "nextcloud"
}

- Content Providers:
Files 1.3.2
{
    "files_local": "1",
    "files_external": "0",
    "files_group_folders": "0",
    "files_encrypted": "0",
    "files_federated": "0",
    "files_size": "100",
    "files_pdf": "1",
    "files_office": "1",
    "files_image": "0",
    "files_audio": "0"
}

There are at least 5 office files on the server, so this is a bit odd.

1 Like
4

Would it be possible to upgrade Elasticsearch to (at least) v6.6.x ?

5

I’ve given that a go, but any version later than Elasticsearch v6.6x creates another rather separate problem while compiling the docker image with the ingest attachment:

-> Downloading ingest-attachment from elastic
[=================================================] 100%??
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.sun.java2d.cmm.kcms
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.security.SecurityPermission createAccessControlContext
* java.security.SecurityPermission insertProvider
* java.security.SecurityPermission putProviderProperty.BC
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Exception in thread "main" java.lang.IllegalStateException: unable to read from standard input; is standard input open and a tty attached?
        at org.elasticsearch.cli.Terminal$SystemTerminal.readText(Terminal.java:173)
        at org.elasticsearch.plugins.PluginSecurity.prompt(PluginSecurity.java:74)
        at org.elasticsearch.plugins.PluginSecurity.confirmPolicyExceptions(PluginSecurity.java:67)
        at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:802)
        at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:776)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:231)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
The command '/bin/sh -c bin/elasticsearch-plugin install ingest-attachment' returned a non-zero code: 1

The error code causes the docker build to fail, so I can’t bring up a docker image that’s later than v6.6x and has the ingest attachment.

6

Yes, the second issue is more docker oriented.

We could try to make your ES v6.2 working, if you are willing to try this PR:

After patching your files, you will also need to run this occ command:

./occ config:app:set --value '1' fulltextsearch_elasticsearch es_ver_below66

7

Just a quick update because this comes up in top searches on the topic. I was able to get this working using the below. Note, my example uses an external nginx proxy in a separate docker-compose.

Nextcloud:latest or 17 at the time of this writing
elasticsearch:6.8.8

Had to run this inside the elasticsearch container (install ingest-attachment)…

docker exec -it nextcloud_esearch bash

/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

Indexes could then successfully occur using
docker exec --user www-data nextcloud_app_1 php occ fulltextsearch:index --no-readline.

Make sure you have also filled in and installed the needed search plugins in your nextcloud admin panel. Hint internal docker IP cant be found using docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' nextcloud_esearch

Important thing to note as have been mentioned here. Support for 6.6 or lower might not work (I used 6.8.8), support for 7+ of elasticsearch wouldn’t work for me.

Extend Elasticsearch container…

FROM elasticsearch:6.8.8

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment && \
    yum-config-manager --add-repo https://download.opensuse.org/repositories/home:/Alexander_Pozdnyakov/CentOS_7/ && \
    rpm --import https://build.opensuse.org/projects/home:Alexander_Pozdnyakov/public_key && \
    yum update -y && \
    yum install tesseract {% for language in tessreact_language_packs %}tesseract-langpack-{{language }} {% endfor %} -y && \
    yum clean all

docker build -t nc_elasticsearch

Docker Compose Yaml

version: '2'

networks:
  dmz:
    external:
      name: dmz

services:
  db:
    image: mariadb
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    volumes:
      - /srv/nextcloud/db:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: secure_password
    env_file:
      - db.env
    networks:
      - dmz

  app:
    image: nextcloud:latest
    ports:
      - 8080:80
    links:
      - db
    volumes:
      - /srv/nextcloud/app:/var/www/html
      - /srv/nextcloud_data:/var/www/html/data
    environment:
      MYSQL_HOST: db
      VIRTUAL_HOST: nextcloud.example.com
      LETSENCRYPT_HOST: nextcloud.example.com
      LETSENCRYPT_EMAIL: admin@example.com
    env_file:
      - db.env
    restart: always
    networks:
      - dmz

  esearch:
    # image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2 # 7 doesn't work with nextcloud
    # image: elasticsearch:6.8.8
    image: nc_elasticsearch:latest
    container_name: nextcloud_esearch
    environment:
      - node.name=esearch
      - discovery.type=single-node
    volumes:
      - /srv/esearch:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300
    restart: always
    links:
      - app
    networks:
      - dmz
1 Like
8

maybe better build your own image with the following docker file. better maintain. or?

FROM elasticsearch:{{ docker_elasticsearch_image }}

RUN bin/elasticsearch-plugin install --batch ingest-attachment && \
    yum-config-manager --add-repo https://download.opensuse.org/repositories/home:/Alexander_Pozdnyakov/CentOS_7/ && \
    rpm --import https://build.opensuse.org/projects/home:Alexander_Pozdnyakov/public_key && \
    yum update -y && \
    yum install tesseract {% for language in tessreact_language_packs %}tesseract-langpack-{{language }} {% endfor %} -y && \
    yum clean all

thats from a ansible jinja2 template. hope you can read it.

1 Like
9

Thanks!

I will play around with this. I assumed that the plugin will get wiped if I did a full rebuild and had not tackled that problem. I was assuming I would have to map a volume/dir to preserve the data, but this may be more ideal (build your own image).

My install does not account for PDF’s and tesseract. I was just concerned with plain text so good info, thanks!

10

all the time when you update the image and launch a new container. yes.

11

Final for my solution is…

FROM elasticsearch:6.8.8

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment && \
    yum-config-manager --add-repo https://download.opensuse.org/repositories/home:/Alexander_Pozdnyakov/CentOS_7/ && \
    rpm --import https://build.opensuse.org/projects/home:Alexander_Pozdnyakov/public_key && \
    yum update -y && \
    yum install tesseract {% for language in tessreact_language_packs %}tesseract-langpack-{{language }} {% endfor %} -y && \
    yum clean all

docker build -t nc_elasticsearch

Amending to the original post.

12

It looks like the “approved” way is to extend the image by building a new one like /u/Reiner_Nippes pointed out. I didn’t find a method to preserve plugins using volumes or persistent mount points.

13

Hi, I’m having issues with installing Elastic Search in Nextcloud 18.4. It seems to be a path issue with fulltextsearch looking for /nextcloud. My nextcloud docker doesnt use a subfolder.

I’ve installed the ingest module in Elasticsearch as recommended above. Elastic search fails with an error on test.

root@Nexus:/mnt/user/appdata/nextcloud/www/nextcloud/apps/fulltextsearch_elasticsearch/lib/Service# docker exec --user abc nextcloud php7 /config/www/nextcloud/occ fulltextsearch:check
Full text search 1.4.1

  • Search Platform:
    Elasticsearch 1.5.1
    {
    “elastic_host”: [
    “http://192.168.1.4:9200”
    ],
    “elastic_index”: “nextcloud”,
    “fields_limit”: “10000”,
    “es_ver_below66”: “0”,
    “analyzer_tokenizer”: “standard”
    }

  • Content Providers:
    Deck 1.0.1
    []
    Files 1.4.2
    {
    “files_local”: “1”,
    “files_external”: “1”,
    “files_group_folders”: “1”,
    “files_encrypted”: “0”,
    “files_federated”: “0”,
    “files_size”: “100”,
    “files_pdf”: “1”,
    “files_office”: “1”,
    “files_image”: “0”,
    “files_audio”: “0”
    }
    root@Nexus:/mnt/user/appdata/nextcloud/www/nextcloud/apps/fulltextsearch_elasticsearch/lib/Service# docker exec --user abc nextcloud php7 /config/www/nextcloud/occ fulltextsearch:test

.Testing your current setup:
Creating mocked content provider. ok
Testing mocked provider: get indexable documents. (2 items) ok
Loading search platform. (Elasticsearch) ok
Testing search platform. ok
Locking process ok
Removing test. ok
Pausing 3 seconds 1 2 3 ok
Initializing index mapping. fail
Error detected, unlocking process ok
In Connection.php line 632:

{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“requ
est [/nextcloud] contains unrecognized parameter: [include_type_name]”}],“t
ype”:“illegal_argument_exception”,“reason”:“request [/nextcloud] contains u
nrecognized parameter: [include_type_name]”},“status”:400}

fulltextsearch:test [–output [OUTPUT]] [-j|–json] [-d|–platform_delay PLATFORM_DELAY]

I would be grateful for any input on resolving this.

15

How/where is the tessreact_language_packs variable defined?

16

Consegui instalar usando esse tutorial