Docker-compose restore from backup of db and data results in errors

ℹ️ Support
,
1
Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 18.0.2): 17.0.4
Operating system and version (eg, Ubuntu 20.04): docker-compose fpm-alpine image
Apache or nginx version (eg, Apache 2.4.25): nginx/1.15.9
PHP version (eg, 7.1): 7.3.20

The issue you are facing:
tldr; I am unable to restore from a backup of the db and data volumes.

I have multiple backups of my nextcloud each of which consists of a tarball of the db and data volumes which I create with this command (from a script I wrote to expedite backup) for the db.
Preformatted textdocker run --rm --volume $database:/db --volume $OUTDIR:/backup ubuntu tar cvf backup/db.tar /db
For the data folder…
docker run --rm --volume $nextcloud:/nextcloud --volume $OUTDIR:/backup ubuntu tar cvf backup/nextcloud.tar /nextcloud

After doing a backup of my db and data volumes, I tried to update my nextcloud instance from v17.0.4 to v17.0.8 (I had a notification on my admin panel that said 17.0.8 was available). What I didn’t realize was that I was behind by a couple major releases (I believe v19 was the latest docker image at the time). Obviously nextcloud didn’t like that. After trying to recover via occ I made the decision to start from scratch via my restore script which executes the following commands;

docker-compose down
docker volume prune
docker-compose up -d
docker-compose stop
docker run --rm --volumes-from $database -v $INDIR:/backup ubuntu bash -c "cd /var/lib/mysql && tar -xvf /backup/db.tar --strip 1"
docker run --rm --volumes-from $nextcloud -v $INDIR:/backup ubuntu bash -c "cd /var/www/html && tar -xvf /backup/nextcloud.tar --strip 1"
docker-compose up -d

Before I ran the script, I changed my docker-compose file to refer to the 17-fpm-alpine image to avoid loading the latest version again like it did when I was updating.

The result was this error:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

The only log that shows anything useful is the db log which shows the following:

2020-08-17 18:34:35+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.4+maria~focal started.
2020-08-17 18:34:36+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-08-17 18:34:36+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.4+maria~focal started.
2020-08-17 18:34:37 0 [Note] mysqld (mysqld 10.5.4-MariaDB-1:10.5.4+maria~focal) starting as process 1 ...
2020-08-17 18:34:37 0 [Warning] You need to use --log-bin to make --binlog-format work.
2020-08-17 18:34:37 0 [Note] InnoDB: Using Linux native AIO
2020-08-17 18:34:37 0 [Note] InnoDB: Uses event mutexes
2020-08-17 18:34:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-08-17 18:34:37 0 [Note] InnoDB: Number of pools: 1
2020-08-17 18:34:37 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
2020-08-17 18:34:37 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2020-08-17 18:34:37 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-17 18:34:37 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-17 18:34:37 0 [Note] InnoDB: 128 rollback segments are active.
2020-08-17 18:34:37 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-08-17 18:34:37 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-08-17 18:34:37 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-08-17 18:34:37 0 [Note] InnoDB: 10.5.4 started; log sequence number 27382677626; transaction id 39742994
2020-08-17 18:34:37 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-08-17 18:34:37 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-08-17 18:34:37 0 [Note] Server socket created on IP: '::'.
2020-08-17 18:34:37 0 [Warning] 'user' entry 'root@ddc9f53735cf' ignored in --skip-name-resolve mode.
2020-08-17 18:34:37 0 [Warning] 'proxies_priv' entry '@% root@8e20972e13d2' ignored in --skip-name-resolve mode.
2020-08-17 18:34:37 0 [Note] Reading of all Master_info entries succeeded
2020-08-17 18:34:37 0 [Note] Added new Master_info '' to hash table
2020-08-17 18:34:37 0 [Note] mysqld: ready for connections.
Version: '10.5.4-MariaDB-1:10.5.4+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2020-08-17 18:34:37 0 [Note] InnoDB: Buffer pool(s) load completed at 200817 18:34:37
2020-08-17 18:34:50 3 [Warning] Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:50 4 [Warning] Aborted connection 4 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:50 5 [Warning] Aborted connection 5 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:50 6 [Warning] Aborted connection 6 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:50 7 [Warning] Aborted connection 7 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:50 8 [Warning] Aborted connection 8 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:52 9 [Warning] Aborted connection 9 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:52 10 [Warning] Aborted connection 10 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:52 11 [Warning] Aborted connection 11 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:52 12 [Warning] Aborted connection 12 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:52 13 [Warning] Aborted connection 13 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)
2020-08-17 18:34:52 14 [Warning] Aborted connection 14 to db: 'unconnected' user: 'unauthenticated' host: '172.25.0.3' (This connection closed normally without authentication)

The interesting part here is 'user' entry 'root@....' ignored in --skip-name-resolve mode
It seems like the default username / password etc defined in the docker-compose file are being ignored.
I have checked that the database is loaded in the volume and it is. All tables (as far as I can tell) are there. There is an admin user with privileges to the database. However I have been getting the error (in the mariadb logs) that ‘oc_admin’@‘nextcloud’ access denied.

The strange thing is that I have run this backup -> restore in testing (with smaller data and db files; the data file is 118G) and it works perfectly every time. I can’t understand how the update would affect it since the backup was done before this issue occurred.

I’m not sure what I’m missing… Any ideas?

Here is a copy of my docker-compose file:

version: '3'

services:
  db:
    image: mariadb
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    volumes:
      - db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=
    env_file:
      - db.env

  app:
    image: nextcloud:fpm-alpine
    restart: always
    volumes:
      - nextcloud:/var/www/html
    environment:
      - MYSQL_HOST=db
    env_file:
      - db.env
    depends_on:
      - db

  web:
    build: ./web
    restart: always
    volumes:
      - nextcloud:/var/www/html:ro
    environment:
      - VIRTUAL_HOST=
    depends_on:
      - app
    networks:
      - proxy-tier
      - default

  proxy:
    build: ./proxy
    restart: always
    ports:
      - 80:80
      - 443:443
    volumes:
      - certs:/etc/nginx/certs:ro
      - vhost.d:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro
    networks:
      - proxy-tier
    depends_on:
      - omgwtfssl

  omgwtfssl:
    image: paulczar/omgwtfssl
    restart: "no"
    volumes:
      - certs:/certs
    environment:
      - SSL_SUBJECT=servhostname.local
      - CA_SUBJECT=my@example.com
      - SSL_KEY=/certs/servhostname.local.key
      - SSL_CSR=/certs/servhostname.local.csr
      - SSL_CERT=/certs/servhostname.local.crt
    networks:
      - proxy-tier

volumes:
  db:
  nextcloud:
  certs:
  vhost.d:
  html:

networks:
  proxy-tier:

Is this the first time you’ve seen this error? (Y/N):
Yes

Steps to replicate it:

  1. make backup of db and data volumes
  2. attempt to upgrade skipping major releases
  3. restore backup using steps mentioned above
  4. pull your hair out
  5. bang head on the wall
  6. loose sleep
  7. ask for help

The output of your Nextcloud log in Admin > Logging:

NA

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

See docker-compose.yml above (no?)

The output of your Apache/nginx/system log in /var/log/____:

NA
2

Solved!

It turns out that I was getting the error ‘access denied for ‘oc-admin’@’…’ using password ‘YES’ because the salt and hash were different after creating a new instance of nextcloud.

The steps to restore in my case were as follows:

  1. inspect backup/nextcloud/config/config.php in the nextcloud folder of the backup to verify version of nextcloud that was running when the backup was made.
  2. set the version in docker-compose.yml to the same version as time of backup.
  3. docker-compose down
  4. docker volume prune
  5. docker-compose up -d
  6. create new admin with the same username and password as the old admin (I’m not sure how vital it is that the password be the same for the new admin as it was for the old in the case that the username for the admin is the same as it was in the backup).
  7. docker exec -it nextcloud_app_1 sh
  8. cat config/config.php
  9. copy instanceid, passwordsalt, secret, and dbpassword
  10. docker-compose stop
  11. create config.tmp.php to merge old and new configs
  12. keep everything from backup’s config.php replacing only the instanceid, passwordsalt, secret, and dbpassword with those you copied from the newly created instance.
  13. write over the db volume with the backup *1
  14. write over the data volume with the backup *2
  15. copy the newly created config.tmp.php into the data volume *3
  16. remove /var/www/html/config/config.php from data volume *4
  17. rename config.tmp.php to config.php *5
  18. docker exec -it nextcloud_app_1 sh to spawn a shell inside the app container thus giving access to the data volume. You will be at /var/www/html.
  19. chown www-html config.php to give nextcloud access to this file (you copied it as root so you need to change the permissions back to the www-html user)
  20. exit (to exit the shell)
  21. docker-compose up -d
  22. login as usual

*1 -> sudo docker run --rm --volumes-from nextcloud_db_1 -v $PWD/backup:/backup ubuntu bash -c "cd /var/lib/mysql && tar -xvzf /backup/db.tar.gz --strip 1"

*2 -> sudo docker run --rm --volumes-from nextcloud_app_1 -v $PWD/backup:/backup ubuntu bash -c "cd /var/www/html && tar -xvzf /backup/nextcloud.tar.gz --strip 1"

*3 -> sudo docker run --rm --volumes-from nextcloud_app_1 -v $PWD/backup:/backup ubuntu bash -c "cd /var/www/html/config && cp /backup/config.tmp.php ./"

*4 -> sudo docker run --rm --volumes-from nextcloud_app_1 -v $PWD/backup:/backup ubuntu bash -c "cd /var/www/html/config && rm ./config.php"

*5 -> sudo docker run --rm --volumes-from nextcloud_app_1 -v $PWD/backup:/backup ubuntu bash -c "cd /var/www/html/config && mv ./config.tmp.php ./config.php"