Docker: Unable to upgrade to version 22.0.0

Nextcloud version (eg, 20.0.5): 21.0.3.122.0.0.11
Operating system and version (eg, Ubuntu 20.04): Debian GNU/Linux 9.11 (stretch)
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.38 (Debian)
PHP version (eg, 7.4): 7.4

My Docker Compose file
  nextcloud:
    image: nextcloud:latest
    depends_on:
      - database
    environment:
      POSTGRES_DB: nextcloud
      POSTGRES_USER: nextcloud
      POSTGRES_PASSWORD: [snip]
      POSTGRES_HOST: database
    volumes:
      - /mnt/ssd/nextcloud/files:/var/www/html
    restart: unless-stopped

  database:
    image: postgres:10
    environment:
      POSTGRES_PASSWORD: [snip]
      POSTGRES_USER: nextcloud
      POSTGRES_DB: nextcloud
    volumes:
      - ./conf/nextcloud/database:/var/lib/postgresql/data
    restart: unless-stopped

The issue you are facing:

  • Nextcloud has gone into permanent maintenance mode after failing to update
  • Removing maintenance mode and accessing the website gives an update prompt
  • Both the update prompt and docker-compose exec -u www-data nextcloud php occ upgrade report an error:
php occ upgrade error
whitestrake in ~ at cadmus
➜ dc exec -u 33 nextcloud php occ upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Turned on maintenance mode
Updating database schema
Updated database
Checking for update of app accessibility in appstore
PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 262144 bytes) in /var/www/html/lib/private/Http/Client/DnsPinMiddleware.php on line 77
PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 262144 bytes) in /var/www/html/lib/private/Log/ErrorHandler.php on line 42

It never seems to get past Checking for update of app accessibility in appstore.

Is this the first time you’ve seen this error? (Y/N): Yes, all previous upgrades have been relatively painless

Steps to replicate it:

  1. docker-compose up -d nextcloud database
  2. docker-compose exec -u www-data nextcloud php occ upgrade
    OR
    Browse to website and click the update prompt
  3. Wait several minutes

The output of your Nextcloud log in Admin > Logging:

  • Not currently able to access the site

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

config.php
<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'instanceid' => [snip],
  'passwordsalt' => [snip],
  'secret' => [snip],
  'trusted_domains' =>
  array (
    0 => [snip],
  ),
  'datadirectory' => '/var/www/html/data',
  'overwrite.cli.url' => [snip],
  'overwriteprotocol' => 'https',
  'dbtype' => 'pgsql',
  'version' => '21.0.3.1',
  'dbname' => 'nextcloud',
  'dbhost' => 'database',
  'dbport' => '',
  'dbtableprefix' => '',
  'dbuser' => 'nextcloud',
  'dbpassword' => [snip],
  'installed' => true,
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpauth' => 1,
  'mail_from_address' => 'nextcloud',
  'mail_domain' => [snip],
  'mail_smtphost' => [snip],
  'mail_smtpport' => '465',
  'mail_smtpname' => [snip],
  'mail_smtppassword' => [snip],
  'maintenance' => true,
  'loglevel' => 0,
  'app_install_overwrite' =>
  array (
    0 => 'files_reader',
    1 => 'files_retention',
  ),
  'theme' => '',
);

The output of your Apache/nginx/system log in /var/log/____:

All redirect to /dev/stdout and /dev/stderr. I can grab these from Docker logs, but it’s just a bunch of PROPFINDs mostly from the last few days the Nextcloud instance has been effectively down.


I’ve also tried docker-compose exec -u www-data nextcloud php occ maintenance:repair which seemed to execute just fine and didn’t have any effect on the error.

php occ maintenance:repair results
whitestrake in ~ at cadmus
➜ dc exec -u www-data nextcloud php occ maintenance:repair
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
 - Repair MySQL collation
     - Not a mysql database -> nothing to do
 - Repair mime types
 - Clean tags and favorites
     - 0 tags of deleted users have been removed.
     - 0 tags for delete files have been removed.
     - 0 tag entries for deleted tags have been removed.
     - 0 tags with no entries have been removed.
 - Repair invalid shares
 - Move .step file of updater to backup location
 - Add move avatar background job
     - Repair step already executed
 - Add preview cleanup background jobs
 - Fix potential broken mount points
     - No mounts updated
 - Repair language codes
 - Install new core bundle components
 - Add log rotate job
 - Clear frontend caches
     - Image cache cleared
     - SCSS cache cleared
     - JS cache cleared
 - Clear every generated avatar on major updates
 - Add preview background cleanup job
 - Queue a one-time job to cleanup old backups of the updater
 - Cleanup invalid photocache files for carddav
 - Add background job to cleanup login flow v2 tokens
 - Remove potentially over exposing share links
     - No need to remove link shares.
 - Clear access cache of projects
 - Reset generated avatar flag
 - Keep legacy encryption enabled
 - Check encryption key format
 - Remove old dashboard app config data
 - Add job to cleanup the bruteforce entries
 - Queue a one-time job to check for user uploaded certificates
 - Repair DAV shares
 - Add background job to set the lookup server share state for users
 - Fix component of birthday calendars
     - 2 birthday calendars updated.
 - Regenerating birthday calendars to use new icons and fix old birthday events without year
     - Repair step already executed
 - Fix broken values of calendar objects
    0 [->--------------------------]
 - Registering building of calendar search index as background job
     - Repair step already executed
 - Register building of social profile search index as background job
 - Registering background jobs to update cache for webcal calendars
     - Added 0 background jobs to update webcal calendars
 - Registering building of calendar reminder index as background job
     - Repair step already executed
 - Clean up orphan event and contact data
     - 0 events without a calendar have been cleaned up
     - 0 properties without an events have been cleaned up
     - 0 changes without a calendar have been cleaned up
     - 0 cached events without a calendar subscription have been cleaned up
     - 0 changes without a calendar subscription have been cleaned up
     - 0 contacts without an addressbook have been cleaned up
     - 0 properties without a contact have been cleaned up
     - 0 changes without an addressbook have been cleaned up
 - Remove activity entries of private events
     - Removed 0 activity entries
 - Fix the share type of guest shares when migrating from ownCloud
 - Copy the share password into the dedicated column
 - Set existing shares as accepted
 - Clean up meta table
 - Update OAuth token expiration times
 - Switches from default updater server to the customer one if a valid subscription is available
     - Repair step already executed
 - Send an admin notification if monthly report is disabled
 - Add background job to check for backup codes
 - Populating added database structures for workflows

I was unable to downgrade from nextcloud:latest to nextcloud:21; the container exits with this log output:

Can't start Nextcloud because the version of the data (22.0.0.11) is higher than the docker image version (21.0.3.1) and downgrading is not supported. Are you sure you have pulled the newest image version?

What troubleshooting options do I have from here?

as you referenced this issue it looks you searched the forum already :+1:

from my understanding you hit php memory limits - did you try to edit memory limits in your php.ini?

What’s the best way to do that in the Docker image, and what’s the recommended amount to set to?

I have the exact same issue.
You can set the enviroment variable PHP_MEMORY_LIMIT for the container though changing from the default of 512M to 1024M didn’t fix the issue for me.

I’ve got a bit of RAM to spare, so opted to give it 4GB to see how it would go. Edited my docker-compose.yml with a new environment variable and composed the service back up.

New compose file
  nextcloud:
    image: nextcloud:latest
    depends_on:
      - database
    environment:
      POSTGRES_DB: nextcloud
      POSTGRES_USER: nextcloud
      POSTGRES_PASSWORD: [snip]
      POSTGRES_HOST: database
      PHP_MEMORY_LIMIT: 4096M
    volumes:
      - /mnt/ssd/nextcloud/files:/var/www/html
    restart: unless-stopped

  database:
    image: postgres:10
    environment:
      POSTGRES_PASSWORD: [snip]
      POSTGRES_USER: nextcloud
      POSTGRES_DB: nextcloud
    volumes:
      - ./conf/nextcloud/database:/var/lib/postgresql/data
    restart: unless-stopped

Confirmed PHP_MEMORY_LIMIT=4096M inside the container (with docker-compose exec -u www-data nextcloud env) and ran the upgrade again (docker-compose exec -u www-data nextcloud php occ upgrade).

After approx. 40min, the upgrade exited with the same error messages.

PHP Fatal error: Allowed memory size of 4294967296 bytes exhausted (tried to allocate 262144 bytes) in /var/www/html/lib/private/Http/Client/DnsPinMiddleware.php on line 77
PHP Fatal error: Allowed memory size of 4294967296 bytes exhausted (tried to allocate 262144 bytes) in /var/www/html/lib/private/Log/ErrorHandler.php on line 42

I see several posting of this same php error on the forum. Check this one out:

Thanks for the link.

Presumably I, too, have APCu specified in my config.php:

$ grep "APCu" config/config.php
  'memcache.local' => '\\OC\\Memcache\\APCu',

Although I never added it myself, per se - I’m assuming it’s been there since I first pulled the official Nextcloud image.

I don’t appear to have any file as specified in the linked post. There is no /etc/php/ folder at all.

The official Docker container has a $PHP_INI_DIR of /usr/local/etc/php. Inside I can see ./conf.d/docker-php-ext-apcu.ini:

$ cat /usr/local/etc/php/conf.d/docker-php-ext-apcu.ini
extension=apcu
apc.enable_cli=1

So, as far as I can see, it doesn’t appear to be the case that I am missing apc.enable_cli=1 after all.

Same problem, same issues.
Running Nextcloud in docker, but found: Bug in DnsPinMiddleware.php, unable to resolve hostname · Issue #27870 · nextcloud/server · GitHub

Not sure if I did the absolute correct thing, but did as mentioned in: Link to comment that solved it for me

After that, occ upgrade worked quickly and ran successfully

3 Likes

THANK YOU! I’ve been pulling my hair out for DAYS over this and this fixed it. The upgrade ran INSTANTLY.

Thank you @mrund!

Confirming the solution for me was to edit ./lib/private/Http/Client/DnsPinMiddleware.php in my Nextcloud files and replace:

$dnsTypes = [DNS_A, DNS_AAAA, DNS_CNAME];

with:

$dnsTypes = [DNS_A];

Then run php occ upgrade again.

Absolutely disappointed that my instance has been down for so long because of a bug in DNS. I was about to restore file and database backups from July 5.

Lesson learned - for the official nextcloud Docker image, latest is not stable, pin versions. A shame, because it’s been stable ever since I started using it.

(And keep backups if you don’t already!)

1 Like