Nextcloud 16 upgrade hangs

Nextcloud version: 16.0.0.9
Operating system and version: Kubernetes 1.13.1/Debian (Linux nextcloud-dcf969ffc-5mngs 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux)
Apache or nginx version (eg, Apache 2.4.25): The one included in the nextcloud:16.0.0 image
PHP version (eg, 7.1): The one included in the nextcloud:16.0.0 image

The issue you are facing:

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Change the Docker image from 15.0.7 to 16.0.0
  2. Let the normal startup (rsync) happen
  3. Watch it start to do the upgrade, with the log
Set log level to debug

Nothing else happens; it just hangs.
The output of your Nextcloud log in Admin > Logging:

$ ./occ upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '[redacted]',
  'passwordsalt' => '[redacted]',
  'trusted_domains' => 
  array (
    0 => '[redacted]',
    1 => 'httpd',
  ),
  'datadirectory' => '/var/lib/nextcloud',
  'objectstore' => 
  array (
    'class' => 'OC\\Files\\ObjectStore\\S3',
    'arguments' => 
    array (
      'bucket' => '[redacted]',
      'autocreate' => true,
      'key' => '[redacted]',
      'secret' => '[redacted]',
      'hostname' => '[redacted]',
      'port' => 443,
      'use_ssl' => true,
      'use_path_style' => false,
    ),
  ),
  'version' => '15.0.7.0',
  'dbtype' => 'pgsql',
  'dbhost' => '[redacted]',
  'dbname' => '[redacted]',
  'dbuser' => '[redacted]',
  'dbpassword' => '[redacted]',
  'dbtableprefix' => 'oc_',
  'installed' => true,
  'default_language' => 'en',
  'defaultapp' => 'files',
  'knowledgebaseenabled' => true,
  'enable_avatars' => true,
  'allow_user_to_change_display_name' => true,
  'remember_login_cookie_lifetime' => 1296000,
  'session_lifetime' => 86400,
  'session_keepalive' => true,
  'token_auth_enforced' => false,
  'auth.bruteforce.protection.enabled' => false,
  'skeletondirectory' => '',
  'lost_password_link' => '',
  'mail_domain' => '[redacted]',
  'mail_from_address' => '[redacted]',
  'mail_smtpdebug' => false,
  'mail_smtpmode' => '[redacted]',
  'mail_smtphost' => '[redacted]',
  'mail_smtpport' => '[redacted]',
  'mail_smtptimeout' => 10,
  'mail_smtpauth' => 1,
  'mail_smtpauthtype' => 'PLAIN',
  'mail_smtpname' => '[redacted]',
  'mail_smtppassword' => '[redacted]',
  'overwritehost' => '[redacted]',
  'overwriteprotocol' => 'https',
  'overwritewebroot' => '/',
  'overwritecondaddr' => '',
  'overwrite.cli.url' => '[redacted]',
  'htaccess.RewriteBase' => '/',
  'trashbin_retention_obligation' => 'auto',
  'versions_retention_obligation' => 'auto',
  'appcodechecker' => false,
  'updatechecker' => true,
  'updater.server.url' => 'https://updates.nextcloud.com/updater_server/',
  'has_internet_connection' => true,
  'check_for_working_webdav' => true,
  'check_for_working_wellknown_setup' => true,
  'check_for_working_htaccess' => true,
  'config_is_read_only' => false,
  'log_type' => 'owncloud',
  'logfile' => '/tmp/nextcloud.log',
  'loglevel' => 2,
  'syslog_tag' => 'Nextcloud',
  'log.condition' => 
  array (
    'shared_secret' => '[redacted]',
    'users' => 
    array (
      0 => 'sample-user',
    ),
    'apps' => 
    array (
      0 => 'files',
    ),
  ),
  'logdateformat' => 'F d, Y H:i:s',
  'logtimezone' => 'Europe/Berlin',
  'log_query' => false,
  'cron_log' => true,
  'log_rotate_size' => false,
  'customclient_desktop' => 'https://nextcloud.com/install/',
  'customclient_android' => 'https://play.google.com/store/apps/details?id=com.nextcloud.client',
  'customclient_ios' => 'https://itunes.apple.com/us/app/nextcloud/id1125420102?mt=8',
  'appstoreenabled' => true,
  'appstoreurl' => 'https://apps.nextcloud.com/api/v0',
  'appstore.experimental.enabled' => true,
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'enable_previews' => true,
  'preview_max_x' => 2048,
  'preview_max_y' => 2048,
  'preview_max_scale_factor' => 10,
  'preview_max_filesize_image' => 50,
  'preview_libreoffice_path' => '/usr/bin/libreoffice',
  'preview_office_cl_parameters' => ' --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir ',
  'enabledPreviewProviders' => 
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\GIF',
    3 => 'OC\\Preview\\BMP',
    4 => 'OC\\Preview\\XBitmap',
    5 => 'OC\\Preview\\MP3',
    6 => 'OC\\Preview\\TXT',
    7 => 'OC\\Preview\\MarkDown',
    8 => 'OC\\Preview\\PDF',
  ),
  'ldapUserCleanupInterval' => 51,
  'comments.managerFactory' => '\\OC\\Comments\\ManagerFactory',
  'systemtags.managerFactory' => '\\OC\\SystemTag\\ManagerFactory',
  'maintenance' => false,
  'singleuser' => false,
  'enable_certificate_management' => false,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => 'redis',
    'port' => 6379,
    'timeout' => 0,
    'password' => '',
    'dbindex' => 0,
  ),
  'memcached_servers' => 
  array (
    0 => 
    array (
      0 => 'memcached',
      1 => 11211,
    ),
  ),
  'memcached_options' => 
  array (
    14 => 50,
    15 => 50,
    19 => 50,
    20 => 50,
    8 => 50,
    -1001 => true,
    16 => true,
    18 => true,
  ),
  'cache_path' => '',
  'cache_chunk_gc_ttl' => 86400,
  'sharing.managerFactory' => '\\OC\\Share20\\ProviderFactory',
  'sqlite.journal_mode' => 'DELETE',
  'supportedDatabases' => 
  array (
    0 => 'sqlite',
    1 => 'mysql',
    2 => 'pgsql',
    3 => 'oci',
  ),
  'tempdirectory' => '/tmp',
  'hashingCost' => 10,
  'blacklisted_files' => 
  array (
    0 => '.htaccess',
  ),
  'share_folder' => '/',
  'theme' => '',
  'cipher' => 'AES-256-CFB',
  'minimum.supported.desktop.version' => '1.7.0',
  'quota_include_external_storage' => false,
  'filesystem_check_changes' => 0,
  'part_file_in_storage' => true,
  'mount_file' => '/var/www/nextcloud/data/mount.json',
  'filesystem_cache_readonly' => false,
  'secret' => '',
  'trusted_proxies' => 
  array (
    0 => '203.0.113.45',
    1 => '198.51.100.128',
  ),
  'forwarded_for_headers' => 
  array (
    0 => 'HTTP_X_FORWARDED',
    1 => 'HTTP_FORWARDED_FOR',
  ),
  'max_filesize_animated_gifs_public_sharing' => 10,
  'filelocking.enabled' => true,
  'filelocking.ttl' => 3600,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'upgrade.disable-web' => false,
  'debug' => false,
  'data-fingerprint' => '',
  'updater.release.channel' => 'stable',
  'mail_smtpsecure' => 'tls',
  'mail_sendmailmode' => 'smtp',
);

The output of your Apache/nginx/system log in /var/log/____:

Nothing in there worth sharing.

I couldn’t figure out the problem and eventually downgraded back to 15.0.7, which I could do by editing version.php and changing the installed version to 15.0.7.0 and running /entrypoint.sh apache. This synced the 15.0.7 files back into the /var/www/html directory and effectively recovered my server. I’ll try again off-production by using a snapshot of the database and files and seeing if I can upgrade that.

My question for the gang is, is there a lockfile that gets generated during the ./occ upgrade process that could be blocking subsequent upgrade attempts from even starting? I couldn’t find any evidence of a lockfile, but it would make sense when running in a scaled environment where there are multiple containers serving the same instance.

Just in general, the Nextcloud logfile can be helpful. Just one thing: You seem to have a lot of stuff in your config.php, normally you start with the default config.php and just add things you need to set (that are different from default). So perhaps it is the case for you, but in the past, some tried to add a lot of stuff from the sample file. Could also be related to the docker image.