Internal Server Error - after upgrade to NC16 / missing appdata files

Hello there,

I am running into this issue since I’ve upgraded from NC 15 to 16. I did manual backups of my NC installation and DB; also the upgrade process was done manually.

Pardon the unsplash bg.

This is my NC log taken from my configured and unchanged data dir (debug):
I only included lines with the request ID shown in the error screen.

# sudo -u www-data php -f occ status
Nextcloud is in maintenance mode - no apps have been loaded

  - installed: true
  - version:
  - versionstring: 16.0.3
  - edition: 

PHP 7.3.4-2 (fpm-fcgi) (built: Apr 13 2019 19:05:48)
Debian 4.19.37-5 (2019-06-19)

This is my nginx error.log, that timeout only occured once.

# cat /var/log/nginx/error.log
2019/07/21 13:23:09 [error] 25632#25632: *159 upstream timed out (110: Connection timed out) while reading response header from upstream, client:, server: uwu, request: "GET /apps/files/ HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.3-fpm.sock", host: "uwu"

Did you make changes in PHP as well?

No, only to config/config.php.

Could you please post your config file? Please clear the sensitive information in it to not compromise your instance. What changes in it have been done?


# cat config.php 
$CONFIG = array (
  'instanceid' => 'oce3nnfrkrfe',
  'passwordsalt' => 'removed',
  'secret' => 'removed',
  'trusted_domains' => 
  array (
    0 => 'uwu',
  'datadirectory' => '/opt/nextcloud/',
  'overwrite.cli.url' => 'https://uwu',
  'dbtype' => 'mysql',
  'version' => '',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_yvelia',
  'dbpassword' => 'removed',
  'installed' => true,
  'mail_from_address' => 'cloud',
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_domain' => 'wu',
  'maintenance' => true,
  '' => 'beta',
  'loglevel' => '0',
  'theme' => '',
  'logfile' => '/opt/nextcloud-local/nextcloud.log',
  'mysql.utf8mb4' => true,
  'data-fingerprint' => 'abf399f07057787c54c20c2d748d9b4c',

What are the mentioned changes?

Only log level and file path, when I moved it out of the data-dir.

  1. 21T11:45:36+00:00",“remoteAddr”:“”,“user”:“yvelia”,“app”:“PHP”,“method”:“GET”,“url”:"/apps/files/",“message”:“file_get_contents(/opt/nextcloud/appdata_oce3nnfrkrfe/css/icons/icons-list.template): failed to open stream: No such file or directory at /srv/nextcloud/lib/private/Files/Storage/Local.php#217”,“userAgent”:“Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”,“version”:“”}

  2. {“reqId”:“w2BTXCQom38TaB9dJlWe”,“level”:3,“time”:“2019-07-21T11:45:36+00:00”,“remoteAddr”:“”,“user”:“yvelia”,“app”:“PHP”,“method”:“GET”,“url”:"/apps/files/",“message”:“stat(): stat failed for /opt/nextcloud/appdata_oce3nnfrkrfe/css/icons/icons-list.template at /srv/nextcloud/lib/private/Files/Storage/Local.php#143”,“userAgent”:“Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”,“version”:“”}

Looks like these 2 files cannot be accessed. Are they existing? If yes, can they be accessed by www-data?

No, they dont.

# ls -a appdata_oce3nnfrkrfe/css/icons/
.  ..

I wonder what makes NC expect those files in the first place. Can “<datadir>/appdata_<instanceid>” be healed?

is php7.3-fpm up&running?

Yes, the timeout only occured once. I had to upgrade to php7.3 from 7.0 for NC16 to work, so I forgot to re-tune my fpm config for 7.3, which may have caused the time-out.

# service php7.3-fpm status
● php7.3-fpm.service - The PHP 7.3 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.3-fpm.service; enabled; vendor prese
   Active: active (running) since Sat 2019-07-20 11:56:53 CEST; 2 days ago

and you are sure that your nginx conf is now php7.3 only?

Very much so, NC displayed a page warning for outdated / non-working, also on the command line.
This is what my nginx per-site config looks like for NC:

# head /etc/nginx/sites-available/nextcloud
upstream php-handler {
    server unix:/run/php/php7.3-fpm.sock;

I had trouble as well when I upgraded to PHP 7.2 from 7.0. In my case not all modules were upgraded properly. Can you confirm that all modules are up to date?

I rechecked all the required php modules listed here:
They were all present and upgraded to 7.3 as well.

So the only change was to move the data directory? Did I get you right at this point? If yes, this might contain a solution:

The (external) data directory location as seen in the config.php was kept unchanged.

I am sorry, I cannot help further. Maybe you can report the issue to the developer on Github.

I have a working setup that looks similar to yours but it differs. You could give it a try, but no clue if this will work for you. Note: I am using PHP7.2.

Anyhow, here we go:

Webserver config:

upstream php-handler {
    #server unix:/var/run/php/php7.2-fpm.sock;

Then sudo nano /etc/php/7.2/fpm/php.ini and adapt the following variables. The first 3 values needs to reflect your needs. The memory_limit is the minimum Nextcloud accepts:

upload_max_filesize = 2048M
post_max_size = 2048M
default_charset = "UTF-8"
memory_limit = 512M
always_populate_raw_post_data = -1 
cgi.fix_pathinfo = 0

At the end of the same file I add these lines:

upload_tmp_dir = /srv/http/nextcloud/data
;extension =
apc.enabled = 1
apc.include_once_override = 0
apc.shm_size = 256M


Now the temp dir needs to be created accordingly:

sudo mkdir -p /srv/http/nextcloud/data
sudo chown www-data:www-data /srv/http/nextcloud/data

Now another file needs to be adapted by using sudo nano /etc/php/7.2/fpm/pool.d/www.conf. Variable listen = /var/run/php7.2-fpm.sock needs to be replaced by listen = And I replace the original env[PATH] by the output of command printenv PATH. Example: env[PATH] = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games.

After this, restart FPM and nginx services.

Hope this helps…

Kind regards

1 Like

Thank you for all your efforts, those are good recommendations as well.

I’ll check all my configs again, run full backups and reinstall my NC files and apps just to rule any issues regarding that process out.