Internal Server Error - after upgrade to NC16 / missing appdata files

lyn1337 · July 21, 2019, 1:44pm

This is my nginx error.log, that timeout only occured once.

# cat /var/log/nginx/error.log
2019/07/21 13:23:09 [error] 25632#25632: *159 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 46.114.38.20, server: uwu, request: "GET /apps/files/ HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.3-fpm.sock", host: "uwu"

nephilim · July 21, 2019, 7:09pm

Did you make changes in PHP as well?

lyn1337 · July 22, 2019, 3:22am

No, only to config/config.php.

nephilim · July 22, 2019, 7:34am

Could you please post your config file? Please clear the sensitive information in it to not compromise your instance. What changes in it have been done?

lyn1337 · July 22, 2019, 8:00am

Sure.

# cat config.php 
<?php
$CONFIG = array (
  'instanceid' => 'oce3nnfrkrfe',
  'passwordsalt' => 'removed',
  'secret' => 'removed',
  'trusted_domains' => 
  array (
    0 => 'uwu',
  ),
  'datadirectory' => '/opt/nextcloud/',
  'overwrite.cli.url' => 'https://uwu',
  'dbtype' => 'mysql',
  'version' => '16.0.3.0',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_yvelia',
  'dbpassword' => 'removed',
  'installed' => true,
  'mail_from_address' => 'cloud',
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_domain' => 'wu',
  'maintenance' => true,
  'updater.release.channel' => 'beta',
  'loglevel' => '0',
  'theme' => '',
  'logfile' => '/opt/nextcloud-local/nextcloud.log',
  'mysql.utf8mb4' => true,
  'data-fingerprint' => 'abf399f07057787c54c20c2d748d9b4c',
);

nephilim · July 22, 2019, 9:38am

What are the mentioned changes?

lyn1337 · July 22, 2019, 9:54am

Only log level and file path, when I moved it out of the data-dir.

nephilim · July 22, 2019, 12:56pm

21T11:45:36+00:00",“remoteAddr”:“46.114.38.20”,“user”:“yvelia”,“app”:“PHP”,“method”:“GET”,“url”:"/apps/files/",“message”:“file_get_contents(/opt/nextcloud/appdata_oce3nnfrkrfe/css/icons/icons-list.template): failed to open stream: No such file or directory at /srv/nextcloud/lib/private/Files/Storage/Local.php#217”,“userAgent”:“Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”,“version”:“16.0.3.0”}
{“reqId”:“w2BTXCQom38TaB9dJlWe”,“level”:3,“time”:“2019-07-21T11:45:36+00:00”,“remoteAddr”:“46.114.38.20”,“user”:“yvelia”,“app”:“PHP”,“method”:“GET”,“url”:"/apps/files/",“message”:“stat(): stat failed for /opt/nextcloud/appdata_oce3nnfrkrfe/css/icons/icons-list.template at /srv/nextcloud/lib/private/Files/Storage/Local.php#143”,“userAgent”:“Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”,“version”:“16.0.3.0”}

Looks like these 2 files cannot be accessed. Are they existing? If yes, can they be accessed by www-data?

lyn1337 · July 22, 2019, 1:39pm

No, they dont.

# ls -a appdata_oce3nnfrkrfe/css/icons/
.  ..

I wonder what makes NC expect those files in the first place. Can “<datadir>/appdata_<instanceid>” be healed?

Reiner_Nippes · July 22, 2019, 1:47pm

is php7.3-fpm up&running?

lyn1337 · July 22, 2019, 1:57pm

Yes, the timeout only occured once. I had to upgrade to php7.3 from 7.0 for NC16 to work, so I forgot to re-tune my fpm config for 7.3, which may have caused the time-out.

# service php7.3-fpm status
● php7.3-fpm.service - The PHP 7.3 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.3-fpm.service; enabled; vendor prese
   Active: active (running) since Sat 2019-07-20 11:56:53 CEST; 2 days ago
...

Reiner_Nippes · July 22, 2019, 2:04pm

and you are sure that your nginx conf is now php7.3 only?

lyn1337 · July 22, 2019, 2:10pm

Very much so, NC displayed a page warning for outdated / non-working, also on the command line.
This is what my nginx per-site config looks like for NC:

# head /etc/nginx/sites-available/nextcloud
upstream php-handler {
    #server 127.0.0.1:9000;
    server unix:/run/php/php7.3-fpm.sock;
}

nephilim · July 23, 2019, 4:29am

I had trouble as well when I upgraded to PHP 7.2 from 7.0. In my case not all modules were upgraded properly. Can you confirm that all modules are up to date?

lyn1337 · July 23, 2019, 6:40am

I rechecked all the required php modules listed here:
https://docs.nextcloud.com/server/16/admin_manual/installation/source_installation.html#prerequisites-for-manual-installation
They were all present and upgraded to 7.3 as well.

nephilim · July 23, 2019, 6:47am

So the only change was to move the data directory? Did I get you right at this point? If yes, this might contain a solution:

lyn1337 · July 23, 2019, 7:25am

The (external) data directory location as seen in the config.php was kept unchanged.

nephilim · July 23, 2019, 8:21am

I am sorry, I cannot help further. Maybe you can report the issue to the developer on Github.

I have a working setup that looks similar to yours but it differs. You could give it a try, but no clue if this will work for you. Note: I am using PHP7.2.

Anyhow, here we go:

Webserver config:

upstream php-handler {
    server 127.0.0.1:9000;
    #server unix:/var/run/php/php7.2-fpm.sock;
}

Then sudo nano /etc/php/7.2/fpm/php.ini and adapt the following variables. The first 3 values needs to reflect your needs. The memory_limit is the minimum Nextcloud accepts:

upload_max_filesize = 2048M
post_max_size = 2048M
default_charset = "UTF-8"
memory_limit = 512M
always_populate_raw_post_data = -1 
cgi.fix_pathinfo = 0

At the end of the same file I add these lines:

upload_tmp_dir = /srv/http/nextcloud/data
;extension = apc.so
apc.enabled = 1
apc.include_once_override = 0
apc.shm_size = 256M

opcache.enable=1
opcache.enable_cli=1
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.memory_consumption=128
opcache.save_comments=1
opcache.revalidate_freq=1

Now the temp dir needs to be created accordingly:

sudo mkdir -p /srv/http/nextcloud/data
sudo chown www-data:www-data /srv/http/nextcloud/data

Now another file needs to be adapted by using sudo nano /etc/php/7.2/fpm/pool.d/www.conf. Variable listen = /var/run/php7.2-fpm.sock needs to be replaced by listen = 127.0.0.1:9000. And I replace the original env[PATH] by the output of command printenv PATH. Example: env[PATH] = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games.

After this, restart FPM and nginx services.

Hope this helps…

Kind regards
//neph

lyn1337 · July 23, 2019, 9:13am

Thank you for all your efforts, those are good recommendations as well.

I’ll check all my configs again, run full backups and reinstall my NC files and apps just to rule any issues regarding that process out.

lyn1337 · July 23, 2019, 10:01am

I verified my downloaded archive again.

# sha256sum -c nextcloud-16.0.3.tar.bz2.sha256 
nextcloud-16.0.3.tar.bz2: OK

Stopped nginx, unpacked the tar archive, copied my config.php, manually copied my missing /apps/ and ran the ownership/permission fix over NC and the data dir.
After restarting nginx, I performed the occ upgrade step.
https://pastebin.com/i7FPEk2n

I then updated my apps via occ.

# sudo -u www-data php occ app:update --all

EDIT:
With occ log:watch in the background I kept testing, so far the issue seemed to stay the same.
Until I tried the following:

# sudo -u www-data php -f occ files:scan-app-data
Scanning AppData for files

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 12319   | 27123 | 00:12:47     |
+---------+-------+--------------+

The NC error-level log stays empty now and the page loads without problems.
The missing files in appdata_X/css/icons/ seemed to have been added again as well.