Error 404 after nginx and php upgrade

Garwin · November 2, 2024, 4:53pm

Hi all

Following an warning message in Nextcloud saying that my PHP version was too old, I upgraded to the latest PHP version (8.3.12).
I also decided to upgrade nginx to the latest version.
Sadly, after these upgrade I’ve now a 404 error message when connecting to Nextcloud.
I’ve try many things, spend a lot of time searching on the web any hint to solve this issue, without success.
Nevertheless, my investigations point to an issue with the FastCGI config, but I’m not able to find what.

Is anyone able to guide me ? It would be much appreciated.

Nextcloud version : 30.0.1
Operating system and version : FreeBSD (Truenas Jail)
nginx version : 1.26.2
PHP version : 8.3.12

The issue you are facing:

Is this the first time you’ve seen this error? (Y/N): YES

Steps to replicate it:

Upgrade PHP from 8.1 to 8.3
Upgrade nginx from 1.24 to 1.26.2
Try to connect to Nextcloud

The output of your Nextcloud log in Admin > Logging:

404 Not Found

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/usr/local/www/nextcloud/apps',
      'url' => '/apps',
      'writable' => true,
    ),
    1 =>
    array (
      'path' => '/usr/local/www/nextcloud/apps-pkg',
      'url' => '/apps-pkg',
      'writable' => false,
    ),
  ),
  'logfile' => '/var/log/nextcloud/nextcloud.log',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'one-click-instance' => true,
  'one-click-instance.user-limit' => 100,
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'localhost',
  ),
  'passwordsalt' => '*******',
  'secret' => '*******',
  'trusted_domains' =>
  array (
    0 => 'https://my.domain.com',
    1 => '192.168.1.xxx',
    2 => '192.168.1.yyy',
    3 => '192.168.1.zzz',
    4 => 'my.domain.com',
    5 => '192.168.1.1',
  ),
  'trusted_proxies' =>
  array (
    0 => '192.168.1.xxx',
  ),
  'datadirectory' => '/usr/local/www/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '30.0.1.2',
  'overwriteprotocol' => 'https',
  'overwrite.cli.url' => 'https://my.domain.com',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'oc_ncadmin',
  'dbpassword' => '*******',
  'installed' => true,
  'instanceid' => '*******',
  'mail_from_address' => 'info',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => '*******',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => '*******',
  'mail_smtpport' => '465',
  'mail_smtpname' => '*******',
  'mail_smtppassword' => '*******',
  'mail_smtpsecure' => 'ssl',
  'default_phone_region' => 'FR',
  'theme' => '',
  'loglevel' => 2,
  'maintenance' => false,
  'maintenance_window_start' => 100,
);

The output of your Apache/nginx/system log in /var/log/____:

No error in /var/log/nginx/error.log

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

No error since the upgrade of PHP and nginx

ernolf · November 2, 2024, 4:59pm

Hi @Garwin,

May I ask how you did the upgrade?

Did you edit the nextcloud.conf (or whatever name that file has in your setup) from nginx to point to the new fpm-socket for php8.3?

Much and good luck,
ernolf

ernolf · November 2, 2024, 5:16pm

Could you please provide the output of this command:

for socket in $(ss -lx | awk '/fpm/{print $5}'); do
    echo "socket: $socket"
    grep -Els "^[^#]*$socket" $(nginx -T 2>/dev/null) | sort -u
done

ernolf

Garwin · November 3, 2024, 8:52am

Hi Ernolf.

Thanks a lot for your answer.
So, I did the PHP upgrade by running
pkg install php82 mod_php82 php82-{bcmath,ctype,curl,dom,exif,extensions,fileinfo,filter,gd,iconv,mbstring,mysqli,opcache,pdo,pdo_mysql,pdo_odbc,pdo_sqlite,pecl-imagick,pecl-json_post,pecl-mcrypt,phar,posix,session,simplexml,sodium,sqlite3,tidy,tokenizer,xml,xmlreader,xmlwriter,zip,zlib}

Nginx was upgraded after “pkg update” and “pkg upgrade”

Regarding the socket, it’s still the same as before the upgrade
From my nextcloud.conf :

upstream php-handler {
    server unix:/var/run/nextcloud-php-fpm.sock;
}

Regarding the command you sent me, how can I run it ?
I tried in through a sh file, but I get an error “ss not found”.

Thanks again for your help. Much appreciated

Garwin · November 3, 2024, 9:03am

… and this is my php-fpm conf file for nextcloud

[nextcloud]
user = www
group = www

listen = /var/run/nextcloud-php-fpm.sock
listen.owner = www
listen.group = www

pm = dynamic
pm.max_children = 100
pm.start_servers = 25
pm.min_spare_servers = 25
pm.max_spare_servers = 75

php_admin_value[session.save_path] = "/usr/local/www/nextcloud-sessions-tmp"
; Nextcloud wants PATH environment variable set.
env[PATH] = $PATH

The socket is the same as in my nextcloud.conf

ernolf · November 3, 2024, 9:06am

OK, that’s because I’m too Debian oriented

In FreeBSD “ss” is called “sockstat” so that command looks this way for you:

for socket in $(sockstat -l | awk '/fpm/ {print $NF}'); do
    echo "socket: $socket"
    grep -Els "^[^#]*$socket" $(nginx -T 2>/dev/null) | sort -u
done

hth.

ernolf

Garwin · November 3, 2024, 9:11am

Better but still an error with grep

grep: unrecognized option `----------'
usage: grep [-abcDEFGHhIiLlmnOoPqRSsUVvwxz] [-A num] [-B num] [-C num]
        [-e pattern] [-f file] [--binary-files=value] [--color=when]
        [--context=num] [--directories=action] [--label] [--line-buffered]
        [--null] [pattern] [file ...]

ernolf · November 3, 2024, 9:14am

OK, lets make things easier, what is the output of

sockstat -l | grep fpm

ernolf

Garwin · November 3, 2024, 9:15am

Here it is :

root@Nextcloud-new:~ # sockstat -l | grep fpm
www      php-fpm    28103 7  tcp4   127.0.0.1:9000        *:*
www      php-fpm    28102 7  tcp4   127.0.0.1:9000        *:*
www      php-fpm    28101 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28100 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28099 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28098 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28097 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28096 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28095 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28094 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28093 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28092 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28091 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28090 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28089 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28088 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28087 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28086 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28085 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28084 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28083 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28082 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28081 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28080 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28079 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28078 7  stream /var/run/nextcloud-php-fpm.sock
www      php-fpm    28077 7  stream /var/run/nextcloud-php-fpm.sock
root     php-fpm    28076 9  stream /var/run/nextcloud-php-fpm.sock
root     php-fpm    28076 10 tcp4   127.0.0.1:9000        *:*

ernolf · November 3, 2024, 9:30am

OK, that look OK.

What is the output of

pkg info | grep php

ernolf

Garwin · November 3, 2024, 9:32am

root@Nextcloud-new:~ # pkg info | grep php
mod_php83-8.3.13               PHP Scripting Language (8.3.X branch)
php83-8.3.13                   PHP Scripting Language (8.3.X branch)
php83-bcmath-8.3.13            The bcmath shared extension for php
php83-ctype-8.3.13             The ctype shared extension for php
php83-curl-8.3.13              The curl shared extension for php
php83-dom-8.3.13               The dom shared extension for php
php83-exif-8.3.13              The exif shared extension for php
php83-extensions-1.0           "meta-port" to install PHP extensions
php83-fileinfo-8.3.13          The fileinfo shared extension for php
php83-filter-8.3.13            The filter shared extension for php
php83-gd-8.3.13                The gd shared extension for php
php83-iconv-8.3.13             The iconv shared extension for php
php83-mbstring-8.3.13          The mbstring shared extension for php
php83-mysqli-8.3.13            The mysqli shared extension for php
php83-opcache-8.3.13           The opcache shared extension for php
php83-pdo-8.3.13               The pdo shared extension for php
php83-pdo_mysql-8.3.13         The pdo_mysql shared extension for php
php83-pdo_odbc-8.3.13          The pdo_odbc shared extension for php
php83-pdo_sqlite-8.3.13        The pdo_sqlite shared extension for php
php83-pecl-imagick-3.7.0_4     PHP wrapper to the ImageMagick/GraphicsMagick library
php83-pecl-json_post-1.1.0     JSON POST handler in PHP
php83-pecl-mcrypt-1.0.7        PHP extension for mcrypt, removed in PHP 7.2
php83-phar-8.3.13              The phar shared extension for php
php83-posix-8.3.13             The posix shared extension for php
php83-session-8.3.13           The session shared extension for php
php83-simplexml-8.3.13         The simplexml shared extension for php
php83-sodium-8.3.13            The sodium shared extension for php
php83-sqlite3-8.3.13           The sqlite3 shared extension for php
php83-tidy-8.3.13              The tidy shared extension for php
php83-tokenizer-8.3.13         The tokenizer shared extension for php
php83-xml-8.3.13               The xml shared extension for php
php83-xmlreader-8.3.13         The xmlreader shared extension for php
php83-xmlwriter-8.3.13         The xmlwriter shared extension for php
php83-zip-8.3.13               The zip shared extension for php
php83-zlib-8.3.13              The zlib shared extension for php

Garwin · November 3, 2024, 9:51am

I’m pretty sure it’s a stupid config issue, but I’m not able to identify which one nor how to debug further.

This only thing I’ve been able to find is that it fails here, in my nextcloud.inc conf file at the line

    try_files $fastcgi_script_name =404;

location ~ \.php(?:$|/) {
    # Required for legacy support
    rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;

    fastcgi_split_path_info ^(.+?\.php)(/.*)$;
    set $path_info $fastcgi_path_info;

    try_files $fastcgi_script_name =404;

    include fastcgi_params;
#    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param SCRIPT_FILENAME /usr/local/etc/php/$fastcgi_script_name;
    fastcgi_param PATH_INFO $path_info;
    # fastcgi_param HTTPS on;

    fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
    fastcgi_param front_controller_active true;     # Enable pretty urls
    fastcgi_pass php-handler;

    fastcgi_intercept_errors on;
    fastcgi_request_buffering off;
}

Garwin · November 3, 2024, 10:00am

I don’t know if it can help, but when I want to restart php-fpm service, it’s not finding it. I need to restart pfp_fpm (with an underscore). I don’t remember if it was the case before.

root@Nextcloud-new:~ # service php-fpm restart
php-fpm does not exist in /etc/rc.d or the local startup
directories (/usr/local/etc/rc.d), or is not executable

root@Nextcloud-new:~ # service php_fpm restart
Performing sanity check on php-fpm configuration:
[03-Nov-2024 10:56:51] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful
Stopping php_fpm.
Waiting for PIDS: 28076.
Performing sanity check on php-fpm configuration:
[03-Nov-2024 10:56:51] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful
Starting php_fpm.

ernolf · November 3, 2024, 11:14am

Is there really no entry at all?

What are the entries made in the access.log?
What are the entries in php (error) log files?

Try raising loglevel etc…

ernolf

Garwin · November 3, 2024, 11:29am

Nginx access.log :

192.168.1.135 - xxx.xxx.xxx.xxx [03/Nov/2024:12:11:51 +0100] "GET /cgi-bin/luci/;stok=/locale HTTP/1.1" 404 146 "http://xxx.xxx.xxx.xxx:80/cgi-bin/luci/;stok=/locale" "-" "xxx.xxx.xxx.xxx"rt=0.000 ut=- cs=-

Nginx error.log shows nothing new since 2 days when I tried to change some parameters unsuccessfully. I reverted my changes, and I’ve no error now.

Php error.log only show a succsessful restart :

[03-Nov-2024 10:56:51] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful
[03-Nov-2024 10:56:51] NOTICE: Finishing ...
[03-Nov-2024 10:56:51] NOTICE: exiting, bye-bye!
[03-Nov-2024 10:56:51] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful
[03-Nov-2024 10:56:51] NOTICE: fpm is running, pid 44267
[03-Nov-2024 10:56:51] NOTICE: ready to handle connections

That’s why I’m completely lost…

ernolf · November 3, 2024, 11:48am

Did you make that change?

Because it normaly should be

fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

https://docs.nextcloud.com/server/latest/admin_manual/installation/nginx.html
hth.

ernolf

Garwin · November 3, 2024, 6:22pm

Yes, that was part of the tests I made.
I reverted this change as well, without any effect.

Thanks again for the time you spend to help me

Garwin · November 7, 2024, 9:45am

Is there a way to find or display the value of variable $fastcgi_script_name ?

    try_files $fastcgi_script_name =404;

I pretty sure it comes from that but I don’t know how to debug or get more info ?

Garwin · November 7, 2024, 2:28pm

I’ve activated the debug mode of nginx.
Now in the error log I can see that :

[notice] 24969#100423: *1 "^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy)" does not match "/index.php/login", client: 192.168.1.135, server: _, request: "GET /login HTTP/1.1", host: "***my nextcloud site***"

I don’t understand this message nor if it helps to understand what’s going wrong with my config.

Please help

Garwin · November 7, 2024, 6:09pm

Some good news : I’ve found why I had 404.
I still don’t know how it happened but many files were deleted from nextcloud root directory, including index.php !
I recovered a previous version of my nextcloud folder by now it’s too old for my current php version so it fails to start.
I’ve tried to upgrade my nextcloud using occ command but it fails due to php version.
I’ll try to downgrade my php version from 8.3.13 to 8.3, but not sure how to do.