Need help with php7.4-fpm

Nextcloud version 18.0.3
Operating system and version: Debian 10.3
nginx 1.17.8
PHP 7.4.4

Hosted on Hyper-V vm.
Hardware:
i7-7700k 4C 8T
8GB mem on VM
250GB SSD raid 0

I am having some issues with optimizing my nextcloud for the gallery app. Tried multiple configs for my SQL and FPM config but can’t get it right. It locks up way to often when I open a new folder in the gallery app. and often it stops halfway…

I used the following guide to setup my server:

2020/04/05 14:47:36 [error] 71431#71431: *60749 connect() to unix:/run/php/php7.4-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: ******, server: 10.0.0.4, request: “GET /core/preview?fileId=439492&x=256&y=256&a=false&v=61d6f1ef72f50d6ff0f95b33706caedd HTTP/1.0”, upstream: “fastcgi://unix:/run/php/php7.4-fpm.sock:”, host: “example.com

SQL conf

[client]
default-character-set = utf8mb4
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]
log_error = /var/log/mysql/mysql_error.log
nice = 0
socket = /var/run/mysqld/mysqld.sock

[mysqld]
basedir = /usr
bind-address = 127.0.0.1
binlog_format = ROW
bulk_insert_buffer_size = 16M
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
concurrent_insert = 2
connect_timeout = 5
datadir = /var/lib/mysql
default_storage_engine = InnoDB
expire_logs_days = 7
general_log_file = /var/log/mysql/mysql.log
general_log = 0
innodb_buffer_pool_size = 1024M
innodb_buffer_pool_instances = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 32M
innodb_max_dirty_pages_pct = 90
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 4000
innodb_flush_method = O_DIRECT
key_buffer_size = 128M
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
log_bin = /var/log/mysql/mariadb-bin
log_bin_index = /var/log/mysql/mariadb-bin.index
log_error=/var/log/mysql/mysql_error.log
log_slow_verbosity = query_plan
log_warnings = 2
long_query_time = 1
max_allowed_packet = 16M
max_binlog_size = 100M
max_connections = 200
max_heap_table_size = 64M
myisam_recover_options = BACKUP
myisam_sort_buffer_size = 512M
port = 3306
pid-file = /var/run/mysqld/mysqld.pid
query_cache_limit = 2M
query_cache_size = 64M
query_cache_type = 1
query_cache_min_res_unit = 2k
read_buffer_size = 2M
read_rnd_buffer_size = 1M
skip-external-locking
skip-name-resolve
slow_query_log_file = /var/log/mysql/mariadb-slow.log
slow-query-log = 1
socket = /var/run/mysqld/mysqld.sock
sort_buffer_size = 4M
table_open_cache = 400
thread_cache_size = 128
tmp_table_size = 64M
tmpdir = /tmp
transaction_isolation = READ-COMMITTED
user = mysql
wait_timeout = 600

[mysqldump]
max_allowed_packet = 16M
quick
quote-names

[isamchk]
key_buffer = 16M

PHP www.conf

; Start a new pool named ‘www’.
; the variable $pool can be used in any directive and will be replaced by the
; pool name (‘www’ here)
[www]

; Per pool prefix
; It only applies on the following directives:
; - ‘access.log’
; - ‘slowlog’
; - ‘listen’ (unixsocket)
; - ‘chroot’
; - ‘chdir’
; - ‘php_values’
; - ‘php_admin_values’
; When not set, the global prefix (or /usr) applies instead.
; Note: This directive can also be relative to the global prefix.
; Default Value: none
;prefix = /path/to/pools/$pool

; Unix user/group of processes
; Note: The user is mandatory. If the group is not set, the default user’s group
; will be used.
user = www-data
group = www-data

; The address on which to accept FastCGI requests.
; Valid syntaxes are:
; ‘ip.add.re.ss:port’ - to listen on a TCP socket to a specific IPv4 address on
; a specific port;
; ‘[ip:6:addr:ess]:port’ - to listen on a TCP socket to a specific IPv6 address on
; a specific port;
; ‘port’ - to listen on a TCP socket to all addresses
; (IPv6 and IPv4-mapped) on a specific port;
; ‘/path/to/unix/socket’ - to listen on a unix socket.
; Note: This value is mandatory.
listen = /run/php/php7.4-fpm.sock

; Set listen(2) backlog.
; Default Value: 511 (-1 on FreeBSD and OpenBSD)
;listen.backlog = 511

; Set permissions for unix socket, if one is used. In Linux, read/write
; permissions must be set in order to allow connections from a web server. Many
; BSD-derived systems allow connections regardless of permissions.
; Default Values: user and group are set as the running user
; mode is set to 0660
listen.owner = www-data
listen.group = www-data
;listen.mode = 0660
; When POSIX Access Control Lists are supported you can set them using
; these options, value is a comma separated list of user/group names.
; When set, listen.owner and listen.group are ignored
;listen.acl_users =
;listen.acl_groups =

; List of addresses (IPv4/IPv6) of FastCGI clients which are allowed to connect.
; Equivalent to the FCGI_WEB_SERVER_ADDRS environment variable in the original
; PHP FCGI (5.2.2+). Makes sense only with a tcp listening socket. Each address
; must be separated by a comma. If this value is left blank, connections will be
; accepted from any ip address.
; Default Value: any
;listen.allowed_clients = 127.0.0.1

; Specify the nice(2) priority to apply to the pool processes (only if set)
; The value can vary from -19 (highest priority) to 20 (lower priority)
; Note: - It will only work if the FPM master process is launched as root
; - The pool processes will inherit the master process priority
; unless it specified otherwise
; Default Value: no set
; process.priority = -19

; Set the process dumpable flag (PR_SET_DUMPABLE prctl) even if the process user
; or group is differrent than the master process user. It allows to create process
; core dump and ptrace the process for the pool user.
; Default Value: no
; process.dumpable = yes

; Choose how the process manager will control the number of child processes.
; Possible Values:
; static - a fixed number (pm.max_children) of child processes;
; dynamic - the number of child processes are set dynamically based on the
; following directives. With this process management, there will be
; always at least 1 children.
; pm.max_children - the maximum number of children that can
; be alive at the same time.
; pm.start_servers - the number of children created on startup.
; pm.min_spare_servers - the minimum number of children in ‘idle’
; state (waiting to process). If the number
; of ‘idle’ processes is less than this
; number then some children will be created.
; pm.max_spare_servers - the maximum number of children in ‘idle’
; state (waiting to process). If the number
; of ‘idle’ processes is greater than this
; number then some children will be killed.
; ondemand - no children are created at startup. Children will be forked when
; new requests will connect. The following parameter are used:
; pm.max_children - the maximum number of children that
; can be alive at the same time.
; pm.process_idle_timeout - The number of seconds after which
; an idle process will be killed.
; Note: This value is mandatory.
pm = dynamic

; The number of child processes to be created when pm is set to ‘static’ and the
; maximum number of child processes when pm is set to ‘dynamic’ or ‘ondemand’.
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI. The below defaults are based on a server without much resources. Don’t
; forget to tweak pm.* to fit your needs.
; Note: Used when pm is set to ‘static’, ‘dynamic’ or ‘ondemand’
; Note: This value is mandatory.
pm.max_children = 16
; The number of child processes created on startup.
; Note: Used only when pm is set to ‘dynamic’
; Default Value: (min_spare_servers + max_spare_servers) / 2
pm.start_servers = 4

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to ‘dynamic’
; Note: Mandatory when pm is set to ‘dynamic’
pm.min_spare_servers = 4

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to ‘dynamic’
; Note: Mandatory when pm is set to ‘dynamic’
pm.max_spare_servers = 12

; The number of seconds after which an idle process will be killed.
; Note: Used only when pm is set to ‘ondemand’
; Default Value: 10s
;pm.process_idle_timeout = 10s;

; The number of requests each child process should execute before respawning.
; This can be useful to work around memory leaks in 3rd party libraries. For
; endless request processing specify ‘0’. Equivalent to PHP_FCGI_MAX_REQUESTS.
; Default Value: 0
pm.max_requests = 500

; The URI to view the FPM status page. If this value is not set, no URI will be
; recognized as a status page. It shows the following informations:
; pool - the name of the pool;
; process manager - static, dynamic or ondemand;
; start time - the date and time FPM has started;
; start since - number of seconds since FPM has started;
; accepted conn - the number of request accepted by the pool;
; listen queue - the number of request in the queue of pending
; connections (see backlog in listen(2));
; max listen queue - the maximum number of requests in the queue
; of pending connections since FPM has started;
; listen queue len - the size of the socket queue of pending connections;
; idle processes - the number of idle processes;
; active processes - the number of active processes;
; total processes - the number of idle + active processes;
; max active processes - the maximum number of active processes since FPM
; has started;
; max children reached - number of times, the process limit has been reached,
; when pm tries to start more children (works only for
; pm ‘dynamic’ and ‘ondemand’);
; Value are updated in real time.
; Example output:
; pool: www
; process manager: static
; start time: 01/Jul/2011:17:53:49 +0200
; start since: 62636
; accepted conn: 190460
; listen queue: 0
; max listen queue: 1
; listen queue len: 42
; idle processes: 4
; active processes: 11
; total processes: 15
; max active processes: 12
; max children reached: 0
;
; By default the status page output is formatted as text/plain. Passing either
; ‘html’, ‘xml’ or ‘json’ in the query string will return the corresponding
; output syntax. Example:
;
; By default the status page only outputs short status. Passing ‘full’ in the
; query string will also return status for each pool process.
; Example:

; The Full status returns for each process:
; pid - the PID of the process;
; state - the state of the process (Idle, Running, …);
; start time - the date and time the process has started;
; start since - the number of seconds since the process has started;
; requests - the number of requests the process has served;
; request duration - the duration in µs of the requests;
; request method - the request method (GET, POST, …);
; request URI - the request URI with the query string;
; content length - the content length of the request (only with POST);
; user - the user (PHP_AUTH_USER) (or ‘-’ if not set);
; script - the main script called (or ‘-’ if not set);
; last request cpu - the %cpu the last request consumed
; it’s always 0 if the process is not in Idle state
; because CPU calculation is done when the request
; processing has terminated;
; last request memory - the max amount of memory the last request consumed
; it’s always 0 if the process is not in Idle state
; because memory calculation is done when the request
; processing has terminated;
; If the process is in Idle state, then informations are related to the
; last request the process has served. Otherwise informations are related to
; the current request being served.
; Example output:
; ************************
; pid: 31330
; state: Running
; start time: 01/Jul/2011:17:53:49 +0200
; start since: 63087
; requests: 12808
; request duration: 1250261
; request method: GET
; request URI: /test_mem.php?N=10000
; content length: 0
; user: -
; script: /home/fat/web/docs/php/test_mem.php
; last request cpu: 0.00
; last request memory: 0
;
; Note: There is a real-time FPM status monitoring sample web page available
; It’s available in: /usr/share/php/7.4/fpm/status.html
;
; Note: The value must start with a leading slash (/). The value can be
; anything, but it may not be a good idea to use the .php extension or it
; may conflict with a real PHP file.
; Default Value: not set
;pm.status_path = /status

; The ping URI to call the monitoring page of FPM. If this value is not set, no
; URI will be recognized as a ping page. This could be used to test from outside
; that FPM is alive and responding, or to
; - create a graph of FPM availability (rrd or such);
; - remove a server from a group if it is not responding (load balancing);
; - trigger alerts for the operating team (24/7).
; Note: The value must start with a leading slash (/). The value can be
; anything, but it may not be a good idea to use the .php extension or it
; may conflict with a real PHP file.
; Default Value: not set
;ping.path = /ping

; This directive may be used to customize the response of a ping request. The
; response is formatted as text/plain with a 200 response code.
; Default Value: pong
;ping.response = pong

; The access log file
; Default: not set
;access.log = log/$pool.access.log

; The access log format.
; The following syntax is allowed
; %%: the ‘%’ character
; %C: %CPU used by the request
; it can accept the following format:
; - %{user}C for user CPU only
; - %{system}C for system CPU only
; - %{total}C for user + system CPU (default)
; %d: time taken to serve the request
; it can accept the following format:
; - %{seconds}d (default)
; - %{miliseconds}d
; - %{mili}d
; - %{microseconds}d
; - %{micro}d
; %e: an environment variable (same as $_ENV or $_SERVER)
; it must be associated with embraces to specify the name of the env
; variable. Some exemples:
; - server specifics like: %{REQUEST_METHOD}e or %{SERVER_PROTOCOL}e
; - HTTP headers like: %{HTTP_HOST}e or %{HTTP_USER_AGENT}e
; %f: script filename
; %l: content-length of the request (for POST request only)
; %m: request method
; %M: peak of memory allocated by PHP
; it can accept the following format:
; - %{bytes}M (default)
; - %{kilobytes}M
; - %{kilo}M
; - %{megabytes}M
; - %{mega}M
; %n: pool name
; %o: output header
; it must be associated with embraces to specify the name of the header:
; - %{Content-Type}o
; - %{X-Powered-By}o
; - %{Transfert-Encoding}o
; - …
; %p: PID of the child that serviced the request
; %P: PID of the parent of the child that serviced the request
; %q: the query string
; %Q: the ‘?’ character if query string exists
; %r: the request URI (without the query string, see %q and %Q)
; %R: remote IP address
; %s: status (response code)
; %t: server time the request was received
; it can accept a strftime(3) format:
; %d/%b/%Y:%H:%M:%S %z (default)
; The strftime(3) format must be encapsuled in a %{<strftime_format>}t tag
; e.g. for a ISO8601 formatted timestring, use: %{%Y-%m-%dT%H:%M:%S%z}t
; %T: time the log has been written (the request has finished)
; it can accept a strftime(3) format:
; %d/%b/%Y:%H:%M:%S %z (default)
; The strftime(3) format must be encapsuled in a %{<strftime_format>}t tag
; e.g. for a ISO8601 formatted timestring, use: %{%Y-%m-%dT%H:%M:%S%z}t
; %u: remote user
;
; Default: “%R - %u %t “%m %r” %s”
;access.format = “%R - %u %t “%m %r%Q%q” %s %f %{mili}d %{kilo}M %C%%”

; The log file for slow requests
; Default Value: not set
; Note: slowlog is mandatory if request_slowlog_timeout is set
;slowlog = log/$pool.log.slow

; The timeout for serving a single request after which a PHP backtrace will be
; dumped to the ‘slowlog’ file. A value of ‘0s’ means ‘off’.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_slowlog_timeout = 0

; Depth of slow log stack trace.
; Default Value: 20
;request_slowlog_trace_depth = 20

; The timeout for serving a single request after which the worker process will
; be killed. This option should be used when the ‘max_execution_time’ ini option
; does not stop script execution for some reason. A value of ‘0’ means ‘off’.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_terminate_timeout = 0

; The timeout set by ‘request_terminate_timeout’ ini option is not engaged after
; application calls ‘fastcgi_finish_request’ or when application has finished and
; shutdown functions are being called (registered via register_shutdown_function).
; This option will enable timeout limit to be applied unconditionally
; even in such cases.
; Default Value: no
;request_terminate_timeout_track_finished = no

; Set open file descriptor rlimit.
; Default Value: system defined value
;rlimit_files = 1024

; Set max core size rlimit.
; Possible Values: ‘unlimited’ or an integer greater or equal to 0
; Default Value: system defined value
;rlimit_core = 0

; Chroot to this directory at the start. This value must be defined as an
; absolute path. When this value is not set, chroot is not used.
; Note: you can prefix with ‘$prefix’ to chroot to the pool prefix or one
; of its subdirectories. If the pool prefix is not set, the global prefix
; will be used instead.
; Note: chrooting is a great security feature and should be used whenever
; possible. However, all PHP paths will be relative to the chroot
; (error_log, sessions.save_path, …).
; Default Value: not set
;chroot =

; Chdir to this directory at the start.
; Note: relative path can be used.
; Default Value: current directory or / when chroot
;chdir = /var/www

; Redirect worker stdout and stderr into main error log. If not set, stdout and
; stderr will be redirected to /dev/null according to FastCGI specs.
; Note: on highloaded environement, this can cause some delay in the page
; process time (several ms).
; Default Value: no
;catch_workers_output = yes

; Decorate worker output with prefix and suffix containing information about
; the child that writes to the log and if stdout or stderr is used as well as
; log level and time. This options is used only if catch_workers_output is yes.
; Settings to “no” will output data as written to the stdout or stderr.
; Default value: yes
;decorate_workers_output = no

; Clear environment in FPM workers
; Prevents arbitrary environment variables from reaching FPM worker processes
; by clearing the environment in workers before env vars specified in this
; pool configuration are added.
; Setting to “no” will make all environment variables available to PHP code
; via getenv(), $_ENV and $_SERVER.
; Default Value: yes
;clear_env = no

; Limits the extensions of the main script FPM will allow to parse. This can
; prevent configuration mistakes on the web server side. You should only limit
; FPM to .php extensions to prevent malicious users to use other extensions to
; execute php code.
; Note: set an empty value to allow all extensions.
; Default Value: .php
;security.limit_extensions = .php .php3 .php4 .php5 .php7

; Pass environment variables like LD_LIBRARY_PATH. All $VARIABLEs are taken from
; the current environment.
; Default Value: clean env
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

; Additional php.ini defines, specific to this pool of workers. These settings
; overwrite the values previously defined in the php.ini. The directives are the
; same as the PHP SAPI:
; php_value/php_flag - you can set classic ini defines which can
; be overwritten from PHP call ‘ini_set’.
; php_admin_value/php_admin_flag - these directives won’t be overwritten by
; PHP call ‘ini_set’
; For php_*flag, valid values are on, off, 1, 0, true, false, yes or no.

; Defining ‘extension’ will load the corresponding shared extension from
; extension_dir. Defining ‘disable_functions’ or ‘disable_classes’ will not
; overwrite previously defined php.ini values, but will append the new value
; instead.

; Note: path INI options can be relative and will be expanded with the prefix
; (pool, global or /usr)

; Default Value: nothing is defined by default except the values in php.ini and
; specified at startup with the -d argument
;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f www@my.domain.com
;php_flag[display_errors] = off
;php_admin_value[error_log] = /var/log/fpm-php.www.log
;php_admin_flag[log_errors] = on
;php_admin_value[memory_limit] = 32M

When you say gallery app do you mean photos? Gallery is not supported in 18.0.3 even if you can use the old app.

Uhm yeah you are right the photos app.

Would be useful to have some error logs. I personally stopped using mariadb because of this issue. You can try regular mysql or postgresql.

When you say locks up halfway through are you talking about a few images or several hundred/thousand. The context here matters.

This is the nginx error I get when it stops generating pictures. Using a folder with 780 photos to test with.

2020/04/05 14:47:36 [error] 71431#71431: *60749 connect() to unix:/run/php/php7.4-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: ******, server: 10.0.0.4, request: “GET /core/preview?fileId=439492&x=256&y=256&a=false&v=61d6f1ef72f50d6ff0f95b33706caedd HTTP/1.0”, upstream: “fastcgi://unix:/run/php/php7.4-fpm.sock:”, host: “example.com

PHP only gives some warnings but this is to be expected considering the amount of pm children used.

[05-Apr-2020 14:25:16] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 12 total children
[05-Apr-2020 14:25:30] WARNING: [pool www] server reached pm.max_children setting (16), consider raising it

I tried using it with the following according to the install guide but this is using way to much memory and immediately starts using serval GB of the swap.

pm.max_children = 240
pm.start_servers = 20
pm.min_spare_servers = 10
pm.max_spare_servers = 20

Here are some redis logs

812:M 05 Apr 2020 14:41:33.149 * Background saving terminated with success
812:M 05 Apr 2020 14:43:39.628 * 10000 changes in 60 seconds. Saving…
812:M 05 Apr 2020 14:43:39.629 * Background saving started by pid 86559
86559:C 05 Apr 2020 14:43:39.642 * DB saved on disk
86559:C 05 Apr 2020 14:43:39.643 * RDB: 0 MB of memory used by copy-on-write
812:M 05 Apr 2020 14:43:39.732 * Background saving terminated with success
812:M 05 Apr 2020 14:45:17.678 * 10000 changes in 60 seconds. Saving…
812:M 05 Apr 2020 14:45:17.678 * Background saving started by pid 86565
86565:C 05 Apr 2020 14:45:17.696 * DB saved on disk
86565:C 05 Apr 2020 14:45:17.697 * RDB: 0 MB of memory used by copy-on-write
812:M 05 Apr 2020 14:45:17.779 * Background saving terminated with success
812:M 05 Apr 2020 14:46:18.010 * 10000 changes in 60 seconds. Saving…
812:M 05 Apr 2020 14:46:18.010 * Background saving started by pid 86567
86567:C 05 Apr 2020 14:46:18.016 * DB saved on disk
86567:C 05 Apr 2020 14:46:18.017 * RDB: 0 MB of memory used by copy-on-write
812:M 05 Apr 2020 14:46:18.111 * Background saving terminated with success
812:M 05 Apr 2020 14:51:19.062 * 10 changes in 300 seconds. Saving…
812:M 05 Apr 2020 14:51:19.070 * Background saving started by pid 87473
87473:C 05 Apr 2020 14:51:19.079 * DB saved on disk
87473:C 05 Apr 2020 14:51:19.080 * RDB: 0 MB of memory used by copy-on-write
812:M 05 Apr 2020 14:51:19.172 * Background saving terminated with success
812:M 05 Apr 2020 15:20:04.339 * 1 changes in 900 seconds. Saving…
812:M 05 Apr 2020 15:20:04.378 * Background saving started by pid 94934
94934:C 05 Apr 2020 15:20:04.420 * DB saved on disk
94934:C 05 Apr 2020 15:20:04.420 * RDB: 0 MB of memory used by copy-on-write
812:M 05 Apr 2020 15:20:04.478 * Background saving terminated with success
812:M 05 Apr 2020 15:25:05.057 * 10 changes in 300 seconds. Saving…
812:M 05 Apr 2020 15:25:05.057 * Background saving started by pid 94951
94951:C 05 Apr 2020 15:25:05.062 * DB saved on disk
94951:C 05 Apr 2020 15:25:05.063 * RDB: 0 MB of memory used by copy-on-write
812:M 05 Apr 2020 15:25:05.157 * Background saving terminated with success

And if you disable redis does it work? You have to start ruling out things.

I don’t have a folder with 750 pictures so I can’t verify this. Might be a good idea to report this on the photos section of github.

Edit: I also don’t think 240 PHP processes is feasible on a 4 core CPU with 8GB ram but to each their own.

tried to disable redis just now but that didn’t do anything.

Okay, time to make a bug report then. You can also use the preview generator app.