Redis / PHP-FPM issues with Nextcloud

Hi,

I have PHP-FPM setup as static, after a while when browsing a lot of photo’s Nextcloud stops loading the previews.

Opening a file / picture will make it load an amount of pictures from the preview cache, but then stops again. This behaviour seems to be related to Redis. As soon as I restart the service the file list / previews are being loaded correctly.

Anyone got a clue?

Im running on:
Centos 7
Nginx, PHP-FPM, Mariadb, Redis.
8 GB RAM
6 Core

The Redis error log:
17456:M 16 Jan 12:59:14.040 * 10 changes in 300 seconds. Saving…
17456:M 16 Jan 12:59:14.040 * Background saving started by pid 17641
17641:C 16 Jan 12:59:14.042 * DB saved on disk
17641:C 16 Jan 12:59:14.042 * RDB: 0 MB of memory used by copy-on-write
17456:M 16 Jan 12:59:14.140 * Background saving terminated with success
17456:M 16 Jan 13:04:15.006 * 10 changes in 300 seconds. Saving…
17456:M 16 Jan 13:04:15.006 * Background saving started by pid 17838
17838:C 16 Jan 13:04:15.008 * DB saved on disk
17838:C 16 Jan 13:04:15.009 * RDB: 0 MB of memory used by copy-on-write
17456:M 16 Jan 13:04:15.107 * Background saving terminated with success

PHP-FPM error log:
[16-Jan-2018 11:02:20] NOTICE: fpm is running, pid 10020
[16-Jan-2018 11:02:20] NOTICE: ready to handle connections
[16-Jan-2018 11:02:20] NOTICE: systemd monitor interval set to 10000ms

Snip from Nextcloud log:
{“reqId”:“GBjKbSjOGOuf5Du2IEog”,“level”:4,“time”:“2018-01-16T11:47:04+00:00”,“remoteAddr”:“xxx.xxx.xxx.xxx”,“user”:“userx”,“app”:“webdav”,“method”:“MKCOL”,“url”:"/remote.php/webdav/xxx/xxx/xxx/",“message”:“Exception: {“Exception”:“Sabre\\DAV\\Exception\\Conflict”,“Message”:“Parent node does not exist”,“Code”:0,“Trace”:”#0 \/xxx\/xxx\/xxx\/xxx\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(590): Sabre\\DAV\\Server->createCollection(‘xxx\/xxx\/xxx’, Object(Sabre\\DAV\\MkCol))\n#1 [internal function]: Sabre\\DAV\\CorePlugin->httpMkcol(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#2 \/xxx\/xxx\/xxx\/xxx\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#3 \/xxx\/xxx\/xxx\/xxx\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit(‘method:MKCOL’, Array)\n#4 \/xxx\/xxx\/xxx\/xxx\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#5 \/xxx\/xxx\/xxx\/xxx\/apps\/dav\/appinfo\/v1\/webdav.php(76): Sabre\\DAV\\Server->exec()\n#6 \/xxx\/xxx\/xxx\/xxx\/remote.php(162): require_once(’\/xxx\/xxx\/xxx\/o…’)\n#7 {main}",“File”:"\/xxx\/xxx\/xxx\/xxx\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php",“Line”:1167}",“userAgent”:“Mozilla/5.0 (Android) ownCloud-android/2.0.0”,“version”:“12.0.4.3”}
{“reqId”:“r9HQkSZnBoBcOUwfuJoI”,“level”:3,“time”:“2018-01-16T12:52:46+00:00”,“remoteAddr”:“xxx.xxx.xxx.xxx”,“user”:“userx”,“app”:“PHP”,“method”:“GET”,“url”:"/index.php/apps/files/api/v1/thumbnail/256/256/Photos/2014/12/2014-12-11%2023.44.44.mov",“message”:“unlink(/tmp/oc_tmp_COYwqt): No such file or directory at /xxx/xxx/xxx/xxx/lib/private/Preview/Movie.php#111”,“userAgent”:“Mozilla/5.0 (Android) ownCloud-android/2.0.0”,“version”:“12.0.4.3”}
{“reqId”:“jkpIqoyqAA4IPyDUMIAy”,“level”:3,“time”:“2018-01-16T12:52:47+00:00”,“remoteAddr”:“xxx.xxx.xxx.xxx”,“user”:“userx”,“app”:“PHP”,“method”:“GET”,“url”:"/index.php/apps/files/api/v1/thumbnail/256/256/Photos/2014/12/2014-12-11%2023.45.01.mov",“message”:“unlink(/tmp/oc_tmp_C3fizW): No such file or directory at /xxx/xxx/xxx/xxx/lib/private/Preview/Movie.php#111”,“userAgent”:“Mozilla/5.0 (Android) ownCloud-android/2.0.0”,“version”:“12.0.4.3”}
{“reqId”:“45zH5Fa5cBSppfQZ48yi”,“level”:3,“time”:“2018-01-16T12:54:41+00:00”,“remoteAddr”:“xxx.xxx.xxx.xxx”,“user”:“userx”,“app”:“PHP”,“method”:“GET”,“url”:"/index.php/apps/files/api/v1/thumbnail/256/256/Photos/2014/12/2014-12-19%2022.21.44.mov",“message”:“unlink(/tmp/oc_tmp_9D8E0A): No such file or directory at /xxx/xxx/xxx/xxx/lib/private/Preview/Movie.php#111”,“userAgent”:“Mozilla/5.0 (Android) ownCloud-android/2.0.0”,“version”:“12.0.4.3”}

Worth to mention, I have set previews to 512x512 in config using:
‘enable_previews’ => true,
‘enabledPreviewProviders’ => array(
‘OC\Preview\PNG’,
‘OC\Preview\JPEG’,
‘OC\Preview\GIF’,
‘OC\Preview\BMP’,
‘OC\Preview\XBitmap’,
‘OC\Preview\MP3’,
‘OC\Preview\TXT’,
‘OC\Preview\MarkDown’,
‘OC\Preview\Movie’
),
‘preview_max_x’ => 512,
‘preview_max_y’ => 512,
‘preview_max_scale_factor’ => 1,
‘filesystem_check_changes’ => 1,
‘memcache.local’ => ‘\OC\Memcache\APCu’,
‘memcache.locking’ => ‘\OC\Memcache\Redis’,
‘redis’ =>
array (
‘host’ => ‘/var/run/redis/redis.sock’,
‘port’ => 0,
),

1 Like

No one? So I changed from static PHP-FPM to dynamic and ondemand. All php settings should be correct, but the gallery app is still hanging… Any advice is welcome!

My php-fpm config:

include=/etc/php-fpm.d/*.conf

[global]
;pid = /run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/error.log
;syslog.facility = daemon
;syslog.ident = php-fpm
;log_level = notice
emergency_restart_threshold = 3
emergency_restart_interval = 1m
process_control_timeout = 5s
;process.max = 128
;process.priority = -19
daemonize = yes
;rlimit_files = 1024
;rlimit_core = 0
;events.mechanism = epoll
;systemd_interval = 10

www.conf:

[www]
;prefix = /path/to/pools/$pool
group = nginx
listen = 127.0.0.1:9000
;listen.backlog = 511
;listen.owner = nobody
;listen.group = nobody
;listen.mode = 0660
;listen.acl_groups =
listen.allowed_clients = 127.0.0.1
; process.priority = -19
pm = ondemand
pm.max_children = 25
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 20
;pm.process_idle_timeout = 10s;
; pm.max_requests = 500
;pm.status_path = /status
;ping.path = /ping
;ping.response = pong
;access.log = log/$pool.access.log
;access.format = "%R - %u %t \"%m %r%Q%q\" %s %f %{mili}d %{kilo}M %C%%"
slowlog = /var/log/php-fpm/www-slow.log
;request_slowlog_timeout = 0
;request_terminate_timeout = 0
;rlimit_files = 1024
;rlimit_core = 0
;chroot = 
;chdir = /var/www
;catch_workers_output = yes
clear_env = no
env[NEXTCLOUD_CONFIG_DIR] = /var/www/html/owncloud/config
;security.limit_extensions = .php .php3 .php4 .php5 .php7
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f www@my.domain.com
;php_flag[display_errors] = off
;php_admin_value[error_log] = /var/log/php-fpm/www-error.log
;php_admin_flag[log_errors] = on
;php_admin_value[memory_limit] = 128M
;php_value[session.save_handler] = files
;php_value[session.save_path]    = /var/lib/php/session
;php_value[soap.wsdl_cache_dir]  = /var/lib/php/wsdlcache
;php_value[opcache.file_cache]  = /var/lib/php/opcache
php_admin_value[upload_tmp_dir] = /var/www/html/owncloud/tmp
php_value[upload_max_filesize] = 16G
php_value[post_max_size] = 16G
php_admin_value[memory_limit] = 1536M
You have new mail in /var/spool/mail/root

php.ini:

;user_ini.filename = ".user.ini"
;user_ini.filename =
;user_ini.cache_ttl = 300
engine = On
short_open_tag = Off
precision = 14
output_buffering = Off
;output_handler =
;url_rewriter.tags
;url_rewriter.hosts
zlib.output_compression = Off
;zlib.output_compression_level = -1
;zlib.output_handler =
implicit_flush = Off
unserialize_callback_func =
serialize_precision = -1
;open_basedir =
disable_functions =
disable_classes =
;highlight.string  = #DD0000
;highlight.comment = #FF9900
;highlight.keyword = #007700
;highlight.default = #0000BB
;highlight.html    = #000000
;ignore_user_abort = On
;realpath_cache_size = 16k
;realpath_cache_ttl = 120
zend.enable_gc = On
;zend.multibyte = Off
;zend.script_encoding =
expose_php = On
max_execution_time = 30
max_input_time = 60
;max_input_nesting_level = 64
; max_input_vars = 1000
memory_limit = 1024M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
;report_zend_debug = 0
track_errors = Off
;xmlrpc_errors = 0
;xmlrpc_error_number = 0
html_errors = On
;docref_root = "/phpmanual/"
;docref_ext = .html
;error_prepend_string = "<span style='color: #ff0000'>"
;error_append_string = "</span>"
;error_log = syslog
;arg_separator.output = "&amp;"
;arg_separator.input = ";&"
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
;enable_post_data_reading = Off
post_max_size = 100M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
;internal_encoding =
;input_encoding =
;output_encoding =
;include_path = ".:/php/includes"
;include_path = ".;c:\php\includes"
doc_root =
user_dir =
; extension_dir = "./"
; extension_dir = "ext"
; sys_temp_dir = "/tmp"
enable_dl = Off
;cgi.force_redirect = 1
;cgi.nph = 1
;cgi.redirect_status_env =
;cgi.fix_pathinfo=1
;cgi.discard_path=1
;fastcgi.impersonate = 1
;fastcgi.logging = 0
;cgi.rfc2616_headers = 0
;cgi.check_shebang_line=1
file_uploads = On
;upload_tmp_dir =
upload_max_filesize = 100M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
;from="john@doe.com"
;user_agent="PHP"
default_socket_timeout = 60
;auto_detect_line_endings = Off
;extension=modulename.extension
;extension=msql.dll
;extension=msql.so
;extension=/path/to/extension/msql.so

[CLI Server]
cli_server.color = On

[Date]
;date.timezone =
;date.default_latitude = 31.7667
;date.default_longitude = 35.2333
;date.sunrise_zenith = 90.583333
;date.sunset_zenith = 90.583333

[filter]
; http://php.net/filter.default
;filter.default = unsafe_raw

; http://php.net/filter.default-flags
;filter.default_flags =

[iconv]
;iconv.input_encoding =
;iconv.internal_encoding =
;iconv.output_encoding =

[intl]
;intl.default_locale =
;intl.error_level = E_WARNING
;intl.use_exceptions = 0

[sqlite3]
;sqlite3.extension_dir =

[Pcre]
;pcre.backtrack_limit=100000
;pcre.recursion_limit=100000
pcre.jit=0

[Pdo]
;pdo_odbc.connection_pooling=strict
;pdo_odbc.db2_instance_name

[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=

[Phar]
;phar.readonly = On
;phar.require_hash = On
;phar.cache_list =

[mail function]
sendmail_path = /usr/sbin/sendmail -t -i
;mail.force_extra_parameters =
mail.add_x_header = On
;mail.log =
; Log mail to syslog;
;mail.log = syslog

[SQL]
sql.safe_mode = Off

[ODBC]
;odbc.default_db    =  Not yet implemented
;odbc.default_user  =  Not yet implemented
;odbc.default_pw    =  Not yet implemented
;odbc.default_cursortype
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
;birdstep.max_links = -1

[Interbase]
; Allow or prevent persistent links.
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
;ibase.default_db =
;ibase.default_user =
;ibase.default_password =
;ibase.default_charset =
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"

[MySQLi]
mysqli.max_persistent = -1
;mysqli.allow_local_infile = On
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off

[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
;mysqlnd.debug =
;mysqlnd.log_mask = 0
;mysqlnd.mempool_default_size = 16000
;mysqlnd.net_cmd_buffer_size = 2048
;mysqlnd.net_read_buffer_size = 32768
;mysqlnd.net_read_timeout = 31536000
;mysqlnd.sha256_server_public_key =

[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

[bcmath]
bcmath.scale = 0

[browscap]
;browscap = extra/browscap.ini

[Session]
session.save_handler = files
;session.save_path = "N;/path"
;session.save_path = "N;MODE;/path"
;session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
;session.cookie_secure =
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.sid_length = 26
session.trans_sid_tags = "a=href,area=href,frame=src,form="
session.sid_bits_per_character = 5

[Assertion]
zend.assertions = -1

[mbstring]

[gd]

[exif]

[Tidy]
tidy.clean_output = Off

[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5

[sysvshm]
;sysvshm.init_mem = 10000

[ldap]
ldap.max_links = -1

[mcrypt]

[dba]

[curl]

[openssl]
; End:

Do you have any idea @jospoortvliet?

I was looking at that twice now, but I fear I don’t understand enough of that (especially fpm) to help.

So only two ideas:

  • maybe it is worth checking for optimized values for fpm (some guides on the internet maybe)
  • could /tmp be too small?

Why I think about /tmp; there are these error messages mentioning that directory and maybe files are deleted by the system (because size limit of tmp too low?) before Nextcloud can actually unlink/ remove them?

...GET”,“url”:"/index.php/apps/files/api/v1/thumbnail/256/256/Photos/2014/12/2014-12-11%2023.45.01.mov",“message”:“unlink(/tmp/oc_tmp_C3fizW): No such file or directory at /xxx/xxx/xxx/xxx/lib/private/Preview/Movie.php#111”,“userAgent”:“Mozilla/5.0 (Android) ownCloud-android/2.0.0”,“version”:“12.0.4.3”}

More or less guessing. Hopefully it helps anyway. Sorry.

Hi @Schmu , thanks for your input. The /tmp dir is not partitioned, so the space should not be of any problem:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       296G  188G   93G  67% /
devtmpfs        2.0G     0  2.0G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           2.0G  209M  1.8G  11% /run
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           396M     0  396M   0% /run/user/0

As soon as I restart php-fpm & redis the images load as they supposed to be. @jospoortvliet, do you any clue?

Issue still present … @jospoortvliet ?

As your issue seem to be quite specific, I would recommend to open an issue on github in parallel and link to two open topics vise versa. On github you usually get faster attention from the devs, which seems necessay in your case, where most of us seem to be not able to help: https://github.com/nextcloud/server/issues

1 Like

Upgraded to 13 yet? I’d try that. Otherwise - you’re a hosting provider, right? Perhaps just get a support contract :wink:

Hi @jospoortvliet , I upgraded to 13 which did not resolve the issue. Also cleared out the whole dir and fetched 13 clean. I’m using this privately so don’t see the need for a support contract tbh. Service configs have been stock cent-os with the modifications recommended. My guess the process dies. I can imagine NC would like to know the root of the issue as im running a pretty standard setup.

Out of interest @Influenist, if you run sudo service php7.0-fpm restart does it resolve the issue (even temporarily)?

See if nero’s thing worked. Otherwise, I am rather clueless - the redis port as “0”, is that correct?

Hmmm, looking at my own redis config, yes:

  array (
    'host' => '/var/run/redis/redis.sock',
    'port' => 0,
    'timeout' => 0,
    'password' => '',
    'dbindex' => 0,
  ),

but you can try those settings.

Redis gives no other error messages? This all seems quite boring and correct… The error in the Nextcloud log seems to claim there’s no file, as in, no thumbnails? You could use the thumbnail generator app to create them? See apps.nextcloud.com

Sorry, this isn’t my expertise either…

Hi, yes it temporary resolved this issue as noted above. Also restarting redis helps now and then.

Seems like TCP backlog setting for redis was overruled by somaxconn.

Hence this seems to be working for now:
sysctl -w net.core.somaxconn=1024

@jospoortvliet problem is still valid and seems to be related to the open files limit.

redis-benchmark -c 6000 -q -n 10 -t get,set
Could not connect to Redis at 127.0.0.1:6379: Can't create socket: Too many open files

Increasing the open files limit temporary resolves this issue:

ulimit -n 999999

Yet I guess there is smth wrong in the code as this issue is not only reported by me. Seen:

https://github.com/nextcloud/server/issues/6459

Problem is solved by turning of “files_system_check_changes”. As already noted back in on Github:

2 Likes