Hi All, I’ve been banging my head against a wall for a while now. I’m pretty new to all of this, so any help I can get would be much appreciated.
The issue:
I am running Memories and Recognize apps to process data on an external USB hard drive. Issues about exhausted PHP memory keep appearing in my log at a rate of 2 - 4 errors per hour.
{"reqId":"cHIWkvhED3lOQO4YxRVg","level":3,"time":"2023-09-03T22:02:26+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Allowed memory size of 536870912 bytes exhausted (tried to allocate 163840 bytes) at /var/www/html/custom_apps/recognize/lib/Clustering/MrdBallTree.php#449","userAgent":"--","version":"27.0.2.1","data":{"app":"PHP"},"id":"64f50ad27a0cd"}
To me, it looks like Recognize thinks it only has about 530mb of memory available. However I have set the memory and upload limit to 6.3gb, which is reflected in the System tab of Administrator Settings. Am I reading this correctly?
If so…
Before I raise a ticket with recognize, is there anything else I could do with Nextcloud Server to make sure Recognize is aware of the “actual” memory limit?
Setup
Nextcloud (apache) version: 27.0.2
Operating system and version: ubuntu 22.04 LTS running on a Lenovo Ideapad 510ISK
PHP version: 8.2.9
Recognize: v4.3.2
Docker compose file
version: '3'
services:
db:
image: mariadb:latest
container_name: nextcloud_mariadb
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
restart: unless-stopped
environment:
- MYSQL_ROOT_PASSWORD=XXXXXX
- MYSQL_PASSWORD=XXXXXXX
- MYSQL_DATABASE=XXXXXXX
- MYSQL_USER=XXXXXXX
volumes:
- /mnt/path-to-internal-ssd/nextcloud/mariadb:/var/lib/mysql
- /etc/localtime:/etc/localtime:ro
redis:
image: redis:latest
container_name: nextcloud_redis
restart: unless-stopped
volumes:
- /mnt/path-to-internal-ssd/nextcloud/redis:/data
- /etc/localtime:/etc/localtime:ro
app:
build:
context: .
dockerfile: nextcloud.Dockerfile
container_name: nextcloud_app
privileged: true
restart: unless-stopped
ports:
- 8888:80
depends_on:
- db
- redis
volumes:
- /mnt/path-to-internal-ssd/nextcloud/nextcloud:/var/www/html
- /mnt/path-to-external-hdd-2/docker/nextcloud/nextcloud/userdata:/var/www/html/data #external usb hdd 2 where NC user data is stored
- /mnt/path-to-external-hdd-1:/mnt/external_one #External usb hdd 1 containing shared files
- /mnt/path-to-internal-ssd:/mnt/internal_two
- /etc/localtime:/etc/localtime:ro
environment:
- MYSQL_PASSWORD=XXXXXXX
- MYSQL_DATABASE=XXXXXXX
- MYSQL_USER=XXXXXXX
- MYSQL_HOST=db
- MYSQL_PORT=3306
- REDIS_HOST=redis
- PHP_MEMORY_LIMIT=6000M
- PHP_UPLOAD_LIMIT=6000M
# command: nvidia-smi
deploy:
resources:
limits: #https://docs.nextcloud.com/server/20/admin_manual/installation/server_tuning.html#reducing-system-load (to stop NC using swap memory)
memory: 7000m
reservations:
devices:
- driver: nvidia
capabilities: [compute, video, utility, graphics]
memswap_limit: 7000m #based off https://stackoverflow.com/questions/70095537/how-to-use-swap-memory-in-docker-compose
devices:
- /dev/dri:/dev/dri
cron:
image: nextcloud:latest
container_name: nextcloud_cron
restart: unless-stopped
user: root
volumes_from: #https://help.nextcloud.com/t/different-file-permissions-in-cron-docker/95793/2
- app
entrypoint: /cron.sh
depends_on:
- app
Dockerfile
FROM nextcloud:apache
RUN set -ex; \
\
apt-get update; \
apt-get install -y --no-install-recommends \
ffmpeg \
ghostscript \
libmagickcore-6.q16-6-extra \
procps \
smbclient \
supervisor \
# libreoffice \
; \
rm -rf /var/lib/apt/lists/*
RUN set -ex; \
apt-get update; \
apt-get install -y --no-install-recommends \
libbz2-dev \
libc-client-dev \
libkrb5-dev \
libsmbclient-dev
RUN set -ex; \
docker-php-ext-configure imap --with-kerberos --with-imap-ssl; \
docker-php-ext-install \
bz2 \
imap
RUN set -ex; \
pecl install smbclient; \
docker-php-ext-enable smbclient;
RUN mkdir -p \
/var/log/supervisord \
/var/run/supervisord
COPY supervisord.conf /
ENV NEXTCLOUD_UPDATE=1
CMD ["/usr/bin/supervisord", "-c", "/supervisord.conf"]
config.php file:
P<?php
$CONFIG = array (
'htaccess.RewriteBase' => '/',
'memcache.local' => '\\OC\\Memcache\\APCu',
'apps_paths' =>
array (
0 =>
array (
'path' => '/var/www/html/apps',
'url' => '/apps',
'writable' => false,
),
1 =>
array (
'path' => '/var/www/html/custom_apps',
'url' => '/custom_apps',
'writable' => true,
),
),
'memcache.distributed' => '\\OC\\Memcache\\Redis',
'memcache.locking' => '\\OC\\Memcache\\Redis',
'redis' =>
array (
'host' => 'redis',
'password' => '',
'port' => 6379,
),
'instanceid' => 'XXXXXXXXXXXX',
'passwordsalt' => 'XXXXXXXXXXXX',
'secret' => 'XXXXXXXXXXXX',
'trusted_domains' =>
array (
0 => 'LOCAL.IP.ADDRESS:8888',
),
'datadirectory' => '/var/www/html/data',
'dbtype' => 'mysql',
'version' => '27.0.2.1',
'overwrite.cli.url' => 'http://LOCAL.IP.ADDRESS:8888',
'dbname' => 'XXXXXXXXXXXX',
'dbhost' => 'XXXXXXXXXXXX',
'dbport' => '',
'dbtableprefix' => 'oc_',
'mysql.utf8mb4' => true,
'dbuser' => 'XXXXXXXXXXXX',
'dbpassword' => 'XXXXXXXXXXXX',
'installed' => true,
'app_install_overwrite' =>
array (
0 => 'metadata',
1 => 'camerarawpreviews',
2 => 'memories',
3 => 'mediadc',
4 => 'cloud_py_api',
5 => 'files_downloadactivity',
),
'memories.exiftool' => '/var/www/html/custom_apps/memories/exiftool-bin/exiftool-amd64-glibc',
'memories.vod.path' => '/var/www/html/custom_apps/memories/exiftool-bin/go-vod-amd64',
'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
'memories.vod.ffprobe' => '/usr/bin/ffprobe',
'memories.gis_type' => 1,
'enabledPreviewProviders' =>
array (
0 => 'OC\\Preview\\Image',
1 => 'OC\\Preview\\HEIC',
2 => 'OC\\Preview\\Movie',
3 => 'OC\\Preview\\MKV',
4 => 'OC\\Preview\\MP4',
5 => 'OC\\Preview\\AVI',
6 => 'OC\\Preview\\TIFF',
),
'preview_max_memory' => 5000,
'preview_max_filesize_image' => 50,
'preview_max_x' => 1024,
'preview_max_y' => 1024,
'memories.vod.nvenc.temporal_aq' => false,
'maintenance' => false,
'preview_max_scale_factor' => 1,
'memories.vod.disable' => false,
'loglevel' => 2,
);
Could this somehow be related to this other issue error that pops into the log from time to time?
{"reqId":"PXfPzsJp9k92R5IXoGLO","level":3,"time":"2023-09-03T13:48:15+00:00","remoteAddr":"192.168.20.57","user":"Jasha","app":"PHP","method":"GET","url":"/apps/memories/api/system-status","message":"Undefined array key \"preview_ffmpeg_path\" at /var/www/html/custom_apps/memories/lib/Util.php#363","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36","version":"27.0.2.1","data":{"app":"PHP"},"id":"64f513860d735"}
Thank you!
