"Random" crash of host VM, second occurance in 2 weeks

Nextcloud version (eg, 20.0.5): 28.0.3
Operating system and version : Ubuntu 20.04.6 LTS
Apache or nginx version: Apache 2.4.41
PHP version : 8.1.27

I’m running Nextcloud in a KVM Virtual Machine in Proxmox. Nothing else is in that VM, not even the database. Today, for the second time in 2 weeks, the VM just silently stopped - and just as last time, almost exactly on the hour (last time 1pm, this time 11am). I performed the last update on 7th march.

The system logs show nothing but the usual nextcloud php cron executions. It’s 2 cron jobs:

  1. (www-data) CMD (php --define apc.enable_cli=1 -f /var/www/nextcloud/cron.php)
  2. (www-data) CMD (php /var/www/nextcloud/occ preview:pre-generate)

However, the logs state only one of these sessions was closed, while usually, both cron sessions are closed (of course). So I assume the VM stopped _during the execution of one of those cron jobs`.

I know this is very vague, but if you have any idea how to debug this I’d be very grateful. Of course, feel free to request additional info if you know anything that may be of interest.

The output of your Nextcloud log in Admin > Logging:
There is exactly one log entry at the time in question.

{
  "reqId": "LIGBJImO30UzZQTAKlCt",
  "level": 2,
  "time": "2024-04-08T09:00:04+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "no app in context",
  "method": "",
  "url": "--",
  "message": "error while discovering ocm provider",
  "userAgent": "--",
  "version": "28.0.3.2",
  "exception": {
    "Exception": "GuzzleHttp\\Exception\\ConnectException",
    "Message": "cURL error 7: Failed to connect to my.nextcloud.domain port 443: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://my.nextcloud.domain/ocm-provider/",
    "Code": 0,
    "Trace": [
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php",
        "line": 158,
        "function": "createRejection",
        "class": "GuzzleHttp\\Handler\\CurlFactory",
        "type": "::",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php",
        "line": 110,
        "function": "finishError",
        "class": "GuzzleHttp\\Handler\\CurlFactory",
        "type": "::"
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php",
        "line": 47,
        "function": "finish",
        "class": "GuzzleHttp\\Handler\\CurlFactory",
        "type": "::"
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php",
        "line": 137,
        "function": "__invoke",
        "class": "GuzzleHttp\\Handler\\CurlHandler",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php",
        "line": 121,
        "function": "GuzzleHttp\\{closure}",
        "class": "GuzzleHttp\\Middleware",
        "type": "::",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php",
        "line": 35,
        "function": "OC\\Http\\Client\\{closure}",
        "class": "OC\\Http\\Client\\DnsPinMiddleware",
        "type": "->",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php",
        "line": 31,
        "function": "__invoke",
        "class": "GuzzleHttp\\PrepareBodyMiddleware",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php",
        "line": 71,
        "function": "GuzzleHttp\\{closure}",
        "class": "GuzzleHttp\\Middleware",
        "type": "::",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php",
        "line": 63,
        "function": "__invoke",
        "class": "GuzzleHttp\\RedirectMiddleware",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php",
        "line": 75,
        "function": "GuzzleHttp\\{closure}",
        "class": "GuzzleHttp\\Middleware",
        "type": "::",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php",
        "line": 331,
        "function": "__invoke",
        "class": "GuzzleHttp\\HandlerStack",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php",
        "line": 168,
        "function": "transfer",
        "class": "GuzzleHttp\\Client",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php",
        "line": 187,
        "function": "requestAsync",
        "class": "GuzzleHttp\\Client",
        "type": "->",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/nextcloud/lib/private/Http/Client/Client.php",
        "line": 230,
        "function": "request",
        "class": "GuzzleHttp\\Client",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/OCM/OCMDiscoveryService.php",
        "line": 91,
        "function": "get",
        "class": "OC\\Http\\Client\\Client",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/files_sharing/lib/External/Storage.php",
        "line": 87,
        "function": "discover",
        "class": "OC\\OCM\\OCMDiscoveryService",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Mount/MountPoint.php",
        "line": 168,
        "function": "__construct",
        "class": "OCA\\Files_Sharing\\External\\Storage",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Mount/MountPoint.php",
        "line": 193,
        "function": "createStorage",
        "class": "OC\\Files\\Mount\\MountPoint",
        "type": "->",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Mount/MountPoint.php",
        "line": 294,
        "function": "getStorage",
        "class": "OC\\Files\\Mount\\MountPoint",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Config/UserMountCache.php",
        "line": 83,
        "function": "getStorageRootId",
        "class": "OC\\Files\\Mount\\MountPoint",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/SetupManager.php",
        "line": 297,
        "function": "registerMounts",
        "class": "OC\\Files\\Config\\UserMountCache",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/SetupManager.php",
        "line": 221,
        "function": "afterUserFullySetup",
        "class": "OC\\Files\\SetupManager",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/SetupManager.php",
        "line": 498,
        "function": "setupForUser",
        "class": "OC\\Files\\SetupManager",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Mount/Manager.php",
        "line": 231,
        "function": "setupForProvider",
        "class": "OC\\Files\\SetupManager",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Node/Root.php",
        "line": 445,
        "function": "getMountsByMountProvider",
        "class": "OC\\Files\\Mount\\Manager",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Node/LazyUserFolder.php",
        "line": 80,
        "function": "getByIdInPath",
        "class": "OC\\Files\\Node\\Root",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/files_sharing/lib/SharedStorage.php",
        "line": 144,
        "function": "getById",
        "class": "OC\\Files\\Node\\LazyUserFolder",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/files_sharing/lib/SharedStorage.php",
        "line": 526,
        "function": "init",
        "class": "OCA\\Files_Sharing\\SharedStorage",
        "type": "->"
      },
      {
        "function": "getSourceStorage",
        "class": "OCA\\Files_Sharing\\SharedStorage",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php",
        "line": 524,
        "function": "call_user_func_array"
      },
      {
        "file": "/var/www/nextcloud/apps/files_sharing/lib/Cache.php",
        "line": 87,
        "function": "__call",
        "class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Cache/Wrapper/CacheJail.php",
        "line": 80,
        "function": "getRoot",
        "class": "OCA\\Files_Sharing\\Cache",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Cache/Wrapper/CacheJail.php",
        "line": 121,
        "function": "getSourcePath",
        "class": "OC\\Files\\Cache\\Wrapper\\CacheJail",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/files_sharing/lib/Cache.php",
        "line": 126,
        "function": "get",
        "class": "OC\\Files\\Cache\\Wrapper\\CacheJail",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/View.php",
        "line": 1324,
        "function": "get",
        "class": "OCA\\Files_Sharing\\Cache",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/View.php",
        "line": 1374,
        "function": "getCacheEntry",
        "class": "OC\\Files\\View",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Node/Root.php",
        "line": 203,
        "function": "getFileInfo",
        "class": "OC\\Files\\View",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Node/Folder.php",
        "line": 135,
        "function": "get",
        "class": "OC\\Files\\Node\\Root",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/Files/Node/Folder.php",
        "line": 144,
        "function": "get",
        "class": "OC\\Files\\Node\\Folder",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/memories/lib/Service/Index.php",
        "line": 124,
        "function": "nodeExists",
        "class": "OC\\Files\\Node\\Folder",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/memories/lib/Service/Index.php",
        "line": 191,
        "function": "indexFolder",
        "class": "OCA\\Memories\\Service\\Index",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/memories/lib/Service/Index.php",
        "line": 110,
        "function": "indexFolder",
        "class": "OCA\\Memories\\Service\\Index",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/memories/lib/Cron/IndexJob.php",
        "line": 92,
        "function": "indexUser",
        "class": "OCA\\Memories\\Service\\Index",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/User/Manager.php",
        "line": 656,
        "function": "OCA\\Memories\\Cron\\{closure}",
        "class": "OCA\\Memories\\Cron\\IndexJob",
        "type": "->",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/nextcloud/apps/memories/lib/Cron/IndexJob.php",
        "line": 100,
        "function": "callForSeenUsers",
        "class": "OC\\User\\Manager",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/memories/lib/Cron/IndexJob.php",
        "line": 63,
        "function": "indexAllUsers",
        "class": "OCA\\Memories\\Cron\\IndexJob",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/public/BackgroundJob/Job.php",
        "line": 81,
        "function": "run",
        "class": "OCA\\Memories\\Cron\\IndexJob",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php",
        "line": 102,
        "function": "start",
        "class": "OCP\\BackgroundJob\\Job",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php",
        "line": 92,
        "function": "start",
        "class": "OCP\\BackgroundJob\\TimedJob",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/cron.php",
        "line": 152,
        "function": "execute",
        "class": "OCP\\BackgroundJob\\TimedJob",
        "type": "->"
      }
    ],
    "File": "/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php",
    "Line": 210,
    "message": "error while discovering ocm provider",
    "exception": [],
    "remote": "https://nc.a-0.me",
    "CustomMessage": "error while discovering ocm provider"
  },
  "id": "6614234710a91"
}

I verified that https://my.nextcloud.domain is reachable from the nextcloud VM. Even though only HTTP is exposed locally, it works since the domain is resolved to the public IP which exposes https. The log entry therefore may indicate that nextcloud already stopped working, while this process still tried to curl it.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'redacted',
  'passwordsalt' => 'redacted',
  'secret' => 'redacted',
  'trusted_domains' => 
  array (
    0 => 'some local IP',
    1 => 'local IP of reverse proxy machine',
    2 => 'my.nextcloud.domain',
  ),
  'trusted_proxies' => 
  array (
    0 => 'local reverse proxy IP',
  ),
  'allow_local_remote_servers' => true,
  'datadirectory' => '/mnt/data',
  'dbtype' => 'pgsql',
  'version' => '28.0.3.2',
  'overwriteprotocol' => 'https',
  'overwrite.cli.url' => 'http://local IP/nextcloud',
  'htaccess.RewriteBase' => '/',
  'dbname' => 'nextclouddb',
  'dbhost' => '...',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => 'redacted',
  'dbpersistent' => true,
  'installed' => true,
  'maintenance' => false,
  'has_rebuilt_cache' => true,
  'theme' => '',
  'loglevel' => 2,
  'logfile' => '/var/log/nextcloud/nextcloud.log',
  'updater.release.channel' => 'stable',
  'default_phone_region' => 'DE',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'app_install_overwrite' => 
  array (
    0 => 'previewgenerator',
    1 => 'fulltextsearch',
    2 => 'files_fulltextsearch',
    3 => 'files_fulltextsearch_tesseract',
    4 => 'fulltextsearch_elasticsearch',
  ),
  'preview_max_x' => '2048',
  'preview_max_y' => '2048',
  'jpeg_quality' => '60',
  'trashbin_retention_obligation' => '30, 360',
  'preview_max_memory' => 4096,
  'preview_max_filesize_image' => 256,
  'memories.exiftool' => '/var/www/nextcloud/apps/memories/bin-ext/exiftool-amd64-glibc',
  'enabledPreviewProviders' => 
  array (
    0 => 'OC\\Preview\\Image',
    1 => 'OC\\Preview\\HEIC',
    2 => 'OC\\Preview\\TIFF',
    3 => 'OC\\Preview\\Movie',
    4 => 'OC\\Preview\\MKV',
    5 => 'OC\\Preview\\MP4',
    6 => 'OC\\Preview\\AVI',
  ),
  'memories.ffmpeg_path' => '/usr/bin/ffmpeg',
  'memories.ffprobe_path' => '/usr/bin/ffprobe',
  'memories.transcoder' => '/var/www/nextcloud/apps/memories/exiftool-bin/go-vod-amd64',
  'memories.no_transcode' => false,
  'memories.qsv' => false,
  'mail_domain' => 'base domain',
  'mail_from_address' => 'nextcloud',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'mail.mydomain',
  'mail_smtpport' => '25',
  'mail_smtpname' => 'nextcloud@mydomain',
  'mail_smtppassword' => 'redacted',
  'activity_expire_days' => 180,
  'memories.vod.path' => '/var/www/nextcloud/apps/memories/bin-ext/go-vod-amd64',
  'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
  'memories.vod.ffprobe' => '/usr/bin/ffprobe',
  'davstorage.request_timeout' => 90,
  'memories.gis_type' => 2,
);

The output of your Apache/nginx/system log in /var/log/____:
Nothing at that time