Internal server error - out of the blue!

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 12.0.2): 16.0.4.1
Operating system and version (eg, Ubuntu 17.04): Ubuntu 18.04.3 LTS (AWS)
Apache or nginx version (eg, Apache 2.4.25): 2.4.29
PHP version (eg, 7.1): 7.2.24-0ubuntu0.18.04.1

The issue you are facing:

So I’ve got a NC server running in AWS, and it’s been happy for months. I’m the only user, and most of the time all it’s doing is catching auto-upload pictures from my phone, GPS data from the phone-track app, and syncing contacts and Joplin notes.
This is running in AWS Lightsail with a secondary disk attached for data only, running zfs.

Yesterday I started getting the below message when trying to log in to the web interface:

Internal Server Error

The server was unable to complete your request.
If this happens again, please send the technical details below to the server administrator.
More details can be found in the server log.

I’m also getting connection errors from my various apps. I can still SSH in to the O/S, and if I put NC in to maintenance mode I get the maintenance mode message at the web portal instead of the error. While scanning the logs, I feel this might be due to the system thinking it’s run out of storage, but I’ve emptied the trashbin, and there should be plently of room in the ncdata directory.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Navigate to my NC login page while NC is not in maintenance mode.

The output of your Nextcloud log in Admin > Logging: (This is actually the last line of my nextcloud.log file. This page crashed when I tried to paste the whole thing.)

{"reqId":"bnaaMxoDeSEwNCgcUmYJ","level":4,"time":"2019-11-24T01:57:42+00:00","remoteAddr":"49.195.186.200","user":"Jay","app":"webdav","method":"PUT","url":"\/remote.php\/dav\/uploads\/Jay\/8a9d6c7de88219b98bbcb8a665ad5378\/0000000000000000-0000000001023999","message":{"Exception":"Sabre\\DAV\\Exception","Message":"Error while copying file to target location (copied bytes: 0, expected filesize: 1024000 )","Code":0,"Trace":[{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Upload\/UploadFolder.php","line":45,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["0000000000000000-0000000001023999",null]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":1096,"function":"createFile","class":"OCA\\DAV\\Upload\\UploadFolder","type":"->","args":["0000000000000000-0000000001023999",null]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php","line":525,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["uploads\/Jay\/8a9d6c7de88219b98bbcb8a665ad5378\/0000000000000000-0000000001023999",null,null]},{"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https:\/\/cloud.my.domain\/remote.php\/dav\/uploads\/Jay\/8a9d6c7de88219b98bbcb8a665ad5378\/0000000000000000-0000000001023999","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpPut"],[{"absoluteUrl":"https:\/\/cloud.my.domain\/remote.php\/dav\/uploads\/Jay\/8a9d6c7de88219b98bbcb8a665ad5378\/0000000000000000-0000000001023999","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:PUT",[{"absoluteUrl":"https:\/\/cloud.my.domain\/remote.php\/dav\/uploads\/Jay\/8a9d6c7de88219b98bbcb8a665ad5378\/0000000000000000-0000000001023999","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https:\/\/cloud.my.domain\/remote.php\/dav\/uploads\/Jay\/8a9d6c7de88219b98bbcb8a665ad5378\/0000000000000000-0000000001023999","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Server.php","line":316,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/remote.php","line":163,"args":["\/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php"],"function":"require_once"}],"File":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php","Line":218,"CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.0","version":"16.0.4.1"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'passwordsalt' => 'XXXXXXX',
  'secret' => 'XXXXXXX',
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => '172.26.9.48',
    2 => 'cloud.my.domain',
    3 => 'cloud.my.domain',
  ),
  'datadirectory' => '/mnt/ncdata',
  'dbtype' => 'pgsql',
  'version' => '16.0.4.1',
  'overwrite.cli.url' => 'https://cloud.my.domain/',
  'dbname' => 'nextcloud_db',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'ncadmin',
  'dbpassword' => 'XXXXXXX',
  'installed' => true,
  'instanceid' => 'ocjtymakppni',
  'mail_smtpmode' => 'smtp',
  'remember_login_cookie_lifetime' => '1800',
  'log_rotate_size' => '10485760',
  'trashbin_retention_obligation' => 'auto, 180',
  'versions_retention_obligation' => 'auto, 365',
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'filelocking.enabled' => true,
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'port' => 0,
    'timeout' => 0.5,
    'dbindex' => 0,
    'password' => 'XXXXXXX',
  ),
  'htaccess.RewriteBase' => '/',
  'maintenance' => false,
);

The output of your Apache/nginx/system log in /var/log/____:

[Sun Nov 24 06:25:05.382352 2019] [mpm_event:notice] [pid 8329:tid 139955154111424] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Sun Nov 24 06:25:05.382377 2019] [core:notice] [pid 8329:tid 139955154111424] AH00094: Command line: '/usr/sbin/apache2'
[Sun Nov 24 06:42:06.041620 2019] [proxy_fcgi:error] [pid 10001:tid 139954922620672] [client 49.195.158.190:16545] AH01067: Failed to read FastCGI header
[Sun Nov 24 17:29:51.951902 2019] [mpm_event:notice] [pid 8329:tid 139955154111424] AH00491: caught SIGTERM, shutting down
[Sun Nov 24 17:30:50.896311 2019] [mpm_event:notice] [pid 4933:tid 140134821243840] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Sun Nov 24 17:30:50.931128 2019] [core:notice] [pid 4933:tid 140134821243840] AH00094: Command line: '/usr/sbin/apache2'
[Sun Nov 24 17:33:05.652742 2019] [mpm_event:notice] [pid 4933:tid 140134821243840] AH00491: caught SIGTERM, shutting down
[Sun Nov 24 17:33:22.368253 2019] [mpm_event:notice] [pid 6291:tid 140504640781248] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Sun Nov 24 17:33:22.368334 2019] [core:notice] [pid 6291:tid 140504640781248] AH00094: Command line: '/usr/sbin/apache2'
[Sun Nov 24 17:34:57.940828 2019] [mpm_event:notice] [pid 6291:tid 140504640781248] AH00491: caught SIGTERM, shutting down
[Sun Nov 24 18:19:39.820733 2019] [mpm_event:notice] [pid 4419:tid 140238532160448] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Sun Nov 24 18:19:39.840866 2019] [core:notice] [pid 4419:tid 140238532160448] AH00094: Command line: '/usr/sbin/apache2'
[Sun Nov 24 21:29:27.726944 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16339] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:29:47.757242 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16347] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:30:07.741903 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16357] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:30:27.742029 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16358] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:30:48.317038 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16368] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:31:08.337228 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16382] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:31:28.331920 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16383] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:31:48.382301 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16385] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:32:08.352272 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16388] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:32:28.382169 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16390] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:32:48.386789 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16395] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:33:08.407022 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16399] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:33:28.406627 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16403] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:33:48.392001 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16411] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:34:08.441890 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16412] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:34:28.431683 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16414] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:34:48.447292 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16416] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:35:08.422105 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16461] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:35:28.436810 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16493] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:35:48.461936 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16534] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:36:09.471776 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16575] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:36:29.462116 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16615] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:36:49.496672 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16624] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:37:09.511778 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16647] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:37:29.506899 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16679] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:37:49.496992 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16692] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:38:09.481752 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16694] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:38:29.506621 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16710] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:38:49.511782 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16739] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:39:09.581901 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16743] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:39:29.532197 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16753] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:39:49.556871 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16770] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:40:09.582434 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16785] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:40:29.566865 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16797] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:40:49.562182 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16804] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:41:09.646715 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16812] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:41:29.561998 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16819] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:41:49.641862 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16827] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:42:09.632137 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16834] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:42:29.681942 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16845] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:42:49.672190 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16851] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:43:09.676989 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16860] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:43:29.732033 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16869] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:43:49.747260 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16878] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:44:09.646817 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16888] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:44:29.781674 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16894] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:44:49.747120 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16901] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:45:09.791683 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16906] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:45:29.767096 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16963] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:45:49.782264 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16985] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:46:09.817194 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16997] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:46:29.921660 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:17006] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:46:49.887547 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:17016] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:47:09.906942 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:17026] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:47:29.911997 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:17032] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:47:49.956546 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:17038] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:48:09.971958 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:17045] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:48:29.911811 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:17054] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:48:49.971908 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:17072] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:49:10.082136 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:17086] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:49:29.972067 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:17090] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:49:50.067206 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:17100] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:50:10.072114 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:17105] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:50:30.016881 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:17122] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:50:50.061965 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:17137] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:51:10.061914 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:17151] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:51:30.051929 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16151] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:51:50.101413 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16166] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:52:10.126893 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16183] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:52:30.131490 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16192] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:52:50.151941 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16200] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:53:10.226584 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16211] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:53:30.172008 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16216] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:53:50.196465 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16229] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:54:10.226709 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16238] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:54:30.226333 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16243] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:54:50.296806 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16246] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:55:10.271743 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16254] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:55:30.287924 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16262] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:55:50.316461 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16267] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:56:10.301700 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16283] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:56:30.391348 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16295] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:56:50.356618 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16306] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:57:10.371444 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16314] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:57:30.361616 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16319] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:57:50.426124 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16326] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:58:10.436355 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16337] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:58:30.431606 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16341] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:58:50.451087 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16348] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:59:10.400938 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16356] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:59:30.495970 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16366] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 21:59:50.481008 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16381] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:00:10.551261 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16397] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:00:30.581011 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16416] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:00:50.576254 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16430] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:01:10.571180 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16439] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:01:30.551870 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16445] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:01:50.601186 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16462] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:02:10.581005 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16469] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:02:30.605953 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16474] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:02:50.631095 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16481] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:03:10.711434 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16490] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:03:31.295953 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16499] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:03:50.701358 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16511] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:04:10.711098 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16525] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:04:30.741531 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16535] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:04:50.750951 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16538] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:05:10.751425 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16540] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:05:30.786215 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16545] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:05:50.811169 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16550] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:06:10.811374 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16554] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:06:30.826319 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16561] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:06:50.830897 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16563] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:07:10.856235 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16568] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:07:30.841336 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16577] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:07:50.871343 2019] [proxy_fcgi:error] [pid 4425:tid 140238250313472] [client 49.195.158.190:16581] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:08:10.891396 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16583] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:08:30.896598 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16587] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:08:50.945904 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16589] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:09:10.976329 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16594] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:09:30.936132 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16619] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:09:53.665757 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16629] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:10:13.655868 2019] [proxy_fcgi:error] [pid 4424:tid 140238124414720] [client 49.195.158.190:16642] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:10:33.675897 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16647] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:10:53.690812 2019] [proxy_fcgi:error] [pid 4425:tid 140238199957248] [client 49.195.158.190:16655] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:11:13.650786 2019] [proxy_fcgi:error] [pid 4425:tid 140238199957248] [client 49.195.158.190:16657] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:11:33.651288 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16670] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:11:53.645801 2019] [proxy_fcgi:error] [pid 4425:tid 140238199957248] [client 49.195.158.190:16673] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:12:13.675762 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16678] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:12:33.650896 2019] [proxy_fcgi:error] [pid 4425:tid 140238199957248] [client 49.195.158.190:16682] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:12:53.675827 2019] [proxy_fcgi:error] [pid 4425:tid 140238199957248] [client 49.195.158.190:16687] AH01071: Got error 'Primary script unknown\n'
[Sun Nov 24 22:13:13.665728 2019] [proxy_fcgi:error] [pid 14484:tid 140238418167552] [client 49.195.158.190:16688] AH01071: Got error 'Primary script unknown\n'
[Mon Nov 25 06:25:04.703964 2019] [mpm_event:notice] [pid 4419:tid 140238532160448] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Nov 25 06:25:04.720975 2019] [mpm_event:notice] [pid 4419:tid 140238532160448] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Mon Nov 25 06:25:04.721002 2019] [core:notice] [pid 4419:tid 140238532160448] AH00094: Command line: '/usr/sbin/apache2'
[Mon Nov 25 10:15:06.760795 2019] [mpm_event:notice] [pid 4419:tid 140238532160448] AH00491: caught SIGTERM, shutting down
[Mon Nov 25 10:16:14.385202 2019] [mpm_event:notice] [pid 8509:tid 140170874350528] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Mon Nov 25 10:16:14.437062 2019] [core:notice] [pid 8509:tid 140170874350528] AH00094: Command line: '/usr/sbin/apache2'
[Mon Nov 25 10:22:17.791062 2019] [mpm_event:notice] [pid 8509:tid 140170874350528] AH00491: caught SIGTERM, shutting down
[Mon Nov 25 10:22:33.652490 2019] [mpm_event:notice] [pid 10366:tid 139834093722560] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Mon Nov 25 10:22:33.652582 2019] [core:notice] [pid 10366:tid 139834093722560] AH00094: Command line: '/usr/sbin/apache2'

Just came back to say that I figured it out!

Seems like I had a bunch on zfs-auto-snapshots going one which had gobbled up all my capacity, without reporting it in that places I was looking!

Deleted all my snapshots, now I’m back up and running :slight_smile: