Fatal error over night: /tmp not present or writable & mysql "permission denied"

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 20.0.5): 27.1.2.1
Operating system and version (eg, Ubuntu 20.04): Ubuntu 22 LTS
nginx version (eg, Apache 2.4.25): nginx 1.18.0
PHP version (eg, 7.4): 8.1

Is this the first time you’ve seen this error? (Y/N): Yes

Steps to replicate it:

  1. it literally broke over night. no input whatsoever.
  2. the only thing that runs overnight is Decatec’s NextcloudBackupRestore bash script and I doubt that’s the culprit, though I’ve seen it fail very ungraciously before when the hdd is just full

The output of your Nextcloud log in Admin > Logging:

Warning	no app in context	Temporary directory /tmp is not present or writable	
2023-10-16T10:48:36+0200
Fatal	no app in context	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql-temptable-591-b0-3e.MAI' (Errcode: 13 "Permission denied")	
2023-10-16T10:48:36+0200
Warning	no app in context	Temporary directory /tmp is not present or writable	
2023-10-16T10:48:36+0200
Fatal	no app in context	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql-temptable-591-af-40.MAI' (Errcode: 13 "Permission denied")	
2023-10-16T10:48:36+0200
Warning	no app in context	Temporary directory /tmp is not present or writable	
2023-10-16T10:48:36+0200
Fatal	no app in context	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql-temptable-591-ae-3d.MAI' (Errcode: 13 "Permission denied")	
2023-10-16T10:48:36+0200
Warning	no app in context	Temporary directory /tmp is not present or writable	
2023-10-16T10:48:36+0200
Fatal	no app in context	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql-temptable-591-ad-3f.MAI' (Errcode: 13 "Permission denied")	
2023-10-16T10:48:36+0200
Warning	no app in context	Temporary directory /tmp is not present or writable	
2023-10-16T10:48:36+0200
Fatal	no app in context	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql-temptable-591-ac-3c.MAI' (Errcode: 13 "Permission denied")	
2023-10-16T10:48:36+0200
Warning	no app in context	Temporary directory /tmp is not present or writable	
2023-10-16T10:48:36+0200

and so on and so forth

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'passwordsalt' => 'blabla',
  'secret' => 'blabla',
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => 'blabla',
  ),
  'datadirectory' => '/var/nc-data',
  'dbtype' => 'mysql',
  'version' => '27.1.2.1',
  'overwrite.cli.url' => 'blabla',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'blabla',
  'installed' => true,
  'default_phone_region' => 'DE',
  'instanceid' => 'blabla',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'redis' =>
  array (
    'host' => '/run/redis/redis.sock',
    'port' => '0',
    'timeout' => '0.0',
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'filelocking.enabled' => 'true',
  'enable_previews' => 'true',
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\GIF',
    3 => 'OC\\Preview\\BMP',
    4 => 'OC\\Preview\\XBitmap',
    5 => 'OC\\Preview\\Movie',
    6 => 'OC\\Preview\\PDF',
    7 => 'OC\\Preview\\MP3',
    8 => 'OC\\Preview\\TXT',
    9 => 'OC\\Preview\\MarkDown',
    10 => 'OC\\Preview\\HEIC',
    11 => 'OC\\Preview\\TIFF',
    12 => 'OC\\Preview\\MP4',
    13 => 'OC\\Preview\\AVI',
    14 => 'OC\\Preview\\MKV',
  ),
  'preview_max_memory' => '4096',
  'preview_max_filesize_image' => '256',
  'preview_max_x' => '2048',
  'preview_max_y' => '1080',
  'preview_max_scale_factor' => '1',
  'auth.bruteforce.protection.enabled' => 'true',
  'trashbin_retention_obligation' => 'auto,7',
  'skeletondirectory' => '',
  'defaultapp' => 'file',
  'activity_expire_days' => '14',
  'integrity.check.disabled' => 'false',
  'updater.release.channel' => 'stable',
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'blabla',
  'mail_domain' => 'blabla',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'bla',
  'mail_smtpport' => 'bla',
  'mail_smtpname' => 'blabla',
  'mail_smtppassword' => 'blabla',
  'maintenance' => true,
  'theme' => '',
  'loglevel' => 2,
  'log_type' => 'file',
  'memories.exiftool_no_local' => true,
  'memories.ffmpeg_path' => '/usr/bin/ffmpeg',
  'memories.ffprobe_path' => '/usr/bin/ffprobe',
  'memories.transcoder' => '/var/www/nextcloud/apps/memories/exiftool-bin/go-vod-amd64',
  'memories.no_transcode' => false,
  'memories.qsv' => true,
  'memories.vod.path' => '/var/www/nextcloud/apps/memories/exiftool-bin/go-vod-amd64',
  'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
  'memories.vod.ffprobe' => '/usr/bin/ffprobe',
  'updater.secret' => 'blablabla',

The output of your nginx error log in /var/log/____:

2023/10/16 10:42:11 [warn] 1148#1148: "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/etc/letsencrypt/live/DOMAIN/fullchain.pem"
2023/10/16 10:42:11 [warn] 1148#1148: "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/etc/letsencrypt/live/DOMAIN/fullchain.pem"

The output of your php8.1-fpm log in /var/log/php8.1-fpm.log:

[15-Oct-2023 00:00:02] NOTICE: error log file re-opened
[16-Oct-2023 10:24:59] NOTICE: Terminating ...
[16-Oct-2023 10:24:59] NOTICE: exiting, bye-bye!
[16-Oct-2023 10:24:59] NOTICE: fpm is running, pid 141996
[16-Oct-2023 10:24:59] NOTICE: ready to handle connections
[16-Oct-2023 10:24:59] NOTICE: systemd monitor interval set to 10000ms
[16-Oct-2023 10:27:42] NOTICE: Terminating ...
[16-Oct-2023 10:27:42] NOTICE: exiting, bye-bye!
[16-Oct-2023 10:27:42] NOTICE: fpm is running, pid 142262
[16-Oct-2023 10:27:42] NOTICE: ready to handle connections
[16-Oct-2023 10:27:42] NOTICE: systemd monitor interval set to 10000ms
[16-Oct-2023 10:41:32] NOTICE: Terminating ...
[16-Oct-2023 10:41:32] NOTICE: exiting, bye-bye!
[16-Oct-2023 10:42:11] NOTICE: fpm is running, pid 1105
[16-Oct-2023 10:42:11] NOTICE: ready to handle connections
[16-Oct-2023 10:42:11] NOTICE: systemd monitor interval set to 10000ms


The output of your NextcloudBackupRestore.log in as mentioned above:

04:05:01: Set maintenance mode for Nextcloud...
Maintenance mode enabled
Done

04:05:02: Stopping web server...
Done

04:05:02: Creating backup of Nextcloud file directory...
Done

04:05:53: Creating backup of Nextcloud data directory...
Ignoring Nextcloud updater backup directory
Done

05:34:44: Backup Nextcloud database (MySQL/MariaDB)...
mysqldump: Error: 'Can't create/write to file '/tmp/#sql-temptable-5b3-3cb8-2.MAI' (Errcode: 13 "Permission denied")' when trying to dump tablespaces
mysqldump: Couldn't execute 'show fields from `oc_accounts`': Can't create/write to file '/tmp/#sql-temptable-5b3-3cb8-6.MAI' (Errcode: 13 "Permission denied") (1)



The output of MariaDB status:

MariaDB [(none)]> status
--------------
mysql  Ver 15.1 Distrib 10.6.12-MariaDB, for debian-linux-gnu (x86_64) using  EditLine wrapper

Connection id:          301
Current database:
Current user:           root@localhost
SSL:                    Not in use
Current pager:          stdout
Using outfile:          ''
Using delimiter:        ;
Server:                 MariaDB
Server version:         10.6.12-MariaDB-0ubuntu0.22.04.1-log Ubuntu 22.04
Protocol version:       10
Connection:             Localhost via UNIX socket
Server characterset:    utf8mb4
Db     characterset:    utf8mb4
Client characterset:    utf8mb4
Conn.  characterset:    utf8mb4
UNIX socket:            /var/run/mysqld/mysqld.sock
Uptime:                 41 min 53 sec

Threads: 2  Questions: 2607  Slow queries: 0  Opens: 67  Open tables: 60  Queries per second avg: 1.037

I’m absolutely stumped. I don’t have any idea how to fix this, but I’m also not an expert. I’m good at following directions and online guides, that’s how I got this far :smiley:

So, as far as I know, nothing particular happened and I didn’t do anything overnight. The only thing that runs overnight that could possibly affect my nc install is decatec’s nextcloudbackuprestore bash script as mentioned above because it knows both the webserveruser and the db user. However, it went through fine the day before and today I wake up to this mess. Just googling the two error messages in NC log (/tmp not available and “Doctrine\DBAL\Exception\DriverException: permission denied”) seems to imply that I’m completely alone with this.

I think there might be a “quick fix” in restoring user/folder permissions, but both the nextcloud data folder and the installation itself under /var/www/nextcloud are recursively owned by the web server user www-data. I have no idea how to resolve the /tmp folder and mysql permission thing.

Edit 1

Because I was stuck in the dreaded Ubuntu boot loop when I actually physically tried to login to the machine I followed this guide and noticed that indeed my /tmp (for everyone else confused: yes, the actual /tmp directory lying around in the root of your ubuntu system on the same level as /var or /mnt) was set to the following:

drwxrwxr-x  23 plex     plex            500 Okt 16 12:09 tmp

Now according to multiple sources (example), /tmp should never be anything other than drwxrwxrwt and owned by root. So

 
sudo chmod 1777 /tmp
sudo chown root:root /tmp

and reboot the machine for good measure … aaaaaaaaand while I managed to physically log in this time, it appears that right after booting up my freshly installed (read: yesterday) Plex docker container managed to AGAIN grab /tmp for its user plex:plex.

WHY?

Well fock.

Soooo it turns out that I was following along this tutorial last night to get an automated home media server going. I was gonna use a bundle of docker containers because that seemed less likely to break something. Anyway, I copy-pasted a bunch of docker compose yaml nonsense including this fine bit here:

  plex: # Media Server
    container_name: plex
    image: cr.hotio.dev/hotio/plex
    restart: unless-stopped
    logging:
      driver: json-file
    environment:
      - PUID=$PUID
      - PGID=$PGID
      - TZ=$TZ
      - UMASK=002
      - DEBUG=no
      - PLEX_CLAIM=blabla
      - PLEX_PASS=$PLEX_PASS
      - ADVERTISE_IP="http://192.168.178.72:32400"
    volumes:
      - ./plex:/config:rw
      - $PLEXMEDIA:/data/media:rw
      - /tmp:/transcode:rw
    ports:
      - 32400:32400
    devices:
      - /dev/dri:/dev/dri #required for Synology users
    network_mode: host

and the trained eye will immediately notice that while I’m usind .env variables for PUID/PGID, those are probably not the default values the author had in mind (since I’m using a plex user) and the docker container for plex also just grabs /tmp as a transcoding directory. which probably isn’t the smartest way to go about this when you could simply not grad the entire root /tmp but create a subfolder in there, but what do I know.

So for now I’ll just axe the plex container and go about this another way.