Maintenance mode enabled and I don't know why

Nextcloud version (eg, 12.0.2): 16.04
Operating system and version (eg, Ubuntu 17.04): Ubuntu 16.0.4
PHP version (eg, 7.1): 7.2.22

The issue you are facing:
I have a backup script that runs everyday at 03.00. It also enables maintenance mode and disables it again afterwards when the backup is done (through the occ command). This is handled with systemd.
Strangely, my instance is in maintanence mode already at about 23ish everyday. I don’t know why and have no clue how to debug this.
I couldn’t find a log output of the occ.
Any hints on where to start?

The output of your Nextcloud log in Admin > Logging:
Not in log

Edit:
Content of systemd timer:

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

first thing that comes to my mind is: have you checked locale-settings/timezone of your server?

good catch, but it seems to be right (set to “de_CH.UTF-8”).
running date also outputs the time of my normal clock

1 Like

Hi,

With
systemctl list-timers
you can output all active timers and see the timezone for each timer. If your timer shows a wrong timezone there, you could define it in your timer config simply by adding it at the end of the line like:

[Timer]
OnCalendar=*-*-* 03:00:00 CEST
Persistent=true

or

[Timer]
OnCalendar=*-*-* 03:00:00 Europe/Zurich
Persistent=true

Thanks for the input!
systemctl list-timersoutput is:

NEXT                          LEFT          LAST                          PASSED
Fre 2019-09-27 03:00:00 CEST  15h left      Don 2019-09-26 03:00:00 CEST  8h ago

Seems to be what I intended…

There are also no other timers that could interfere with this.

Indeed, looking good. Also the “15h left” indicates the timer really wants to start at 3 am first.
Might there be another job setting NC in maintenance?
Are there any cronjobs for

  • www-data
  • root
  • any other user you are using to maintain NC

You can check for example with:
crontab -l -u www-data

I have already checked that.
no crontab for www-data and those of root and “main”-user are just empty…

also ran for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done to output all crontabs: either empty or non-existent

Can you post the complete configuration of your systemd job including the commands for the backup which are executed? Maybe we can put some echo commands in there and output the time to a log file somewhere.

Edit:
One additional question. Did you change the time at some point? Meaning, did you set the backup to be at 23:00 before and changed the time to be 3:00 in the morning later?

RIght.

So I use borg-backup and borgmatic as wrapper. Config is pretty standard.

borgmatic.service:

[Unit]
Description=borgmatic backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/borgmatic

borgmatic.timer: (as above)

[Unit]
Description=Run borgmatic backup

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

Log of service:

Sep 26 03:00:00 this-server systemd[1]: Starting borgmatic backup...
Sep 26 03:00:18 this-server sudo[8556]:     root : TTY=unknown ; PWD=/var/www/cloud ; USER=www-data ; COMMAND=/usr/bin/php /var/www/cloud/occ maintenance:mode --on
Sep 26 03:00:18 this-server sudo[8556]: pam_unix(sudo:session): session opened for user www-data by (uid=0)
Sep 26 03:00:18 this-server borgmatic[7570]: Maintenance mode already enabled
Sep 26 03:00:18 this-server sudo[8556]: pam_unix(sudo:session): session closed for user www-data
Sep 26 03:29:57 this-server sudo[17397]:     root : TTY=unknown ; PWD=/ ; USER=www-data ; COMMAND=/usr/bin/php /var/www/cloud/occ maintenance:mode --off
Sep 26 03:29:57 this-server sudo[17397]: pam_unix(sudo:session): session opened for user www-data by (uid=0)
Sep 26 03:29:57 this-server borgmatic[7570]: Maintenance mode disabled
Sep 26 03:29:57 this-server sudo[17397]: pam_unix(sudo:session): session closed for user www-data
Sep 26 03:36:45 this-server systemd[1]: Started borgmatic backup.

Line 4 is as expected (always like this).

I have set this up quite a while ago, so I don’t remember exactly when I run it first. But maybe i tried to run it once on 23.xx, this is possible. But that shouldn’t make any difference, right?

I have also added a line to echo a timestamp, the user that runs it and what the invoking command was to a file everytime it runs (the actual backup).

Okay, so there was quite a misunderstanding on my side. I was thinking that the backup starts at the wrong time and therefore NC is in maintenance earlier than expected.
In that case no, it doesn’t matter if you configured a different time earlier.

So we need to find out which is setting NC in maintenance.
Could you search your journal for “occ maintenance:mode --on”?
If your last reboot was performed some days ago already, use
journalctl -xe

and then search for the mentioned string to get an idea what is activating the maintenance mode.

only thing that shows up is

Sep 26 08:49:49 this-server sudo[27027]:     root : TTY=pts/8 ; PWD=/var/www/cloud ; USER=www-data ; COMMAND=/usr/bin/php occ log:file

where I tried to find relevant logs

I’ll dig into the rotated logs to see if i can pull something

Found something? Maybe increasing the log level to “Information” or “Debug” could help.

What I’ve been able to get is that it’s also running everyday at 22.46. But I was not yet able to pinpoint a source of that.