AIO can't access root data on external drive after reboot

ā„¹ļø Support šŸ“¦ Appliances (Docker, Snappy, VM, NCP, AIO)
, , ,
1

Hi,

Iā€™m using the nextcloud AIO solution together with an external HDD data directory.

After the working system rebooted (electricity was cut), the system somehow does not properly mount the external data dir to the docker nextcloud-aio-nexctloud.

I checked that:

  • the HDD is correctly mounted and accesible on my linux server
  • the docker compose was restarted after the HDD was mounted

As this is a reboot issue Iā€™m not to sure what could have happened.
To my understanding the ā€œNextcloud AIO v7.0.0ā€ control page seems fine, and does not give any warnings/errors.
Moreover, it seems to correctly display my env vars from the compose.yml file:

[ā€¦ ] Nextcloudā€™s datadir is getting stored in the /mnt/data/ncdata directory [ā€¦]
[ā€¦] The Nextcloud container is getting gets access to the /mnt/data directory and local external storage in Nextcloud is enabled [ā€¦]

The nextcloud php config reads the correct paths as well

[ā€¦] ā€˜datadirectoryā€™ => ā€˜/mnt/ncdataā€™, [ā€¦]

Looking into the nextcloud-aio-nextcloud logs and fs, it seems to be loading a wrong/errorneous path ā€¦ the logs read

  • ā€˜[ā€™ -f /dev-dri-group-was-added ā€˜]ā€™
    ++ find /dev -maxdepth 1 -mindepth 1 -name dri
  • ā€˜[ā€™ -n ā€˜ā€™ ā€˜]ā€™
  • set +x
    Installing imagemagick via apkā€¦
    Enabling Imagickā€¦
    Configuring Redis as session handlerā€¦
    Setting php max childrenā€¦
    Initializing nextcloud 27.0.2.1 ā€¦
    Initializing finished
    New Nextcloud instance.
    Installing with PostgreSQL database
    Starting Nextcloud installationā€¦
    Nextcloud was successfully installed
  • Repair MySQL collation
    • Not a mysql database ā†’ nothing to do
  • Repair mime types
  • Clean tags and favorites
    • 0 tags of deleted users have been removed.
    • 0 tags for delete files have been removed.
    • 0 tag entries for deleted tags have been removed.
    • 0 tags with no entries have been removed.
  • Repair invalid shares
  • Move .step file of updater to backup location
  • Add move avatar background job
    • Add background job
  • Add preview cleanup background jobs
  • Migrate oauth2_clients table to nextcloud schema
    • Update the oauth2_access_tokens table schema.
    • Update the oauth2_clients table schema.
    • Delete clients (and their related access tokens) with the redirect_uri starting with oc:// or ending with *
  • Fix potential broken mount points
    • No mounts updated
  • Repair language codes
  • Add log rotate job
  • Clear frontend caches
    • Image cache cleared
    • JS cache cleared
  • Clear every generated avatar
  • Add preview background cleanup job
  • Queue a one-time job to cleanup old backups of the updater
  • Cleanup invalid photocache files for carddav
  • Add background job to cleanup login flow v2 tokens
  • Remove potentially over exposing share links
    • No need to remove link shares.
  • Clear access cache of projects
  • Reset generated avatar flag
  • Keep legacy encryption enabled
  • Check encryption key format
  • Remove old dashboard app config data
  • Add job to cleanup the bruteforce entries
  • Queue a one-time job to check for user uploaded certificates
  • Repair DAV shares
  • Add background job to set the lookup server share state for users
  • Add token cleanup job
  • Clean up abandoned apps
  • Add possibly missing system config
  • Upgrading Circles App
  • Fix component of birthday calendars
    • 0 birthday calendars updated.
  • Regenerating birthday calendars to use new icons and fix old birthday events without year
    • Adding background jobs to regenerate birthday calendar
  • Fix broken values of calendar objects
    0 [>---------------------------]
  • Registering building of calendar search index as background job
    • Add background job
  • Register building of social profile search index as background job
  • Registering background jobs to update cache for webcal calendars
    • Added 0 background jobs to update webcal calendars
  • Registering building of calendar reminder index as background job
    • Add background job
  • Clean up orphan event and contact data
    • 0 events without a calendar have been cleaned up
    • 0 properties without an events have been cleaned up
    • 0 changes without a calendar have been cleaned up
    • 0 cached events without a calendar subscription have been cleaned up
    • 0 changes without a calendar subscription have been cleaned up
    • 0 contacts without an addressbook have been cleaned up
    • 0 properties without a contact have been cleaned up
    • 0 changes without an addressbook have been cleaned up
  • Remove activity entries of private events
    • Removed 0 activity entries
  • Clean up old calendar subscriptions from deleted users that were not cleaned-up
    0 [->--------------------------]
    • 0 calendar subscriptions without an user have been cleaned up
  • Remove invalid object properties
    • 0 invalid object properties removed.
  • Fix the share type of guest shares when migrating from ownCloud
  • Copy the share password into the dedicated column
  • Set existing shares as accepted
  • Update OAuth token expiration times
  • Switches from default updater server to the customer one if a valid subscription is available
  • Send an admin notification if monthly report is disabled
  • Force-reset all Text sessions before Yjs migration
  • Initialize migration of background images from dashboard to theming app
  • Add background job to check for backup codes
  • Populating added database structures for workflows
    Applying default settingsā€¦
    System config value loglevel set to string 2
    System config value log_type set to string file
    System config value logfile set to string /var/www/html/data/nextcloud.log
    System config value log_rotate_size set to string 10485760
    admin_audit 1.17.0 enabled
    Config value logfile for app admin_audit set to /var/www/html/data/audit.log
    System config value log.condition => apps => 0 set to string admin_audit
    Applying preview settingsā€¦
    System config value preview_max_x set to string 2048
    System config value preview_max_y set to string 2048
    System config value jpeg_quality set to string 60
    Config value jpeg_quality for app preview set to 60
    System config value enabledPreviewProviders deleted
    System config value enabledPreviewProviders => 1 set to string OC\Preview\Image
    System config value enabledPreviewProviders => 2 set to string OC\Preview\MarkDown
    System config value enabledPreviewProviders => 3 set to string OC\Preview\MP3
    System config value enabledPreviewProviders => 4 set to string OC\Preview\TXT
    System config value enabledPreviewProviders => 5 set to string OC\Preview\OpenDocument
    System config value enabledPreviewProviders => 6 set to string OC\Preview\Movie
    System config value enabledPreviewProviders => 7 set to string OC\Preview\Krita
    System config value enable_previews set to boolean true
    Applying other settingsā€¦
    System config value upgrade.disable-web set to boolean true
    System config value mail_smtpmode set to string smtp
    System config value trashbin_retention_obligation set to string auto, 30
    System config value versions_retention_obligation set to string auto, 30
    System config value activity_expire_days set to string 30
    System config value simpleSignUpLink.shown set to boolean false
    System config value share_folder set to string /Shared
    calendar 4.4.4 installed
    calendar enabled
    Error: Could not download app contact
    Applying one-click-instance settingsā€¦
    System config value one-click-instance set to boolean true
    System config value one-click-instance.user-limit set to integer 100
    System config value one-click-instance.link set to string [redacted link]
    support already enabled
    Adjusting log filesā€¦
    System config value upgrade.cli-upgrade-link set to string [redacted link]
    System config value logfile set to string /var/www/html/data/nextcloud.log
    Config value logfile for app admin_audit set to /var/www/html/data/audit.log
    System config value updatedirectory set to string /nc-updater
    Applying network settingsā€¦
    System config value davstorage.request_timeout set to integer 3600
    System config value trusted_domains => 1 set to string my.domain.com
    System config value overwrite.cli.url set to string [redacted link]
    System config value htaccess.RewriteBase set to string /
    .htaccess has been updated
    System config value dbpersistent set to boolean true
    System config value files_external_allow_create_new_local set to boolean true
    nextcloud-aio 0.4.0 enabled
    notify_push 0.6.3 installed
    notify_push enabled
    System config value trusted_proxies => 0 set to string 127.0.0.1
    System config value trusted_proxies => 1 set to string ::1
    Config value base_endpoint for app notify_push set to https[redacted link]/push
    System config value enabledPreviewProviders => 0 set to string OC\Preview\Imaginary
    System config value preview_imaginary_url set to string http[redacted link]9000
    [22-Aug-2023 21:58:27] NOTICE: fpm is running, pid 583
    [22-Aug-2023 21:58:27] NOTICE: ready to handle connections
    now

2023-12-28 19:28:30.971875+00
(1 row)

  • ā€˜[ā€™ -f /dev-dri-group-was-added ā€˜]ā€™
    ++ find /dev -maxdepth 1 -mindepth 1 -name dri
  • ā€˜[ā€™ -n ā€˜ā€™ ā€˜]ā€™
  • set +x
    Configuring Redis as session handlerā€¦
    Setting php max childrenā€¦
    Applying one-click-instance settingsā€¦
    System config value one-click-instance set to boolean true
    System config value one-click-instance.user-limit set to integer 100
    System config value one-click-instance.link set to string https[redacted link]/all-in-one/
    support already enabled
    Adjusting log filesā€¦
    System config value upgrade.cli-upgrade-link set to string https[redacted link]/all-in-one/discussions/2726
    System config value logfile set to string /var/www/html/data/nextcloud.log
    Config value logfile for app admin_audit set to /var/www/html/data/audit.log
    System config value updatedirectory set to string /nc-updater
    Applying network settingsā€¦
    System config value davstorage.request_timeout set to integer 3600
    System config value trusted_domains => 1 set to string [redacted link]com
    System config value overwrite.cli.url set to string https[redacted link]com
    System config value htaccess.RewriteBase set to string /
    .htaccess has been updated
    System config value dbpersistent set to boolean true
    System config value files_external_allow_create_new_local set to boolean true
    notify_push new version available: 0.6.6
    notify_push updated
    System config value trusted_proxies => 0 set to string 127.0.0.1
    System config value trusted_proxies => 1 set to string ::1
    Config value base_endpoint for app notify_push set to https[redacted link]/push
    System config value enabledPreviewProviders => 0 set to string OC\Preview\Imaginary
    System config value preview_imaginary_url set to string http[redacted link]nextcloud-aio-imaginary:9000
    [28-Dec-2023 19:29:30] NOTICE: fpm is running, pid 231
    [28-Dec-2023 19:29:30] NOTICE: ready to handle connections
    Your data directory is invalid.
    Ensure there is a file called ā€œ.ocdataā€ in the root of the data directory.

Your data directory is invalid.
Ensure there is a file called ā€œ.ocdataā€ in the root of the data directory.

Your data directory is invalid.
Ensure there is a file called ā€œ.ocdataā€ in the root of the data directory.

Persmissions on the external data dir:

[root@w3 ncAIO]# ls -al /mnt/data/ncdata/
total 28
drwxr-xā€”. 6 33 root 4096 Dec 28 20:28 .
drwxrwxrwx. 15 root root 4096 Aug 22 21:44 ā€¦
drwxr-xr-x. 4 33 33 4096 Aug 23 00:10 admin
drwxr-xr-x. 10 33 33 4096 Aug 23 01:15 appdata_ocfkwmy8tmff
-rw-rā€“rā€“. 1 33 33 0 Aug 22 23:56 audit.log
drwxr-xr-x. 2 33 33 4096 Aug 22 23:57 files_external
-rw-rā€“rā€“. 1 33 33 542 Aug 22 23:56 .htaccess
-rw-rā€“rā€“. 1 33 33 0 Aug 22 23:56 index.html
drwxr-xr-x. 7 33 33 4096 Aug 23 10:10 marc
-rw-rā€“rā€“. 1 33 33 0 Aug 22 23:56 nextcloud.log
-rw-rā€“rā€“. 1 33 33 0 Aug 22 23:56 .ocdata

Help is very much welcome.
Cheers,
Marc

2

Just for completeness sake, here is the docker setup information:

services:
nextcloud-aio-mastercontainer:
image: nextcloud/all-in-one:latest
init: true
restart: always
container_name: nextcloud-aio-mastercontainer
volumes:
- nextcloud_aio_mastercontainer:/mnt/docker-aio-config
- /var/run/docker.sock:/var/run/docker.sock:ro
- 880:8080

environment:
  - APACHE_PORT=11000
  - APACHE_IP_BINDING=0.0.0.0
  - NEXTCLOUD_DATADIR=/mnt/data/ncdata
  - NEXTCLOUD_MOUNT=/mnt/data
  - NEXTCLOUD_STARTUP_APPS=calendar contact

caddy:
image: caddy:alpine
restart: always
container_name: caddy
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./certs:/certs
- ./config:/config
- ./data:/data
- ./sites:/srv
network_mode: ā€œhostā€

volumes:
nextcloud_aio_mastercontainer:
name: nextcloud_aio_mastercontainer

3

Well this is akward ā€¦

After stopping all containers via the AIO control interface, which I did for this post and the logs, and manually restarting them in the terminal, it seems to work now ā€¦

In a terminal I ran:

docker start nextcloud-aio-apache
docker start nextcloud-aio-notify-push
docker start nextcloud-aio-imaginary
docker start nextcloud-aio-nextcloud
docker start nextcloud-aio-redis
docker start nextcloud-aio-database

Seems to be me that some configuration state was lost during some kind of default/cached docker startup.

(here a picture of the AIO control page with the ā€œstop buttonā€)

image
image625Ɨ625 22.2 KB

1 Like
4

I didnā€™t look at your logs in-depth, but youā€™re running an out of date AIO (and, in turn, NC Server) versions.

Perhaps you encountered an already fixed bug?[1]

[1] Releases Ā· nextcloud/all-in-one Ā· GitHub

5

I doubt this is caused by AIO. I rather think the drive was not mounted before the containers were started. Is the drive mounted via fstab?

2 Likes
6

Yes, Iā€™m using fstab.

And yes again, after checking the boot up order (systemd-analyze plot), I think you are correct, in your assumption, that AIO is fine. Somehow the systemd mount command is missing, was not executed before the docker container loaded.

While other fstab entries loaded correctly and well before docker, in this case the HDD in question, for some reason, did not load at time.

Thanks

PS.: would mind though, if the ā€œaio-nextcloudā€ container would have automatically restarted ; )

1 Like