[NCP] nc-scan / external files gone

After trying to upgrade from my NCP Berryboot on Raspberry Pi 2 to 4 I’m now completely stuck. Let’s see if someone can help me out here…

I first tried to us ncp-dist-upgrade on my Pi2 to make NCP compatible with my Pi4. That failed (
E: Sub-process /usr/bin/dpkg returned an error code (1), also mentioned here in a similar way as problem 3: NextcloudPi - Backup restore or Buster, I can't have both) .

Then I tried to restore the previously made backup using ncp-restore. That didn’t work either (Error: Refusing to restore from /var/www/nextcloud, also mentioned here: "can only restore from ext/btrfs filesystems").

Then I set up a new NCP instance on the Pi4 (using berryboot and the WD pidrive I already used with the Pi2). That worked and is now up and running. When I try to restore the backup I still got the same problem as mentioned above.

So I decided to set up everything manually, created a second user with the same name and password as the one I used on the Pi2 and moved 200GB of data in Windows from the previous NCP-instance to the data-folder of the new user (from F:\data\NextcloudPi.img\var\www\nextcloud\data\admin\files to F:\data\NextcloudPi-2019-12-20.img\var\www\nextcloud\data\admin\files).
I then connected the drive to the Raspberry 4 and booted into NCP. I ran ncp-scan, but no files where found. I then looked manually for the files but apparently they where not visible anymore! Instead the standard files which I deleted via Windows appeared again (Photos, Documents etc)! I verified this when I connected the drive to my PC again. All files invisible (but still 200GB of space used on the drive!)

Any idea how I now can restore the “invisible” files or delete them? Or do I have to format the whole drive and start all over again?
Any help appreciated, I am close to giving up…

NCP does not support NTFS or Dos file system. Only ext4 and btrfs.

Also you can not copy or move files from and to any ****.img files the way you are.

If need further assistance please post link (pastebin or similar) to output of ncp-report.

Hey Oliver, thanks for your quick reply!
To clarify this - I didn’t change the file system at all, the drive is formatted via Linux and NCP was working normally on it for months. I don’t think that this is a new problem, but rather think it might be associated with the use of Berryboot in some way.

Also when I was referring to NextcloudPi.img I was talking about a normal folder named like that by Berryboot and not a file. The image files are stored in a separate folder on Berryboot.

Here is the ncp-report you requested:

NextCloudPi diagnostics

NextCloudPi version  v1.20.1
NextCloudPi image    NextCloudPi_12-20-19
distribution         Raspbian GNU/Linux 10 \n \l
automount            no
USB devices          sda
datadir              /var/www/nextcloud/data
data in SD           yes
data filesystem      overlayfs
data disk usage      193G/344G
rootfs usage         193G/344G
swapfile             none
dbdir                /var/lib/mysql
Nextcloud check      ok
Nextcloud version    17.0.2.1
HTTPD service        up
PHP service          up
MariaDB service      up
Redis service        up
Postfix service      up
internet check       ok
port check 80        open
port check 443       open
IP                   ***REMOVED SENSITIVE VALUE***
gateway              ***REMOVED SENSITIVE VALUE***
interface            eth0
certificates         ***REMOVED SENSITIVE VALUE***
NAT loopback         no
uptime               11:04

Nextcloud configuration

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "0": "localhost",
            "5": "nextcloudpi.local",
            "7": "nextcloudpi",
            "8": "nextcloudpi.lan",
            "11": "2003:c9:f701:2e00:d36e:2f36:ff98:7962",
            "1": "192.168.178.45",
            "12": "***REMOVED SENSITIVE VALUE***",
            "3": "***REMOVED SENSITIVE VALUE***"
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "17.0.2.1",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "tempdirectory": "\/var\/www\/nextcloud\/data\/tmp",
        "mail_smtpmode": "sendmail",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "overwriteprotocol": "https",
        "loglevel": "2",
        "log_type": "file"
    }
}

HTTPd logs

[Mon Jan 27 09:09:43.779564 2020] [ssl:warn] [pid 787:tid 4153220832] AH01882: Init: this version of mod_ssl was compiled against a newer library (OpenSSL 1.1.1d  10 Sep 2019, version currently loaded is OpenSSL 1.1.1c  28 May 2019) - may result in undefined or erroneous behavior
[Mon Jan 27 09:09:43.867441 2020] [ssl:warn] [pid 787:tid 4153220832] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Mon Jan 27 09:09:43.902674 2020] [ssl:warn] [pid 1033:tid 4153220832] AH01882: Init: this version of mod_ssl was compiled against a newer library (OpenSSL 1.1.1d  10 Sep 2019, version currently loaded is OpenSSL 1.1.1c  28 May 2019) - may result in undefined or erroneous behavior
[Mon Jan 27 09:09:43.906556 2020] [ssl:warn] [pid 1033:tid 4153220832] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Mon Jan 27 09:09:43.910368 2020] [mpm_event:notice] [pid 1033:tid 4153220832] AH00489: Apache/2.4.38 (Raspbian) OpenSSL/1.1.1c configured -- resuming normal operations
[Mon Jan 27 09:09:43.910518 2020] [core:notice] [pid 1033:tid 4153220832] AH00094: Command line: '/usr/sbin/apache2'
[Mon Jan 27 20:12:36.321671 2020] [proxy_fcgi:error] [pid 1036:tid 3897955360] [client 192.168.178.20:51180] AH01071: Got error 'PHP message: PHP Notice:  Undefined index: app in /var/www/ncp-web/index.php on line 238PHP message: PHP Notice:  Undefined index: app in /var/www/ncp-web/index.php on line 244'

Database logs

2020-01-27  9:09:59 0 [Note] InnoDB: Uses event mutexes
2020-01-27  9:09:59 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-01-27  9:09:59 0 [Note] InnoDB: Number of pools: 1
2020-01-27  9:09:59 0 [Note] InnoDB: Using generic crc32 instructions
2020-01-27  9:09:59 0 [Note] InnoDB: Initializing buffer pool, total size = 1.5G, instances = 1, chunk size = 128M
2020-01-27  9:09:59 0 [Note] InnoDB: Completed initialization of buffer pool
2020-01-27  9:09:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-01-27  9:10:00 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-01-27  9:10:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-01-27  9:10:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-01-27  9:10:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-01-27  9:10:00 0 [Note] InnoDB: 10.3.17 started; log sequence number 8455409; transaction id 8493
2020-01-27  9:10:00 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-01-27  9:10:00 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-01-27  9:10:00 0 [Note] Server socket created on IP: '127.0.0.1'.
2020-01-27  9:10:01 0 [Note] Reading of all Master_info entries succeeded
2020-01-27  9:10:01 0 [Note] Added new Master_info '' to hash table
2020-01-27  9:10:01 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.3.17-MariaDB-0+deb10u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Raspbian 10
2020-01-27  9:10:03 0 [Note] InnoDB: Buffer pool(s) load completed at 200127  9:10:03

Nextcloud logs

{"reqId":"Xi85lRMkg3jBhwUYBMxGSQAAAFE","level":2,"time":"2020-01-27T19:27:43+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Laptop\/Unterwegs\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi85sBMkg3jBhwUYBMxGSgAAAE4","level":2,"time":"2020-01-27T19:28:10+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Laptop\/Unterwegs\/Wir%20heiraten!%20J%2BJ%20-%20small.jpg","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi85yhMkg3jBhwUYBMxGSwAAAEU","level":2,"time":"2020-01-27T19:28:37+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Laptop\/Unterwegs\/Klimagruppe\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi855RMkg3jBhwUYBMxGTAAAAEg","level":2,"time":"2020-01-27T19:29:04+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Laptop\/Unterwegs\/Klimagruppe\/Name.xlsx","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi86ABMkg3jBhwUYBMxGTQAAAFY","level":2,"time":"2020-01-27T19:29:31+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Doktorarbeit\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi86GxMkg3jBhwUYBMxGTgAAAFQ","level":2,"time":"2020-01-27T19:29:58+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Doktorarbeit\/Grafiken\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi86Nj1ICTr0vbWNtcxolgAAABU","level":2,"time":"2020-01-27T19:30:25+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Doktorarbeit\/Grafiken\/Abb8-Kaplan-Meier.pdf","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi88IxMkg3jBhwUYBMxGUQAAAE8","level":2,"time":"2020-01-27T19:38:38+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PUT","url":"\/remote.php\/webdav\/SofortUpload\/Threema%20Pictures\/threema-20200127-180548-5ab84600023bfb26.jpg","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi88QxMkg3jBhwUYBMxGUwAAAEo","level":2,"time":"2020-01-27T19:39:10+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"MKCOL","url":"\/remote.php\/dav\/uploads\/admin\/e66f408959cdcd02502d9e3fd04a66ee","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi88Xj1ICTr0vbWNtcxomAAAAAg","level":2,"time":"2020-01-27T19:39:37+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/dav\/uploads\/admin\/e66f408959cdcd02502d9e3fd04a66ee","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi882xMkg3jBhwUYBMxGVQAAAFU","level":2,"time":"2020-01-27T19:41:42+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi89phMkg3jBhwUYBMxGVgAAAFE","level":2,"time":"2020-01-27T19:45:05+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Notes\/Vortr%C3%A4ge\/Neue%20Impulse%20f%C3%BCr%20die%20Eurozone.txt","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi9DXRMkg3jBhwUYBMxGXAAAAEA","level":2,"time":"2020-01-27T20:09:28+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi9DXT1ICTr0vbWNtcxomQAAABc","level":2,"time":"2020-01-27T20:09:53+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi9DeBMkg3jBhwUYBMxGXgAAAFM","level":2,"time":"2020-01-27T20:09:55+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Laptop\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi9DXT1ICTr0vbWNtcxomgAAAAw","level":2,"time":"2020-01-27T20:10:18+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PUT","url":"\/remote.php\/webdav\/SofortUpload\/Threema%20Pictures\/threema-20200127-180548-5ab84600023bfb26.jpg","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi9DuhMkg3jBhwUYBMxGYAAAAE0","level":2,"time":"2020-01-27T20:11:01+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi9Duz1ICTr0vbWNtcxomwAAAAE","level":2,"time":"2020-01-27T20:11:26+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Laptop\/Handy\/apks\/whyred.xml","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi9D4xMkg3jBhwUYBMxGYQAAAEo","level":2,"time":"2020-01-27T20:11:42+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Laptop\/Unterwegs\/","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}
{"reqId":"Xi9D-hMkg3jBhwUYBMxGYgAAAFA","level":2,"time":"2020-01-27T20:12:09+00:00","remoteAddr":"93.198.62.115","user":"--","app":"core","method":"PROPFIND","url":"\/remote.php\/webdav\/Laptop\/Unterwegs\/Wir%20heiraten!%20J%2BJ%20-%20small.jpg","message":"Login failed: 'admin' (Remote IP: '93.198.62.115')","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.9.2","version":"17.0.2.1"}

Mainly I was wondering if there is a way to restore the missing files I moved to the data folder.
Or a way to free up the space that is used by these files, so I can copy the files to the drive again (but in which folder would it be safe to store the files so that they can be detected by ncp-scan)?

I don’t know, but maybe the following info also helps:

pi@nextcloudpi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
dev             1.7G     0  1.7G   0% /dev
none            344G  193G  134G  59% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  9.5M  1.9G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1  942M   99M  843M  11% /boot
log2ram          40M  2.6M   38M   7% /var/log
/dev/loop0      562M  562M     0 100% /media/disk
tmpfs           373M     0  373M   0% /run/user/1000
pi@nextcloudpi:~ $ lsblk -f
NAME FSTYPE LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
loop0
     squash                                                  0   100% /media/dis
sda
├─sda1
│
└─sda2
     ext4   berryboot
                  47fa232c-8709-4755-bbb9-c591cba472cf
mmcblk0
│
└─mmcblk0p1
     vfat   BERRYBOOT
                  E27D-E0C6                             842.9M    10% /boot
pi@nextcloudpi:~ $ ls -l /media
total 1048584
drwxr-xr-x 21 root   root           298 Dec 20 23:34 disk
drwxrwxrwx  4 nobody nogroup       4096 Jan 26 20:35 nc-backup
-rw-------  1 root   root    1073741824 Jan 26 20:35 swap
lrwxrwxrwx  1 root   root            11 Jan 27 09:09 USBdrive -> /media/disk
lrwxrwxrwx  1 root   root            16 Jan 27 09:09 USBdrive1 -> /media/nc-backup

Not sure. but berryboot uses squashfs, which might be the problem, if your using the normal ncp image.
There is berryboot for rpi4 and a ncp image for it, but I have not tested it, as I don’t have a rpi4.

Maybe someone who is using a rpi4 can help you further, if not, I’ d start from scratch and do a fresh install.

Ok. But still the question remains: in which folder would it be safe to store the files so that they can be detected by ncp-scan (and are not deleted after reboot)?

@nachoparker, can you help to clarify this?

On the new install files and folders I copied manually into the user folder were recognized correctly by nc-scan.
The only thing now is, that I am not allow to change files or upload new files into these folders because apparently I don’t have the permissions to do so (even as admin account):

You don’t have permission to upload or create files here.

I started nc-fix-permissions what finished without errors within seconds, but I still can not edit or upload to these folders (rebooted in between). Any idea how to fix this?

I think; nc-fix-permissions does not touch external files.

So you’ll need to change ownership to www-data manually.

If it is a fresh install, a new ncp-report could be useful. Best way to restore would to use nc-restore

I finally managed to achieve the permission to change files and folders. I executed nc-scan again AFTER nc-fix-permissions and apparently that worked!
So - back to business now :wink: