My NextcloudPi server completely freaked out tonight, the client on my laptop suddenly started syncing large changes out of the blue and before I knew it, 90% of my files were deleted. I used to have about 25GB of data synced, now the server says only about 480MB remains. Fortunately, I disabled syncing on my desktop when I realized what was happening, so I still have copies of most of the files. Also, the Pi has BTRFS snapshots, so I can see all files are there in the last snapshot before the thing went nuts.
So my question is - how do I properly debug this issue to find out if it’s a software failure, hardware failure or a hack?
The logs are always where investigations start.
I’d start with:
auth.log (for hack)
dmesg (for hardware, or use nc-hdd-test)
ncp-report (general system info and health)
For long logs I use pastebin, edit to remove own IP, post link, if need assistance.
1 Like
Thanks for your reply, OliverV. I’ve shut the system down to prevent escalation if it was indeed hacked. I’ll connect the drive to my workstation and copy all files for further analysis.
Since ncp-report and nc-hdd-test (probably?) only work on a live Nextcloud environment, are there any other logfiles that could be relevant?
I was planning on migrating Nextcloud to a more powerful machine in the near future, so I’m thinking of skipping spending time on the Pi and focusing on the new machine instead.
On the other hand, if what I ran into was a software bug, I feel obliged to do my part and report it.
I appreciate any assistance! 
Since you have been hacked, I’d focus on a fresh install and making it secure.
You most likely did not have the NCP’s most important security features enabled.
ssh and nc should have extra protection, ufw and fail2ban, enabled!?
Mine are heavily under attack all the time, as is visible from auth.logs . Anyone opening a port to public web should be aware of that and take measures to protect from unauthorized access.
For ssh I use RSA certificates to authenticate, and disable password logins completely, or at least disallow for root. Nextcloud has its own safety measures build in, but having fail2ban as an extra measure cant hurt.
On mine, It is mostly ssh that is targeted, so changing external port from default 22 to a higher unused port, deflects some login attempts. Port scanners, may allow hackers to find your ssh port in the end. You can also use fail2ban or ufw to simply block anyone except yourself and your family and friends.
Well that’s the point - I don’t know IF the system was hacked or not. Logfile auth.log shows just one failed login attempt, which was my own when I entered the wrong ssh password.
The system is not accessible by SSH from the web, only from LAN.
The passwords are all long and randomly generated by my password manager, so brute force seems unlikely.
Fail2ban is active on the system (fail2ban.log shows just one external IP being logged) and the system automatically updates as per the default settings of NextcloudPi.
Granted - ufw was not active, but only ports 80 and 443 are forwarded from the router to the system, nothing else.
With all above, I suspect software or hardware failure is more likely than an actual hack.
So the question remains - which (offline) log files could shed more light on what happened?
ls -lhtr /var/log/
will display list of all system logs in time reversed order. (Last one modified, last) Note they get rotated (saved as tar.gz and new one created) automatically. So you can/should check older ones too.
Also check /var/www/nextcloud/data/nextcloud.log or wherever your ncdata directory is.
As said use dmesg
sudo dmesg
to check for hardware issues, or use nc-hdd-test
There is now way, afaik, for NCP to start deleting files spontaneously. Please provide actual logs.