Server died! - how to clean files_trashbin remotely?

My Nextcloud instance is not accessible because my server is frozen due to, I’m led to believe, my root partition becoming completely full (100%!).

I’m not anywhere near my server but I can SSH into it. Running the following:

du -a /var | sort -nr | head -n 10

shows that the Nextcloud trashbin at /var/www/html/nextcloud/data/my_name/files_trashbin is quite full (4324952).

I think that I should be able to give my server a bit of breathing room by cleaning the contents of this trashbin. However, I’m not sure how I can do this remotely (nor even locally!). Could somebody please provide some guidance?

I’m running a Nextcloud 13.something-or-other server on Fedora 28. I’m sorry but, I don’t have specific details because I access my Nextcloud instance web page

Thanks in advance for any help provided :slight_smile:

@Hevii_Guy Assuming you are the only user on this instance, and can clear all trash without worrying about a loss of data of users that deleted files by “accident” and it ending up in trash, you could issue the following:
sudo -u apache php occ trashbin:cleanup --all-users from /var/www/html/nextcloud
You can also use that command and only clear one user’s data by specifying the user like this:
sudo -u apache php occ trashbin:cleanup bob

Remember, if a user’s files are in here by accident you will now remove them permanently and IRREVOCABLY so be ABSOLUTELY sure about this before you do this.

Thanks, Hans. Alas, the command didn’t work. This was the result:

root@heviiMTLServer heviiguy# cd /var/www/html/nextcloud
root@heviiMTLServer nextcloud# sudo -u apache php occ trashbin:cleanup --all-use
rs
PHP Warning:  PHP Startup: Unable to load dynamic library 'xmlrpc.so' (tried: /u
sr/lib64/php/modules/xmlrpc.so (/usr/lib64/php/modules/xmlrpc.so: cannot open sh
ared object file: No such file or directory), /usr/lib64/php/modules/xmlrpc.so.s
o (/usr/lib64/php/modules/xmlrpc.so.so: cannot open shared object file: No such 
file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'igbinary.so' (tried: 
/usr/lib64/php/modules/igbinary.so (/usr/lib64/php/modules/igbinary.so: cannot o
pen shared object file: No such file or directory), /usr/lib64/php/modules/igbin
ary.so.so (/usr/lib64/php/modules/igbinary.so.so: cannot open shared object file
: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'imagick.so' (tried: /
usr/lib64/php/modules/imagick.so (/usr/lib64/php/modules/imagick.so: cannot open
 shared object file: No such file or directory), /usr/lib64/php/modules/imagick.
so.so (/usr/lib64/php/modules/imagick.so.so: cannot open shared object file: No 
such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'radius.so' (tried: /u
sr/lib64/php/modules/radius.so (/usr/lib64/php/modules/radius.so: cannot open sh
ared object file: No such file or directory), /usr/lib64/php/modules/radius.so.s
o (/usr/lib64/php/modules/radius.so.so: cannot open shared object file: No such 
file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'redis.so' (tried: /us
r/lib64/php/modules/redis.so (/usr/lib64/php/modules/redis.so: cannot open share
d object file: No such file or directory), /usr/lib64/php/modules/redis.so.so (/
usr/lib64/php/modules/redis.so.so: cannot open shared object file: No such file 
or directory)) in Unknown on line 0
An unhandled exception has been thrown:
Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occ
urred while executing 'PRAGMA journal_mode = WAL':

SQLSTATE[HY000]: General error: 10 disk I/O error in /var/www/html/nextcloud/lib
/private/DB/Connection.php:64
Stack trace:
#0 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.p
hp(992): OC\DB\Connection->connect()
#1 /var/www/html/nextcloud/lib/private/DB/Connection.php(216): Doctrine\DBAL\Con
nection->executeUpdate('PRAGMA read_unc...', Array, Array)
#2 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.p
hp(623): OC\DB\Connection->executeUpdate('PRAGMA read_unc...')
#3 /var/www/html/nextcloud/lib/private/DB/Connection.php(151): Doctrine\DBAL\Con
nection->setTransactionIsolation(2)
#4 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManage
r.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDO
Sqlite\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\Even
tManager))
#5 /var/www/html/nextcloud/lib/private/DB/ConnectionFactory.php(152): Doctrine\D
BAL\DriverManager::getConnection(Array, Object(Doctrine\DBAL\Configuration), Obj
ect(Doctrine\Common\EventManager))
#6 /var/www/html/nextcloud/lib/private/Server.php(628): OC\DB\ConnectionFactory-
>getConnection('sqlite3', Array)
#7 /var/www/html/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113):
 OC\Server->OC\{closure}(Object(OC\Server))
#8 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(
116): Pimple\Container->offsetGet('OCP\\IDBConnecti...')
#9 /var/www/html/nextcloud/lib/private/ServerContainer.php(132): OC\AppFramework
\Utility\SimpleContainer->query('OCP\\IDBConnecti...')
#10 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(164): OC\ServerContainer->query('OCP\\IDBConnecti...')
#11 /var/www/html/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(109)
: OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Obj
ect(OC\Server))
#12 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(116): Pimple\Container->offsetGet('DatabaseConnect...')
#13 /var/www/html/nextcloud/lib/private/ServerContainer.php(132): OC\AppFramewor
k\Utility\SimpleContainer->query('DatabaseConnect...')
#14 /var/www/html/nextcloud/lib/private/Server.php(1511): OC\ServerContainer->qu
ery('DatabaseConnect...')
#15 /var/www/html/nextcloud/lib/private/Server.php(332): OC\Server->getDatabaseC
onnection()
#16 /var/www/html/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113)
: OC\Server->OC\{closure}(Object(OC\Server))
#17 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(116): Pimple\Container->offsetGet('OC\\Authenticati...')
#18 /var/www/html/nextcloud/lib/private/ServerContainer.php(132): OC\AppFramewor
k\Utility\SimpleContainer->query('OC\\Authenticati...')
#19 /var/www/html/nextcloud/lib/private/Server.php(336): OC\ServerContainer->que
ry('OC\\Authenticati...')
#20 /var/www/html/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113)
: OC\Server->OC\{closure}(Object(OC\Server))
#21 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(116): Pimple\Container->offsetGet('OC\\Authenticati...')
#22 /var/www/html/nextcloud/lib/private/ServerContainer.php(132): OC\AppFramewor
k\Utility\SimpleContainer->query('OC\\Authenticati...')
#23 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(164): OC\ServerContainer->query('OC\\Authenticati...')
#24 /var/www/html/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(109)
: OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Obj
ect(OC\Server))
#25 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(116): Pimple\Container->offsetGet('OC\\Authenticati...')
#26 /var/www/html/nextcloud/lib/private/ServerContainer.php(132): OC\AppFramewor
k\Utility\SimpleContainer->query('OC\\Authenticati...')
#27 /var/www/html/nextcloud/lib/private/Server.php(352): OC\ServerContainer->que
ry('OC\\Authenticati...')
#28 /var/www/html/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113)
: OC\Server->OC\{closure}(Object(OC\Server))
#29 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(116): Pimple\Container->offsetGet('OCP\\IUserSessio...')
#30 /var/www/html/nextcloud/lib/private/ServerContainer.php(132): OC\AppFramewor
k\Utility\SimpleContainer->query('OCP\\IUserSessio...')
#31 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(164): OC\ServerContainer->query('OCP\\IUserSessio...')
#32 /var/www/html/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(109)
: OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Obj
ect(OC\Server))
#33 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(116): Pimple\Container->offsetGet('UserSession')
#34 /var/www/html/nextcloud/lib/private/ServerContainer.php(132): OC\AppFramewor
k\Utility\SimpleContainer->query('UserSession')
#35 /var/www/html/nextcloud/lib/private/Server.php(1380): OC\ServerContainer->qu
ery('UserSession')
#36 /var/www/html/nextcloud/lib/private/Server.php(688): OC\Server->getUserSessi
on()
#37 /var/www/html/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113)
: OC\Server->OC\{closure}(Object(OC\Server))
#38 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(116): Pimple\Container->offsetGet('OC\\App\\AppManag...')
#39 /var/www/html/nextcloud/lib/private/ServerContainer.php(132): OC\AppFramewor
k\Utility\SimpleContainer->query('OC\\App\\AppManag...')
#40 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(164): OC\ServerContainer->query('OC\\App\\AppManag...')
#41 /var/www/html/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(109)
: OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Obj
ect(OC\Server))
#42 /var/www/html/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php
(116): Pimple\Container->offsetGet('AppManager')
#43 /var/www/html/nextcloud/lib/private/ServerContainer.php(132): OC\AppFramewor
k\Utility\SimpleContainer->query('AppManager')
#44 /var/www/html/nextcloud/lib/private/Server.php(1684): OC\ServerContainer->qu
ery('AppManager')
#45 /var/www/html/nextcloud/lib/private/legacy/app.php(332): OC\Server->getAppMa
nager()
#46 /var/www/html/nextcloud/lib/private/legacy/app.php(113): OC_App::getEnabledA
pps()
#47 /var/www/html/nextcloud/lib/base.php(662): OC_App::loadApps(Array)
#48 /var/www/html/nextcloud/lib/base.php(1081): OC::init()
#49 /var/www/html/nextcloud/console.php(46): require_once('/var/www/html/n...')
#50 /var/www/html/nextcloud/occ(11): require_once('/var/www/html/n...')
#51 {main}PHP Fatal error:  Uncaught Doctrine\DBAL\DBALException: Failed to conn
ect to the database: An exception occurred while executing 'PRAGMA journal_mode 
= WAL':

SQLSTATE[HY000]: General error: 10 disk I/O error in /var/www/html/nextcloud/lib
/private/DB/Connection.php:64
Stack trace:
#0 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.p
hp(992): OC\DB\Connection->connect()
#1 /var/www/html/nextcloud/lib/private/DB/Connection.php(216): Doctrine\DBAL\Con
nection->executeUpdate('PRAGMA read_unc...', Array, Array)
#2 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.p
hp(623): OC\DB\Connection->executeUpdate('PRAGMA read_unc...')
#3 /var/www/html/nextcloud/lib/private/DB/Connection.php(151): Doctrine\DBAL\Con
nection->setTransactionIsolation(2)
#4 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManage
r.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDO
Sqlite\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\Even
tMana in /var/www/html/nextcloud/lib/private/DB/Connection.php on line 64
root@heviiMTLServer nextcloud# 

Any further suggestions?

@Hevii_Guy from the very first error, I gather you are missing that library. From this site I gather you could install the RPM, then try again?

This is the line I am referring to. Based on google it seems the above mentioned site contains the RPM to provide this library?

It seems you are missing quite a few libraries. Try to use the dnf whatprovides command to see what packages provides the libraries mentioned in the errors.

Thanks again, Hans.

Although as you’ve pointed out I clearly need it, I’m surprised that Nextcloud runs on my server in spite of not having the requisite library. Yet, I must correct the problem. However, this brings me into waters which are well above my head…

Whereas I think that I can load the rpm locally, my grasp of the CLI is too limited to enable me to do it remotely (from across the Atlantic Ocean) on my tablet! What I’m saying (with considerable shame) is that the requisite cut’n’paste-able commands would really be appreciated if somebody would be so kind and generous as to post them :wink:

Oh, I just remembered: I had tried to install trash-cli earlier but even though the file was relatively puny, I didn’t have enough room on the partition; installation was refused. Hence, even if I had the necessary commands to install the dynamic library, I suspect that it too would be refused.

I’m in a bit of a bind, huh? :grimacing:

Could you perhaps paste the output of php -v? And can you at all remember which repo you used to install php with? (SCL, remi, webtatic etc?) Only then will I be able to help you with the necessary commands.

As luck would have it, I had a Fedora 26 machine on hand :slight_smile:

Quick dnf search turned up the following PHP libraries, hope they are the same on Fedora 28:

sudo dnf search php | grep redis
php-pecl-redis.x86_64 : Extension for communicating with the Redis key-value

sudo dnf search php | grep imagick
php-pecl-imagick.x86_64 : Provides a wrapper to the ImageMagick library
php-pecl-imagick-devel.x86_64 : imagick extension developer files (header)

sudo dnf search php | grep igbinary
php-pecl-igbinary.x86_64 : Replacement for the standard PHP serializer
php-pecl-igbinary-devel.x86_64 : Igbinary developer files (header)

sudo dnf search php | grep radius
php-pear-Auth-radius.noarch : RADIUS support for php-pear-Auth
php-pecl-radius.x86_64 : Radius client library

Which to me seems you need:

sudo dnf install php-pecl-redis php-pecl-imagick php-pecl-igbinary php-pecl-radius

This is a wild guess, but seeing as you are across the Atlantic, it is worth a shot I guess? After you have done this:

systemctl restart httpd.service

Then run the occ command again.
HTH

I appreciate your help, Hans.

Running php-v results in the same fault (unable to load dynamic library ‘xmlrpc.so’ …yada yada…)

However, running which php shows that it’s in /bin/php

So, it seems to me that php is indeed installed. Could it not be running because the root partition is so completely full and thus constipated?

  • I’ve got a rather large (massive) directory within Nextcloud/data which I’m willing to sacrifice (delete, get-rid-of, trash etc). I’m thinking that this should release a considerable amount of partition real estate. If I’m on the right track, can I simply kill this directory with an rm from the CLI or would this screw-up the database?

I just noticed your latest reply. Thanks! I’ll give it a shot but, I’m afraid that the CLI is going to tell me that I don’t have enough room!

I am not sure what emtying the directory would result in regarding the database. I have never done that, so I cannot give you any definitive answer.
Try to see if you can clear up space in /tmp, as a full partition there might also result in some wonky terminal commands.
AFAIK php doesn’t “run” by default, like httpd and firewall etc does. /bin/php merely points to an executable which is called when something needs it to run, like in the case of Nextcloud.

Yip, just as I had suspected: “At least 4MB more space needed on the / filesystem
(which is somewhat of a mystery since df shows me that the partition is 5.1343840 GB yet I’m only using 4.9361096 GB)

Good thing is that I now know that I do indeed require those files. The problem is that I’m not sure how I can load them!

I only have a few .service files in the root tmp directory. Can’t imagine that they’re bunging-up my system. Earlier I had dnf-removed a couple of large programs (FlightGear and Gimp) in an effort to recover some valuable space. Funny thing is that it didn’t seem to have made any difference; the root directory was still showing as 100% full.

I know it sounds foolish, but after the DNF remove, have you rebooted?

Occaisionally (just ‘occaisionally’) I’ve been known as a fool. I’m almost certain that I had rebooted. Nevertheless, I’ll kill and revive the box now…

What you could do is the following:

  • stop httpd.service

  • move something else from /var/www/html/nextcloud/data to somewhere not on the root filesystem, anywhere you have space

  • install the packages

  • run the occ command

  • move content back (which you previously moved out of the directory)

  • start apache

Don’t know if that will help, but maybe worth a go?

Great idea! I’ll give it a shot…

  • rebooting was fruitless :frowning:

I now know… Woo hoo! It worked like a veritable linux version of Milk of Magnesia; the gates opened wide and dumped a massive load of crap thus allowing the remainder of the system to finally function properly! The server is working once again and my wife and I can sync our trans-ocean calendars :earth_americas::earth_africa:

Thank-you so very much, Hans :star_struck:

Glad I could help! All the best! Maybe mark it solved just incase.