VMware appliance constantly losing connectivity -RESOLVED

VMware appliance constantly losing connectivity

Nextcloud version 16.0.2:
Operating system and version (eg, Ubuntu 17.04):
Apache or nginx version Unknown:
PHP version 7.2.19-0ubuntu0.18.04.1:

The issue you are facing:

I’ve deployed (and redeployed) the v16 .ova appliance several times in a robust VMware physical lab.

Everything seems to install correctly including SSL however I am experiencing an issue where the server connectivity via web and client apps.

I think something is going sideways durubg the initial upgrade but not sure. The only changes I have made to the appliance are as follows:

Increased RAM to 8GB
Increased CPUs to 4
Added a 500GB drive and extended the xfs partition.

I have also deployed the Owncloud appliance in the same VMware environment and it is working flawlessly which would rule out my firewall causing the connectivity issues. Any help would be greatly appreciated!

Is this the first time you’ve seen this error? Yes:

Steps to replicate it:

  1. Deploy the ova template in vCenter
  2. Run all of the initial configuration scripts
  3. enjoy intermittent connectivity.

The output of your Nextcloud log in Admin > Logging:

Fatal	webdav	Sabre\DAV\Exception\BadRequest: expected filesize 89852979 got 59916288		2019-07-04T12:42:13-0400
Fatal	webdav	Sabre\DAV\Exception\BadRequest: expected filesize 100000000 got 35684352		2019-07-04T12:42:13-0400
Error	no app in context	Sabre\DAV\Exception\BadRequest: expected filesize 89852979 got 59916288		2019-07-04T12:42:13-0400
Error	no app in context	Sabre\DAV\Exception\BadRequest: expected filesize 100000000 got 35684352		2019-07-04T12:42:13-0400
Fatal	webdav	Doctrine\DBAL\Exception\DeadlockException: An exception occurred while executing 'UPDATE "oc_filecache" SET "mtime" = GREATEST("mtime", ?), "etag" = ? WHERE ("storage" = ?) AND ("path_hash" IN ('d41d8cd98f00b204e9800998ecf8427e', '45b963397aa40d4a0063e0d85e4fe7a1', '2e20f9270005581ccd5151ae797b1a74', 'd95b4b83ca2fbb2c11c91305de237cd6'))' with params [1562258445, "5d1e2c0de106d", 3]: SQLSTATE[40P01]: Deadlock detected: 7 ERROR: deadlock detected DETAIL: Process 8350 waits for ShareLock on transaction 4529; blocked by process 8353. Process 8353 waits for ShareLock on transaction 4530; blocked by process 8350. HINT: See server log for query details. CONTEXT: while updating tuple (15,113) in relation "oc_filecache"		2019-07-04T12:40:46-0400
Error	index	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'UPDATE "oc_filecache" SET "mimepart" = ?, "mimetype" = ?, "mtime" = ?, "size" = ?, "etag" = ?, "storage_mtime" = ?, "permissions" = ?, "checksum"=? WHERE ("mimepart" <> ? OR "mimetype" <> ? OR "mtime" <> ? OR "size" <> ? OR "etag" <> ? OR "storage_mtime" <> ? OR "permissions" <> ? OR "checksum" <> ? OR "mimepart" IS NULL OR "mimetype" IS NULL OR "mtime" IS NULL OR "size" IS NULL OR "etag" IS NULL OR "storage_mtime" IS NULL OR "permissions" IS NULL OR "checksum" IS NULL) AND "fileid" = ? ' with params [19, 19, 1562257632, false, "5d1e28e004f31", 1562257632, 11, "", 19, 19, 1562257632, false, "5d1e28e004f31", 1562257632, 11, "", 113]: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input syntax for integer: ""		2019-07-04T12:27:12-0400
Error	PHP	filesize(): stat failed for /mnt/ncdata/appdata_oc65lc0lmz8d/preview/49/500-500-max.png at /var/www/nextcloud/lib/private/Files/Storage/Local.php#169		2019-07-04T12:27:12-0400
Warning	core	Login failed: 'admin' (Remote IP: '74.117.147.174')		2019-07-04T12:26:38-0400
Error	PHP	file_put_contents(/mnt/ncdata/.htaccess): failed to open stream: Permission denied at /var/www/nextcloud/lib/private/Setup.php#560		2019-07-04T11:56:57-0400
Debug	core	starting upgrade from 16.0.1.1 to 16.0.2.1		2019-07-04T11:56:57-0400

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
/**
 ---<lines removed for brievity>---
 */

namespace OC;

/**
 * This class is responsible for reading and writing config.php, the very basic
 * configuration file of Nextcloud.
 */
class Config {

        const ENV_PREFIX = 'NC_';

        /** @var array Associative array ($key => $value) */
        protected $cache = array();
        /** @var string */
        protected $configDir;
        /** @var string */
        protected $configFilePath;
        /** @var string */
        protected $configFileName;

        /**
         * @param string $configDir Path to the config dir, needs to end with '/'
         * @param string $fileName (Optional) Name of the config file. Defaults to config.php
         */
        public function __construct($configDir, $fileName = 'config.php') {
                $this->configDir = $configDir;
                $this->configFilePath = $this->configDir.$fileName;
                $this->configFileName = $fileName;
                $this->readData();
        }

        /**
         * Lists all available config keys
         *
         * Please note that it does not return the values.
         *
         * @return array an array of key names
         */
        public function getKeys() {
                return array_keys($this->cache);
        }

        /**
         * Returns a config value
         *
         * gets its value from an `NC_` prefixed environment variable
         * if it doesn't exist from config.php
         * if this doesn't exist either, it will return the given `$default`
         *
         * @param string $key key
         * @param mixed $default = null default value
         * @return mixed the value or $default
         */
        public function getValue($key, $default = null) {
                $envValue = getenv(self::ENV_PREFIX . $key);
                if ($envValue !== false) {
                        return $envValue;
                }

                if (isset($this->cache[$key])) {
                        return $this->cache[$key];
                }

                return $default;
        }

        /**
         * Sets and deletes values and writes the config.php
         *
         * @param array $configs Associative array with `key => value` pairs
         *                       If value is null, the config key will be deleted
         */
        public function setValues(array $configs) {
                $needsUpdate = false;
                foreach ($configs as $key => $value) {
                        if ($value !== null) {
                                $needsUpdate |= $this->set($key, $value);
                        } else {
                                $needsUpdate |= $this->delete($key);
                        }
                }

                if ($needsUpdate) {
                        // Write changes
                        $this->writeData();
                }
        }

        /**
         * Sets the value and writes it to config.php if required
         *
         * @param string $key key
         * @param mixed $value value
         */
        public function setValue($key, $value) {
                if ($this->set($key, $value)) {
                        // Write changes
                        $this->writeData();
                }
        }

        /**
         * This function sets the value
         *
         * @param string $key key
         * @param mixed $value value
         * @return bool True if the file needs to be updated, false otherwise
         */
        protected function set($key, $value) {
                if (!isset($this->cache[$key]) || $this->cache[$key] !== $value) {
                        // Add change
                        $this->cache[$key] = $value;
                        return true;
                }

                return false;
        }

        /**
         * Removes a key from the config and removes it from config.php if required
         * @param string $key
         */
        public function deleteKey($key) {
                if ($this->delete($key)) {
                        // Write changes
                        $this->writeData();
                }
        }

        /**
         * This function removes a key from the config
         *
         * @param string $key
         * @return bool True if the file needs to be updated, false otherwise
         */
        protected function delete($key) {
                if (isset($this->cache[$key])) {
                        // Delete key from cache
                        unset($this->cache[$key]);
                        return true;
                }
                return false;
        }

        /**
         * Loads the config file
         *
         * Reads the config file and saves it to the cache
         *
         * @throws \Exception If no lock could be acquired or the config file has not been found
         */
        private function readData() {
                // Default config should always get loaded
                $configFiles = array($this->configFilePath);

                // Add all files in the config dir ending with the same file name
                $extra = glob($this->configDir.'*.'.$this->configFileName);
                if (is_array($extra)) {
                        natsort($extra);
                        $configFiles = array_merge($configFiles, $extra);
                }

                // Include file and merge config
                foreach ($configFiles as $file) {
                        $fileExistsAndIsReadable = file_exists($file) && is_readable($file);
                        $filePointer = $fileExistsAndIsReadable ? fopen($file, 'r') : false;
                        if($file === $this->configFilePath &&
                                $filePointer === false) {
                                // Opening the main config might not be possible, e.g. if the wrong
                                // permissions are set (likely on a new installation)
                                continue;
                        }

                        // Try to acquire a file lock
                        if(!flock($filePointer, LOCK_SH)) {
                                throw new \Exception(sprintf('Could not acquire a shared lock on the config file %s', $file));
                        }

                        unset($CONFIG);
                        include $file;
                        if(isset($CONFIG) && is_array($CONFIG)) {
                                $this->cache = array_merge($this->cache, $CONFIG);
                        }

                        // Close the file pointer and release the lock
                        flock($filePointer, LOCK_UN);
                        fclose($filePointer);
                }
        }

        /**
         * Writes the config file
         *
         * Saves the config to the config file.
         *
         * @throws HintException If the config file cannot be written to
         * @throws \Exception If no file lock can be acquired
         */
        private function writeData() {
                // Create a php file ...
                $content = "<?php\n";
                $content .= '$CONFIG = ';
                $content .= var_export($this->cache, true);
                $content .= ";\n";

                touch ($this->configFilePath);
                $filePointer = fopen($this->configFilePath, 'r+');

                // Prevent others not to read the config
                chmod($this->configFilePath, 0640);

                // File does not exist, this can happen when doing a fresh install
                if(!is_resource ($filePointer)) {
                        // TODO fix this via DI once it is very clear that this doesn't cause side effects due to initialization order
                        // currently this breaks app routes but also could have other side effects especially during setup and exception handling
                        $url = \OC::$server->getURLGenerator()->linkToDocs('admin-dir_permissions');
                        throw new HintException(
                                "Can't write into config directory!",
                                'This can usually be fixed by giving the webserver write access to the config directory. See ' . $url);
                }

                // Try to acquire a file lock
                if(!flock($filePointer, LOCK_EX)) {
                        throw new \Exception(sprintf('Could not acquire an exclusive lock on the config file %s', $this->configFilePath));
                }

                // Write the config and release the lock
                ftruncate ($filePointer, 0);
                fwrite($filePointer, $content);
                fflush($filePointer);
                flock($filePointer, LOCK_UN);
                fclose($filePointer);

                if (function_exists('opcache_invalidate')) {
                        @opcache_invalidate($this->configFilePath, true);
                }
        }
}

The output of your Apache/nginx/system log in /var/log/____:

root@ztcloud-nc:/var/log/apache2# less error.log | grep error
[Thu Jul 04 12:35:39.663017 2019] [:error] [pid 1894:tid 139706922673920] [client 69.213.236.57:51802] [client 69.213.236.57] ModSecurity: Request body (Content-Length) is larger than the configured limit (13107200). [hostname "ztcloud-nc.sddatacenter.com"] [uri "/remote.php/dav/uploads/ztaylor/3947277039/00000001"] [unique_id "XR4q2zK47Ks7yf2SbT6xfAAAAEY"]
[Thu Jul 04 12:35:39.685640 2019] [:error] [pid 1894:tid 139706922673920] [client 69.213.236.57:51802] [client 69.213.236.57] ModSecurity: Request body (Content-Length) is larger than the configured limit (13107200). [hostname "ztcloud-nc.sddatacenter.com"] [uri "/remote.php/dav/uploads/ztaylor/3947277039/00000001"] [unique_id "XR4q2zK47Ks7yf2SbT6xfAAAAEY"]
[Thu Jul 04 12:35:43.897455 2019] [:error] [pid 1893:tid 139706645845760] [client 69.213.236.57:18395] [client 69.213.236.57] ModSecurity: Request body (Content-Length) is larger than the configured limit (13107200). [hostname "ztcloud-nc.sddatacenter.com"] [uri "/remote.php/dav/uploads/ztaylor/2096965540/00000000"] [unique_id "XR4q35J3kcvAL6zPwnUSNQAAABU"]
[Thu Jul 04 12:35:43.921888 2019] [:error] [pid 1893:tid 139706645845760] [client 69.213.236.57:18395] [client 69.213.236.57] ModSecurity: Request body (Content-Length) is larger than the configured limit (13107200). [hostname "ztcloud-nc.sddatacenter.com"] [uri "/remote.php/dav/uploads/ztaylor/2096965540/00000000"] [unique_id "XR4q35J3kcvAL6zPwnUSNQAAABU"]
[Thu Jul 04 12:35:44.127348 2019] [:error] [pid 1893:tid 139706671023872] [client 69.213.236.57:40846] [client 69.213.236.57] ModSecurity: Request body (Content-Length) is larger than the configured limit (13107200). [hostname "ztcloud-nc.sddatacenter.com"] [uri "/remote.php/dav/uploads/ztaylor/1750122773/00000000"] [unique_id "XR4q4JJ3kcvAL6zPwnUSNgAAABI"]
Jul  5 10:27:35 ztcloud-nc kernel: [76652.933193] device-mapper: ioctl: error adding target to table
Jul  5 10:35:59 ztcloud-nc systemd-resolved[1302]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jul  5 10:36:38 ztcloud-nc systemd-resolved[1302]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jul  5 10:38:55 ztcloud-nc kernel: [77332.822644] device-mapper: ioctl: error adding target to table
Jul  5 10:40:10 ztcloud-nc kernel: [   13.070336] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro
Jul  5 10:41:09 ztcloud-nc apt-helper[2210]: E: Sub-process nm-online returned an error code (1)

Finally figured out this was due to an ip conflict. Please mark as closed.