Help digging into an unknown locking problem with large (100+GB) drop uploads

N

Nextcloud version: 16.01
Operating system and version: CentOS 7.6
Apache or nginx version:NGINX 1.16
PHP version: 7.3

The issue you are facing:
My logs are showing a lock issue when uploading very large files. In this instance, just over 100GB. Smaller, but still large files (43GB) seem to work without issue. I’m looking for some pointers of where to look for more information to resolve this. I have plenty of disk space, I believe. Both data and tmp directories are on NFS mounts with more than 10TB free space.
Thanks in advance!

Scenario: Uploading a file > 100GB to a drop share.

Steps to replicate it:

  1. Create share
  2. Mark share as drop, upload only
  3. Upload file larger than 100GB

The output of your Nextcloud log in Admin > Logging:

 "reqId": "ttsPtMGyWM1h3rMHmlsG",
    "level": 3,
    "time": "2019-05-21T08:39:24+00:00",
    "remoteAddr": "XXX.XXX.XXX.XXX",
    "user": "--",
    "app": "no app in context",
    "method": "PUT",
    "url": "\/public.php\/webdav\/vm_diskimage.qcow2",
    "message": {
        "Exception": "OCP\\Lock\\LockedException",
        "Message": "\"files\/aef6206389df1df3ffd0825b208a83e0\" is locked, existing lock on file: none",
        "Code": 0,
        "Trace": [
            {
                "file": "\/nextcloud\/install\/path\/lib\/private\/Files\/Storage\/Common.php",
                "line": 779,
                "function": "changeLock",
                "class": "OC\\Lock\\MemcacheLockingProvider",
                "type": "->",
                "args": [
                    "files\/aef6206389df1df3ffd0825b208a83e0",
                    2
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php",
                "line": 615,
                "function": "changeLock",
                "class": "OC\\Files\\Storage\\Common",
                "type": "->",
                "args": [
                    "files\/Client Uploads\/vm_diskimage.qcow2",
                    2,
                    {
                        "__class__": "OC\\Lock\\MemcacheLockingProvider"
                    }
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php",
                "line": 615,
                "function": "changeLock",
                "class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
                "type": "->",
                "args": [
                    "files\/Client Uploads\/vm_diskimage.qcow2",
                    2,
                    {
                        "__class__": "OC\\Lock\\MemcacheLockingProvider"
                    }
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php",
                "line": 615,
                "function": "changeLock",
                "class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
                "type": "->",
                "args": [
                    "files\/Client Uploads\/vm_diskimage.qcow2",
                    2,
                    {
                        "__class__": "OC\\Lock\\MemcacheLockingProvider"
                    }
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php",
                "line": 615,
                "function": "changeLock",
                "class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
                "type": "->",
                "args": [
                    "files\/Client Uploads\/vm_diskimage.qcow2",
                    2,
                    {
                        "__class__": "OC\\Lock\\MemcacheLockingProvider"
                    }
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/lib\/private\/Files\/View.php",
                "line": 1968,
                "function": "changeLock",
                "class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
                "type": "->",
                "args": [
                    "files\/Client Uploads\/vm_diskimage.qcow2",
                    2,
                    {
                        "__class__": "OC\\Lock\\MemcacheLockingProvider"
                    }
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/apps\/dav\/lib\/Connector\/Sabre\/Node.php",
                "line": 392,
                "function": "changeLock",
                "class": "OC\\Files\\View",
                "type": "->",
                "args": [
                    "\/vm_diskimage.qcow2",
                    2
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/apps\/dav\/lib\/Connector\/Sabre\/File.php",
                "line": 241,
                "function": "changeLock",
                "class": "OCA\\DAV\\Connector\\Sabre\\Node",
                "type": "->",
                "args": [
                    2
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php",
                "line": 156,
                "function": "put",
                "class": "OCA\\DAV\\Connector\\Sabre\\File",
                "type": "->",
                "args": [
                    null
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php",
                "line": 1096,
                "function": "createFile",
                "class": "OCA\\DAV\\Connector\\Sabre\\Directory",
                "type": "->",
                "args": [
                    "vm_diskimage.qcow2",
                    null
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php",
                "line": 525,
                "function": "createFile",
                "class": "Sabre\\DAV\\Server",
                "type": "->",
                "args": [
                    "vm_diskimage.qcow2",
                    null,
                    null
                ]
            },
            {
                "function": "httpPut",
                "class": "Sabre\\DAV\\CorePlugin",
                "type": "->",
                "args": [
                    {
                        "absoluteUrl": "https:\/\/transfer.newestech.com\/public.php\/webdav\/vm_diskimage.qcow2",
                        "__class__": "Sabre\\HTTP\\Request"
                    },
                    {
                        "__class__": "Sabre\\HTTP\\Response"
                    }
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php",
                "line": 105,
                "function": "call_user_func_array",
                "args": [
                    [
                        {
                            "__class__": "Sabre\\DAV\\CorePlugin"
                        },
                        "httpPut"
                    ],
                    [
                        {
                            "absoluteUrl": "https:\/\/transfer.newestech.com\/public.php\/webdav\/vm_diskimage.qcow2",
                            "__class__": "Sabre\\HTTP\\Request"
                        },
                        {
                            "__class__": "Sabre\\HTTP\\Response"
                        }
                    ]
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php",
                "line": 479,
                "function": "emit",
                "class": "Sabre\\Event\\EventEmitter",
                "type": "->",
                "args": [
                    "method:PUT",
                    [
                        {
                            "absoluteUrl": "https:\/\/transfer.newestech.com\/public.php\/webdav\/vm_diskimage.qcow2",
                            "__class__": "Sabre\\HTTP\\Request"
                        },
                        {
                            "__class__": "Sabre\\HTTP\\Response"
                        }
                    ]
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php",
                "line": 254,
                "function": "invokeMethod",
                "class": "Sabre\\DAV\\Server",
                "type": "->",
                "args": [
                    {
                        "absoluteUrl": "https:\/\/transfer.newestech.com\/public.php\/webdav\/vm_diskimage.qcow2",
                        "__class__": "Sabre\\HTTP\\Request"
                    },
                    {
                        "__class__": "Sabre\\HTTP\\Response"
                    }
                ]
            },
            {
                "file": "\/nextcloud\/install\/path\/apps\/dav\/appinfo\/v1\/publicwebdav.php",
                "line": 107,
                "function": "exec",
                "class": "Sabre\\DAV\\Server",
                "type": "->",
                "args": []
            },
            {
                "file": "\/nextcloud\/install\/path\/public.php",
                "line": 79,
                "args": [
                    "\/nextcloud\/install\/path\/apps\/dav\/appinfo\/v1\/publicwebdav.php"
                ],
                "function": "require_once"
            }
        ],
        "File": "\/nextcloud\/install\/path\/lib\/private\/Lock\/MemcacheLockingProvider.php",
        "Line": 129,
        "CustomMessage": "--"
    },
    "userAgent": "Mozilla\/5.0 (X11; Fedora; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/66.0.3359.170 Safari\/537.36",
    "version": "16.0.1.1"
}

There were no errors in the nginx errors log.