Strange issue with uploads

ANCU · July 14, 2024, 6:43pm

Hello,

Long time listener, first time caller.

I’d appreciate your support in resolving this issue. Please let me know if you need any more information.

I have had NextCloud working since December 2022 on my setup:

Physical Windows 2022 Server. (Host, with multiple raid drives):
-OS on Volume 1,
-VMs on Volume 2,
-storage on Volume 3
Ubuntu 2204 Virtual Machine (Guest)
Docker Compose (using NginX Proxy Manager)

I followed this guide to build the current setup
https://www.smarthomebeginner.com/docker-media-server-2022/

I previously followed an older version of this guide so I was also using NextCloud before 2022

Nextcloud (and other Ubuntu based containers) can successfully read and write to shared storage that’s hosted on the Windows Server.

As of a few days ago, other containers continue to read and write to the shared storage, however, it seems that NextCloud can only read.

Uploads (attempts to write) from an Android phone and a Windows laptop fail.

On Android, I’ve uninstalled the regular NextCloud client and installed NextCloud Dev to access logs.

On Android, the notification says You are not permitted to upload to this folder.
From NextCloud Dev, the logs include:
NotifyUploadResult with resultCode: FORBIDDEN

status=UPLOAD_FAILED

(if anyone can advise me on how to copy the logs out of the app and paste them here, that would be appreciated)

This got me thinking there may be an issue with my drive mapping (/etc/fstab) or permissions (chown/chmod).

I’ve had multiple checks/reviews and re-applied the permissions.
I’ve then restarted the NextCloud container.

All appears okay but the uploads still fail.

I spent a few days searching the web and this website for similar issues and a fix but I’ve been unsuccessful so far.
I’d appreciate your support in resolving this issue. Please let me know if you need any more information.

The NextCloud section of my docker-compose looks like this

NextCloud – Your Own Cloud Storage

nextcloud:
container_name: nextcloud
hostname: nextcloud
restart: always
image: lscr.io/linuxserver/nextcloud:version-29.0.3
volumes:
- $DOCKERDIR/appdata/nextcloud/www/html/config:/config
- $DOCKERDIR/appdata/nextcloud/www/nextcloud:/app/www/public
- $DOCKERDIR/appdata/nextcloud/www/html:/nextcloud
- $DOCKERDIR/appdata/nextcloud/www/html/custom_apps:/apps
- /media/nextclouddata:/data
- ${USERDIR}/docker/shared:/shared
ports:
- “444:443”
environment:
- MYSQL_DATABASE=nextcloud
- MYSQL_HOST=192.168.x.xx:3306
- MYSQL_USER=*****
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
- PUID=${PUID}
- PGID=${PGID}

ANCU · July 16, 2024, 7:53pm

I’ve also realised that there doesn’t seem to be any issue with uploading contacts daily.

wwe · July 16, 2024, 8:06pm

you should first isolate the problem:

review if the problem occurs with other clients as well (browser, desktop client)
- if yes, likely server issue
- if no client issue
review container logs (docker inspect {container name})
review application logs (nextcloud.log in you data directory)

you run very uncommon setup I would recommend switching to native Linux and Docker. It’s nothing wrong in general but in case of issues nested virtualization and Windows add complexity and troubleshooting more complex. Just in case it turns out the application could not write inside of container you need to troubleshoot the whole chain: if the data directory is writable inside of container, in the VM and last on the host

ANCU · July 16, 2024, 10:04pm

Thanks for the response wwe.

I’ve confirmed that uploads fail from:

Android and Windows on my account.
Android on my friend’s account.

Therefore, I’m inclined to believe it’s a server issue.

I’ll review the logs in the morning.

ANCU · July 17, 2024, 10:02pm

I’ve inspected the container as advised, I don’t think I see anything unusual:

As a new user of this website, there are apparently limits to how much I’m allowed to post so I’ll paste the section that I think concerns permissions:

“Binds”: [
“/home/XxXxX/docker/appdata/nextcloud/www/nextcloud:/app/www/public:rw”,
“/home/XxXxX/docker/appdata/nextcloud/www/html:/nextcloud:rw”,
“/home/XxXxX/docker/appdata/nextcloud/www/html/custom_apps:/apps:rw”,
“/media/nextclouddata:/data:rw”,
“/home/XxXxX/docker/shared:/shared:rw”,
“/home/XxXxX/docker/appdata/nextcloud/www/html/config:/config:rw”
],

The whole thing can be viewed here f876c2f3 - XxXxX@uhts:~/docker$ sudo docker inspect nextcloud [sudo] password for XxXxX:...

ANCU · July 17, 2024, 10:35pm

The end of the logs is here a6eb590a - Log

From what I see, it still appears to concern permissions but I don’t see the issue.

(Remote IP: ‘xx.xx.xxx.xxx’)“,“userAgent”:“Mozilla/5.0 (Android) Nextcloud-android/3.29.2”,“version”:“29.0.3.4”,“data”:{“app”:“core”}}
{“reqId”:“D84cS14k5vOTGLJweIZn”,“level”:3,“time”:“2024-07-17T11:09:59+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“PHP”,“method”:“POST”,“url”:”/remote.php/dav/bulk",“message”:“file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test.txt): Failed to open stream: No such file or directory at /app/www/public/lib/private/Files/Storage/Local.php#339”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“app”:“PHP”}}
{“reqId”:“D84cS14k5vOTGLJweIZn”,“level”:3,“time”:“2024-07-17T11:09:59+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“no app in context”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“Could not create path "/user1/files/InstantUpload/Camera/2024/07/Test.txt"”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“path”:“/InstantUpload/Camera/2024/07/Test.txt”}}
{“reqId”:“D84cS14k5vOTGLJweIZn”,“level”:3,“time”:“2024-07-17T11:09:59+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“PHP”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test2.txt): Failed to open stream: No such file or directory at /app/www/public/lib/private/Files/Storage/Local.php#339”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“app”:“PHP”}}
{“reqId”:“D84cS14k5vOTGLJweIZn”,“level”:3,“time”:“2024-07-17T11:09:59+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“no app in context”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“Could not create path "/user1/files/InstantUpload/Camera/2024/07/Test2.txt"”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“path”:“/InstantUpload/Camera/2024/07/Test2.txt”}}
{“reqId”:“soklH42ZDfYRFRS6mlAj”,“level”:3,“time”:“2024-07-17T15:15:39+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“PHP”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test.txt): Failed to open stream: No such file or directory at /app/www/public/lib/private/Files/Storage/Local.php#339”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“app”:“PHP”}}
{“reqId”:“soklH42ZDfYRFRS6mlAj”,“level”:3,“time”:“2024-07-17T15:15:39+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“no app in context”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“Could not create path "/user1/files/InstantUpload/Camera/2024/07/Test.txt"”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“path”:“/InstantUpload/Camera/2024/07/Test.txt”}}
{“reqId”:“soklH42ZDfYRFRS6mlAj”,“level”:3,“time”:“2024-07-17T15:15:39+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“PHP”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test2.txt): Failed to open stream: No such file or directory at /app/www/public/lib/private/Files/Storage/Local.php#339”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“app”:“PHP”}}
{“reqId”:“soklH42ZDfYRFRS6mlAj”,“level”:3,“time”:“2024-07-17T15:15:39+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“no app in context”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“Could not create path "/user1/files/InstantUpload/Camera/2024/07/Test2.txt"”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“path”:“/InstantUpload/Camera/2024/07/Test2.txt”}}
{“reqId”:“whVW4NDVPzjlVcjJEZYm”,“level”:3,“time”:“2024-07-17T17:15:51+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“PHP”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test.txt): Failed to open stream: No such file or directory at /app/www/public/lib/private/Files/Storage/Local.php#339”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“app”:“PHP”}}
{“reqId”:“whVW4NDVPzjlVcjJEZYm”,“level”:3,“time”:“2024-07-17T17:15:51+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“no app in context”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“Could not create path "/user1/files/InstantUpload/Camera/2024/07/Test.txt"”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“path”:“/InstantUpload/Camera/2024/07/Test.txt”}}
{“reqId”:“whVW4NDVPzjlVcjJEZYm”,“level”:3,“time”:“2024-07-17T17:15:51+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“PHP”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test2.txt): Failed to open stream: No such file or directory at /app/www/public/lib/private/Files/Storage/Local.php#339”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“app”:“PHP”}}
{“reqId”:“whVW4NDVPzjlVcjJEZYm”,“level”:3,“time”:“2024-07-17T17:15:51+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“no app in context”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“Could not create path "/user1/files/InstantUpload/Camera/2024/07/Test2.txt"”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“path”:“/InstantUpload/Camera/2024/07/Test2.txt”}}
{“reqId”:“TsbfLHJltmkfq2Ucs6Ai”,“level”:3,“time”:“2024-07-17T21:41:17+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“PHP”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test.txt): Failed to open stream: No such file or directory at /app/www/public/lib/private/Files/Storage/Local.php#339”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“app”:“PHP”}}
{“reqId”:“TsbfLHJltmkfq2Ucs6Ai”,“level”:3,“time”:“2024-07-17T21:41:17+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“no app in context”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“Could not create path "/user1/files/InstantUpload/Camera/2024/07/Test.txt"”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“path”:“/InstantUpload/Camera/2024/07/Test.txt”}}
{“reqId”:“TsbfLHJltmkfq2Ucs6Ai”,“level”:3,“time”:“2024-07-17T21:41:17+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“PHP”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test2.txt): Failed to open stream: No such file or directory at /app/www/public/lib/private/Files/Storage/Local.php#339”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“app”:“PHP”}}
{“reqId”:“TsbfLHJltmkfq2Ucs6Ai”,“level”:3,“time”:“2024-07-17T21:41:17+00:00”,“remoteAddr”:“xx.xx.xxx.xxx”,“user”:“user1”,“app”:“no app in context”,“method”:“POST”,“url”:“/remote.php/dav/bulk”,“message”:“Could not create path "/user1/files/InstantUpload/Camera/2024/07/Test2.txt"”,“userAgent”:“Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,“version”:“29.0.3.4”,“data”:{“path”:“/InstantUpload/Camera/2024/07/Test2.txt”}}

jtr · July 17, 2024, 10:35pm

Please check your Nextcloud Server log. If there is a permissions problem there should be hints there as to the cause. The client logs only indicate there is a problem, but won’t tell you anything in detail since it’s a server-side matter. There’s no need for us all to keep guessing.

I’ve also realised that there doesn’t seem to be any issue with uploading contacts daily.

I’m not 100% clear on where External Storage is entering the picture here. Where in Nextcloud is the Docker bind mount /home/xxx/shared:/shared mounted? Just at /InstantUpload/ as a Local type mount within Nextcloud?

As of a few days ago, other containers continue to read and write to the shared storage, however, it seems that NextCloud can only read.

You’re using the third-party LSIO image, not any of the project’s Nextcloud images, which introduces another variable because it has its own way of handling for ownership (via its PUID/GUID mechanism). I guess that could be a factor.

Do you know what changed a few days ago? Based on your container inspect output it appears the container was recreated on 7/15. Was that the time all this started?

jtr · July 17, 2024, 10:41pm

Based on the path, this is going to your /media/nextclouddata:/data mount, not your /shared volume. Is that what you expect?

You’d mentioned External Storage, but I don’t see any External Storage involved here.

ANCU · July 17, 2024, 10:48pm

Hello,

Thanks for your response.

I’ve posted from the server log about the same time you responded.

You’re right, the container was recreated on 15 July as part of troubleshooting the issue but the issue started around 10 July.

The data directory (/media/nextclouddata:/data) is external to the Ubuntu server.
It’s a shared folder on the Windows host.

ANCU · July 17, 2024, 10:49pm

Yes, that’s what I’m expecting and how it’s worked for years.

The data directory (/media/nextclouddata:/data) is external to the Ubuntu server.
It’s a shared folder on the Windows host. Hence it’s ‘external’ to Ubuntu.

jtr · July 17, 2024, 11:02pm

Indications here are this is indeed a file/folder permissions/ownership issue outside of Nextcloud. The “could not create path” is only occurring after a fairly basic file_put_contents() and touch() fails. Besides permissions, these types of things sometimes arise from weird filesystem issues from the different lower layers.

What happens if you run this command in your Nextcloud container:

php -r 'var_dump(touch("/data/users1/files/InstantUpload/Camera/2024/07/TestCommandLine.txt"));'

Make sure to test it while exec’d in to the container as the PUID you have configured in your Compose in order for it to be valid test. e.g.:

docker compose exec -uPUID bash

wwe · July 18, 2024, 8:08am

looking through you log snippet (btw please avoid editing log files using rich editor - all quotes are garbage in your message " is replaced with ”) I see the system fails to create the file

here I extracted one request:

{"reqId":"soklH42ZDfYRFRS6mlAj","level":3,"time":"2024-07-17T15:15:39+00:00","remoteAddr":"xx.xx.xxx.xxx","user":"user1","app":"PHP","method":"POST","url":"/remote.php/dav/bulk","message":"file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test.txt): Failed to open stream: No such file or directory at /app/www/public/lib/private/Files/Storage/Local.php#339","userAgent":"Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"29.0.3.4","data":{"app":"PHP"}}
{"reqId":"soklH42ZDfYRFRS6mlAj","level":3,"time":"2024-07-17T15:15:39+00:00","remoteAddr":"xx.xx.xxx.xxx","user":"user1","app":"no app in context","method":"POST","url":"/remote.php/dav/bulk","message":"Could not create path "/user1/files/InstantUpload/Camera/2024/07/Test.txt","userAgent":"Mozilla/5.0 (Windows) mirall/3.13.23.12-Win64 (build 20240708) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"29.0.3.4","data":{"path":"/InstantUpload/Camera/2024/07/Test.txt"}}

message part tells you:

file_put_contents(/data/user1/files/InstantUpload/Camera/2024/07/Test.txt)
Could not create path “/user1/files/InstantUpload/Camera/2024/07/Test.txt”

which clearly indicates the path doesn’t exists or is not accessible for the system (webserver user account). Likely this there some issue with the volume mount.

I would recommend you double check the mounts “chain” from your server into the container… I don’t need to say external storage adds useless complexity to the system I would recommend you use local storage for your bind mounts.

as you are using custom linuxserverio image - I would recommend you consult linuxserver.io Nextcloud docs and reach out to linuxserver.io Support

ANCU · August 4, 2024, 4:39pm

Hello again,

The underlying storage raid volume was undergoing a rebuild so I thought it best to let that complete before resuming troubleshooting (as a few comments have indicated that this may be a complication).

MegaRaid is now reporting that the volume is optimal again.

I’ve restarted the Container and the Ubuntu VM and the symptom remains the same from clients so I understand that the issue persists.

I’ll follow the steps you’ve recommended and let you know how it goes.

ANCU · August 8, 2024, 8:41pm

Command run:

php -r ‘var_dump(touch(“/data/users1/files/InstantUpload/Camera/2024/07/TestCommandLine.txt”));’

Result:

~$ sudo docker-compose exec nextcloud php -r ‘var_dump(touch(“/data/users1/files/InstantUpload/Camera/2024/07/TestCommandLine.txt”));’
no configuration file provided: not found

And indeed, the July folder doesn’t exist.

ANCU · August 10, 2024, 5:55pm

@jtr, any further guidance please?

(I didn’t mean to call in public but I can’t figure out how/don’t have permission to send direct messages.)

jtr · August 10, 2024, 9:00pm

That didn’t run the command.

Please run it more like this:

docker compose exec -u{PUID_from_your_Compose} nextcloud bash
php -r 'var_dump(touch("/data/users1/files/InstantUpload/Camera/2024/07/TestCommandLine.txt"));'

Be extra cautious about types of quote marks. They’re very important.

The data directory (/media/nextclouddata:/data) is external to the Ubuntu server.
It’s a shared folder on the Windows host.

So is it a CIFS volume mounted via /etc/fstab then?

ANCU · August 11, 2024, 11:01am

@jtr

Blockquote So is it a CIFS volume mounted via /etc/fstab then?

Yes, it is.
I can confirm that the volume remains mounted and can be browsed from the Ubuntu VM.
Nextcloud is also successfully saving contacts (.Contacts-Backup) to it.

I’ve confirmed (as thought) that the PUID is 1000.

I did this by running the id command.

Also, from inspecting my nextcloud container in Portainer

However, this is as far as I’m getting.

I think (at this stage) I’m likely not entering the commands quite right.

Please let me know what’s not right.

You can also see that after the permission is denied, I’m returned to abc@nextcloud (in Terminal)

Thanks.