PROPFIND results for some files are not XML formatted?

ℹ️ Support
1

[details=“Support intro”]

Nextcloud version (eg, 20.0.5): 24.0.5
Operating system and version (eg, Ubuntu 20.04): FreeBSD 13.1-RELEASE
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.54
PHP version (eg, 7.4): 8.1

The issue you are facing:

I am trying to use rclone to mount my nextcloud user directory directly. This works, mostly. The problem I’m seeing is in the rclone logs it complains about a small subset of files saying they have illegal characters.

2023/01/07 22:02:07 ERROR : Drive/home/scott/machine_info/wannabe.house/system.txt.weeklu.18Dec-12PM.gz: Failed to copy: read metadata failed: XML syntax error on line 2: illegal character code U+001F

Digging into this, it seems like rclone is making a PROPGET request to the nextcloud DAV server and expecting a valid XML response. The response it gets back is NOT XML. I’m including a sample request and response inline.

Request:

PROPFIND /remote.php/dav/files/scott/Drive/home/scott/machine_info/wannabe.house/system.txt.weekly.18Dec-12PM.gz HTTP/1.1
Host: nextcloud.acknak.org
User-Agent: rclone/
Content-Length: 287
Authorization: XXX
Depth: 1
Referer: https://nextcloud.acknak.org/remote.php/dav/files/scott/
Accept-Encoding: gzip
^M
<?xml version="1.0"?>
<d:propfind  xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
 <d:prop>
  <d:displayname />
  <d:getlastmodified />
  <d:getcontentlength />
  <d:resourcetype />
  <d:getcontenttype />
  <oc:checksums />
 </d:prop>
</d:propfind>

Non-XML response:

Content-Length: 433
Cache-Control: no-store, no-cache, must-revalidate
Content-Encoding: x-gzip
Content-Security-Policy: default-src 'none';
Content-Type: application/xml; charset=utf-8
Date: Sun, 08 Jan 2023 05:58:58 GMT
Dav: 1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nextcloud-checksum-update, nc-calendar-search, nc-enable-birthday-calendar
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Referrer-Policy: no-referrer
Server: Apache/2.4.54 (FreeBSD) OpenSSL/1.1.1o-freebsd
Set-Cookie: XXX
Set-Cookie: XXX
Set-Cookie: XXX
Set-Cookie: XXX
Set-Cookie: XXX
Set-Cookie: XXX
Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
Vary: Brief,Prefer
X-Content-Type-Options: nosniff
X-Debug-Token: tyOKKmpcdLRqRlfQ6MsG
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: tyOKKmpcdLRqRlfQ6MsG
X-Robots-Tag: none
X-Xss-Protection: 1; mode=block
^M
^_<8B>^H^@^@^@^@^@^@^C}<92><DB>n<9C>0^P<86><EF><F3>^T<88><EB>^F<DB><C0>&[<C4>^REZ<B5><91><A2>^\<A4><AC>
z[^Y3^F+`[x<D8>C<9E>><A6><CB>6<CB>J<ED><8D><F9><C7><FE>=<F9>ݾk<83>-<F4>N^Y<BD>
YDû<E2>*<AF><B2>nhQ9<E4>8<B8><C0>#<DA>e<D5>*\<DF><FF><CA><C2>)t<AB><B0>A<B4>^Y!<8E><97>=T|ڝ^@#<issue-59
<FE>^Rf<A7>Ek<86><EA><82><D0>_<84><86>=ΐ<C2>ߡ^Gg<8D>v0<AE><9B>^^dAz<E8>^LBd|A"U^K<8E>8a^PɺW[ <8D><E9>`<
<DA><E8><B8>h<94><86><DF>JKCv\k^BԘ<C1>y<E0><E0>^P<BA>^H<F7>^X<ED>^@<DE><DB>CĖk^P<D7>,~}<8A>ꏜL<F5>|]<DB
<DB>Z<8F>s^M<D8>r<87><9D><A9><94>TP^Uo<83><FE>^V<B0>e<E0>%<82><98>Ʊ^_2<CA>2J<83><9F>O<9B>Q<EC>2<E1>issu
<A8>!<8C>F<D0>؂<AE><B1>)^X<BB>I<93>d<A2><E7>GG+<CC><D0>^K<C0><83>^E2<CF>^_<B7>
nm<AB>^DG<FF><8B>d^?]^?(;W<FA>^C<E5>Fd<A2>^A<F1><EE><86><CE>͢<E2><ED><E1><9E>e|<B1><90><E9>-<A4>RRq<C3><
<E2>tQR^<C6>    K<92><EF><89>,<A9>L^VR<DE><CA>e<9C><93><F3><D4>Y<E4>eɗQ<C7><FE>)^^6<9B>W<C2>"<E6>}<A1><
<C1><CB><E3>HL''<F8><DC><E1>K<B7>+<E5>l<CB>^O<9A>w<E3><CB><FF><A3><9E><D2>4x6^X<FC>0<83><AE><FE>Y<84><9
<9C><F7>^U<99>5zq<F5>   ^X<B8>^Z(^O^C^@^@

This response is abnormal. When I make a PROPGET request for a different filepath, the response is valid XML:

Sample “correct” XML response for a different file (truncated for bevity):

Cache-Control: no-store, no-cache, must-revalidate
Content-Security-Policy: default-src 'none';
Content-Type: application/xml; charset=utf-8
Date: Sun, 08 Jan 2023 06:01:05 GMT
Dav: 1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nextcloud-checksum-update, nc-calendar-search, nc-enable-birthday-calendar
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Referrer-Policy: no-referrer
Server: Apache/2.4.54 (FreeBSD) OpenSSL/1.1.1o-freebsd
Set-Cookie: XXX
Set-Cookie: XXX
Set-Cookie: XXX
Set-Cookie: XXX
Set-Cookie: XXX
Set-Cookie: XXX
Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
Vary: Brief,Prefer
X-Content-Type-Options: nosniff
X-Debug-Token: qvIC1e6PoT4dxvPdBCaZ
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: qvIC1e6PoT4dxvPdBCaZ
X-Robots-Tag: none
X-Xss-Protection: 1; mode=block
^M
<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns"><d:response><d:href>/remote.php/dav/files/scott/Drive/</d:href><d:propstat><d:prop><d:getlastmodified>Sun, 08 Jan 2023 05:57:30 GMT</d:getlastmodified><d:resourcetype><d:collection/></d:resourcetype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:displayname/><d:getcontentlength/><d:getcontenttype/><oc:checksums/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/files/scott/Drive/.backup_policy</d:href><d:propstat><d:prop><d:getlastmodified>Mon, 06 Jun 2022 22:29:21 GMT</d:getlastmodified><d:getcontentlength>107</d:getcontentlength><d:resourcetype/><d:getcontenttype>application/octet-stre
am</d:getcontenttype><oc:checksums><oc:checksum>SHA1:2100ebae0d5a023892d540082dc6b4dc4f9d37b4</oc:check
sum></oc:checksums></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:dis
playname/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:
href>/remote.php/dav/files/scott/Drive/18098%20N%20Shore%20Dr.%20Leavenworth%2c%20WA%2098826/</d:href><
d:propstat><d:prop><d:getlastmodified>Sat, 07 Jan 2023 17:29:27 GMT</d:getlastmodified><d:resourcetype>
<d:collection/></d:resourcetype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><
d:prop><d:displayname/><d:getcontentlength/><d:getcontenttype/><oc:checksums/></d:prop><d:status>HTTP/1
.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/files/scott/Dr
ive/2279%20Kira%20Ct.%20Toms%20River%2c%20NJ%2008755/</d:href><d:propstat><d:prop><d:getlastmodified>Sa
t, 07 Jan 2023 04:44:21 GMT</d:getlastmodified><d:resourcetype><d:collection/></d:resourcetype>.....

Both requests appear to succeed in the apache access logs (status 207):

10.0.0.223 - scott [07/Jan/2023:22:04:54 -0800] "PROPFIND /remote.php/dav/files/scott/Drive/.backup_policy HTTP/1.1" 207 397
10.0.0.223 - scott [07/Jan/2023:22:05:48 -0800] "PROPFIND /remote.php/dav/files/scott/Drive/home/scott/machine_info/wannabe.house/system.txt.weekly.18Dec-12PM.gz HTTP/1.1" 207 433

Neither request seems to produce anything in the nextcloud.log file.

Is this the first time you’ve seen this error? (Y/N): Yes

Steps to replicate it: Not entirely sure; I have copied about 24k files to this server and only a handful of them seem to exhibit this problem.

I am not including logs or configs unless someone has a question about a setting. The logs are too large; the request shows no output in nextcloud.log at all; I only see it in the apache access.log (see above).

If anyone has advice, a workaround or a suggestion about what other information would be useful to debug this please let me know.

Thanks!
Scott

3

Looking at this, the “non XML” response says it’s gzip encoded whereas the other one isn’t. The request indicates that gzip encoding is ok. Maybe this is a bug with rclone not nextcloud?

4

Sorry to reply to my own post but I guess I have to say “nevermind” now. I verified that when I set the --no-gzip-encoding flag on rclone it works fine. I suspect the “bad” XML was a gzipped response from the server and rclone has some kind of a bug where it verifies the XML before it decompresses it? Weird. I’ll leave this post up in case someone else runs into a similar problem. Sorry for the noise.