File Upload: Performance improvements and bug fixes

💻 Development
1

Hi All,

I also ran into a few problems lately, all related to file upload. For some, I have found references (eg, the 504 error assembling chunks):

  • Chunk assembly error
  • Slow uploads of the WebUI (only over http/2)
  • Total unresponsiveness of the server (for other users/requests) while one request is finishing the upload (assembling chunks) of a big file.
  • Very slow assembly of chunks.

The details of my installation are not so important here I think. Maybe one observation: It only happens frequently and reproducibly on a server set up with http2 (h2) connecting to an apache 2.4 backend also accessed over http2 (h2c), and with an internet connection of 500 MBit/s.

I have a question here in the hope that some core / files backend developer might read this:

Can you point to the right files on the front- and backend code that handle chunked uploads? I’m an experienced PHP dev, so yes, I’ve searched a bit the code, stumbling upon several probably involved classes and interfaces, but it would be super great if someone already involved could just share a quick list of the involved JS and PHP files / event model in the WebUI / DAV (chunked) Upload flow. I’d give it a look and possibly contribute fixes / ideas (I actually do have a few ideas, but not plenty of time).

Thanks in advance and Best Regards,

Lorenzo

2

Might help you to get started with the chunks code. Happy coding :sunglasses:

3

Thanx a lot! Actually, From that thread I found a few cross-linked threads brought me to the already present fix: php occ config:app:set files max_chunk_size --value 0
Disabling chunked uploads makes all uploads faster. Of course this also effectively disables resuming an upload.

But for some set ups - ironically, particularly those with big files and high bandwidth - it practically solves most upload performance problems. Of course this means tuning the whole proxy-backend-chain to higher timeouts and a high LimitRequestBody value (apache). But yeah, I can live with that!

I just hope that disabling chunked uploading remains in place for the time being. A better solution for resuming would indeed be using Range HTTP headers. But this also means saying bye-bye to the old multipart/form-data format for uploading (from whichever client).

Best,

Lorenzo

Best,

Lorenzo