Windows illegal filename characters - can't find the github bug report

alexanderdd · June 7, 2021, 9:42am

Hello,

pretty sure that a bug report on github already exists, but now I could not find it, also not here on the forum. Can someone help me find it?

steps to reproduce:

in webinterface, create file with * in filename (or other filename characters that are illegal on windows)
file will not be synced to Windows machines because the character is illegal

Thank you.

Btw these are illegal characters for filenames in windows:

  < (less than)
  > (greater than)
  : (colon - sometimes works, but is actually NTFS Alternate Data Streams)
  " (double quote)
  / (forward slash)
  \ (backslash)
  | (vertical bar or pipe)
  ? (question mark)
  * (asterisk)

wwe · June 7, 2021, 12:38pm

but what is your expectation? In my eyes the

one solution would be to rename the file on the client… but then you have two different files… or you have to rename it on the server as well
or you apply limits to file names on the server already (must exclude invalid characters on all supported clients). what should happen if the client tries to upload a file valid on this OS but invalid on server/other clients…

I think the problem must have been reported already or maybe addressed from the beginning by software architecture. your report shows the problem exist at least in some cases and could result in data loss. in my eyes it’s worth creating new issue to have discussion and better documentation.

Larry_Boyd · June 15, 2021, 6:44pm

Yeah, not really sure what you are expecting to be done about this. Its a limitation of the substandard OS you have chosen to use, not a limitation of nextcloud.

alexanderdd · June 16, 2021, 8:35am

Expectation is to limit allowed characters in Nextcloud to those allowed in all three OSes Win/Mac/Linux by default.
Maybe have an option to allow all UTF8 characters, including a warning that this will lead to sync problems on Windows machines.

Reason: Nextcloud is a file syncing solution. Windows is the most popular OS at the moment. Nextcloud claims to use sane defaults. So the defaults should not create problems in syncing to Win machines.

…Still hoping that someone can point me to the issue on github…? I’ve seen it before but can’t find it anymore

bb77 · June 16, 2021, 9:14am

I would second that. Because even when you not are using Windows for your self or in your organisation, you probably want to share files with people outside your home or organisation. Chances are high that these people use Windows. Default settings that take this into account would definitely be a useful thing.

Larry_Boyd · June 18, 2021, 3:40pm

No it isn’t. Android (Linux) is.
https://gs.statcounter.com/os-market-share

RuudschMaHinda · July 8, 2021, 11:46am

If you find the issue on github, please let me know. I have quite a few “special” users, who repeatedly use characters that shouldn’t be used - after two emails already, and after several windows machines stopped synching because of this. (Very heterogenous client environment)

wwe · July 8, 2021, 6:48pm

if you don’t find any issue on github just open one. maybe it worked good enough till now - so may it’s worth to give the devs a hint to existing edge cases you experience in real life… if they don’t fix them at least we got a reference and documentation on expected behavior.

RuudschMaHinda · July 9, 2021, 9:20am

Issue has been created as a feature request:

github.com/nextcloud/server

Disallow illegal characters / ? < > \ : * | " and so on in file and folder names

opened 09:19AM - 09 Jul 21 UTC

RuudschMaHinda

0. Needs triage enhancement

Nextcloud allows characters in file names and folder names, which are illegal in… some operating systems, or all operating systems. For some reason users in one of my installs use Asterisks, some use colons, I even found pipes and question marks, let alone quotation marks, and ampersands. Nextcloud itself should not allow naming files or folders with those characters in them. Windows clients basically refuse to work if a folder name has a colon in them. Renaming said folder stops synching alltogether because windows can't handle it. I suggest to not allow those special characters when creating files and folders within the webinterface. Files and folders with those characters in the sync folder should be renamed and said character removed before it gets distributed to other clients on different OSs. To be honest I don't even know how some users manage to get some of these characters in there. It's not been the first time, and I just wrote a third email about this issue to this team. They are very learning resistant.

RuudschMaHinda · August 9, 2021, 11:37am

I still maintain that this is something that must be addressed. Until then, I created a workaround for myself - maybe this helps others as well, so I will post it here - as well as on github:

Using Flow External Scripts I created the following rule:

When file is created, renamed, copied
and User is not member of Temp-Ban-Group
Run script
rename 's/[\?\<\>\:\@\*\|\$]/_/g' /path/to/datafolder/%n && php /path/to/nextcloud/occ files:scan %o

This replaces any of the offending characters with an underscore, and then rescans the userfolder. This can take a while depending on the amount of files and folders. It also depends on the hardware. In my case I had to install rename with apt install rename.

This also seems to be working with group folders.

Edit: This does NOT work with folder names - only files! Flow currently can’t be triggered by folders being created/uploaded/renamed.

RuudschMaHinda · October 23, 2021, 12:59pm

Welp, the windows client now at least gives error messages about illegal characters. I also gathered some commands and installed “rename”. Will write a script with these commands that triggers every night.

The commands I have gathered (also includes some legal characters):

find /path/to/data/ -name '*\?*' | rename -v -n 's/[\?\!\<\>\:\@\*\|]/_/' #replaces ? with _
find /path/to/data/ -name '*\:*' | rename -v -n 's/[\?\!\<\>\:\@\*\|]/_/' #replaces : with _
find /path/to/data/ -name '*\\*' | rename -v -n 's/[\?\!\<\>\:\@\*\|]/_/' #replaces \ with _ -- might be useless?
find /path/to/data/ -name '*\**' | rename -v -n 's/[\?\!\<\>\:\@\*\|]/_/' #replaces * with _
find /path/to/data/ -name '*\>*' | rename -v -n 's/[\?\!\<\>\:\@\*\|]/_/' #replaces > with _
find /path/to/data/ -name '*\<*' | rename -v -n 's/[\?\!\<\>\:\@\*\|]/_/' #replaces < with _
find /path/to/data/ -name '*\"*' | rename -v -n 's/[\?\!\<\>\:\@\*\|]/_/' #replaces " with _
find /path/to/data/ -name '*\|*' | rename -v -n 's/[\?\!\<\>\:\@\*\|]/_/' #replaces | with _
find /path/to/data/ -name '*\!*' | rename -v -n 's/[\?\!\<\>\:\@\*\|]/_/' #replaces ! with _
find /path/to/data/ -name '* ' | rename -v -n 's/ *$//' #deletes trailing spaces
find /path/to/data/ -name ' *' | rename -v -d -n 's/ *//' #deletes leading spaces
find /path/to/data/ -name '*.' -type f,d | rename -v -n 's/.$//' #deletes trailing periods
sudo -u www-data php7.4 /var/www/nextcloud/occ files:scan --all &&  sudo -u www-data php7.4 /var/www/nextcloud/occ files:scan-app-data

Observe the -n after the rename command … this makes this a test run! Once you’re happy with the shown result, remove the -n.
Also the rename commands need to be cleaned up still for just the characters searched for?

Some commands must be run several times until all - for example - trailing periods have been deleted. i.e. “foldername with open sentence…” needs the trailing periods command to be run three times. Haven’t worked out the script yet.

alexanderdd · March 30, 2023, 9:24am

The issue was closed by the stale bot a year ago, but imo it is still relevant and not solved.

Can someone please reopen and triage the bug Disallow illegal characters `/ ? < > \ : * | "` and so on in file and folder names · Issue #27891 · nextcloud/server · GitHub ? Or does someone know who to ping so it can be triaged? What info is still needed (label needs info)?