I did notice that with a shared folder [where delete permission is granted on the folder], one is able to delete the files within the folder but not the folder itself.
In this sense it follows standard file system permissions where if one is given write access to a directory, they are able to delete files and directories within that directory, but not the directory itself.
Following a standard file system, one would never be able to give delete privileges just to a file itself. One would need to give write access to the directory containing the file.
So maybe this is by design. Just surprised no documentation supporting this [that I was able to find], and that no one has jumped in to say more or less the same.