Photoshop PSD, PSB, TIFF large files previews

[rou]

Hi,
Recently I upgraded NC from version 20 to 25.0.4.1 in 5 consecutive updates. We use it to save and share large photoshop files, mainly tiff, psd and psb extensions. I use previewgenerator with the enablePreviewsProviders setup as follows:

array (
    0 => 'OC\\Preview\\Movie',
    1 => 'OC\\Preview\\PNG',
    2 => 'OC\\Preview\\JPEG',
    3 => 'OC\\Preview\\GIF',
    4 => 'OC\\Preview\\BMP',
    5 => 'OC\\Preview\\XBitmap',
    6 => 'OC\\Preview\\MP3',
    7 => 'OC\\Preview\\MP4',
    8 => 'OC\\Preview\\TXT',
    9 => 'OC\\Preview\\MarkDown',
    10 => 'OC\\Preview\\PDF',
    11 => 'OC\\Preview\\TIFF',
    12 => 'OC\\Preview\\Photoshop',
    13 => 'OC\\Preview\\PhotoshopPSB',
}

I created the PhotoshopPSB provider and inserted it into lib/private/PreviewManager.php:

$imagickProviders = [
                                'SVG' => ['mimetype' => '/image\/svg\+xml/', 'class' => Preview\SVG::class],
                                'TIFF' => ['mimetype' => '/image\/tiff/', 'class' => Preview\TIFF::class],
                                'PDF' => ['mimetype' => '/application\/pdf/', 'class' => Preview\PDF::class],
                                'AI' => ['mimetype' => '/application\/illustrator/', 'class' => Preview\Illustrator::class],
                                'PSD' => ['mimetype' => '/application\/x-photoshop/', 'class' => Preview\Photoshop::class],
                                'PSB' => ['mimetype' => '/application\/octet-stream/', 'class' => Preview\PhotoshopPSB::class],
                                'EPS' => ['mimetype' => '/application\/postscript/', 'class' => Preview\Postscript::class],
                                'TTF' => ['mimetype' => '/application\/(?:font-sfnt|x-font$)/', 'class' => Preview\Font::class],
                                'HEIC' => ['mimetype' => '/image\/hei(f|c)/', 'class' => Preview\HEIC::class],
                                'TGA' => ['mimetype' => '/image\/t(ar)?ga/', 'class' => Preview\TGA::class],
                                'SGI' => ['mimetype' => '/image\/sgi/', 'class' => Preview\SGI::class],
                        ];

lib/private/Preview/PhotoshopPSB.php:

class PhotoshopPSB extends Bitmap {
        /**
         * {@inheritDoc}
         */
        public function getMimeType(): string {
                return '/application\/octet-stream/';
        }
}

as the uploaded psb files has this mimetype. The thumbnails of the PSD and PSB files are created correctly through the ‘occ preview:generate-all’ command.

But the tiff files are not creating thumbnails.

The server is Debian 10, with php7.4 and php7.4-imagick extension, and ImageMagic 6 instaled on the system as well. I have increased quite a lot the setings of the /etc/ImageMagic6/profile.xml settings as files are huge, I mean, up to 3Gb some of them. PSD and PSB thumbnails are created quite fast, TIFF thumbnails are slow and I don’t see them after the process preview:generate-all finish.

I also created new config/mimetypemapping.json and config/mimetypealiases.json and run the
occ maintenance:mimetype:update-js
However I can’t make the thumbnails preview to open on lightbox. They just start downloading after I click on them.

I also installed CameraRawPreviews app to see if I could view tiff previews, but I don’t, as I think they are not created by the previewgenerator.

Any Help? I’m few days reopening old forum threads looking for solutions, but they just don’t work for me. I don’t know if my system has become unstable after updating from Nextcloud 20 to 25, or what could be happening.

As I said, PSD and PSB thumbnails are being created well, even the size of up to 3Gb of some of the PSB files. The problem seems to be on the TIFF thumbnails generation and then, the previews on the browser are not working on any of the three extensions.

[rou]

WIP (Work In Progress):

Edited apps/viewer/js/viewer-main.js by hand and inserted “image/vnd.adobe.photoshop” after “image/tiff”.

Added config/mimetypealiases.json with

{
        "application/x-photoshop": "image",
        "image/vnd.adobe.photoshop": "image",
        "application/octect-stream": "image"
}

Added config/mimetypemapping.json:

{
        "psb": ["image/vnd.adobe.photoshop"],
        "psd": ["image/vnd.adobe.photoshop"]
}

Run

sudo -u www-data php7.4 occ maintenance:mimetype:update-js

And now I can see the image previews for PSD and PSB. However, this change will be overwritten next time I upgrade. So:

Whats going on that the viewer-main.js doesn’t include the extensions of the mimetypemapping.json images that I have to edit the viewer-main.js directly?

[ernolf]

Hi @rou

I just want to comment on one detail:

@rou

application/octect-stream

The application/octet-stream MIME type is a binary file format used to indicate that a file is not in a specific format or does not have a standard format that can be recognized by a computer or software. The term “octet” refers to a sequence of eight bits, which are the building blocks of binary data.

In practical terms, the application/octet-stream MIME type is often used when a file contains data that does not fit into any other MIME type category, or when the specific type of data is unknown or cannot be determined. This MIME type is commonly used for executable files, binary data files, compressed files, and other types of files that are not meant to be interpreted by web browsers or other applications.

When a web server sends a file with the application/octet-stream MIME type to a client, the client is typically prompted to download the file rather than attempting to display it in the browser. This is because the browser does not know how to handle the file, and downloading it is the safest option.

For the server the mimetype defined in /config/mimetypemapping.json for the file extension is used. As long as nothing has been defined for a certain file, it is handeled as aplication/octet stream. Therefore, you should remove anything related to the mime type aplication/octet-stream. You don’t want all files with file extensions unknown to the server to be forwarded to imagick to create thumbnails and cause errors.

Finaly, when you add

"psd": ["image/vnd.adobe.photoshop"]

to config/mimetypemapping.json, you could break the preview generation for the .psd files, since, as you can see, it works internaly with application/x-photoshop.

When you made changes to config/mimetypemapping.json (adding, removing or changing mimetypes) you should not only run

occ maintenance:mimetype:update-js

but as well

occ maintenance:mimetype:update-db

I don’t want to claim that I’m 100% correct on all points, but I suspect problems with the points I named.

I think the best way will be, to make a feature (or pull) request anyway.

Happy hacking

[rou]

Hi @ernolf ,

Thanks for the clarifications, I didn’t know the second command to update the database of mimetypes, I think that could be the final solution instead of editing the viewer-main.js file directly.

As per the octect-stream, you might be right, but that is the mime tipe returned by the previewgenerator app when I added an echo on one of the functions executed to generate the thumbnails. So I added that into the aliases. However, I think this is wrong. And now when I run the preview:generate-all I get image/x-dcraw instead. It could be that I was missing some filesystem package as the server is mine and it is a fresh install of a Debian.

Anyway. The dirty hack to the viewer-main.js file + the modification of the PreviewManager to add a new provider seems to work as I need.

The thing should be more easy to add new extensions and providers to generate and preview image files.

Thanks.

[ernolf]

@rou

As per the octect-stream, you might be right, but that is the mime tipe returned by the previewgenerator app when I added an echo on one of the functions executed to generate the thumbnails.

First of all, you should use better analyzing-tools when hacking.

I made a tool to get verbose file information about any file at any moment:

It is based on the file_script app

It passes file information about selected files, say “file[i]”, gathered by the server (from a server side point of view) and gives it to a script, that echoes it back into a file “file[i].fileinfo.txt” in the same directory and in adition it gives information gathered by system tools such as file and mimetype (os point of view). Here an example of the output:

Give it a try:
Create a file /usr/local/bin/nc-get-fileinfo:

#!/bin/bash

echo "$(date +"%Y-%m-%d %H:%M:%S %Z")"

if [ $# -lt 10 ]; then
  digits=1
elif [ $# -lt 100 ]; then
  digits=2
else
  digits=3
fi

# iterate over the arguments
for (( i=0; i<=$#; i++ )); do
  # format the index with leading zeros
  index=$(printf "%0*d" $digits $i)
  # get the i-th argument
  arg=${!i}
  # echo the argument with its index
  echo "  -  arg $index = $arg"
  if echo "$arg" | grep -q "meta.local_path"; then
      lp="${arg##*= }"
  fi
done
echo


echo "information gathered by file:"
echo "  file                         = $(file -bL "$lp")"
echo "     -k, --keep-going          = $(file -bLk "$lp")"
echo "     -z, --uncompress          = $(file -bLz "$lp")"
echo "     -Z, --uncompress-noreport = $(file -bLZ "$lp")"
echo "     -i, --mime                = $(file -bLi "$lp")"
echo "     --mime-type               = $(file -bL --mime-type "$lp")"
echo "     --extension               = $(file -bL --extension "$lp")"
echo
echo "information gathered by mimetype:"
echo "  mimetype                     = $(mimetype -b "$lp")"
echo "     -a, --all                 = $(mimetype -ba "$lp" | uniq | sed ':a;N;$!ba;s/\n/, /g')"
echo "     --file-compat             = $(mimetype -b --file-compat "$lp")"

exit 0

(updated 08.03.2023; 19:15)

and make it executable.

Now go to the admin section of “File actions” (/settings/admin/files_scripts), create a New action, name it “Get Fileinfo”, paste this lua script into the code-window:

-- Process the files
local files = get_input_files()

local arguments = {}
for _, input_file in ipairs(files) do
  local output_filename = input_file.name .. '.fileinfo.txt'
  local out_folder = get_parent(input_file)

  -- Check if the output file already exists
  if exists(out_folder, output_filename) then
    abort('Output file "' .. output_filename .. '" already exists. Skipping...')
    -- Skip to the next iteration of the loop
    goto continue
  end
  local arguments = {}
  if is_file(input_file) then
-- only for analyzing purpose:
    local users = users_find(nil, nil)
    local num_users = #users
-- user invoking this actionscript
    local current_user = users_find(nil, nil)[1].uuid
-- to enable UID's with spaces:
--    local current_user = '"' .. users_find(nil, nil)[1].uuid .. '"'
-- finding out nextclouds installation dir with shell_command `pwd`
    local current_dir = shell_command('pwd').output
-- provides full path as seen by nextcloud
    local filename = "'" .. full_path(input_file) .. "'"
-- to enable filenames (input_files) with spaces:
--    local filename = '"' .. full_path(input_file):gsub('"', '\\"') .. '"'
-- meta_data part:
    local metadata = meta_data(input_file)
    local size = metadata.size or ""
    local mimetype = metadata.mimetype or ""
    local etag = metadata.etag or ""
    local utime = metadata.utime or ""
    local mtime = metadata.mtime or ""
    local can_read = metadata.can_read or ""
    local can_delete = metadata.can_delete or ""
    local can_update = metadata.can_update or ""
    local storage_path = metadata.storage_path or ""
    local local_path = metadata.local_path or ""
    local owner_id = metadata.owner_id or ""
-- tags part.
    local tags = get_file_tags(input_file)
    local assigned_tags = #tags
    local metadata_args = {
      "'found_users       = " .. num_users .. "'",
      "'invoking_nc_user  = " .. current_user .. "'",
-- obtained from shell_command('pwd')
      "'nextcloud_dir     = " .. current_dir .. "'",
-- full_path(input_file):
      "'filename          = " .. filename .. "'",
-- gathered from meta_data:
      "'meta.size         = " .. size .. "'",
      "'meta.mimetype     = " .. mimetype .. "'",
      "'meta.etag         = " .. etag .. "'",
      "'meta.utime        = " .. utime .. "'",
      "'meta.mtime        = " .. mtime .. "'",
      "'meta.can_read     = " .. tostring(can_read) .. "'",
      "'meta.can_delete   = " .. tostring(can_delete) .. "'",
      "'meta.can_update   = " .. tostring(can_update) .. "'",
      "'meta.storage_path = " .. storage_path .. "'",
      "'meta.local_path   = " .. local_path .. "'",
      "'meta.owner_id     = " .. owner_id .. "'",
      "'assigned_tags     = " .. assigned_tags .. "'",
    }
 -- Add each tag as its own string in the metadata_args array
    for i, tag in ipairs(tags) do
      local tag_str = "'  - tag-" .. i .. " details = id=" .. tag.id .. ", name=" .. tag.name .. ", user_assignable=" .. tostring(tag.user_assignable) .. ", user_visible=" .. tostring(tag.user_visible) .. ", access_level=" .. tag.access_level .. "'"
      table.insert(metadata_args, tag_str)
    end

-- Concatenate the metadata_args array into a single string with spaces between each argument
    local arguments_string = table.concat(metadata_args, " ")
    table.insert(arguments, arguments_string)
  end

  -- Construct the command string with quotes around each argument
  local command = "/usr/local/bin/nc-get-fileinfo " .. table.concat(arguments, " ")

  -- Run the command
  local result = shell_command(command)

  -- Create output file with the same name as the input file but with .fileinfo.txt extension
  local output_file = new_file(out_folder, output_filename, result.output .. "\n" .. result.errors)
  if output_file == nil then
    abort("Could not create output file: " .. output_filename)
  end
  ::continue::
end

(updated 08.03.2023; 19:15)

activate and save it.

Now you can go to whatever file you want in your cloud and you’ll find “More actions” in the context menu, when you choose “Get Fileinfo” and click “Execute”, the magic happens.

---

Now to your Issue. I got the same result without messing around in the codebase of the server, only withChanges to config/mimetypemapping.json

The problematic of your approach is that you don’t understand yourself, why one or the other works now. Messing around in the codebase of the server gives errors like “Some files have not passed the integrity check.” etc.

You should always first completely understand how things work together and then, when you want to change things, create a theme in the /themes directory and activate it.

To create previews of the .psb files, it only needed this line to be added to config/mimetypemapping.json:

	"psb": ["application/x-photoshop"],

thats all.

After you did occ maintenance:mimetype:update-js, you will see, that as soon as you change a file in your cloud to *.psb, and you look (with my analising tool) to the mimetype as seen by the server, you’ll see that it now is seen as “application/x-photoshop” which is automaticaly recognized to create previews. No aditional preview provider, no changes to the code in lib/private/PreviewManager.php, no aditional file lib/private/Preview/PhotoshopPSB.php, nothing.

My advice for further experiments:
Think thoroughly first and only make changes if you really understand how everything interlocks

Happy hacking

[rou]

Hi,

Good news to read… many thanks for your tool. I’ll give it a try ASAP. And also thanks for the clarification on the mimetypes and how to make it to work. This is the correct place to say “Less is more”…

Thanks…