Developing an app, with python module - is it ok?

đź’» Development app dev
1

Good day.
My friend and i decided to develop an app for finding duplicate media(image,photos).

I need it for my NC instance(have huge amount of duplicate photos/videos with different resolutions, sizes and formats). And we want to publish app on GitHub and in App NC store. We think many will like it to use it.
Frontend and backend will be written as recommended in guides, but we need a part that will be written in python.
And that part need access for our app DB tables(we need create 3 tables for this app). What is right way to do that?(using database)
App needs to supports minimum of 3 types of db(sqlite, mysql/maria and postgre) - so python module will read NC config, get type of db and connect to it.

Is it a right way?

I found only recommended ways to work with DB for php in guides. I can rewrite what I wrote in python in php, but it will be slower and I try to optimise code as much as possible.
Also a third way when php will load data from tables to pass that data to python looks like very ugly for me.

1 Like
2

Hi,

you’ll mostly have to explore this yourself. The Python dependency will mean your app will only run in some environments where the right Python with the right modules is installed.

You can use proc_open and similar to invoke a Python script from php, just as an idea.

Cheers

3

I don’t know much systems(and i know many linux distros, some FreeBSD clones and macOS) where is a problem to install python3.
And during app install we can do it automatically(or throw web interface), and install all module needs(there are only a few amount we need) in venv.

Thanks for your attention and reply. :slight_smile:

4

There is a python library for ownCloud (and worked/works with Nextcloud) that uses the Nextcloud webdav interface: GitHub - owncloud/pyocclient: ownCloud client library for Python
Perhaps some functions have to be implemented if needed (e.g. tagging). This way, you could use the webdav-interface and the NC-APIs and you don’t need to deal with the database.

5

You cannot directly access a shared sqlite with python, since sqlite doesn’t use a server. The others, yes. Those will be fine.

However to be honest, I think you’d be better off leaving the database access in nextcloud/php, and passing data between them.

I’m not trying to talk you out of using python, but what is the reason you need it? Curious.

6

Thanks, forgot about that sqlite “feature”.

Leaving access to files/db from php and pass all data from php to python seems a bit overhead.
Then it will become an app, I wish not to use myself. And then someone can write app in full php and it will be better.

If you interested why we choose to use python this is a brief number of reasons:

  • Better support for different hashing methods(whash, dhash)
  • Future support of CNN networks, most of such libraries written in Python
  • Admin will be able to restrict python script, for max cpu load/memory consumption or even host it on another machine if needed, php will commutate with it with REST API.
  • Python script will be able to work in two modes:
    a) As service, processing user tasks, with/without unloading hashing data(8/32/128 bytes for each unique image(depends what option selected), and if it is video then multiply value by 2-3) That will be useful for instances like mine, with currently 10k+ photos.
    b) As on demand start to perfom task and quit
    That will lead to maximum of speed and customisation. I dont know how to write in php first variant, I am not so good at php to write the best code for this situation.
7

Oh, ok, so this isn’t actually a plugin for nextcloud, its a completely separate system that shares the database. Gotcha.

8

We are close to alpha release, need some GUI improvements and adjustments, doc writing for admins and finish video processing part - maybe in two-three weeks we will finish it, as alpha version and publish on Github.
Based on the results of, it will be a nextcloud plugin, not a separate system.
OS where we are testing it: clear install of Centos, Debian and Ubuntu and WSL(without pip and otherm only python that ships with).
Mysql,Maria,Postgre support.(in future maybe we add Oracle if needed)
Requirements will be: Python >= 3.6 and it is all.
All other modules will be auto installed locally(if user prefer this method) or user can follow guide and install all it manually.
Trying to make it maximum user friendly and easy to use.

P.S: a question about nextcloud-core - do we need to check if file is locked? we properly handle situation when file is invalid/damaged - and we prefer to skip those files. Currently we don’t do that check for lock - do we need it? Files are open only for read - can it cause some unknown issues?

9

We finished first version of it. If someone will be interested, here is out repo:
https://github.com/andrey18106/mediadc

2 Likes
10

Thanks for the app. and more importantly, opening the doors of Python to Next Cloud.

An off topic question though,
I want to synch my self hosted Kanboard with nextcloud tasks.
Any idea how can I do that?
PS. Not a professional developer is talking here.

11

I haven’t any idea for that, i am not familiar with nextcloud app named tasks. Maybe, if you will provide some more info about this…
I just don’t understand for what reason you need python to integrate Kanboard? =)
Did you tried this app? [https://apps.nextcloud.com/apps/external]

Currently we are working on nextcloud python framework, and will rewrite our MediaDC app after that.
Framework will allow:

  1. Call from php python modules with arguments and return value in json.
  2. Automatically install required python modules by app if they have packages on pip, if not then gcc compilers must be installed on system(that requires sudo access) and they will build from sources.
  3. Provide API for Python for NC FS access(list, read, write, create, delete) (in alpha state already, almost finished) (not a webdav)
  4. Provide API for Python for NC DB access(not done/tested yet, in process)
  5. Framework will install Python 3.9.9, but there will be ability to specify own interpreter for admins(currently we are working on this, in a few days will finish if there will be no more bugs during tests)
  6. There will be option in feature(not in a first version) to run python module for app on separate machine. Steps 3/4 are able to work through unix sock or network socket. Read File access will be slower in this case.
  7. Very nice configuration page, and no need in sudo access to install something, ability to do silent auto-install install in most cases.
  8. It will run on Debian/Centos family and currently we fighting with musli linuxes to support from box Alpine family without glibc. Arm 64bit CPU’s will be supported from box, too.
    And for some reason no one installs nextcloud on Mac, but it works on Mac too…

We can’t say yet how fast it will be, php has too many restrictions for threadings, but what we have tested it looks promising.

12

Currently we are working on nextcloud python framework, and will rewrite our MediaDC app after that.

Is there any news regarding this framework for python?

13

Development is not going very fast, current plans have changed.
In a few days, a new beta version of MediaDC will be released without the need to install python or its dependencies (and support for snap), and if its reviews are good and it works for everyone, we will think further about how to expand this so that others can write applications in python.

1 Like
14

I’m curious to see the result! will you eventually upload it to the app store?

15

Oh, man, I’d love to write my app in python and integrate it with nextcloud.
I suppose there are others too but don’t know about this topic here.

Regards.

16

Good time of day.

Good news:

We were able to start working on framework again, here are some spoilers:

  • first version is coming soon(4-8 days, imho)
  • we were able to build MediaDC that works without installing python dependencies(works on SNAP(only for NC25+)), beta currenly is present in our repository releases, sources temporary unvalailable(i am rewriting code it to use framework).
  • with the next release of MediaDCC(0.3.0) - i prepared repos to allow other build similar app and packing them with Nuitka.
def process_directory_images(dir_id: int, task_settings: dict) -> list:
    dir_info = db.get_paths_by_ids([dir_id])
    file_mounts = []
    if dir_info:
        file_mounts = get_mounts_to(dir_info[0]["storage"], dir_info[0]["path"])
    fs_records = db.get_directory_data_image(
        dir_id, task_settings["mime_dir"], task_settings["mime_image"], file_mounts
    )
    if not fs_records:
        return []
    ignore_files = get_ignore_flag(fs_records)
    apply_exclude_list(fs_records, task_settings)
    sub_dirs = extract_sub_dirs(fs_records, task_settings["mime_dir"])
    if ignore_files:
        fs_records.clear()
    dc_process_images(task_settings, fs_records)
    if fs_records:
        db.increase_processed_files_count(task_settings["id"], len(fs_records))
    return sub_dirs

new code that uses API from framework:

def process_directory_images(directory: FsNodeInfo, task_settings: dict) -> list[FsNodeInfo]:
    fs_objs = fs_list_directory(directory["id"])
    fs_apply_ignore_flags(fs_objs) # .noimage, .nomedia
    fs_apply_exclude_lists(fs_objs, task_settings["exclude_fileid"], task_settings["exclude_mask"])
    sub_dirs = fs_extract_sub_dirs(fs_objs)
    fs_filter_by(fs_objs, "mimepart", [mimetype.IMAGE]) # we want only images
    process_images(task_settings, fs_objs)
    if fs_objs:
        increase_processed_files_count(task_settings["id"], len(fs_objs))
    return sub_dirs

Bad news:

  • Andrey18106 has currently problems with the electricity due to air strikes.
  • When we publish 0.3.0 MediaDC and before publishing the second version of the framework (there will be support for writing files from python), we need to know if this hard work will be benefited(in cash) or not. We want o ship second release of the framework with Apache Thrift technology, and implementing it, writing all the CI & CD tests, and documentation is really hard work for full time.

With best regards, Alexander Piskun, bigcat88@icloud.com

1 Like
17

thank you so much for your hard work on this. Sending all my support to Andrey

1 Like