Script "nc-apps" - a helpful and more verbose complement to `occ app:list`

ernolf · June 18, 2023, 5:31pm

occ app:list gives you a list of enabled and disabled apps but you only get the app-ids listed.
In the app store, however, apps are not sorted, searched and found by app-id, but by its name. These names are sometimes far away from the app-id, which is a pitfall, or at least a point of great confusion especially for newer admins. It’s easy to get one app mixed up with another, or unintentionally install two apps that are actually mutually exclusive.
There are a number of shipped apps that are automatically supplied with the server and its update. These apps are in turn divided into those that are always enabled, those that are enabled by default and those that can be enabled by the admin. This is not clearly visible with the provided tools.
After an update from e.g. 26 to 27, one like to use the app_install_overwrite function for some apps because you either already know that the app runs without problems or you just want to try it first. Once an app is included in the app_install_overwrite array, many admins simply forget about it and leave it there. However, you can also delete the app_install_overwrite array after the app has been installed and activated. Nevertheless, it is sometimes desirable to get an overview of which apps are actually supported by your server version, which min/max php version is required and which databases are supported etc.

To counter all this I wrote `nc-apps`

It is a program written in bash that is intended to close this gap(s).

It’s intentionally written in bash so that anyone can read and understand the code, maybe use parts of it for their own routines.

Once you get used to the wealth of information that this tool provides, you will hardly want to miss it.

It also gives you the ability to manipulate the version; If you are running Nextcloud 25, for example, you can tell the tool to simulate version 26 in order to see which of the apps already installed are suitable for 26 without any further updates. (~~It doesn’t show all updatable apps, which only become visible from the actually installed new version, but it still gives a good preliminary overview.~~) Because of the appstore integration, this filter is applicable on not installed apps too.

The tool also has the option of “cleaning up” an existing app_install_overwrite array (if present) in config/config.php, i.e. removing those app ids that are meanwhile supported by the current version and even without being listed in the app_install_overwrite array, can be deactivated and reactivated again.

The script can handle a configuration spread over multiple *.config.php files.

But what use a thousand words, it’s best to just try it out. The script can’t do any harm to your system, even if you want to clean up the app_install_overwrite array, it will first list everything that will be changed if you explicitly answer “yes”. Only then changes are applied.

Restriction:

This script is intended for use on conventional “barremetal” linux server installations or dedicated virtual machines. It is not tested with containerized solutions like snap, lxc, docker, aio etc.

Here you can read the code: https://global-social.net/script/nc-apps

and this is how you get the script installed:

sudo wget -qO /usr/local/bin/nc-apps https://global-social.net/script/nc-apps
chmod +x /usr/local/bin/nc-apps

Explanation:

the first line downloads the script into the directory /usr/local/bin
the second line makes it executable

You only need to install it once.
As all of my scripts, it comes signed and does an integrity check on the first run and checks for updates on every startup.

Once installed, you can simply run it

nc-apps

The help-screen will guide you through your various queries:

  nc-apps - version 2024-02-22 14:07 (latest version)

  Usage:
    nc-apps -h|--help
    nc-apps -H|--filters
    nc-apps install_overwrite [ -u [ --allow-unstable ]]
    nc-apps shipped [ *'section' ] [ --version=NC-VER ] [ --min_version=NC-VER ] [ --php=PHP-VERSION ] [ --nopager ]
    nc-apps not_shipped [ *'section' ] [ --version=NC-VER ] [ --min_version=NC-VER ] [ --php=PHP-VERSION ] [ --(un)supported ] [ --markdown ] [ --nopager ]
    nc-apps all_apps [ *'section' ] [ --version=NC-VER ] [ --min_version=NC-VER ] [ --php=PHP-VERSION ] [ --(un)supported ] [ --markdown ] [ --nopager ]
    nc-apps id=[pattern] [ --version=NC-VER ] [ --min_version=NC-VER ] [ --php=PHP-VERSION ] [ --(un)supported ] [ --markdown ]
    nc-apps name=[pattern] [ --version=NC-VER ] [ --min_version=NC-VER ] [ --php=PHP-VERSION ] [ --(un)supported ] [ --markdown ]
    nc-apps lang=[ISO 639-1]
    nc-apps updatable [ *'section' ] [ --version=NC-VER ] [ --min_version=NC-VER ] [ --php=PHP-VERSION ]
    nc-apps update [ *'section' ] [ --allow-unstable ] [ --php=PHP-VERSION ]
    nc-apps rebuild_db
    nc-apps integrity_check

  Options:

    *) 'section' can be eather one of:
                            enabled  - installed and enabled apps (default)
                            disabled - installed but disabled apps
                            local    - enabled and disabled combined
                            store    - not installed, available in appstore
                            all      - all sections

   -h | --help              this page

   -H | --filters           examples of JSON filters (for experts)

   install_overwrite [ -u [ --allow-unstable ]]
                            write actual 'app_install_overwrite' array in config/*config.php file.
                            If invoked with -u, it will first perform an update of al installed apps.
                            This script supports multiple *.config.php files.

   shipped [section*)]      list shipped apps (no 'store' section)

   not_shipped [section*)]  list not shipped apps

   all_apps [section*)]     list all apps

   name=[pattern]           list apps where name matches "pattern"

   id=[pattern]             list apps where id matches "pattern"

   lang=[ISO 639-1]         language to be used for translations instead of the detected locale on your computer, which is "de"

   updatable [section*)]    list updatable apps (no store section)

   update [section*)]       update enabled, disabled or local apps

   --allow-unstable         allow updating to unstable releases (only for update and install_overwrite with option -u)

   --supported              only list apps supported by version

   --unsupported            only list apps not supported by version

   --nopager                don't use pager (only for long listing functions)

   --markdown               list as markdown table code

   --php=PHP-VERSION        use another installed php version for updates. Possible (installed) PHP-VERSION: 8.1
                            example: --php=8.1

    *) NC-VER = only 2 digits

   --version=NC-VER*)       simulates an assumed nextcloud version instead of actual installed real nextcloud version.
   --min_version=NC-VER*)   simulates an assumed minimum nextcloud version. Defaults to 11
                            example: --version=26 and/or --min_version=25
                            '--(min_)version=' has no effect on install_overwrite, shipped and update

   rebuild_db               Delete database and create new from scratch.
                            Use this after updates, (de)installation of packages or simply when results start to get weird.

   integrity_check          Verify the integrity of this script with signature

Enjoy learning more about your apps.
ernolf

ibizaman · June 24, 2023, 3:45am

That is so useful! Thank you for sharing.

I’m always a bit worried about downloading an executable this way. Would you mind providing a hash of the executable so we can verify we downloaded the correct script? I can read bash but it’s long enough that a review is probably not enough to spot something is wrong.

Being able to simulate a version bump is a killer feature.

ernolf · June 25, 2023, 1:57am

I made an update with an integrated integrity check, the code now comes gpg-signed.

ibizaman · June 26, 2023, 5:29am

I had no idea you could make it self signed, that’s pretty neat.

ernolf · August 7, 2023, 4:12pm

I’ve greatly expanded the script in the meantime. Now it includes an integrated appstore browser/analyzer.

Since the beautified output is very slow with large masses of apps (currently 389 apps are listed in the Appstore), I added a (much faster) interface that outputs pure json.

The apps.json from apps.nextcloud.com, stored and maintained in data/appdata_$instanceid/appstore/ is used as the data source and filtered to the extent that only the translation of your locale (or the one passed as an argument) is displayed and only the latest version of an app.

I have created a help output that provides the most common filters as examples, so that not only command line junkies can get by with them:

  # Query single app "appname" by id:
    appstore '.[] | select(.id == "appname")'

  # Field contains (int)
    appstore '.[] | select(.phpMinIntSize | contains(64))'

  # Field contains ("string")
    appstore '.[] | select(.id | contains("string"))'

  # Field contains ("string") Case insensitive!
    appstore '.[] | select(.name | ascii_downcase | contains("string"))'

  # Field equals "string"
    appstore map(select(.id == "string"))

  # All apps where NCmax is greater than "27"
    appstore 'map(select(.NCmax > "27"))'

  # Apps with NCmax less than "26" and NCmin greater than "20":
    appstore 'map(select(.NCmax < "26" and .NCmin > "20"))'

  # All apps where phpMax is less than 8.2:
    appstore 'map(select(.phpMax < "8.2"))'

  # All apps where phpMax is less than 8.2 and phpMax is not empty:
    appstore 'map(select(.phpMax < "8.2" and (.phpMax | length > 0)))'

  # All apps where phpMin is greater than 7.0 and phpMin is not empty:
    appstore 'map(select(.phpMin > "7.0" and (.phpMin | length > 0)))'

  # Apps with phpMax less than 8.2 and phpMax is not empty, and phpMin greater than 7.0 and phpMin is not empty:
    appstore 'map(select(.phpMax < "8.2" and (.phpMax | length > 0) and .phpMin > "7.0" and (.phpMin | length > 0)))'

  # Apps with phpMax less than 8.2 or phpMin greater than 7.0 and either phpMax or phpMin is not empty:
    appstore 'map(select((.phpMax < "8.2" or .phpMin > "7.0") and ((.phpMax | length > 0) or (.phpMin | length > 0))))'

  # Apps with phpMax less than 8.2 and phpMin greater than 7.0, and both phpMax and phpMin are not empty:
    appstore 'map(select(.phpMax < "8.2" and .phpMin > "7.0" and (.phpMax | length > 0) and (.phpMin | length > 0)))'

  # Apps with a specific pattern in the name or id:
    appstore 'map(select(.name | test("pattern")))'
    appstore 'map(select(.id | test("pattern")))'

  # Apps with specific categories:
    appstore 'map(select(.categories | contains("category1") or contains("category2")))'

  # Apps with NCmax less than 26 and NCmin greater than 20, and phpMax less than 8.2:
    appstore 'map(select(.NCmax < "26" and .NCmin > "20" and .phpMax < "8.2"))'

  # phpExtensions is not empty
    appstore 'map(select(.phpExtensions | length > 0))'

  # Show only selected fields (id, name, and NCmax) for each app:
    appstore 'map({id, name, NCmax})'

  # in Combination
    appstore 'map(select(.NCmax > "26" and .NCmin > "20" and .phpMax > "8.1" and (.phpMax | length > 0) and (.phpExtensions | length > 0))) | map({id, name, NCmax})'

  # Sort by last modified, oldest first
    appstore 'sort_by(.lastModified)'

  # newest first
    appstore 'sort_by(.lastModified) | reverse'

  # in Combination
    appstore 'map(select(.NCmax > "26" and .NCmin > "20" and .phpMax < "8.2")) | sort_by(.lastModified)'

  # Apps created after January 1, 2023:
    appstore 'map(select(.created > "2023-01-01"))'

  # Apps last modified after January 1, 2023:
    appstore 'map(select(.lastModified > "2023-01-01"))'

  # Apps created before January 1, 2023:
    appstore 'map(select(.created < "2023-01-01"))'

  # Apps last modified before January 1, 2023:
    appstore 'map(select(.lastModified < "2023-01-01"))'

  # Apps that haven't been modified after January 1, 2023 (possibly "stale" apps):
    appstore 'map(select(.lastModified < "2023-01-01"))'

  # Apps created before January 1, 2023 and haven't been modified after that date (potentially "stale" apps):
    appstore 'map(select(.created < "2023-01-01" and .lastModified < "2023-01-01"))'

Professionals who want a json object to work with get pure (presorted) validated json with this script.

Besides updating apps and maintaining a custom install_overwrite array in my config.php (fully automatic), searching for single app informations like website or issue-tracker with the options name=“pattern” or id=“pattern” has proven to be extremely helpful in day-to-day use.

ernolf

Ghafla · December 21, 2023, 11:30am

Integrated the script into my nextcloud container (via Dockerfile +) and it throws this:

root@NCapp:/var/www/html# nc-apps 

     First run, checking integrity:
     ===========================================================================
gpg:                using RSA key 3EE43A84D2BA635A94E1033B0B145139A170715C
     ---------------------------------------------------------------------------
     - Integrity check failed. The script may have been modified or tampered with!
     ---------------------------------------------------------------------------
     You should run:

     sudo wget -O /usr/local/bin/nc-apps https://global-social.net/apps/raw/s/nc-apps

     to fix this.
     ===========================================================================

Of course I didn’t modify the script.
Any idea how to circumvent this?

ernolf · December 21, 2023, 12:09pm

That is because it needs gpg to verify the integrity.

You can circumvent the force integrity check by setting v=true in the second to last line.

But you are on your own! I did never test that script in a docker container!

Much luck,
ernolf

Ghafla · December 21, 2023, 9:18pm

gpg is installed along with the other dependencies xmlstarlet jq sqlite3 sudo less.
v=true helped get past the integrity check but somehow the directory structure of the container does not resemble the one you seem to expect in a bare metal installation:

root@NCapp:/var/www/html# nc-apps all_apps

  This step is required once after a new installation or sometimes after an update:

  Hello, this is the first run wizard of "nc-apps"
  enter the directory where your Nextcloud is installed (or [q] to quit):
  --> /var/www/html

  - first run wizard completed successfully.

ls: cannot access '/usr/local/bin/php[78].[0-4]': No such file or directory

no php with version number tacked:

root@NCapp:/var/www/html# ls -la /usr/local/bin/
total 5856
drwxr-xr-x 1 root root        3 Dec 21 22:06 .
drwxr-xr-x 1 root root        4 Dec 16 05:15 ..
-rwxrwxr-x 1 root root      122 Dec 16 05:12 docker-php-entrypoint
-rwxrwxr-x 1 root root     1449 Dec 16 05:12 docker-php-ext-configure
-rwxrwxr-x 1 root root     2669 Dec 16 05:12 docker-php-ext-enable
-rwxrwxr-x 1 root root     2963 Dec 16 05:12 docker-php-ext-install
-rwxrwxr-x 1 root root      587 Nov 27 23:12 docker-php-source
-rwxr-xr-x 1 root root    69459 Dec 21 22:06 nc-apps
-rwxr-xr-x 1 root root      817 Dec 16 05:15 pear
-rwxr-xr-x 1 root root      838 Dec 16 05:15 peardev
-rwxr-xr-x 1 root root      751 Dec 16 05:15 pecl
lrwxrwxrwx 1 root root        9 Dec 16 05:15 phar -> phar.phar
-rwxr-xr-x 1 root root    15245 Dec 16 05:15 phar.phar
-rwxr-xr-x 1 root root 18901488 Dec 16 05:15 php
-rwxr-xr-x 1 root root     3046 Dec 16 05:15 php-config
-rwxr-xr-x 1 root root     4535 Dec 16 05:15 phpize

A pity. Seemed promising.
Thanks for sharing anyway!

Edit:
The script could not be tricked into working by creating a symlink lrwxrwxrwx 1 root root 18 Dec 21 22:22 php8.2 -> /usr/local/bin/php either:

--------------------------------------------------------------------------------
command = "all_apps enabled", markdown = false, nopager = false, lang = 
Versions: assumed = 27.1.5, is = 27.1.5.1, min = 11.0.8, php = 8.2
--------------------------------------------------------------------------------
================================================================================
. . . . . . . . . . .  create  and  update  json  objects  . . . . . . . . . . .
  json  objects  created
================================================================================
 . . . . . . . . . . .  stocking  up  arrays  with  data  . . . . . . . . . . .
  all arrays stocked up
================================================================================

ernolf · December 21, 2023, 10:14pm

Thank you for your feedback!

That is a good one. I added support for multiple php versions, to enable handling apps with older php requirements (specially thought for the nearer future, when people start to switch to php8.3 and lots of apps still require php8.2).
The script needs php for some tasks, like reading config values from config.php etc.

I will look at this and create a fix for you next days.

ernolf

ernolf · December 23, 2023, 6:28pm

I updated it and it should work now without php$version

That does not mean, that it works inside of docker. You will have to test.
The script needs a tmp directory under /tmp to work!

ernolf

Ghafla · December 25, 2023, 6:23pm

…which it still doesn’t.
/tmp is owned by www-data:www-data so that should be no issue.

Anyway, thanks for trying to help here and Happy Holidays!

ernolf · January 12, 2024, 5:55pm

If you are planing to update your server to Nextcloud 28 and you want to monitor, if all of your installed apps are already supported and you don’t want to search in extensive tables like → here ← but you want verbose information, simply run:

Show, which of your installed, not shipped apps are NOT (yet) supported for Nextcloud 28, taking into account all installed and enabled apps supported by minimum Nextcloud 26:

nc-apps --version=28 --min_version=26 --unsupported not_shipped enabled --nopager

List all shipped apps for Nextcloud 28, taking into account all enabled and disabled apps:

nc-apps --version=28 shipped all --nopager

That will show you, that there are 48 shipped apps for nextcloud 28, while there are 49 for nextcloud 27. Then it is easy to compare and find out, that the files_rightclick app has stopped to be an app on its own, since it is integrated into the nextcloud files app as of Nextcloud 28.

Here some more random examples:

List all installed and enabled apps:

nc-apps all_apps enabled --nopager

List all updatable apps:

nc-apps updatable all

Update all updatable apps, allow to update to unstable (beta) app-versions:

nc-apps update all --allow-unstable

List all apps with the pattern “text” in App-Id:

nc-apps id="text"

and much much more documented and hidden features…

Write me, if you need more functionality,
ernolf

Script "nc-apps" - a helpful and more verbose complement to `occ app:list`

To counter all this I wrote nc-apps

Restriction:

This script is intended for use on conventional “barremetal” linux server installations or dedicated virtual machines. It is not tested with containerized solutions like snap, lxc, docker, aio etc.

Explanation:

Show, which of your installed, not shipped apps are NOT (yet) supported for Nextcloud 28, taking into account all installed and enabled apps supported by minimum Nextcloud 26:

List all shipped apps for Nextcloud 28, taking into account all enabled and disabled apps:

List all installed and enabled apps:

List all updatable apps:

Update all updatable apps, allow to update to unstable (beta) app-versions:

List all apps with the pattern “text” in App-Id:

To counter all this I wrote `nc-apps`