Upcoming changes in the server repository for compiled assets

💻 Development
1

Dear Nextcloud contributors, as you may know, we commit compiled assets on the server repository.
We do this for various reasons, and we’re aware that many of you rely on this by directly using the server repository for quick test setups and end-to-end tests.

However, compiling assets on pull requests has proven problematic - like it often causes conflicts in pull requests,
requiring you to rebase, recompile, and re-run the CI. To fix this,
we’re moving asset compilation out of the pull request workflow and into a dedicated workflow that runs after a pull request is merged.

We are aware that this might require some adjustments on your side, so to avoid disrupting you too much,
we will roll out these changes with a grace period.

We plan to deploy the new process at the beginning of August.

What changes for you as a developer

You no longer need to compile assets on your pull requests, so the overall contribution process should feel smoother and less cumbersome.
We will add a workflow to check that no compiled assets are included, so once we switch to the new process, you’ll only need to commit your source changes.

What changes for you as a user

If you directly use the server repository for whatever reason, there might now be a very small time gap where sources and compiled assets are out of sync.
So if you use the server repository - e.g. for end-to-end tests - you may need to compile it yourself, or wait for the post-merge compilation to finish.

Once we’ve switched over to the new process, you can check whether assets are still compiling here:

Future plans

Once we’ve deployed the changes in server, we will evaluate the results. If the feedback is positive, we might also adjust other repositories that currently enforce assets to be compiled within pull requests.

2

Hello and thanks for the information @susnux!

May I ask a few small questions for clarification:

  1. The devs will check in only the sources. The CI will build the corresponding compiled artifacts, also in the long time plans (if evaluation tells nothing different). So, there are no plans to completely drop the artifacts from the repo at the moment?
  2. Again for clarification: Every server commit should contain a valid set of assets. These might be somewhat outdated compared to the sources in the server but nevertheless fully functional. If I had waited 5 more minutes, the CI build would probably have terminated and my checkout would have had the newly built assets included. Correct?
  3. Are the CI Jobs already in place or at least in a PR present? I would like to have a look to understand what is going on :wink:
  4. Is there going to be some documentation on this? When does it make sense to update existing documentation like the tutorials (maintained by @edward) and the dev docs on docs.nextclout.com?
  5. You mentioned a grace period. This is just for communication when this is going live, correct? There is no kind of soft start.
  6. Will this short time desync of sources and assets affect only the master branch or also stable branches (especially with backported features)?
  7. Just put of curiosity: what might/will happen if two PRs are merged in quick succession?

Sorry for the questions. I did not meant to interrogate you. Just seen this and seen some changes coming towards me as I am one of the users of the repo, e.g. during tests in the CI and for local development. So, I might need to adjust and find a way to ensure a reproducible and stable operation.

Cheers
Chris

3

Sure @christianlupus ! Let me try to clarify your questions:

So, there are no plans to completely drop the artifacts from the repo at the moment?

No there are no plans to do so at the moment. We are aware that they cause the repository to grow very big, but for now we have no plans to remove them from the repository.

If I had waited 5 more minutes, the CI build would probably have terminated and my checkout would have had the newly built assets included. Correct?

Lets say you clone the server repo or pull from the main branch, then there might be a case where a PR has been merged that updated the frontend but the assets are not yet compile (we expect a delay of max 5 minutes). Usually this is not a problem but in rare situations this can be a problem like for example backend now requires a password-confirmation but the frontend is not yet compiled so running that code will not work for that API call.

We try to monitor this for any issues, e.g. apps using the server repo directly for e2e tests.
In such cases a mtime check dist vs src/** could help to decide if you need to wait / compile yourself. But also as mentioned one could just check if a compile job is running using Github API.

Are the CI Jobs already in place or at least in a PR present?

Not yet in place. Its the same as used already in the text repository.
There is a draft: ci: compile on CI after merge to reduce conflicts and total CI time by susnux · Pull Request #60589 · nextcloud/server · GitHub

Is there going to be some documentation on this?

For app development nothing changes - except if you use the server for e2e tests.
For server development we will - of course - adjust the documentation.
If you have any specific documentation in mind which might be outdated now, please go ahead I would be interested!

You mentioned a grace period. This is just for communication when this is going live, correct?

Yes, its so that you are aware that the server repository might change this and you might have to check if you are using it directly in any place. So that you are not surprised by the change as we highly value our community contributors :slight_smile:

Will this short time desync of sources and assets affect only the master branch or also stable branches

It will only affect master for now, but with stable35 it will then also be used on stable branches. Of course we will make sure any tagged commit for (pre-)releases have all assets compiled!

(especially with backported features)?

This will become even better with this change! Currently if you backport things they will often not work because /backport can fail when trying to apply your changes due to conflicting assets. With this new compile-after-merge you only need to backport sources meaning less likely to conflict.

Just put of curiosity: what might/will happen if two PRs are merged in quick succession?

  • PR1 and PR2 got merged in that order.
  • PR1 merge triggers a compile (C1)
  • C1 starts running
  • PR2 merge triggers a compile too (C2)
  • If C1 is still running C2 will cancel C1 and compile assets for both change

This has the advantage that the number of “compile assets” commits will be reduced meaning the overall repository size will grow slower than with the current approach.

Sorry for the questions. I did not meant to interrogate you.

No worries! Questions are fine, thats why I posted it here so you can clarify all of them :slight_smile:

I might need to adjust and find a way to ensure a reproducible and stable operation.

If there are any problems, please reach out to use, e.g. for direct communication you can also use the community chat on our instance.

4

Hello @susnux,

thanks for this change, taking compilation out of the PR loop removes a lot of daily friction.
I would even advocate going one step further: not committing compiled assets at all, and putting the compiled directories behind .gitignore.

The post-merge model fixes the conflicts, but it keeps the underlying oddity that build output lives in version control. That is where the remaining machinery comes from: the desync window on master, an extra asset commit after every merge (with the repository growing forever), a CI check that has to police PRs for accidentally committed assets, and special rules for stable branches and tags. All of that exists only to manage artifacts that git was never meant to hold.

The classic argument for committing them is that a git checkout must be runnable without a build toolchain. I think that argument has quietly expired: master currently requires Node 24 and npm 11, which hardly any distribution ships today. So “runnable from checkout” already depends on a very specific toolchain, and the real problem was never the compilation itself, it was setting up the toolchain.

That problem has a simple solution: build in throwaway containers. I maintain ncmake, a single generic Makefile for Nextcloud apps. Its concept, which I have spent a good amount of design work on, is strictly DRY: everything is derived from the repository itself, the app id and version from appinfo/info.xml, the PHP image from the declared min-version, the Node image from engines.node in package.json. composer and npm run in disposable podman or docker containers, the host needs neither PHP nor Node, and uid/gid mapping keeps the produced files owned by you. The same make build behaves identically on a laptop, on a test server and inside GitHub Actions, so local builds and CI cannot drift apart. My apps use this one Makefile for building, dist tarballs, deploying into a live instance or an AIO container, App Store publishing and the release workflow itself.

Applied to server, the same concept would mean: compiled assets are ignored, and make builds them from any commit with exactly the Node version the repo declares, in a container. The desync window disappears, the post-merge asset commits disappear, the new CI check becomes unnecessary because there is nothing left to police, and testing a PR no longer requires a local Node 24 setup. Tagged releases would of course keep shipping precompiled tarballs, produced by the same make target developers use. The one honest cost: running master from a bare checkout then needs podman or docker (or a matching local Node). In CI that cost does not even exist, GitHub’s own ubuntu-latest runners ship both podman and docker preinstalled, and self-hosted runners (like nextclouds ubuntu-latest-low) can be provisioned the same way. Daemonless, rootless podman is a particularly good fit for throwaway build containers, it adds next to no overhead. For the audience that actually runs master, that seems like a fair trade.

The “derive everything, repeat nothing” approach also covers a neighbouring chore: server#62407 updates eight workflow files by hand from the organization templates. In my app repos this is already automated end to end: a small GitHub App runs ncmake’s make workflows-update, which refreshes the managed workflows from their upstream templates (nextcloud/.github plus ncmake’s own) and opens a PR when something changed, leaving locally modified workflows untouched. twofactor_oath#52 shows what that looks like. No repo needs its own dependabot on .github/workflows for this, dependabot already watches nextcloud/.github, and the updater propagates from that single source. The same would work at organization scale.

ncmake is still evolving and more targets are planned, but the app side of the concept is running in production today. If there is interest I would be happy to prototype a server profile: I have already sketched what it would need, repository type detection, containerized npm ci and build through build/demi.sh, and an assets status target for the post-merge model. ncmake is MIT licensed, so if Nextcloud would rather adapt or copy the idea than depend on my repository, that is perfectly fine with me.

ernolf

5

Thank you for your insights @ernolf !
Seems like you have quite some nice tools created for automation :rocket:

The classic argument for committing them is that a git checkout must be runnable without a build toolchain. I think that argument has quietly expired: master currently requires Node 24 and npm 11, which hardly any distribution ships today.

That would be more of an argument for keeping the assets, because then you would not need to compile yourself :wink: But for Node we recommend since a couple years to use tools like NVM or volta so you are not bound to your distributions provided versions and also can easily switch between versions so you can work on different stable branches on the same time.

That problem has a simple solution: build in throwaway containers.

This is basically how our release process we use for our non-shipped apps, and some community apps, works. Versions are fetched from package.json and appinfo and then used by our building tool (krankerl) to build and publish the apps from CI.

The desync window disappears, the post-merge asset commits disappear, the new CI check becomes unnecessary because there is nothing left to police, and testing a PR no longer requires a local Node 24 setup.

We are aware of this, but there are still some reasons for us internally to keep the compiled assets in the repository.
This is a first step to reduce the frictions when contributing to Nextcloud, we will further evaluate how we can improve the process afterwards.

server#62407 updates eight workflow files by hand from the organization templates. In my app repos this is already automated end to end

For apps this is also already automated in the Nextcloud GitHub organization.
Meaning any app within the organization can just use this workflow to automate the update and even allow applying patched for customization:

But the server is not an app, the repository layout and requirements are completely different (e.g. for installing the proper PHP version etc).
So we only reuse some of the workflows and heavily adjusted custom ones.

6

Thanks for the pointers @susnux! I know krankerl, ncmake actually grew out of wanting that idea in a strictly DRY form that behaves the same on my laptop, on a test server and in CI, with no dedicated build tool to install, GNU make is already on every system. I spent several years professionally writing nothing but Makefiles, so that route was simply the one closest to me.

The patch reapply mechanism in sync-workflow-templates.yml I was not aware of, and I will happily adopt that idea. My updater serves repos outside the organization, where the templates need some rewriting on the way in anyway, org runner labels like ubuntu-latest-low do not exist out there.

Understood on the internal reasons, this first step already removes real friction, and my offer stands should the further evaluation ever point toward toolchain-free builds.

Cheers, ernolf

7

Me too and I found myself regularly rejecting the PRs and manually merging the stuff. Will elaborate on this. :smiley: