Hub 4 pioneers ethical AI integration for a more productive and collaborative future

Originally published at: https://nextcloud.com/blog/hub-4-pioneers-ethical-ai-integration-for-a-more-productive-and-collaborative-future/

6 years ago, Nextcloud was just a file-sync-and-share solution similar to Dropbox.

Today, we are a Content Collaboration Platform giving you a complete online productivity platform with integrated features from Talk, Office, Groupware and hundreds of apps.

Every day, you choose us to be your Content Collaboration Platform, and every year we provide you with product releases that make you more productive.

Today, we are excited to announce a major step forward with Hub 4 – the very first on-premises collaboration platform to integrate intelligent features across its applications.

“Nextcloud Hub 4 is the first collaboration platform to integrate intelligent features comprehensively across its applications, while addressing the privacy and control challenges that come with AI technologies.”

Frank Karlitschek, CEO and Founder of Nextcloud

The Nextcloud Hub 4 release comes at an exciting time, with the recent breakthrough of ChatGPT and AI technology being at the epicenter of mainstream news and consumer interest.

Now more than ever, businesses need to learn how to leverage the way they work with AI technologies to increase their productivity and collaboration.

At the same time, we are all too aware of the many failings of AI. Be it related to discrimination, privacy, ecological footprint or the generation of just plain nonsense, the challenges are numerous.

Our society is just beginning to comprehend the potential, impact and limitations of AI in the work world, however, while we digest this powerful technology, we’ve chosen to embrace it and take it a step forward, while taking our ethical obligations to you seriously.

Integrating AI into Nextcloud’s collaboration platform is going to transform the way you think, work and collaborate with others.

That is not all

But there is much, much more. We introduce beta apps for Talk on the desktop and Virtual Files on Mac OS. We introduce recording and breakout rooms in Talk, file attachments in Calendar, advanced versioning in Files, the new Tables app, we bring the Notes app into Nextcloud, make Deck more collaborative – the list is long. Please, read on, share your thoughts on this release with us in the comments or on social media and tell friends, colleagues and family that it is possible to have a great collaboration experience WITHOUT giving your data to Big Tech!

Intelligent features: Introducing the ‘Smart Picker’

The best way to introduce our new, intelligent features is through the Smart Picker – a cross-application feature enabling users to use AI (and much more) in their daily work flow.

For the first time, users have the opportunity to work with AI in a personal and collaborative way.

To activate the Smart Picker, just type a slash (/) in the chat bar and you’re instantly given a wide selection of options.

With the Smart Picker, you can talk to, create with and work faster thanks to AI.

Talk to AI: Speech-to-text

Sometimes, it’s easier to say something out loud. With Nextcloud’s speech-to-text feature utilizing Whisper via Replicate, you can formulate text through your voice. Use your spoken words to send a message in a Nextcloud Talk chat, in an email and through many more applications.

Example of using AI Speech-to-text (by Whisper via Replicate) in Nextcloud Mail

Create with AI: Image generation

Your cloud is full of photos, and we’ve now made it one step easier to access them in a flash. With AI image generation, you can type in keywords to find the image you’re looking for. Insert an AI generated image into a Talk chat, Notes and much more.

Example of using AI image generation (by OpenAI’s Dall-E 2 and StableDiffusion) in Nextcloud Talk

Nextcloud gives you two options to generate images: the online Dall-E 2 and an on-premises StableDiffusion based generator.

Work with AI: ChatGPT text generation

Recently, all the rage has been centered around the novel ChatGPT, so we’ve decided to integrate its relative right into Nextcloud. ChatGPT, which uses GPT-3 technology by OpenAI, is known for crafting intelligent, professional text and our version is now readily available to use in Nextcloud Talk, Office, Mail and more.

Use the Smart Picker to generate fresh text by giving the tool instructions on what you want to know. For instance, in Nextcloud Talk, ask a question that will help you construct a message to the team you are managing.

The GPT-3 text generation delivers 1 result by default, however the advanced settings allows for as many results as you want and you can even choose the model of data to use for optimal results.

Example of using ChatGPT text generation (by OpenAI) in Nextcloud Office

Smarter collaboration with the Smart Picker

Now that we’ve introduced the AI features and how they work, let’s talk more about the Smart Picker. The Smart Picker was created to enhance your workflow and to better collaborate. It is conveniently integrated across all your Nextcloud applications resulting in seamless collaboration.

The Smart Picker is available in Nextcloud Talk, Deck, Mail, Notes, Text, Collectives, Office and other applications.

In addition to the AI speech-to-text, image generation and ChatGPT text generation, you can use the Smart Picker across all applications to:

  • Share tasks from a Deck board, card or comment
  • Share Nextcloud Tables
  • Share a map location (by OpenStreetMap)
Example of using map location (by OpenStreetMap) in Nextcloud Text
  • Share a knowledge database items from Collectives pages
  • Share videos from PeerTube
  • Create Text templates or pre-defined text snippets
Example of using a Text template in Nextcloud Mail
  • Share GIFs (by Giphy)
  • Share Mastadon people, toots and hashtags
  • Share Movies, series and people (by The Movie Database)
Example of Movies, series and people (by The Movie Database) AI feature
  • Share Files
  • Share any link
  • Share GitHub issues, pull requests and comments
  • Share GitLab repositories, issues and mergers
  • Translate with Nextcloud Translate (via DeepL)
Example of the Translation in Nextcloud AI feature

The Smart Picker was designed in a way to be easily expandable, so we look forward to many more apps on the store that can help you insert smart information into your documents, mail and more!

Ethical AI Rating

We thought long and hard about the ethical issues with AI. As you might remember, our previous releases did all AI related work on the server. We did our very best to find ways to classify images, recognize faces and recommend you files without leaking any data to a third party server.

But AI is moving fast, and we do want to give you the option to take advantage of innovations from third party companies like Deepl, OpenAI and others.

We can’t fix all problems in the world, but we can give you a choice, and provide transparency. So, to address the obvious privacy and control challenges that come with artificial intelligence, the AI-powered features are transparently classified with Nextcloud’s Ethical AI Rating. The Ethical AI Rating is based on the availability of code, model and training data so you can make informed choices. We share all about the conditions, classifications and methodology of the Ethical AI Rating in our blog.

Tables App: Alternative to Microsoft SharePoint


Use the Smart Picker for Tables in Nextcloud Text

Use the Smart Picker for Tables in Nextcloud Text

Example using Tables: Vacation Requests

Example using Tables: Vacation Requests

Creating a table

Creating a table

List of your tables

List of your tables

Nextcloud Tables is an on-premises, open source SharePoint alternative that lets customers track data that is relevant to their organization, such as expenses, projects or sales leads.

It enables customers to build, manage and work with data structures, putting together easy ‘apps’ and connecting them to various other components in Nextcloud for automation. The flexibility of Tables allows users to tailor the table to their specific needs.

“The initiative by Nextcloud to offer a digitally sovereign, open-source alternative to Microsoft SharePoint is commendable. That is why we are working with Nextcloud to optimize Nextcloud Tables.”

Ralf Sutorius, Senior IT Architect, Stadt Köln (City of Cologne)

Hub 4 Product Improvements

Files, Talk, Groupware and Office are Nextcloud’s key products and each have many exciting improvements and updates!

What’s more, all products are integrated with the new Smart Picker with AI collaboration that will transform the way you work.

Nextcloud Files

Advanced Versioning

In Nextcloud Files, we kept older versions so you never lose the original text or one in between. With Hub 4, we’re taking it to the next level giving users more control over the smart versioning mechanism in Nextcloud. It is now possible to name a version, so you can easily go back to the original copy or one created by a colleague at a certain date and time.


Name the version

Save version name

Files settings

Nextcloud will keep versions using a time based algorithm. This keeps less precise versions over time, for example, keeping 1 version per minute, 1 per hour, 1 per day and so on. The system also acts on available file storage, cleaning up older versions when storage gets low. But named versions are always kept!

💡 AI component: Recommended files are shown at the top of your files list and you can share them based on old sharing behaviors.

Desktop client & mobile apps

The Files desktop client has a major release on the way, however a preview version is out now of virtual files on MacOS. As for the upcoming version, a default browser selection is introduced, as well as the ability to create accounts from the command line. That is of course very useful in large organizations.

As for the Android files app, document scanning with multi-page support to PDF and PNG is now available.

The iOS Files app delivers improvements to document scanning, image editing before uploading, and office document editing improvements.

Nextcloud Talk

Jump start conversation in breakout rooms

Breakout rooms are excellent for diversifying large meetings and virtual learning environments.

In this release, Nextcloud provides full functionality to set up a breakout room, manage it, send messages and much more.


Configure breakout rooms

Manage breakout rooms

Send message to a breakout room

Send message to all breakout rooms

Record and track every call

For everyday work calls, there’s now a recording option with a time watcher. Easily keep track of time in a call with the time watcher and never be out of the loop with session recording. After the call, simply send the recording in a chat and it will pop-up for all participants to see and utilize.


Start a recording

Start a recording

Recording started

Recording started

Send recording in a chat

Send recording in a chat

Nextcloud Talk reduced loading times, server load and notification delays by up to 99%.

More in Talk

🎮 Several administrator features to control permissions

🔊 Conversation ownership transfer

📱 A dedicated iPad interface

💡 AI component: Use the Smart Picker in Nextcloud Talk for Nextcloud Dictate (speech-to-text), Nextcloud Translate (with DeepL), Dall-E (image generation), ChatGPT technology and much more!


Smart Picker view in Talk

Smart Picker view in Talk featuring the AI image generator with Dall-E & StableDiffuse technology

AI speech-to-text

AI image generator with Dall-E & StableDiffuse technology

Image generated in Talk example

Nextcloud Groupware

Shared mailboxes and S/MIME support

We also introduce shared mailboxes so your enterprise or organization can be more efficient and better organize its teams and workflow.

To improve suitability for large enterprise and government organizations, the Mail App in Hub 4 introduces S/MIME (Secure/Multipurpose Internet Mail Extensions) for digitally signing and encrypting messages.

S/MIME is a security protocol that ensures secure email communication so your information is not compromised.

💡 AI component: Use the Smart Picker in Mail for Nextcloud Dictate (speech-to-text), Nextcloud Translate (with DeepL), Image Generation (Dall-E), ChatGPT technology and much more!

Smart Picker Mail highlights

  • Text templates – excellent for recurring sales or support messages

Email text templates

  • GIF picker (by Giphy) – to lighten the mood in an email chain
  • Movies, series and persons (by The Movie Database) – Be part of the movie conversation!
Example of using ChatGPT text generation (by OpenAI) in Nextcloud Mail

Other improvements in Groupware

  • Support for attachments when making an appointment in Calendar
  • XOAUTH2 support in Mail to improve interoperability with Outlook and Gmail

💡 AI component: The Smart Inbox in Mail is getting improvements, training the local neural network also on mail subjects to be able to better recommend you important emails.


Attachment support when making an appointment in Calendar

Attachment support when making an appointment in Calendar

Nextcloud Office

E2EE improvements

End-to-end encryption is designed to keep a sub-set of super confidential data safe, even if a server breach occurs. Imagine one of the hundreds of universities that use Nextcloud. The collaboration between students should be overseen by teachers and system administrators. But the HR department, dealing with very sensitive employee information, should be able to work on documents without even system administrators being able to view them. This is what End-to-End encryption makes possible. We made two improvements in this release.

First, we introduce sharing. Before, after picking a folder for end-to-end encryption, users could work in this folder and access it from all their clients, mobile and desktop, but not share it. It is now possible for the HR team to share their folders with confidential dossiers with each other!

You may already use File Drop – the Nextcloud feature that allows your external personnel, clients and partners to easily upload files to a secure cloud folder through a hyperlink.

With the new Nextcloud Hub 4 release, File Drop now includes the option to upload files to End-to-End Encrypted folders for increased security in your organization.

Receive confidential files in a secure, trustworthy way without worrying about breaches. It’s simple, secure and a no fuss file exchange.

Going back to the HR department – the team can now send job applicants a link where they can securely upload their resume’s directly in an end-to-end encrypted folder, shared in the HR team.

New improvements with Text and Collectives

Nextcloud Office has improved integration and performance, document compatibility and a reworked UI.

Real-time cursor tracking

While editing a document, you can now see others who are working on the document too in real-time. As they write and move about the document, you’ll see their name highlighted on their cursor. Watch the additions and corrections others make to better help you work on a shared document.

Collectives

In Nextcloud Office, the knowledge base application Collectives improves the creation of new collectives and moving pages between them. It also links collectives to Files, allowing users to find the files that make up a collective and the collective associated to files more easily.

💡 AI component: Use the Smart Picker in Nextcloud Office for Nextcloud Dictate (speech-to-text), Nextcloud Translate (with DeepL), Dall-E (image generation), ChatGPT technology and much more!

Improved text & Smart Picker in Deck

The project management app Nextcloud Deck now also uses the Nextcloud Text functionality for editing the text on cards. This makes it easier for users to add rich content and supports the new Smart Picker.

Deck also features live editing of boards by multiple people, exporting boards to CSV files and creating new tags when assigning them.

Smart Picker in Deck

More in Office

📹 Embedding videos

✒️ Font management

☑️ Watermarking official documents

Performance improvements

Performance improvements

In this release, the loading of folder mounts and container handling has been improved by up to 20%. OCS API requests are up to 3x faster and Nextcloud Talk in particular reduced loading times, server load and notification delays with up to 99%.

New Developer Program

As an extension of this release, Nextcloud is introducing a new developer program to super-charge application development. The Nextcloud App Store has already over 300 apps in circulation, however there are countless more apps we’d love to offer and we need your help.

You can work with AI right now

Microsoft just announced its AI tool Copilot for cross-application collaboration, Google its AI in Workspace, but they are only available for a few select customers. Nextcloud’s Smart Picker with AI features is available right now for you to talk to, create with and work with.

And, where Microsoft fired it’s Ethical AI team, we are taking our responsibility serious. We do our best to provide you on-premises solutions where possible, and if that is not possible – we make it clear when a AI solution leaks data to third parties, is not built on an open data set, or not open source.

Let AI assist you in writing a more captivating message, let your words drive your next email or just engage with it to save time on mundane tasks.

This is your Nextcloud. Your opportunity to use AI. Let it work for how you want it to, without compromising your privacy.

Upgrade to Hub 4

Nextcloud Hub 4 is available immediately and ready for use.

This release supports the latest PHP 8.2, making it ready for deployment on newer Linux distributions and taking advantage of the improvements in this release.

Get Hub 4

4 Likes

Nice update, just missing a lot of documentation. E.g. why is call recording not showing up in my talks? I saw in the admin settings, that one could add a recording backend there. But what’s that? Where do I get it from? How does that have to be configured?

Everything sounds great. Instantly done upgraded from 25… but… where are these Features, especially the “Smart Picker” and AI stuff? Missing it totally, or are these only “Enterprise Features”?

Thanks for getting clear…

Find them at Integration - App Store - Nextcloud

3 Likes

You need high performance backend and a recording server

2 Likes

mmh okay… sorry,… now i get it… after a big announcement i have to figure out that i have to install other apps and also have to figure out which ones and everything else…

like configs, requirements, etc…

not really helpful, and particullary not well documented.

ironic mode off

but thanks anyway for that hint.

To give the benefit of the doubt here, it’s always been the case that big new features are not automatically enabled when upgrading, else people who didn’t want them would have to set them up as part of the upgrade, only to find and turn them off again.

I hope these new features are either turned on or easy to find and turn on for new installations, and I do agree it would be nice if each feature headline had a link to some setup documentation.

I see integrations for OpenAI and Replicate, and the readme for each simply says it has a place to enter an API key. I’m assuming each has some kind of a disclaimer in place, telling users that this is a third-party API. Am I correct?

Being able to use self-hosted models would be great, but as mentioned in the quote, might not yet be possible to set up across Nextcloud’s many supported environments. I just hope it’s made sufficiently clear that this it is a third-party service being used for now.

1 Like

Hey,
During the presentation of NC Hub 4, there was a mention about Talk desktop app (Finally, yeeeeehy!!! :smile: ), but it’s dissappeared from the release page. :frowning: nooooo!
What’s up with that? Did it got puled at the last minute? Where can I find it?

1 Like

It’s still in development, but you can download preview releases from GitHub: GitHub - nextcloud/talk-desktop: 💬💻 Nextcloud Talk Desktop Client Preview

3 Likes

Thank you!

2 Likes

Are there plans to support other AI as well, like BLOOM https://bigscience.huggingface.co/ for example? Or is it just ChatGPT because it was easy to integrate and there was a feel like “it has to be there”? :slight_smile:

1 Like

there are some non-concrete ideas floating around, but with the smart picker we also tried to make it super super accessible for anyone to develop the integrations that they need, so maybe you feel like giving the integration with bloom a try yourself? If you have questions about the coding we will try to answer them in the developer section of this forum :slight_smile: You can find a tutorial for the smart picker at Develop for Nextcloud

1 Like

I’ve forwarded your feedback to the writer of the blog to consider!

I get that AI is a hot topic right now, and I see the appeal, but personally I have no interest in these new features. I wish Nextcloud would make the basics (file-syncing) flawless before adding so many extra features. For me, Nextcloud is almost useless while issues #2467 and #2786 are still open.

Nextcloud is awesome and I really respect what they are doing. I hope I don’t sound too much like an entitled free-software user.

3 Likes

100% agree.

AI may be great for specific tasks (help with programming, write basic advertising texts, generate funny pictures…), but I am a bit sceptical if integrating it in Nextcloud will add much benefit (which outweights the effort to keep it under control).

And about other problems: I can add 2 problems:

  • “Rich workspaces make file list jump around” - posted it yesterday in this Forum, but don’t really expect any answer.

  • Edit an image in the browser: Just wanted to rotate a jpg by 90° - made the browser tab (Firefox) crash … ok I don’t use that edit feature any more, no problem.

2 Likes

So don’t use them.

My impression is that most of these issues occur in combination with the External Storage App, and I think many of these issues would disappear if people would use an appropriately sized server with local SSD storage. But trying to integrate an existing SMB server, by mounting who knows how many SMB Shares into Nextcloud via External Storage app, possibly even on rather poorly sized hardware, will almost certainly lead to issues, especially if many users are accessing the same files on the same SMB shares.

1 Like

Thank you for your feedback. I am not using external storage though. I have a large ZFS pool on the same physical system where I am hosting Nextcloud. The person who submitted that issue is also not using external storage. I am not sure where this is coming from. I am also pretty much the only user of my server, so no issues with multiple users accessing the same file. Now, it could be multiple machines logged into the same account, but this is why I installed Nextcloud: to synchronize files between multiple machines. My point with my original post is that while I love Nextcloud I wish they would spend a little more time making their bread a butter rock solid.

Ok, not sure then. I use the Nextcloud Desktop client since ever, and I rarely had any issues. There were occasional sync conflicts, but I never experienced any major issues that caused thousands of conflicts or even data loss, as experienced by other people. Maybe I was just lucky.

I don’t think this is an either or question. The people who mainly work on the desktop client are mostly different people than those who work on the A.I. features. The developer community is quite large around Nextcloud, so the fact that A.I. features are beeing added will most likely not conflict with the development of the desktop client, or vice versa.

2 Likes

It’s not so easy, every new feature requires additional maintenance.
So if there are long running bugs or real missing features for which there are already proposed solutions it’s IMO not really productive to introduce even more things.
You can of course be of a different opinion :slight_smile:

1 Like

Any information on setting up the recording server? I have not been able to find any information about it, just the entry in the settings under talk. I do have a HPB, is there a way to enable it there? I did upgrade to the latest signaling server.