[Discussion] Why is contact syncing so . . . convoluted?

[summersab]

@dugite-code

My understanding is that DAVx5 is a syncadapter and is what “adapts” webdav communications into an android addressbook device side.

So, my understanding of SyncAdapters is completely different (and I could be completely wrong). From Create a sync adapter  |  Connectivity  |  Android Developers

The sync adapter component in your app encapsulates the code for the tasks that transfer data between the device and a server. Based on the scheduling and triggers you provide in your app, the sync adapter framework runs the code in the sync adapter component.

From what I gather, SyncAdapters are like cron jobs + some sort of data transfer protocol (I have yet to see an example of that format). They provide asynchronous data transfer and updates between an app and a cloud service, not a means for apps to communicate with the Android system. It is what Nextcloud is currently using to sync and update files between the device and the Nextcloud instance.

The API call that DAVx5 and other apps use to interact with the address book is ContactsContract. That is what allows contacts to be created, read, updated, and deleted by third-party apps.

My suggestion is to eliminate the need to use CardDAV protocol on Android since it is non-native. Instead, use the native SyncAdapter API call (protocol?) to send the data to the server. Then, since Nextcloud uses CardDAV (and since it is FAR easier to implement the CardDAV protocol in PHP), create a “bridge” between what is received by the server-side SyncAdapter listener and Nextcloud. That way, when data is received, it will be converted it to vCard format and sent to Nextcloud over CardDAV (via a localhost connection).

This keeps everything native to the respective platforms, would likely integrate much more seamlessly, and wouldn’t break any existing functionality. Let me know if I’ve misunderstood how these APIs and protocols work - since I only just learned about the ones on Android last week, it’s entirely possible.

[dugite-code]

From what I gather, SyncAdapters are like cron jobs + some sort of data transfer protocol

Yes that is my understanding. But you still need to translate said data into something that can be used. You need to do this with any API, For example how would android know what to do with a json response? At the very least you need to dump it into a database for later use.

It is what Nextcloud is currently using to sync and update files between the device and the Nextcloud instance.

Yes the Nextcloud app uses Webdav in order to sync files, the same (extended http) protocol DAVx5 uses to get your contacts

The API call that DAVx5 and other apps use to interact with the address book is ContactsContract. That is what allows contacts to be created, read, updated, and deleted by third-party apps.

Ok, but you don’t communicate with a remote API the same way you communicate with a device side API. There are a few considerations like data sanitation, security concerns and general efficiency that prevent doing that.

My suggestion is to eliminate the need to use CardDAV protocol on Android since it is non-native.

The Webdav protocol is just and extended HTTP request, nothing non-native about that. WebDAV - Wikipedia Even if you used a Rest API you would still need to tell Android what to do with it. So why not use a good web standard.

create a “bridge” between what is received by the server-side SyncAdapter listener and Nextcloud.

Saberdav (aka Nextcloud Webdav library) is the “SyncAdapter listener” as you call it. There is no such thing as a server-side “SyncAdapter listener”, SyncAdapters are only an Android thing. Google has it’s own SyncAdapter talking to their proprietary contact back end then they translate that data into Android contacts. That said they may still be using Cardav themselves, I haven’t checked if that’s still available for a few years.

This keeps everything native to the respective platforms, would likely integrate much more seamlessly, and wouldn’t break any existing functionality.

This is actually the whole point of using a Web standard like Webdav. All data translation is done device side natively where things can change often, the Webdav standard is well defined and been around a long time so is a stable target. The DAVx5 project has years of work built in dealing with all the changes Android has had, along with the device specific fixes, hence why the Nexcloud dev’s wouldn’t want to re-do all the work already done. Even if you didn’t use Webdav you would still need to process the data on Android’s end anyway, you would just be re-doing all the existing work for little to no benefit.

As always this is based on my best understanding, I’m only a novice coder

[summersab]

No, I get what CardDAV/WebDAV are (though admittedly, I have no direct experience utilizing them). My comment about CardDAV being non-native to Android was the fact that it provides no interface for it - something like DAVx5 is required. As you said in another message, it’s a complicated app. Since that’s the case, why do the conversion to vCard and use a CardDAV app on the client? just in order to say, “We sent the data via CardDAV - huzzah?” That’s silly and complicated.

What format does a ContactsContract call use to dump contracts/what format does a SyncAdapter use when sending data? Is it just JSON? Something else? Why not send the data to the server in the format that Android provides and then convert/sync it using CardDAV server-side? My whole thought is to avoid having to use a third-party app just for the simple task of syncing contacts whilst maintaining adherence to CardDAV. just use the most default protocols provided by Android and offload all of the difficult stuff to the server.

[dugite-code]

@summersab

simple task of syncing contacts

I think this is the main misunderstanding. Syncing data as varied and extensible as contacts is NOT simple. You also have to deal with edits, deletes and potentially thousands of contacts, that’s a lot of technical problems to deal with.

@summersab

why do the conversion to vCard and use a CardDAV app on the client? just in order to say, “We sent the data via CardDAV - huzzah?” That’s silly and complicated.

Because CardDAV is a Standard specifically designed for this task. You would basically just be re-inventing the wheel if you wanted to do something else and that would be even more complicated, mainly because you would have to write your own specification.

@summersab

Why not send the data to the server in the format that Android provides

Typically how data is presented by internal API’s is not conclusive for transport over the network. When you sync over the network you want to do it as lightly as possible by using UID’s and other things rather than dumping a large blob of data back and forth. Something you deal with by using a Webdav client

@summersab

just use the most default protocols provided by Android and offload all of the difficult stuff to the server

There is no default protocol. So I’m not sure what you mean by this? DAVx5 is just doing http requests through the SyncAdapter, this isn’t anything unusual.

[summersab]

Well, how do other apps do it without requiring a third-party app like DAVx5? Maybe we should start there. Dozens if not hundreds of apps are able to do this in a way that is seamless and straightforward for typical end-users. You can tell me till you’re blue in the face that the majority of Nextcloud users are technically-competent and prefer F-Droid, and I’ll respond by saying that if you want to appeal to a broader user base that extends beyond such people, things like this really need to be addressed. So, how might it be addressed? I put forth a (potentially half-baked yet well-meaning) suggestion. Got any ideas?

[anon9582441]

Perhaps a middle way is to build a link between DAVx5 and the Nextcloud client so that it automatically can find the user’s account? This is how some of the other Nextcloud apps work.

[summersab]

@anon9582441

Perhaps a middle way is to build a link between DAVx5 and the Nextcloud client so that it automatically can find the user’s account? This is how some of the other Nextcloud apps work.

Which apps? I’m curious.

I’d think that the preferable solution would be to avoid the need for DAVx5 altogether if possible, but anything is better than nothing . . .

[anon9582441]

I think that Nextcloud Talk and SMS/phone sync does it.

Here is how it looks like in Talk. See the Import… link?

[summersab]

Good thought, but it says that it’s pulling contacts from the Nextcloud app . . . which is using the convoluted DAVx5 setup that I think needs to die. It’s not connecting directly to Android’s contact storage or using its own management system.

I did a search of the code and couldn’t find any calls to ContactsContract or DAVx5, but I did find where it is connecting to the local Nextcloud app to pull contacts. So, that’s unfortunately not too helpful.

On a side note with Phone Sync, I would love to see it be able to send texts much like how https://messages.android.com works. Turns out that the KDE Connect team is almost done with their own standalone SMS syncing app. I tried to revitalize a feature request for Phone Sync over here:
https://github.com/nextcloud/ocsms/issues/4

Maybe I should start a new thread about this? It’s #2 on my Nextcloud wish list.

[anon9582441]

I’m not sure I follow you. It’s simple. Currently the Nextcloud app does not have a builtin contact client. Instead you should use the stand-alone DAVx5, or another CardDAV client to sync contacts.

As for Android, it has no native support for migrating contacts between different accounts. This is not unique to Nextcloud. Perhaps some apps can do it (Facebook, Skype, etc) but it isn’t an Android feature.

So I am thinking that you want Nextcloud to builds in contact support in the Nextcloud app? It would be good. Another option, perhaps easier, is to adapt DAVx5 so it can import accounts from the Nextcloud App?

Edit: you can support here https://forums.bitfire.at/topic/2121/please-support-nextclouds-singlesignon-sso/8

[summersab]

@anon9582441

I’m not sure I follow you. It’s simple

Imagine you’re an employee at Hobby Lobby, you have just enough technical know-how to click “deploy” on a cloud-based VM of Nextcloud, and your primary computing device is your phone. Is it still simple?

[anon9582441]

It can be improved. If you follow the link to DAVx5 you see they are adding support. It works pretty easy now
SSO is still not there though.

[denNorske]

Hello!

I would consider myself somewhat techical competent and I want to join the ship saying that CardDav/Nextcloud/Davx5 stuff is hard to set up as it is today, hands down. The amount of effort required to do a simple thing (sync contacts) seems to be unituitive. Although I cannot come with concrete suggestions as you may have, I want to share my experience doing it. Reason I want to share my experience is because I yet haven’t figured out how to set it up properly and…:

1. Which way the contacts sync and what is required to move entirely over to it
2. Which way calendars sync, and why can’t i sync my local calendars on my phone up to nextcloud? (tried this).
3. Why do we use dav for it? As mentioned above, can an alternative method be used and baked in to the nextcloud app on the mobile devices, and be an alternative you flip in your settings to enable/disable? What is limiting that today? About the google discussion above, I would say a huge amount of us are using google play services as long as we use android. The people that is anti-google 100%, feel free to continue using webdav because i respect that

Although, I am not directly criticising nextcloud because I am super happy with it, but I am just curious and want to follow this thread forward. @summersab very interesting topic you brought up.
den

[summersab]

Thanks, @denNorske. Again, I’m really not trying to be a jerk about this. I just think that the current state of contact syncing stands in stark contrast to the highly-polished product that is Nextcloud, and I believe there is room for discussion for how to make it better.

[anon9582441]

Just saw this : https://nextcloud.com/blog/nextcloud-3.8.1-for-android-is-out-plus-a-tip/

Contacts Backup…

[denNorske]

This dumps a contacts file (.vcf) and uploads this to .contacts_backups folder on your instance. I use this already. It sadly doesn’t sync contacts. But the interesting part --> How do they get it? Can’t it be built further on?

[anon9582441]

That was my thinking too.

The contacts web ui has a import feature. So t least one time imports are possible this way.

[denNorske]

would this make sense to anyone? I bet there is more to it than this. Could the VCF’s be delta based that it only syncs changes?

[summersab]

Right - so, I just spun up a brand-new Nextcloud VM, connected to it, installed DAVx5, synced my contacts, and . . . why did it sync nothing? Why doesn’t it pull contacts from my Android contact storage? You know, all of the hundreds of people I’ve been contacting via Gmail all these years and stored in my phonebook, but I’d now like to use on Nextcloud so I can switch to Protonmail? Oh, right - this is a separate contact list and isn’t connected to the one in Android because . . . well, I have no idea why. Yeah, this isn’t simple. The average user would have abandoned the whole thing, at this point.

@denNorske, that’s kinda what I was suggesting in a round-about way.

[denNorske]

This sounds very similar to what i experienced with my calendar syncing. I had hoped to have my phone calendars show up nicely in the web interface on nextcloud.