Z-Push Sync and how it works

I donā€™t know if i post this now in the right place, but some of the Admins could move the topic, if they want. Iā€™m relative new to Nextcloud, so please forgive e, if made some mistakes in my description, but i would do my best to describe the things.

My goal was it, to have a private Cloud, including Protocols like CalDAV, CardDAV, IMAP and also Exchange ActiveSync, since im coming from a Windows World and have have used Outlook.com since a very long time. All my setup is done on a small Raspberry Pi 2 with Raspbian installed. I use a ā€œMiniban Imageā€, since i use this Box only as a Server. You should have a Nextcloud installed up and running. As a Webserver, i use Nginx with PHP-FPM. For Nextcloud, i have installed additionally MariaDB as a MySQL Backend. Currently Nextcloud is installed as a Subdirectory under Webroot. This will change later. Its planned to move the installation to the Webroot of a Subdomain of my DynDNS Setup.

To get ActiveSync to work, i use Z-Push. I have used this before with a Zarafa/Kopano Installation and like it very much. Z-Push does not only work with Zarafa/Kopano, it also works with other Backendā€™s to. It can work with BackendIMAP, BackendCalDAV, BackendCardDAV and other i didnā€™t remember now. Best of all, there is a BackendCombinded, which allow it to combine several Backends. In my case, i use BackendCombined, to combine BackendIMAP for Mails via IMAP, BackendCalDAV for the Calendars and Tasks and BackendCardDAV for Contacts. Since they are combined, i have only one possibility to enter User Credentials and it is a Multi-User-Thing -> All Backends must share exactly the same Credentials.

Since i tried the Installation, i have setup first every Backend for itself, to have a better insight, where are Problems and what is working. I started with BackendIMAP to get my Mails to work. For this, you must install first Z-Push. You can download it via wget from http://download.z-push.org/final/. For example you can use

wget http://download.z-push.org/final/2.3/z-push-2.3.3.tar.gz

After this, you should unpack the archive to the directory /usr/share/ and rename the resulting folder to ā€˜z-pushā€™. Make this folder owned by your webserveruser with ā€˜chown -R www-data:www-data /usr/share/z-pushā€™

Now you can edit the file /usr/share/z-push/config.php. Later i will call this file ā€œGlobal Configā€, since there are config.php files for every Backend. First change the Timezone to your Timezone. In my case i use
define('TIMEZONE', 'Europe/Berlin');

If you must Login to your IMAP Server with full E-Mail Address, set
define('USE_FULLEMAIL_FOR_LOGIN', true);

Now specify two directory for state and logfiles:

define('STATE_DIR', '/var/lib/z-push/'); define('LOGFILEDIR', '/var/log/z-push/');

Both directorys must be writeable by the webserveruser. As next you define the used Backend. For this first test, we would like to use IMAP and so we use

define('BACKEND_PROVIDER', 'BackendIMAP');

For now, you can ignore the rest of the Global Config file and save your changes. Do not forget to allow write access for the state and log file directory.

As next configure the BackendIMAP. For this, edit the file /usr/share/z-push/backend/imap/config.php. Here change the Server to the Server you would like to use. Ask your Mail Provider for the correct values. In my case, i must enter the server address, the port 993 and as option i use only ā€˜/tlsā€™. If /tls did not work, try /ssl. You can try to determine the settings from your IMAP Server, if you connect via telnet to it.

Now have a look at the IMAP folder settings in config.php for IMAP, to set the correct folders. In my case, i must change some folder names, maybe its needed in your case to. You could look at the foldernames via telnet or any normal IMAP Client. You could try also the Webmail from Nextcloud or the Rainloop App. In my case, i use Rainloop.

Its now time to change your nginx config. You must add a new location in your virtual server directive to set the Alias directory for /Microsoft-Server-ActiveSync as described in the readme from Z-Push for Apache. Itā€™s a little bit more in nginx and i dont know if my Config is the best for this, but it works. So lets define such a location:

location /Microsoft-Server-ActiveSync {
		root /usr/share/z-push/;
		add_header Strict-Transport-Security max-age=63072000;		
		add_header X-Frame-Options DENY;		
		add_header X-Content-Type-Options nosniff;
		fastcgi_connect_timeout 300;                             
		fastcgi_send_timeout    300;                           
		fastcgi_read_timeout    300;
		fastcgi_pass unix:/var/run/php5-fpm.sock;
		include         fastcgi_params;
		fastcgi_param   SCRIPT_FILENAME    $document_root/index.php;
		fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
	}

I set here a timeout with 300, because the first sync (and sometimes syncs in future) could take a long time on our small Raspberry Pi. With this, you would see the timeout error not so often as with other settings, because its time enough to make a Sync. You should also change the timeout setting in your php.ini for fpm, to get no timeouts.

Now you could try to setup a first test on your Client Device. I use a Android Phone with the App ā€œNineā€ and the Windows Mail App. Both can use EAS. For the first tests, its good to have some Logfile on the Screen running, For this i open multiple Putty Windows or one Putty Window with some ā€œscreenā€-Sessions. So i have a look at the Error and Access Logs of nginx and Z-Push. In my case it starts directly with some missing PHP Libraries. I had to install the following modules: libawl-php, php-gettext, php5-curl, php5-gd, php5-imap, php5-json, php5-mcrypt, php5-mysql (since i save my STATES in MariaDB), php5-readline and php5-xsl. After installing the missing modules, the sync of E-Mails should work fine. Maybe it needs some time, if you have a lot of Mails.

If the Mails are working you can look at the Calendar Data. For this to make work, you should change first the correct Backend. In my case this is BackendCalDAV. Open /usr/share/z-push/backend/caldav/config.php and change the needed settings:

define('CALDAV_PROTOCOL', 'https');
define('CALDAV_SERVER', 'your.domain.net');
define('CALDAV_PORT', '443');
define('CALDAV_PATH', '/nextcloud/remote.php/dav/calendars/%u/'); //if nextcloud is installed in a subfolder
define('CALDAV_PERSONAL', 'personal');
define('CALDAV_SUPPORTS_SYNC', true);
define('CALDAV_MAX_SYNC_PERIOD', 2147483647);

Change this to your server and if your installation is in other directory, change also the path. Im not sure, if CalDAV of NextCloud does support Sync, but i thinks so. Someone shoud correct this, if im wrong.

Next, change the Backend in the Global Config to BackendCalDAV instead of BackendIMAP and start a fresh Sync. Your Calendars should then appear on your Device.
Attention: If you use the Standard EAS on a Samsung Device, you see only one Calendar! Sometimes this is a merge of all your Calendars on the Server and sometimes its only the Maincalendar (the one which comes first). This is a problem of the Samsung EAS Implementation. I dont know, if this also the Case with the GMail App, since i havent used this before. With the App ā€œNineā€, all is OK and you have all your Calendars andā€¦ Tata!!! ā€¦Tasksā€¦

If Calendar and Mails do work, you could have a look at the Contacts. First change the config in /usr/share/z-push/backend/carddav/config.php:

define('CARDDAV_PROTOCOL', 'https');
define('CARDDAV_SERVER', 'your.domain.net');
define('CARDDAV_PORT', '443');
define('CARDDAV_PATH', '/nextcloud/remote.php/dav/addressbooks/users/%u/contacts/');
define('CARDDAV_DEFAULT_PATH', '/nextcloud/remote.php/dav/addressbooks/users/%u/contacts/');
define('CARDDAV_GAL_MIN_LENGTH', 5);
define('CARDDAV_CONTACTS_FOLDER_NAME', '%u Addressbook');
define('CARDDAV_SUPPORTS_SYNC', true);
define('CARDDAV_SUPPORTS_FN_SEARCH', false);
define('CARDDAV_URL_VCARD_EXTENSION', '.vcf');

Its nearby the same as for CalDAV, also here, please change your path and server do your needs. After this, change the Backend in the Global Config from BackendCalDAV to BackendCardDAV and restart the Sync. Only one Adressbook is synced, if you have more than one Adressbook, you could only sync he first one.

If Sync of Contacts do also work, you should edit /usr/share/z-push/backend/combined/config.php.

First change the array of available Bankends to your backends, like this:

'backends' => array(
                'i' => array(
                'name' => 'BackendIMAP',
				),
				'c' => array(
					'name' => 'BackendCalDAV',
				),
				'd' => array(
					'name' => 'BackendCardDAV',
				),
            ),

and map the folders to the correct Backend as in my example:

'folderbackend' => array(
                SYNC_FOLDER_TYPE_INBOX => 'i',
				SYNC_FOLDER_TYPE_DRAFTS => 'i',
				SYNC_FOLDER_TYPE_WASTEBASKET => 'i',
				SYNC_FOLDER_TYPE_SENTMAIL => 'i',
				SYNC_FOLDER_TYPE_OUTBOX => 'i',
				SYNC_FOLDER_TYPE_TASK => 'c',
				SYNC_FOLDER_TYPE_APPOINTMENT => 'c',
				SYNC_FOLDER_TYPE_CONTACT => 'd',
				SYNC_FOLDER_TYPE_NOTE => 'c',
				SYNC_FOLDER_TYPE_JOURNAL => 'c',
				SYNC_FOLDER_TYPE_OTHER => 'i',
				SYNC_FOLDER_TYPE_USER_MAIL => 'i',
				SYNC_FOLDER_TYPE_USER_APPOINTMENT => 'c',
				SYNC_FOLDER_TYPE_USER_CONTACT => 'd',
				SYNC_FOLDER_TYPE_USER_TASK => 'c',
				SYNC_FOLDER_TYPE_USER_JOURNAL => 'c',
				SYNC_FOLDER_TYPE_USER_NOTE => 'c',
				SYNC_FOLDER_TYPE_UNKNOWN => 'i',
            ),

If this is done, change the Backend in the Global Config from BackenCalDAV to BaeckendCombined. After this, resync you device and you should have Mails, Calendars and Contacts synced.

6 Likes

Will you continue this? Would be great to use this with my nextcloud install!

Yes of course, but i hadnā€™t the time for this: Maybe at the Weekend.

Would be great! And a big thanx for the first post!

I hope my little HowTo could help some people a little bit. Im not a professional User. I have done this the first time and there may some quirks in the tutorial, but it should work. If you have additions, feel free to add them here.

1 Like

Quite an interesting project which could be useful to many. I use Z-Push together with Kopano as itā€™s already integrated with NextCloud through Kopano Files (which maybe could be an idea to optimise your setup :wink: ) but your post gave me ideas on how to use Z-Push to sync other applications. Thanks

Hi @Dist1958,

I am involved in the Z-Push project and also wanted to drop by and say thanks for the howto. If it does not find a place in a Nextcloud Wiki, I am sure we can arrange a subsection for it over at the official Z-Push wiki.

Itā€™s also worth noting that we are currently searching for new/additional maintainers for the backends that you are using. You and others interested in it can find more information at: https://forums.zarafa.com/showthread.php?13210-Developer-Wanted.

I would also have some notes on your instructions:

Since Z-Push 2.3.0 we now also have packages and repositories for various distributions, we now even recommend the use of the repositories over the use of the tar archive (as you have explained in your howto). Once you have the repositories added you could install your desired packages with:

sudo apt install z-push-backend-caldav z-push-backend-carddav z-push-backend-combined z-push-backend-imap z-push-common z-push-ipc-sharedmemory

Since you are using states in MySQL you also install z-push-state-sql, for users of Apache z-push-config-apache could also be interesting. We currently donā€™t have a package for Nginx configuration, but are open to contributions in this regard (the same can be said for a Nextcloud/OwnCloud meta package imho).

If you have any additional questions feel free to ask.

Hi Felix, nice to see you here. We was in contact back in the days from Zarafa Test Days. I will write you via PM on the Zarafa Forum :slight_smile: Installing from repositories does not work in my case, since im on a Raspberry Pi. There are no official Packages for Z-Push, Zarafa or Kopano at Raspbian. I would be more than happy, to see a Repo for Raspberry Pi, official or not. I know, there are Packages for Arch Linux made by Pietma, but i didnā€™t like Arch Linux because of its Rolling Release Philosophy. If there is a Repo up, for Kopano or Z-Psuh, i would love to see the Address here.

If you have some ideas, how to tweak the settings a for Raspberry Pi, i would love to see your Feedback. Currently i fight with some Timeouts for the Ping Command and a small TimeZone Issue. Manfred have seen my Problems other there in the Zarafa Forum.

While there probably never will be packages for Kopano for the Pi (imho way to underpowered to be really of use here), the Z-Push packages do not contain any binary parts and are therefore architecture independent. So you are free to use them.

As for the timeouts: Mobile connections are ideally really long lasting, so better donā€™t define any timeouts in your php-engine, or set them to a high value. Under good network connections mobile devices can keep connections open for up to an hour (with keepalive in between).

Currently, i try 3780 as the timeout. I hope thatā€™s enough. If i have a bad network connection, is the connection longer active or shorter? I think its shorter, if the signal is lostā€¦ is this correct?

A slight wrinkle now when using IMAP auth on nextcloud and specifying the acceptable domain name in the nextcloud conf - the nextcloud username and caldav/carddav paths will use local part of full email, but youā€™ll porbably still need z-push configured to use full email for login.

Result - %u => full email address.

Auth still works, but carddav and caldav paths will be wrong.

This howto seems to be very useful in regard to Windows 10 and its non-support of generic Caldav and Carddav. The only way to use these directly with Windows 10 is to do an ugly hack of using the Apple ICloud account type and change the URLs afterwards.

Running a Z-Push server as an intermediary and setting it up as an Exchange account in Windows 10 seems to be a better way after all.

I think this guide should be part of the howto section of the forum, after all.

1 Like

Categories have since been re-jigged a bit and this was prior to that, so missed. Rectifying now :sunglasses:

The only way I got this working was to modify the z-push backend backend caldav.php and carddav.php files to use local part of email name for the urls in their Logon methods.
e.g.: $this->_caldav_path = str_replace(ā€™%uā€™, Utils::GetLocalPartFromEmail($username), CALDAV_PATH);

Be warned though, each sync check in z-push (30 secs by default) results in about 7 imap logins - for each user and device. So you seem to get about 14/15 imap logins per user device per minute. Any serious number of users and youā€™ll want to make sure your imap server is well-tuned for lots of logins! Dovecot can be configured for this if required.

Hi! Is there a way to let z-push let authorize the users against nextcloud/ owncloud?

Andy

I succesfully setup z-push with NC. Everything seems OK, except one (drive me crazy) issue. When someone try to accept (maybe decline too) an events on Android Phone, the incoming mail vanished (which is ok), BUT event is not in Calendar. I will make some tests with iPhone or other androidā€™s mail client and so.

1 Like