Federation: cannot add trusted server

We have the exact same Problem with two of our Nextcloud servers. Both run with Version 11.0.2. Both of the Servers are in the same LAN with no firewall between them. Have you solved your problem or does anyone have a hint what the problem could be?

I think this is caused by a issue I fixed here: https://github.com/nextcloud/server/pull/3825

I plan to backport it to stable11 as soon as it merged. For the issue we discuss here, I think this particular change will fix it: https://github.com/nextcloud/server/pull/3825/commits/1f1e36ac6560161ed6c26b8dbe980bfc62733581#diff-ec69af4ceadfc4207fcfd3373d0b9338

If you feel comfortable enough you could apply this changes to all the involved servers. Remove the trusted servers and add them again. (You only need the changes for apps/federation/lib/Middleware/AddServerMiddleware.php and can ignore the changes to the test file)

Dear Björn,
Thank you very much for your fast reply. I have change the mention script. After this I have removed every single server from the trusted list within the web interface. After re adding them the behavior is the same. The symbol stays yellow, and the log file on both servers show again “…[status code] 400 [reason phrase] Bad request…”. Have you any other idea what could cause this problem?
Many Thanks

@IZA not really at the moment. Just one thought: Do you use self signed SSL certificates on the server. This could also create problems.

On one server we use a wildcard cert from Thawte and on the other server a Lets Encrypt cert. But both are valid and the host systems (Ubuntu 16.04) has both root certs in the local store.

Another thought: is it possible to manually bypass/manipulate the token validation to force the federation between servers I know? This would not solve the problem but would make it possible to use the federation for now.

This three-way-handshake is needed in order to exchange a shared secret which is than used as passphrase to sync the users. What you could do in theory is to manipulate the database directly and add a shared_secret. But I didn’t tested it and I don’t recommend it. It would be better to find out why this three way handshake fails on your setup

Can you enable “debug” mode and try it again? Maybe this way we get more information?

If this doesn’t lead to more useful information I could prepare some additional debug outputs which you could add to your code. Btw, make sure that check the nextcloud.log on both servers.

Dear Björn,
Sorry for not replying. At the moment I´m busy with another project. But I made a simple test; I have created two demo instants of Nextcloud https://demo.nextcloud.com/iza-test001/ and https://demo.nextcloud.com/iza-test002/. When I try to configure a federation between this two servers, the behavior is the same like on my servers. The server will be listed as a trusted server but with a yellow symbol, on both sides. Maybe it is the configuration of the demo instances or there is a general problem with the federation.
Best

Update, and the same errors in the (web-)log:

server-001:
“GuzzleHttp\Exception\ClientException: Client error response [url] https://demo.nextcloud.com/iza-test002//ocs/v2.php/apps/federation/api/v1/request-shared-secret?format=json [status code] 400 [reason phrase] Bad request”

That’s expected because the fix is not yet applied on the demo server. As soon as they are on 11.0.3 I will test it and report back. Please ping me here in case the demo servers are updated and you don’t hear anything from me.

I have updated both of our servers to the latest Nextcloud version. If I now try to establish a federation between these two servers I get the following errors in the log files:

Server 1:
{“reqId”:“QTy5ab8y8oE3x85E5K5i”,“remoteAddr”:"",“app”:“federation”,“message”:“https://server2 refused to ask for a shared secret.”,“level”:1,“time”:“2017-04-25T14:00:03+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}

Server 2:
{“reqId”:“FCBV73zzNSgMpY6KelrL”,“remoteAddr”:"",“app”:“federation”,“message”:“https://Server1 refused to ask for a shared secret.”,“level”:1,“time”:“2017-04-25T14:00:15+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}

@IZA can you check the database on both servers? On server1 you should have a entry for server2 and on server2 you should have a entry for server1 in the oc_trusted_servers table. Can you check the ‘token’ in both tables and post them here? Thanks!

There is no token in the database. The following is the SQL export of the table:

Server 1:
INSERT INTO oc_trusted_servers (id, url, url_hash, token, shared_secret, status, sync_token) VALUES
(26, ‘https://cloud.server2.org’, ‘****************************************’, ‘’, NULL, 2, NULL);

Server 2:
INSERT INTO oc_trusted_servers (id, url, url_hash, token, shared_secret, status, sync_token) VALUES
(23, ‘https://cloud.server1.org’, ‘****************************************’, ‘’, NULL, 2, NULL);

Did you removed the server from the trusted servers list after update and tried again from scratch?
Also please try again:

  1. Remove server1 as trusted server on server2
  2. Remove server2 as trusted server on server1
  3. check the oc_trusted_servers table on both servers, both entries should be gone
  4. add the trusted servers again on server1 and server2
  1. Done
  2. Done
  3. Tables are empty on both servers
  4. Done

Immediately after I add the trusted server I see the following database entries, with a token:

Server 1:
INSERT INTO oc_trusted_servers (id, url, url_hash, token, shared_secret, status, sync_token) VALUES
(27, ‘https://cloud.server2.org’, ‘***************************************’, ‘Jtwk328CJJ1TKucx’, NULL, 2, NULL);

Server 2:
INSERT INTO oc_trusted_servers (id, url, url_hash, token, shared_secret, status, sync_token) VALUES
(24, ‘https://cloud.server1.org’, ‘***************************************’, ‘35spNYqsMXsqTqSZ’, NULL, 2, NULL);

Nothing happens in the web interface or in the log file, the entries are still yellow. I think the reason for this is that I use system cron. After the cron job is executed I get the known error in the log file and the token in the database is lost again.

Server 1:
Nextcloud Log:
{“reqId”:“lvhxwoD2MoQDzsCXwiUZ”,“remoteAddr”:"",“app”:“federation”,“message”:“https://cloud.server2.org refused to ask for a shared secret.”,“level”:1,“time”:“2017-05-03T12:00:02+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}
SQL:
INSERT INTO oc_trusted_servers (id, url, url_hash, token, shared_secret, status, sync_token) VALUES
(27, ‘https://cloud.server2.org’, ‘***************************************’, ‘’, NULL, 2, NULL);

Server 2:
Nextcloud Log:
{“reqId”:“6F/v9WQCsaouF2xPUWRO”,“remoteAddr”:"",“app”:“federation”,“message”:“https://cloud.server1.org refused to ask for a shared secret.”,“level”:1,“time”:“2017-05-03T12:00:03+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}
SQL:
INSERT INTO oc_trusted_servers (id, url, url_hash, token, shared_secret, status, sync_token) VALUES
(24, ‘https://cloud.server1.org’, ‘***************************************’, ‘’, NULL, 2, NULL);

Is this the only thing you have in the log file? I just had another look in the code.

At the same time server2 creates this log message (and the same, the other way around):

{"reqId":"6F\/v9WQCsaouF2xPUWRO","remoteAddr":"","app":"federation","message":"https:\/\/cloud.server1.org refused to ask for a shared 

server1 should create one of this log message:

Either:

remote server not trusted (https://cloud.server2.org) while requesting shared secret

Or:

remote server (https://cloud.server2.org) presented lower token. We will initiate the exchange of the shared secret.

because this are the two situations we return a 403 which will lead to the other message you posted. If non of the two messages show up in your look file, there seems to be another code path on your system which results in a 403.

When I search the lines around the log entry I get this:

Comand:
tac nextcloud.log | grep -A 5 -B 5 'server2.org' | head -20

Server 1:
{“reqId”:“lvhxwoD2MoQDzsCXwiUZ”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OC\Command\CommandJob job with ID 1562”,“level”:0,“time”:“2017-05-03T12:00:02+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}
{“reqId”:“lvhxwoD2MoQDzsCXwiUZ”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Federation\BackgroundJob\RequestSharedSecret job with ID 1561”,“level”:0,“time”:“2017-05-03T12:00:02+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}
{“reqId”:“lvhxwoD2MoQDzsCXwiUZ”,“remoteAddr”:"",“app”:“federation”,“message”:“https://cloud.server2.org refused to ask for a shared secret.”,“level”:1,“time”:“2017-05-03T12:00:02+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}
{“reqId”:“lvhxwoD2MoQDzsCXwiUZ”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Federation\BackgroundJob\RequestSharedSecret job with ID 1561”,“level”:0,“time”:“2017-05-03T12:00:02+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}
{“reqId”:“lvhxwoD2MoQDzsCXwiUZ”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\NextcloudAnnouncements\Cron\Crawler job with ID 1339”,“level”:0,“time”:“2017-05-03T12:00:02+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}

Server 2:
{“reqId”:“xzONce+B9sQ4gG45/8jA”,“remoteAddr”:"***",“app”:“user_ldap”,“message”:“Ready for a paged search”,“level”:0,“time”:“2017-05-3T12:00:35+00:00”,“method”:“GET”,“url”:"/ocs/v2.php/apps/notifications/api/v1/notifications?format=json",“user”:"***",“version”:“11.0.3.2”}
{“reqId”:“xzONce+B9sQ4gG45/8jA”,“remoteAddr”:"**",“app”:“user_ldap”,“message”:"initializing paged search for Filter objectClass= base Array\n(\n [0] => uid=***\n)\n attr Array\n(\n [0] => jpegPhoto\n)\n limit 500 offset 0",“level”:0,“time”:“2017-05-03T12:00:35+00:00”,“method”:“GET”,“url”:"/ocs/v2.php/apps/notifications/api/v1/notifications?format=json",“user”:"***",“version”:“11.0.3.2”}
{“reqId”:“6F/v9WQCsaouF2xPUWRO”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Federation\BackgroundJob\RequestSharedSecret job with ID 48”,“level”:0,“time”:“2017-05-03T12:00:03+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}
{“reqId”:“6F/v9WQCsaouF2xPUWRO”,“remoteAddr”:"",“app”:“federation”,“message”:“https://cloud.server1.org refused to ask for a shared secret.”,“level”:1,“time”:“2017-05-03T12:00:03+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}
{“reqId”:“6F/v9WQCsaouF2xPUWRO”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Federation\BackgroundJob\RequestSharedSecret job with ID 48”,“level”:0,“time”:“2017-05-03T12:00:03+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}
{“reqId”:“6F/v9WQCsaouF2xPUWRO”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\User_LDAP\Jobs\CleanUp job with ID 18”,“level”:0,“time”:“2017-05-03T12:00:03+00:00”,“method”:"–",“url”:"–",“user”:"–",“version”:“11.0.3.2”}

I have now tested this behavior with Nextcloud 12, with the same results. I have also done the test with two Nextcloud demo (v12) sessions, with the same result. In both tests the federation entry has a yellow symbol, and the user list will not be exchanged.

Now tree days after my tests I noticed that the federation is no working and is displayed with a green symbol. I guess I had to wait a little bit longer, although I waited until the cronjob was finished. So I guess Nextcloud 12 solved the problem for me. Thanks!

I‘m facing the same issue - for YEARS to be honest, even back in old owncloud times. I never managed to get other nextclouds to be green.

Indicators:

  • my Nextcloud log says „[remote server] refused to ask for a shared secret“ at the time when trying to establish the trust for the first time
  • also in NC log: „Remote server presented lower token. We will initiate the exchange of the shared secret.“
  • database table „nc_trusted_servers“ contains entries for all remote servers including a token and status 2, but columns „sync_token“ and „shared_secret“ (!!!) are empty“

What else to check or try? Any ideas?