LDAP connection issue

jrevillard · October 14, 2019, 11:39am

Dear all,

I’m using nextcloud/owncloud for years now but I’m facing an issue that I cannot solve.

I think that this appeared where I updated to nextcloud 16.0 but I cannot guaranty this. This is a non deterministic behavior. I would say that more than 90% of the time, where users try to login into nextcloud they get the “Internal server error message” and the root cause is this:

{"reqId":"PrwcdqQ6SPwuTtzARQZu","level":3,"time":"2019-10-14T11:29:04+00:00","remoteAddr":"37.164.255.44","user":"--","app":"index","method":"POST","url":"\/login","message":{"Exception":"OC\\ServerNotAvailableException","Message":"Lost connection to LDAP server.","Code":0,"Trace":[{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/LDAP.php","line":388,"function":"processLDAPError","class":"OCA\\User_LDAP\\LDAP","type":"->","args":[null]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/LDAP.php","line":311,"function":"postFunctionCall","class":"OCA\\User_LDAP\\LDAP","type":"->","args":[]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/LDAP.php","line":203,"function":"invokeLDAPMethod","class":"OCA\\User_LDAP\\LDAP","type":"->","args":["*** sensitive parameters replaced ***"]},{"function":"search","class":"OCA\\User_LDAP\\LDAP","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/Access.php","line":1104,"function":"call_user_func_array","args":[[{"__class__":"OCA\\User_LDAP\\LDAP"},"*** sensitive parameter replaced ***"],["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***"]]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/Access.php","line":1125,"function":"OCA\\User_LDAP\\{closure}","class":"OCA\\User_LDAP\\Access","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/Access.php","line":1160,"function":"invokeLDAPMethod","class":"OCA\\User_LDAP\\Access","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/Access.php","line":1317,"function":"executeSearch","class":"OCA\\User_LDAP\\Access","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***",500,"*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/Access.php","line":1002,"function":"search","class":"OCA\\User_LDAP\\Access","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/Access.php","line":903,"function":"searchUsers","class":"OCA\\User_LDAP\\Access","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/Access.php","line":877,"function":"fetchListOfUsers","class":"OCA\\User_LDAP\\Access","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/User_LDAP.php","line":174,"function":"fetchUsersByLoginName","class":"OCA\\User_LDAP\\Access","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/User_LDAP.php","line":191,"function":"getLDAPUserByLoginName","class":"OCA\\User_LDAP\\User_LDAP","type":"->","args":["*** sensitive parameter replaced ***"]},{"function":"checkPassword","class":"OCA\\User_LDAP\\User_LDAP","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/User_Proxy.php","line":81,"function":"call_user_func_array","args":[[{"__class__":"OCA\\User_LDAP\\User_LDAP"},"checkPassword"],["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***"]]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/Proxy.php","line":152,"function":"walkBackends","class":"OCA\\User_LDAP\\User_Proxy","type":"->","args":["*** sensitive parameter replaced ***","checkPassword",["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***"]]},{"file":"\/var\/www\/html\/apps\/user_ldap\/lib\/User_Proxy.php","line":196,"function":"handleRequest","class":"OCA\\User_LDAP\\Proxy","type":"->","args":["*** sensitive parameter replaced ***","checkPassword",["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***"]]},{"file":"\/var\/www\/html\/lib\/private\/User\/Manager.php","line":212,"function":"checkPassword","class":"OCA\\User_LDAP\\User_Proxy","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/lib\/private\/Authentication\/Login\/UidLoginCommand.php","line":49,"function":"checkPasswordNoLogging","class":"OC\\User\\Manager","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/lib\/private\/Authentication\/Login\/ALoginCommand.php","line":39,"function":"process","class":"OC\\Authentication\\Login\\UidLoginCommand","type":"->","args":[{"__class__":"OC\\Authentication\\Login\\LoginData"}]},{"file":"\/var\/www\/html\/lib\/private\/Authentication\/Login\/UserDisabledCheckCommand.php","line":57,"function":"processNextOrFinishSuccessfully","class":"OC\\Authentication\\Login\\ALoginCommand","type":"->","args":[{"__class__":"OC\\Authentication\\Login\\LoginData"}]},{"file":"\/var\/www\/html\/lib\/private\/Authentication\/Login\/ALoginCommand.php","line":39,"function":"process","class":"OC\\Authentication\\Login\\UserDisabledCheckCommand","type":"->","args":[{"__class__":"OC\\Authentication\\Login\\LoginData"}]},{"file":"\/var\/www\/html\/lib\/private\/Authentication\/Login\/PreLoginHookCommand.php","line":52,"function":"processNextOrFinishSuccessfully","class":"OC\\Authentication\\Login\\ALoginCommand","type":"->","args":[{"__class__":"OC\\Authentication\\Login\\LoginData"}]},{"file":"\/var\/www\/html\/lib\/private\/Authentication\/Login\/Chain.php","line":108,"function":"process","class":"OC\\Authentication\\Login\\PreLoginHookCommand","type":"->","args":[{"__class__":"OC\\Authentication\\Login\\LoginData"}]},{"file":"\/var\/www\/html\/core\/Controller\/LoginController.php","line":298,"function":"process","class":"OC\\Authentication\\Login\\Chain","type":"->","args":[{"__class__":"OC\\Authentication\\Login\\LoginData"}]},{"file":"\/var\/www\/html\/lib\/private\/AppFramework\/Http\/Dispatcher.php","line":170,"function":"tryLogin","class":"OC\\Core\\Controller\\LoginController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/lib\/private\/AppFramework\/Http\/Dispatcher.php","line":99,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OC\\Core\\Controller\\LoginController"},"tryLogin"]},{"file":"\/var\/www\/html\/lib\/private\/AppFramework\/App.php","line":126,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OC\\Core\\Controller\\LoginController"},"tryLogin"]},{"file":"\/var\/www\/html\/lib\/private\/AppFramework\/Routing\/RouteActionHandler.php","line":47,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OC\\Core\\Controller\\LoginController","tryLogin",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"_route":"core.login.tryLogin"}]},{"function":"__invoke","class":"OC\\AppFramework\\Routing\\RouteActionHandler","type":"->","args":[{"_route":"core.login.tryLogin"}]},{"file":"\/var\/www\/html\/lib\/private\/Route\/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"OC\\AppFramework\\Routing\\RouteActionHandler"},{"_route":"core.login.tryLogin"}]},{"file":"\/var\/www\/html\/lib\/base.php","line":1000,"function":"match","class":"OC\\Route\\Router","type":"->","args":["\/login"]},{"file":"\/var\/www\/html\/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"\/var\/www\/html\/apps\/user_ldap\/lib\/LDAP.php","Line":349,"CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/77.0.3865.90 Safari\/537.36","version":"17.0.0.9"}

As you can see in the logs, I just updated to nextcloud 17.0.0 to see if the problem persist but unfortunately yes.

When I login using the admin account, if I go in the ldap configuration, everything is green: I can retrieve users, groups etc…

I spent hours to try to understand but I’m completely lost.

Here is the ldap cnfiguration (openldap):

+-------------------------------+--------------------------------------------------------------------------------------+
| Configuration                 | s01                                                                                  |
+-------------------------------+--------------------------------------------------------------------------------------+
| hasMemberOfFilterSupport      | 0                                                                                    |
| homeFolderNamingRule          |                                                                                      |
| lastJpegPhotoLookup           | 0                                                                                    |
| ldapAgentName                 | cn=admin,dc=maatg,dc=fr                                                              |
| ldapAgentPassword             | ***                                                                                  |
| ldapAttributesForGroupSearch  | cn;description                                                                       |
| ldapAttributesForUserSearch   | uid                                                                                  |
| ldapBackupHost                |                                                                                      |
| ldapBackupPort                |                                                                                      |
| ldapBase                      | dc=maatg,dc=fr                                                                       |
| ldapBaseGroups                | dc=maatg,dc=fr                                                                       |
| ldapBaseUsers                 | dc=maatg,dc=fr                                                                       |
| ldapCacheTTL                  | 600                                                                                  |
| ldapConfigurationActive       | 1                                                                                    |
| ldapDefaultPPolicyDN          |                                                                                      |
| ldapDynamicGroupMemberURL     |                                                                                      |
| ldapEmailAttribute            | mail                                                                                 |
| ldapExperiencedAdmin          | 1                                                                                    |
| ldapExpertUUIDGroupAttr       |                                                                                      |
| ldapExpertUUIDUserAttr        |                                                                                      |
| ldapExpertUsernameAttr        |                                                                                      |
| ldapExtStorageHomeAttribute   |                                                                                      |
| ldapGidNumber                 | gidNumber                                                                            |
| ldapGroupDisplayName          | cn                                                                                   |
| ldapGroupFilter               | objectclass=groupOfNames                                                             |
| ldapGroupFilterGroups         |                                                                                      |
| ldapGroupFilterMode           | 0                                                                                    |
| ldapGroupFilterObjectclass    |                                                                                      |
| ldapGroupMemberAssocAttr      | member                                                                               |
| ldapHost                      | ldap.gnubila.fr                                                                      |
| ldapIgnoreNamingRules         |                                                                                      |
| ldapLoginFilter               | (|(&(objectclass=inetOrgPerson)(uid=%uid))(&(objectclass=inetOrgPerson)(mail=%uid))) |
| ldapLoginFilterAttributes     |                                                                                      |
| ldapLoginFilterEmail          | 0                                                                                    |
| ldapLoginFilterMode           | 0                                                                                    |
| ldapLoginFilterUsername       | 1                                                                                    |
| ldapNestedGroups              | 0                                                                                    |
| ldapOverrideMainServer        | 0                                                                                    |
| ldapPagingSize                | 500                                                                                  |
| ldapPort                      | 389                                                                                  |
| ldapQuotaAttribute            |                                                                                      |
| ldapQuotaDefault              |                                                                                      |
| ldapTLS                       | 1                                                                                    |
| ldapUserAvatarRule            | default                                                                              |
| ldapUserDisplayName           | cn                                                                                   |
| ldapUserDisplayName2          |                                                                                      |
| ldapUserFilter                | objectclass=inetOrgPerson                                                            |
| ldapUserFilterGroups          |                                                                                      |
| ldapUserFilterMode            | 0                                                                                    |
| ldapUserFilterObjectclass     |                                                                                      |
| ldapUuidGroupAttribute        | auto                                                                                 |
| ldapUuidUserAttribute         | auto                                                                                 |
| turnOffCertCheck              | 0                                                                                    |
| turnOnPasswordChange          | 0                                                                                    |
| useMemberOfToDetectMembership | 1                                                                                    |
+-------------------------------+--------------------------------------------------------------------------------------+

Could someone guide me please to solve this issue ?

Thanks in advance,
Best,
Jerome

KarlF12 · October 14, 2019, 12:03pm

Are there any errors on the LDAP server?

I would run a packet capture of the LDAP connection when users can’t log in to see if anything sticks out.

jrevillard · October 14, 2019, 3:24pm

Thanks @KarlF12 for your help.

I managed to get some data with tcpdump / wireshark but I cannot analyze it… what should I look for ?

Best.

KarlF12 · October 14, 2019, 3:55pm

Timeouts or failed queries, I would expect. Or lack of a query being sent when they log in. Any sort of LDAP error.

Basically:

Does the server send the LDAP query
Does the LDAP server reply
Does the reply contain the requested info

jrevillard · October 15, 2019, 10:35am

Does the server send the LDAP query

YES…

Does the LDAP server reply

There is so much traffic that it’s complicated to follow

Does the reply contain the requested info

I cannot understand it… could I post the tcpdump output here ? does it contain sensitive info ?

Best,

jrevillard · October 15, 2019, 10:36am

Ah by the way, I was looking at our ldap, let me check tcpdump on the nextcloud server

jrevillard · October 15, 2019, 10:42am

At least, the only thing that I can tell is that the LDAP server is answering…

Also ldap:check-user always work,

docker exec --user www-data nextcloud_nextcloud_1 php occ ldap:check-user -vvv XXXXXXXXXXX
The user is still available on LDAP.

The problem seems to only be password checking isn’t it ?

KarlF12 · October 15, 2019, 1:44pm

You would need to open the tcpdump in Wireshark to really be able to see what’s going on. And yes, it most likely does contain sensitive data.

If you have no problems listing LDAP users, then I’m not sure where the problem is going to be. I have limited experience with OpenLDAP and have only integrated Nextcloud with Active Directory.

Edit: Also, make sure that you restart your memcache in case that could be part of the issue. According to the docs, Nextcloud does use it to cache LDAP info.

jrevillard · October 15, 2019, 9:30pm

Thanks, redis is already setup.

I don’t know at all how to fix the issue …

jrevillard · October 16, 2019, 10:32am

I’m making some progress!

I forgot to say that I’m running nextcloud using docker (official docker image) and I made some tests with the ldap communication.

Outside of the docker container, in the same host, running ldapsearch works:

# ldapsearch -x -LLL -H ldaps://ldap.gnubila.fr:636 -D cn=admin,dc=maatg,dc=fr -w UcacVenifHakyab1 -b dc=maatg,dc=fr | wc -l
3672

But inside the container it start to retrive results and crash in the middle:

# ldapsearch -x -LLL -H ldaps://ldap.gnubila.fr:636 -D cn=admin,dc=maatg,dc=fr -w UcacVenifHakyab1 -b dc=maatg,dc=fr | wc -l
ldap_result: Can't contact LDAP server (-1)
51

Any idea about the potential problem ?