Users encryption key is never generated

Nextcloud version: 11.0.2
Operating system and version: Debian 9
PHP version: 7.0
OpenSSL: 1.1.0e

Can you reliably replicate it? (If so, please outline steps): yes
This issue is a new one, on a fresh installation of v11.

The issue you are facing:
0. Encryption is activated.

  1. Create a new user.

What happens:
At the first login I first got the message No public key found for user XY, after logging in again I got the message “missing private key of encryption app” and I am directed to the settings where I can enter my old password to re-encrypt the files. However, there is no old password. This is a new account. However, if I still go there and enter some fake passwords and click on “update private key” I get a 503 error in the network log and the returned HTML (which causes a JSON parse error, as it is a HTML file) contains the error “Private Key missing for user: XY”.

What should happen:
Generate the private key & stuff on first login.

The output of your Nextcloud log in Admin > Logging:

Error encryption Encryption Library, symmetrical encryption failed no content given 5 minutes ago
Error PHP Undefined variable: data at /[…]/apps/files_accesscontrol/lib/StorageWrapper.php#259 5 minutes ago
Error PHP Missing argument 2 for OCA\FilesAccessControl\StorageWrapper::file_put_contents(), called in /[…]/lib/private/Files/View.php on line 1124 and defined at /[…]/apps/files_accesscontrol/lib/StorageWrapper.php#257 5 minutes ago
Error encryption Encryption Library:error:02001002:system library:fopen:No such file or directory 5 minutes ago
Error encryption Encryption library couldn’t export users private key, please check your servers OpenSSL configuration.“no user given” 5 minutes ago
Error encryption Encryption library openssl_pkey_new() fails: error:2006D080:BIO routines:BIO_new_file:no such file 5 minutes ago
Error encryption Encryption Library couldn’t generate users key-pair for “no user given” 5 minutes ago
Warning encryption no public key found for user “[…]”, user will not be able to read the file 5 minutes ago
Error encryption Encryption Library, symmetrical encryption failed no content given 5 minutes ago
Error PHP Undefined variable: data at /[…]/apps/files_accesscontrol/lib/StorageWrapper.php#259 5 minutes ago
Error PHP Missing argument 2 for OCA\FilesAccessControl\StorageWrapper::file_put_contents(), called in /www/lib/private/Files/View.php on line 1124 and defined at /[…]/apps/files_accesscontrol/lib/StorageWrapper.php#257 5 minutes ago
Error encryption Encryption Library:error:2006D080:BIO routines:BIO_new_file:no such file 5 minutes ago
Error encryption Encryption library couldn’t export users private key, please check your servers OpenSSL configuration.“no user given” 5 minutes ago
Error encryption Encryption library openssl_pkey_new() fails: error:0E064002:configuration file routines:CONF_load:system lib 5 minutes ago
Error encryption Encryption Library couldn’t generate users key-pair for “no user given”

Actually when enabling encryption with the command line everything works. Could it be that NextCloud somehow needs access to the openssl binary or so?

Another log when creating a new user:

Warning	encryption	no public key found for user "ghut", user will not be able to read the file	a few seconds ago
Error	encryption	Encryption Library, symmetrical encryption failed no content given	a few seconds ago
Error	PHP	Undefined variable: data at /www/lib/private/Files/Storage/Wrapper/Wrapper.php#251	a few seconds ago
Error	PHP	Missing argument 2 for OC\Files\Storage\Wrapper\Wrapper::file_put_contents(), called in /www/lib/private/Files/View.php on line 1124 and defined at /www/lib/private/Files/Storage/Wrapper/Wrapper.php#250	a few seconds ago
Error	encryption	Encryption Library:error:02001002:system library:fopen:No such file or directory	a few seconds ago
Error	encryption	Encryption library couldn't export users private key, please check your servers OpenSSL configuration."no user given"	a few seconds ago
Error	encryption	Encryption library openssl_pkey_new() fails: error:2006D080:BIO routines:BIO_new_file:no such file	a few seconds ago
Error	encryption	Encryption Library couldn't generate users key-pair for "no user given"	a few seconds ago
Error	encryption	Encryption Library, symmetrical encryption failed no content given	a minute ago
Error	PHP	Undefined variable: data at /www/lib/private/Files/Storage/Wrapper/Wrapper.php#251	a minute ago
Error	PHP	Missing argument 2 for OC\Files\Storage\Wrapper\Wrapper::file_put_contents(), called in /www/lib/private/Files/View.php on line 1124 and defined at /www/lib/private/Files/Storage/Wrapper/Wrapper.php#250	a minute ago
Error	encryption	Encryption Library:error:02001002:system library:fopen:No such file or directory	a minute ago
Error	encryption	Encryption library couldn't export users private key, please check your servers OpenSSL configuration.admin	a minute ago
Error	encryption	Encryption library openssl_pkey_new() fails: error:2006D080:BIO routines:BIO_new_file:no such file	a minute ago
Error	encryption	Encryption Library couldn't generate users key-pair for ADMINUSER	a minute ago [NOTE: Admin already has a key pair, no idea why it tries to regenerate it]
Info	admin_audit	User created: "ghut"