Nextcloud no more accessible: "ERR_CONNECTION_TIMED_OUT"

ℹ️ Support
, , ,
1

Hi and Happy new year!

My current setup

  • Nextcloud version: 25.0.2 (obtained via podman exec <nextcloud-container> printenv NEXTCLOUD_VERSON since my Nextcloud app is no more accessible)
  • *Operating system and version: Ubuntu 22.04.3
  • Podman version: 3.4.4
  • nginx-proxy version: nginx-proxy 1.0.3, based on nginx 1.23.2 (based on the logs from podman logs <nginx-proxy container>
  • PHP version: PHP 8.1.13 (obtained via podman exec <nextcloud-container> php -v)
  • Docker-compose file: pastebin link
  • Dockerfile for nextcloud-app:
FROM nextcloud:apache
RUN apt update && apt install -y smbclient libsmbclient-dev && rm -rf /var/lib/apt/lists/* 
RUN pecl install smbclient 
RUN docker-php-ext-enable smbclient
  • Dockerfile for nginx-proxy:
FROM nginxproxy/nginx-proxy:1.3-alpine
COPY uploadsize.conf /etc/nginx/conf.d/uploadsize.conf

The issue I am facing

  • Is this the first time you’ve seen this error?: Yes

  • Steps to replicate it:

    1. docker-compose up -d in the directory that contains the docker-compose file
    2. Try to get to the Nextcloud Web page (nc.mydomain.com): it results in ERR_CONNECTION_TIMED_OUT error from the browser.

  • Description of the issue
    It has been working well for months but, suddenly, without changing anything on purpose, Nextcloud is no more accessible from inside or outside the home network.
    The browser returns the error ERR_CONNECTION_TIMED_OUT.

Logs and traces

  • The output of your Nextcloud log in Admin > Logging: I can’t access the Web page.

  • The output of your config.php file: pastebin link

  • The output of your Apache/nginx/system log in /var/log/____: Logs obtained via podman logs <nextcloud-container> since apache2 logs are redirected to stdout and stderr:

Configuring Redis as session handler
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.89.1.5. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.89.1.5. Set the 'ServerName' directive globally to suppress this message
[Tue Jan 16 06:12:03.682341 2024] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.54 (Debian) PHP/8.1.13 configured -- resuming normal operations
[Tue Jan 16 06:12:03.682570 2024] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

  • Logs from nginx-proxy container (obtained via podman logs <nginx-proxy container>): pastebin link

  • Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors: pastebin link here (shortened to the past 2 weeks). I don´t know exactly when it stopped working but it should be less than 1 week.
    By looking at these logs, I suspect an issue with connection to the DB. But I’m not sure.

Thank you for your help!

2

Agreed. Check your db container status and logs. Your MariaDB server is offline or something.

3

Thank you @jtr .

I don’t see anything wrong in the db container:

2024-01-16 06:11:49+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.16+maria~ubu2004 started.
2024-01-16 06:11:50+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-01-16 06:11:50+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.16+maria~ubu2004 started.
2024-01-16 06:11:50+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2024-01-16  6:11:50 0 [Note] Starting MariaDB 10.6.16-MariaDB-1:10.6.16+maria~ubu2004 source revision b83c379420a8846ae4b28768d3c81fa354cca056 as process 1
2024-01-16  6:11:50 0 [Warning] You need to use --log-bin to make --binlog-format work.
2024-01-16  6:11:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-01-16  6:11:50 0 [Note] InnoDB: Number of pools: 1
2024-01-16  6:11:50 0 [Note] InnoDB: Using generic crc32 instructions
2024-01-16  6:11:50 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2024-01-16  6:11:50 0 [Note] InnoDB: Using Linux native AIO
2024-01-16  6:11:50 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2024-01-16  6:11:50 0 [Note] InnoDB: Completed initialization of buffer pool
2024-01-16  6:11:50 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=26478914723,26478914723
2024-01-16  6:11:53 0 [Note] InnoDB: To recover: 155 pages
2024-01-16  6:11:53 0 [Note] InnoDB: 128 rollback segments are active.
2024-01-16  6:11:53 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2024-01-16  6:11:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2024-01-16  6:11:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-01-16  6:11:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2024-01-16  6:11:53 0 [Note] InnoDB: 10.6.16 started; log sequence number 26487480441; transaction id 40429143
2024-01-16  6:11:53 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-01-16  6:11:53 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-01-16  6:11:53 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2024-01-16  6:11:53 0 [Note] Server socket created on IP: '0.0.0.0'.
2024-01-16  6:11:53 0 [Note] Server socket created on IP: '::'.
2024-01-16  6:11:53 0 [Note] InnoDB: Buffer pool(s) load completed at 240116  6:11:53
2024-01-16  6:11:53 0 [Note] mariadbd: ready for connections.
Version: '10.6.16-MariaDB-1:10.6.16+maria~ubu2004'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

Do you have any idea on how to analyze deeper?

4

Looking a bit closer, it looks like all the db errors are from your cron jobs / your cron container presumably and those go back several weeks (and are still ongoing).

The first error in your app container is one about not being able to access some folders under your user’s home folder (e.g. something under /var/www/html/data/jeremy.

What happens if you run occ config:list system or something like that within your app container?

I would also suggest trying to make an connection to your db server with the mariadb client from within both your cron and app containers to verify connectivity/etc - e.g. https://mariadb.com/docs/server/connect/clients/mariadb-client/

FROM nextcloud:apache

May not be related to your current problems, but your images are very fragile the way you have them:

  • Your Dockerfile doesn’t reference a particular version (tag). If anything was to trigger a pull your rebuilt image would break your entire environment because it would try to deploy NC28 over your NC25 environment
  • Your cron container has a similar problem
5

It seems to work as expected. The output is the config.php file (it seems correct):

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "*** REMOVED SENSITIVE VALUE***",
            "*** REMOVED SENSITIVE VALUE***",
            "192.168.1.16"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.2.3",
        "overwrite.cli.url": "*** REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "bulkupload.enabled": false,
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***"
    }
}

Thanks. Indeed, I change it.

7

Sorry, where do you see that?

8

How can I test the connection between the db container and the nextcloud container? ping command doesn’t work from the nextcloud container (“Operation not permitted”).

I’m really stuck and don’t even know where to look. Logs and errors don’t talk to me :confused:

Thanks a lot for your help.