EDIT: Nextcloud un-reachable (The connection has timed out) - ONLY ON HOME NETWORK

Nextcloud version (eg, 20.0.5): 23.0.4
Operating system and version (eg, Ubuntu 20.04): Ubuntu 20.04.4
Apache or nginx version (eg, Apache 2.4.25): nginx/1.18.0
PHP version (eg, 7.4): 8.0.19

Just since this morning I cannot connect to my nextcloud via the browser or client. In the browser I get “The connection has timed out” in the desktop client it indicates “Offline”. The Android client just hangs with the spinning circle.

A strange (to me) thing is that my Uptime Doctor email notifications show that it is UP occationally. The most resent notification told me that the site was DOWN for 01hr 15min, but when I attemt to connect (as soon as I got the email notifiation) I still got the “The connection has timed out” message.

EDIT: Another strange thing is I can connect with ssllabs.com and get an “A” rating. But with a warning that " HTTP request to this server failed, see below for details."
The detail is that:
https://www.domain.net/ (Request failed)

Just to be sure I checked that my ports were still open using ‘nmap localhost’ and that does not seem to be a problem:

Starting Nmap 7.80 ( https://nmap.org ) at 2022-06-06 13:13 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0000080s latency).
Other addresses for localhost (not scanned): ::1
Not shown: 991 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
80/tcp    open  http
443/tcp   open  https
631/tcp   open  ipp
3306/tcp  open  mysql
3551/tcp  open  apcupsd
4000/tcp  open  remoteanything
7001/tcp  open  afs3-callback
20000/tcp open  dnp

Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. navigate to my nextcloud domain, or open a nextcloud client

The output of your Nextcloud log in Admin > Logging: Since I can’t access the instance I output occ log:tail instead, listing only events since yesterday (when everything was working just fine).

------- ------ ---------------------------------------------------------------------------- --------------------------- 
  Level   App    Message                                                                      Time                       
 ------- ------ ---------------------------------------------------------------------------- ---------------------------                                                  
                                                                                                                         
  Error   dav    Sabre\VObject\Recur\NoInstancesException:                                    2022-06-05T04:06:38+00:00  
                 This recurrence rule does not generate any                                                              
                 valid instances at                                                                                      
                 3rdparty/sabre/vobject/lib/Recur/EventIterator.php                                                      
                 line 192                                                                                                
                                                                                                                         
                 0. .../CalDavBackend.php line 2797                                                                      
                    Sabre\VObject\Recur\EventIterator->__construct(                                                      
                                                                                                                         
                    )                                                                                                    
                 1. .../CalDavBackend.php line 1260                                                                      
                    OCA\DAV\CalDAV\CalDavBackend->getDenormalizedData(                                                   
                                                                                                                         
                    )                                                                                                    
                 2. .../RefreshWebcalService.php line 144                                                                
                    OCA\DAV\CalDAV\CalDavBackend->createCalendarObject(                                                  
                                                                                                                         
                    )                                                                                                    
                 3. .../RefreshWebcalJob.php line 115                                                                    
                    OCA\DAV\CalDAV\WebcalCaching\RefreshWebcalService->refreshSubscription(                              
                                                                                                                         
                    )                                                                                                    
                 4. .../Job.php line 79                                                                                  
                    OCA\DAV\BackgroundJob\RefreshWebcalJob->run(                                                         
                                                                                                                         
                    )                                                                                                    
                 5. .../RefreshWebcalJob.php line 108                                                                    
                    OCP\BackgroundJob\Job->execute(                                                                      
                                                                                                                         
                    )                                                                                                    
                 6. cron.php line 150                                                                                    
                    OCA\DAV\BackgroundJob\RefreshWebcalJob->execute(                                                     
                                                                                                                         
                    )                                                                                                    
                                                                                                                         
  Error   dav    Sabre\VObject\Recur\NoInstancesException:                                    2022-06-06T04:13:36+00:00  
                 This recurrence rule does not generate any                                                              
                 valid instances at                                                                                      
                 3rdparty/sabre/vobject/lib/Recur/EventIterator.php                                                      
                 line 192                                                                                                
                                                                                                                         
                 0. .../CalDavBackend.php line 2797                                                                      
                    Sabre\VObject\Recur\EventIterator->__construct(                                                      
                                                                                                                         
                    )                                                                                                    
                 1. .../CalDavBackend.php line 1260                                                                      
                    OCA\DAV\CalDAV\CalDavBackend->getDenormalizedData(                                                   
                                                                                                                         
                    )                                                                                                    
                 2. .../RefreshWebcalService.php line 144                                                                
                    OCA\DAV\CalDAV\CalDavBackend->createCalendarObject(                                                  
                                                                                                                         
                    )                                                                                                    
                 3. .../RefreshWebcalJob.php line 115                                                                    
                    OCA\DAV\CalDAV\WebcalCaching\RefreshWebcalService->refreshSubscription(                              
                                                                                                                         
                    )                                                                                                    
                 4. .../Job.php line 79                                                                                  
                    OCA\DAV\BackgroundJob\RefreshWebcalJob->run(                                                         
                                                                                                                         
                    )                                                                                                    
                 5. .../RefreshWebcalJob.php line 108                                                                    
                    OCP\BackgroundJob\Job->execute(                                                                      
                                                                                                                         
                    )                                                                                                    
                 6. cron.php line 150                                                                                    
                    OCA\DAV\BackgroundJob\RefreshWebcalJob->execute(                                                     
                                                                                                                         
                    )                                                                                                    
                                                                                                                         
  Error   cron   Doctrine\DBAL\Exception\DriverException: An                                  2022-06-06T15:30:46+00:00  
                 exception occurred while executing a query:                                                             
                 SQLSTATE[70100]: <<Unknown error>>: 1927                                                                
                 Connection was killed at                                                                                
                 .../dbal/src/Driver/API/MySQL/ExceptionConverter.php                                                    
                 line 114                                                                                                
                                                                                                                         
                 0. .../Connection.php line 1780                                                                         
                    Doctrine\DBAL\Driver\API\MySQL\ExceptionConverter->convert(                                          
                                                                                                                         
                    )                                                                                                    
                 1. .../Connection.php line 1719                                                                         
                    Doctrine\DBAL\Connection->handleDriverException(                                                     
                                                                                                                         
                    )                                                                                                    
                 2. .../Connection.php line 1067                                                                         
                    Doctrine\DBAL\Connection->convertExceptionDuringQuery(                                               
                                                                                                                         
                    )                                                                                                    
                 3. .../Connection.php line 237                                                                          
                    Doctrine\DBAL\Connection->executeQuery(                                                              
                                                                                                                         
                    )                                                                                                    
                 4. .../QueryBuilder.php line 345                                                                        
                    OC\DB\Connection->executeQuery(                                                                      
                                                                                                                         
                    )                                                                                                    
                 5. .../QueryBuilder.php line 287                                                                        
                    Doctrine\DBAL\Query\QueryBuilder->execute(                                                           
                                                                                                                         
                    )                                                                                                    
                 6. .../JobList.php line 211                                                                             
                    OC\DB\QueryBuilder\QueryBuilder->execute(                                                            
                                                                                                                         
                    )                                                                                                    
                 7. cron.php line 144                                                                                    
                    OC\BackgroundJob\JobList->getNext(                                                                   
                                                                                                                         
                    )                                                                                                    
                                                                                                                         
                 Caused by Doctrine\DBAL\Driver\PDO\Exception:                                                           
                 SQLSTATE[70100]: <<Unknown error>>: 1927                                                                
                 Connection was killed at                                                                                
                 3rdparty/doctrine/dbal/src/Driver/PDO/Exception.php                                                     
                 line 26                                                                                                 
                                                                                                                         
                 0. .../Statement.php line 94                                                                            
                    Doctrine\DBAL\Driver\PDO\Exception::new(                                                             
                                                                                                                         
                    )                                                                                                    
                 1. .../Connection.php line 1057                                                                         
                    Doctrine\DBAL\Driver\PDO\Statement->execute(                                                         
                                                                                                                         
                    )                                                                                                    
                 2. .../Connection.php line 237                                                                          
                    Doctrine\DBAL\Connection->executeQuery(                                                              
                                                                                                                         
                    )                                                                                                    
                 3. .../QueryBuilder.php line 345                                                                        
                    OC\DB\Connection->executeQuery(                                                                      
                                                                                                                         
                    )                                                                                                    
                 4. .../QueryBuilder.php line 287                                                                        
                    Doctrine\DBAL\Query\QueryBuilder->execute(                                                           
                                                                                                                         
                    )                                                                                                    
                 5. .../JobList.php line 211                                                                             
                    OC\DB\QueryBuilder\QueryBuilder->execute(                                                            
                                                                                                                         
                    )                                                                                                    
                 6. cron.php line 144                                                                                    
                    OC\BackgroundJob\JobList->getNext(                                                                   
                                                                                                                         
                    )                                                                                                    
                                                                                                                         
                 Caused by PDOException: SQLSTATE[70100]: <<Unknown                                                      
                 error>>: 1927 Connection was killed at                                                                  
                 3rdparty/doctrine/dbal/src/Driver/PDO/Statement.php                                                     
                 line 92                                                                                                 
                                                                                                                         
                 0. .../Statement.php line 92                                                                            
                    PDOStatement->execute(                                                                               
                                                                                                                         
                    )                                                                                                    
                 1. .../Connection.php line 1057                                                                         
                    Doctrine\DBAL\Driver\PDO\Statement->execute(                                                         
                                                                                                                         
                    )                                                                                                    
                 2. .../Connection.php line 237                                                                          
                    Doctrine\DBAL\Connection->executeQuery(                                                              
                                                                                                                         
                    )                                                                                                    
                 3. .../QueryBuilder.php line 345                                                                        
                    OC\DB\Connection->executeQuery(                                                                      
                                                                                                                         
                    )                                                                                                    
                 4. .../QueryBuilder.php line 287                                                                        
                    Doctrine\DBAL\Query\QueryBuilder->execute(                                                           
                                                                                                                         
                    )                                                                                                    
                 5. .../JobList.php line 211                                                                             
                    OC\DB\QueryBuilder\QueryBuilder->execute(                                                            
                                                                                                                         
                    )                                                                                                    
                 6. cron.php line 144                                                                                    
                    OC\BackgroundJob\JobList->getNext(                                                                   
                                                                                                                         
                    )                                                                                                    
                                                                                                                         
 ------- ------ ---------------------------------------------------------------------------- --------------------------- 

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'x',
  'passwordsalt' => 'x',
  'secret' => 'x',
  'trusted_domains' => 
  array (
    0 => 'www.domain.net',
  ),
  'datadirectory' => '/usr/share/nginx/nextcloud-data',
  'default_phone_region' => 'CA',
  'dbtype' => 'mysql',
  'version' => '23.0.4.1',
  'overwrite.cli.url' => 'https://www.domain.net',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost:3306',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextclouduser',
  'dbpassword' => 'x',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'mail_from_address' => 'domain',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'startmail.com',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'smtp.mailserver.com',
  'mail_smtpport' => '465',
  'mail_smtpname' => 'user@mailserver.com',
  'mail_smtppassword' => 'x',
  'mail_smtpsecure' => 'ssl',
  'maintenance' => true,
  'updater.release.channel' => 'stable',
  'theme' => '',
  'loglevel' => 2,
  'app_install_overwrite' => 
  array (
    0 => 'cookbook',
    1 => 'radio',
    2 => 'podcast',
    3 => 'apporder',
    4 => 'flow_notifications',
    5 => 'files_mindmap',
    6 => 'integration_whiteboard',
  ),
);

The output of your Apache/nginx/system log in /var/log/nginx/error.log:

2022/06/06 10:07:05 [notice] 1908020#1908020: signal process started
2022/06/06 10:07:07 [notice] 1908029#1908029: signal process started
2022/06/06 11:31:23 [warn] 846#846: "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/etc/letsencrypt/live/collabora.dysoncloud.net/fullchain.pem"
2022/06/06 11:31:23 [warn] 846#846: "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/etc/letsencrypt/live/www.dysoncloud.net/fullchain.pem"
2022/06/06 11:31:23 [warn] 1020#1020: "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/etc/letsencrypt/live/collabora.dysoncloud.net/fullchain.pem"
2022/06/06 11:31:23 [warn] 1020#1020: "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/etc/letsencrypt/live/www.dysoncloud.net/fullchain.pem"
2022/06/06 11:44:53 [notice] 21140#21140: signal process started

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

The results of ‘tail /var/log/nginx/nextcloud.error’

2022/05/26 18:41:11 [error] 1789541#1789541: *19070 access forbidden by rule, client: 3.23.128.99, server: www.domain.net, request: "GET /.git/config HTTP/1.1", host: "www.domain.net"
2022/05/28 12:57:38 [error] 1803283#1803283: *7120 access forbidden by rule, client: 3.22.188.50, server: www.domain.net, request: "GET /.git/config HTTP/1.1", host: "www.domain.net"
2022/06/03 02:58:09 [error] 1873191#1873191: *23035 access forbidden by rule, client: 20.230.179.180, server: www.domain.net, request: "GET /.env HTTP/1.1", host: "www.domain.net"
2022/06/06 11:29:32 [error] 1908030#1908030: *612 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 185.10.57.32, server: www.domain.net, request: "HEAD / HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.0-fpm.sock:", host: "www.domain.net", referrer: "http://www.uptimedoctor.com/monitored-by-user-dysonsphere"
2022/06/06 11:30:13 [error] 1908030#1908030: *614 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 67.225.218.51, server: www.domain.net, request: "HEAD / HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.0-fpm.sock:", host: "www.domain.net", referrer: "http://www.uptimedoctor.com/monitored-by-user-dysonsphere"
2022/06/06 12:10:27 [error] 43534#43534: *7 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Call to a member function getRequest() on null in /usr/share/nginx/nextcloud/remote.php:52
Stack trace:
#0 /usr/share/nginx/nextcloud/remote.php(170): handleException()
#1 {main}
  thrown in /usr/share/nginx/nextcloud/remote.php on line 52" while reading response header from upstream, client: 52.96.51.157, server: www.domain.net, request: "GET /remote.php/dav/public-calendars/TWrHTToarfPR9MxA?export HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.0-fpm.sock:", host: "www.domain.net"

UPDATE:
I called my ISP to troubleshoot and it turns out that connection to the nextcloud is possible from OUTSIDE my home network, either on mobile data, an outside internet connection, or a VPN.

There was an update to the router initiated by the ISP and since then the nextcloud is not accessible from within the home network.

So My ISP’s Modem/Router will not let me configure hairpin NAT or DNS servers! So I cannot use the advice found here: https://help.nextcloud.com/t/cannot-access-nextcloud-from-local-network-but-works-from-other-networks/93122.
Looks like I will set the router to bridge mode and use a decent router to control my network.