I use Nextcloud(18 currently). Everything was working fine until this day when i got this message when accessing my cloud via the webbrowser.
Internal Server Error The server encountered an internal error and was unable to complete your request. Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report. More details can be found in the server log.
I am using docker, or more specific docker compose for almost 2 years now. fpm alpine, splitted into 5 parts: mariadb, redis, app, web, and cron.
version: '3'
services:
nextcloud_db:
image: mariadb
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- nextcloud_db:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=
env_file:
- db.env
nextcloud_redis:
image: redis:alpine
restart: always
nextcloud_app:
build: ./app
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- nextcloud:/var/www/html
environment:
- MYSQL_HOST=nextcloud_db
- REDIS_HOST=nextcloud_redis
env_file:
- db.env
depends_on:
- nextcloud_db
- nextcloud_redis
nextcloud_web:
build: ./web
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- nextcloud:/var/www/html:ro
environment:
- VIRTUAL_HOST=
- LETSENCRYPT_HOST=
depends_on:
- nextcloud_app
nextcloud_cron:
image: nextcloud:fpm-alpine
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- nextcloud:/var/www/html
entrypoint: /cron.sh
depends_on:
- nextcloud_db
- nextcloud_redis
volumes:
nextcloud_db:
nextcloud:
The mariadb one has an error currently.
nextcloud_db_1 | 2020-05-15 01:28:44+02:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.4.13+maria~bionic started.
nextcloud_db_1 | 2020-05-15 01:28:49+02:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
nextcloud_db_1 | 2020-05-15 01:28:49+02:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.4.13+maria~bionic started.
nextcloud_db_1 | 2020-05-15 1:28:50 0 [Note] mysqld (mysqld 10.4.13-MariaDB-1:10.4.13+maria~bionic) starting as process 1 ...
nextcloud_db_1 | 2020-05-15 1:28:50 0 [Warning] You need to use --log-bin to make --binlog-format work.
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: Using Linux native AIO
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: Uses event mutexes
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: Number of pools: 1
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: Using SSE2 crc32 instructions
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: Completed initialization of buffer pool
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
nextcloud_db_1 | 2020-05-15 1:28:51 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=8312011756
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] InnoDB: Creating shared tablespace for temporary tables
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] InnoDB: Waiting for purge to start
nextcloud_db_1 | 2020-05-15 01:28:52 0x7f010affd700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.13/storage/innobase/include/fut0lst.ic line 67
nextcloud_db_1 | InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
nextcloud_db_1 | InnoDB: We intentionally generate a memory trap.
nextcloud_db_1 | InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
nextcloud_db_1 | InnoDB: If you get repeated assertion failures or crashes, even
nextcloud_db_1 | InnoDB: immediately after the mysqld startup, there may be
nextcloud_db_1 | InnoDB: corruption in the InnoDB tablespace. Please refer to
nextcloud_db_1 | InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
nextcloud_db_1 | InnoDB: about forcing recovery.
nextcloud_db_1 | 200515 1:28:52 [ERROR] mysqld got signal 6 ;
nextcloud_db_1 | This could be because you hit a bug. It is also possible that this binary
nextcloud_db_1 | or one of the libraries it was linked against is corrupt, improperly built,
nextcloud_db_1 | or misconfigured. This error can also be caused by malfunctioning hardware.
nextcloud_db_1 |
nextcloud_db_1 | To report this bug, see https://mariadb.com/kb/en/reporting-bugs
nextcloud_db_1 |
nextcloud_db_1 | We will try our best to scrape up some info that will hopefully help
nextcloud_db_1 | diagnose the problem, but since we have already crashed,
nextcloud_db_1 | something is definitely wrong and this may fail.
nextcloud_db_1 |
nextcloud_db_1 | Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic
nextcloud_db_1 | key_buffer_size=134217728
nextcloud_db_1 | read_buffer_size=2097152
nextcloud_db_1 | max_used_connections=0
nextcloud_db_1 | max_threads=102
nextcloud_db_1 | thread_count=4
nextcloud_db_1 | It is possible that mysqld could use up to
nextcloud_db_1 | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760255 K bytes of memory
nextcloud_db_1 | Hope that's ok; if not, decrease some variables in the equation.
nextcloud_db_1 |
nextcloud_db_1 | Thread pointer: 0x7f00fc000c08
nextcloud_db_1 | Attempting backtrace. You can use the following information to find out
nextcloud_db_1 | where mysqld died. If you see no messages after this, something went
nextcloud_db_1 | terribly wrong...
nextcloud_db_1 | stack_bottom = 0x7f010affcc08 thread_stack 0x49000
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] InnoDB: 10.4.13 started; log sequence number 8312011765; transaction id 20582626
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] InnoDB: !!! innodb_force_recovery is set to 1 !!!
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
nextcloud_db_1 | 2020-05-15 1:28:52 0 [Note] Plugin 'FEEDBACK' is disabled.
nextcloud_db_1 | mysqld(my_print_stacktrace+0x2e)[0x561046981e8e]
nextcloud_db_1 | mysqld(handle_fatal_signal+0x515)[0x5610463fd915]
nextcloud_db_1 | /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f0148c84890]
nextcloud_db_1 | /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f01475b8e97]
nextcloud_db_1 | /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f01475ba801]
nextcloud_db_1 | mysqld(+0x57c53c)[0x56104610a53c]
nextcloud_db_1 | mysqld(+0xb35506)[0x5610466c3506]
nextcloud_db_1 | mysqld(+0xb35d5d)[0x5610466c3d5d]
nextcloud_db_1 | mysqld(+0xb39a55)[0x5610466c7a55]
nextcloud_db_1 | mysqld(+0xb1c16e)[0x5610466aa16e]
nextcloud_db_1 | /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f0148c796db]
nextcloud_db_1 | /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f014769b88f]
nextcloud_db_1 |
nextcloud_db_1 | Trying to get some variables.
nextcloud_db_1 | Some pointers may be invalid and cause the dump to abort.
nextcloud_db_1 | Query (0x0):
nextcloud_db_1 | Connection ID (thread ID): 2
nextcloud_db_1 | Status: NOT_KILLED
nextcloud_db_1 |
nextcloud_db_1 | Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
nextcloud_db_1 |
nextcloud_db_1 | The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
nextcloud_db_1 | information that should help you find out what is causing the crash.
nextcloud_db_1 | Writing a core file...
nextcloud_db_1 | Working directory at /var/lib/mysql
nextcloud_db_1 | Resource Limits:
nextcloud_db_1 | Limit Soft Limit Hard Limit Units
nextcloud_db_1 | Max cpu time unlimited unlimited seconds
nextcloud_db_1 | Max file size unlimited unlimited bytes
nextcloud_db_1 | Max data size unlimited unlimited bytes
nextcloud_db_1 | Max stack size 8388608 unlimited bytes
nextcloud_db_1 | Max core file size unlimited unlimited bytes
nextcloud_db_1 | Max resident set unlimited unlimited bytes
nextcloud_db_1 | Max processes unlimited unlimited processes
nextcloud_db_1 | Max open files 1048576 1048576 files
nextcloud_db_1 | Max locked memory 16777216 16777216 bytes
nextcloud_db_1 | Max address space unlimited unlimited bytes
nextcloud_db_1 | Max file locks unlimited unlimited locks
nextcloud_db_1 | Max pending signals 1545098 1545098 signals
nextcloud_db_1 | Max msgqueue size 819200 819200 bytes
nextcloud_db_1 | Max nice priority 0 0
nextcloud_db_1 | Max realtime priority 0 0
nextcloud_db_1 | Max realtime timeout unlimited unlimited us
nextcloud_db_1 | Core pattern: co...
nextcloud_db_1 | Fatal signal 11 while backtracing
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=0
max_threads=102
thread_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760255 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fa650000c08
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fa674ff0c08 thread_stack 0x49000
2020-05-15 0:01:32 0 [Note] InnoDB: 10.4.13 started; log sequence number 8312011765; transaction id 20582626
2020-05-15 0:01:32 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
mysqld(my_print_stacktrace+0x2e)[0x55e945e98e8e]
mysqld(handle_fatal_signal+0x515)[0x55e945914915]
2020-05-15 0:01:32 0 [Note] Plugin 'FEEDBACK' is disabled.
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fa6aeddc890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fa6ad710e97]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fa6ad712801]
mysqld(+0x57c53c)[0x55e94562153c]
mysqld(+0xb35506)[0x55e945bda506]
mysqld(+0xb35d5d)[0x55e945bdad5d]
mysqld(+0xb39a55)[0x55e945bdea55]
mysqld(+0xb1c16e)[0x55e945bc116e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fa6aedd16db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fa6ad7f388f]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 4
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 1048576 1048576 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 1545098 1545098 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: co...
Fatal signal 11 while backtracing
this is in the docker logs. this sounds way too complicated for me and i have no clue.
I have not made any changes to the setup. As I said i am using docker for almost 2 years, with some changes made to the docker-compose.yml file over the time. But this issue just appeared and i havent changed anything for about 3 or 4 months.
Is it possible, that it has something to do with a big database? I am using Nextcloud News and do not purge any items because i use them for machine learning data processing. So the database might be a little big. After amysqldump the oc_news_items table is about 170MB in size and the last automated dump (this morning) doesnt seem corrupted or something like this, but it has suspuciously 99991 rows. So maybe there might be a problem with over 99999 rows of nextcloud news items in the db? Or maybe only coincidence.
Anyways I need my system working. It is in productive use.
let me know if you know anything related to this problem or if I should provide further data or what i should look for to get this fixed. Thank you.