Cannot allocate Memory for Bufferpool (MariaDB) on fresh Raspberry Pi 4 (8GB)

Hi, Im trying to setup nextcloud on my brandnew Raspberry Pi 8GB but getting an error on the initial setup after nc-limit on the restart of the maria DB telling me that it can not allocate memory.

I have installed Hypriot on my fresh RPI4 and entered following command for setting up the container:
docker run -d -p 4443:4443 -p 443:443 -p 80:80 -v /mnt/CarazzaIII/nextcloud:/data --name nextcloudpi ownyourbits/nextcloudpi-armhf $192.168.2.100

Here is the log im getting from it:


Initializing empty volume..
Making /usr/local/etc/ncp-config.d persistent ...
Making /etc/services-enabled.d persistent ...
Making /etc/letsencrypt persistent ...
Making /etc/shadow persistent ...
Making /etc/cron.d persistent ...
Making /etc/cron.daily persistent ...
Making /etc/cron.hourly persistent ...
Making /etc/cron.weekly persistent ...
Making /usr/local/bin persistent ...
Making /etc/apache2 persistent ...
Starting PHP-fpm
Starting Apache
Starting mariaDB
2020-07-12  5:21:09 0 [Note] mysqld (mysqld 10.3.22-MariaDB-0+deb10u1) starting as process 101 ...
Starting Redis
Starting Cron
Starting Postfix
Uninitialized instance, running nc-init...
Running nc-init
Setting up a clean Nextcloud instance... wait until message 'NC init done'
Setting up database...
Setting up Nextcloud...
Nextcloud was successfully installed
Set mode for background jobs to 'cron'
System config value tempdirectory set to string /var/www/nextcloud/data/tmp
System config value mysql.utf8mb4 set to boolean true
System config value trusted_domains => 11 set to string 87.102.170.96
System config value trusted_domains => 1 set to string 172.17.0.2
System config value trusted_domains => 5 set to string nextcloudpi.local
System config value trusted_domains => 7 set to string nextcloudpi
System config value trusted_domains => 8 set to string nextcloudpi.lan
System config value mail_smtpmode set to string sendmail
System config value mail_smtpauthtype set to string LOGIN
System config value mail_from_address set to string admin
System config value mail_domain set to string ownyourbits.com
nextcloudpi enabled
calendar installed
calendar enabled
contacts installed
contacts enabled
notes installed
notes enabled
tasks installed
tasks enabled
news installed
news enabled
calendar enabled
contacts enabled
notes enabled
tasks enabled
news enabled
previewgenerator enabled
Config value squareSizes for app previewgenerator set to 32 256
Config value widthSizes for app previewgenerator set to 256 384
Config value heightSizes for app previewgenerator set to 256
System config value preview_max_x set to string 2048
System config value preview_max_y set to string 2048
System config value jpeg_quality set to string 60
Config value jpeg_quality for app preview set to 60
System config value overwriteprotocol set to string https
Following columns will be updated:

* mounts.storage_id
* mounts.root_id
* mounts.mount_id

This can take up to hours, depending on the number of files in your instance!
Check indices of the share table.
Check indices of the filecache table.
Check indices of the twofactor_providers table.
Check indices of the login_flow_v2 table.
Check indices of the whats_new table.
Check indices of the cards table.
Check indices of the cards_properties table.
Check indices of the calendarobjects_props table.
Adding calendarobject_calid_index index to the calendarobjects_props table, this can take some time...
calendarobjects_props table updated successfully.
Check indices of the schedulingobjects table.
Adding schedulobj_principuri_index index to the schedulingobjects table, this can take some time...
schedulingobjects table updated successfully.
NC init done
System config value trusted_domains => 1 set to string 172.17.0.2
System config value trusted_domains => 6 set to string 92.168.2.100
Provisioning
Provisioning Redis password
Updating NextCloud config with Redis password
Provisioning MariaDB password
Updating NextCloud config with MariaDB password
Running nc-limits
Using 6304066560B for PHP
Using 4 PHP threads
Using 3362168832 memory for the database
Stopping MariaDB database server: mysqld.
Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
Stopping redis-server: redis-server.
Starting redis-server: redis-server.
Init done

And here is the log i get from MariaDB startup when i try to manually restart:


2020-07-12  5:28:52 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-forma
t/
2020-07-12  5:28:52 0 [Note] InnoDB: Using Linux native AIO
2020-07-12  5:28:52 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-07-12  5:28:52 0 [Note] InnoDB: Uses event mutexes
2020-07-12  5:28:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-07-12  5:28:52 0 [Note] InnoDB: Number of pools: 1
2020-07-12  5:28:52 0 [Note] InnoDB: Using generic crc32 instructions
2020-07-12  5:28:52 0 [Note] InnoDB: Initializing buffer pool, total size = 3.25G, instances = 1, chunk size = 128M
2020-07-12  5:28:52 0 [ERROR] InnoDB: mmap(136118272 bytes) failed; errno 12
2020-07-12  5:28:52 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2020-07-12  5:28:52 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-07-12  5:28:52 0 [Note] InnoDB: Starting shutdown...
double free or corruption (out)
200712  5:28:53 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.3.22-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 466218 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /data/database
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             unlimited            unlimited            processes
Max open files            1048576              1048576              files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       63600                63600                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: core

I have tried to reduce the amount on memory but it resets the value after reboot. Also i tried to reduce the max memory with nc-limits also with no success. Furthermore i tried to update to the latest version hoping there was a bugfix (docker exec -it nextcloudpi ncp-update).

Im now googling for a solution since 2 days. I have found an entry in the github issue-Page with the exact same problem so i think im not the only one with that problem, but there is not a good solution for it.
https://github.com/nextcloud/nextcloudpi/issues/1141
SD Card is also new and im using the official Pi4 power supply

So now im out of ideas an I hope one of you guys can help me out here

Big thank in advance for helping me

How is that volume mounted?
output of:

df -lh
and
cat /etc/fstab

Also after init done, check dmesg and look for input/output errors.
Unfamiliar with Hypriot, I assume other docker containers run without errors? if not check with docker run hello-world

Hi Oliver, thanks alot for replying:
Here are the outputs you requested:

df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/root 112G 2.9G 105G 3% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
tmpfs 4.0G 8.6M 4.0G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 4.0G 0 4.0G 0% /sys/fs/cgroup
/dev/mmcblk0p1 64M 52M 13M 81% /boot
/dev/sda1 13T 5.0T 7.1T 41% /mnt/CarazzaIII
tmpfs 802M 0 802M 0% /run/user/1000

$ cat /etc/fstab
proc /proc proc defaults 0 0
PARTUUID=96ce6a0b-01 /boot vfat defaults 0 0
PARTUUID=96ce6a0b-02 / ext4 defaults,noatime 0 1
UUID=“9b8c4392-634b-4d90-b55d-8e570279e459” /mnt/CarazzaIII ext4 defaults 0 2

dmesg output

[Mon Jul 13 16:52:38 2020] docker0: port 1(veth5cc1bfb) entered disabled state
[Mon Jul 13 16:52:38 2020] veth7c6ff2d: renamed from eth0
[Mon Jul 13 16:52:38 2020] docker0: port 1(veth5cc1bfb) entered disabled state
[Mon Jul 13 16:52:38 2020] device veth5cc1bfb left promiscuous mode
[Mon Jul 13 16:52:38 2020] docker0: port 1(veth5cc1bfb) entered disabled state
[Mon Jul 13 16:54:07 2020] docker0: port 1(vethd874eec) entered blocking state
[Mon Jul 13 16:54:07 2020] docker0: port 1(vethd874eec) entered disabled state
[Mon Jul 13 16:54:07 2020] device vethd874eec entered promiscuous mode
[Mon Jul 13 16:54:07 2020] IPv6: ADDRCONF(NETDEV_UP): vethd874eec: link is not ready
[Mon Jul 13 16:54:08 2020] eth0: renamed from vethd3127aa
[Mon Jul 13 16:54:08 2020] IPv6: ADDRCONF(NETDEV_CHANGE): vethd874eec: link becomes ready
[Mon Jul 13 16:54:08 2020] docker0: port 1(vethd874eec) entered blocking state
[Mon Jul 13 16:54:08 2020] docker0: port 1(vethd874eec) entered forwarding state

I have 2 other conainter running on it without problems.
Could the problem be related to the fact, that hypriot is running in 32bit? I heard that when running linux on 32bit (with pae) it can only allocate 3GB memory. If thats the case is it possible to redce the amount of memory nextcloud sets for innodb-buffer-pool? I tried with nc-limit without success.

I tried a memtest an also there it can not allocate more than 2.8GB:

$ free
total used free shared buff/cache available
Mem: 8208420 222428 5201644 11652 2784348 7721468
Swap: 0 0 0

$ sudo memtester 3300 1
memtester version 4.3.0 (32-bit)
Copyright © 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffff000
want 3300MB (3460300800 bytes)
got  2896MB (3037020160 bytes), trying mlock ...locked.
Loop 1/1:
  Stuck Address       : ok
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : ok

Can try unmounting the drive

sudo umount /dev/sda1

edit fstab replace

defaults 0 2

with

rw,users 0 0

mount

sudo mount -a

or reboot
check with df -lh if mounted properly

remove the container and start a new one

I removed the container changed fstab remounted the device.
Now i get another error:

$ docker logs -f nextcloudpi
Initializing empty volume..
Making /usr/local/etc/ncp-config.d persistent ...
Making /etc/services-enabled.d persistent ...
Making /etc/letsencrypt persistent ...
Making /etc/shadow persistent ...
Making /etc/cron.d persistent ...
Making /etc/cron.daily persistent ...
Making /etc/cron.hourly persistent ...
Making /etc/cron.weekly persistent ...
Making /usr/local/bin persistent ...
/run-parts.sh: line 47: /etc/services-enabled.d/010lamp: Permission denied
/run-parts.sh: line 47: /etc/services-enabled.d/020nextcloud: Permission denied
Init done

Here output of df lh after i changed fstab an mounted the device

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       112G  2.8G  105G   3% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           4.0G     0  4.0G   0% /dev/shm
tmpfs           4.0G  8.6M  4.0G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           4.0G     0  4.0G   0% /sys/fs/cgroup
/dev/mmcblk0p1   64M   52M   13M  81% /boot
tmpfs           802M     0  802M   0% /run/user/1000
/dev/sda1        13T  5.0T  7.1T  41% /mnt/CarazzaIII

I change the entry in fstab to
UUID="9b8c4392-634b-4d90-b55d-8e570279e459" /mnt/CarazzaIII ext4 rw,users,exec 0 0
Now i get the same error MariaDB error again

Just in case someone else has the same problem.
It seems that i have a workaround.

After Init Done with the failed startup of MariaDB message do following:

stop the containter
sudo docker stop nextcloudpi

And remove it:
sudo docker rm nextcloudpi

Change the nc-limits.sh script in the data-directory under bin/ncp/CONFIG by replacing the line under DATABASE MEMORY
AUTOMEM=$(( TOTAL_MEM * 40 / 100 ))
with
AUTOMEM=$(( TOTAL_MEM * 20 / 100 ))

Then Recreate the container (with the same data-directory without deleting the content!)
docker run -d -p 4443:4443 -p 443:443 -p 80:80 -v {data-directory}:/data --name nextcloudpi ownyourbits/nextcloudpi-armhf ${DOMAIN}

I think limits.sh does not work for 32-Bit OS with 8GB memory cause it tries to allocate more than 3GB of memory which is not possible and leads to the MariaDB failing on startup.

For 64-Bit OS there should be no problem.
If there is a better solution please report

2 Likes

After spending hours trying to figure out my problem with my new Rasperry 4 8GB/RAM, your message solved my problem in two minutes.
Thank you! :grinning: :clap: :clap: :clap:

Happy i could help out.

Keep in mind its a Workaround and it has the drawback whenever nextcloud is updated it will overwrite the changes in the nc-limits.sh script. If thats the case and you restart the container the error will raise again.

You can fix that again by removing the container, overwriting the script an recreating the container like in the description above.

After getting the error and fixing it too many times, I just upgraded the kernel to 64-bit.