Update to NC 21 - php out of memory ("killed")

Hi all,

I did an update of Ubuntu from 18 to 20.04 and from Nextcloud 18 to NC 19 to NC 20 and things seemed to be fine. Then I did the upgrade to NC 21 with the updater and it got stuck after executing the occ upgrade. it only stated “Killed” and went back to the prompt. When I try to get out of the maintenance mode , the same thing happens

sudo -u www-data php occ maintenance:mode --off
Killed

dmesg:

[ 591.696678] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 591.696816] [ 1162] 33 1162 399482 363495 3186688 0 0 php

[ 591.696821] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-4.scope,task=php,pid=1162,uid=33
[ 591.696857] Out of memory: Killed process 1162 (php) total-vm:1597928kB, anon-rss:1451624kB, file-rss:2356kB, shmem-rss:0kB, UID:33 pgtables:3112kB oom_score_adj:0
[ 591.806036] oom_reaper: reaped process 1162 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

I did try to re-run the updater, but it didn’t help
sudo -u www-data php /var/www/nextcloud/updater/updater.phar

Any idea how to get rid of the “killed” message and continue with the update? I’ve a VPS with 2GB of RAM (27% used). memcache set to local

I would also consider staying on NC 20, Ideally by simply canceling the update and not going through the restore process. Not sure if that is possible

thanks in advance.

[  570.182824] oom_reaper: reaped process 1089 (php7.4), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[  591.696366] mysqld invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[  591.696377] CPU: 0 PID: 799 Comm: mysqld Not tainted 5.4.0-72-generic #80-Ubuntu
[  591.696378] Hardware name:  KVM Server, BIOS VPS 
[  591.696384] Call Trace:
[  591.696440]  dump_stack+0x6d/0x8b
[  591.696457]  dump_header+0x4f/0x1eb
[  591.696459]  oom_kill_process.cold+0xb/0x10
[  591.696476]  out_of_memory.part.0+0x1df/0x3d0
[  591.696478]  out_of_memory+0x6d/0xd0
[  591.696486]  __alloc_pages_slowpath+0xd5e/0xe50
[  591.696489]  __alloc_pages_nodemask+0x2d0/0x320
[  591.696494]  alloc_pages_current+0x87/0xe0
[  591.696501]  __page_cache_alloc+0x72/0x90
[  591.696503]  pagecache_get_page+0xbf/0x300
[  591.696505]  filemap_fault+0x6b2/0xa50
[  591.696512]  ? unlock_page_memcg+0x12/0x20
[  591.696514]  ? page_add_file_rmap+0xff/0x1a0
[  591.696521]  ? xas_load+0xd/0x80
[  591.696523]  ? xas_find+0x17f/0x1c0
[  591.696525]  ? filemap_map_pages+0x24c/0x380
[  591.696546]  ext4_filemap_fault+0x32/0x50
[  591.696552]  __do_fault+0x3c/0x130
[  591.696554]  do_fault+0x24b/0x640
[  591.696555]  __handle_mm_fault+0x4c5/0x7a0
[  591.696557]  handle_mm_fault+0xca/0x200
[  591.696578]  do_user_addr_fault+0x1f9/0x450
[  591.696580]  __do_page_fault+0x58/0x90
[  591.696581]  do_page_fault+0x2c/0xe0
[  591.696586]  do_async_page_fault+0x39/0x70
[  591.696591]  async_page_fault+0x34/0x40
[  591.696598] RIP: 0033:0x55af54bedeb0
[  591.696604] Code: Bad RIP value.
[  591.696605] RSP: 002b:00007f838e54b5c8 EFLAGS: 00010202
[  591.696613] RAX: 000055af56023ca0 RBX: 0000000000000410 RCX: 00007f8370000bac
[  591.696614] RDX: 00007f8370005bd0 RSI: 0000000000000058 RDI: 0000000000000047
[  591.696615] RBP: 00007f838e54b600 R08: 00007f8370005bc0 R09: 0000000000001670
[  591.696616] R10: 0000000000000000 R11: 0000000000000286 R12: 00007f8370005be0
[  591.696616] R13: 0000000000000047 R14: 0000000000000058 R15: 0000000000000078
[  591.696618] Mem-Info:
[  591.696629] active_anon:474093 inactive_anon:4714 isolated_anon:0
                active_file:60 inactive_file:66 isolated_file:0
                unevictable:4634 dirty:0 writeback:0 unstable:0
                slab_reclaimable:5160 slab_unreclaimable:10256
                mapped:6788 shmem:4916 pagetables:1885 bounce:0
                free:3256 free_pcp:119 free_cma:0
[  591.696634] Node 0 active_anon:1896372kB inactive_anon:18856kB active_file:240kB inactive_file:264kB unevictable:18536kB isolated(anon):0kB isolated(file):0kB mapped:27152kB dirty:0kB writeback:0kB shmem:19664kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[  591.696635] Node 0 DMA free:7576kB min:44kB low:56kB high:68kB active_anon:8312kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:16kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  591.696641] lowmem_reserve[]: 0 1884 1884 1884 1884
[  591.696643] Node 0 DMA32 free:5448kB min:5532kB low:7460kB high:9388kB active_anon:1888060kB inactive_anon:18856kB active_file:240kB inactive_file:264kB unevictable:18536kB writepending:0kB present:2080620kB managed:2019472kB mlocked:18536kB kernel_stack:2640kB pagetables:7524kB bounce:0kB free_pcp:476kB local_pcp:476kB free_cma:0kB
[  591.696646] lowmem_reserve[]: 0 0 0 0 0
[  591.696647] Node 0 DMA: 0*4kB 1*8kB (M) 1*16kB (U) 0*32kB 2*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 2*1024kB (UM) 0*2048kB 1*4096kB (M) = 7576kB
[  591.696654] Node 0 DMA32: 2*4kB (UM) 0*8kB 200*16kB (UM) 70*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5448kB
[  591.696669] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  591.696670] 7103 total pagecache pages
[  591.696674] 0 pages in swap cache
[  591.696674] Swap cache stats: add 0, delete 0, find 0/0
[  591.696675] Free swap  = 0kB
[  591.696675] Total swap = 0kB
[  591.696676] 524153 pages RAM
[  591.696676] 0 pages HighMem/MovableOnly
[  591.696677] 15308 pages reserved
[  591.696677] 0 pages cma reserved
[  591.696677] 0 pages hwpoisoned
[  591.696678] Tasks state (memory values in pages):
[  591.696678] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[  591.696683] [    338]     0   338    14917     1010   131072        0          -250 systemd-journal
[  591.696685] [    365]     0   365     5366      890    69632        0         -1000 systemd-udevd
[  591.696687] [    366]   100   366     4600      694    77824        0             0 systemd-network
[  591.696692] [    511]     0   511    70052     4498    94208        0         -1000 multipathd
[  591.696699] [    556]   101   556     5974     1541    86016        0             0 systemd-resolve
[  591.696705] [    557]   112   557    22557      499    77824        0             0 systemd-timesyn
[  591.696707] [    585]     0   585    59679      846    98304        0             0 accounts-daemon
[  591.696712] [    592]     0   592     2493      585    49152        0             0 cron
[  591.696719] [    593]   103   593     1904      521    49152        0          -900 dbus-daemon
[  591.696725] [    599]     0   599    21392      546    53248        0             0 lxcfs
[  591.696734] [    602]     0   602     8060     2719    94208        0             0 networkd-dispat
[  591.696739] [    612]     0   612     1611      597    49152        0             0 qemu-ga
[  591.696741] [    613]   102   613    56137      827    77824        0             0 rsyslogd
[  591.696743] [    621]     0   621   156690     3569   212992        0          -900 snapd
[  591.696749] [    627]     0   627     4218      926    73728        0             0 systemd-logind
[  591.696754] [    629]     0   629     1004      569    49152        0             0 atd
[  591.696759] [    664]     0   664     2247      414    45056        0             0 agetty
[  591.696766] [    666]     0   666    58236      844    81920        0             0 polkitd
[  591.696767] [    670]     0   670     3071      982    69632        0         -1000 sshd
[  591.696777] [    674]     0   674    27766     2713   114688        0             0 unattended-upgr
[  591.696784] [    681]   111   681   320732    85623  1040384        0             0 mysqld
[  591.696789] [    682]     0   682    68248     3131   225280        0             0 apache2
[  591.696791] [    686]    33   686    70934     7005   274432        0             0 apache2
[  591.696793] [    687]    33   687    70903     6944   274432        0             0 apache2
[  591.696794] [    688]    33   688    71449     8546   282624        0             0 apache2
[  591.696796] [    689]    33   689    70932     7212   278528        0             0 apache2
[  591.696797] [    690]    33   690    70935     7707   278528        0             0 apache2
[  591.696799] [    779]     0   779     3502      665    65536        0             0 sshd
[  591.696801] [    831]    33   831    70929     7064   274432        0             0 apache2
[  591.696807] [    833]     0   833     4600     1126    81920        0             0 systemd
[  591.696809] [    834]     0   834    25824      833    98304        0             0 (sd-pam)
[  591.696811] [    942]     0   942     2921     1004    57344        0             0 bash
[  591.696813] [   1052]    33  1052    70930     6928   274432        0             0 apache2
[  591.696815] [   1161]     0  1161     3119      736    57344        0             0 sudo
[  591.696816] [   1162]    33  1162   399482   363495  3186688        0             0 php
[  591.696818] [   1163]     0  1163     5366      678    61440        0             0 systemd-udevd
[  591.696820] [   1164]     0  1164     5366      697    61440        0             0 systemd-udevd
[  591.696821] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-4.scope,task=php,pid=1162,uid=33
[  591.696857] Out of memory: Killed process 1162 (php) total-vm:1597928kB, anon-rss:1451624kB, file-rss:2356kB, shmem-rss:0kB, UID:33 pgtables:3112kB oom_score_adj:0
[  591.806036] oom_reaper: reaped process 1162 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

removing the following line from config/config.php has fixed the issue and the update could continue. I’ve re-added the line after installation.
‘memcache.local’ => ‘\OC\Memcache\APCu’,

1 Like