Upgraded to 29.01. and now invalid hash on some files

I just upgraded 29.0.1 and now this file:

js/NcCheckboxRadioSwitch-hel9-sAH-DWj7ZO5U.mjs

is reported as having na invalid hash:

Technical information
=====================
The following list covers which files have failed the integrity check. Please read
the previous linked documentation to learn more about the errors and how to fix
them.

Results
=======
- activity
	- INVALID_HASH
		- js/NcCheckboxRadioSwitch-hel9-sAH-DWj7ZO5U.mjs

Raw output
==========
Array
(
    [activity] => Array
        (
            [INVALID_HASH] => Array
                (
                    [js/NcCheckboxRadioSwitch-hel9-sAH-DWj7ZO5U.mjs] => Array
                        (
                            [expected] => f32ced32fb04b2fce3fb4bb7b691690abc204a923d0d8fcd82b91de7c52ad02e6f70186dcbe3bdd2406dcbbbf9e89312ad928ffb6498715bbab7b8fc226d77c9
                            [current] => 6dce400e70fadcb003805d8551ee03965dbe5949b860d6ca8d355c69e11818b534fa2933a6b6f4d7cd3a995c3e9f28b53f0d83d92e331f5aaf98a1a9f026ce32
                        )

                )

        )

)

I’ve checked the docs:

https://docs.nextcloud.com/server/29/admin_manual/issues/code_signing.html

I get it. I know what it’s about. But I don’t know why an update caused it and above all I don’t know how to fix it. Clearly I need a copy of js/NcCheckboxRadioSwitch-hel9-sAH-DWj7ZO5U.mjs from 29.0.1 and presumably still have my 29.0.0 copy because something went wrong!

Something did sort of go wrong. The download was so slow I took the kids out to their karate lesson and when I returned it had failed the next step the integrity check. So I just restarted it and it went through cleanly and upgraded and the site seems functional … but for this odd new warning.

You can use this copy. You can check SHA512 sum e.g. here or with sha512sum on linux.

Normally the download can not be the problem because you download a zip file from here. Unlikely that the zip file is unpackable but only has a broken file.

Thanks enormously, I shall of course give this a try and next convenience (within a day I hope, but wanted to share my gratitude sooner). I agree and am seriously puzzled how this one file could have somehow broken when the downloader downloads a zip, unpacks it and checks its integrity …

As noted though the symptom I recall is, the download integrity check failing and so I just reloaded the page and tried again, and then it proceeded without complaint and the Administration page says I’m on 29.0.1, but this one file is reported as corrupt … hmmm, leads me to suspect some logic error in the downloader/installer, because the only hypothesis to spring to mind is that it:

1 - Downloaded the zip
2 - Unpacked the zip
3 - Tested the integrity (all the file hashes) and spotted an error (quite
possibly this one file)
4 - On restarting it, completed without complaint.

Realistically if 3. is caused by this one file now reported on Administration page then I would argue it should on a failed integrity check, have attempted to download again, and unpack again, and test again …

The integrity check is there for these errors. It is possible, for example, that the hard disk is full when unpacking and this leads to strange side effects.

Of course. I’m not critical of it. It’s essential. The opposite in fact. I’m amazed that it completed the install with a failed integrity check! Seems mild in this instance but could be crippling.

I downloaded the NcCheckboxRadioSwitch-hel9-sAH-DWj7ZO5U.mjs file and see it’s part of the activity app. Alas there is a file called https://raw.githubusercontent.com/nextcloud/activity/stable29/js/NcCheckboxRadioSwitch-hel9-sAH-DWj7ZO5U.mjs.map I ignored that and just updated the .mjs file and rescanned and the error has disappeared.

Perhaps on a dedicated thread you can share your updater.log (from the original run and the retry). I’d be interested it and I’m fairly familiar with this area of code.

No worries. Found it in the data dir, and it charmingly has the whole history in it (thumbs up to Nextcloud for that). I extracted this update (so my 29.0.1 upgrade log). Alas, it’s a little long for inclusion in a post, and the forum is a bit brain-dead and won’t let me attach a .log, a .txt or a .zip file, but will permit .odt so here is the log extract attached in the (unnecessarily larger) .odt format (my criticism of the forum here is on the basis that someone somewhen decided .odt was safe or useful and .txt or .log - a common text file people might want to share on this forum, or .zip which is at least small, are not - c’est la vie, .odt it is). It is only 304 lines though and admirably devoid of any security related information like passwords, local URLs and such, so if you prefer I can add it to this post as preformatted text?

Nextcloud 29.0.1 update.log.odt (53.9 KB)

Thanks very much, BTW for taking a look. I have worked in software for decades, and IMHO, the installers and updaters are mission-critical for good UX (they are to software much like menu items are to a chef … very little praise for a good job, but everyone remembers and is scared by negative experiences).

Addendum: Ooops, not sure how to do that on a dedicated thread so that you see it?

I don’t see any apparent explanation in the log. It would also be really odd that only one file was impacted by… well, I can’t even image what caused this at the moment other than a lower level bug in ZipArchive (what we use in PHP to extract) or an OS/filesystem or hardware problem.

That definitely wasn’t a stray file from a prior version of Nextcloud either, because the last part of the filename changes regularly. That particular filename would have first appeared in v29.0.1RC1. Even so it doesn’t match the prior similar file’s hash either.

The Updater checks the signature of the entire downloaded zip file before doing anything with it. It passed that. So anything that went wrong was presumably later.

Any chance you still have a copy of the corrupted NcCheckboxRadioSwitch-hel9-sAH-DWj7ZO5U.mjs? Maybe you can upload it as a gist @ https://gist.github.com? I’m curious what the heck is in it.

What version of PHP? And what distribution/OS? Also, what’s the underlying filesystem? (There have been some ZipArchive and underlying libzip bugs over the years, but other than some older RHEL/RHEL-derived distros, I’m not aware of any shipping ancient libzip versions that many people should still be using…).

Just checked. Was sure I kept the old one around for reference as NcCheckboxRadioSwitch-hel9-sAH-DWj7ZO5U.mjs.old but alas not (well I think I did and probably removed it when happy with the fix, doh - als checked and tis is one of the servers I still have on ext4 not btrfs so lack a timely snapshot between the install and now).

I did look at it and it looked internally similar to the good one, like compressed JavaScript (unnecessary whitespace no more) and so it was tricky to diff, which I did with meld but of course the lines are very very long and so took a long time to diff with not very useful summary resulting.

PHP:

$ php --version
PHP Warning:  JIT is incompatible with third party extensions that override zend_execute_ex(). JIT disabled. in Unknown on line 0
PHP 8.1.2-1ubuntu2.17 (cli) (built: May  1 2024 10:10:07) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.2, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.2-1ubuntu2.17, Copyright (c), by Zend Technologies
    with Xdebug v3.1.2, Copyright (c) 2002-2021, by Derick Rethans

as to OS and distro:

$ inxi -Fz
System:
  Kernel: 5.15.0-102-generic x86_64 bits: 64 Console: pty pts/0
    Distro: Ubuntu 22.04.4 LTS (Jammy Jellyfish)
Machine:
  Type: Desktop Mobo: Intel model: D54250WYK v: H13922-305 serial: <superuser required>
    UEFI: Intel v: WYLPT10H.86A.0051.2019.0322.1320 date: 03/22/2019
CPU:
  Info: dual core model: Intel Core i5-4250U bits: 64 type: MT MCP cache: L2: 512 KiB
  Speed (MHz): avg: 1297 min/max: 800/2600 cores: 1: 1297 2: 1297 3: 1297 4: 1297
Graphics:
  Device-1: Intel Haswell-ULT Integrated Graphics driver: i915 v: kernel
  Display: server: No display server data found. Headless machine? tty: 322x76
  Message: GL data unavailable in console. Try -G --display
Audio:
  Device-1: Intel Haswell-ULT HD Audio driver: snd_hda_intel
  Device-2: Intel 8 Series HD Audio driver: snd_hda_intel
  Sound Server-1: ALSA v: k5.15.0-102-generic running: yes
Network:
  Device-1: Intel Ethernet I218-V driver: e1000e
  IF: eno1 state: up speed: 1000 Mbps duplex: full mac: <filter>
Drives:
  Local Storage: total: 2.05 TiB used: 499.61 GiB (23.8%)
  ID-1: /dev/sda vendor: Samsung model: SSD 840 EVO 250GB mSATA size: 232.89 GiB
  ID-2: /dev/sdb type: USB vendor: LITE-ON model: 1 UITRA1 size: 1.82 TiB
Partition:
  ID-1: / size: 113.32 GiB used: 25.37 GiB (22.4%) fs: ext4 dev: /dev/dm-0
  ID-2: /boot size: 973.4 MiB used: 252.9 MiB (26.0%) fs: ext4 dev: /dev/sda2
  ID-3: /boot/efi size: 511 MiB used: 6.1 MiB (1.2%) fs: vfat dev: /dev/sda1
Swap:
  ID-1: swap-1 type: file size: 4 GiB used: 104.8 MiB (2.6%) file: /swap.img
Sensors:
  System Temperatures: cpu: 29.8 C mobo: 27.8 C
  Fan Speeds (RPM): N/A
Info:
  Processes: 169 Uptime: 50d 13h 48m Memory: 15.51 GiB used: 1.61 GiB (10.4%) Init: systemd
  runlevel: 5 Shell: Bash inxi: 3.3.13

It does indeed appear to be a mystery. Leaves me wondering if a bad disk sector perhaps caused a hiccup. fsck is useless on the running server, and haven’t time to bring it down and boot off a USB right now to use that on the root filesystem, but SMART is possible:

$ sudo smartctl /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-102-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

ATA device successfully opened

Use 'smartctl -a' (or '-x') to print SMART (and more) information

cirrus@nephele:/var/www/html/nextcloud/apps/activity/js$ sudo smartctl -a /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-102-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     Samsung SSD 840 EVO 250GB mSATA
Serial Number:    S1KPNEAF203696L
LU WWN Device Id: 5 002538 844584d30
Firmware Version: EXT41B6Q
User Capacity:    250,059,350,016 bytes [250 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
TRIM Command:     Available
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jun  6 09:07:20 2024 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		( 4800) seconds.
Offline data collection
capabilities: 			 (0x53) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (  80) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   076   076   000    Old_age   Always       -       119267
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       364
177 Wear_Leveling_Count     0x0013   093   093   000    Pre-fail  Always       -       78
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   077   049   000    Old_age   Always       -       23
195 Hardware_ECC_Recovered  0x001a   200   200   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0012   099   099   000    Old_age   Always       -       273
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       41570598293

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

and I see Runtime_Bad_Block is at Pre-fail. Not sure that explains anything though as at the SMART level I’d expect any bad blocks not to present as a write issue in the user space (i.e. I’d expect it’s handled quietly by lower layers and just works in the user space).

Are you actively using Xdebug?

Unfortunately, it’s not clear why the file mismatch occurred in your environment.

I guess on the other hand we can look at the upside: the hash checking did its job. :slight_smile:

I have a similar issue. All of a sudden, after upgrading, the integrity check fails. It’s still there, even on the latest version and I can’t get rid of it.

@akr Your situation is a different (but interesting in its own way hah). Responded in your thread.