S3 external storage excessive network traffic

bjo · December 27, 2023, 12:13am

I stumbled upon the same issue. I’m running 28.0.1 with juicefs on Backblaze B2 for storage and was wondering why every day in the evening a lot of egress traffic appears. Some digging in the logs showed:

{"reqId":"LF9xJW9in00pBvwc4Tw9","level":0,"time":"2023-12-26T18:12:41+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"Run OC\\Core\\BackgroundJobs\\GenerateMetadataJob job with ID 119439","userAgent":"--","version":"28.0.1.1","data":{"app":"cron"}}

The corresponding reqId in audit.log shows access for a lot of files for hours. Shouldn’t the metadata update job run only for an hour if I understand

github.com

nextcloud/server/blob/master/core/BackgroundJobs/GenerateMetadataJob.php#L79


      
          			if ($startTime === null) {
          				// Skip all user before lastMappedUser.
          				if ($lastMappedUser !== $user->getUID()) {
          					continue;
          				}
          
          				$startTime = time();
          			}
          
          			// Stop if execution time is more than one hour.
          			if (time() - $startTime > 60 * 60) {
          				return;
          			}
          
          			$this->scanFilesForUser($user->getUID());
          			$this->config->setAppValue('core', 'metadataGenerationLastHandledUser', $user->getUID());
          		}
          
          		$this->jobList->remove(GenerateMetadataJob::class);
          	}

correctly?
And does

github.com

nextcloud/server/blob/master/core/BackgroundJobs/GenerateMetadataJob.php#L50


      
          		ITimeFactory $time,
          		private IConfig $config,
          		private IRootFolder $rootFolder,
          		private IUserManager $userManager,
          		private IFilesMetadataManager $filesMetadataManager,
          		private IJobList $jobList,
          		private LoggerInterface $logger,
          	) {
          		parent::__construct($time);
          
          		$this->setTimeSensitivity(\OCP\BackgroundJob\IJob::TIME_INSENSITIVE);
          		$this->setInterval(24 * 3600);
          	}
          
          	protected function run(mixed $argument): void {
          		$users = $this->userManager->search('');
          		$lastMappedUser = $this->config->getAppValue('core', 'metadataGenerationLastHandledUser', '');
          
          		if ($lastMappedUser === '') {
          			$user = array_key_first($users);
          			if ($user === null) {

scan now every file every day?
Why isn’t the metadata updated when running occ files:scan -all?

But all in all it’s bad when paying for egress traffic and NC seems to download every file every day. B2 has 3x storagr as egress for free, but this doesn’t also fit then.