Subscribed calendars takes long time on refresh

Matias · January 11, 2024, 1:07pm

Support intro

Sorry to hear you’re facing problems

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can

Nextcloud version (eg, 20.0.5): 27.1.4
Operating system and version (eg, Ubuntu 20.04): Ubuntu 22.04
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.52
PHP version (eg, 7.4): 8.1.2

I have a question about refresh subscribed calendars(like google calendar)
An user have a big google calendar and when cron.php runs takes like 10 minutes to process.
Debugging this times i see that is related to this subscribed calendar.
Running some sql queries i see when the cron.php is running this calendar is like recreates every time. I use this SQL querie to debug

SELECT calendarid,calendartype,  COUNT(*) FROM `oc_calendarobjects` GROUP BY calendarid,calendartype ORDER BY COUNT(*);

and i see that calendar 52 is recreating all time:

calendarid calendartype count
| 52 | 1 | 7818 |

now has 7800+ calendar objects but when the cron starts again go to 0 objects and stat to add all again.

The question i have if this is normal behavior that recreates the objects of the subscribed calendar every refresh.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

Subscribe to big calendar
wait cron.php runs
check calendar objects related to this subscribed calendar(using the querie provided above)

The output of your Nextcloud log in Admin > Logging:

I don’t see anything related to this.
some errors like:

{"reqId":"0","level":3,"time":"2024-01-11T12:55:23+00:00","remoteAddr":"","user":"--","app":"dav","method":"","url":"--",
"message":"Unable to create calendar object from subscription 52","userAgent":"--","version":"27.1.4.1",
"exception":{"Exception":"Sabre\\VObject\\Recur\\NoInstancesException","Message":"This recurrence rule does not generate any valid instances","Code":0,"
Trace":[{"file":"/var/www/cloud/apps/dav/lib/CalDAV/CalDavBackend.php","line":2824,"function":"__construct","class":"Sabre\\VObject\\Recur\\EventIterator",
"type":"->"},{"file":"/var/www/cloud/apps/dav/lib/CalDAV/CalDavBackend.php","line":1229,"function":"getDenormalizedData","class":"OCA\\DAV\\CalDAV\\CalDavBackend","
type":"->"},{"file":"/var/www/cloud/apps/dav/lib/CalDAV/WebcalCaching/RefreshWebcalService.php","line":129,"function":"createCalendarObject","class":"OCA\\DAV\\CalDAV\\CalDavBackend",
"type":"->","args":["*** sensitive parameters replaced ***"]}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

Default config

The output of your Apache/nginx/system log in /var/log/____:

nothing related

ernolf · January 11, 2024, 6:34pm

Could you please check the refreshrate of your subscribed calendars with the
nc-subscribed_calendar script, if everything looks OK:

Install it with these commands:

sudo wget -qO /usr/local/bin/nc-subscribed-calendar https://global-social.net/raw/s/nc-subscribed-calendar
sudo chmod +x /usr/local/bin/nc-subscribed-calendar

then simply run the script:

nc-subscribed-calendar

ernolf

Matias · January 25, 2024, 2:30pm

Hi thanks for the reply …at the end the user deleted the subscribed calendar. so now i can’t test it, but i take notes for the future. regards.

jokerigno · February 8, 2024, 4:17pm

Hi tried to use your script but I see this message:

alpine-nextcloud:~# nc-subscribed-calendar
readlink: unrecognized option: e
BusyBox v1.36.1 (2023-11-07 18:53:09 UTC) multi-call binary.

Usage: readlink [-fnv] FILE

Display the value of a symlink

        -f      Canonicalize by following all symlinks
        -n      Don't add newline
        -v      Verbose

  Dependency not met!

  missing dependencies:
  ==============================================================================
  Bynary          Package         Description                                                                                                                                                
  ------------------------------------------------------------------------------
  gpg             gpg             GNU Privacy Guard                                                                                                                                          
  jq              jq              command-line JSON processor                                                                                                                                
  ------------------------------------------------------------------------------

  please install first

installed via apk but now I see:

tail: can't open '': No such file or directory
tail: no files
/usr/local/bin/nc-subscribed-calendar: line 137: declare: `': not a valid identifier

  Hello, this is the first run wizard of "nc-subscribed-calendar"
  enter the directory where your Nextcloud is installed (or [q] to quit):
  -->

I tried different path but without success. Any clue for what is the correct info to put here?

Thanks

ernolf · February 8, 2024, 7:50pm

The script should be run by bash (and readlink etc. from coreutils). It is not build to run with busybox. Busybox comes with a more dash like sh and is not compatible.
If your busybox is linked to /bin/bash as well, then that is erroneous. You should fix that.

Are you running your server on an embedded linux or why do you use busybox?
If you have got some space, I would install some decent binary versions instead of busybox, which is meant for more special cases, where space is expensive.

ernolf

jokerigno · February 8, 2024, 8:19pm

I run Nextcloud within my proxmox server and I choose the alpine-Linux (that should be lighter) using tteck script coming from here

There’s a way to make it work or I have to start fro scratch using another Nextcloud script?

ernolf · February 8, 2024, 8:30pm

I have updated the script and made it more busybox conform.
The auto update will not work this time so, please download it again exactly as you did before and run again.
I hope it works now.
Please give me feedback.

Much and good luck,
ernolf

jokerigno · February 8, 2024, 8:56pm

this is the new output:

alpine-nextcloud:~# nc-subscribed-calendar

     First run, checking integrity:
     ===========================================================================
gpg:                using RSA key 3EE43A84D2BA635A94E1033B0B145139A170715C
gpg: Good signature from "[ernolf] Raphael Gradenwitz (codesigning) <raphael.gra                                                                                                             denwitz@googlemail.com>" [unknown]
     ---------------------------------------------------------------------------
     Integrity check passed. The script is authentic.
     ===========================================================================
Press Any key to continue or [q] to quit

  Hello, this is the first run wizard of "nc-subscribed-calendar"
  enter the directory where your Nextcloud is installed (or [q] to quit):

I think something positive happened right?

ernolf · February 8, 2024, 9:03pm

Yess! Looks perfect!

ernolf

jokerigno · February 8, 2024, 9:05pm

great! now can you help me in understaning what

directory where your Nextcloud is installed

means?

I looked at my config php and tried several path (in bold) without success:

$CONFIG = array (
‘datadirectory’ => ‘/var/lib/nextcloud/data’,
‘logfile’ => ‘/var/log/nextcloud/nextcloud.log’,
‘logdateformat’ => ‘F d, Y H:i:s’,
‘log_rotate_size’ => 104857600,
‘apps_paths’ =>
array (
0 =>
array (
‘path’ => ‘/usr/share/webapps/nextcloud/apps’,
‘url’ => ‘/apps’,
‘writable’ => false,
),
1 =>
array (
‘path’ => ‘/var/lib/nextcloud/apps’,
‘url’ => ‘/apps-appstore’,
‘writable’ => true,
),
),

ernolf · February 8, 2024, 9:45pm

The directory where occ and config/config.php resides.
That’s not misleading, is it?

ernolf

jokerigno · February 8, 2024, 10:02pm

my config.php (posted above) is in

/etc/nextcloud

but there’s no trace there of occ.

occ is in

/usr/bin

ernolf · February 8, 2024, 11:32pm

OMG, Thats that crazy alpine package. I’m sory for that. I cannot say, if that one is supported.

The installation directory is /usr/share/webapps/nextcloud

No guarantee if that works. I did not develop the script for Alpine. Maybe I find some time tomorrow to adapt it.

ernolf

jokerigno · February 9, 2024, 6:47am

another step made, another block:

  enter the directory where your Nextcloud is installed (or [q] to quit):
  --> /usr/share/webapps/nextcloud

  - first run wizard completed successfully.

This script looks up for subscribed calendars in the database of your nextcloud
and it gives you the means to set an individual refreshrate per calendar
/usr/local/bin/nc-subscribed-calendar: line 279: mysql: command not found
There ain't no subscribed calendars in database

ernolf · February 9, 2024, 9:39am

Could you please run the script again, it will offer you to upgrade. Look if it now works

Thanks again for your feedback. It helps make my scripts more portable overall.
I have now adjusted the search of the config directory. To do this, lib/base.php is now integrated instead of being hardcoded to config/config.php. This should now make it possible to run the script on Alpine.

EDIT:

I have now taken a look at the APKBUILD script of the Alpine Nextcloud package and realized that this step was unnecessary, as a symlink is created on the config-dir.

So this error must be broken down:

The error message suggests that the nextcloud-mysql package is in use but the mariadb-client package is not installed.

If so, please install the mariadb-client package. Then everything should work.

ernolf

jokerigno · February 9, 2024, 9:31pm

ok I installed it and I have made a step further:

But I’m still stuck.

The refreshrate must be a valid ISO 8601 duration interval
as described here: https://www.php.net/manual/dateinterval.construct.php
with the limitation that Y (year) and M (month) are not supported.
  Examples:
    PT5M      -  every 5 minutes
    PT29M     -  every 29 minutes
    PT1H      -  every hour
    PT1H30M   -  every 1,5 hour
    P4W3DT8H  -  12 times a year (instead of 'P1M' for every month)
    P13W      -  4 times a year
    P26W      -  twice a year (instead of 'P6M')
    P52W      -  once in a year (instead of 'P1Y')
Enter the refreshrate you want to set for calendar 3 or 'NULL' to unset ([q] to quit): PT5M
grep: unrecognized option: P
BusyBox v1.36.1 (2023-11-07 18:53:09 UTC) multi-call binary.

Usage: grep [-HhnlLoqvsrRiwFE] [-m N] [-A|B|C N] { PATTERN | -e PATTERN... | -f FILE... } [FILE]...

Search for PATTERN in FILEs (or stdin)

        -H      Add 'filename:' prefix
        -h      Do not add 'filename:' prefix
        -n      Add 'line_no:' prefix
        -l      Show only names of files that match
        -L      Show only names of files that don't match
        -c      Show only count of matching lines
        -o      Show only the matching part of line
        -q      Quiet. Return 0 if PATTERN is found, 1 otherwise
        -v      Select non-matching lines
        -s      Suppress open and read errors
        -r      Recurse
        -R      Recurse and dereference symlinks
        -i      Ignore case
        -w      Match whole words only
        -x      Match whole lines only
        -F      PATTERN is a literal (not regexp)
        -E      PATTERN is an extended regexp
        -m N    Match up to N times per file
        -A N    Print N lines of trailing context
        -B N    Print N lines of leading context
        -C N    Same as '-A N -B N'
        -e PTRN Pattern to match
        -f FILE Read pattern from file
You must enter a valid ISO 8601 duration (without P$nM or P$nY) or 'NULL'

It’s very strange that “PT5M” is not recognized.

ernolf · February 9, 2024, 10:41pm

I am very sory, but that is again because of the limitation of the BusyBox, the grep implementation in this case, that comes without the -P option to parse Perl-compatible regular expressions (PCREs).

Please install grep package to solve that. You will need it anyway.

ernolf

jokerigno · February 9, 2024, 11:27pm

apk add grep did the trick.

Now let’s wait 5 min and see if it finally changes

EDIT: i waited 5min but nothing. I refreshed after other few minutes and now both my calendar are updated. Thank you!

ernolf · February 10, 2024, 12:38am

Thank you again for your feedback!

Much and good luck,
ernolf

ernolf · February 10, 2024, 8:19am

Important for understanding:

The renewal interval that you can set with the script refers to the interval at which new tasks are added to the queue. They then have to be executed by the cron job. So if you only run the cron job every 10 minutes, for example, then the subscribed calendar will only be synchronized every 10 minutes as well. This means that the shortest interval depends on the cron interval.
So if a cron job is executed every 5 minutes and the last one was at 10:00, for example, and a new synchronization job was queued at 10:01, then the actual synchronization will be processed in the cron job that is triggered at 10:05

ernolf