„X-Robots-Tag“-HTTP-Header not configured with „noindex, nofollow“ since NC 26.0.0

chrissi55 · March 22, 2023, 10:45am

Upgrade NC 25.0.4 → 26.0.0.11

After integrity:check-core i get

But in nginx → conf.d → nextcloud.conf i set

X-config

The error occur the first time → earlier NC Versions 23.x → 25.x didn’t have problems with this nextcloud.conf

The “none” option contains “noindex, nofollow”
see: Spezifikationen für Robots-Meta-Tags | Google Search Central | Dokumentation | Google Developers

Thanks for ideas what causes this error

You could check your Nextcloud headers by yourself as well with the following website:

https://headers.4tools.net/

In my case

As Workaround you can modify the *.conf file of your nginx server for serving your nextcloud files by setting

add_header X-Robots-Tag “none” always;
to
add_header X-Robots-Tag “noindex, nofollow” always;

after recheck (integrity:check-core) the error is not shown.

by the way the setting
add_header X-Robots-Tag “noindex, nofollow, nosnippet, noarchive” always;
gives the same error → NC 26.x seems to unlike the nosnippet, noarchive options …

ernolf · March 22, 2023, 12:31pm

Because “none” is indeed equivalent to “noindex, nofollow” for Google, but seems to be not supported by Bing and probably other search engines.
Look here: https://github.com/nextcloud/server/pull/36689

Since Nextcloud 26, the shiped .htaccess file is changed in some lines:

old:

    Header always set X-Robots-Tag "none"

new:

    Header always set X-Robots-Tag "noindex, nofollow"

old:

  # Add cache control for static resources
  <FilesMatch "\.(css|js|svg|gif|png|jpg|ico|wasm|tflite)$">
    Header set Cache-Control "max-age=15778463"
  </FilesMatch>

  <FilesMatch "\.(css|js|svg|gif|png|jpg|ico|wasm|tflite)(\?v=.*)?$">
    Header set Cache-Control "max-age=15778463, immutable"
  </FilesMatch>

new:

  # Add cache control for static resources
  <FilesMatch "\.(css|js|svg|gif|png|jpg|ico|wasm|tflite)$">
    <If "%{QUERY_STRING} =~ /(^|&)v=/">
      Header set Cache-Control "max-age=15778463, immutable"
    </If>
    <Else>
      Header set Cache-Control "max-age=15778463"
    </Else>
  </FilesMatch>

old:

ErrorDocument 403 /
ErrorDocument 404 /

new:

ErrorDocument 403 /index.php/error/403
ErrorDocument 404 /index.php/error/404

For people who disabled .htaccess parsing (with AllowOverride None) for performance reasons or nginx users must change their webserver conf files accordingly. I know how to do it for apache2 but have no clue how to translate this to nginx. I would say, that your “Workaround” is more than that, it looks like the solution.

bb77 · March 22, 2023, 12:52pm

Yup. The documentation has been updated as well: Replace X-Robots-Tag header value with "noindex, nofollow" by MichaIng · Pull Request #9635 · nextcloud/documentation · GitHub

https://docs.nextcloud.com/server/stable/admin_manual/installation/nginx.html

chrissi55 · March 22, 2023, 12:53pm

The part

# Add cache control for static resources
  <FilesMatch "\.(css|js|svg|gif|png|jpg|ico|wasm|tflite)$">
    <If "%{QUERY_STRING} =~ /(^|&)v=/">
      Header set Cache-Control "max-age=15778463, immutable"
    </If>
    <Else>
      Header set Cache-Control "max-age=15778463"
    </Else>
  </FilesMatch>

should be for nginx in “*.conf” (e.g. nextcloud.conf)

location ~ \.(?:css|js|svg|gif|png|jpg|ico|wasm|tflite)$ {
try_files $uri /index.php$request_uri;
expires 6M;
access_log off;
}

If i’m right.

Instead of

expires 6M;

you also can try

expires max;

also see:

mar-el15 · March 31, 2023, 12:49pm

On Nginx, if I change the line
add_header X-Robots tag "none" always;
to
add_header X-Robots tag "noindex nofollow" always;
, the next restart of Nginx fails with the following message:

nginx: [emerg] invalid number of arguments in "add_header" directive in /etc/nginx/conf.d/nextcloud.conf:36

Can someone help me with this please?

chrissi55 · March 31, 2023, 12:54pm

Hi there,

your line shows
add_header X-Robots tag "noindex nofollow" always;

but it should be
add_header X-Robots tag "noindex, nofollow" always;

(comma separated)
i think this will solve your prob

mar-el15 · March 31, 2023, 1:47pm

Thanks for the quick reply, but unfortunately the error still occurs. When I enter “noindex,nofollow” the warning does not disappear, so everything is the same and when I enter “noindex, nofollow” (i.e. a space after the comma) the same error appears as above.

chrissi55 · March 31, 2023, 1:50pm

try this

add_header X-Robots-Tag “noindex, nofollow” always;

there is a “-” between Robots and Tag as well

mar-el15 · March 31, 2023, 2:13pm

Thanks, but the “-” was already there because I only changed what was in the “”.

chrissi55 · April 1, 2023, 6:51am

ok was is this line 36 exactly now?
can you post the part of the config again, please

/etc/nginx/conf.d/nextcloud.conf:36

forget a semicolon at the end or something else?

McFex · April 5, 2023, 12:48am

I had the same problem with nginx as user mar-el15.

The solution provided by user chrissi55 is almost correct. But there is no “SPACE” after the colon insode the “”.

add_header X-Robots-Tags "noindex,nofollow" always;

With this nginx starts without any error.

But nextcloud still gives out the security warning.

chrissi55 · April 5, 2023, 5:36am

Just follow the admin manual that is linked above.

Now to the warning in nextcloud → please clear the cache of your browser and try again.
Alternative is to use the occ command to check integrity again and after that use the browser twice.

Another trap for all who use a reverse proxy in front of nextcloud
If you contact your WebUI of the nextcloud server direct from inner LAN / DOMAIN you may bypass all reverse proxy settings so as the headers.
Be sure that your backend nginx settings (the nginx on the same server than the nextcloud files) serve the same headers as well → or check your nextcloud WebUI from really “outside” by using another device (mobile device only using mobile data e.g.).

Ggeawan · April 8, 2023, 12:17am

After Nextcloud update to 26, had the same error, fix above worked but had to edit this file from my nextcloud jail shell

cd /usr/local/etc/nginx/conf.d

edit nextcloud.inc

change:
add_header X-Robots-Tag “none” always;
to
add_header X-Robots-Tag “noindex, nofollow” always;

JackDinn · May 4, 2023, 8:49pm

Hi,
Using Nextcloud on Truenas.

So what was the answer here?

add_header X-Robots-Tag “noindex, nofollow” always;

and I get

root@nextcloud-jail:/usr/local/etc/nginx/conf.d # service nginx start
Performing sanity check on nginx configuration:
nginx: [emerg] invalid number of arguments in "add_header" directive in /usr/local/etc/nginx/conf.d/nextcloud.inc:23
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
Starting nginx.
nginx: [emerg] invalid number of arguments in "add_header" directive in /usr/local/etc/nginx/conf.d/nextcloud.inc:23
/usr/local/etc/rc.d/nginx: WARNING: failed to start nginx

If i change to
add_header X-Robots-Tag “noindex,nofollow” always;

I get this on the webUI
The "X-Robots-Tag" HTTP header is not set to "noindex, nofollow". This is a potential security or privacy risk, as it is recommended to adjust this setting accordingly.

I tried occ integrity:check-core
root@nextcloud-jail:/usr/local/etc/nginx/conf.d # occ -vv integrity:check-core No errors found

I have cleared the browser cache many times.

I have read all the links on this page but i am still in the dark?

What is the nextcloud.inc line supposed to be?

“noindex,nofollow”
“noindex, nofollow” or
“noindex nofollow”

and then what do i do exactly?

Cheers.

ernolf · May 4, 2023, 10:18pm

just a suspicion:

try

"noindex, nofollow"

the difference are the double quotes: " "
do you see the difference? You are using curly quotes “ ”.

The Unicode code point for the double quotation mark " is U+0022 and the Unicode code point for the left and right double quotation marks “ ” are U+201C and U+201D, respectively.

Double quotes " are the standard ASCII characters used to enclose strings and indicate literal values. They are typically used in programming languages, configuration files, and other technical contexts where exact syntax and punctuation are important.

Curly quotes “ ”, on the other hand, are a typographic convention used in many written languages but not suitable for nginx configuration files.

JackDinn · May 4, 2023, 10:35pm

Yes, you are correct and it’s all working now. I’m so grateful thank you, I would have never noticed.

It’s a bit odd as I copied the text directly from one of the posts above on this thread (so as not to make a silly spelling mistake or something). Well, that didn’t work ^^

I just need to get the SSL security cert sorted so people dont need to create an exception when they come to my server via the web, but i thought i’d see if i could get this far first. It looks complicated for me

Goxgrue · May 9, 2023, 8:59am

Hi can anyone tell me where to find this file

I am using Nextcloud on TrueNas, adn I don’t know which file is it that I want to edit to remove that error

JackDinn · May 9, 2023, 9:26am

See if this file is the one (i assume you are in ssh of the NC jail).
root@nextcloud-jail:/ # cat /usr/local/etc/nginx/conf.d/nextcloud.inc

mmjoy · May 27, 2023, 9:18pm

I have tried everything so far, but nextcloud just keeps complaining:

The "X-Robots-Tag" HTTP header is not set to "noindex, nofollow". This is a potential security or privacy risk, as it is recommended to adjust this setting accordingly.

I am using docker installation, and separate docker for Nginx-proxy-manager. I have tried everything I found so far, without luck. My current config is:

Within nextcloud docker, /nextcloud/nginx/site-confs/default.conf has

 # HTTP response headers borrowed from Nextcloud `.htaccess`
    add_header Referrer-Policy                      "no-referrer"   always;
    add_header X-Content-Type-Options               "nosniff"       always;
    add_header X-Download-Options                   "noopen"        always;
    add_header X-Frame-Options                      "SAMEORIGIN"    always;
    add_header X-Permitted-Cross-Domain-Policies    "none"          always;
    add_header X-Robots-Tag                         "noindex, nofollow"
 always;
    add_header X-XSS-Protection                     "1; mode=block" always;

In NPM docker, Nginx-Proxy-Manager-Official/data/nginx/proxy_host/1.conf has:

# HSTS (ngx_http_headers_module is required) (63072000 seconds = 2 years)
  add_header Strict-Transport-Security "max-age=63072000;includeSubDomains; preload" always;
  add_header X-Robots-Tag "noindex, nofollow" always;

    # Force SSL
    include conf.d/include/force-ssl.conf;

  access_log /data/logs/proxy-host-1_access.log proxy;
  error_log /data/logs/proxy-host-1_error.log warn;

  location / {

  # HSTS (ngx_http_headers_module is required) (63072000 seconds = 2 years)
  add_header Strict-Transport-Security "max-age=63072000;includeSubDomains; preload" always;
  add_header X-Robots-Tag "noindex, nofollow" always;

Integrity check in nextcloud cli returns no issues.

// edit
This was unexpected. I found another site where someone advised that he had it configured inside nextcloud docker and npm docker also. Removing it from npm worked and now it passes checks just fine. Does not make much sense to me, but as long as it is good, I am happy

alexbergsland · June 4, 2023, 9:36am

Thanks @mmjoy .
I am using the docker images Swag and Nextcloud from linuxserver.io and had added the robots tag manually in my nextcloud.conf file but removing that got rid of the warning for me.
It is likely already present in the nextcloud image and it seems you get the same warning for having it twice.
I had a hunch it was something like this because https://headers.4tools.net showed the robot tag twice.