https://nginx.org/en/docs/http/ngx_http_auth_request_module.html
It expects a 2xx response for authorized, and 401 or 403 for unauthorized. Every other status code is considered an error, and is thrown away with an 500 status to the client.
I experimented further, I think I now know how it should be working. The issue is that NC returns a 303 with the location of the login page for the configured client, when the request is not authorized. Which makes sense, but this nginx module considers this an error. Of course, NC’s oauth2 client settings page gives zero flexibility with those status codes, and the docs mention nothing either. On the nginx side, it doesn’t seem to be possible to override the 303 status code by configuration, it’s apparently ignored. So, I’m stuck at this.
No one ever tried to make this work?