Fail2Ban on NCP

NextcloudPi Documentation
, ,
1

ncp-docs_banner14-docs690x51, 100%
ncp-docs_banner14-docs690x51, 100%4000×300 113 KB

How fail2ban works

(Source): GitHub - How Fail2Ban Works
Sergey G. Brester edited this page on 5 May · 11 revisions

Fail2Ban scans log files resp. journals (using specified regular expressions also known as filter-rules) and executes configured actions to ban failures having too many attempts (matched specified filter-rules). It does this e. g. by updating system firewall rules to reject new connections from those IP addresses, for a configurable amount of time. But you can write resp. configure your own action to ban something other as host/IP, like user or e-mail.

Fail2Ban comes out-of-the-box ready to read many standard log files, such as those for sshd and Apache, and is easy to configure to read any log file you choose, for any error you choose.

But fail2ban is just a tool, so it should be properly configured.
Must be manually configured in Docker

Best practice

(Source): GitHub - Best Practices Fail2Ban
Sergey G. Brester edited this page on 23 Aug 2019 · 2 revisions

HowTo’s

FAQ

Clone this wiki locally

Here are the advises and common smart practices in order to drive Fail2ban operating faster as well as to minimize a load of the system through Fail2Ban:

  • If you are familiar with RE-engine, optimize the regex’s (by this are meant all regex’s Fail2Ban uses - especially failregex but also prefregex, ignoreregex and datepattern). This may imply also other precautions you may take by changing of default filters or configuration files, for example:
    • use more precise predefined tags as <HOST>, if you don’t expect hostname in log-message (e. g. <ADDR> for IP-addresses only) or disable usedns for such jails
    • use more precise regex parts as predefined in filter or common config-files substitutions, e. g. redefine __prefix_line with your own shorter and more efficient RE exactly matching related part of your log-messages, because interpolation of default RE for the __prefix_line may be too long (due to all targeting systems or distributions and backwards-compatibility reasons) and therefore not optimal for application on your system
    • potentially remove regex’s or its optional parts that are not valid at all for your system (so practically don’t occur, related to the application and/or distribution version).
  • If log-file or journal show several different types of messages or some pre-filtering may be advantageous, use common prefregex (and short and efficient as possible);
  • Reorganize the regex-order by occurrence in log (frequently as first). This is valid for all regex too.
  • Try to reduce count of log-messages (especially unneeded or unrelated information, which is called “parasitic” further) in observing log-file or journal. See section Reduce parasitic log-traffic below.
  • Use incremental bantime (since v.0.11) as well as other Fail2Ban new features might help you to minimize intrusions attempt count resp. to reduce attacks in the long perspective.
  • Under DDOS-attack similar circumstances use other tools and make special arrangements too, may be in combination with Fail2Ban and/or information it could provide you. Please note, Fail2Ban is just a tool and it depends on what you make out of it.

Reduce parasitic log-traffic

It is always recommended to take diverse precautions in order to minimize the unrelated and therefore uninteresting logging messages in log-files or journals that Fail2Ban is monitoring.

This is application- or system-related and could depend on kind of protection is needed, for example:

  • using re-targeting in rsyslog, etc (filtering and forwarding into different log-file)
  • decreasing the log-level, priority or facility lists (in order to hide uninteresting “parasitic” messages)
  • by means of extracting the failures only by application/service into separate log-file or journal, if it is directly possible in corresponding application (e. g. in nginx via extra location for 40x|50x-error codes with own extra access-log set in location), see an example below.

Several applications as well as web-servers write the failures in access or error logs together with other messages, which may be unrelated for some jails of Fail2Ban (one can have more as one jail for some application) or even totally don’t interest Fail2Ban at all.

It is important to keep in mind that if multiple failregex are configured and there is no prefregex (pre-filtering messages or it matches too) each parasitic message would cause a check for possible match across all specified regex’s for the jail.

Especially an observing of the access-log directly may be too heavy for your system (well visited web-server generating too many entries in access-log may produce high load by Fail2Ban). Therefore it is recommended to restrict it in web-server.

Here are examples how it might be implemented in nginx using conditional access_log or by supplying the failed requests to extra location via postaction resp. via error_page to extra internal location (with extra access-log) which would then get all the requests with 404/403/401/500 etc.:

How to install or upgrade fail2ban manually

(Source:) GitHub - How to install Fail2Ban manually
Sergey G. Brester edited this page on 19 Dec 2020 · 3 revisions

Fail2Ban scans log files like /var/log/auth.log and bans IP addresses conducting too many failed login attempts. It does this by updating system firewall rules to reject new connections from those IP addresses, for a configurable amount of time. Fail2Ban comes out-of-the-box ready to read many standard log files, such as those for sshd and Apache, and is easily configured to read any log file of your choosing, for any error you wish.

user@hostname:~ $ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:	3
|  |- Total failed:	10
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	7
   |- Total banned:	7
   `- Banned IP list:	94.232.43.63 137.184.56.35 87.251.67.98 94.232.47.130 45.9.20.71 185.219.52.171 185.217.1.122

Though Fail2Ban is able to reduce the rate of incorrect authentication attempts, it cannot eliminate the risk presented by weak authentication. Set up services to use only two factor, or public/private authentication mechanisms if you really want to protect services.

Must be manually configured in Docker

Many distributions already provide fail2ban packages, so better you’d install this built from your maintainers version of fail2ban, especially if you are not familiar with linux.

You don’t need install fail2ban for the test attempts or to try some new functionality, so firstly read How to test newer fail2ban version resp. use fail2ban standalone instance.

Required pre-installation steps

If you already have fail2ban installed from your distribution:

  • backup your current configuration /etc/fail2ban and run-time database /var/lib/fail2ban/fail2ban.sqlite3

  • you can also store the dump of your current configuration to be able to compare it later:
    fail2ban-client -d > /tmp/f2b-dump-of-prev-version.txt

  • if your fail2ban version departs significantly or you decided install from source, stop and uninstall obsolete version of fail2ban

Manual installation of debian upstream-package (released here)

If you have debian-based distribution, you could try to install debian packages from newest releases. This upstream package is very similar the package maintaining for the latest debian stable version, but can also work for the most other debian-based distributions.

:: Note ::
Although in opposite to installation from source, it would also install service units, man-files, bash-completion etc, the package configuration can also deviate from configuration provided by maintainers of your distribution. Additionally note that the upstream packages are provided without the test-suite (fail2ban-testcases, fail2ban.test python module).

Manual installation from source

Manual install/upgrade is very similar to aforementioned testing of standalone instance, just you have to do more steps to install it, like copy and enable service units, copy man-files etc pp.

Now we have to configure fail2ban (also see Proper fail2ban configuration):

  • copy your .local files from backup to /etc/fail2ban (or create new jail.local and set-up jails you need to enable).

  • take a look in your backup copy of /etc/fail2ban/jail.conf for includes made for your distribution:

[INCLUDES] 
before = paths-debian.conf

  • copy this file and set the same line in new jail.conf or jail.local in /etc/fail2ban.

  • test it
    sudo fail2ban-client --test

Now you can start fail2ban via service:

sudo service fail2ban start

How to test newer fail2ban version resp. use fail2ban standalone instance

(Source): How to test newer Fail2Ban version resp. use Fail2Ban-standalone instance
Sergey G. Brester edited this page on 14 Mar 2019 · 3 revisions

Sometimes you’ll need to test a fix for some issue, new functionality or some features provided only in the newer version, another different fail2ban configuration, etc.

To test another version of fail2ban, you do not need to install it at all.

You can create a standalone or test instance parallel to your stock fail2ban (fail2ban works basically without installation).

Below you’ll find an example, how to create such test instance without installation, e. g. 0.10-th version of fail2ban, regardless of original fail2ban version, that may be installed parallel on the same host:

  • download or git checkout a 10th branch, e. g. to directory /tests/f2b10:

git clone --branch 0.10 https://github.com/fail2ban/fail2ban.git /tests/f2b10

  • enter your new fail2ban directory:

cd /tests/f2b10
f2b10dir="$(pwd)"

  • if you will use python3, first execute ./fail2ban-2to3
    (don’t do that if python --version returns 2.x)

  • if you want, you can execute the test suite (works only on clean copy, as long as you do not made some customisation, resp. no *.local files was created):

PYTHONPATH=. ./bin/fail2ban-testcases --fast --no-network

  • create separate directory run for your files this fail2ban instance uses:

mkdir run

  • create ./config/fail2ban.local and write there this another paths:
[Definition]
logtarget = /tests/f2b10/run/fail2ban.log
socket = /tests/f2b10/run/fail2ban.sock
pidfile = /tests/f2b10/run/fail2ban.pid
dbfile = /tests/f2b10/run/fail2ban.sqlite3
  • create ./config/jail.local and write there your own customizations only, e. g. your backend, enable there your jails, etc.:
[DEFAULT] 
backend = pyinotify 
default_backend = pyinotify 

[sshd] 
enabled = true 

[pam-generic] 
enabled = true
  • copy your fail2ban database (contains last positions inside logs, current bans, etc.):

cp /var/lib/fail2ban/fail2ban.sqlite3 /tests/f2b10/run/fail2ban.sqlite3

  • :warning: [ important ] stop original stock fail2ban
    service fail2ban stop or fail2ban-client stop

:: Note ::
If you want to leave your original stock fail2ban running, see warning at end of this article for more informations.

  • define alias for your new test fail2ban instance:

alias f2b10="sudo PYTHONPATH=$f2b10dir $f2b10dir/bin/fail2ban-client -c $f2b10dir/config/"

  • start your test fail2ban instance:

sudo PYTHONPATH=. ./bin/fail2ban-client -c $(pwd)/config/ start f2b10 start

  • to stop this fail2ban instance:

sudo PYTHONPATH=. ./bin/fail2ban-client -c $(pwd)/config/ stop f2b10 stop

  • path for configs /tests/f2b10/config (corresponds to /etc/fail2ban)
  • path for log /tests/f2b10/run/fail2ban.log

Easy and without any installation…

:warning: [ Important ] you should normally always shutdown your original version before you start this standalone instance and vice versa. But you can leave your original stock fail2ban running, and try to run this test (standalone) version parallel side by side with original version of fail2ban. In this case you should never use the same jail names as configured in jail.local (jail.conf) from /etc/fail2ban/, because otherwise, through equal name convention, you may get conflict in tables, chains or rules in your system actions (like iptables, etc.)

Proper fail2ban configuration

(Source): GitHub - Proper Fail2Ban Configuration
Thorian93 edited this page on 17 Oct 2020 · 4 revisions

Clone this wiki locally

If not configured manually, Fail2ban will load configuration files from the directory /etc/fail2ban. You can find many files called *.conf there.
Before you start the fail2ban service for the first time, you should do some configuration appropriate to your system. The least would be to enable jails for the services that you want to protect with fail2ban.

[Q] Should I make my configuration directly in jail.conf and fail2ban.conf?
[A] No. You should avoid to change .conf files, created by fail2ban installation. Instead, you should create new files with a .local extension.

Details

Since these stock files may be overwritten by the package upgrades, or because your changes may be incompatible with some future versions, you shouldn’t edit it in-place.
So to set your jail configuration, don’t change jail.conf. To customize some filter configuration, don’t change filter.conf. Instead, create a new file with a .local extension and insert only the settings you want to override or the settings you want to append to the default configuration. For example any values defined in jail.local will override those in jail.conf in the same sections (e. g. [DEFAULT]).

So for example if original .conf file contains:

[DEFAULT] logpath = /path/to/log [section1] logpath = /other/path enabled = true [section2] enabled = true

And you’ll create a .local file contains:

[DEFAULT] logpath = /my-path/to/log

The value of parameter logpath in section1 will be still /other/path.
But value of parameter logpath in section2 will be changed to /my-path/to/log (because it was not specified in the section itself, so the new default value will be used).

[Q] What configuration is necessary to let fail2ban protect a service?

Answer

[A] You should create a jail.local file and at least enable the corresponding jails (all jails are disabled by default) or overwrite the default settings which you want to change, or even create your own jails (and/or) filters, that are not available in the default configuration of fail2ban.

For example if your intention is to monitor authorization failures occurring in sshd and nginx, but the error.log of your your nginx-instance is configured as /var/log/my-nginx/error.log you should set the logpath parameter appropriately additionally to enabled in section [nginx].

So your jail.local looks like:

[nginx] 
logpath = /var/log/my-nginx/error.log 
enabled = true 

[sshd] 
enabled = true

If you use another version of fail2ban as provided by the maintainers of your distribution, you should check other parameters (that may be normally specified in some distribution config files), like:

  • several path-parameters of fail2ban service itself (specified in fail2ban.conf or includes):
[Definition] 
logtarget = /var/log/fail2ban.log 
socket = /var/run/fail2ban/fail2ban.sock 
pidfile = /var/run/fail2ban/fail2ban.pid 
dbfile = /var/run/fail2ban/fail2ban.sqlite3
  • other jail parameters (jail.conf or includes) like backend (e. g. usage of systemd journals expected systemd backend), action resp. banaction (e. g. you can’t use iptables if your system does not support it), logpath, etc.

You can also control resp. configure another optional configurations parameters, like ignoreip, etc.

[Q] How I can see the current (merged) configuration, that fail2ban will use

Answer

[A] You can dump your current configuration (all the parameters that fail2ban loads on startup) with the following commands:

# dump parameters:
fail2ban-client -d
# verbose: output config files will be loaded and dump parameters:
fail2ban-client -vd
fail2ban-client -vvd

[Q] How I can notify fail2ban, that the configuration was changed

Answer

[A] You should execute fail2ban-client reload (in previous versions before 0.10 fail2ban-client restart).

You can also get and set parameters individually during runtime, using the fail2ban client-server communication protocol. For example:

fail2ban-client set pam-generic logencoding UTF-8
fail2ban-client set nginx findtime 10m

[Q] How should I correctly modify log file locations other than in the jail settings or messing with master .conf files?

Answer

[A] To make a modification to the default log file locations you should create a .local file of paths-common.conf or paths-debian.conf (whichever you are using in jail.local) and make changes only in your .local files which keeps it nicely structured for your jail(s) settings and avoids problems when Fail2Ban is updated

To create your .local file

Please don’t copy it:
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
cp /etc/fail2ban/paths-common.conf /etc/fail2ban/paths-common.local
Just create and edit it with your preferred editor.

Now if you want for example an Nginx filter to read all your Nginx Access Logs for multiple web sites

  • Either do it in jail.local:
    [nginx]
    logpath = /var/log/nginx/*access*.log
    enabled = true

  • Or instead of using in your jail:
    Edit the line in paths-common.local or paths-debian.local (whichever you are using) and add the entry with nginx_access_log line as follows
    [DEFAULT]
    nginx_access_log = /var/log/nginx/*access*.log
    Then in your jail you would rather use
    logpath = %(nginx_access_log)s

About the Author

This article was worked on by @ZendaiOwl. Thank you so much for all you do! You may write your own articles by joining our wiki and documentation team on the forum.

2

A post was merged into an existing topic: About moving the articles