I think I managed to limit the RAM usage by changing the MPM configuration of Apache (in /etc/apache2/mods-enabled/mpm_prefork.conf):
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxRequestWorkers: maximum number of server processes allowed to start
# MaxConnectionsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 3
MaxRequestWorkers 3
MaxConnectionsPerChild 0
</IfModule>
The default values were way too high for my Raspberry Pi 3. For instance, when loading the Photos tab, one Apache process takes 150 MB. With the default values, Apache tries to run up to 150 processes. I reduced it to 3, which is the maximum my Raspberry can handle. As a consequence, loading the Photos page for instance is very slow, but at least now my server doesn’t crash. The other pages load well, because they contain less heavy files.
To conclude, I agree with @esbeeb on the fact that a Raspberry Pi 3 might not be enough to use Nextcloud comfortably. I don’t think it is related to the version of Nextcloud, it is just a matter of configuring Apache accordingly to the server’s capacity, which is weak in my case.