I have exactly the same problem on 24.0.3. The list is more than full (over 12.000 enties) and my Login takes around half a minute. Thanx for posting this solution.
A proper solution from official side would be even better ;).
There should be an option for max valid time a session can be valid, without any action. And I tought there is a setting in config.php but even so I set
‘session_lifetime’ => 60 * 60 * 2,
All sessions stay in the list (including the caldav syncs with my phone).