Android App cannot connect to local instance when using Wifi

I am using ssh port forwarding to access my restricted Nextcloud instance. This works well on my desktop and also on my android device except for when I am using wifi. As soon as I connect to a wifi the Android app won’t connect anymore. I can still access the instance via the browser which tells me that the port forwarding is working.

I already tried to dig a bit and used logcat to get some logs, that look like:

07-20 01:26:36.073  6191 12553 D RefreshFolderOperation: Checking changes in xxxx@localhost:50004/
07-20 01:26:36.076  6191 12553 D OwnCloudClient #0: REQUEST PROPFIND /remote.php/dav/files/xxxx//
07-20 01:26:36.893  6191  8857 I OpenGLRenderer: Davey! duration=9223369567573ms; Flags=0, FrameTimelineVsyncId=180310, IntendedVsync=2469269068601, Vsync=2469269068601, InputEventId=0, HandleInputStart=2469270487701, AnimationStart=2469270490670, PerformTraversalsStart=2469270751399, DrawStart=2469270942389, FrameDeadline=2469302401933, FrameInterval=2469270479576, FrameStartTime=16666666, SyncQueued=2469271103899, SyncStart=2469283194109, IssueDrawCommandsStart=2469283552755, SwapBuffers=2469297647027, FrameCompleted=9223372036854775807, DequeueBufferDuration=10408335, QueueBufferDuration=1188021, GpuCompleted=9223372036854775807, SwapBuffersCompleted=2469299835725, DisplayPresentTime=0, 
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: Read file / failed: Connect timeout exception
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 5000 ms
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:155)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at com.owncloud.android.lib.common.OwnCloudClient.executeMethod(OwnCloudClient.java:199)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at com.owncloud.android.lib.common.OwnCloudClient.executeMethod(OwnCloudClient.java:168)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at com.owncloud.android.lib.resources.files.ReadFileRemoteOperation.run(ReadFileRemoteOperation.java:84)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:187)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at com.owncloud.android.operations.RefreshFolderOperation.checkForChanges(RefreshFolderOperation.java:365)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at com.owncloud.android.operations.RefreshFolderOperation.run(RefreshFolderOperation.java:235)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at com.owncloud.android.lib.common.operations.RemoteOperation.run(RemoteOperation.java:363)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at java.lang.Thread.run(Thread.java:920)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: Caused by: java.net.SocketTimeoutException: failed to connect to localhost/127.0.0.1 (port 50004) from /127.0.0.1 (port 44867) after 5000ms
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at libcore.io.IoBridge.connectErrno(IoBridge.java:236)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at libcore.io.IoBridge.connect(IoBridge.java:180)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:142)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:390)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:230)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:212)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:436)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at java.net.Socket.connect(Socket.java:621)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at java.lang.reflect.Method.invoke(Native Method)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
07-20 01:26:37.687  6191 12525 E ReadFileRemoteOperation: 	... 15 more

Any ideas how I can fix this?