I assume that server-side asset deletions are applied to client libraries. I.e. If I take a picture on my phone, but then later delete the picture from immich on another device, it will then also remove the original copy on the client (phone) that took it.

Based on the above comment, I think you are looking for ~/.var/app.

This is correct. But the port wont reply to anything but a valid client, so it should not be too apparent to a would-be attacker that you have a port open at all.