Good practice. Good luck then! Let us know if you need anything, on matrix I’m @scrubbles:halflings.chat, feel free to DM with questions

That’s how I started too, so very good plan, and good way of thinking ahead. Ssd will be fast so the app will load fast for your users, and images can take a few seconds and no one will mind.

Make sure you have a solid backup plan for both

Basically for a cloud provider s3 storage is just any storage. It’s not a disk that needs to be high availability with programs reading and writing to it with an OS on top, its just blobs of data. Images, video, isos, whatever. Its meant for access that is lower than what a VM would need for an active program.

For matrix this is ideal for its content. An image uploaded will be read a fee dozen times, and then less and less until eventually it isn’t really needed ever unless someone scrolls and scrolls up.

So for hosting, if you store that on a disk you’re saying “this is critical to the operation of the software and must be highly available and optimized for vms reading and writing to it.”. Think like m.2 ssds. Blob storage then analogous to us home labbers to throwing it on a giant nas. Its there, may take a bit to load, but its there.

Then s3 has classes too, where if you need your data even less you can pay even less trading off access times, you can get even better rates if you know you need it extremely infrequently, like audit logs. Tape drives are actually used quite a bit for those opt-in low access tiers because if you think about it the data storage is incredibly dense, but opening up a tape can be minutes or longer to access. No problem if you’re pulling up some archive from 20 years ago.

If you’re running locally on your own system then yes you can use your own. You can use something like MinIO or Garage to self-host an S3 bucket, and then point Matrix to that

Docs would be helpful, I can’t find much of anything, I think you honestly did the best writeup.

Star Trek quotes is hilarious and perfect!

I’m not sure your level of understanding of cloud infrastructure, so let me know if you need me to go into more detail. Disk storage, like what is attached to a VPS/VM is very expensive, and it’s the 100GB drive you have attached. What is much cheaper is object/blob storage, known in AWS and most cloud providers as S3. This is far far cheaper for many reasons.

Matrix (and really I should say Synapse, what I use) can be configured to save images, photos, uploads, etc to save to a blob storage “bucket” instead of disk. So you can lower your disk from 100 down to something lower because your data is stored in blob storage (fully encrypted). For synapse, the module you need is here: https://github.com/matrix-org/synapse-s3-storage-provider

Thank you! I’m going to start playing with this and see what I can figure out! I’ll be referencing this frequently!

How do you do that, I’m very interested! Also good to see you Admiral!

You can set it to use object storage instead, much cheaper

My friend group has already moved to Matrix, and we’ve been happy

I Will never understand why the open source community hates the GPL license. Maybe they just haven’t seen themselves how big corporations taking advantage of free individual independent developers. I still remember the core.js developer, whose code is in pretty much every giant framework out there basically begging for any sort of income for his work while his family was going hungry in Eastern Europe. Angular, react, all major frameworks absolutely depend on it and never gave them anything.

But that’s not decentralized. It’s the same as discord, only instead of a corporation now you have a server admin.

I completely support you moving off of Discord, and I completely support you setting up Matrix. I tried a lot, I think it has the most feature parity. That being said, the biggest thing I regret when setting it up is that I went with Synapse for my backend Matrix server, when there are others.

I’ve heard very good things about Conduit (https://conduit.rs/), mostly that it’s easier to stand up and easier to maintain.

Either way, I think it’s a smart move, and it’s worth the investment. It’s not the easiest to stand up, but operationally our communication should be our own. Expect trial and error, getting one piece up and running, then the next, and then the next. Celebrate small wins like “Today I got it running” and then “Today I got federation working”, and then “Now I have voice working!”. Otherwise it’s going to feel overwhelming.

I believe in the cause, so feel free to DM me if you have any questions, or send me a DM on Matrix :)

Oh, and a very useful tool - https://federationtester.matrix.org/

This will tell you exactly what is wrong with your federation.

That’s too bad, that’s a hard line for me, it has to have the option of federation.

Also, so it’s a direct server, so it is centralized, there’s nothing decentralized about it

Decentralized (no abuse of power and doesn’t have a single point of failure)

There is a direct server though, is it federated? The readme doesn’t say it’s federated at all

nerd herd

I understood that reference!

I’ve heard positive things about Dito, if I was doing it over again I think I’d start there

Can confirm, I host Matrix (homeserver synapse) and Element. Voice is a pain to get set up but I hear there are other matrix services which will do this for you easier. It’s a process though. You can get text chat up in a day, voice is going to be a bit after that, a lot of tinkering.

If you’re only at 2 nodes, then I think host paths with node selectors are what you should go with. That gets you up and running in the short term, but know that the conversion later to something like Longhorn will be a process. (Creating the volumes, then copying all the data over, ensuring correct user access, etc).

So you have a classic issue of datastorage on kubernetes. By design, kubernetes is node-agnostic, you simply have a pile of compute resources available. By using your external hard drive you’ve introduced something that must be connected to that node, declaring that your pod must run there and only there, because it’s the only place where your external is attached.

So you have some decisions to make.

First, if you want to just get it started, you can do a hostPath volume. In your volumes block you have:

volumes:
  - name: immich-volume
    hostPath:
      path: /mnt/k3s/immich-media # or whatever your path is

The gotcha is that you can only ever run that pod on the node with that drive attached, so you need a selector on the pod spec.
You’ll need to label your node with something like kubectl label $yourNodeName anylabelname=true, like kubectl label $yourNodeName localDisk=true Then you can apply a selector to your pod like:

    spec:
      nodeSelector:
        localDisk=true

This gets you going, but remember you’re limited to one node whenever you want data storage.

For multi-node and true clusters, you need to think about your storage needs. You will have some storage that should be local, like databases and configs. Typically you want those on the local disk attached to the node. Then you may have other media, like large files that are rarely accessed. For this you may want them on a NAS or on a file server. Think about how your data will be laid out, then think about how you may want to grow with it.

For local data like databases/configs, once you are at 3 nodes, your best bet with k3s is Longhorn. It is a HUGE learning curve, and you will screw up multiple times as a warning, but it’s the best option for managing tiny (<10GB) drives that are spread across your nodes. It manages provisioning and making sure that your pods can access the volumes underneath, without you managing nodes specifically. It’s the best way to abstract away not only compute, but also storage.

For larger files like media and linux ISOs, then really the best option is NFS or block storage like MinIO. You’ll want a completely separate data storage layer that hosts large files, and then following a guide like this you can enable mounting of NFS shares directly into your pods. This also abstracts away storage, you don’t care what node your pod is running on, just that it connects to this store and has these files available.

I won’t lie, it’s a huge project. It took about 3 months of tinkering for me to get to a semi-stable state, simply because it’s such a huge jump in infrastructure, but it’s 100% worth it.

Helm has worked well for me, what’s the problem you had?