sylver_dragon

do any of you hate how self-hosting services like photo- or document-management systems, or even a simple rss tool, forces you to sort your stuff out, and put your decades old files in order?!

What is this “sort” thing you speak of? I don’t sort anything, I have NextCloud syncing my entire photos, videos and documents folders and they are just as messy as ever. Granted, I do go through my photos and videos once a year and dump them in a folder named for the year they were taken. Occasionally, I’ll go hog wild and try to sort some of a year’s photos/videos into folders named after events. Though, that hasn’t happened in a number of years. I setup NextCloud so I could have everything synced to my own server and just forget, not have to deal with labeling my data.

As for bookmarks. I already keep those in folders; but, I don’t sync those. I use my desktop far more than I use my phone for web browsing. And the types of things I use my phone for (mostly recipes), I just keep bookmarked there.

No, if you open a terminal and run:
sudo dmesg

You should get a long output which is the kernel log. Assuming the crash happened recently, there may be something in the last few lines (bottom of the output) which could indicate why the process died (or was killed). Keep in mind that this is a running log; so, if it’s been a while since the crash, the entries for it may be higher up in the log. It’s often best (if you can) to trigger the problem then immediately go run the sudo dmesg command and look at the output. With luck, there will be useful logs. If not, you may need to look elsewhere.

On my system (Arch), if I have too much running, the kernel can kill processes based on resource starvation. It’s quite possible you’re running into a similar limitation. There should be a message in dmesg when this happens.

I tried that, I ended up with this weird “Windows 11” adware installed and couldn’t get rid of it. There was also a problem with odd programs and advertising showing up in my Start Menu, even after I removed them. Also, my settings would occasionally just change, without my knowledge or permission.

It depends on the environment. I’ve been in a couple of places which use Linux for various professional purposes. At one site, all systems with a network connection were required to have A/V, on-access scanning and regular system scans. So, even the Linux systems had a full A/V agent and we were in the process of rolling out EDR to all Linux based hosts when I left. That was a site where security tended to be prioritized, though much of it was also “checkbox security”. At another site, A/V didn’t really exist on Linux systems and they were basically black boxes on the network, with zero security oversight. Last I heard, that was finally starting to change and Linux hosts were getting the full A/V and EDR treatment. Though, that’s always a long process. I also see a similar level of complacency in “the cloud”. Devs spin random shit up, give it a public IP, set the VPS to a default allow and act like it’s somehow secure because, “it’s in the cloud”. Some of that will be Linux based. And in six months to a year, it’s woefully out of date, probably running software with known vulnerabilities, fully exposed to the internet and the dev who spun it up may or may not be with the company anymore. Also, since they were “agile”, the documentation for the system is filed under “lol, wut?”

Overall, I think Linux systems are a mixed bag. For a long time, they just weren’t targeted with normal malware. And this led to a lot of complacency. Most sites I have been at have had a few Linux systems kicking about; but, because they were “one off” systems and from a certain sense of invulnerability they were poorly updated and often lacked a secure baseline configuration. The whole “Linux doesn’t get malware” mantra was used to avoid security scrutiny. At the same time, Linux system do tend to default to a more secure configuration. You’re not going to get a BlueKeep type vulnerability from a default config. Still, it’s not hard for someone who doesn’t know any better to end up with a vulnerable system. And things like ransomware, password stealers, RATs or other basic attacks often run just fine in a user context. It’s only when the attacker needs to get root that things get harder.

In a way, I’d actually appreciate a wide scale, well publicized ransomware attack on Linux systems. First off, it would show that Linux is finally big enough for attackers to care about. Second, it would provide concrete proof as to why Linux systems should be given as much attention and centrally managed/secured in the Enterprise. I know everyone hates dealing with IT for provisioning systems, and the security software sucks balls; but, given the constant barrage of attacks, those sorts of things really are needed.

It depends on what your goals are.

• Ventoy is good for having an alternate OS on a Thumbdrive. Even with a USB 3 device, you may encounter I/O blocking and find this isn’t suitable as a “daily driver” OS. However,. for booting something like Tails or Windows/Linux for OS specific hardware/applications, it can be a good solution.
• Dualbooting is a good way to “test drive” an alternate OS and also have a way to fallback to the other OS if you regularly need access to some software which only runs on that OS. This is likely to have better performance than the USB/Ventoy setup at the cost of Windows fucking up the bootloader config from time to time.
• Windows/Linux with a Linux/Windows VM is useful when you know what OS you want to run on a day to day basis, but have some reason to reach into the other OS on occasion and aren’t too worried about performance and hardware access in the alternate OS.

Ultimately, it’s going to come down to what you are trying to do and why you want to run multiple Operating Systems. For example, my main system is running Linux. But, I want the ability to run Windows malware in a controlled sandbox (not a euphemism, I work in cybersecurity and lab some stuff for fun). So, I have KVM setup to run Virtual Machines, including Windows.

For another example, prior to making the switch to Linux, I had Windows as my primary OS and booted Linux on a USB stick (not Ventoy, but close enough). This let me gain confidence that I would be able to make the jump.

I don’t have a good example for dual booting. Maybe something like a SteamDeck where you want a stable, functional OS most of the time; but, have some games which will only run in Windows.

It makes little sense why it works on an offsite WiFi, but not mobile data.

I’d agree with unbuckled above, it’s a DNS issue. If your mobile device is capable, use nslookup or dig to see what responses you are getting in different scenarios. It’s possible that your VPN software is leaking DNS queries out to the mobile data provider’s DNS servers while you are on mobile data and only using the correct DNS settings when you are on wifi. Possibly look for split tunnel settings in the VPN software, as this can create this type of situation.

You can also confirm this from the pihole side. Connect to the VPN via mobile data and browse to some website you don’t use often, but is not your own internal stuff. Then open the query log on your pihole and see if that domain shows up. I’d put money on that query not showing in the pihole query log.

Along with the things others have said (Backups, Linux, Docker, Networking) I’d also recommend getting comfortable with server and network security. A lot of this is wrapped up in the simple mantra “install your goddamn updates!” But, there is more to it than that. For example, if you go with Nextcloud, read through their hardening guide and seriously consider implementing all of the recommendation. Also think through how you intend to manage both the server and instance. If this is all local, then it is easier as you can keep SSH access to the server firewalled off from the internet. If you host part of your stuff “in the cloud”, you’ll want to start looking at limiting down access and using keys to login (which is good practice for all situations). Also, never use default credentials. You may also want to familiarize yourself with the logs provided by the applications and maybe setup some monitoring around them. I personally run Nextcloud and I feed all my logs into Splunk (you can run a free instance in a docker container). I have a number of dashboards I look at every morning to keep an eye on things. E.g. Failed/successful logins, traffic sources, URI requests, file access, etc. If your server is attached to the internet it will be under attack constantly. Fail2Ban on my wireguard container banned 112 IP addresses over the last 24 hours, for 3 failed attempts to login via SSH. Less commonly, attackers try to log in to my Nextcloud instance. And my WordPress site is under constant attack. If you choose to run Wordpress, be very careful about the plugins you choose to install, and then keep them up to date. Wordpress itself is reasonably secure, the plugins are a shit-show and worse when they aren’t kept up to date.

I’m sure there are several out there. But, when I was starting out, I didn’t see one and just rolled my own. The process was general enough that I’ve been able to mostly just replace the SteamID of the game in the Dockerfile and have it work well for other games. It doesn’t do anything fancy like automatic updating; but, it works and doesn’t need anything special.

I see containers as having a couple of advantages:

1. Separation of dependencies - while not as big of issue as it used to be, just knowing that you won’t end up with the requirements for one application conflicting with another is one less issue to worry about. Additionally, you can do anything you want to one container, without having an effect on another container. You don’t get stuck wanting to reboot or revert the system, but not wanting to break a different running service.
2. Portability - Eventually, you are going to replace the OS of that VM (at least, you should). Moving a container to a new OS is dead simple. Re-installing an application on a new OS, moving data and configs can be anywhere from easy to a pain in the arse, depending on the software.
3. Easier fall back - Have you ever upgraded an application and had everything go to shit? In my years working as a sysadmin, I lost way too many evenings to this sort of bullshit. And while VM snapshots should make reverting easy, sometimes it just didn’t work out that way. Containers force enough separation of applications that you can do just about anything to one container and not effect others.
4. Less dependency on a single install - Have you ever had a system just get FUBAR, and after a few hours of digging the answer seems to be, just format the drive and start over? Maybe you tried some weird application out and the uninstall wasn’t really clean. By having all that crap happen in containers, you can isolate the damage. Nuke the container, nuke the image, and the base OS is still clean.
5. Easier version testing - Want to try out upgrading to version 2 of an application, but worried that it may not be fully baked yet or the new configs may take a while to get right? Do it off in a separate container on a copy of the data. You can do this with VMs and snapshots; but, I find containers to be less overhead.

That all said, if an application does not have an official container image, the added complexity of creating and maintaining your own image can be a significant downside. One of my use cases for containers is running game servers (e.g. Valheim). There isn’t an official image; so, I had to roll my own. The effort to set this up isn’t zero and, when trying to sort out an image for a new game, it does take me a while before I can start playing. And those images need to be updated when a new version of the game releases. Technically, you can update a running container in a lot of cases; but, I usually end up rebuilding it at some point anyway.

I’d also note that, careful use of VMs and snapshots can replicate or mitigate most of the advantages I listed. I’ve done both (decade and a half as a sysadmin). But, part of that “careful use” usually meant spinning up a new VM for each application. Putting multiple applications on the same OS install was usually asking for trouble. Eventually, one of the applications would get borked and having the flexibility to just nuke the whole install saved a lot of time and effort. Going with containers removed the need to nuke the OS along with the application to get a similar effect.

At the end of the day, though. It’s your box, you do what you are most comfortable with and want to support. If that’s a monolithic install, then go for it. While I, or other might find containers a better answer for us, maybe it isn’t for you.

My list of items I look for:

• A docker image is available. Not some sort of make or build script which make gods know what changes to my system, even if the end result is a docker image. Just have a docker image out on Dockerhub or a Dockerfile as part of the project. A docker-compose.yaml file is a nice bonus.
• Two factor auth. I understand this is hard, but if you are actually building something you want people to seriously use, it needs to be seriously secured. Bonus points for working with my YubiKey.
• Good authentication logging. I may be an outlier on this one, but I actually look at the audit logs for my services. Having a log of authentication activity (successes and failures) is important to me. I use both fail2ban to block off IPs which get up to any fuckery and I manually blackhole entire ASNs when it seems they are sourcing a lot of attacks. Give me timestamps (in ISO8601 format, all other formats are wrong), IP address, username, success or failure (as a independent field, not buried in a message or other string) and any client information you can (e.g. User-Agent strings).
• Good error logging. Look, I kinda suck, I’m gonna break stuff. When I do, it’s nice to have solid logging giving me an idea of what I broke and to provide a standardized error code to search on. It also means that, when I give up and post it as an issue to your github page, I can provide you with some useful context.

As for that hackernews response, I’d categorically disagree with most of it.

An app, self-contained, (essentially) a single file with minimal dependencies.

Ya…no. Complex stuff is complex. And a lot of good stuff is complex. My main, self-hosted app is NextCloud. Trying to run that as some monolithic app would be brain-dead stupid. Just for the sake of maintainability, it is going to need to be a fairly sprawling list of files and folders. And it’s going to be dependent on some sort of web server software. And that is a very good place to NOT roll your own. Good web server software is hard, secure web server software is damn near impossible. Let the large projects (Apache/Nginx) handle that bit for you.

Not something so complex that it requires docker.

“Requires docker” may be a bit much. But, there is a reason people like to containerize stuff, it avoids a lot of problems. And supporting whatever random setup people have just sucks. I can understand just putting a project out as a container and telling people to fuck off with their magical snowflake setup. There is a reason flatpak is gaining popularity.
Honestly, I see docker as a way to reduce complexity in my setup. I don’t have to worry about dependencies or having the right version of some library on my OS. I don’t worry about different apps needing different versions of the same library. I don’t need to maintain different virtual python environments for different apps. The containers “just work”. Hell, I regularly dockerize dedicated game servers just for my wife and I to play on.

Not something that requires you to install a separate database.

Oh goodie, let’s all create our own database formats and re-learn the lessons of the '90s about how hard databases actually are! No really, fuck off with that noise. If your app needs a small database backend, maybe try SQLite. But, some things just need a real database. And as with web servers, rolling your own is usually a bad plan.

Not something that depends on redis and other external services.

Again, sometimes you just need to have certain functionality and there is no point re-inventing the wheel every time. Breaking those discrete things out into other microservices can make sense. Sure, this means you are now beholden to everything that other service does; but, your app will never be an island. You are always going to be using libraries that other people wrote. Just try to avoid too much sprawl. Every dependency you spin up means your users are now maintaining an extra application. And you should probably build a bit of checking into your app to ensure that those dependencies are in sync. It really sucks to upgrade a service and have it fail, only to discover that one of it’s dependencies needed to be upgraded manually first, and now the whole thing is corrupt and needs to be restored from backup. Yes, users should read the release notes, they never do.
The corollary here is to be careful about setting your users up for a supply chain attack. Every dependency or external library you add is one more place for your application to be attacked. And just because the actual vulnerability is in SomeCoolLib.js, it’s still your app getting hacked. You chose that library, you’re now beholden to everything it gets wrong.

At the end of it all, I’d say the best app to write is the one you are interested in writing. The internet is littered with lots of good intentions and interesting starts. There is a lot less software which is actually feature complete and useful. If you lose interest, because you are so busy trying to please a whole bunch of idiots on the other side of the internet, you will never actually release anything. You do you, and fuck all the haters. If what you put out is interesting and useful, us users will show up and figure out how to use it. We’ll also bitch and moan, no matter how great your app is. It’s what users do. Do listen, feedback is useful. But, also remember that opinions are like assholes: everyone has one, and most of them stink.