𞋴𝛂𝛋𝛆

  • 4 Posts
  • 41 Comments
Joined 2 years ago
cake
Cake day: June 9th, 2023

help-circle
  • The UEFI boot system is tricky and you need to get along with Secure Boot to do this. Secure Boot is outside of the Linux kernel. Both Fedora and Ubuntu have systems for this. Fedora uses the Anaconda system and I believe they do it best. I have had a W11 partition for 2 years and never used it once. It can’t even get on the internet with my firewall setup, but it is there and never had any issues the 3 times I logged into it.

    I think all of the Fedora systems support the shim key and secure boot but I know Workstation does. For Ubuntu I think it is just the regular vanilla Ubuntu desktop that the shim supports. This may be somewhat sketchy with Nvidia or maybe not. Nvidia “”““open sourced””“” their kernel code but the actual nvcc compiler required to build the binaries is still proprietary crap.

    I have a 3080Ti gaming laptop. It isn’t half bad with 16 GB of video RAM from all the way back in 2021. Nvidia is artificially holding back the vram because of monopoly nonsense. The new stuff has very little real consumer value as a result, at least with AI stuff I run. The hardware is a little faster, but more vram is absolutely critical and new stuff that is the same or worse than what I have from 3 generations and nearly 5 years ago is ridiculous.

    The battery life blows and the GPU likely won’t even work on battery. It will get donkey balls hot with AI workloads, especially any kind of image gen. This results in lots of thermal throttling. All AI packages run as servers on your network. If you are thinking along these lines if running your own models, get a tower and run the thing remotely.

    I manage, and need the ergonomics for physical disability reasons, but I still would prefer to have a separate tower to run models from.

    Anyways, you can sign your own UEFI keys to use any distro, but this can be daunting for some people. The US defense department has a good PDF guide on setting your own keys. The UEFI bootloader for the machine may not have all key signing features implemented. There is a way to boot into UEFI directly and set the keys manually but this is not easy to find great guides on how to do it step by step. Gentoo has a tutorial on this, but it assumes a high level of competency.

    Other than signing your own keys, the shim keys mentioned are special keys signed by Microsoft for the principal maintainer of the distro. These slide under the Microsoft key to keep secure boot enabled.

    If you boot any secure boot enabled OS, the bootloader is required to delete any bootable unsigned code it finds. It does not matter if it is a shimmed Fedora or W11. If you have any other OS present in the boot list, it should be deleted. W11 is SB only, and this is where the real issues arise.


  • 𞋴𝛂𝛋𝛆@lemmy.worldtoLinux@lemmy.mlSecurity Focused Daily Driving Distros?
    link
    fedilink
    English
    arrow-up
    4
    arrow-down
    3
    ·
    edit-2
    25 days ago

    Are you insane? Debian is a base distro like any other and runs more hardware than any other. It has all of the bootstrapping tools to get hardware working.

    Canonical is a server company and Ubuntu server is literally the product.

    Arch is absolute garbage for most users unless you have a CS degree or you have entirely too much time on your hands and don’t mind an OS as your life project. Arch abhors tutorial content in all documentation and therefore dumps users into a rabbit hole regularly. Pacman is the worst package manager as it will actively break a system and present the user with the dumbest of choices at random because the maintainers are ultimately sadistic and lackadaisical. Arch is nearly identical to Gentoo with Arch binaries often based on Gentoo builds, yet Gentoo provides relevant instruction and documentation with any changes that require user intervention and does so at a responsible and ethical level that shows kindness, respect, and consideration completely absent from Arch. Arch is a troll by trolls for trolls. I’m more than capable of running it now, but I would never bother with such inconsiderate behavior.







  • Why are you confrontational? I’m just casually tossing out ideas and learning. Of course I understand what you are saying. However, busybox covers the core of a POSIX system and with the size constraints, it is likely standardising something like this. On Gentoo, such a change might be more straight forward instead of some sloppy hack with a wrapper.

    I imagine you must be good at memorizing a lot of information. I am not. I am good at abstraction and must explore in abstraction to understand heuristically. I understand heuristic connections better than most people. Neither method is better or worse. Being toxic about interchanges of information is useless nonsense. I know far more than I let on, but I’m well aware that I am a jack of all trades and expert of none. All the projects don’t matter relative to those that are used the most. If most projects can be colorized, it will motivate others to fall in line or prompt rewrites assuming such a change was popular. Colorized manpages and help pages should be standard and should have been a decade ago. No one is using an IDE without syntax highlighting. The terminal is an extension of the abstracted language of Linux. Without universal syntax highlighting for new users in these spaces, Linux is presenting an outdated language format ripe for deprecation. These details have long term consequences.










  • I haven’t looked into the issue of PCIe lanes and the GPU.

    I don’t think it should matter with a smaller PCIe bus, in theory, if I understand correctly (unlikely). The only time a lot of data is transferred is when the model layers are initially loaded. Like with Oobabooga when I load a model, most of the time my desktop RAM monitor widget does not even have the time to refresh and tell me how much memory was used on the CPU side. What is loaded in the GPU is around 90% static. I have a script that monitors this so that I can tune the maximum number of layers. I leave overhead room for the context to build up over time but there are no major changes happening aside from initial loading. One just sets the number of layers to offload on the GPU and loads the model. However many seconds that takes is irrelevant startup delay that only happens once when initiating the server.

    So assuming the kernel modules and hardware support the more narrow bandwidth, it should work… I think. There are laptops that have options for an external FireWire GPU too, so I don’t think the PCIe bus is too baked in.


  • 𞋴𝛂𝛋𝛆@lemmy.worldtoSelfhosted@lemmy.worldConsumer GPUs to run LLMs
    link
    fedilink
    English
    arrow-up
    9
    arrow-down
    1
    ·
    edit-2
    3 months ago
    Anything under 16 is a no go. Your number of CPU cores are important. Use Oobabooga Textgen for an advanced llama.cpp setup that splits between the CPU and GPU. You'll need at least 64 GB of RAM or be willing to offload layers using the NVME with deepspeed. I can run up to a 72b model with 4 bit quantization in GGUF with a 12700 laptop with a mobile 3080Ti which has 16GB of VRAM (mobile is like that).

    I prefer to run a 8×7b mixture of experts model because only 2 of the 8 are ever running at the same time. I am running that in 4 bit quantized GGUF and it takes 56 GB total to load. Once loaded it is about like a 13b model for speed but is ~90% of the capabilities of a 70b. The streaming speed is faster than my fastest reading pace.

    A 70b model streams at my slowest tenable reading pace.

    Both of these options are exponentially more capable than any of the smaller model sizes even if you screw around with training. Unfortunately, this streaming speed is still pretty slow for most advanced agentic stuff. Maybe if I had 24 to 48gb it would be different, I cannot say. If I was building now, I would be looking at what hardware options have the largest L1 cache, the most cores that include the most advanced AVX instructions. Generally, anything with efficiency cores are removing AVX and because the CPU schedulers in kernels are usually unable to handle this asymmetry consumer junk has poor AVX support. It is quite likely that all the problems Intel has had in recent years has been due to how they tried to block consumer stuff from accessing the advanced P-core instructions that were only blocked in microcode. It requires disabling the e-cores or setting up a CPU set isolation in Linux or BSD distros.

    You need good Linux support even if you run windows. Most good and advanced stuff with AI will be done with WSL if you haven’t ditched doz for whatever reason. Use https://linux-hardware.org/ to see support for devices.

    The reason I mentioned avoid consumer e-cores is because there have been some articles popping up lately about all p-core hardware.

    The main constraint for the CPU is the L2 to L1 cache bus width. Researching this deeply may be beneficial.

    Splitting the load between multiple GPUs may be an option too. As of a year ago, the cheapest option for a 16 GB GPU in a machine was a second hand 12th gen Intel laptop with a 3080Ti by a considerable margin when all of it is added up. It is noisy, gets hot, and I hate it many times, wishing I had gotten a server like setup for AI, but I have something and that is what matters.