bad chart

shockingly enough, corporations always act in their own self-interest

it’s additional rules for the subreddit on top of the site wide rules for all of reddit

Q4 will give you like 98% of quality vs Q8 and like twice the speed + much longer context lengths.

If you don’t need the full context length, you can try loading the model at shorter context length, meaning you can load more layers on the GPU, meaning it will be faster.

And you can usually configure your inference engine to keep the model loaded at all times, so you’re not loosing so much time when you first start the model up.

Ollama attempts to dynamically load the right context lenght for your request, but in my experience that just results in really inconsistent and long time to first token.

The nice thing about vLLM is that your model is always loaded, so you don’t have to worry about that. But then again, it needs much more VRAM.

In my experience anything similar to qwen-2.5:32B comes closest to gpt-4o. I think it should run on your setup. the 14b model is alright too, but definitely inferior. Mistral Small 3 also seems really good. anything smaller is usually really dumb and I doubt it would work for you.

You could probably run some larger 70b models at a snails pace too.

Try the Deepseek R1 - qwen 32b distill, something like deepseek-r1:32b-qwen-distill-q4_K_M (name on ollama) or some finefune of it. It’ll be by far the smartest model you can run.

There are various fine tunes that remove some of the censorship (ablated/abliterated) or are optimized for RP, which might do better for your use case. But personally haven’t used them so I can’t promise anything.

I don’t know, feddit.nl is pretty chill. I always see everything and barely anything objectable

True, but the newest mistral model is already pretty great

let’s just rebrand instances to superleddits and communities to subleddits 🤣

“Reddit but you can block the part that annoys you”

does anyone know why this sudden uptick?

why would I want to stream myself peeing??

Every Linux user has to go through a period of compulsive distro hopping. Don’t worry, eventually you’ll grow tired of it and just settle on one workhorse distro.

How long until CPUs and GPUs just merge into one thing

Fixing anything in industry is like fighting a very big, very lazy elephant seal bull

The only real advantage of local AI is privacy and that it’s much cheaper if you use it a lot.

The only consumer use case I see in the wild with some real momentum behind it is role play.

All the local AI communities I browse are 50% people trying to find usecases for it at their job (like me; unsuccessfully I might add) and 50% people interested in role play.

People will apparently spend thousands to jerk off to a soulless machine demon simulacrum shell of a human.

To be fair, I can see the appeal of local AI for video games, like RPGs. There is this really fun game called “Suck Up”, where you are a vampire trying to convince AI to let you inside their house. That is the one real “killer” application I see atm.

I personally see a lot of other useful usecases for local AI, but from my experience at work, I would estimste it will take another 5 years until any of it is anywhere near consumer ready.

local AI is cool and all, but neither the hardware nor the models are really ready for your average consumer

why are you so interested in logs? are you like a lumberjack?

I’m so glad to see X as a service fail miserably everywhere

It is kind of interesting how open machine learning already is without much explicit advocacy for it.

It’s the only field I can think of where the open version is just a few months behind SOTA in all of IT.

Open training pipelines and open data are the only aspects that could still use improvements in ML, but there are plenty of projects that are near-SOTA and fully open.

ML is extremely open compared to consumer mobile or desktop apps that are always ~10 years behind SOTA

Congratulations 🤠🥾er