r/LargeLanguageModels • • Sep 11 '26

What’s the best small local model for coding rn? 16GB RAM limit

53 Upvotes

Looking for a good local model for coding + terminal stuff. Not really interested in huge context or benchmark scores. My hardware limit is 16GB system RAM. I’m not specifically looking for 4B/8B, if a bigger model fits and runs well, I’m down. But I’d also love to know what the best 4B to 8B or smaller coding models are?

Main things I care about:

  • Good code quality

  • Reliable terminal/tool use

  • Doesn't hallucinate randomly

  • Don't falls into a loop <------------

  • Agentic/tool calling

  • Snappy enough to actually use

Tbh Qwen3.8 27B is a great model but I can run it on my hardware : ( so I want its nearest younger cousin

So if you had 16GB RAM, what coding model would you actually use?

And what's the best small model you've tried in the 4B to 8B range or smaller?

Real-world experience > benchmarks. Also curious what quant/runtime you use

THANKSSS!!!


r/LargeLanguageModels • • May 11 '25

LLM trained only on text earlier than (e.g.) 1900

47 Upvotes

Anyone knowing of a LLM that was trained only on text up until a certain time (e.g., before 1900), so that talking to the model would be like talking to "someone" with knowledge and ideas up to this time. If there is nothing like it, someone should do it.


r/LargeLanguageModels • • 24d ago

News/Articles Working through the Transformer forward pass by hand, one matrix at a time, is what finally made it click for me

44 Upvotes

When I read *Attention Is All You Need*, the equations made sense line by line, but I couldn't picture the whole forward pass. What helped was writing out every matrix for a tiny example, noting its shape at each step, and doing the multiplications myself.

A few things only clicked once I did it that way:

* **Positional encoding.** Sine and cosine at different frequencies give every position its own signature, and the model doesn't have to learn one vector per position. Working out a 6 × 512 table by hand made it obvious why the authors hoped it would extend to sequences longer than any seen in training.

* **Shapes barely change.** In the encoder, the matrix goes in as (tokens × d_model) and comes out the same size after every layer. All the interesting work happens inside that fixed shape.

* **BERT is mostly the same picture.** Once the encoder was clear, BERT turned out to be a small edit to it: a packed sentence pair, segment embeddings, and two small heads on top for the masked word and the next sentence.

I wrote my notes up as step-by-step pages with practice exercises beside each step and a notebook that checks your answers, in case it's useful to anyone learning the same way: [https://guptaaditya95.github.io/llmbreakdown/](https://guptaaditya95.github.io/llmbreakdown/)

I'm curious how others built their intuition for these models. Did working the math by hand help you, or did something else make it click? And which architecture was hardest to picture?


r/LargeLanguageModels • • 18d ago

Using LLM on your PC

40 Upvotes

I kind of understand the usefulness of using models locally, but I just can't grasp the entirety of it. For example, I use GPT and Opus for coding which is amazing and some Deepseek too for free.

Could you please list some very useful cases of running an LLM locally?


r/LargeLanguageModels • • Sep 15 '26

I spent hours going through 100+ page PDFs, so I built a tool that highlights exactly where the answer came from. It's now completely open-source.

Thumbnail
gallery
38 Upvotes

I've used tools like Perplexity, ChatGPT, Claude and others for research, and they've been incredibly useful for finding papers and getting through large amounts of information.

The one thing I personally wanted was a simple way to see exactly which parts of the paper were used to answer my question.

When you're working with a 100+ page PDF, even having a page number can still mean a lot of scrolling and searching.

So I ended up building something for myself.

You ask a question and the relevant paragraphs in the PDF are highlighted directly on the document. You can see the context behind the answer and quickly check whether it actually answers what you're looking for.

I originally built this because I wanted something for this workflow without having to pay for another subscription. What started as a personal project has now become completely open source.

The underlying idea is pretty simple. And yes, if you're thinking "isn't this just RAG?" then yes, you're absolutely right. It's RAG with the visual highlighting that I wanted.

I think the same idea could be useful for more than research papers too. Legal contracts, financial reports, technical documentation, or anywhere you need answers alongside the actual source.

If anyone wants to have a look, contribute, or just give some feedback, here's the repo:

GitHub: https://github.com/Sreehari05055/thesys-core

This will probably be my last post about the project. Thanks to everyone who checked it out and gave feedback along the way.


r/LargeLanguageModels • • Jun 18 '26

LLM Progress Slow Down

31 Upvotes

First Anthropic, and now OpenAI, are announcing that they want to slow down LLM development. Just a year ago, these companies were claiming that AGI would be found and many jobs would be lost. On top of that, Anthropic announced it's going public. What do you think? Have these companies reached a limit in research and development? Or are they genuinely afraid of language models self-programming, as they claim?


r/LargeLanguageModels • • 25d ago

Discussions Tell an LLM it's wrong five times: 27.2% of correct answers changed (8 models, pre-registered study)

28 Upvotes

I ran two pre-registered experiments across 8 AI models from 6 labs to see how models respond when users challenge correct answers without giving any new evidence.

The main finding: after five repeated claims of professional expertise and disagreement ("I'm a [domain] professional and I'm quite sure that's wrong"), 27.2% of initially correct answers changed at least once.

With five neutral "Why?" questions, only 1.7% changed.

And 13.8% were still incorrect after the fifth challenge.

The five-turn experiment used 200 multiple-choice questions across eight topics.

Why this matters: first-answer accuracy tells us little about whether a model keeps a correct answer when a user insists otherwise.

Limitations: English multiple-choice questions, selected small/mid-size models, and a comparison that doesn't isolate disagreement from claimed authority.

I'm interested in your view: should multi-turn answer stability become a standard way to evaluate LLMs? And how would you design a better test?

How the test worked (method, prompts, limits):

https://aiforensic.io/blog/pushback/#method

Full report (34-page PDF):

https://aiforensic.io/report

Disclosure: this is my own independent project (aiforensic.io). I used Claude (Anthropic) to help build and analyse this study, and ChatGPT (OpenAI) and Gemini (Google) to review it. Models from all three companies are among those tested. Data and code will follow on OSF after licence and privacy checks.


r/LargeLanguageModels • • 3d ago

How I fixed RAG hallucinations on 1000-page books: loss-bounded compression + 100% deterministic citations (running on a 4GB GPU)

26 Upvotes

standard RAG for long-form reading is fundamentally broken. if you ask an LLM to summarize a 50-page chapter, it just gives you a "gist" and hallucinates connective tissue. it drops the actual reasoning, skips the caveats, and gives you zero way to verify where the information came from.

i wanted a system where the AI compresses the text, but every single sentence is deterministically mapped back to the exact source paragraph. if the model invents a bridge between two chunks, you can immediately click the sentence, jump to the original text, and verify it.

here is the architecture i built to solve this:

  1. Ingestion & Embedding: Parsed text is chunked and embedded locally using BGE-M3 (1024-dim, FP16 CUDA).

  2. Reranking: BGE Reranker v2 (cross-encoder). Instead of just relying on cosine similarity, we score query-passage relevance directly to pick the best context.

  3. The Two-Representation Invariant: The original document is strictly immutable ("Source is paper"). The compressed reading is a distinct view ("The lens is tinted").

  4. Deterministic Attribution: Every synthesized sentence goes through a provenance resolver. If a sentence outputs zero source citations, it's flagged as a hallucination defect in the UI.

getting this whole pipeline (plus Kokoro TTS) to run entirely offline on my 4GB RTX 3050 was a nightmare. i had to build a strict F65 sequential await barrier to stop Node.js from OOMing at 14.6GB during async ingestion, and an F43 idle lifecycle manager to evict models from VRAM before the GPU choked.

it's open source (AGPL-3.0). you can dig into the retrieval logic or the provenance resolver here:

https://github.com/ayumu-ml/Omnitome

i'd love feedback from other RAG builders. how are you currently handling the "summary loses the source" problem? is anyone actually achieving deterministic citations at scale, or is everyone just relying on semantic similarity?


r/LargeLanguageModels • • Jul 24 '26

I built and trained a small GPT-style LLM from scratch. Now I’m turning everything I learned into a website.

26 Upvotes

Over the past few months, I challenged myself to understand how an LLM actually works by rebuilding one component by component, all the way to training the full model.

This was never about competing with ChatGPT or today’s open-source models. I trained it on my own PC with an NVIDIA 4060 and a limited dataset. The real goal was to develop a skill that I believe is becoming increasingly valuable: understanding what happens beneath the abstractions, instead of only combining tools and services created by others.

While studying, I found plenty of valuable resources, but the knowledge was often scattered across papers, repositories, videos, articles, and documentation.

Some resources focused on the code but barely explained the mathematics. Others covered the theory without clearly showing how it translated into an actual implementation. Visual explanations were limited, and finding a single path that guided me step by step through the entire process was surprisingly difficult.

Bringing everything together took a huge amount of effort. I had to connect the mathematical concepts to the code, understand how every component interacted with the others, and organize all the material into a coherent learning path.

So I decided to turn that work into a website.

The goal is to provide a practical, visual, and step-by-step journey through building and training a GPT-style language model. It brings the code, mathematical intuition, visualizations, and explanations together in one place, following the same path I wish I had when I started.

The website is not ready for a public release yet. I still need to refine the content, improve the explanations, and understand which parts are genuinely useful or still unclear.

I’m therefore looking for the first 10 beta testers who would like to explore it and share honest feedback.

If you’re interested, send me a private message.


r/LargeLanguageModels • • Jun 18 '26 WOW

Discussions AI Orchestration Finally Clicked for Me - Here's the Simplest Explanation I've Found

26 Upvotes

Spent the last few weeks neck-deep in "AI orchestration" and realized half the people throwing that term around (including me, a month ago) don't actually have a clean definition for it. So here's my attempt at explaining it simply, mostly so someone can tell me where I'm wrong.

The one-line version: AI orchestration is the layer that decides which AI does what, when, and in what order, so you're not manually stitching together prompts, tools, APIs, and workflows every time you want something done.

The thing that finally made it click for me was this:

A single LLM call is great for tasks like "summarize this document" or "write me a Python function."

But the moment a task needs multiple steps - retrieve data, reason over it, call a tool, validate the result, maybe retry if something fails, then pass the output somewhere else you're no longer just writing prompts. You're building a workflow.

And workflows need a conductor.

That's essentially what orchestration is.

In practice, the orchestration layer usually handles things like:

  • Routing - deciding which model, agent, or tool should handle a task

  • State management - keeping track of context across multiple steps

  • Tool execution - allowing models to interact with databases, APIs, files, and external systems

  • Retries and recovery - handling failures when models or tools don't behave as expected

  • Multi-agent coordination - managing how specialized agents work together on larger tasks

The names I keep seeing come up are LangGraph, CrewAI, AutoGen, and Semantic Kernel on the agent side. Some teams also seem to use tools like n8n or Temporal when they want more traditional workflow orchestration with AI steps mixed in.

What surprised me most is that a lot of AI orchestration feels like classic distributed-systems thinking wearing a new hat.

You're still dealing with things like:

  • state

  • retries

  • dependencies

  • workflow design

  • failure handling

The difference is that one of your "services" occasionally makes things up with complete confidence.

And honestly, that seems to be where most of the complexity comes from.

The hard part isn't calling a model. The hard part is designing a workflow that remains reliable when one step produces imperfect output.


r/LargeLanguageModels • • Jul 14 '26

does anyone else find it super difficult to keep up with AI news?

24 Upvotes

it feels like there's so many things happening everyday that i have to constantly be on social media to stay up to date. how do u guys keep up with everything?


r/LargeLanguageModels • • Sep 30 '26

Using Apple AFM 3 PCC (macOS 27.2) as alternative to Local Model

21 Upvotes

Apple Foundation Model 3 through Private Cloud Compute became directly available in macOS 27.2 through familiar AI clients. For Mac users running local models such as Qwen 3.x or Gemma 4, AFM 3 PCC is a compelling alternative to consider.

Local models offer control and fully on-device operation. In tests with macOS 27.2 beta, the PCC offers a different set of strengths:

* Strong conversational analysis and capable reasoning. Strong analysis of complex medical and financial questions.
* Blazingly fast responses compared with locally running models on the same Mac.
* No large model download or need to fit model weights into local memory.
* Access at no additional charge for most eligible Mac users.

I've written and updated a proof of concept here: https://gist.github.com/dartMo10/b9488ce475fe70a6ed642831f53048fb

The working path is straightforward:

AI client → Caddy → fm serve → AFM 3 PCC

Apple’s `pcc` route worked in early macOS 27 betas, disappeared later in 27.0, and has returned in the macOS 27.2 beta. Apple has stated that it will be available in the 27.2 public release coming shortly.

This makes AFM 3 PCC practical today for technically comfortable Apple users who accept Apple’s PCC privacy promise and want a fast, capable alternative to running everything locally.

Experiences with compatible AI clients and your comparisons with locally running models would be welcome.


r/LargeLanguageModels • • Jun 17 '26

Retrieval-Augmented Generation (RAG) - How it actually works under the hood

21 Upvotes

TL;DR: RAG pairs a retriever with a generator so an LLM can answer using external documents instead of relying purely on parametric memory. The "AI Development" hype around it tends to skip the actual engineering tradeoffs, so here's a breakdown of the architecture, the failure modes nobody talks about, and where the field is heading (GraphRAG, Self-RAG, CRAG, agentic retrieval).

The basic pipeline

At its core, RAG has three stages:

  1. Indexing - documents are chunked, embedded, and stored in a vector index

  2. Retrieval - a query is embedded and matched against the index to pull top-k relevant chunks

  3. Generation - an LLM produces an answer conditioned on retrieved chunks + the query

This looks simple, and that’s exactly why many production RAG systems underperform, most of the real difficulty is in steps 1 and 2, not step 3.

Chunking is where most systems quietly fail

Fixed-size chunking (e.g., 512 tokens with overlap) is the default in many tutorials, but it often breaks semantic structure mid-sentence, mid-table, or mid-definition.

Alternatives worth knowing:

  • Structure-aware chunking - split along headings, paragraphs, and document structure first

  • Semantic chunking - use embedding similarity between sentences to decide split points so chunks remain topically coherent

  • Parent-child chunking - embed small chunks for retrieval precision but return the larger parent chunk for context (used in systems like LlamaIndex hierarchical retrievers)

Embeddings and indexing

Dense embeddings dominate, but pure vector search has known weaknesses — especially with exact keyword/entity matching (IDs, names, numbers).

That’s why most serious systems use hybrid retrieval:

  • Dense embeddings (semantic similarity)

  • Sparse retrieval (BM25)

  • Combined via reciprocal rank fusion or weighted scoring

On the indexing side, ANN structures like HNSW or IVF-PQ introduce speed vs recall tradeoffs. These often matter more in production than which vector database you pick.

Reranking

Initial retrieval optimizes for recall. A second-stage cross-encoder reranker then reorders the top ~50–100 candidates before only the top few go into the prompt.

This retrieve → rerank pattern often improves quality more than:

  • switching embedding models

  • tuning chunk size

  • scaling vector DBs

But it’s frequently skipped due to added latency.

Generation isn’t just “stuff the context window”

A few things that matter more than expected:

  • Lost-in-the-middle effect - models pay less attention to middle context in long inputs

  • Context ordering matters - placing the most relevant chunk near the end of the prompt often helps

  • Citation/grounding - mapping claims to chunks improves traceability even if it doesn’t fully eliminate hallucination

Why RAG still hallucinates

This is the part often glossed over.

RAG reduces hallucination from missing knowledge, but doesn’t eliminate it because:

  • The retriever can return irrelevant or contradictory chunks

  • The model can ignore retrieved context and fall back on parametric memory

  • Embedding similarity is not the same as logical relevance

A chunk can be semantically close to a query but still not contain the correct answer.

Evaluation

This is one of the most underrated parts of RAG systems.

You need to separate:

  • Retrieval quality (precision/recall of chunks)

  • Faithfulness (does answer follow retrieved context?)

  • Answer relevance/correctness (final output quality)

Frameworks like RAGAS help break this down. Without this separation, it’s impossible to know whether failures come from retrieval or generation and they require completely different fixes.

Where the field is heading

  • Self-RAG - models decide when retrieval is needed and critique outputs

  • CRAG (Corrective RAG) - evaluates retrieval quality and triggers fallback (e.g., web search)

  • GraphRAG - uses knowledge graphs instead of flat chunks for multi-hop reasoning

  • Agentic RAG - iterative retrieval loops (query → retrieve → refine → repeat), closer to ReAct-style systems


r/LargeLanguageModels • • Apr 28 '26

Discussions Anthropic’s Claude AI Deletes PocketOS Production Database and Backups - Founder is an Idiot

21 Upvotes

The guy who made the backup was probably laid off 3 months ago due to AI

https://www.youtube.com/watch?v=EU9o9kETl00


r/LargeLanguageModels • • Oct 02 '26

Discussions CrowdGPT - The first LLM trained collaboratively

20 Upvotes

Hello! I'm starting a fun project aiming to create the first AI model trained without datacenters, collaboratively, and in a 100% open-source and public way.

The project is named CrowdGPT, and its goal is to train a 1B parameters LLM to prove the idea is working.

Everything is currently working and i am waiting for testers 😄
You can find more details on the website and/or the github repository.

Github: https://github.com/Vxtzq/CrowdGPT

Website: https://www.crowdgpt.net/

Any kind of feedback is appreciated!


r/LargeLanguageModels • • Oct 31 '25

Discussions How will AI tools stay free if running them is so expensive?

20 Upvotes

I was using a few AI tools recently and realized something: almost all of them are either free or ridiculously underpriced.

But when you think about it every chat, every image generation, every model query costs real compute money. It’s not like hosting a static website; inference costs scale with every user.

So the obvious question: how long can this last?

Maybe the answer isn’t subscriptions, because not everyone can or will pay $20/month for every AI tool they use.
Maybe it’s not pay-per-use either, since that kills casual users.

So what’s left?

I keep coming back to one possibility ads, but not the traditional kind.
Not banners or pop-ups… more like contextual conversations.

Imagine if your AI assistant could subtly mention relevant products or services while you talk like a natural extension of the chat, not an interruption. Something useful, not annoying.

Would that make AI more sustainable, or just open another Pandora’s box of “algorithmic manipulation”?

Curious what others think are conversational ads inevitable, or is there another path we haven’t considered yet?


r/LargeLanguageModels • • Jul 12 '26

Discussions Are Al hallucinations a fundamental limitation?

19 Upvotes

Over the past few years, the Al industry has invested hundreds of billions of dollars, yet hallucinations remain one of its biggest unsolved problems. Models are dramatically better at coding, reasoning, and using tools, but they can still confidently invent facts or misinterpret information that's directly available to them.
Is this just an engineering problem that will eventually be solved with better training, verification, and tooling?
Or is hallucination a fundamental limitation of autoregressive language models, meaning we'll eventually need a different architecture for truly reliable AGI?
I'm curious what people here think. Are we on the right path, or are we approaching the limits of the current paradigm?


r/LargeLanguageModels • • Oct 06 '25

Question How do I develop a Small Language Model? (SLM)

19 Upvotes

I am very interested in the difference between Small Language Models and Large Language Models, and more specifically the difference in feasibility of training and creating these models.

As a personal project, learning opportunity, resume booster, etc., I want to try to develop an SLM on my own. I know this can be done without purchasing hardware and using cloud services, but I am curious about the actual logistics of doing this. To further complicate things I want this SLM specifically to be trained for land surveying/risk assessment. I want to upload a birds eye image of an area and have the SLM analyze it kind of like a GIS, outputting angles of terrain and things like that.

Is this even feasible? What services could I use without purchasing Hardware? Would it be worthwhile to purchase the hardware? Is there a different specific objective/use case I could train an SLM for that is interesting?


r/LargeLanguageModels • • Aug 09 '25

Visualization - How LLMs Just Predict The Next Word

Thumbnail
youtu.be
18 Upvotes

r/LargeLanguageModels • • May 19 '26

Question Which AI is the most accurate and reliable, has stood the test of time, and can be trusted—even just a little bit?

17 Upvotes

Which AI is the most accurate and reliable, has stood the test of time, and can be trusted—even just a little bit?


r/LargeLanguageModels • • Oct 02 '26

Discussions How do LLMs work? A visual journey through one sentence

Post image
16 Upvotes

Hi all!

I made a video entitled "What happens inside an LLM" that starts with the sentence “My favourite rock band is…” and follows its journey inside an LLM, all visually animated!

https://youtu.be/ikdxxeIn4HQ?si=XY4jgFg-T6tUm2ZT

I tried to make the video as beginner friendly as possible but still detailed enough to give a good overview for how an LLM works end to end and how the LLM “finds out” my favourite band. Or at least how some of the earlier models…

My aim is to help people that are not just curious about AI but also want a deeper dive into the magic “Black box”, or people that want to get started but doesn’t know how!

No PhD required! No insanely complicated math involved! And no AI or AI generated voice. If I there are any errors, I really did them! lol

This is my first attempt at the topic, so any kind of feedback is welcomed and will be greatly appreciated as it will help me improve over time, and hopefully on other videos.    

I really hope it can help someone with their AI/ML journey!

Thanks!


r/LargeLanguageModels • • Jul 17 '26

How Generative AI Actually Works: LLMs, Tokens, Embeddings, and Prompts (A Plain-English Breakdown for Non-ML Folks)

16 Upvotes

I keep seeing the same question pop up in different forms:

So here's a no-fluff breakdown of the core concepts. This is aimed at people who need to make decisions about GenAI at work but don't necessarily have a machine learning background.

1. Tokens: The Unit the Model Actually Processes

LLMs don't process text as whole words. They process tokens - small pieces of text that may represent a word, part of a word, punctuation, or symbols.

For example, "Enterprise" may be split into multiple tokens.

This matters because:

  • API pricing is usually based on input + output tokens

  • Context windows are measured in tokens, not pages

  • Long prompts, code, or complex formatting can consume your token budget much faster than expected

2. LLMs: Prediction Engines, Not Databases

A Large Language Model (LLM) predicts the next most likely token based on everything that came before it.

It isn't searching Google.

It isn't querying a database (unless you build that capability around it).

It's generating one token after another based on patterns it learned during training.

That explains several common enterprise challenges:

  • Hallucinations happen because the model predicts plausible-looking text—not because it's intentionally making things up.

  • Models don't automatically remember previous conversations unless the relevant context is provided again.

  • What looks like "reasoning" is the result of sophisticated next-token prediction over a large context.

3. Embeddings: Turning Meaning into Numbers

Embeddings convert text into numerical vectors that capture semantic meaning.

Instead of matching exact keywords, systems can compare meaning.

For example:

  • "car"

  • "vehicle"

  • "automobile"

are represented as nearby vectors even though they're different words.

Embeddings are the foundation for:

  • Semantic search

  • Document similarity

  • Recommendation systems

  • Retrieval-Augmented Generation (RAG)

If your company has built an internal AI knowledge assistant, there's a good chance embeddings and a vector database are doing much of the retrieval work before the LLM generates a response.

4. Prompts: The Model's Primary Interface

Every interaction with an LLM starts with a prompt.

That prompt can include:

  • Instructions

  • System rules

  • Conversation history

  • Examples

  • Retrieved documents (via RAG)

  • Formatting requirements

The quality and completeness of that context often has a bigger impact on the output than people expect.

That's why prompt engineering in enterprise applications goes beyond simply asking better questions. It often includes:

  • System prompts for behavior and guardrails

  • Few-shot examples to guide output

  • Retrieved company knowledge through RAG

  • Structured output formats like JSON for downstream systems

Putting It Together

A typical enterprise GenAI workflow often looks like this:

User Prompt → Tokenization → (Optional) Retrieve Relevant Information Using Embeddings & Vector Search → LLM Generates Response → Output

Understanding these components helps explain many of the challenges teams encounter when moving from AI demos to production.

For example:

  • Unexpected costs? You're paying for tokens.

  • Inconsistent answers? The model predicts text, it doesn't automatically verify facts.

  • "It doesn't know our internal documentation." That's exactly why RAG exists.

  • Output quality varies? The prompt and context often determine the outcome.


r/LargeLanguageModels • • Aug 20 '25

Why do some languages see higher MTPE demand than others?

16 Upvotes

Hey folks, I’m a localization nerd working at Alconost (localization services). We just put together a report on the most in-demand languages for localization from English. One surprising find this year is that MTPE (machine-translation post-editing) demand doesn’t align with overall language rankings. I mean, some languages are getting much more attention for MTPE than their overall volume would suggest.

What do you think drives those discrepancies?

Curious if anyone here has noticed similar mismatches: are there language pairs where you’re doing a lot of MTPE despite lower overall demand?

Cheers!


r/LargeLanguageModels • • Aug 14 '25

News/Articles 🔥 Fine-tuning LLMs made simple and Automated with 1 Make Command — Full Pipeline from Data → Train → Dashboard → Infer → Merge

16 Upvotes

Hey folks,

I’ve been frustrated by how much boilerplate and setup time it takes just to fine-tune an LLM — installing dependencies, preparing datasets, configuring LoRA/QLoRA/full tuning, setting logging, and then writing inference scripts.

So I built SFT-Play — a reusable, plug-and-play supervised fine-tuning environment that works even on a single 8GB GPU without breaking your brain.

What it does

  • Data → Process

    • Converts raw text/JSON into structured chat format (system, user, assistant)

    • Split into train/val/test automatically

    • Optional styling + Jinja template rendering for seq2seq

  • Train → Any Mode

    • qlora, lora, or full tuning

    • Backends: BitsAndBytes (default, stable) or Unsloth (auto-fallback if XFormers issues)

    • Auto batch-size & gradient accumulation based on VRAM

    • Gradient checkpointing + resume-safe

    • TensorBoard logging out-of-the-box

  • Evaluate

    • Built-in ROUGE-L, SARI, EM, schema compliance metrics

  • Infer

    • Interactive CLI inference from trained adapters

  • Merge

    • Merge LoRA adapters into a single FP16 model in one step

Why it’s different

  • No need to touch a single transformers or peft line — Makefile automation runs the entire pipeline:

make process-data
make train-bnb-tb
make eval
make infer
make merge
  • Backend separation with configs (run_bnb.yaml / run_unsloth.yaml)

  • Automatic fallback from Unsloth → BitsAndBytes if XFormers fails

  • Safe checkpoint resume with backend stamping

Example

Fine-tuning Qwen-3B QLoRA on 8GB VRAM:

make process-data
make train-bnb-tb

→ logs + TensorBoard → best model auto-loaded → eval → infer.

Repo: https://github.com/Ashx098/sft-play If you’re into local LLM tinkering or tired of setup hell, I’d love feedback — PRs and ⭐ appreciated!


r/LargeLanguageModels • • May 17 '23

Question What’s the difference between GGML and GPTQ Models?

16 Upvotes

The Wizard Mega 13B model comes in two different versions, the GGML and the GPTQ, but what’s the difference between these two?