Been messing around with a different approach to NLP. Everyone seems to be fine-tuning massive LLMs or calling APIs, but for a lot of structured text tasks, that feels like overkill. Stuff like email classification, intent detection, ticket routing, why should we throw a 100B+ param model at it when a small, purpose-built model works just as well?

So we built SmolModels, small AI models that run locally or via API. No huge datasets, no cloud lock-in, just lightweight models that do one thing well. Open-sourced it here: SmolModels GitHub.

Curious if anyone else is working with smaller NLP models, what’s been your experience?

Hi r/LanguageTechnology:

After a year of work, I published my free NLP course. The course is intended to help anyone who knows Python and a bit of math go from the basics to today's mainstream models and frameworks.

I strive to balance theory and practice, so every module consists of detailed explanations and slides along with a Colab notebook putting the ideas into practice (in most modules).

The course is divided into two parts. In part one, we cover text preprocessing, how to turn text into numbers, and multiple ways to classify and search text using "classical" approaches. And along the way, we'll pick up valuable bits on how to use tools such as spaCy and scikit-learn.

In part two, we dive into deep learning for NLP. We start with neural network fundamentals and go through embeddings and sequence models until we arrive at transformers and the mainstream models of today.

The notebooks cover how to accomplish everyday NLP tasks including extracting key information, document search, text similarity, text classification, finding topics in documents, summarization, translation, generating text, and question answering.

No registration required: https://www.nlpdemystified.org/

In our most recent paper, we show that language models are few-shot learners even if they have far less than 175B parameters. Our method (combining PET and ALBERT) performs similar to GPT-3 on SuperGLUE after training on 32 examples with just 0.1% of its parameter count: https://arxiv.org/abs/2009.07118 - I would be happy about any feedback :)

Want to introduce “The NLP Index”, a new asset in NLP code discovery. It's free and open to the public.

It houses over 3,000 code repositories that one can search including a side bar with some of the most important topics in NLP today. The engine is search as you type and typo tolerant (it’s crazy fast). The index includes the arxiv research paper PDF, ConnectedPapers link, and its GitHub repo.

https://index.quantumstat.com/

First two words are the seeds

• your wife didn't marry you because we've understood all along . Deep Fucking Value . Salute !

• gamestop is still a 300% green day . Hold the line , the funds most likely rebought

• the retard you must not be earth shattering dump . Don’t worry everyone always wipes their first

• the retard strength in this nonsense lmao ! Priceless . Good shit Edit: In discovery we could

• stonks can only place I can warm my tendies . Can someone explain how this movie 2

• your wife and her boyfriend extra close tonight . J . Simpson to do another paper trading

• your wife may have covered the shorts to have joined . I can’t buy more shares

• stonks can only hope for the word "tendies" . # HOLD HOLD HOLD ! 💎🙌 . Guys

• gamestop is going to $100 Million . DO NOT SELL 💎🙌🏻 , BUY ! ! ! ! Available shares just got bodied

If you're new to NLP, these first six modules will get you up and running with the basics.

We'll cover:

• what makes NLP challenging and what we'll learn in the course.

• various ways to preprocess our text depending on our goals including tokenization, lemmatization, part-of-speech-tagging, and more.

• how to turn our text into numbers with basic bag-of-words approaches and performing document similarity and search.

• how to use spaCy and scikit-learn to do all of the above.

No registration needed but you can sign up for updates. Check it out at nlpdemystified.org and let me know what you think!

Hi all, I've been learning about BERTopic recently, an incredibly simple to use library that allows us to do some seriously cool stuff in topic modeling. I dived into the details in this article, in short:

• Uses transformer models to generate meaningful vector representations of text

• Fascinating techniques like UMAP and HDBSCAN are used to produce clusters from these vector representations

• A modified TF-IDF (called c-TF-IDF) finds the most relevant keywords for each cluster and assigns these as the cluster topics

There's a huge amount to BERTopic, and the most recent release means it supports even more components. So, there's much more to cover, but I think this article just about covers everything you need to get started. I hope it's helpful!

Wanted to introduce chai to y'all - it's a free, open-source platform for developers to build and deploy their own chat AIs. Currently over 100 AIs are hosted on the platform, which has over 12,000 conversations 😱

Following the tutorials at https://chai.ml/docs you can deploy your own in under 10 minutes. I think this is such a cool way to learn NLP and it's really fun seeing your score go up on the developer platform.

Hi everyone. I’m not sure if you remember me, but I’m the guy who was practically living on soju and whisky while waiting for the last ACL results. Well, I’m back, and unfortunately, the peer review system has given me another reason to reach for the bottle.

Just went through the ARR March Cycle results, and I am beyond speechless.

As a Corresponding Author, I received a comment that made my heart drop for a second:

"Seems to be a hallucinated reference, duplicate/erroneous references..." followed by a list of supposedly "faked" citations.

Being accused of fabricating references is a grave Ethical allegation. I immediately went into a full-blown panic and spent the last few hours cross-referencing every single entry in our Bibliography.

Here’s the kicker: None of the "hallucinated references" listed by the reviewer actually exist in our manuscript. 🤷‍♂️

The situation is clear: The Reviewer used an LLM to generate the review and blindly Copy-pasted the output without even opening our PDF. The AI hallucinated a list of non-existent errors, and the reviewer had the audacity to give themselves a Confidence 4 while accusing me of academic misconduct based on a hallucination.

It is the height of Irony and Unprofessionalism. A reviewer, entrusted to safeguard the Integrity of a top-tier venue, used an LLM to accuse an author of "hallucinating" a flaw that only existed in the reviewer's own lazy workflow.

I’ve heard the horror stories about the declining Quality of Peer Review in AI research, but this is a new low. We are at a point where "experts" aren't even reading the papers anymore; they are just letting stochastic parrots make serious ethical accusations for them.

How do you even approach a Rebuttal when a "Confidence 4" reviewer hasn't engaged with a single word of your actual work? The Peer Review system is officially broken. I’m so incredibly frustrated that I’ll have to go grab a drink again tonight.

The original transformer model consisted of both encoder and decoder stages. Since that time, people have created encoder-only models, like BERT, which have no decoder at all and so function well as base models for downstream NLP tasks that require rich representations.

Now we also have lots of “decoder-only“ models, such as GPT-*. These models perform well at creative text generation (though I don’t quite understand how or why).

But in many (all?) use cases of text generation, you start with a prompt. Like the user could ask a question, or describe what it wants the model to do, and the model generates a corresponding response.

If the model’s architecture is truly decoder-only, by what mechanism does it consume the prompt text? It seems like that should be the role of the encoder, to embed the prompt into a representation the model can work with and thereby prime the model to generate the right response?

So yeah, do “decoder-only” models actually have encoders? If so, how are these encoders different from say BERT’s encoder, and why are they called “decoder-only”? If not, then how do the models get access to the prompt?

😁 Happy to introduce one of the most comprehesive ChatGPT cheat sheets: a 30 pg. paper highlighting various prompts to manage ChatGPT for generating text. The document not only highlights what ChatGPT can generate but also how it can generate it! Here is the TOC:

1. NLP Tasks

2. Code

3. Structured Output Styles

4. Unstructured Output Styles

5. Media Types

6. Meta ChatGPT

7. Expert Prompting

Google Doc: https://drive.google.com/file/d/1OcHn2NWWnLGBCBLYsHg7xdOMVsehiuBK/view?usp=share_link

Hey guys, I just translated my curated list for learning Deep NLP

It was basically written in Korean, but I decided to translate this into English to share more people who want to study Deep NLP.

It contains Mathematics, Deep Learning, and Natural Language Processing courses and books.

And also introduces famous blogs that publish Deep NLP posts, useful libraries, and Famous Researchers !

I hope you guys enjoy it and if you have any recommendations about this project please submit an issue !

https://github.com/Huffon/NLP101/blob/master/README_EN.md

We just released the Last Translation Benchmark paper. In a massive crowdsourcing effort we collected 3456 unique hard-to-translate examples that break state-of-the-art translation models, and which can be used for more reliable evaluation.

I wanted to share this new library I've been working on and that I open-sourced!.

here are some links to the library:

💻 GitHub Repository

🐍 PyPi project

as the title suggests, you can increase the inference speed of any pretrained T5 model and also decrease the models' size, in a single line of code.

The library can be installed with pip install fastt5. This code snippet from the repository's README gives a concise overview:

from fastT5 import export_and_get_onnx_model
from transformers import AutoTokenizer

model_name = 't5-small'
model = export_and_get_onnx_model(model_name)

tokenizer = AutoTokenizer.from_pretrained(model_name)
t_input = "translate English to French: The universe is a dark forest."
token = tokenizer(t_input, return_tensors='pt')

tokens = model.generate(input_ids=token['input_ids'],
               attention_mask=token['attention_mask'],
               num_beams=2)

output = tokenizer.decode(tokens.squeeze(), skip_special_tokens=True)
print(output)

The fastT5 library exports the T5 model to onnx with past_key_values, then quantizes it and runs it on onnxruntime.

The exported onnx models support the generate() method of huggingface transformers for inferencing.

for more information on the project refer to the repository here.

Discussion thread for ACL 2026 decisions

*UPDATE* Super Duper NLP Repo

Hi, we have updated and added 52 new NLP notebooks, bringing us to 233 total! Several interesting topics from information retrieval to knowledge graphs included in this update. Thank you for contributing: Manu Romero, Abhishek Mishra, Nikhil Narayan, Oleksii Trekhleb, Chris Tran, Prasanna Kumar & Cristiano De Nobili.

Have a good day.

https://notebooks.quantumstat.com/

I'm in the process of pivoting toward a career in NLP. After recently watching the documentary The Social Dilemma, I'm really concerned about the privacy abuses of FAANG and other big tech companies, and I'm wondering how many NLP jobs are to be found outside of such companies.

I don't want to spend my remaining career fine-tuning sentiment analysis models to optimize "engagement" (i.e. addiction) for social media.

What are good ethical careers/employers in this space?

Introducing the Big Bad Database. Helping NLP and ML developers gain access to nearly 200 datasets for natural language processing.

Read about it here: https://medium.com/towards-artificial-intelligence/nlp-dataset-library-14443ddd3084

Database: https://quantumstat.com/dataset/dataset.html

Random thought: NLP in 2010 and NLP in 2020 already felt like two different worlds. The jump was huge.

Now its growing even faster.

So Iam curious how do you think NLP will look in 2030?

What big shifts do you expect? Will it still be mostly scaling transformers or will something completely new take over?

I have compiled a list of NLP papers published on arXiv last month that I enjoyed reading: April 2020 NLP Papers: Reading List

Hope some of you find this helpful 😊 Let me know if you think an important paper is missing!

A lot of buzz around ChatGPT –but not too much in-depth content on the topic...

Actually, OpenAI did not release much technical details so far unfortunately, even though the methodology is based on previous research.

I tried my best to describe in some detail how ChatGPT works and how it was trained in this article:

https://www.assemblyai.com/blog/how-chatgpt-actually-works/

It also contains my personal criticism on some technical aspects of the methodology, as well as relevant references and a link to a cool open source repo for RLHF!

What is the best piece of technical content you read so far about ChatGPT?

Everyday is Christmas and I find it hard to keep up-to-date with all the developments in the field. What is your go-to source on recent developments? I don't want to miss smaller innovations that didn't make a headline in pop-blogs.