​While training a 253M parameter language model across 4 anchor Dravidian languages (Kannada, Tamil, Telugu and Malayalam), 4 low resource Dravidian languages (Tulu, Koya, Kuvi and Kolami) and English. We ran into two things on the tokenization and multilingual training side that might be useful for anyone working on Indic scripts.

​1st, standard byte level BPE tokenizers shred South Indian scripts into short UTF-8 fragments (around 3.67 bytes per token on a typical 48k GPT style baseline). Using Unicode NFC normalization and a pre-tokenization regex that groups \[\\p{L}\\p{M}\\u200c\\u200d\]+ with a 32768 vocabulary brought our token-weighted average up to 9.680 UTF-8 bytes per token across Kannada (9.717), Tamil (9.654), Telugu (8.910) and Malayalam (10.531). Across all 8 Dravidian languages, it averages 8.693 bytes per token, compared to 3.159 on English.

​2nd, raw training loss hid a clear regression when we mixed in STEM, math and Python data during pretraining. Training loss dropped from \~2.9 to \~2.0, while held out bits per byte got worse across all 9 languages, including a 7% jump on Kuvi and Kannada moving from 0.489 to 0.500. Switching the next phase to a 50/50 split with multilingual replay recovered BPB across all 11 validation sets.

​​On a 300 prompt held out check the final model gets 100% script purity across Kannada, Tamil, Telegu and Malayalam(80/80) and 96.25% format conformance (77/80 with all three misses in Tamil) alongsside 17.18% average (25.00% best) on Belebele Dravidian. Tulu, Koya, Kuvi and Kolami are still experimental because clean, commercially usable text and validation sets are thin.

​The tokenizer regex, byte per token numbers and eval notes are written up in the technical documentation.

For anyone else working on agglutinative or low resource Indic languages: have you found that akshara level pre-tokenization hurts morphological generalization compared to finer subwords annd what replay ratios or data sources have worked for you when mixing in English code or STEM?

I've recently started planning my moving abroad and unfortunately I do not know anything about universities/countries where CL is in demand nowadays. Actually Ik 3 languages except latin (which ive just started learning at uni). They are – Russian (Native), English (I passed the Cambridge exam 7 years ago for B2), German (A2).

Hi there! I'm a senior undergrad student who's going to apply for a Natural Language Processing Master or some other related field. I have a strong interest in investigation and I'm trying my best to learn CS skills. In order to strengthen my math profile I also took the GRE subject math test and got a 600(32% percentile). This isn't my highest level but this test is kinda hard even for math majors.

Do you guys think that it's still worth to study a Master in NLP or CL? Or does AI have already taken all of its future? If it's screwed, should I just turn to study analytics or quant to not waste my time?

How are you evaluating transcription accuracy when speakers switch languages mid-conversation?

I've been looking at how multilingual transcription should be tested in real-world conditions.

A recording with one speaker using one language is relatively straightforward to evaluate.

Things become more interesting when:

- A speaker switches languages in the middle of a sentence.
- Proper nouns are pronounced differently across languages.
- Two speakers use different languages.
- Technical terminology remains in English.
- Background noise affects one language more than another.

Word error rate is useful, but I'm not convinced it captures every failure that matters to the reader.

For example, a transcript could have a reasonable overall error rate while badly misrepresenting a key decision made during a language switch.

For anyone working with multilingual speech recognition, what additional evaluation methods have proved useful?

I've been helping a small archive turn a run of early-1900s newspapers into something researchers can search and read. Getting the text off the page turned out to be the easy half.

The pages are six to eight columns, ads break across gutters, headlines span several columns, and articles jump to another page. Plain OCR reads in horizontal strips, so you get line one of every column, then line two, and so on. The words are all there, but no article can be read start to finish, and nobody has time to reassemble them by hand.

The other loss is structure. A table of shipping arrivals or election returns comes out as a paragraph of numbers. A caption gets separated from the engraving it describes. For research use, those relationships matter as much as the text.

What has helped so far:

- running layout detection before recognition, so each column block is read on its own

- keeping a small set of the worst pages (boxed sidebars, a full-width engraving between columns) as a regression set

- spot-checking reading order on a sample from every reel instead of trusting a confidence score

For the hardest pages I've been testing OXR from SoMarkAI, an open-weight parser that handles recognition and reading order in one pass and keeps the layout model as a separate, swappable part. Too early to say whether it holds up across a full reel.

For anyone who has done large newspaper or archival runs: how do you check reading order at scale? Human sampling, or have you found an automated check that catches the bad reorderings?

Most steps of a conversational procedure aren't tool calls but things the agent has to say: asking for information, confirming, telling the result. Saying them, in the right order, is a third axis of an agent's reliability, alongside factual accuracy and correct tool use, and one that has been largely overlooked. For example, an agent can fetch the forecast correctly and never tell you, or issue the right refund without first asking you to confirm it: everything it said was true, the tool calls were correct, and still a step of the procedure was skipped.

Since we felt this gap was worth working on, in collaboration between Idiap and Logitech, we decided to take a first step: we built a dataset of over 36k dialogues, defined three tasks on it, from finding the turn where an agent drifts to comparing whole sets of dialogues, and wrote a paper with first baselines for each. Together, they make DialFork, to our knowledge the first benchmark for this procedural hallucination.

Three tasks, from the turn to the whole set:

• The turn (Task 3): find the turn where the drift starts, on the whole dialogue or turn by turn as it unfolds.

• The dialogue (Task 2): measure "how far" (distance) a dialogue drifts from a correct one. Both talk about the same thing, so this needs a procedural distance, not semantic similarity.

• The set (Task 1): a single procedural distance between two sets of dialogues, e.g. an agent before and after a model upgrade.

The reported baselines, from embedding-based distances to streaming detectors and zero-shot LLMs, leave plenty of room. Each task has its own track on a public leaderboard on Hugging Face. If you'd like to try your own method, all it takes to add it is a CSV file of predictions.

• Leaderboard: https://huggingface.co/spaces/DialFork/leaderboard

• Paper: https://arxiv.org/abs/2610.21561

• Code: https://github.com/idiap/dialfork

• Dataset: https://huggingface.co/datasets/DialFork/dialfork

I'm one of the authors btw. Since the problem lies in what the agent says (language) rather than in its tool calls, we thought people here might find it useful, and feedback, including critical, is very welcome.

I've been running doc OCR benchmarks since the old opencv+tesseract days. Vision-language models were supposed to kill that whole stack. They didn't. Getting clean markdown out of a two-column academic PDF with tables and footnotes still wrecks most tools I try.

The handoff problem is what gets me. Detection runs, then crops go to recognition, then some geometry heuristic guesses reading order. By the time a figure spans columns or a table nests inside another, you're already toast. I hit this again this week on the same stress-test PDFs I always use. Table+figure+caption groupings split apart or land in the wrong order. Same story as three years ago.

I've been looking at OXR from SoMarkAI. Single pass for recognition plus reading order, and the layout part is swappable without touching the backbone. That's the part I'm actually curious about. I still run my own hybrid pipeline for production stuff. The question is whether anyone here has moved to a single-pass setup for real workloads, and whether pluggable layout is worth the tradeoffs in practice. What breaks when you actually own the maintenance?

hi I’m a social science researcher interested in NLP. for the past few years I thought this was the state of the art method for social science researchers but now with general LLMs being amazingly good at annotations, is there any need in using more traditional methods? or are they still relevant due to cost, accuracy or reproducibility? If we have all the money in the world to use general llms, what Is the reason to use and learn other methods for information retrieval, text analysis, annotation, sentiment analysis etc? very confused and lost!

I spent two months on the wrong layer. Retrieval looked fine, the embeddings looked fine, and the agent still made up relationships between numbers that were never in the same table. When I finally dumped the Markdown that was going into the context window, the tables were paragraphs, footnotes were spliced into sentences, and two-column pages were interleaved line by line. The model was doing its best with broken input.

The tool history, for anyone on the same path. PyMuPDF: fast, fine for simple text, falls over on two-column papers with margin notes. Marker: better structure, but it dropped equations and mangled nested tables for me. Commercial parsing APIs: worked, but at my volume the bill was hard to justify. Training my own layout model: I estimated six months to reach parity with things that already exist, so no.

What I'm testing now is OXR from SoMarkAI (SoMarkAI/OXR on GitHub). Open weights, 1.3B parameters, released last month. The design is a separate layout model you can swap out, then recognition and reading order in a single pass, and they post-trained it with a recursive self-improvement loop. They credit the results to both pieces together. In SoMark's comparison against published scores it ranks first on olmOCR-Bench (87.84), OmniDocBench (97.18) and DocParsingBench (87.76). License is Apache 2.0 with extra terms, including a commercial license above USD 2M in funding or revenue, so read it before you build a product on it. Self-hosting wants about 24GB of VRAM, 48GB recommended.

I'm running it against my set of worst documents before I switch anything in production. So far the formula and math handling is the clearest improvement over my current stack, which matters because that's where my old output was least usable.

The question I keep going back and forth on is whether parsing is a layer worth owning at all. Some teams I know pay for an API and never think about it. Others run everything local down to the embeddings. Where did you land, and did changing the parser move your error rate?

Recently returned from EMNLP, and one thing that surprised me was how many papers were accepted that I would previously have considered somewhat outside the scope of a main NLP conference paper -for example, work focused primarily on audio or vision-only models, purely empirical evaluations, or datasets generated using GPT-4o with limited or no human annotation.

One example I came across was a hate-speech paper where the main contribution was fine-tuning existing models on one dataset and evaluating them on another (i.e., transfer analysis). The work may still be useful and interesting, but the contribution was primarily empirical.

I discussed this with several PhD students who struggled for ~half a year to get papers accepted despite introducing what seemed like more substantial methodological contributions, such as new training methods. Of course, novelty alone does not automatically make a paper better, and empirical work can also be highly valuable. Still, the contrast made me wonder whether our expectations for what constitutes a main-conference contribution have changed.

I wanted to make this thread to hear from more experienced researchers - Do you think that we should introduce more distinct tracks at EMNLP/ACL conferences for empical introducing new task, empirical introducing new evaluation method, purely methodological, empirical-only without new shown effects, dataset/resource, multimodal, and evaluation-focused work?
Probably I got the wrong impression from many works presented at EMNLP this year specifically.

UPD:
This thread was created with the purpose distinct from complaing that EMNLP publishes empirical papers.
The problem in this thread that I wanted to discuss - what is empirical study nowadays? and how we should handle the introduced difference in what we consider empirical now, especially given increasing N of papers: introducing precise different tracks, for instance. For me, empirical contribution remains, in line with previous editions of EMNLP- a new type of evaluation of benchmarks or models, new task with existing datasets, new evaluation approach, new rubric or metric, or new finding which questions the ones shown before or introduces a new problem. Similar to previous years, I think probably most of works should somehow connect to existing language studies or language models, purely vision or audio models analyses were quite at rare previous editions, but not this one.

So, I created this thread to discuss why and how the defintion of what is empirical according to publishing norms transforms, and the scope of EMNLP specifically transforms

Researchers have demonstrated that recurrent neural network architectures are not necessary for building powerful deep learning models for NLP. The Transformer architecture introduced Self-Attention, which allows each token to interact with other tokens in a sequence and capture contextual relationships.

Language is highly dependent on context. The meaning and grammatical role of a word can change depending on its position and relationships with other words.

For example, in Arabic, sentences can be classified into nominal sentences (sentences that begin with a noun) and verbal sentences (sentences that begin with a verb). Furthermore, the grammatical role of a word can depend on its relationships with other words in the sentence.

Another interesting aspect of Arabic is its rich morphology. A single root, such as K-T-B (كتب), can generate a huge number of related words and forms. In fact, Arabic word formation can produce millions of possible forms (sometimes estimated at around 12 million, depending on how forms are counted). For example:

• Kataba (كتب): He wrote.

• Kitaab (كتاب): Book.

• Kaateb (كاتب): Writer.

• Maktaba (مكتبة): Library.

These forms share a common root but have different meanings and grammatical roles.

Therefore, a language model should not rely only on the isolated identity of each token. It should develop contextual representations that encode useful syntactic, semantic, and grammatical information.

The goal is to learn rich contextual representations that help the model understand not only what each token is, but also what role it plays in a given context.

I am curious to understand from NLP practitioners what the current state of the art is for implementing language models within the following type of pipeline, especially when optimizing for speed

Raw/unstructured text -> NER/multi word phrase extraction -> normalize/link to canonical entities/wordforms

I’ve got a recording in another language that I need to understand, and I’m kinda stuck.
I know a few words but nowhere near enough to follow the whole thing. It’s a longer recording too, so manually going through it and translating bits one by one sounds like a nightmare.I mainly just want to get an English version of what’s being said, then I can go back to the audio if something doesn’t sound right. Anyone dealt with this before? What did you end up using? tnx guys

When an AI system encounters incomplete, inconsistent, or contradictory information, how should it determine whether there is enough evidence to produce a reliable answer?

This is an important problem in natural language processing, information extraction, and AI-assisted decision-making. A system may retrieve information from multiple sources, encounter missing values, or find conflicting claims that cannot be resolved using the available evidence.

One possible approach is to use evidence-first, fail-closed decision rules. Under this approach, a system would not proceed with scoring or evaluation when essential evidence or required conditions remain unresolved. Missing information would remain explicitly missing rather than being silently replaced with assumed values.

I am exploring these principles through the Organic Intelligence Protocol (OIP), a research-stage framework focused on evidence requirements, auditability, and claim discipline. The framework has not been independently validated as a general machine learning method, and its documented audits do not establish predictive superiority or verified real-world AI safety.

I would appreciate technical perspectives on three questions.

1. How can evidence sufficiency be formally defined and measured in an NLP pipeline?

2. What evaluation methods could determine whether fail-closed rules reduce unsupported conclusions without unnecessarily rejecting valid answers?

3. How should a system distinguish missing information, contradictory evidence, and genuine uncertainty?

I am particularly interested in approaches involving information extraction, knowledge representation, retrieval-augmented generation, and uncertainty evaluation.

What existing methods or experimental designs would be most appropriate for investigating these questions?

Did anyone get papers to review for the ARR October cycle?

Update: They assigned 10 Papers, damn.

Context: I come from a linguistics background and have several years of experience in my specialty within the field. I passed the first round of interviews for an EdTech learning company for a high-ish level role. They gave me an "assignment" to see how I would handle what I assume are issues they are facing regarding scoring, mapping of scores, comparisons, etc.

The assignment has multiple parts and I have been explicitly asked for all of my thinking regarding my choices--whys, why nots, deep explanations, etc. Essentially, they want me to show all of my steps.

I'm not even halfway done, and this "assignment" has taken several hours and is multiple pages.

Having not experienced this before, I'm concerned that this ask from the company is more so them fishing for information they can feed into an AI. Are assignments of this scale common within the EdTech and Language Tech arena?

Hello all!

I graduated earlier this year with a MS in Human Language Technology from UA. I mention the school because the program is heavily focused on giving students practical, job-like experience. I completed an internship during which I worked on a live project and had the opportunity to present at the Language Technology for All conference in France last year. In addition, I have professional experience in language technology, primarily in text chatbot work that was more manual (e.g. working with patterns that are manually programmed and not inferred using ML) as well as experience with AI and Data Analysis. I have had an impossible time finding a job that is actually relevant and I would really like some advice.

Currently, I am on contract at Meta as a DLA, though I have been tasked with more complex linguistics work, specifically reporting on results of assessments and creating assessment guidelines for various models, both text and voice.

Is there a place I am not looking? I will share my docs in the comments.