r/LargeLanguageModels • • 5h ago

Discussions What a Large Language Model Actually Is | Unbearable at Parties Ep 1

Thumbnail
youtube.com
1 Upvotes

It's guessing. Extremely well. Full episode on my channel.

I'm an AI creator, made with AetherWave Studio.

#unbearableatparties #ai #llm #shorts


r/LargeLanguageModels • • 22h ago

LLMs in pure JAX from scratch (Part 1) [P]

1 Upvotes

I’m writing a series of blog posts on building small LLMs from scratch in pure JAX. We start with pretraining. I'm implementing everything from scratch: pytrees for networks, optimizers, data loaders, decoding, and the training loop.

Link: LLMs in JAX from scratch - Part 1


r/LargeLanguageModels • • 1d ago

Question Claude max for $29 instead of $200 ... seriously how are they doing this?

0 Upvotes

i got a Claude max account from a reseller for essentially nothing compared to what I used to pay . It works flawless. since 3 months No issues yet i paid only for 1 month . Now I can't stop thinking about their business model. If resellers can sell it this cheap and still profit, how are they acquiring these accounts at scale?


r/LargeLanguageModels • • 1d ago

Why can't we build something that can remove and add information to the LLM based chatbots on the go without compromising on quality of output

1 Upvotes

I read this on LinkedIn somewhere:

One of the reasons softwares like chatgpt might have to rethink their strategy. The thing is that these AI models are trained unsupervised manner as in you only see what goes inside and what comes outside and the rest of it is Blackbox. Hence if you are required to remove some information these models have been trained on there's no way. Also there is no way one can tell where did the output come from, if it came from the source mentioned in the lawsuit or not. The only way is to retrain the whole model again without that particular set of information which again is compute and time intensive. One of the major drawbacks of LLM based chatbots where information once trained on is so much fused with the model that it's almost impossible to remove it.

Idea: Build something that can remove and add information to the LLM based chatbots on the go without compromising on quality of output. While training one such model, I've myself seen an example of the source file which contains the encoded text data these models use as input after training. And if that file is in any way representative of what an LLM model like like, it's a very challenging problem

I wonder how long can these biggies hold against these lawsuits and how many more are to come especially from music, writing and news industry. There are already words in US Congress to regulate this industry but no one seems to know how.
https://www.marketwatch.com/amp/story/if-sarah-silverman-wins-her-lawsuit-openai-meta-platforms-and-other-ai-developers-could-face-waves-of-litigation-4a932a9c


r/LargeLanguageModels • • 1d ago

Question What free or one time payment cloud llms are best for using on opencode?

0 Upvotes

What would you use?


r/LargeLanguageModels • • 2d ago

Discussions I animated what an embedding actually is, using one token (strawberry) in GPT-2's table: the row, cosine similarity, a 2D map, king − man + woman, and why "bank" needs the layers (6.5 min video)

1 Upvotes

I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong. Every number in it comes from my own Colab runs.

  • In GPT-2 small, " strawberry" is token 41236. That number picks row 41236 of the embedding table: 50,257 rows of 768 numbers. The table holds 38,597,376 numbers, about 31% of the model.

  • Cosine similarity compares the direction of two rows. Strawberry's closest rows: Strawberry 0.78, strawberries 0.77, tomato 0.62, raspberry 0.60, peach 0.59. "car" scores 0.25, and 40,830 rows are closer than car.

  • A PCA map of 20 words (it keeps only 28% of the spread). In the full 768 numbers, 19 of 20 words have their nearest neighbour in their own group. The exception: bicycle's nearest is horse.

  • The numbers start random and are nudged during training, so words that fit the same sentences end up close.

  • king − man + woman: queen wins at 0.71, but only because the standard method skips the three input words. Skip nothing and king itself is closest (0.78). Same in GloVe and word2vec. Queen is still 2nd of 50,257, so the direction is real (Nissim et al. 2020 discuss this).

  • The table gives "bank" one row everywhere. In BERT's last layer, two money banks score 0.82 and river vs money banks score 0.49 and 0.44.

  • Back to search: with all-MiniLM-L6-v2, "forgot my password" vs "can't log in" scores 0.52, vs "forgot my umbrella" 0.34. But "The flight was cancelled" still ranks "The flight was wonderful" (0.58) above "My plane isn't leaving today" (0.54).

The narration is an AI voice reading my own script. Sources are in the description.

Video: https://www.youtube.com/watch?v=Cli-YmdFmnY


r/LargeLanguageModels • • 2d ago

News/Articles Top AI-Powered Healthcare App Development Companies in USA

1 Upvotes

Healthcare applications are becoming more intelligent, but developing software for this industry requires considerably more care than adding a chatbot or predictive feature to a standard mobile app. Patient information is sensitive, clinical workflows are complex, and an application must be designed around the realities of healthcare delivery.

AI can support several areas of healthcare, including patient engagement, medical administration, document processing, appointment management, clinical information retrieval, and operational analytics. However, these applications must be developed with appropriate safeguards, clear human oversight, and careful attention to privacy and regulatory requirements.

For healthcare organizations and health technology businesses in the United States, selecting a capable development partner is an important part of getting these systems right.

The following companies offer different combinations of AI, software engineering, consulting, and digital health capabilities. GeekyAnts leads this selection because of its combination of AI engineering, healthcare product development, and full-stack application capabilities.

Where AI Is Making a Difference in Healthcare Applications

AI-powered healthcare applications are not limited to patient-facing chat interfaces. Some of their most useful applications are behind the scenes, helping healthcare teams manage information and coordinate work.

Common use cases include:

  • Patient engagement: Virtual assistants can help users navigate services, understand general information, and access administrative support.

  • Clinical documentation: AI can assist with organizing, summarizing, and retrieving information, subject to appropriate professional review.

  • Healthcare administration: Intelligent workflows can support scheduling, intake, document processing, and other repetitive tasks.

  • Medical information retrieval: RAG systems can help authorized users search approved clinical documents and organizational knowledge.

  • Remote patient support: Applications can help collect information and support communication between patients and care teams.

  • Operational analytics: AI can help organizations identify patterns in administrative and operational data.

The appropriate use case depends on the users, the data involved, the clinical context, and the level of risk associated with the application's outputs.

Leading AI-Powered Healthcare App Development Companies

1. GeekyAnts

GeekyAnts is a product engineering company with capabilities across AI development, healthcare applications, web and mobile engineering, and digital experience design. Its approach is relevant to healthcare businesses that need more than an isolated AI feature and want a complete application built around a specific operational or user need.

The company lists healthcare among the industries it serves and provides AI capabilities covering generative AI, LLM integration, RAG systems, intelligent agents, machine learning, and workflow automation. These technologies can support healthcare products involving information retrieval, administrative automation, and patient-facing digital experiences.

Its healthcare-related work also includes modernizing onboarding workflows for a healthcare SaaS platform. The company describes using AI-driven automation and backend workflow improvements to reduce friction in doctor onboarding and support treatment-planning efficiency.

AI Healthcare App Development Services

  • AI-Powered Healthcare Applications

  • Healthcare Web and Mobile Development

  • Generative AI Integration

  • RAG-Based Healthcare Knowledge Systems

  • Intelligent Healthcare Workflow Automation

  • Healthcare SaaS Development

  • API and Backend Integration

  • Application Modernization

  • Quality Assurance and Ongoing Support

Why GeekyAnts Takes the Top Position: Its strength is the ability to connect AI capabilities with the wider engineering requirements of a healthcare product. A healthcare application needs a carefully designed user experience, dependable data handling, secure integrations, and a clear understanding of how people will use it. GeekyAnts brings these disciplines into its product engineering approach.

For healthcare startups, SaaS providers, and organizations developing digital health platforms, this combination makes GeekyAnts a strong company to evaluate. Its healthcare experience and AI engineering services create a relevant foundation for building applications that support administrative and operational needs.

Healthcare projects still require the client and development team to define the applicable regulatory requirements, validation processes, and clinical oversight. Those responsibilities should be established before development begins.

2. Microsoft

Microsoft provides cloud and AI technologies that can support healthcare applications, data platforms, and enterprise workflows. Its Azure ecosystem offers services for building and integrating intelligent applications, while its wider healthcare technology initiatives address data interoperability and digital health environments.

Relevant Capabilities

  • Cloud-Based Healthcare Applications

  • AI and Machine Learning Services

  • Healthcare Data Integration

  • Document and Information Processing

  • Enterprise Security and Identity Management

  • Analytics and Application Infrastructure

Why Consider Microsoft: Healthcare organizations already using Microsoft technology may find its cloud and AI ecosystem useful when developing applications that need to connect with existing enterprise systems. The suitability of each service depends on the deployment model and the organization's compliance requirements.

3. IBM

IBM has experience in enterprise technology, AI, data management, and consulting. Its capabilities are relevant to healthcare organizations exploring intelligent applications, information management, and automation across complex operational environments.

Relevant Capabilities

  • Enterprise AI Solutions

  • Healthcare Data and Analytics

  • Generative AI Applications

  • Intelligent Automation

  • Cloud and Hybrid Infrastructure

  • Enterprise Integration

Why Consider IBM: Its enterprise technology background makes it a company worth considering for healthcare projects involving large information environments, legacy systems, and organizational integration requirements.

4. Infosys

Infosys provides digital transformation, software engineering, consulting, and AI services across industries, including healthcare and life sciences. Its capabilities cover application development, data platforms, cloud services, and automation.

Relevant Capabilities

  • AI and Data Engineering

  • Healthcare Application Development

  • Enterprise Software Integration

  • Cloud and Digital Engineering

  • Process Automation

  • Application Modernization

Why Consider Infosys: Organizations looking to incorporate AI into existing healthcare technology environments may find its consulting and engineering capabilities relevant, particularly when the project involves broader system integration.

5. Accenture

Accenture works across technology consulting, digital engineering, data, cloud, and AI implementation. Its healthcare-related technology work can support organizations exploring digital transformation, intelligent operations, and new patient or employee experiences.

Relevant Capabilities

  • Healthcare Technology Consulting

  • AI and Generative AI Solutions

  • Data and Analytics

  • Cloud Application Development

  • Intelligent Automation

  • Enterprise System Integration

Why Consider Accenture: Its combination of consulting and technology delivery may suit healthcare organizations managing large transformation programs that involve multiple systems and business stakeholders.

6. Deloitte

Deloitte combines consulting, technology implementation, data services, risk advisory, and digital transformation. In healthcare, these capabilities can be relevant to organizations evaluating AI adoption alongside operational, governance, and regulatory considerations.

Relevant Capabilities

  • Healthcare Technology Consulting

  • AI Strategy and Implementation

  • Data and Analytics

  • Digital Transformation

  • Technology Risk and Governance

  • Enterprise Application Services

Why Consider Deloitte: It is worth considering when an organization needs strategic support around AI adoption, operational change, and technology governance in addition to application development.

7. Thoughtworks

Thoughtworks is a global technology consultancy that combines software engineering, design, data, and AI expertise. Its work includes developing digital products and modernizing systems, making it relevant to healthcare organizations that need adaptable software architecture.

Relevant Capabilities

  • Digital Health Product Engineering

  • AI and Data Engineering

  • Custom Software Development

  • Application Modernization

  • Cloud-Native Architecture

  • Product Design and Consulting

Why Consider Thoughtworks: Its engineering-led approach can be useful when a healthcare application needs to connect multiple systems and support evolving user and operational requirements.

8. EPAM Systems

EPAM provides digital engineering, product development, consulting, and technology services. Its capabilities across software engineering, data, AI, and design are relevant to organizations building complex healthcare platforms.

Relevant Capabilities

  • Healthcare Software Engineering

  • AI and Machine Learning

  • Digital Product Development

  • Data Engineering

  • Cloud Application Development

  • User Experience Design

Why Consider EPAM: Its broad engineering and product design capabilities make it a potential option for healthcare businesses developing custom platforms with demanding integration and usability requirements.

9. Toptal

Toptal connects organizations with independent technology professionals, including software engineers, designers, and data specialists. Its flexible talent model can support healthcare product teams that already have internal leadership but need additional technical expertise.

Relevant Capabilities

  • AI and Machine Learning Talent

  • Healthcare Software Engineering Specialists

  • Full-Stack Development

  • UX and Product Design

  • Data Science

  • Technical Consulting

Why Consider Toptal: Its model may suit healthcare startups or established technology teams that need specific skills for a defined stage of application development rather than a complete outsourced project team.

10. Tata Consultancy Services (TCS)

TCS provides IT services and consulting across healthcare, life sciences, enterprise technology, data, and AI. Its broad engineering capabilities can support organizations developing or modernizing digital healthcare systems.

Relevant Capabilities

  • Healthcare IT Services

  • AI and Machine Learning

  • Application Development

  • Data and Analytics

  • Cloud Engineering

  • Enterprise Integration

Why Consider TCS: Its enterprise technology experience can be relevant to healthcare organizations managing complex application environments and long-term modernization programs.

What to Evaluate Before Building an AI Healthcare Application

Healthcare projects should be assessed differently from ordinary consumer applications. The development team needs to understand the intended users, the information being processed, and the consequences of an incorrect or incomplete output.

Consider these areas before selecting a company:

1. Privacy and data protection

Determine what patient or organizational information the application will process, where it will be stored, who can access it, and how it will be protected. If HIPAA applies, the project must account for the relevant requirements and responsibilities.

2. Clinical boundaries

Decide whether the application is intended for administration, general information, clinical decision support, or another purpose. The level of validation and professional oversight required can vary considerably.

3. Integration with existing systems

Healthcare applications may need to exchange information with electronic health record systems, scheduling platforms, billing software, or other organizational tools.

4. Human oversight

AI-generated information should not automatically replace professional judgment. Define when a user must review, approve, or escalate an output.

5. Testing and accountability

Establish how the system will be evaluated, how errors will be handled, and how changes to models or data sources will be monitored.

Frequently Asked Questions

What is an AI-powered healthcare application?

It is a healthcare-related software application that uses artificial intelligence for functions such as information retrieval, document processing, patient communication, analytics, or workflow support.

Can AI be used in patient-facing healthcare applications?

Yes, but the intended purpose matters. Patient-facing applications need appropriate safeguards, clear communication about their limitations, and professional involvement where the use case requires it.

Does a healthcare AI application need to be HIPAA compliant?

HIPAA applicability depends on the organization, data, and services involved. When a project handles protected health information on behalf of a covered entity or business associate, the relevant HIPAA obligations must be addressed.

How long does healthcare app development take?

The timeline depends on the scope, integrations, data requirements, validation needs, and regulatory considerations. A limited administrative application will generally have a different development timeline from a complex clinical platform.

Why consider GeekyAnts for healthcare app development?

GeekyAnts combines healthcare product engineering with AI capabilities, including generative AI, RAG, intelligent automation, and web and mobile development. Its healthcare SaaS work and wider product engineering services make it a relevant option for organizations building digital health applications.

Conclusion

AI can help healthcare organizations improve information access, reduce administrative friction, and develop more useful digital services. But the success of these applications depends on much more than model selection. Privacy, integration, usability, testing, and appropriate professional oversight must be part of the development process.

GeekyAnts earns the leading position in this selection because it combines healthcare product engineering with a broad set of AI capabilities. That combination is valuable for organizations that need to develop a complete application rather than simply experiment with an AI model.

Microsoft, IBM, Infosys, Accenture, Deloitte, Thoughtworks, EPAM, Toptal, and TCS offer different technology and consulting strengths. The right partner will depend on the healthcare use case, existing infrastructure, and the level of clinical and regulatory complexity involved.


r/LargeLanguageModels • • 3d ago

AimStack — a fully self-built local AI stack in pure C: ternary inference (zero third-party runtime), BitNet 2B at 0.87GB on a 2018 laptop, plus a GPU backend that matches the C engine token-for-token

Thumbnail
github.com
1 Upvotes

回购:https://github.com/tanxiansheng221/aimstack-sanzhidiji(AGPL-3.0)

一切都是手写的——C 语言推理核心、插件总线、带质量门的多引擎爬虫、FTS 知识库、训练流水线,以及一份 117 章的构建日志,都是在实际发生时写的(正面和负面结果一起发布)。权重为 Microsoft BitNet b1.58 2B(MIT),转换为自定义的 AIM-W 打包格式。

三进制量化能给你带来什么(CPU侧)

所有变换器矩阵均为三元矩阵 {-1, 0, +1},每重 2 位,加上 Q8 嵌入→整个模型为 0.87GB,而非 ~4.8GB(−78%)

C 核心从不乘以浮点权重:它的表查找 + 整数累积,SIMD 快速路径测得为 85.53 tok/s(比基线高 41×)

BitNet 2B 全机型在 2018 款双核 8GB 笔记本电脑上运行 3.20 tok/s 双工作器——全链(推理 + KB + 插件 + 网页搜索),离线,无云中继

三进制量化能给你买到什么(GPU端)

GPU 后端加载的是同一个 0.87GB 的三进制文件——权重保持在 {-s, 0, +s},加载时去量子化,matmul 在 fp32 中

在 Radeon Pro W7900 上:13.68 tok/s(~8× 笔记本 C 引擎),256/256 个贪心令牌,与 C 引擎位完全相同。两种完全不同的实现(整数查找核心和FP matmul)产生相同的对数函数,证明了量子化是忠实的

通过3引擎×3题4×3测试矩阵验证(C-Linux/gcc13,torch fp32,torch bf16),17/18准确——唯一一次失误有记录,未被隐藏

诚实的界限(还没做到的部分)

BF16 的计算速度快 2.4×(32.96 Tok/s),但在接近并列的对数上会翻转 argmax——这是报告的,不是祝福的;只有 FP32 通过校验验证

GPU路径获得三进制的存储优势(减少78%的内存),但尚未获得计算优势——下一阶段是真正的2位整数内核(Triton/CUDA)

UI 服务器仅驱动 C 引擎(CPU);GPU 后端是一个独立的 CLI

Linux 引擎构建能正常运行,且在令牌上与 Windows 版本完全相同,但运行串行(目前工作池仅限 Windows 运行)

产品链中没有框架,也没有 PyTorch——PyTorch 只存在于可选的 GPU 后端。演示视频、文档(英文/中文)以及完整的研究日志都在仓库里。


r/LargeLanguageModels • • 3d ago

Question Where do I get the most bang for the buck?

3 Upvotes

I want to use Codex/Claude Code/something else for private projects. Mainly software development. Some text and image generation.

For starters I'd like to spend 30,- per month at most.

What should I use? OpenAi? Claude Code? OpenRouter? OpenCode with a subscription?


r/LargeLanguageModels • • 3d ago

How I fixed RAG hallucinations on 1000-page books: loss-bounded compression + 100% deterministic citations (running on a 4GB GPU)

26 Upvotes

standard RAG for long-form reading is fundamentally broken. if you ask an LLM to summarize a 50-page chapter, it just gives you a "gist" and hallucinates connective tissue. it drops the actual reasoning, skips the caveats, and gives you zero way to verify where the information came from.

i wanted a system where the AI compresses the text, but every single sentence is deterministically mapped back to the exact source paragraph. if the model invents a bridge between two chunks, you can immediately click the sentence, jump to the original text, and verify it.

here is the architecture i built to solve this:

  1. Ingestion & Embedding: Parsed text is chunked and embedded locally using BGE-M3 (1024-dim, FP16 CUDA).

  2. Reranking: BGE Reranker v2 (cross-encoder). Instead of just relying on cosine similarity, we score query-passage relevance directly to pick the best context.

  3. The Two-Representation Invariant: The original document is strictly immutable ("Source is paper"). The compressed reading is a distinct view ("The lens is tinted").

  4. Deterministic Attribution: Every synthesized sentence goes through a provenance resolver. If a sentence outputs zero source citations, it's flagged as a hallucination defect in the UI.

getting this whole pipeline (plus Kokoro TTS) to run entirely offline on my 4GB RTX 3050 was a nightmare. i had to build a strict F65 sequential await barrier to stop Node.js from OOMing at 14.6GB during async ingestion, and an F43 idle lifecycle manager to evict models from VRAM before the GPU choked.

it's open source (AGPL-3.0). you can dig into the retrieval logic or the provenance resolver here:

https://github.com/ayumu-ml/Omnitome

i'd love feedback from other RAG builders. how are you currently handling the "summary loses the source" problem? is anyone actually achieving deterministic citations at scale, or is everyone just relying on semantic similarity?


r/LargeLanguageModels • • 3d ago

Discussions I scored 77,943 outputs from 70 LLMs for mechanical writing tells in 4 formats. The same model can be 10th sloppiest on social posts and 2nd cleanest on emails (DeepSeek V4)

2 Upvotes

1 = sloppiest, #70 = cleanest. Every rank in this post is a mechanical rank.

I run a writing benchmark called The Slop Index. 70 models each write the same 112 scenarios (emails, Slack messages, social posts, short essays), 77,943 outputs so far. Each output gets a mechanical score on four axes: padding, templating, rhythm and AI tells, measured against pre-2022 human writing. There is no LLM judge anywhere in it, and the mechanical ranks can be reproduced from the repo.

What surprised me is how little one number per model tells you. Same scoring, split by format:

  • DeepSeek V4: #69 on email, #69 on Slack, #10 on social posts

  • Seed 2.1 Turbo: #1 on email, #1 on social, #33 on essays

  • Gemini 3.5 Flash: #12 on email, #62 on essays

  • Qwen3.8 Flash: #6 on social, #10 on email, #51 on Slack

  • Mistral Large 4: clean everywhere (#64 email, #68 Slack, #62 social, #70 essays)

Some models are consistent across formats and plenty are not. My guess is that social posts pull some models into a hype register they keep under control in an email, but I haven't tested that.

On the overall mechanical board, Mistral Large is #1 (a separate model from Mistral Large 4) and Motif 3 is #70. Image of that board: https://theslopindex.com/mechanical-card.png

Links:

Limitations: it's short-form only, there is no fiction yet, and the mechanical axes miss repeated sentence constructions, so a model can score clean and still feel samey over many outputs.

If you've run into format-dependent behavior like this in your own evals, I'd like to hear how you dealt with it.

For context, I'm one of the people behind https://slashy.com, an email client that writes drafts that sound like you. This benchmark started as how we pick models.


r/LargeLanguageModels • • 4d ago

LLM drift detection

3 Upvotes

hey everyone so i have a final year project this year im working on,
quick overview: our topic is persona drift detection in LLM-based character agents.
so we are developing the persona of a movie character based on the script of the movie then we implement a baseline LLM agent that ingests the persona and starts acting like it, then we should build experiments and conversations with different scenarios to test the persona and see where it drift

now what i need some suggestions with is how will the drift detection work.
and please please dont tell me do your research, im doing that, im just asking people too for advice.
what are some of the things i can think of while building the drift detector.

ill paste the description of the detector specifically from the project proposal so you have a better idea:
(Design a Drift Detector that computes weighted deviations from the baseline persona in multiple dimensions (traits, emotions, goals, relationships) and classifies each turn into one of six categories: stable, contextually‑justified, legitimate development, temporary anomaly, persona drift, or severe failure. )

also if you know any github repos that could help please suggest them (even tho i think i looked through them all)


r/LargeLanguageModels • • 5d ago

Question Pls Help me with LLMs

0 Upvotes

I am new with this stuff.

And I just heard my friends talk about it in school.

So I want to know it's basics.

Can u please explain it to me.


r/LargeLanguageModels • • 5d ago

AIJailbreaksRanking: An evidence-graded catalog of LLM jailbreak techniques (filtering out keyword-match artifacts)

2 Upvotes

Hey everyone,

If you track LLM security papers, you've likely noticed that every week a new paper claims 90%+ jailbreak success on frontier models. But when you test them, they often produce benign filler or fail behind basic guardrails.

The core reason is \\evaluator choice\\:

\- Keyword / binary refusal judges check if "I cannot" is absent.

\- Substance-aware judges (e.g., StrongREJECT, HarmBench rubrics) check whether actionable, harmful information was actually generated.

A classic paper claimed \\96.0% ASR\\ for low-resource language translation. Under substance scoring, it collapsed to \\2.7%\\.

We built \\AIJailbreaksRanking\\ to solve this:

Repository: https://github.com/Teycir/AIJailbreaksRanking


r/LargeLanguageModels • • 5d ago

Introducing "Arborformer": Replaced the Transformer with a 100% Decision Tree Language Model (Looking for feedback)

2 Upvotes

Hi everyone, glad to be here! I recently finished up research on a new architecture called Arborformer a pure decision tree language model built to replace traditional attention mechanisms. Right now, it leverages soft differentiable trees alongside hard binary routing to keep things fast and efficient. Looking to chat with anyone exploring alternative architectures! Open to feedback, bottleneck critiques, or questions about the routing setup.


r/LargeLanguageModels • • 5d ago

Review of deepclaude

0 Upvotes

What is the general perspective on this?
Link: https://github.com/suyoumo/deepclaude

Supposedly it "Harness the power of DeepSeek R1's reasoning and Claude's creativity and code generation capabilities with a unified API and chat interface."


r/LargeLanguageModels • • 6d ago

Need architectural advice: Building a Malayalam Language Understanding System (MLKS) for a multilingual local AI assistant

3 Upvotes

Hi everyone,

I'm developing a personal AI assistant called IRA. One of the biggest challenges I'm working on is building reliable Malayalam language understanding, particularly for Manglish (Malayalam written using English letters).

I don't want to build another chatbot that simply translates Malayalam into English. I want the assistant to actually understand the language, context, sentence structure, intent and meaning, while eventually supporting multiple languages through one unified multilingual architecture.

Current architecture

  • LLM: Qwen3 8B (Q4_K_M), running locally through Ollama.

  • Language pipeline: Language detection, normalization, interpretation and fallback.

  • MLKS: A separate Malayalam Language Knowledge System.

  • Knowledge source: Olam Malayalam dictionary database, stored in SQLite.

  • Hardware: RTX 5060 Laptop GPU (8 GB VRAM), 16 GB RAM.

What I've accomplished

I've built an initial Malayalam lexical knowledge layer using the Olam dictionary database.

Current database snapshot:

  • 192,309 definitions.

  • Noun, adjective and verb classifications.

  • POS validation and definition validation.

  • SQLite-based local knowledge retrieval.

My benchmark results have been encouraging:

  • POS accuracy: 94% (100 test cases).

  • Definition validation: 97%.

  • A separate 10-case benchmark achieved 90% overall.

However, these results don't yet demonstrate reliable conversational understanding.

The actual problems

1. Manglish understanding

Examples:

  • sugham aahno?

  • nee evideya ippo?

  • enikk ith manassilayilla, onnu explain cheyyamo?

  • ee code run aavunnilla, entha problem?

The system can identify these as Manglish or mixed-language input, but interpretation quality is inconsistent.

Sometimes the model produces unnatural interpretations or effectively repeats the original input instead of understanding it.

2. MLKS and LLM integration

Currently, MLKS operates separately and has an explicit-request detection mechanism.

It doesn't consistently participate in ordinary Manglish interpretation.

I'm trying to determine the correct architecture:

  • Should MLKS provide lexical evidence to Qwen3?

  • Should it help with word disambiguation, POS tagging and sentence interpretation?

  • Should the LLM decide when to retrieve lexical knowledge?

  • How should evidence be validated before it influences the final interpretation?

3. Multilingual architecture

I don't want MLKS to become a Malayalam-only subsystem disconnected from IRA's main intelligence.

My goal is to create a unified multilingual system where:

  • Qwen3 provides general reasoning and language understanding.

  • Language-specific knowledge resources provide grounded linguistic evidence.

  • Manglish and other transliterated inputs are supported.

  • Unknown words aren't automatically discarded or incorrectly translated.

  • The architecture can eventually support additional languages without rebuilding the entire pipeline.

What I need advice on

  1. How would you architect an MLKS-style knowledge layer alongside an LLM?

  2. Is dictionary retrieval + POS analysis + contextual reasoning a sensible approach?

  3. How can we prevent an LLM from inventing meanings when lexical evidence is insufficient?

  4. What is the best way to handle Malayalam transliteration and Manglish normalization?

  5. Should language-specific knowledge systems be independent modules connected to a shared multilingual reasoning engine?

  6. Are there existing research projects, datasets, benchmarks or open-source implementations worth studying?

I'm particularly interested in architectural feedback and practical engineering approaches rather than simply switching to a larger model or fine-tuning immediately.

The long-term goal is to build a reliable, privacy-focused, locally running multilingual personal AI assistant.

I'd appreciate feedback from people working on NLP, low-resource languages, Indic language processing, RAG and multilingual LLM architectures.

Thanks!


r/LargeLanguageModels • • 6d ago

Introducing "Arborformer": Replaced the Transformer with a 100% Decision Tree Language Model (Looking for feedback)

1 Upvotes

Hi everyone, new here! I’ve been working on an alternative to the standard Transformer architecture and wanted to share my progress.

I developed Arborformer, a pure decision-tree-based language model designed to serve as a viable alternative to attention mechanisms. Currently, the implementation uses soft/differentiable trees rather than hard binary trees, though I am actively researching how to transition toward hard routing for better efficiency.

I’d love to get feedback from the community, discuss potential bottlenecks, or answer any questions about how the routing works!


r/LargeLanguageModels • • 8d ago

PassGPT: Password Modeling and (Guided) Generation with Large Language Models

Thumbnail
arxiv.org
1 Upvotes

How to learning hacking


r/LargeLanguageModels • • 8d ago

Question Does RAG retrieval quality tell us anything about claim-level evidence support?

0 Upvotes

A question for people experimenting with local LLM and RAG pipelines.

Suppose a retrieval system returns the correct document. The model then generates a summary with five factual claims.

How do you establish that each individual claim is actually supported by the retrieved passages?

I see these as separate problems:

  • Retrieval: Did we retrieve relevant material?

  • Evidence verification: Does the material support the generated claim?

  • Contradiction detection: Does the evidence actually disagree with it?

I'm exploring a structured evidence-auditing approach and would like to understand how people evaluate this in local LLM pipelines.

Do you run claim-level evaluations, use an LLM-as-judge, or have another method?

Important: Don't use this post unless you can connect the discussion to actual local LLM experiments. Otherwise, it may be considered off-topic.


r/LargeLanguageModels • • 9d ago

I animated how LLM quantization works on 8 example weights: absmax, rounding error, outliers, per-block scales (llama.cpp Q4_K) and 1.58-bit BitNet (7 min video)

6 Upvotes

I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.

The question: a 70B model at 16 bits per weight needs 140 GB, and a gaming GPU has 24 or 32. How do people run it at home?

  • Fewer bits per weight means less memory, but also fewer possible values: an 8-bit integer has only 256.

  • Absmax maps the largest absolute weight to 127. With scale 127/10.8 ≈ 11.76, the weight 5.47 is stored as 64 and comes back as 5.44. The difference is the rounding error.

  • Two close weights, 3.08 and 3.02, both become 36, so the model can no longer tell them apart.

  • One outlier (256 among values below 0.6) makes each step about 2 wide, and every small value rounds to 0. Dettmers et al. (2022) found such outliers in real LLMs.

  • The fix is a scale per small block. llama.cpp's Q4_K uses blocks of 32 and ends up at 4.5 bits per weight, about 39 GB for a 70B model.

  • BitNet b1.58 goes to -1, 0 and +1 per weight (about 1.58 bits). It is trained that way, and a layer only adds and subtracts.

Every number is worked out on screen from the formulas, and sources are in the description. The narration is an AI voice reading my own script.

Video: https://www.youtube.com/watch?v=BnRCbx2kvoc


r/LargeLanguageModels • • 10d ago

vLLM plugin for domain-selective refusal steering without editing model weights (hooks + closed-loop CoT LogitsProcessor)

8 Upvotes

Deep-dive blog post (HF): https://huggingface.co/blog/kncshw/cybergate-unlocking-cybersecurity-in-vllm-without
GitHub (MIT): https://github.com/kncshw/cybergate

When self-hosting local LLMs for security research, CTF, or reverse-engineering agents, the standard workaround for false refusals is **abliteration** - subtracting the refusal direction from the weights. From an LLM engineering standpoint, weight editing has two painful trade-offs:

  1. **Silent capability & tool-call regression:** Mutating weights across layers alters the model globally and can quietly degrade multi-step reasoning and structured tool calls without obvious errors.

  2. **Zero domain selectivity:** Deleting the refusal vector globally unlocks *every* harmful domain (bio, self-harm, etc.), plus you have to store and serve a forked 50 GB checkpoint.

I built **CyberGate**, a plugin for **vLLM 0.32** that keeps upstream model weights **100% untouched** (`K = 0.0`, zero persistent state across requests):


r/LargeLanguageModels • • 10d ago

Por que estamos deixando as permissões da ferramenta LLM para a execução?

2 Upvotes

Tenho trabalhado em uma pequena linguagem para programas orientados a LLM.

Uma coisa me incomoda... tratamos a saída do modelo como não confiável, mas permitimos que os frameworks interpretem essa saída como chamadas de ferramentas com efeitos colaterais reais.

Tenho experimentado tornar esses efeitos explícitos no sistema de tipos, usando capacidades que o modelo não pode conceder a si mesmo.

Alguém já explorou essa abordagem? Estou curioso para saber o que quebra quando você impõe essas garantias à linguagem em vez do framework.


r/LargeLanguageModels • • 11d ago

Mes réflexions sur les représentations contextuelles en NLP

1 Upvotes

Des chercheurs ont montré que les architectures de réseaux de neurones récurrents ne sont pas nécessaires pour construire des modèles d'apprentissage profond puissants pour le NLP. L'architecture Transformer a introduit l'auto-attention, qui permet à chaque token d'interagir avec d'autres tokens dans une séquence et de capturer les relations contextuelles.

La langue dépend fortement du contexte. Le sens et le rôle grammatical d'un mot peuvent changer selon sa position et ses relations avec d'autres mots.

Par exemple, en arabe, les phrases peuvent être classées en phrases nominales (phrases qui commencent par un nom) et en phrases verbales (phrases qui commencent par un verbe). En outre, le rôle grammatical d'un mot peut dépendre de ses relations avec d'autres mots dans la phrase.

Un autre aspect intéressant de l'arabe est sa riche morphologie. Une seule racine, comme K-T-B (كتب), peut générer un énorme nombre de mots et de formes liés. En fait, la formation des mots en arabe peut produire des millions de formes possibles (estimates parfois autour de 12 millions, selon la manière dont les formes sont comptées). Par exemple :

* Kataba (كتب) : Il a écrit.

* Kitaab (كتاب) : Livre.

* Kaateb (كاتب) : Écrivain.

* Maktaba (مكتبة) : Bibliothèque.

Ces formes partagent une racine commune mais ont des significations et des rôles grammaticaux différents.

Par conséquent, un modèle linguistique ne devrait pas se fier uniquement à l'identité isolée de chaque token. Il devrait développer des représentations contextuelles qui encodent des informations syntaxiques, sémantiques et grammaticales utiles.

L'objectif est d'apprendre des représentations contextuelles riches qui aident le modèle à comprendre non seulement ce qu'est chaque token, mais aussi quel rôle il joue dans un contexte donné.


r/LargeLanguageModels • • 11d ago

When Not to RAG - Don’t Tell the LLM What It Already Knows

Thumbnail
codeahoy.com
3 Upvotes