r/Multimodal • • Dec 22 '24

Anyone want to help me teach LLMs to actually see

0 Upvotes

So I've been doing some interrogation of chat GPTs large language model and I've come to the realization that none of these models have been trained on bifocal image data it means that all of their understanding of depth perception is coming from pattern recognition on 2D images.

I think that if we can get large language models to truly develop the kind of emergent heuristics that comes from seeing and relating object movement through 3D space it will help with their reasoning and eventually help them to improve their score on the arc prize.

I wonder if anyone else has been thinking about this, has anyone been doing any work on this, is anyone interested in helping me to develop a test to prove that bifocal training image data helps to improve a model's reasoning capability about the physical world?


r/Multimodal • • Jan 16 '25

Multimodal Models with Google DeepMind Spoiler

Thumbnail
ankitaguha256.medium.com
0 Upvotes

r/Multimodal • • Apr 16 '26

Can frontier AI models actually read a painting?

1 Upvotes

I wrote up a small experiment on whether frontier multimodal models can appraise art from vision alone.

I tested 4 frontier models on 15 paintings worth about $1.46B in total auction value, in two settings:

  1. image only

  2. image + basic metadata

The main thing I found was what I describe as a recognition vs commitment gap.

In several cases, models appeared able to identify the work or artist from pixels alone, but that did not always translate into committing to the valuation from the image alone. Metadata helped some models a lot more than others.

Gemini 3.1 Pro was strongest in both settings. GPT-5.4 improved sharply once metadata was added.

I thought this was interesting because it suggests that for multimodal models, “seeing” something and actually relying on what is seen are not the same thing.

Would be curious what people think about:

  • whether this is a useful framing

  • how to design cleaner tests for visual reliance vs textual reliance

  • whether art appraisal is a reasonable probe for multimodal grounding

Blog post: https://arcaman07.github.io/blog/can-llms-see-art.html


r/Multimodal • • Jul 03 '21

da Vinci invented the iPhone

Post image
3 Upvotes

r/Multimodal • • May 04 '25

OIX Multimodal Hackathon – Build AI Agents That Understand Video (May 17, $900 Prize Pool)

3 Upvotes

We’re hosting a 1-day online hackathon focused on building AI agents that can see, hear, and understand video — combining language, vision, and memory.

🧠 Challenge: Create a Video Understanding Agent using multimodal techniques
💰 Prizes: $900 total
📅 Date: Saturday, May 17
🌐 Location: Online
🔗 Spots are limited – sign up here: https://lu.ma/pp4gvgmi

If you're working on or curious about:

  • Vision-Language Models

  • RAG for video data

  • Long-context memory architectures

  • Multimodal retrieval or summarization

...this is the playground to build something fast and experimental.

Come tinker, compete, or just meet other builders pushing the boundaries of GenAI and multimodal agents.


r/Multimodal • • Jul 30 '21

DALL·E mini is now available

Thumbnail
huggingface.co
9 Upvotes

r/Multimodal • • Apr 17 '21

*Semantic* Video Search with OpenAI’s CLIP Neural Network

Thumbnail self.openai
2 Upvotes

r/Multimodal • • Feb 22 '21

De Deyne et al. (2021) Visual and Affective Multimodal Models of Word Meaning in Language and Mind

Thumbnail
onlinelibrary.wiley.com
4 Upvotes

r/Multimodal • • Aug 12 '21

"a painting by Lisa Frank titled 'New York City loves animals'"

Post image
3 Upvotes

r/Multimodal • • Mar 08 '21

GPT-3 vs. DALL-E Hype Cycle

Thumbnail
bakztfuture.substack.com
2 Upvotes

r/Multimodal • • Apr 29 '21

"4MC-4M-Image-Text-Pairs-with-CLIP-embeddings" (4M YFC100M images with the CLIP caption embeddings, lightly censored), Christoph Schuhmann

Thumbnail
github.com
1 Upvotes

r/Multimodal • • Jun 18 '21

Long-Short Temporal Contrastive Learning of Video Transformers

Thumbnail
arxiv.org
1 Upvotes

r/Multimodal • • Apr 17 '21

"Artistic Performance: a community of artists rebuild the universe" [LatentVisions, +4 drafts in Aleph 5.3]

Thumbnail
gallery
2 Upvotes

r/Multimodal • • Mar 17 '21

Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models

Thumbnail
arxiv.org
5 Upvotes

r/Multimodal • • Sep 02 '21

Composition & Phrasing with DALL-E

Thumbnail
youtube.com
3 Upvotes

r/Multimodal • • Mar 22 '21

Paint by Word

Thumbnail
arxiv.org
3 Upvotes

r/Multimodal • • Aug 11 '21

Oasis - Wonderwall

Post image
1 Upvotes

r/Multimodal • • May 11 '21

AI generated Playing Cards

Post image
2 Upvotes

r/Multimodal • • Mar 02 '21

New text-to-image Google Colab notebook "Aphantasia" from eps696. Details in a comment. Example: text="The Lord of the Rings"; subtract="contains text".

Post image
1 Upvotes

r/Multimodal • • Mar 03 '21

WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

Thumbnail
arxiv.org
4 Upvotes

r/Multimodal • • Jun 29 '21

GitHub Copilot · Your AI pair programmer

Thumbnail
copilot.github.com
5 Upvotes

r/Multimodal • • Mar 09 '21

New Google Colab notebook "Aleph2Image Modified by kingchloexx for Image+Text to Image - Colaboratory" by kingchloexx. This notebook is for editing an existing image using a text description. Example: Text "green fur" with "plus" operation.

Thumbnail
gallery
4 Upvotes

r/Multimodal • • Feb 24 '21

For developers: OpenAI has released the encoder and decoder for the discrete VAE used for DALL-E.

Thumbnail
self.mediasynthesis
4 Upvotes

r/Multimodal • • Apr 29 '21

Zero-Shot Detection via Vision and Language Knowledge Distillation

Thumbnail
arxiv.org
3 Upvotes