ReadBench evaluates bypassing text tokenization by rendering long-context documents as images for vision-language models

hilarious if labs run DeepSeek-OCR style context compression on the back end, of course derived from much older works

@suchenzang The trick is to choose the winning team long before anyone knows it's the winning team and then later keep rubbing it in with "i told you so"s. ezpz

@giffmana it's just not fair that vision got to utility first with orders of magnitude less compute

absolutely rigged

@giffmana I see your CLIPPO and counter with ReadBench

@xeophon Like two decades (in this day and age) later, but very cool yeah. Did you know that one of the authors also works at PI now?

@giffmana If i remember correctly. Deepseek was also experimenting with this on their Deepseek OCR a year or so ago, and people were all excited about the compression possibilities.

@giffmana @suchenzang "I told you so" doesn't bring you friends nor respect.

If you are right too early no one cares.

@giffmana True but anyway to overcome this?

“Disclaimer: the tradeoff is serious: it is lossy, and exact strings like IDs, hashes, secrets, or exact numbers can be silently misread. 🙈”

@giffmana hey you are making progress on your schmidhuber posting quest 😂💪💪💪 baby steps, baby steps

@rihim_s *as far as you know*!

@giffmana i swear i've seen the exact same finding published multiple times and it just doesn't get used anywhere

@SwayStar123 @giffmana it's bullshit imo, If you ablated this with a token compressor you could get better compression if you stay in languge token space

@giffmana You know the favorite thing of a LLM researcher is looking at ancient vision papers (or accidentally re-discovering them)

@7uomoki I wonder though if he takes himself as seriously as I do myself.

@giffmana https://arxiv.org/pdf/2212.08045

@jqlive Ohhh yeah i vaguely remember now

@isharduld Just scale it up

@JFPuget @giffmana @suchenzang yessir 🫡, this is a lesson im learning

@giffmana Didn’t DeepSeek publish a detail paper about that around a year ago? I remember a lot of fuzz about it

Visual tokenization as an inference optimization is well-documented in academic literature, but the real bottleneck has been hardware-side memory bandwidth, not software alone. If frontier labs are just rediscovering this, the market may be overpricing their near-term efficiency gains.