Parsing a .xlsx file is still a nightmare. We have spent the last few years pretending that the “context window” solves everything, but the reality is that most of us are just shoving stripped-down, flattened text into a prompt and hoping the model can reconstruct the original logic. When you strip a complex spreadsheet of its structure, you aren’t giving the model a clean version of the data; you’re giving it a ghost of the data.
The problem is that the industry has leaned far too heavily on Markdown as the universal translator. If it can be turned into a table in Markdown, the LLM can probably handle it. But real-world office documents aren’t clean tables. They are nested XML nightmares wrapped in zip containers (and we’ve all spent too many hours doing exactly that). A .docx file isn’t a document; it’s a directory of XML files pretending to be a page.
This is where the Office Comprehension Bench (OCB) comes in. Instead of relying on the “convert to text and pray” method, OCB looks at how LLM systems handle Word, Excel, and PowerPoint files in their actual native formats. According to the paper https://arxiv.org/abs/2607.01245, the benchmark evaluates the ability to actually comprehend the structure and content of .docx, .xlsx, and .pptx files.
Why are we still pretending that a CSV is the same thing as a formatted Excel workbook? It isn’t. A CSV is a list; a workbook is a relational database with a visual layer. When a model fails to “understand” a spreadsheet, it’s rarely a failure of reasoning. It’s a failure of perception. Trying to parse a complex Excel file using a generic text-splitter is like trying to read a book that’s been put through a shredder and then taped back together in a random order. You might recognize the words, but the narrative is gone.
The OCB introduces two tracks to separate the wheat from the chaff. By testing native formats, it forces developers to stop ignoring the pre-processing pipeline. Most “AI agents” today are just wrappers around a very fragile parser. If the parser trips on a merged cell in Excel or a floating text box in PowerPoint, the model is hallucinating based on broken input before it even starts thinking.
The parser is the problem.
Here is the take: this benchmark isn’t actually testing the “intelligence” of the LLM. It is testing the quality of the middleware. If a model scores poorly on OCB, it doesn’t mean the model is stupid; it means the developer is using a mediocre library to feed it data. We have a tendency to blame the brain when the eyes are the ones failing.
The current approach of “flatten everything to text” is a lazy shortcut that has reached its limit. We’ve seen this before with PDF parsing—everyone claimed it was solved until they hit a multi-column layout with embedded images, and the whole thing fell apart. The same thing is happening with Office files. We are treating highly structured binary-adjacent formats as if they were simple strings.
If we want agents that can actually operate in a corporate environment, we need to stop treating the file format as a hurdle to be cleared and start treating it as a primary data source. This means moving away from the “convert to Markdown” obsession and toward a system where the model can query the XML structure directly or use a specialized representation that preserves spatial and relational logic.
I suspect the results of OCB will initially look depressing for the major labs. But that’s a good thing. It highlights the massive gap between “chatting with a document” and actual comprehension.
By Q4, we will see a specific, high-performance preprocessing library emerge—likely from a smaller, focused team—that increases OCB scores across the board by at least 15% without requiring a single change to the underlying model weights. Once that happens, the conversation will shift from “can the model read this” to “how fast can the parser run.” For now, we’re just staring at the mess.
