Checking the numbers an assistant reads out of a long PDF

The first evening I took Claude seriously for my profession, one of the things I gave it was a long technical document of my own. I uploaded the workbook of results that goes with it first, the spreadsheet where the runs are reduced to conditions and results, and then the document itself as a PDF. It saw more in the document than in the workbook. I think that is expected, because the workbook only holds the results, and the methods and the characterization are written in the document.

Then it gave me a package of documents built on it, and the figure it made much of was one measured value. I wrote that document, and I still have not opened it and checked that number against the page.

In the same month I asked what tokens are, because I did not know, and got the explanation of tokens against words and what context length means. The way I keep it in my head as a chemical engineer: tokens are the pieces of words the model works in, and the context window is a vessel of fixed volume. The whole PDF goes into that vessel together with everything else in the conversation. Anthropic’s documentation says each page of a PDF is processed as text and as an image, that a dense PDF with small fonts, complex tables or heavy graphics can fill the context window before the page limit is reached, and it suggests splitting a large document into sections.11Anthropic, “PDF support”. A long report full of tables is that kind of document. There is also a study by Liu and others where models had to find one relevant item in a long input, and they did best when the item was at the beginning or the end and worse when it was in the middle.22Liu et al. (2023), abstract. The results chapters of a long report are in the middle. So I cannot tell from the reply whether a number was read from the right row of the right table.

About three months later I went back to the same material with Claude, and this time I caught a few things. A block of the data was missing from the file it had consolidated, so I uploaded the source file for that block again. Then one component that matters in that work was missing from the charts it drew. For one table I gave it screenshots of the pages instead of the file.

A few days before that, in another project, I had asked for a literature review and for one paper to reproduce. The link in its citation pointed to the wrong paper, and I had already downloaded it on its word. I told it that it needs to learn to cite its claims correctly.

Anthropic’s guidance for long documents is to have the model pull out word for word quotes first, give a quote for each claim, and take back a claim when it cannot find one.33Anthropic, “Reduce hallucinations”. For a technical document I would write it like this.

For every number you report from this document, give the page,
the table or figure it comes from, and the sentence or table row
quoted word for word. If you cannot find a quote, say so and do
not report the number.

Footnotes

1. Anthropic, “PDF support”. ↩

2. Liu et al. (2023), abstract. ↩

3. Anthropic, “Reduce hallucinations”. ↩