I had a model running offline on my own computer in LM Studio, and I was wondering if a local model can learn from online resources. I asked an assistant, and the options were web search added on top of the local model, RAG over my own documents, or fine-tuning. RAG means a program searches your documents for the passages that match your question and hands those passages to the model along with the question, so the model itself is not changed. I said yes to that one.

So I enabled the local server in LM Studio, installed AnythingLLM, connected it to the server, made a workspace and uploaded research papers. The first long document failed with this:

Trying to keep the first 7280 tokens when context the overflows. However, the model is loaded with context length of only 4096 tokens.

The model was loaded with a context length of 4096 tokens, and the document needed 7280. The fix was to eject the model in LM Studio, reload it with a larger context length, and then update the AnythingLLM connection. I think context length is the first setting to raise before feeding a local model documents.

At that point I did not know what a token was. I asked two days later. A token is a piece of text the model counts in, often a whole short word or a part of a longer one, so a document has more tokens than words. Context length is how many tokens the model can hold at one time, and the question, the passages from the documents and the answer all have to fit in it.

I was putting together a package of documents with Claude, over more than one chat. One afternoon I picked it up again, so I asked Claude to retrieve where the last chat ended and deliver the complete set of documents for Word. It did that, with a summary.

Then the chat hit its length limit. A long chat fills up, and a new chat starts empty and cannot see the old one, so I was about to lose the context.

So before leaving it, I had Claude write a handoff file in markdown, meant for a fresh tab, with the instruction to look at the project files. A handoff note written at the end carries the work into the next chat.

Nine days later I did the same thing in a different chat, on a research idea. I had it write a summary of the conversation as a markdown file to continue elsewhere, then in another chat I asked for step by step instructions and started on step 1. That same day I had also asked Claude what tokens are, and got the explanation of tokens against words and what context length means.