I had a model running offline on my own computer in LM Studio, and I was wondering if a local model can learn from online resources. I asked an assistant, and the options were web search added on top of the local model, RAG over my own documents, or fine-tuning. RAG means a program searches your documents for the passages that match your question and hands those passages to the model along with the question, so the model itself is not changed. I said yes to that one.
So I enabled the local server in LM Studio, installed AnythingLLM, connected it to the server, made a workspace and uploaded research papers. The first long document failed with this:
Trying to keep the first 7280 tokens when context the overflows. However, the model is loaded with context length of only 4096 tokens.The model was loaded with a context length of 4096 tokens, and the document needed 7280. The fix was to eject the model in LM Studio, reload it with a larger context length, and then update the AnythingLLM connection. I think context length is the first setting to raise before feeding a local model documents.
At that point I did not know what a token was. I asked two days later. A token is a piece of text the model counts in, often a whole short word or a part of a longer one, so a document has more tokens than words. Context length is how many tokens the model can hold at one time, and the question, the passages from the documents and the answer all have to fit in it.