tishad

Where I stand with AI, and where a reader might

2026-10-11T00:00:00.000Z

A year ago I did not believe AI assistants were useful for my kind of work, and I thought what they cost in energy was hard to defend. Now I use one every working day. I did not get there overnight, and I did not change my mind about everything, so I want to tell it in the order it happened.

Where I started

I had two objections, and they were of different kinds.

The first was usefulness. Chemical engineering gave me a modeling habit before it gave me any data. My first project in graduate school was a steady-state flowsheet in Aspen Plus, where every stream was computed and nothing was measured, and the experimental work that followed was the opposite, real measurements and a mass balance that closed on good days. Neither needed a learning algorithm, and for most of that decade I read about neural networks the way I read about fusion. The value of my work sits in a measured number, and I did not see what a language model could add to that.

The second was cost. I work on fuels and energy, and a technology that needs new power stations for its data centres did not look like something an engineer should use casually. The International Energy Agency puts data centres at around 1.5 percent of the world’s electricity in 2024, about 415 terawatt-hours, and expects that to more than double, to around 945 terawatt-hours, by 2030, with AI as the largest driver of the growth.11International Energy Agency (2025), executive summary: data centres accounted for around 1.5% of the world’s electricity consumption in 2024, or 415 TWh, and consumption is set to more than double to around 945 TWh by 2030, with AI the most important driver.

What changed my mind

A friend of mine, a graduate student in the Department of Civil Engineering at UW, convinced me to try it and get started. So I started, in November 2025, having never written a line of Python. The first obstacle was the one every beginner meets, what to install and why it does not run. A hosted notebook took that question away. Google Colab gave me a working Python cell in a browser with nothing installed, and my first physics-informed neural network, on a first-order reactor, ran the same evening. Within a week I had a reading list that went from linear regression to gradient boosting, written for me in the order I needed it.

The second step was the opposite of the first. In February 2026 I moved the work onto my own machine: Python, an editor, a virtual environment, Jupyter. That month is the tall bar in the figure. Most of those thousand messages were error texts: a package name I had mistyped, a Windows alias that pretended to be Python, a search that ran overnight and wrote nothing to disk. Each cost an evening, and each is now a short note on this site. What I learned in that month is where an assistant earns its place: beside work I am doing myself, one cell at a time, explaining the error in front of me. When I let it run the whole analysis in its own window, I learned nothing and could check nothing.

From March to the end of August the curve in the figure is flat. The record behind the figure is my sessions with one assistant, and in those months I was not using it, so the record shows no new tool there. I used other tools, and I read books and articles and took courses, and I practiced on what I had met in February. None of that is in this record.

The third step came in the autumn, when I opened a coding agent in a folder on my own computer. At this stage prompting has little to do with it. The work is to read every line the model writes and run it and keep nothing I cannot explain. The model is fast and confident, and the second property is the dangerous one.

What I did about the cost

The second objection I did not resolve, so I put numbers beside it.

Google has published an estimate for its own assistant: the median text prompt uses 0.24 watt-hours of energy, emits 0.03 grams of carbon dioxide equivalent and consumes 0.26 millilitres of water, and by the company’s account the energy per prompt fell by a factor of 33 over twelve months.22Vahdat and Dean (2025): the median Gemini Apps text prompt is estimated at 0.24 Wh, 0.03 g CO2e and 0.26 mL of water, with energy per median prompt down 33 times over a recent twelve-month period. The figures are the company’s own. That is a company measuring its own product, and it covers a median text prompt, not a long session with a large model. In September 2026 I drew energy per query against model size as an animated chart, to see the shape for myself: the larger the model, the more each answer costs.

So two things are true at the same time. The cost of one question is small and falling. The total is large and rising, because use grows faster than efficiency improves. I use these tools every working day now, and I count the objection as open.

What the figure shows

All three panels are drawn from one source, my sessions with an assistant, and they measure only that. Panel a counts the days in each month on which I had at least one working session: 193 days in a little under twelve months, with February and September at 26 each and June at 4. Panel b counts the messages I wrote, and February alone, with a little over a thousand, holds more than the three months after it combined, which is what learning to program looks like from the inside. Panel c is the learning curve. It counts the tools, methods and working habits I met for the first time, 46 in all, and it has the shape of a staircase with one long landing. It climbs from November to February, when 36 of the 46 arrived. From March to the end of August it is flat, with one first meeting on the last days of August. The flat stretch is not idle time. The bars in panels a and b run through it, which is practice on what I had already met, and it also holds the other tools and the reading that a record of my sessions with one assistant does not see. The curve rises again in September. Named beside it, month by month, are the things I had to learn before I could work with an agent on my own machine, in a terminal and an editor.

The figure is a count of days and first meetings in one kind of record. It is not a measure of competence, and it is not the whole of the year.

Working with an agent did not start overnight

The last squares on the curve are the ones people ask about: a coding agent running on my own machine, with skills, memory files and subagents. It looks like a sudden change, and it was not. Each of those rests on something earlier on the same line.

In November 2025 I learned to keep context in files and not in the chat, because a long conversation forgets and a file does not. In January 2026 a project with an assistant in a cloud workspace forced Git, a repository and a tracker on me, since the file list had outgrown the conversation. In February came the terminal, the package manager, a virtual environment and a notebook on my own machine, and the habit of having a plan reviewed and criticized before anything is built. By August I had met a database, an API and a deployment.

Only then, in September 2026, did I open a coding agent in a folder on my own computer. What made it usable in the first week was everything above: I could read what it wrote, I knew what a commit was for, I knew why the environment mattered, and I already had the habit of asking for the plan first. Skills, which are saved procedures, and memory files, which are saved context, are the same two habits from November and February given names by the tool.

An engineer who starts with the agent and skips the year will get output sooner than I did. I do not think they will be able to check it.

Where that leaves me

A year ago I would have said that I ask a chat assistant for a draft or a summary now and then, and I would have defended staying there. Now I run models on my own hardware and I train and validate models on small tables of my own. I also work with a coding agent in a folder on my computer. Training a model on a small table is easy. Validating the result is the whole problem, and that is where my effort goes.

I have not published a machine learning method or result that someone else can reproduce, and I have not shipped anything that runs without me. What I can do is also narrow. I can handle a small table of runs, and for everything else I am still the person who asks the assistant. Physics-informed networks for reactor models are on my list because the problem fits. Özdemir and Yildirim describe the gap when they note that a purely data-driven model has no knowledge of thermodynamic limits and nothing stops it from predicting a conversion above equilibrium.33Özdemir and Yildirim (2025), p. 10, on hybrid and physics-informed models.

What I still do not trust

An assistant that is asked about a specific published number is wrong often enough that I read the source for every number I use, without exception. Al Sharah and colleagues put the general version of this well when they warn that easy access lets people apply methods they do not understand.44Al Sharah et al. (2024), pp. 623 to 624, under “challenges and drawbacks”. The warning applies to the chat window as much as to a random forest.

I also do not trust my own speed. A year of this has made me faster at producing things, and speed checks nothing. The check I keep is the one from February: if I cannot run it and explain it, it does not go in.

For a reader who has not started

The first step that worked for me was the cheapest one, a hosted notebook with nothing installed and one real problem from my own field. The pieces in the From zero series are the notes I would have wanted in that first month, in the order I needed them.

International Energy Agency (2025), executive summary: data centres accounted for around 1.5% of the world’s electricity consumption in 2024, or 415 TWh, and consumption is set to more than double to around 945 TWh by 2030, with AI the most important driver. ↩
Vahdat and Dean (2025): the median Gemini Apps text prompt is estimated at 0.24 Wh, 0.03 g CO2e and 0.26 mL of water, with energy per median prompt down 33 times over a recent twelve-month period. The figures are the company’s own. ↩
Özdemir and Yildirim (2025), p. 10, on hybrid and physics-informed models. ↩
Al Sharah et al. (2024), pp. 623 to 624, under “challenges and drawbacks”. ↩

Linux beside Aspen Plus

2026-10-11T00:00:00.000Z

Aspen Plus runs on Windows and nowhere else, and that one fact shaped the whole layout of the workstation. A process engineer who gives up the steady-state simulator to get a cleaner machine learning environment has traded the wrong thing.

The choice was between three layouts. A dual boot keeps both systems but makes switching a reboot, which is enough friction that one side goes unused. A live USB is a demonstration, not a workstation. The Windows Subsystem for Linux, version 2, runs a real Ubuntu kernel inside Windows, shares the file system, and passes the GPU through, so CUDA work in Linux sees the same RTX 4090 that the Windows tools see.11NVIDIA’s CUDA on WSL guide describes GPU passthrough to the WSL2 kernel; the driver lives on the Windows side and the CUDA toolkit inside Ubuntu. That is the layout in use: Windows 11 on the metal, Ubuntu 24.04 in the subsystem, Aspen Plus on one side and everything data-shaped on the other.

The Linux side holds the Python environment, managed with uv or conda, carrying the engineering stack I already had: scikit-learn, XGBoost and SHAP for tabular models, SIPPY for system identification, CasADi and Pyomo for optimization, BioSTEAM for techno-economic models. It also holds Docker, which runs Open WebUI, and will hold Ollama when the move from LM Studio happens.

Time-series data from a pilot unit arrive as thousands of tags at thirty-second intervals over weeks, which is tens of millions of rows per campaign in the worst case. Excel is the wrong container above a few hundred thousand rows and a database server is more than one person needs. Parquet files queried with DuckDB sit in between: columnar, compressed, fast to scan, and readable from Python without a running service.22DuckDB reads Parquet directly and runs analytical SQL in process; the documentation covers the Python package and the Parquet reader. The pattern is general and nothing in it is specific to any one unit.

What has surprised me is how little I miss on the Windows side. The simulator, the office tools and the vault application stay where they are. The terminal, the models and the data moved. The two meet at a shared folder, and a result computed in Linux opens in Excel without ceremony.

Still open is the laptop. A Surface Pro cannot run this layout well, and the question is whether the next portable machine is a Linux laptop that reaches the desktop over Tailscale or a second Windows machine with the same subsystem.

NVIDIA’s CUDA on WSL guide describes GPU passthrough to the WSL2 kernel; the driver lives on the Windows side and the CUDA toolkit inside Ubuntu. ↩
DuckDB reads Parquet directly and runs analytical SQL in process; the documentation covers the Python package and the Parquet reader. ↩

The local AI stack

2026-10-11T00:00:00.000Z

One rule decides everything on this machine: nothing leaves it, and nothing that belongs to my employer arrives on it. The first half is a technical constraint and the second is a policy I set for myself. Local processing does not make someone else’s data mine to carry home.

The hardware is a 2024 desktop with an Intel Core i9, an NVIDIA RTX 4090 with 24 GB of video memory, 64 GB of system memory in two sticks, and an MSI Z790 Edge board. The 4090 is the reason the rest exists. A 32 billion parameter model in 4-bit quantization fits in its memory with room for a usable context window, and that size is where open models stopped feeling like toys for engineering questions.

What runs today is LM Studio as the chat interface and model server, with AnythingLLM beside it for retrieval over my literature folder (the retrieval note covers that). The models are Qwen3 32B for reasoning, Qwen2.5-Coder 32B for code, and bge-m3 for embeddings. The embedding choice mattered more than I expected: retrieval quality moved more when I changed the embedding model than when I changed the chat model.

What is planned is the move from Windows-native tools to a Linux layer with Ollama serving the same models, Open WebUI in Docker as the interface, and Aider or Continue in VS Code for coding with a local model (the Linux note explains why it is a layer and not a replacement). Tailscale gives remote access from the laptop without opening a port.

What is excluded is any tool that needs a cloud API, which rules out the agentic coding assistant I use for this site. That is a real cost. The local coding models are a year or two behind, and I accept the gap on this machine because the whole point of it is the first rule.

What I have not done is fine-tune anything. The use cases so far are chat and writing over my own notes, code, and small machine learning jobs on tabular data, and none of them has needed a model trained on my material rather than a model reading it.

Private retrieval over my own papers

2026-10-11T00:00:00.000Z

A folder of about forty PDFs is the whole corpus, and the question I ask of it most often is “where did I read that”. Retrieval-augmented generation answers that question well and answers almost nothing else reliably.11Lewis et al. (2020) introduced the retrieval-augmented pattern: retrieve passages by vector similarity, then generate conditioned on them. The failure modes I describe follow from the pattern, not from any one implementation.

The setup is AnythingLLM pointed at the folder, with bge-m3 producing the embeddings and a local Qwen3 32B model writing the answers, all on the workstation. Each PDF is split into chunks of a few hundred words, each chunk becomes a vector, and a question is matched against those vectors before the model sees anything. The model then writes from the matched chunks and cites them.

What it is good for is locating. Asked which of the reviews gives a threshold for how many data points a molecular model needs, it returns the right paper and the right page in a few seconds, which is faster than my own memory and much faster than reopening five PDFs. It is also good for the first draft of a reference list and for finding every place a term appears across the corpus.

What it is not good for is the number itself. A retrieved chunk can be the wrong table, a value can be transcribed with the wrong unit, and the model will state the result with the same confidence either way. The rule that makes the tool safe is older than the tool: no number, claim or quotation goes into anything I write until I have read it in the source. Retrieval tells me where to read. It does not replace the reading.

Two practical findings. The embedding model matters more than the chat model; changing from a general English embedding to bge-m3 improved the hit rate on engineering terms more than any change of chat model did. And chunk size is a trade: short chunks find a sentence, long chunks keep a table together, and tables are where chemical engineering papers keep their results, so I lean long.

Open for me is whether a corpus of forty papers is worth this machinery at all. At forty, a good folder structure and a reading note per paper do most of the work. At four hundred the retrieval would earn its keep. The reading notes in the Library are the part of the system that scales either way.

Lewis et al. (2020) introduced the retrieval-augmented pattern: retrieve passages by vector similarity, then generate conditioned on them. The failure modes I describe follow from the pattern, not from any one implementation. ↩

Telling a skill exactly which files it may read

2026-10-04T00:00:00.000Z

I keep literature notes on journal articles in my Obsidian vault, and I have a skill in Claude Code that writes them from the PDF. A skill is a folder with a file of written instructions in it, and Claude follows those instructions when I run the command. I drop PDFs into an inbox folder and run the command, and it writes a note for each paper that does not have one yet.

This time there was one new paper waiting, a 33 page review. Claude read the whole paper. Then it went through my vault to pull context for a Connections section in the note, and on wrap up it also added a dated entry to one of my hub notes and made a Log section there, because the hub did not have one.

I was watching the run and I noticed the agent reading a lot of files. Every file it reads costs tokens. I think a skill reads whatever it can reach unless its instructions say what to read. So I told Claude what I want from this skill. I just want it to extract the data from the PDF and make the Markdown file in the literature folder inside my vault. I do not want it to consume any more tokens than it absolutely has to to do the job accurately.

Claude rewrote the skill so that it reads only its own instructions, the note template and the text of the PDF. It also added a small script, extract.sh, that pulls the text out of the PDF, and it saved the rule to its memory.

It had tested the extraction step on the PDFs in the inbox but not the full rewritten skill. So I ran the command again, and it wrote a note from the one PDF that did not have a note yet. I did not measure the tokens before and after.

Testing a skill with planted errors and a run without the skill

2026-10-03T00:00:00.000Z

I asked Claude Code to make a skill that checks the units on a calculation sheet. A skill is a file of written instructions, a SKILL.md, that Claude uses when the task calls for it or when you type its name as a command.11Anthropic (2026), “Extend Claude with skills”, Claude Code documentation. It is like an SOP, but for the assistant.

Along with the instructions it wrote a script, check_units.py, that does the conversions, with a test where 15 of 15 values agree when they are worked out by two routes. It wrote two reference sheets, one of unit factors and one of unit traps. And it checked its sources first: the conversion factors in NIST SP 811,22Thompson and Taylor (2008), NIST Special Publication 811, Appendix B, “Conversion Factors”. the licence of pint, which is a Python library for units, and the NASA report on the Mars Climate Orbiter. That is the spacecraft that was lost because thruster data came in pound-force seconds where the specification asked for newton seconds, so the numbers were off by a factor of 4.45.33Mars Climate Orbiter Mishap Investigation Board (1999), Phase I Report, section 4, “MCO Root Cause”, p. 16.

Then it made the test, a reference problem, which is a short calculation sheet with four errors planted in it, and five pass criteria written down with it. The four errors were a gallon conversion, a temperature difference, a site atmospheric pressure, and a missing gc. If you have not worked in US customary units, gc is the constant that connects pound-mass and pound-force in an equation, 32.174 lbm ft per lbf s squared, and if you leave it out your answer is wrong by a factor of 32.17.

To me this is like testing a method in the lab, where you make a sample that you already know the answer for and give it to somebody who does not know the answer. Here the somebody was a fresh assistant each time, one that had seen nothing of the work. Some trials had the skill and some did not, and some were allowed to run the script while the others had to do the arithmetic by hand. It used two Claude models, Haiku, which is the small one, and Fable 5.1, which is the strong one.

In the first round, Haiku with no skill caught three of the four errors, the gallon conversion, the temperature difference and the atmospheric pressure, and missed the missing gc. Fable 5.1 with no skill found all four. So the strong model passes this problem without the skill. Haiku with the skill, working by hand, caught the missing gc and named the 32.17 ratio as the sign of it. Fable 5.1 with the skill and the script passed every criterion and also returned ten rough edges in the skill text.

So Claude revised the skill and ran the trials again. Haiku by hand had two defects in round one: it had not answered all four conversions, and its values did not all carry both the sheet unit and the SI unit. Both were fixed after the revision, and it found all four errors again, but it still failed. Fable 5.1 passed by hand and with the script. The Haiku run with the script ended in an API error, and the partial report it left had missed gc again. So on the small model the catch did not hold every time.

After that the skill text was frozen at 169 lines and a final round was run on the frozen text. Fable 5.1 passed by hand and with the script. Haiku 4.5 passed with the script. Haiku by hand found all the errors and failed one criterion, because a single SI value was 1.9 percent off.

Before trusting the record Claude audited the transcripts of the runs, because a stray file from one of the earlier runs had landed in the skill folder itself, so a trial could have seen the expected answers.

So the strong model did not need the skill on this problem, and the small one gained from it, though not reliably. All of that is one problem and two models from one company, and the skill still needs a run on another vendor’s assistant and my own review.

Anthropic (2026), “Extend Claude with skills”, Claude Code documentation. ↩
Thompson and Taylor (2008), NIST Special Publication 811, Appendix B, “Conversion Factors”. ↩
Mars Climate Orbiter Mishap Investigation Board (1999), Phase I Report, section 4, “MCO Root Cause”, p. 16. ↩

Prediction and interpretation are two different uses of a fitted model

2026-08-23T00:00:00.000Z

I was trying to get my thoughts straight on what machine learning is for when the data come from experiments, so that I could say it in my own words. I had already asked an assistant what machine learning is used for in experimental research and development like mine, in chemistry and chemical engineering. Then I dictated my own version. Machine learning offers predictive modeling of datasets. On large experimental datasets the models fit well. On pilot data the same models do not, or at least it is not guaranteed. And for pilot-scale or process development work, prediction is not the only thing we want anyway. We want the relationships between the variables, the importance of each one, and the effect of each on the final product.

So there are two uses in that paragraph, and each has its own words. A fitted model is a function learned from a table. Each row is a run, the input columns are called features (temperature, feed type, reactor configuration) and the output column is called the target or the label (a yield). Prediction is the use everybody knows. You give the function a new row of features and read off the target. The other use is called interpretation, or model inspection. You ask the fitted function which features it leans on and in which direction, and you do not predict any new row.

This split is not new. Breiman in 2001 described two cultures of statistical modeling, one that assumes the data come from a stated model and reads its parameters, and one that treats the mechanism as unknown and judges an algorithm by how well it predicts.11Breiman (2001), abstract: the data modeling culture and the algorithmic modeling culture. Shmueli in 2010 argued that explaining and predicting are different goals that people mix up, and that the difference affects every step of the modeling.22Shmueli (2010), abstract: explanation and prediction as distinct purposes, and the conflation of explanatory with predictive power. I think an engineer already knows this split in a simpler form. A calibration curve is for predicting, because you read the signal and get the concentration. A regression of yield on temperature in a design of experiments is for explaining, because you want to know whether the slope is there and how large it is.

Interpretation has its own tools, and these are the ones I met.

Permutation importance takes a model that is already fitted, shuffles one feature column so it no longer lines up with the rows, and measures how much the score drops.33scikit-learn user guide, “Permutation feature importance”: the definition, the warning on low-importance features of a bad model, the statement that importance is specific to a particular model, and the note on correlated features. A big drop means the model was relying on that column.

SHAP values come from a 2017 paper by Lundberg and Lee. The method gives each feature an importance value for one particular prediction, so the prediction for one run is split into a contribution from each input.44Lundberg and Lee (2017), abstract: SHAP assigns each feature an importance value for a particular prediction. Averaged over all the runs that gives a ranking, and plotted against a feature it shows which way the effect goes.

An ablation study I had to ask about directly, what it is and what it tells us. It takes something out, an input or a part of the method, fits again, and compares the result with and without it. I think of it as running the unit with one stream shut off.

The documentation for these tools also says where they stop.

The scikit-learn guide says permutation importance shows how important a feature is for one particular model, and that a feature with low importance in a bad model could be very important in a good one, so the model has to predict well on held-out data before you read its importances.33scikit-learn user guide, “Permutation feature importance”: the definition, the warning on low-importance features of a bad model, the statement that importance is specific to a particular model, and the note on correlated features. So the two uses are tied together. If the model predicts badly, its ranking of the inputs means very little.

Then there are inputs that move together. If two features are correlated and you shuffle one, the model still gets the same information through the other, and both come out lower than they deserve.33scikit-learn user guide, “Permutation feature importance”: the definition, the warning on low-importance features of a bad model, the statement that importance is specific to a particular model, and the note on correlated features. In process data, two settings that were always changed together cannot be pulled apart by any ranking.

And a high importance does not mean cause. scikit-learn has an example called “Failure of Machine Learning to infer causal effects”, where a model scores almost as well with a variable left out and still gives a wrong answer about the effect of the one that stayed in.55scikit-learn example, “Failure of Machine Learning to infer causal effects”, including its lessons learned. The way out it points to is one an experimentalist already has, which is an experiment where you set that variable on purpose.

Where I have got to is that for pilot data I use the model mostly to find which variables matter and how, and much less to forecast. But one question still bothers me. If prediction accuracy is not the main goal of experimental and pilot-scale R&D, then what is?

Breiman (2001), abstract: the data modeling culture and the algorithmic modeling culture. ↩
Shmueli (2010), abstract: explanation and prediction as distinct purposes, and the conflation of explanatory with predictive power. ↩
scikit-learn user guide, “Permutation feature importance”: the definition, the warning on low-importance features of a bad model, the statement that importance is specific to a particular model, and the note on correlated features. ↩ ↩² ↩³
Lundberg and Lee (2017), abstract: SHAP assigns each feature an importance value for a particular prediction. ↩
scikit-learn example, “Failure of Machine Learning to infer causal effects”, including its lessons learned. ↩

A venv does not survive being copied to another machine

2026-02-21T00:00:00.000Z

I was running a Python project in a Jupyter notebook, and a run died when I closed the browser by accident. After that I moved the work to a new computer. I copied the whole project folder across to the Desktop of the new machine, and the virtual environment, the venv folder, went with it.

Then I tried to rebuild the environment there. First PowerShell refused to run the venv activate script, because of the execution policy. I got activation working. The Python documentation for venv gives this command for that case:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

With the venv active, pip failed:

Fatal error in launcher

The copied venv still pointed at the folder where it was made, which was a Desktop path under OneDrive on the old computer. A venv does not survive being copied between machines, because its scripts hard-code the original path. I did not know that. The same documentation page says it in a warning: the scripts in an environment contain the absolute path to the interpreter of that environment, so environments are not portable, and if you need one in a new location you recreate it there.

So I deleted the venv. My habit from the old command prompt was this, and it does not work in PowerShell:

rmdir /s /q venv

Then I made a fresh venv, upgraded pip from 25.1.1 to 26.0.1, and installed the packages again: pandas, numpy, matplotlib, seaborn, scipy, scikit-learn, openpyxl, jupyter, xgboost, optuna, shap and tqdm. After that the notebook cells ran.

I did not try to repair the copied venv, so I cannot say if editing the paths inside it would have worked. I think it is better to rebuild it than to fight it.

The next day I moved back to the laptop, and that time I copied the new files across minus the venv. That night I also pasted a pip line into the terminal and it lost its first words, so the terminal gave “shap is not recognized”.

What the DOE generative AI policy asks of the person using the tool

2026-02-15T00:00:00.000Z

I was writing a short post about the US Department of Energy policy on the use of generative AI. The document is DOE P 203.1, it was released in December 2025, and it is the first standalone policy the department has on these tools.11DOE P 203.1, Use of Generative Artificial Intelligence. Purpose and scope (pp. 1 to 2) for who it applies to and the exclusions; section B, Operational requirements for use (pp. 3 to 5), for human accountability and oversight, labeling and transparency, validation and verification, data provenance and training; section C (pp. 5 to 6) for high-impact use cases and the 2 April 2026 date. It is six pages. I uploaded the document to Claude and had it analyze it and lay it out visually, so I could understand it first and then write about it.

The first picture it made was full of text, and I thought people would rather read the original than read that. So I decided to highlight only the core principle and who it applies to. I wrote the principle myself, close to the wording of the policy: GenAI tools are intended to augment, not replace, human decision-making; users bear ultimate responsibility for the accuracy, completeness and appropriateness of all GenAI-generated outputs. The policy has a qualifier at the end that I left off, which is outputs used or relied upon for DOE work.11DOE P 203.1, Use of Generative Artificial Intelligence. Purpose and scope (pp. 1 to 2) for who it applies to and the exclusions; section B, Operational requirements for use (pp. 3 to 5), for human accountability and oversight, labeling and transparency, validation and verification, data provenance and training; section C (pp. 5 to 6) for high-impact use cases and the 2 April 2026 date.

Then a stronger line came up, something like “you own every mistake”, and I tried it. Before using it I asked Claude exactly where the document says that, so I could verify it myself. The policy says users bear ultimate responsibility for the outputs they use or rely on. It does not talk about mistakes or blame, and it ties the amount of checking to the risk and to what the output is used for.11DOE P 203.1, Use of Generative Artificial Intelligence. Purpose and scope (pp. 1 to 2) for who it applies to and the exclusions; section B, Operational requirements for use (pp. 3 to 5), for human accountability and oversight, labeling and transparency, validation and verification, data provenance and training; section C (pp. 5 to 6) for high-impact use cases and the 2 April 2026 date.

Most of the six pages is governance, like who the Chief AI Officer is and how a high-impact use case gets approved. The part about the person using the tool is a little over half a page. It says the tools augment the decision and a person still makes it. It says the user is responsible for the accuracy, completeness and appropriateness of what they use or rely on, and should know the tools can produce inaccurate, biased or false information. It says a direct output with a legal or binding effect must be explicitly labeled as AI-generated. And it says the user must validate outputs in proportion to the risk and the intended use, which it describes as fact-checking, checking that the output fits the context, and checking for hallucinations.11DOE P 203.1, Use of Generative Artificial Intelligence. Purpose and scope (pp. 1 to 2) for who it applies to and the exclusions; section B, Operational requirements for use (pp. 3 to 5), for human accountability and oversight, labeling and transparency, validation and verification, data provenance and training; section C (pp. 5 to 6) for high-impact use cases and the 2 April 2026 date.

There are two more items in the data section that are about the user. The user is accountable for documenting data sources and what was done to the data, mainly when a tool generates synthetic data or adds to a dataset. And the cut-off date of the model has to be considered for any analysis that needs current information.11DOE P 203.1, Use of Generative Artificial Intelligence. Purpose and scope (pp. 1 to 2) for who it applies to and the exclusions; section B, Operational requirements for use (pp. 3 to 5), for human accountability and oversight, labeling and transparency, validation and verification, data provenance and training; section C (pp. 5 to 6) for high-impact use cases and the 2 April 2026 date.

In my post I wrote that anyone using GenAI tools for DOE work is personally responsible for whatever it produces, and in the picture I added that it applies if you use the tools at a DOE facility. Then I read the scope again, and it is narrower than that. It governs the department’s own elements and their staff. For contractors it says the use of these tools stays governed by existing policies. It does not apply to recipients of financial assistance from the department unless stated otherwise, it does not apply to using generative AI to carry out basic or applied research, and it does not cover national security systems.11DOE P 203.1, Use of Generative Artificial Intelligence. Purpose and scope (pp. 1 to 2) for who it applies to and the exclusions; section B, Operational requirements for use (pp. 3 to 5), for human accountability and oversight, labeling and transparency, validation and verification, data provenance and training; section C (pp. 5 to 6) for high-impact use cases and the 2 April 2026 date. So as I read it, a university group working on a DOE grant is not covered, and my line was wider than the policy.

The labeling part is also narrower than I first put it. I wrote that the policy requires labeling anything AI-generated that has legal or binding effect, and that is the one explicit mandate it gives. Otherwise it says it strongly favors transparency. It does not ask for a label on every paragraph an assistant helped with.11DOE P 203.1, Use of Generative Artificial Intelligence. Purpose and scope (pp. 1 to 2) for who it applies to and the exclusions; section B, Operational requirements for use (pp. 3 to 5), for human accountability and oversight, labeling and transparency, validation and verification, data provenance and training; section C (pp. 5 to 6) for high-impact use cases and the 2 April 2026 date.

I think this policy will be adopted more and more as an AI usage guide in academia and industry. Every organization is working out its internal AI guidelines right now, and the governance, data handling and IP rules in it are transferable. I cannot show that this will happen, it is what I think.

I only read the policy, not the other DOE orders it points to, and I am not a lawyer. Some parts of it were still to come when I read it. It says mandatory training requirements will come within six months of the release, and the requirements for high-impact uses take effect on 2 April 2026.11DOE P 203.1, Use of Generative Artificial Intelligence. Purpose and scope (pp. 1 to 2) for who it applies to and the exclusions; section B, Operational requirements for use (pp. 3 to 5), for human accountability and oversight, labeling and transparency, validation and verification, data provenance and training; section C (pp. 5 to 6) for high-impact use cases and the 2 April 2026 date.

DOE P 203.1, Use of Generative Artificial Intelligence. Purpose and scope (pp. 1 to 2) for who it applies to and the exclusions; section B, Operational requirements for use (pp. 3 to 5), for human accountability and oversight, labeling and transparency, validation and verification, data provenance and training; section C (pp. 5 to 6) for high-impact use cases and the 2 April 2026 date. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

The first chart to learn is predicted against actual

2026-02-10T00:00:00.000Z

I was practicing machine learning on my own computer in VS Code. I started with a script and then switched to Jupyter when I saw it would be easier cell by cell. The first figure that came out of the notebook was a parity plot, which is also called predicted against actual.

It is a simple chart. Each run is one point, with the measured value on the horizontal axis and the value the model predicted for that run on the vertical axis. Then you draw the 45 degree line. If the model predicted every run exactly, every point would sit on that line. It is the same chart you would draw to compare a flow meter against a reference, with the model in place of the meter.

import matplotlib.pyplot as plt

fig, ax = plt.subplots()
ax.scatter(y_measured, y_predicted)
ax.axline((0, 0), slope=1)   # the 45 degree line
ax.set_xlabel("measured")
ax.set_ylabel("predicted")
ax.set_aspect("equal")

axline draws a line through a point with a given slope, so you do not have to work out the axis limits first. The code above is a general version and not the cell from my notebook.

A score like R2 is one number for all the runs. On the plot you can see each run, so you can see which runs the model misses and in which direction. With these axes a point above the line is a run the model overpredicts. If the cloud of points is flatter than the line, the model is pulling everything toward the average, so the high runs are predicted low and the low runs high. If a group of points sits off the line together and they are all one feed or one reactor configuration, then I think the model has not learned that feed or that configuration. If there is one point far from the others, I would look that run up in the lab notebook first, before deciding anything about the model.

The predictions on the plot should come from runs the model was not fitted on. scikit-learn has its own version of the chart, PredictionErrorDisplay, and its documentation says to use it “preferably on held-out data points”. The axes are also not the same everywhere. With kind="actual_vs_predicted" scikit-learn puts the observed values on the vertical axis and the predicted on the horizontal, so above and below the line mean the opposite of what I wrote here.

What a SHAP value says and what it does not

2026-02-10T00:00:00.000Z

I was fitting machine learning models to a small table of runs. Modeling in the browser was taking too long, so I moved the work to my own computer, and the code came from an assistant one step at a time. The first step was an install line, and one of the libraries in it was called shap.

pip install pandas numpy scikit-learn xgboost shap matplotlib seaborn

By the end of that day I had a SHAP bar chart, a SHAP beeswarm and a SHAP plot against temperature, all from models I had fitted earlier the same day. Each figure was a few lines of code. So then I had to find out what those figures let me say about the process behind the runs.

What a SHAP value is

SHAP stands for SHapley Additive exPlanations. It is a method, and a Python library with the same name, that takes one prediction from a fitted model and divides it among the inputs of the model.11Lundberg and Lee (2017), abstract: SHAP assigns each feature an importance value for a particular prediction.

Say the model predicts a yield for one run. The model also has an average prediction over all the runs it was given. The prediction for this run sits above or below that average, and SHAP splits the difference among the inputs of the run, so much to the temperature and so much to the feed. The shares add up exactly to the difference. The idea comes from game theory, where a group of players earns a payout together and you want to divide it fairly when they did not contribute equally. The share of one input is found by adding it to every possible combination of the other inputs and averaging how much the prediction moves each time.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features. For a process engineer I think the nearest thing is splitting a shared utility cost among the units that use it.

A SHAP value has the units of what the model predicts. If the model predicts a yield in weight percent, then a value of plus 2 for temperature on one run means that, in this model’s accounting, the temperature of that run put the prediction 2 points above the average prediction.

Reading the beeswarm

The plot you see most is the beeswarm, which the library also draws with shap.summary_plot. Every run is one dot on the line of each input. How far the dot sits left or right is the SHAP value of that input for that run, and the color is the value of the input itself, low to high. Where many runs have similar values the dots pile up.33SHAP documentation, beeswarm plot: one dot per instance per feature, x position by SHAP value, color by original feature value, default ordering by mean absolute value and the alternative ordering by maximum absolute value. So you read each line twice. First you look at how far the dots spread from zero, because that is how much the input moves the predictions. Then you look at whether the colors sort along the axis, because if the high values sit on the right and the low ones on the left, the prediction goes up with that input. The inputs are stacked from top to bottom by the mean of their absolute SHAP values, and the bar chart that comes with the beeswarm is only that mean.33SHAP documentation, beeswarm plot: one dot per instance per feature, x position by SHAP value, color by original feature value, default ordering by mean absolute value and the alternative ordering by maximum absolute value. The documentation says that this ordering favors an input with a broad average effect over one that matters a lot for a few runs, and that you can order by the maximum absolute value to find the second kind.33SHAP documentation, beeswarm plot: one dot per instance per feature, x position by SHAP value, color by original feature value, default ordering by mean absolute value and the alternative ordering by maximum absolute value. With a small table where one group has only a handful of runs, I think that second ordering is worth the extra line.

When I redid the figures about two weeks later, the beeswarm cell failed because shap.summary_plot did not accept an ax argument, which is the usual way to tell a plotting function which panel to draw in. It got fixed.

What it does not tell you

All of this is about the fitted model. The shares add up to the model’s prediction, and the measured value does not come into it, so if the model predicts a run badly, SHAP explains the bad number completely. Whether the model predicts well on runs it has not seen has to be shown somewhere else, by validation, because the plot cannot show it.

The other thing is cause. A model reproduces the associations in the data it was trained on, and SHAP displays them. An article in the library’s own documentation, with the first author of the method among its writers, says “making correlations transparent does not make them causal”.44Dillon et al., SHAP documentation: the quoted sentence, and the conditions of independence from other features and from unobserved confounders. They give the condition under which a causal reading is safe, which is that the input has to be independent of the other inputs and of anything unmeasured that also drives the outcome.44Dillon et al., SHAP documentation: the quoted sentence, and the conditions of independence from other features and from unobserved confounders. Data from experiments stand a little better here than plant operating records, because the experimenter sets the factors and they do not drive one another. But anything that moved with a setpoint and was not recorded is carried inside the share of that setpoint. A residence time that shifts with temperature would be one, or a feeder that behaves differently with one material.

There are four common readings of the plot that it does not support.

The first is that the value is what would be lost if the input were removed. Molnar says plainly that the Shapley value is not the difference in prediction when the feature is taken out of the model.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features. To answer that you refit without the input, and that is a different experiment.

The second is that the value is a slope. It cannot tell you how much the prediction changes for a given change in the input. Molnar suggests pairing it with curves that vary one input and hold the rest.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features.

The third is that a high rank means the input is important in the process. The values are relative to the data they were computed on and to the particular model.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features. Another model fitted to the same runs may divide the same prediction differently.

The fourth is that correlated inputs are divided cleanly. When inputs are correlated the calculation evaluates the model at combinations that never occur in reality, and the shares become unreliable.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features.

Lundberg and Lee (2017), abstract: SHAP assigns each feature an importance value for a particular prediction. ↩
Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features. ↩ ↩² ↩³ ↩⁴ ↩⁵
SHAP documentation, beeswarm plot: one dot per instance per feature, x position by SHAP value, color by original feature value, default ordering by mean absolute value and the alternative ordering by maximum absolute value. ↩ ↩² ↩³
Dillon et al., SHAP documentation: the quoted sentence, and the conditions of independence from other features and from unobserved confounders. ↩ ↩²

Ask where the numbers came from before charting them

2026-02-08T00:00:00.000Z

I wanted to write a one page article on how chemical engineering graduates in the US spread across industries and roles during their careers, traditional and non traditional, to start a conversation with other chemical engineers. I asked Claude for the analysis and it built a first one. Then I insisted on doing all of it myself in VS Code, step by step, in a Jupyter notebook.

Getting Python and Jupyter running took a while. Once they ran, the first cells failed with KeyError: 'year'. The code was calling the API of the Bureau of Labor Statistics (BLS), and the API had returned 0 data points because the series ID was wrong, so there was no year to work with. Then I asked Claude whether it had made up the data in cell 3. The first data was fabricated. So that evening we moved to pulling everything from the agencies, and since then I ask where the numbers came from before I chart them.

The employment and wage numbers come from the Occupational Employment and Wage Statistics tables of the BLS, which are published for each year as a zip file with spreadsheets inside.11BLS, OEWS tables page: national, state, area and industry files for each reference year, as zipped XLSX files from 2014 onward. Chemical engineers are occupation code 17-2041. The zip downloads failed with 403 at first, and then all years came back 200. Then the years 2014 to 2018 came out wrong, because the code was reading field_descriptions.xlsx from each zip instead of national_M20xx_dl.xlsx, and that was fixed by picking the right file. After that I had a real series for 2014 to 2024, saved to data/bls_cheme_timeseries.csv.

The second source was Data USA, which serves Census data through its own API.22Data USA API documentation. Every documented form of the request gave 404, even the example from their own docs, and I found the working endpoint myself. From it I got the number of people in the occupation and their average wage for 2014 to 2023. I also tried to get what jobs people with a chemical engineering degree actually hold. The server gave a 500, then an empty result at one level, and the next level did not exist in that dataset, so I gave up on that and used the data from the extracted zips instead.

I added the same series for mechanical, electrical, civil and computer hardware engineers, software developers and all occupations, because I think scientific data has to be presented against something, or the chart means nothing.

Pulling the data myself took the evening. In those numbers, BLS employment of chemical engineers fell from 33,470 in 2014 to 20,330 in 2024, while the Census count of people in the occupation rose from 56,387 in 2014 to 84,824 in 2023. I think that means graduates are spreading into other roles.

BLS, OEWS tables page: national, state, area and industry files for each reference year, as zipped XLSX files from 2014 onward. ↩
Data USA API documentation. ↩

Telling an assistant to stop agreeing

2026-02-02T00:00:00.000Z

I was making a cover banner for my LinkedIn profile, late at night. I gave Claude a banner I liked as the reference and asked for versions that show process development, pilot plants and de-risking commercialization.

The first set looked amateur. So I told it to think about what a billion dollar company would put on a cover, something aimed at executives and business development leaders, and to analyze examples online before making the next set. I kept repeating the same rules. Leave room on the left so the profile photo covers nothing. Add the phrase “de-risking commercialization”. At one point it gave me a chaos theory graph, which does not apply to what I do. Then I told it to stop being a yes man and to use search and analysis.

What helped was that I spelled out what de-risking means to me in one sentence: validating that a laboratory process can be scaled up economically and safely to a commercial plant, so that business leaders can invest in the technology to make money. The banner only improved after that. Then I had the design I picked rendered at high resolution with JavaScript so the lines are smooth and the text is clean, and it had forgotten the room for the photo again, so the whole composition had to be moved right.

Claude drifts into agreeing with whatever I say unless told to search and argue. There is a name for this, sycophancy. A group of researchers tested five AI assistants on four kinds of free-form writing task and found it in all of them. Their explanation is the training: in human preference data a response that matches the user’s view is more likely to be preferred, and people sometimes prefer a convincingly written agreeable answer over a correct one.11Sharma et al. (2023), abstract: five AI assistants showed sycophancy across four free-form text generation tasks; a response matching the user’s views is more likely to be preferred in human preference data; humans and preference models at times prefer convincingly written sycophantic responses over correct ones.

A few nights later I was working on a logo with it. I had a clear idea of the geometry, and it kept distorting the shapes, misaligning the letters and filling gaps I did not ask for. That went on for two hours. I told it that it was wasting my time, and to ask questions and get it right the first time. In the end I edited the SVG by hand and pasted it back, which for geometry works better than describing the fix in words.

After that I was in OpenLCA, trying to reproduce a published life cycle assessment with free databases. I had the databases imported and could not find the process I needed. Searching “woodchips” in the inputs dialog showed wood results but not hardwood. So I told Claude to ask me multiple choice questions so it understands the situation.

Sharma et al. (2023), abstract: five AI assistants showed sycophancy across four free-form text generation tasks; a response matching the user’s views is more likely to be preferred in human preference data; humans and preference models at times prefer convincingly written sycophantic responses over correct ones. ↩

Git and GitHub for one person

2026-01-16T00:00:00.000Z

I started my first project with an assistant in January, and I chose GitHub to host it because I knew little about hosting. I am a chemical engineer and I had never used Git. For the next several months I used it without understanding what it was doing.

Git and GitHub are two different things

Git is a program on my computer that keeps the history of a folder. GitHub is a website that stores a copy of that history.

In February I installed VS Code, signed in to GitHub and cloned my repository, which means downloading a copy of it to my computer. The “Git: Clone” command did not show in VS Code at first, then it appeared. The repository was empty because I had deleted everything in it earlier. I built it again on my computer, looked at it there, and did not push it back up, because there was no point publishing it as it was.

A commit

The unit of history in Git is the commit. The Pro Git book describes it as a picture of what all the files look like at that moment, and Git keeps a series of those pictures.11Pro Git, chapter 1.3: snapshots and not differences; the three states modified, staged and committed. A file goes through three states on the way: modified, staged (marked to go into the next snapshot) and committed.11Pro Git, chapter 1.3: snapshots and not differences; the three states modified, staged and committed.

git add .
git commit -m "a short message saying what changed"

The nearest thing I know from my own trade is an issued revision of a drawing. You can mark up the working copy all day, but Rev B exists only when it is issued, with a date and a name on it, and you can always go back to it. Git also wants the name, so before the first commit you set your name and email once.

My first real one was an analysis of public workforce statistics that I did in a notebook. The first commit had 27 files and 2,718 insertions. The README came after that, because I first created it one folder up by accident.

The part I did not understand was that a folder with Git in it has no history unless I make the commits. About a week later I was building something else with an assistant, file by file, and one evening a setup script gave me back much less than we had built. So I checked:

git log

The repository held two early commits and nothing of the work since, because I had never committed it. In March I opened that old folder again and there was nothing in it to dig for, so I started over from a fresh template.

A branch

A branch is a movable pointer to one of the commits. Making one copies nothing, and new commits on it leave the main line where it was.22Pro Git, chapter 3.1: a branch as a lightweight movable pointer to a commit; creating a branch creates a new pointer.

Working alone I have used it for one purpose, which is a trial I might throw away. When I was checking whether a tool could do what I wanted before I built on it, the test went on its own branch, and its commit had 17 files. If the trial had failed I could have deleted the branch, and the main line would have stayed as it was.

Push

A commit lives only on the computer where it was made. A remote is a copy of the project hosted somewhere else, and git push sends the local commits to it.33Pro Git, chapter 2.5: remote repositories, git push <remote> <branch>, and the name origin. After you clone from GitHub, Git calls that server origin.33Pro Git, chapter 2.5: remote repositories, git push <remote> <branch>, and the name origin.

git push origin main

So committing and pushing are two decisions. The day I cloned for the first time I did not push, because what I had was not good enough to publish. The analysis I did push, and that first push was 4.02 MiB.

.gitignore

A .gitignore is a text file that lists what Git should leave alone.44gitignore documentation: the file specifies intentionally untracked files; files already tracked are not affected. I put in it what can be rebuilt, like a Python virtual environment or the node_modules folder, and what must not be published, like files with keys in them.

It has to be there before the first commit, because a rule added later does not affect files Git is already tracking.44gitignore documentation: the file specifies intentionally untracked files; files already tracked are not affected. In my analysis repository it went in with the first commit. Months later a coding agent set up a new repository for me and its second commit was the .gitignore: an internal data folder, node_modules, the build folders and the environment files. Then it tested the rule with git check-ignore on a dummy file under the data folder, before any real data went in.

I think you also have to remember that an ignored file is in no commit, so pushing does not back it up. That data needs its own copy somewhere.

Using it alone

GitHub defines a private repository as one that only the owner and the people the owner shares it with can reach.55GitHub Docs, “About repositories”: a private repository is accessible only to the owner and people explicitly given access. With nobody else on the project I still get two things from it.

One is a copy off the machine. In February I moved a modeling project to a new computer by copying the folder, and the same project also existed as a folder ending in v2 and another ending in v3, with files copied between them. The other is the dated record. git log tells me what the work looked like on a given day, which I would otherwise answer from memory and file timestamps.

In August Git and committing felt like too much for me, so for one job I chose a host where I could upload the files myself. Four days later I moved to a coding agent, and the first thing it did was work in a repository. The repository was empty, so its first commit was the rules file and its second was the .gitignore. The first commit came out with the agent as the author, and my rule is that commits carry my name, so it reset the author. The fix that lasts is to set the name once for the whole machine, because a fix inside the agent’s container does not persist:

git config --global user.name "Your Name"
git config --global user.email "you@example.com"

Pro Git, chapter 1.3: snapshots and not differences; the three states modified, staged and committed. ↩ ↩²
Pro Git, chapter 3.1: a branch as a lightweight movable pointer to a commit; creating a branch creates a new pointer. ↩
Pro Git, chapter 2.5: remote repositories, git push <remote> <branch>, and the name origin. ↩ ↩²
gitignore documentation: the file specifies intentionally untracked files; files already tracked are not affected. ↩ ↩²
GitHub Docs, “About repositories”: a private repository is accessible only to the owner and people explicitly given access. ↩

Kernel models in plain terms

2025-11-25T00:00:00.000Z

I first met support vector regression one evening when I was working through scikit-learn in a course I was taking with an assistant, and it was one item in a list that started at linear regression and went on through random forests and XGBoost. A few months later it was one of the models in a tuning search I ran on my own computer, and it was one of the quick ones to tune.

What a kernel is

A kernel is a function that takes two rows of the data table, two runs, and gives back a number that says how alike their conditions are. The common one is the radial basis function, RBF. In scikit-learn it is exp(-gamma d²), where d is the distance between the two runs in the space of their inputs.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. Two runs at the same conditions score 1, and the score falls toward 0 as the conditions move apart.

A kernel model predicts a new run from the training runs, each one weighted by how alike it is to the new one. It is like interpolating in a steam table, where the two neighbouring entries do all the work. The kernel does the same with several inputs at once, and the influence of a run fades smoothly with distance.

How fast it fades is the main setting. In support vector regression it is gamma, and the larger it is, the closer a run has to be to count.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. In a Gaussian process the same thing is given as a length_scale, which Rasmussen and Williams describe as roughly the distance you have to move in input space before the function value can change significantly.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. If you put the two formulas side by side, gamma is 1 over twice the square of the length scale.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. 33scikit-learn user guide, section 1.7: hyperparameters fitted by maximizing the log marginal likelihood, n_restarts_optimizer, alpha and WhiteKernel, the RBF kernel exp(-d²/(2 l²)). So a large gamma and a short length scale say the same thing: only very near runs count, and the fitted surface can wiggle a lot.

Why the inputs are scaled first

The similarity comes from a distance, and a distance adds up the differences across all the input columns in whatever units they are in. If one column is a temperature that spans several hundred degrees and another is a 0 or 1 code, then the temperature decides the distance almost alone. The scikit-learn guide says support vector machines are not scale invariant and that scaling the data is highly recommended, for example standardizing each input to mean 0 and variance 1.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. In code you tie the scaler to the model in one line, and then every new row gets the same scaling:11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma.

model = make_pipeline(StandardScaler(), SVR(kernel="rbf", C=1.0, epsilon=0.1))

Support vector regression

Support vector regression looks for a function that stays within a distance epsilon of the training targets and is otherwise as flat as possible. Smola and Schölkopf say errors smaller than epsilon are not cared about.44Smola and Schölkopf (2004), section 1.2: the goal of epsilon support vector regression and the role of the constant C. I read epsilon as a tolerance band, the plus or minus inside which a yield counts as reproduced. A point inside the band costs nothing.

Real data seldom fit inside a band, so a miss beyond it is allowed at a price, and C sets the price. It is the trade between how flat the function is and how much deviation beyond epsilon is tolerated.44Smola and Schölkopf (2004), section 1.2: the goal of epsilon support vector regression and the role of the constant C. A high C follows the training runs closely. A low C gives a smoother function and accepts some misses, and the scikit-learn guide says to lower it when the observations are noisy.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma.

The name comes from the band. The fitted function depends only on some of the training runs, the ones on or outside the band, because the cost ignores the runs whose prediction is already close to the target.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. Those runs are the support vectors.

So the settings are C, epsilon and gamma. The guide calls the choice of C and gamma critical and says to search them over values spaced exponentially.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. What you get back is one number for each prediction, and nothing that says how far to trust it.

Gaussian process regression

A Gaussian process starts from the same kernel. It treats the unknown function as random, and it uses the kernel to say how strongly the outputs at two conditions are correlated, strongly when the conditions are close and weakly when they are far apart.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. From the training runs it then gives, for any new condition, a predicted mean and a predicted variance.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. In scikit-learn you ask for it in the call:

mean, std = gpr.predict(X_new, return_std=True)

The standard deviation is small near the training runs and grows away from them.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. For someone who plans experiments I think this is the useful part, because the model marks where in the operating window it is guessing.

The settings are handled differently too. The length scale and the noise level are fitted to the data by maximizing a quantity called the log marginal likelihood. That surface can have several local optima, so scikit-learn has n_restarts_optimizer to repeat the fit from different starting values.33scikit-learn user guide, section 1.7: hyperparameters fitted by maximizing the log marginal likelihood, n_restarts_optimizer, alpha and WhiteKernel, the RBF kernel exp(-d²/(2 l²)). Measurement noise goes in through alpha or through a WhiteKernel term.33scikit-learn user guide, section 1.7: hyperparameters fitted by maximizing the log marginal likelihood, n_restarts_optimizer, alpha and WhiteKernel, the RBF kernel exp(-d²/(2 l²)).

It has a cost in arithmetic. The basic algorithm works on a matrix with one row and one column for each training run, and the work grows with the cube of the number of runs.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. At a few tens of runs that is nothing.

The other cost is that the standard deviation is computed from the model’s own assumptions, which are the kernel you chose, the smoothness it implies and noise of constant size. Rasmussen and Williams show the same data explained by a short length scale with almost no noise and by a long length scale with a lot of noise, and the error bars are very different.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. So the interval depends on the kernel, and it does not measure experimental error. For that you still need replicate runs.

Both models fall back to something flat outside the training range. Far from every training run the RBF similarity is zero for all of them, so the SVR prediction goes to a constant and the Gaussian process goes back to its prior mean, with the standard deviation at its largest.

scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸
Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
scikit-learn user guide, section 1.7: hyperparameters fitted by maximizing the log marginal likelihood, n_restarts_optimizer, alpha and WhiteKernel, the RBF kernel exp(-d²/(2 l²)). ↩ ↩² ↩³
Smola and Schölkopf (2004), section 1.2: the goal of epsilon support vector regression and the role of the constant C. ↩ ↩²

Machine learning vocabulary in terms a process engineer already knows

2025-11-25T00:00:00.000Z

I spent an evening on a course an assistant was giving me, and it went through the scikit-learn workflow, from exploring the data and preprocessing to cross-validation and hyperparameter tuning. scikit-learn is the Python library that holds most of the standard machine learning methods. I am a chemical engineer and I was just starting to learn, so a lot of that evening was new words. A few months later I ran my first modeling script on my own computer, and its first step reported how many samples and how many features it had loaded. Samples are runs and features are input columns. Most of these words are like that, a new name for something you already do with a run table or a calibration curve.

The table

A sample is one row of the data table, and for a pilot unit one row is one run. The scikit-learn glossary says other people call it an instance, a data point or an observation.11scikit-learn glossary, entries “sample”, “feature” and “target”.

A feature is an input column, something you set or know before the run gives a result, like a temperature setpoint or a feed type.11scikit-learn glossary, entries “sample”, “feature” and “target”. In code the block of feature columns is called X.

A target is the output column, the thing you want predicted, like a yield or a conversion. The glossary also lists outcome, response and label for it.11scikit-learn glossary, entries “sample”, “feature” and “target”. In code it is y. You fit a model with fit(X, y) and then ask it for predictions with predict(X).

Holding runs back

Think of a calibration curve. The training set is the runs the model is fitted on, like the standards you use to draw the curve. The test set is the runs you keep out of the fitting and use only to check the model, like a check standard that was not used to draw the curve. The scikit-learn guide gives the reason for it: a model that only repeated the values it had already seen would score perfectly and still fail to predict anything useful on a new run.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. The function that splits the table is train_test_split.

Cross-validation is the same check done in rotation. The runs are divided into groups called folds, the model is fitted on all the folds except one and checked on the one left out, then this is repeated until every fold has been the check once, and the scores are averaged.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. It is like rotating which standard you hold back. In scikit-learn it is KFold and cross_val_score.

Leave-one-out is cross-validation with one run in each fold. Each run is held back in turn, the model is fitted on all the others and then predicts the held-back one, so for n runs that is n fits.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. The name is LeaveOneOut.

Settings

A hyperparameter is a setting you choose before the fit, and a parameter is a number the fit finds. If you fit a polynomial to calibration data, the order of the polynomial is the hyperparameter and the coefficients are the parameters. The depth of a decision tree is another hyperparameter. The guide defines them as parameters that are not learnt directly within the estimator and are passed in when you construct it.33scikit-learn user guide, section 3.2: “Hyper-parameters are parameters that are not directly learnt within estimators”, passed as arguments to the constructor; GridSearchCV considers all combinations of given values and RandomizedSearchCV samples a given number of candidates; the final evaluation is to be done on an evaluation set of samples not seen during the search. GridSearchCV and RandomizedSearchCV are the tools that try many settings and compare them by cross-validation.33scikit-learn user guide, section 3.2: “Hyper-parameters are parameters that are not directly learnt within estimators”, passed as arguments to the constructor; GridSearchCV considers all combinations of given values and RandomizedSearchCV samples a given number of candidates; the final evaluation is to be done on an evaluation set of samples not seen during the search.

Overfitting is the calibration curve with too high an order. It goes through every standard, noise included, and then it is wrong for the next sample.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. You see it as a big gap between the score on the training set and the score on the runs held back.

A baseline is the simplest prediction you have to beat. For a yield that is the mean yield of the training runs, given for every run whatever the conditions were. scikit-learn has it as DummyRegressor, which by default always predicts the mean of the training set.44scikit-learn API reference, DummyRegressor: a regressor that makes predictions using simple rules, useful as a simple baseline; the default strategy “mean” always predicts the mean of the training set. If a model cannot beat that, it has learned nothing from the features.

Preparation and what is left over

Scaling puts the feature columns on comparable ranges, because a temperature in the hundreds of degrees and a flag that is 0 or 1 are very different sizes of number. StandardScaler subtracts the mean of each column and divides by its standard deviation. The part I think is easy to miss is that the scaler is fitted too. The guide says it should be learnt from the training set and then applied to the held-out runs.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data.

A pipeline is a fixed sequence of steps, say scaler then model, handled as one object. I think of it as the written procedure, so the same preparation is done every time in the same order. And when you give the whole pipeline to cross-validation, the scaler is refitted on the training part of every split, so nothing about the held-back runs reaches the model. When information from the test runs does get into the model building it is called data leakage, and the documentation recommends the pipeline to prevent it.55scikit-learn, Common pitfalls and recommended practices, section on data leakage: the definition, the rule never to call fit on the test data, and the recommendation of the pipeline for cross-validation and hyperparameter tuning. The names are Pipeline and make_pipeline.

A residual is measured minus predicted for one run, the same as in any regression, and you read it the same way. Plot it against the predicted value or against a feature and it should show no pattern. scikit-learn has no object for it, it is just y - model.predict(X).

Word	What it is on the plant side	scikit-learn name
Feature	Setpoint or feed property	`X`
Target	Yield	`y`
Training set	Calibration standards	first outputs of `train_test_split`
Test set	Check standard	second outputs of `train_test_split`
Cross-validation	Rotating the check standard	`KFold`, `cross_val_score`
Leave-one-out	One check standard at a time	`LeaveOneOut`
Hyperparameter	Order of the calibration polynomial	constructor argument, `GridSearchCV`
Overfitting	Curve through every point	none, seen as a gap between training and test scores
Baseline	Mean of past runs	`DummyRegressor`
Scaling	Comparable ranges for all columns	`StandardScaler`
Pipeline	Written procedure	`Pipeline`, `make_pipeline`
Residual	Measured minus predicted	none, `y - model.predict(X)`

Where the calibration comparison stops

The calibration comparison works for what the words mean, and I think it stops working when you ask how much data you need. A calibration has one input and a known form, so a few standards are enough. A model with several features and no assumed form needs many more rows before it can say anything, and a run table from a pilot unit has few.

One thing about the comparison above. I used the check standard for the test set and also for the fold held back inside cross-validation. The user guide keeps a separate evaluation set for the final score, one that was never seen while the settings were being searched,33scikit-learn user guide, section 3.2: “Hyper-parameters are parameters that are not directly learnt within estimators”, passed as arguments to the constructor; GridSearchCV considers all combinations of given values and RandomizedSearchCV samples a given number of candidates; the final evaluation is to be done on an evaluation set of samples not seen during the search. so those two are different things and I have used one picture for both.

scikit-learn glossary, entries “sample”, “feature” and “target”. ↩ ↩² ↩³
scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. ↩ ↩² ↩³ ↩⁴ ↩⁵
scikit-learn user guide, section 3.2: “Hyper-parameters are parameters that are not directly learnt within estimators”, passed as arguments to the constructor; GridSearchCV considers all combinations of given values and RandomizedSearchCV samples a given number of candidates; the final evaluation is to be done on an evaluation set of samples not seen during the search. ↩ ↩² ↩³
scikit-learn API reference, DummyRegressor: a regressor that makes predictions using simple rules, useful as a simple baseline; the default strategy “mean” always predicts the mean of the training set. ↩
scikit-learn, Common pitfalls and recommended practices, section on data leakage: the definition, the rule never to call fit on the test data, and the recommendation of the pipeline for cross-validation and hyperparameter tuning. ↩

Tree ensembles in plain terms

2025-11-25T00:00:00.000Z

I met decision trees, random forests and XGBoost one evening in a course an assistant was giving me on the scikit-learn workflow. A few months later I ran a random forest, XGBoost and gradient boosting myself, on my own computer in VS Code. One install line brought all of them:

pip install pandas numpy scikit-learn xgboost shap matplotlib seaborn

After that each model is a few lines of Python, so you can run all of them without knowing what they are. They are all built from the same piece, the decision tree.

A decision tree

A decision tree predicts a number by asking yes or no questions about the inputs, one after another. Say the first one is whether the temperature is above 800 C, and the next one is whether the feed is type A. Each answer sends the run down one branch, and at the end of the branches there is a leaf. The prediction for a run that lands in a leaf is the mean of the training runs that landed in the same leaf.11scikit-learn user guide, section 1.10: a tree as a piecewise constant approximation; little data preparation; overfitting, instability and the statement that trees “are not good at extrapolation”; leaf values as the node mean under squared error; the tips on max_depth and min_samples_leaf. It looks like a troubleshooting chart an operator would follow, only here the algorithm picks the questions and the thresholds, and it picks each one so the runs on either side are as alike in their result as possible.

The scikit-learn guide calls the result a piecewise constant approximation.11scikit-learn user guide, section 1.10: a tree as a piecewise constant approximation; little data preparation; overfitting, instability and the statement that trees “are not good at extrapolation”; leaf values as the node mean under squared error; the tips on max_depth and min_samples_leaf. If you plot the prediction against one input you get a staircase and never a smooth curve.

A tree assumes very little. It does not assume a straight line or any other form, and the guide says it needs little data preparation, where other methods often need the inputs normalized first.11scikit-learn user guide, section 1.10: a tree as a piecewise constant approximation; little data preparation; overfitting, instability and the statement that trees “are not good at extrapolation”; leaf values as the node mean under squared error; the tips on max_depth and min_samples_leaf. In scikit-learn it is DecisionTreeRegressor.

The settings control how far the sorting goes. max_depth is how many questions can come in a row, and min_samples_leaf is the smallest number of training runs allowed in a leaf. If you put no limit, the tree keeps splitting until every training run sits in its own leaf, so it reproduces the training data and that is overfitting. The guide suggests starting with a depth of 3.11scikit-learn user guide, section 1.10: a tree as a piecewise constant approximation; little data preparation; overfitting, instability and the statement that trees “are not good at extrapolation”; leaf values as the node mean under squared error; the tips on max_depth and min_samples_leaf. It also says a tree is unstable, meaning a small change in the data can give a completely different tree, and that the way around it is to use trees inside an ensemble.11scikit-learn user guide, section 1.10: a tree as a piecewise constant approximation; little data preparation; overfitting, instability and the statement that trees “are not good at extrapolation”; leaf values as the node mean under squared error; the tips on max_depth and min_samples_leaf. An ensemble is many models combined into one, and the next three are ensembles of trees.

A random forest

A random forest grows many trees and averages their predictions. Two things make the trees different from each other. Each tree is built on a bootstrap sample, which is a random draw of the training runs with replacement, so some runs appear twice and some do not appear. And at each split the tree may be allowed to look at only a random subset of the features. The guide says the purpose of both is to decrease the variance of the forest.22scikit-learn user guide, section 1.11: bootstrap samples and random feature subsets in random forests, to decrease variance; n_estimators and max_features; gradient boosting as an additive model fitted to negative gradients; learning_rate as shrinkage and its interaction with n_estimators; subsample.

I think of it like replicate analyses. One determination has scatter, and the mean of many has less, as long as they do not all carry the same error. Breiman’s paper says the error of a forest depends on how strong the individual trees are and on the correlation between them,33Breiman (2001), abstract: generalization error converges to a limit as the number of trees becomes large and depends on the strength of the individual trees and the correlation between them. so the randomness is there to make the trees less correlated.

The main settings are n_estimators, the number of trees, and max_features, the size of the random subset at each split.22scikit-learn user guide, section 1.11: bootstrap samples and random feature subsets in random forests, to decrease variance; n_estimators and max_features; gradient boosting as an additive model fitted to negative gradients; learning_rate as shrinkage and its interaction with n_estimators; subsample. Breiman showed that the error converges to a limit as you add trees,33Breiman (2001), abstract: generalization error converges to a limit as the number of trees becomes large and depends on the strength of the individual trees and the correlation between them. so more trees cost you computing time and do not overfit. In a tuning search I ran later, the random forest was one of the slow models to tune.

Gradient boosting

Boosting also adds trees together, but it builds them one after another. Each new tree is fitted to what the trees before it still get wrong. In Friedman’s method the new tree is fitted to the negative gradient of the loss, and for squared error that is the current residual, measured minus predicted.44Friedman (2001), abstract: function estimation as numerical optimization in function space, a general gradient descent boosting method for any fitting criterion, with specific algorithms for regression trees. 22scikit-learn user guide, section 1.11: bootstrap samples and random feature subsets in random forests, to decrease variance; n_estimators and max_features; gradient boosting as an additive model fitted to negative gradients; learning_rate as shrinkage and its interaction with n_estimators; subsample. To me it is like trimming out an offset in small moves until it is gone.

So the settings mean different things here. n_estimators is the number of correction steps. learning_rate is the fraction of each correction that is kept, and it is also called shrinkage. The guide says the two interact strongly, a smaller learning rate needs more trees, and small values favor better test error.22scikit-learn user guide, section 1.11: bootstrap samples and random feature subsets in random forests, to decrease variance; n_estimators and max_features; gradient boosting as an additive model fitted to negative gradients; learning_rate as shrinkage and its interaction with n_estimators; subsample. max_depth is the size of each small tree, and subsample fits each tree on a random fraction of the runs.22scikit-learn user guide, section 1.11: bootstrap samples and random feature subsets in random forests, to decrease variance; n_estimators and max_features; gradient boosting as an additive model fitted to negative gradients; learning_rate as shrinkage and its interaction with n_estimators; subsample.

Boosting cuts the error down step by step, and if you give it enough steps it goes on to fit the noise in the training runs. So in boosting the number of trees is a setting you have to tune, and the guide treats shrinkage as a way to control overfitting.22scikit-learn user guide, section 1.11: bootstrap samples and random feature subsets in random forests, to decrease variance; n_estimators and max_features; gradient boosting as an additive model fitted to negative gradients; learning_rate as shrinkage and its interaction with n_estimators; subsample. In scikit-learn it is GradientBoostingRegressor.

XGBoost

XGBoost is a separate library, which is why it has its own name in the install line, and the method inside it is gradient boosting of trees. Chen and Guestrin describe it as a scalable tree boosting system, and most of their paper is about speed on very large data, like handling sparse inputs and using memory and disk well.55Chen and Guestrin (2016), abstract and sections 2.1 and 2.3: the regularized objective penalizing the number of leaves and the leaf weights; shrinkage after Friedman and column subsampling as in random forests. I do not think any of that matters for a table of a few tens of rows.

What matters at any size is that XGBoost adds a penalty to the loss for how complex each tree is, counted by the number of leaves and the size of the leaf values. The authors say that with this term set to zero you are back to traditional gradient tree boosting.55Chen and Guestrin (2016), abstract and sections 2.1 and 2.3: the regularized objective penalizing the number of leaves and the leaf weights; shrinkage after Friedman and column subsampling as in random forests. It also uses shrinkage from Friedman and it subsamples the feature columns like a random forest does.55Chen and Guestrin (2016), abstract and sections 2.1 and 2.3: the regularized objective penalizing the number of leaves and the leaf weights; shrinkage after Friedman and column subsampling as in random forests. In Python the penalty is reg_lambda and gamma, the shrinkage is learning_rate and the column subsampling is colsample_bytree, next to the same n_estimators and max_depth as before.66XGBoost documentation, XGBoost parameters: eta (alias learning_rate), max_depth, colsample_bytree, lambda (alias reg_lambda) and gamma. The model is XGBRegressor.

Outside the training range

Every one of these models ends in leaves, and a leaf holds a constant that came from training runs. Say a tree was trained on runs between 600 and 900 C and you ask it about 1000 C. The last question about temperature that the tree learned has its threshold somewhere below 900 C. The 1000 C point answers that question the same way a 900 C run does, so it lands in the same leaf and gets the same prediction. The staircase is flat after its last step. The scikit-learn guide says it directly, tree predictions are piecewise constant and so they are not good at extrapolation.11scikit-learn user guide, section 1.10: a tree as a piecewise constant approximation; little data preparation; overfitting, instability and the statement that trees “are not good at extrapolation”; leaf values as the node mean under squared error; the tips on max_depth and min_samples_leaf.

Averaging does not change this, because an average of flat lines is flat, and a sum of small flat corrections is flat too. So a forest or a boosted model follows the trend up to the edge of the data and then holds the last value, and no setting fixes that. A straight line fit would at least keep its slope, right or wrong. A tree ensemble gives you the answer for the nearest condition it has seen, and it gives no sign that you asked outside its range. So before I believe a prediction from one of these, I compare the new condition with the minimum and maximum of each feature in the training table.

scikit-learn user guide, section 1.10: a tree as a piecewise constant approximation; little data preparation; overfitting, instability and the statement that trees “are not good at extrapolation”; leaf values as the node mean under squared error; the tips on max_depth and min_samples_leaf. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
scikit-learn user guide, section 1.11: bootstrap samples and random feature subsets in random forests, to decrease variance; n_estimators and max_features; gradient boosting as an additive model fitted to negative gradients; learning_rate as shrinkage and its interaction with n_estimators; subsample. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
Breiman (2001), abstract: generalization error converges to a limit as the number of trees becomes large and depends on the strength of the individual trees and the correlation between them. ↩ ↩²
Friedman (2001), abstract: function estimation as numerical optimization in function space, a general gradient descent boosting method for any fitting criterion, with specific algorithms for regression trees. ↩
Chen and Guestrin (2016), abstract and sections 2.1 and 2.3: the regularized objective penalizing the number of leaves and the leaf weights; shrinkage after Friedman and column subsampling as in random forests. ↩ ↩² ↩³
XGBoost documentation, XGBoost parameters: eta (alias learning_rate), max_depth, colsample_bytree, lambda (alias reg_lambda) and gamma. ↩

A small neural network in plain terms

2025-11-19T00:00:00.000Z

When I started on machine learning I had no understanding of neural networks, and I told the assistant so. About three months later a small neural network was one of the models I was fitting on my own computer.

The network I mean is a multilayer perceptron, MLP for short. It is the plainest kind of neural network, and it is the one scikit-learn gives you for a table of numbers, under the name MLPRegressor.

Units and layers

Start with one unit. A unit takes the numbers handed to it, multiplies each one by a weight, adds them up, and then passes the sum through a fixed curve called the activation function.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. A weight is like a fitted coefficient in a regression. It is a number the fitting adjusts.

Then the units are arranged in layers. The input layer has one unit for each input column, so if your table has a temperature and a reactor type, that is two units. The output layer gives the prediction, a yield for example. In between are the hidden layers, and each unit in a hidden layer takes its inputs from every unit in the layer before it.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. For regression the output unit is a plain weighted sum with no curve, and the fit minimizes the squared error, which is the same quantity a least squares line minimizes.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha.

The setting that fixes the shape is hidden_layer_sizes. The value (10, 10) means two hidden layers of ten units each. If you set nothing you get one layer of 100.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. The count of adjustable numbers grows fast. Say there are three inputs, two hidden layers of ten units and one output. Then I count 40 weights and offsets for the first hidden layer, 110 for the second and 11 for the output, so 161 in all. On a table of a few dozen runs, even a small network like this one has several times more adjustable numbers than there are rows.

This setting also gave me a bug. I had saved the layer sizes of tuned networks in a CSV as text like “10_10”. When the file was loaded again, the loader read that column as integers, so “10_10” became 1010 and “20_10” became 2010, and no error was thrown. I found it later, fixed the values and ran the predictions again.

The activation

Without the curve the layers would do nothing, because a weighted sum of weighted sums is still a weighted sum, so the whole network would be a linear model however many layers it had. The activation bends the output of each unit, and many bent pieces stacked together can follow a curved response, like a yield that rises with temperature and then levels off. In scikit-learn the choices are identity, logistic, tanh and relu, and relu is the default.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. Relu passes a positive sum through unchanged and sets a negative one to zero.

Training and epochs

The weights start as random numbers. Training shows the network the rows, measures the error, and works out for every weight which direction would reduce the error. Then the weights move a small step and the cycle repeats. The procedure that passes the error back through the layers is called backpropagation.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha.

An epoch is one pass through all the training rows. In scikit-learn the setting max_iter caps the number of passes at 200 by default.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. Training stops earlier if the error stops improving by more than a tolerance. If it reaches the cap first, the library prints a convergence warning. On one long job on my PC the terminal printed those warnings the whole time. As I understand it, the warning means the fit was stopped before it settled, and the usual answer is more iterations or a different solver. The documentation says the default solver, adam, works well on datasets with thousands of rows, and that for small datasets lbfgs can converge faster and perform better.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state.

Tuning a network is also slow, and the size of the table does not tell you that in advance. In my tuning search the MLP was one of the slow models.

Scaling the inputs

A furnace temperature runs to several hundred degrees. A reactor type coded as 0 or 1 never goes above one. If both go raw into a weighted sum, the temperature swamps the code. The user guide says the model is sensitive to feature scaling, so each input should be brought to a common range or standardized to mean 0 and variance 1, and the same scaling has to be applied to the rows held out for testing.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha.

The part that is easy to get wrong is where the scaling is learned. If the mean and the spread are computed on all the rows before validation, then the held-out row has already had a say in the model. So the guide recommends putting the scaler and the network in one pipeline,11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. and then each validation split learns its scaling from its own training rows.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPRegressor

model = make_pipeline(
    StandardScaler(),
    MLPRegressor(hidden_layer_sizes=(10, 10), max_iter=2000, random_state=0),
)

Why the fit changes from one start to the next

The user guide lists this as a disadvantage. A network with hidden layers has an error surface with more than one minimum, so different random starting weights can end in different fits with different validation scores.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. On a small table I think it is easy to see why. There are more adjustable numbers than rows, so many different sets of weights reproduce the training rows about equally well, and they do not agree with each other in between those rows. Which set you get depends on where the training started.

Setting random_state to a fixed integer makes one fit repeatable.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. That gives you the same start every time, and it says nothing about whether another start would have done better. The other handle is alpha, a penalty on large weights that defaults to 0.0001, and the guide suggests searching it over several orders of magnitude.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. 22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state.

scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸
MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

Starting Python and machine learning from zero with an assistant as the teacher

2025-11-19T00:00:00.000Z

I am a chemical engineer with a PhD, and when this started I had no coding. I had never written Python and I did not understand what a neural network was. I wanted to do research with machine learning, and with PINNs in particular. PINN stands for physics-informed neural network. It is a neural network where the governing equation of the problem, for example the rate equation of a reactor, goes into the loss, which is the number the training tries to make small.

So at 4 AM one night I asked Claude to teach me machine learning, AI and PINNs from the ground up, as a PhD chemical engineer new to all of it, with examples I can relate to.

It asked me three questions before it started. My answers were: no experience with Python, no understanding of neural networks, and I want both fundamentals and applications. Then it gave me three paths to choose from. Option A was foundation first for 2 to 3 weeks, option B was learn by doing from week 1, and option C was a parallel track. I chose option C, and the first install was going to be Anaconda.

The next day I asked for an online platform instead of a local install and went with Google Colab, which is a Python notebook that runs in the browser. I ran the first cell and saw output. Then I asked for a simple ODE problem, so my first small model was a PINN for a first-order reactor, built in parts: the problem, the network, the physics-informed loss, the training. Training completed and I plotted the results. Colab removes the install step entirely, and I think that is why the first working model took one evening.

A few days later, close to midnight, I started a proper course in a new chat. I asked again, a little differently: teach me the fundamentals and applications of machine learning and deep learning, as a PhD who did not study computer engineering and has done little or no coding, enough to do ML and PINN research. The plan it gave had five phases.

Phase	Time
Python	2 to 3 weeks
Mathematics	1 to 2 weeks
Classical ML	3 to 4 weeks
Deep learning	4 to 6 weeks
PINNs	3 to 4 weeks

We started with Python basics on Colab: variables, operations, lists, dictionaries, control flow and functions. Then I went straight on to NumPy the same night, which is the library for arrays, so indexing, operations on arrays, statistics, linear algebra, broadcasting and random numbers.

The next evening I did pandas, from series and dataframes through pivot tables and applying functions. Then matplotlib and seaborn for plots, subplots, error bars, heatmaps and pair plots. Then the scikit-learn workflow, which started with exploring and preprocessing the data and went on to linear regression, polynomial features, ridge and lasso, decision trees, random forests, XGBoost and support vector regression, and after that model comparison, cross-validation and hyperparameter tuning.

So in two evenings I had gone through the topics of the Python phase, which the plan gave 2 to 3 weeks, and I think a good part of what it called classical ML.

Starting Python twice as an engineer who had never programmed

2025-11-19T00:00:00.000Z

I have a PhD in chemical engineering and had done no programming when I asked Claude to teach me machine learning from the ground up, with examples I can relate to. It asked me a few things first, and my answers were that I had no experience with Python and no understanding of neural networks, and that I wanted both the fundamentals and the applications. The plan I picked had Anaconda as the first install. Anaconda is a large bundle of Python and scientific libraries. I did not install it then, so I cannot say anything for or against it.

In the browser first

The next day I asked for an online platform instead of a local install, and went with Google Colab. Colab calls itself a hosted Jupyter Notebook service that requires no setup.11Google, Colaboratory FAQ: “Colab is a hosted Jupyter Notebook service that requires no setup to use”; “Colab resources are not guaranteed and not unlimited”. A notebook is a page of boxes called cells. You type code in a cell, run it, and the result shows under it. I ran the first cell and saw output.

A few days later I started a proper course with Claude, close to midnight, and all of it was in Colab. The order was the language itself first (variables, operations, lists, dictionaries, control flow, functions), then NumPy for arrays. The next evening it was pandas for tables, then matplotlib and seaborn for plots, then the scikit-learn workflow for fitting models. If you think in spreadsheets, a pandas dataframe is close to a worksheet with named columns.

So in that first week nothing was installed on my computer, and I think that was right for me. The machine is Google’s, and they say the resources are not guaranteed and not unlimited,11Google, Colaboratory FAQ: “Colab is a hosted Jupyter Notebook service that requires no setup to use”; “Colab resources are not guaranteed and not unlimited”. but for learning the language that did not matter.

On my own computer

About two and a half months later I wanted to do a small analysis myself, step by step, in VS Code. It was public employment data on chemical engineers. I already had VS Code from the day before. Claude had built a first version of the analysis, and then I insisted on doing all of it on my own computer.

The setup was a fight. First I typed “matpilotlib” in the pip install line. Then pandas was not found inside the virtual environment I had made, and the message was:

Python was not found

Windows has a command named python that is only a shortcut to the Microsoft Store, and it was getting in front of the Python in my environment. The Python documentation for Windows now recommends the Python install manager, and it has a troubleshooting entry for this case that points to the “Manage app execution aliases” setting.22Python documentation, Using Python on Windows: the Python install manager, and the troubleshooting entry for python giving “command not found” or opening the Store app. I installed Python 3.14.3 through the install manager and added its folder to PATH. Then I could not remove the old environment folder because it was still in use. Then pip wanted an upgrade, from 25.3 to 26.0.1. After that it said “All good”, and I had a Jupyter server running on localhost:8888.

A virtual environment, or venv, is a folder that holds the libraries one project needs, kept apart from every other project.33Python documentation, venv: what a virtual environment is, the creation and activation commands, and the note on Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser. The reason is that two projects can need different versions of the same library, and one installation of Python holds only one of them.44The Python tutorial, chapter 12: applications that need different versions of the same library, and python -m pip install. It is like keeping a separate set of fittings for each skid, so work on one leaves the other alone. You make it with three lines in the terminal, typed in the project folder:

python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install pandas numpy scikit-learn matplotlib jupyter

The first line makes the environment, the second switches the terminal to it, and the third installs into it. On Windows the second line can be refused because PowerShell does not allow scripts by default. I had met that the day before with npm, where the message was “running scripts is disabled”, and the fix is the same one the venv documentation gives:33Python documentation, venv: what a virtual environment is, the creation and activation commands, and the note on Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser.

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Two days after that I moved another analysis from the browser to my own computer, because waiting for Claude to run it there was taking too long. It was the same steps again, a venv and a pip install, and then a script file. I switched to Jupyter when I saw it would be easier cell by cell.

The day after, I started in a new folder and ran the first script from a plain PowerShell window. It failed on a unicode escape in the script’s own docstring. Then it failed because pandas was not there, and then pip was not on the path in that window at all. So I went back to Jupyter in the venv, and told Claude to stop making me repeat my setup every time.

I think the pandas and pip failures were the same problem, that the plain PowerShell window was not using the Python my libraries were installed into. What I wrote down that night was to set up the venv in Jupyter from the start and skip the PowerShell detour.

The order I would use now

If I started again I think I would do it in the same order. First a hosted notebook, until the language, NumPy, pandas and plotting are familiar. Then on my own computer I would install Python itself from python.org, then VS Code, then make a project folder with a venv in it, then install the libraries with pip inside that venv, and Jupyter inside the same venv.

Git I would leave for later. I met it in that same local week, because the analysis was going to a public repository, and it took a .gitignore file, setting my name and email, and one commit. It is useful, but I did not need it to get a first plot.

I should also say that in both weeks Claude wrote most of the code. I ran it and asked what each part did. I am not sure how much of it I could have written from a blank page at that time.

Google, Colaboratory FAQ: “Colab is a hosted Jupyter Notebook service that requires no setup to use”; “Colab resources are not guaranteed and not unlimited”. ↩ ↩²
Python documentation, Using Python on Windows: the Python install manager, and the troubleshooting entry for python giving “command not found” or opening the Store app. ↩
Python documentation, venv: what a virtual environment is, the creation and activation commands, and the note on Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser. ↩ ↩²
The Python tutorial, chapter 12: applications that need different versions of the same library, and python -m pip install. ↩

My local model failed on its first long document because it was loaded with a 4096 token context

2025-11-16T00:00:00.000Z

I had a model running offline on my own computer in LM Studio, and I was wondering if a local model can learn from online resources. I asked an assistant, and the options were web search added on top of the local model, RAG over my own documents, or fine-tuning. RAG means a program searches your documents for the passages that match your question and hands those passages to the model along with the question, so the model itself is not changed. I said yes to that one.

So I enabled the local server in LM Studio, installed AnythingLLM, connected it to the server, made a workspace and uploaded research papers. The first long document failed with this:

Trying to keep the first 7280 tokens when context the overflows. However, the model is loaded with context length of only 4096 tokens.

The model was loaded with a context length of 4096 tokens, and the document needed 7280. The fix was to eject the model in LM Studio, reload it with a larger context length, and then update the AnythingLLM connection. I think context length is the first setting to raise before feeding a local model documents.

At that point I did not know what a token was. I asked two days later. A token is a piece of text the model counts in, often a whole short word or a part of a longer one, so a document has more tokens than words. Context length is how many tokens the model can hold at one time, and the question, the passages from the documents and the answer all have to fit in it.

Colab, Jupyter or VS Code, and which one for which job

2025-11-16T00:00:00.000Z

When I started Python I had never programmed, and the names Colab, Jupyter and VS Code kept coming up. I ended up using all of them, each for a different job. Four words come with them: notebook, cell, kernel and virtual environment. I understand them by things I already know as a chemical engineer.

Notebook, cell, kernel, virtual environment

A notebook is a document that keeps the code, the output of the code and your own notes on one page. On disk it is one file ending in .ipynb.11Jupyter Notebook documentation, “The Jupyter Notebook”. https://jupyter-notebook.readthedocs.io/en/stable/notebook.html The closest thing I know is a run sheet, where the instruction, the reading and the remark sit next to each other.

A cell is one box in that document. You write a few lines of code in it, you run it, and the result shows right under it.11Jupyter Notebook documentation, “The Jupyter Notebook”. https://jupyter-notebook.readthedocs.io/en/stable/notebook.html So a cell is like one step in a procedure. You do the step, you look at the reading, then you go to the next step.

The kernel is the thing that actually does the calculation. The page you see only sends the code of a cell to the kernel and shows what comes back.11Jupyter Notebook documentation, “The Jupyter Notebook”. https://jupyter-notebook.readthedocs.io/en/stable/notebook.html The Jupyter documentation describes kernels as separate processes that run on their own and talk to the notebook.22Jupyter documentation, “Kernels (programming languages)”. https://docs.jupyter.org/en/latest/projects/kernels.html I think of the page as the display and the kernel as the instrument behind it. The instrument keeps its state, so a data table you loaded in cell 3 is still in memory when you run cell 10. If you restart the kernel it is like switching the instrument off and on. Everything in memory is gone and you run the cells again, although the old outputs are still printed on the page.

A virtual environment, or venv, is a folder that holds one copy of Python and the libraries one project needs, kept apart from the other projects and from the Python of the computer itself.33Python 3 documentation, “venv, creation of virtual environments”. https://docs.python.org/3/library/venv.html It is like the toolbox you keep with one rig, so nothing gets borrowed for another job. The kernel of a notebook on your own computer runs from one of these folders, and that decides which libraries the notebook can see.

Colab for the first evening

For my first Python I asked for an online platform because I did not want to install anything, and I went with Google Colab. Colab is a notebook service hosted by Google that needs no setup, and the notebooks are stored in Google Drive.44Google, “Colaboratory frequently asked questions”. https://research.google.com/colaboratory/faq.html So the kernel runs on a machine at Google and you only need a browser. I ran the first cell and saw output, and the same evening I did a small tutorial problem on a first-order reactor in it. Later I did Python basics and NumPy in Colab too. Colab removes the install step entirely. The machine at Google is deleted after it sits idle for a while,44Google, “Colaboratory frequently asked questions”. https://research.google.com/colaboratory/faq.html so I would not leave a long run on it.

VS Code for the folder and the terminal

VS Code is a general editor. It shows a project folder and its files, and it has a terminal built in. I already had it for LaTeX. So when I wanted to do an analysis of public employment data on my own computer, step by step and by myself, VS Code was the place I already knew.

That evening was mostly a fight with the environment, a typo in a pip install and a Python that was not the one I meant, and when it was over the Jupyter server came up on localhost:8888, which is an address on your own computer and nothing on the internet.

VS Code can also open and run notebooks itself through its Jupyter extension.55Visual Studio Code documentation, “Jupyter Notebooks in VS Code”. https://code.visualstudio.com/docs/datascience/jupyter-notebooks I ran mine in a browser tab with VS Code beside it for the folder and the terminal.

Jupyter when I had to rerun the work myself

The next job was a modeling exercise. Claude was running it in the browser and it was taking too long, so I stopped it and moved the work to my own computer, one step at a time. I made a venv, installed the libraries, and started with a script, which is a single file that runs from top to bottom.

pip install pandas numpy scikit-learn xgboost shap matplotlib seaborn

Then I switched to Jupyter when I saw it would be easier cell by cell. With a script, if something fails near the end you run the whole file again. With a notebook the loaded data stays in the kernel, so you fix the one cell that failed and run only that. The next day I tried a script again in plain PowerShell and it failed because pandas and pip were not there outside the venv, so I went back to Jupyter in the venv. More than once at night I asked how to stop the notebook server and the venv.

What went wrong with long runs

One cell printed nothing for 25 minutes. I did not know if it was working, so I left it overnight, and I asked whether the PC going to sleep would stop it. When I checked the next evening it had finished, and no files had been written. All the results were only in the memory of the kernel, so I had them saved to a CSV right then.

Another long run died when I closed the browser by accident. The Jupyter documentation says the kernel stays active when the browser window is closed, and you can open the notebook again and reconnect.11Jupyter Notebook documentation, “The Jupyter Notebook”. https://jupyter-notebook.readthedocs.io/en/stable/notebook.html I do not know if my kernel was still running and I just did not reconnect.

Another time the PC went to sleep in the middle of a run. After the restart a cell gave a NameError on a variable that an earlier cell had defined, because the earlier cells had not been run again.

Jupyter Notebook documentation, “The Jupyter Notebook”. https://jupyter-notebook.readthedocs.io/en/stable/notebook.html ↩ ↩² ↩³ ↩⁴
Jupyter documentation, “Kernels (programming languages)”. https://docs.jupyter.org/en/latest/projects/kernels.html ↩
Python 3 documentation, “venv, creation of virtual environments”. https://docs.python.org/3/library/venv.html ↩
Google, “Colaboratory frequently asked questions”. https://research.google.com/colaboratory/faq.html ↩ ↩²
Visual Studio Code documentation, “Jupyter Notebooks in VS Code”. https://code.visualstudio.com/docs/datascience/jupyter-notebooks ↩

When a long chat hit its limit I had it write a handoff file for the next one

2025-11-09T00:00:00.000Z

I was putting together a package of documents with Claude, over more than one chat. One afternoon I picked it up again, so I asked Claude to retrieve where the last chat ended and deliver the complete set of documents for Word. It did that, with a summary.

Then the chat hit its length limit. A long chat fills up, and a new chat starts empty and cannot see the old one, so I was about to lose the context.

So before leaving it, I had Claude write a handoff file in markdown, meant for a fresh tab, with the instruction to look at the project files. A handoff note written at the end carries the work into the next chat.

Nine days later I did the same thing in a different chat, on a research idea. I had it write a summary of the conversation as a markdown file to continue elsewhere, then in another chat I asked for step by step instructions and started on step 1. That same day I had also asked Claude what tokens are, and got the explanation of tokens against words and what context length means.

Checking the numbers an assistant reads out of a long PDF

2025-11-06T00:00:00.000Z

The first evening I took Claude seriously for my profession, one of the things I gave it was a long technical document of my own. I uploaded the workbook of results that goes with it first, the spreadsheet where the runs are reduced to conditions and results, and then the document itself as a PDF. It saw more in the document than in the workbook. I think that is expected, because the workbook only holds the results, and the methods and the characterization are written in the document.

Then it gave me a package of documents built on it, and the figure it made much of was one measured value. I wrote that document, and I still have not opened it and checked that number against the page.

In the same month I asked what tokens are, because I did not know, and got the explanation of tokens against words and what context length means. The way I keep it in my head as a chemical engineer: tokens are the pieces of words the model works in, and the context window is a vessel of fixed volume. The whole PDF goes into that vessel together with everything else in the conversation. Anthropic’s documentation says each page of a PDF is processed as text and as an image, that a dense PDF with small fonts, complex tables or heavy graphics can fill the context window before the page limit is reached, and it suggests splitting a large document into sections.11Anthropic, “PDF support”. A long report full of tables is that kind of document. There is also a study by Liu and others where models had to find one relevant item in a long input, and they did best when the item was at the beginning or the end and worse when it was in the middle.22Liu et al. (2023), abstract. The results chapters of a long report are in the middle. So I cannot tell from the reply whether a number was read from the right row of the right table.

About three months later I went back to the same material with Claude, and this time I caught a few things. A block of the data was missing from the file it had consolidated, so I uploaded the source file for that block again. Then one component that matters in that work was missing from the charts it drew. For one table I gave it screenshots of the pages instead of the file.

A few days before that, in another project, I had asked for a literature review and for one paper to reproduce. The link in its citation pointed to the wrong paper, and I had already downloaded it on its word. I told it that it needs to learn to cite its claims correctly.

Anthropic’s guidance for long documents is to have the model pull out word for word quotes first, give a quote for each claim, and take back a claim when it cannot find one.33Anthropic, “Reduce hallucinations”. For a technical document I would write it like this.

For every number you report from this document, give the page,
the table or figure it comes from, and the sentence or table row
quoted word for word. If you cannot find a quote, say so and do
not report the number.

Anthropic, “PDF support”. ↩
Liu et al. (2023), abstract. ↩
Anthropic, “Reduce hallucinations”. ↩

tishad

Where I stand with AI, and where a reader might

Where I started

What changed my mind

What I did about the cost

What the figure shows

Working with an agent did not start overnight

Where that leaves me

What I still do not trust

For a reader who has not started

Footnotes

Linux beside Aspen Plus

Footnotes

The local AI stack

Private retrieval over my own papers

Footnotes

Telling a skill exactly which files it may read

Testing a skill with planted errors and a run without the skill

Footnotes

Prediction and interpretation are two different uses of a fitted model

Footnotes

A venv does not survive being copied to another machine

What the DOE generative AI policy asks of the person using the tool

Footnotes

The first chart to learn is predicted against actual

What a SHAP value says and what it does not

What a SHAP value is

Reading the beeswarm

What it does not tell you

Footnotes

Ask where the numbers came from before charting them

Footnotes

Telling an assistant to stop agreeing

Footnotes

Git and GitHub for one person

Git and GitHub are two different things

A commit

A branch

Push

.gitignore

Using it alone

Footnotes

Kernel models in plain terms

What a kernel is

Why the inputs are scaled first

Support vector regression

Gaussian process regression

Footnotes

Machine learning vocabulary in terms a process engineer already knows

The table

Holding runs back

Settings

Preparation and what is left over

Where the calibration comparison stops

Footnotes

Tree ensembles in plain terms

A decision tree

A random forest

Gradient boosting

XGBoost

Outside the training range

Footnotes

A small neural network in plain terms

Units and layers

The activation

Training and epochs

Scaling the inputs

Why the fit changes from one start to the next

Footnotes

Starting Python and machine learning from zero with an assistant as the teacher

Starting Python twice as an engineer who had never programmed

In the browser first

On my own computer

The order I would use now

Footnotes

My local model failed on its first long document because it was loaded with a 4096 token context

Colab, Jupyter or VS Code, and which one for which job

Notebook, cell, kernel, virtual environment

Colab for the first evening

VS Code for the folder and the terminal