Ask where the numbers came from before charting them

I wanted to write a one page article on how chemical engineering graduates in the US spread across industries and roles during their careers, traditional and non traditional, to start a conversation with other chemical engineers. I asked Claude for the analysis and it built a first one. Then I insisted on doing all of it myself in VS Code, step by step, in a Jupyter notebook.

Getting Python and Jupyter running took a while. Once they ran, the first cells failed with KeyError: 'year'. The code was calling the API of the Bureau of Labor Statistics (BLS), and the API had returned 0 data points because the series ID was wrong, so there was no year to work with. Then I asked Claude whether it had made up the data in cell 3. The first data was fabricated. So that evening we moved to pulling everything from the agencies, and since then I ask where the numbers came from before I chart them.

The employment and wage numbers come from the Occupational Employment and Wage Statistics tables of the BLS, which are published for each year as a zip file with spreadsheets inside.11BLS, OEWS tables page: national, state, area and industry files for each reference year, as zipped XLSX files from 2014 onward. Chemical engineers are occupation code 17-2041. The zip downloads failed with 403 at first, and then all years came back 200. Then the years 2014 to 2018 came out wrong, because the code was reading field_descriptions.xlsx from each zip instead of national_M20xx_dl.xlsx, and that was fixed by picking the right file. After that I had a real series for 2014 to 2024, saved to data/bls_cheme_timeseries.csv.

The second source was Data USA, which serves Census data through its own API.22Data USA API documentation. Every documented form of the request gave 404, even the example from their own docs, and I found the working endpoint myself. From it I got the number of people in the occupation and their average wage for 2014 to 2023. I also tried to get what jobs people with a chemical engineering degree actually hold. The server gave a 500, then an empty result at one level, and the next level did not exist in that dataset, so I gave up on that and used the data from the extracted zips instead.

I added the same series for mechanical, electrical, civil and computer hardware engineers, software developers and all occupations, because I think scientific data has to be presented against something, or the chart means nothing.

Pulling the data myself took the evening. In those numbers, BLS employment of chemical engineers fell from 33,470 in 2014 to 20,330 in 2024, while the Census count of people in the occupation rose from 56,387 in 2014 to 84,824 in 2023. I think that means graduates are spreading into other roles.

Footnotes

1. BLS, OEWS tables page: national, state, area and industry files for each reference year, as zipped XLSX files from 2014 onward. ↩

2. Data USA API documentation. ↩