Here are some questions that I’ve had and some answers I've found. They focus on the uv toolset of environment management and packaging. This post doesn’t say anything about setuptools, even though setuptools, setup.py, and setup.cfg are still present in a lot of well-built projects, especially pre-2025 ones.

I expect this list of questions to grow and evolve over time as I learn. My primary audience is a clueless future me, but I hope it helps you too.

How do I name projects? Packages? Modules? Repositories?

As the saying goes, naming things is one of the hardest problems in computer science. This is particularly true when it comes to packaging. There are a lot of different entities to be named, and it’s unclear sometimes what names belong to which. There is a repository name, a top level directory name, a project name, and a package name. All of these can be different. For extra confusion, they can get confused with module names and function names as well.

The Python interpreter and build tools have no problem knowing whether a particular name is supposed to reference the project or the package. They determine this from context. For human brains, especially the ones new to packaging, this can be a lot to keep track of. To save yourself unnecessary hassle, a good trick is to use the same name for all these things. A brief, memorable, all-lowercase name is ideal. The one way in which these names will differ is in how they handle multiword names. For project, repository, and top level directory names, separate words with a hyphen, as in my-amazing-tool. For Python packages and modules separate multiple words with an underscore, for example my_amazing_tool. This keeps things consistent with the conventions of the various tools and communities.

But don’t worry too much if you feel the need to deviate from this. Plenty of smart people have differing opinions and it’s a matter of convention only. The PEP 8 recommendation is to give packages single-word names, without underscores, but this can be challenging to do in a readable way. Everyone ignores this.

If you plan to distribute it publically on https://pypi.org, check it first to make sure the package name isn't already taken.

What are wheels and sdists?

An sdist is a source distribution and a wheel is a binary distribution. I have no idea why it's called a wheel. Source is the code itself—.py files and their supporting cast. It comes in a single .tar.gz archive which has to be unzipped with tar -xvf before it can be properly read. Detail on sdists here.

The wheel is the compiled version of the source, containing only the files needed to actually run the code. Because compilation is platform specific, a wheel is tied to a particular operating system, processor architecture, and Python version. A single project can have many wheels if it's meant to be run on many platforms and architectures. The big caveat here is that Python files don't get pre-built into binaries. The local Python interpreter does that at runtime. So if it's a purely Python package, then one wheel is usually sufficient for all platforms and OS's. Detail on wheels here.

What are build tools and why do they matter?

Build tools do the work of taking the original files and the information from pyproject.toml and using them as ingredients for building the sdist and the wheel.

There are two parts to this, a build frontend and a build backend. For the purposes of this post, the frontend is uv build it does some gathering and interpretation of the files and prepares them for the next step. pip and build are other popular build frontends.

There are a few common build backend tools, including hatchling, setuptools, and uv's own uv_build. Here's a short guide for choosing between them, but when in doubt hatchling is a good choice. The backend needs to be called out in pyproject.toml like this

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Here are some examples for the other backends as well.

Why have a src directory?

There are two common patterns for organizing projects flat and src layouts.

A flat layout is intuitive. The package code sits at the top level of the project and is more straightforward to access. But because of how imports work it's easy to lose track of whether your other code, like tests, are referencing the working copy of your code in the project or a previously installed version of the package. It can result in maddening bugs.

A src layout alleviates this. Because the package sits one level lower, it's not so easily reachable for direct import. Any import would need an installed version of the package. Using an editable install makes sure that your most recent changes to the code are what is run.

The src layout gives the benefit of protecting us from ourselves a bit more. And it comes at the cost of a slightly more complex file structure and an extra step to ensure an editable install.

How do I make a package visible across the project?

To make a package a visible from other locations in a project that are outside the package directories, for instance in tests/, the most reliable way is to do an editable install. From within the top level directory of the project run

uv pip install -e .

Modules outside your package shouldn't need __init__.py files in each directory. But now they will be able to import mypackage and go to town.

Note that it's also totally valid to include tests\ within the package. In that case they will very much need their __init__.py files. More detail in pytest best practices.

How do I add other file types to the package? And how do I access them from the code?

The easiest way is to include them under the package directory tree. By default, hatchling includes in the sdist any non-Python files under the mypackage directory that are not in .gitignore. This behavior can be aribitrarily modified for both types of build targets, sdists and wheels. Examples here. They can be instructed to include files from outside the project as well.

From within the code, these files can be accessed by their absolute path The __file__ attribute gives the absolute path of a given module. It can the be modified to point to the data instead. For example for this structure

myproject
├── pyproject.toml
└── src
    └── mypackage
        ├── __init__.py
        ├── mymodule.py
        └── data
            └── mydata.json

within mymodule

import os

mymodule_abspath = __file__
mydata_abspath = os.path.join(mymodule_abspath, 'data')
mydata_absfilename = os.path.join(mydata_abspath, 'mydata.json')
with open(mydata_absfilename, 'rt') as f:
    mydata = f.read()

What goes into pyproject.toml?

While pyproject.toml files are powerful and flexible and can be quite long, a minimal pyproject.toml contains just some basic project and build system information, like this.

[build-system]
requires = ['hatchling']
build-backend = "hatchling.build"

[project]
name = 'myproject'
version = '0.1.0'

There are some other commonly used fields for the project table, including description, license, authors, and keywords. project-dependencies will list other packages that yours depends on. The classifiers field gives a set of standard tags that can help humans and software tools alike make better use of your package. The full list of classifiers is lengthy, but some especially helpful ones are

• Development Status
• Intended Audience
• Topic
• Programming language

Resources

These are the references that I find most useful when answering these questions.

• python.org packaging
• uv project configuration
• build frontends
• build backends
• hatch configuration
• pyproject.toml configuration

It’s time to get more specific about what this ALM will do, what its inputs and outputs will be, so that I can actually start building it. It’s also time to define some tests and performance measures. I’ll need them when I have something running and I make a change; I’ll need some way to measure whether it got better.

Inputs

After the ALM has been trained, I’ll want to feed it text to proofread. To start with I’ll plan to do this through the simplest and clunkiest way I can think of: passing it the path to a text file containing the text to be proofread. In an actual professional product built for users, this is probably not ideal, but it’s a nice generalizable front door that slicker user interfaces can be built around in the future.

Outputs

After the proofreader has done its job and identified segments that might need correction, those segments will be reported as positions in the input text. Specifically, when the text file is read in as a string, the index of starting character position and ending character will be used to tag segments for inspection and correction. Position of the suspected error will be reported as a pair of indices. This collection of start/finish pairs will be the output of the proofreader.

There are a lot of other things that could be enhanced about this to provide a good user experience, and I leave the door open to add those later. For instance, the text file could be modified to include special characters marking the suspect segments. Or a fancier graphical UI could simply highlight the potential errors or underline them, as is common in word processors. An even fancier extension could propose corrections and offer the user a single keystroke way to select from a number of suggestions. But all of these could be built on top of a pair of start and end markers for each proofreading note.

Evaluation

I’ll need a way to evaluate the progress. If I try to enhance my proofreading model, how will I know if it worked? The variety of all possible text to be proofread is so large, it’s impossible to test every variation exhaustively. The best I can hope to do is come up with is a representative sample.

This falls somewhere in between the traditional software engineering practice of testing, where the notion of right and wrong answers and what a function must do is fairly clean cut, and bench marking, which is a consensus driven measure for comparing solutions in a broadly recognized way. It’s what has come to be known in LLM development as evaluations or, more affectionately, evals. Evals are a reasonable sample of the space in which a language model is expected to work. They lack the recognition and respect has a full-blown benchmark, and also lack the rigor and confidence of carefully designed tests. But despite having the worst of both worlds, evals operate in a space that we cannot ignore, and for which there is no better solution that I know of.

In practice, evals are organic. They grow to cover new use cases and new failure modes during the development process. But it’s helpful think through a reasonable initial set. For proofreading there are several classes of errors that are important to cover.

• Spelling. Febuary. Febrewary. Februry. Fabuwary.
• Word choice. Catching when it should be "imply" and when it should be "infer". To/two/too. There/their/they're.
• Punctuation. Appropriate sentence termination. Comma usage. Quotation mark usage.
• White space. Extra spaces. Spaces around punctuation. Weird indents.
• Capitalization.
• Grammar. Verb tense. Pronoun-antecedent agreement. Preposition choice.

These aren’t exhaustive, but there’s no need for evals to be exhaustive in order to be useful. I will almost certainly add more later as I discover new categories that aren’t getting picked up well. But they are a good place to start.

Creating evals for each category

In practice, to test how well a given language model performs in each of these areas, I’ll need to create an evaluation data set. For each of the areas above, I’ll pull five paragraphs arbitrarily from an evaluation text (Frankenstein by Mary Wollstonecraft (Godwin) Shelley) and throw 10 errors into the text of a given type. Having five paragraphs full of spelling errors gives a total of 50 spelling errors to detect. Each paragraph will come with its own answer key, the beginning and end of each word or phrase containing the error. After the proofreading model processes the paragraph, the errors it detects will be compared against the ground truth.

• A ground truth error that is overlapped by at least one model detected-error is considered detected (true positive). This is not quite the same thing as
• A model-detected error that overlaps at least one ground truth error. This is considered an accurate detection, but there may be several of these per ground truth error. I can't use this as the true positive count because it could result in inflated counts.
• A model-detected error that doesn’t overlap a ground truth error will be considered a false positive.
• A ground truth error that is not overlapped by at least one model detected error will be considered a false negative.

Recall will be defined as the total number of model-detected ground truth errors (true positives) over the total number of ground truth errors (true positives plus false negatives).

Accuracy will be a total number of model-detected ground truth errors (true positives) divided by the total number of true positives plus false positives. When there are no true positives or false positives, accuracy will be undefined.

Creating the evaluation dataset

Putting this into computer-readable form required creating a Python script with some error-ridden example text and the locations of the errors. I created the initial set of evals for spelling errors, but held off on creating evals for the other error types (punctiation, capitalization, etc.) for now. By the time you read this, there is a good chance it will already have evolved. If that's the case, you can find the latest version here. The evaluation dataset is a list of dicts, each of which contain a paragraph of text taken from a different chapter of Frankenstein which I modified to contain ten spelling mistakes. It also contains a list of ten dicts, each containing

• the mis-spelled word
• the index of its first and last character
• the corrected version of the word

Here's a snippet of the result

evaluation_dataset = [
    {
        'source': 'Frankenstein',
        'chapter': 'L1',
        'paragraph': """
I am already far north of London, and as I walk in the streets of
Petersburgh, I feel a cold northern breeze play upon my cheeks, which
braces my nerves and fills me with delight. Do you understand this
feeling? This breeze, which has travelled from the regions towards
which I am advancing, gives me a foretaste of those icey climes.
Inspirited by this wind of promise, my daydreams become more fervent
and vivid. I try in vain to be persuaded that the pole is the seat of
frost and desolation; it ever presents itself to my imagniation as the
region of beauty and delight. There, Margaret, the sun is for ever
visible, its broad disk just skirting the horizon and diffusing a
perpettual splendour. There—for with your leave, my sister, I will put
...
requisite; or by ascertaining the secret of the magnet, which, if at
all possible, can only be effected by an undertaking such as mine.
                """,
        'mistakes': [
            {
                'first_char': 322,
                'last_char': 325,
                'wrong_text': 'icey',
                'correct_text': 'icy',
            },
            {
                'first_char': 526,
                'last_char': 536,
                'wrong_text': 'imagniation',
                'correct_text': 'imagination',
            },
            {
                'first_char': 678,
                'last_char': 687,
                'wrong_text': 'perpettual',
                'correct_text': 'perpetual',
            },
            ...
        ]
    },
]

End-to-end prototyping

It’s fair to ask why I don’t go ahead and complete the evaluation datasets for the other types of errors. It seems logical to completely finish this step before moving onto the next. We can imagine this as a breadth-first solution to the problem, thoroughly working through one stage of development, putting some polish on it before moving to the next. When this work is spread across teams, this is called waterfall style development. One team completes a whole stage of the project like design or backend support before passing it onto the next.

The alternative to this is a depth-first development strategy. Building a bare bones end-to-end solution and then adding breadth and sophistication to it in subsequent passes. Starting with an end-to-end prototype means leaving a lot of things incomplete in the first pass. It means creating something that you would be embarrassed to show to your friends. If you’re working across multiple teams, it means a whole lot more communication up front.

In theory, both of these approaches are valid and will produce a good result in similar timeframes. But in practice, they don’t. The waterfall approach assumes that all of the work done at each stage gets to remain in its final form. In fact, every additional stage teaches us things we didn’t know about what needed to come before. This requires a lot of rework on stages that we had previously thought were complete. In an end-to-end prototyping approach this rework happens quickly. The whole project has a lot less momentum and can pivot more gracefully. It is more agile.

This lesson can take a long time to learn, and in some cases, it is in managers' interest to ignore it, depending on the incentives of the organization. But since I am all of the engineering teams and all of the levels of management for this project I get to decide: We’re going to start with a lightweight end-to-end prototype.

So now that the spelling evals are done, onto the next stage—building a model to detect misspelled words.

Tutorials, projects, code, and thoughts collected into topic groups I've generously called Book Projects.

Highlights

New Releases

• Being a Staff+ Data Scientist in 2026
• Build a custom tokenizer
• Artisanal Language Models

Most Visited

• Transformers from scratch
• Setting up an ssh server
• How to convert RGB color images to grayscale
• Convolution in one dimension

Most Loved

• Choose your professional path
• What to do when a leader does something wrong
• On microsuffering

I'm most proud of

• Solving an easy reinforcement learning problem on hard mode: Inverting a pendulum
• Naive Cartographer: A Markov Decision Process Learner
• Ziptie: Learning Useful Features

I've tried to survey the job description of data science a couple of times with varying degrees of success, most recently to go with some informal recommendations for creating data science degree programs. Together with a group of colleages we tried to summarize what data scientists do and the data science subtypes of maker, oracle, detective, generalist. But in the face of changing expectations this doesn't feel like enough anymore. It's time for a refresh.

A brief and biased history of the Data Scientist role

In the beginning...

The field of data science was named in 1997, and the discipline has existed by other names for a very long time. After all, people have been answering questions using data for thousands of years.

When data science first got huge, organizations expected data scientists to spin straw into gold—to transform unorganized data archives into profit. Big Data, it was believed, held inherent value, which only needed to be coaxed into cash form. This rarely panned out, so the approach evolved into a) data scientists produce "insights" and then b) "insights" generate profit. This also proved elusive in the end. Eventually data scientists settled into various niches involving answering questions using data, and some companies decided they didn't need as many data scientists as they had originally thought.

The Neural Network era

Then came the Neural Network revolution, where the machine learning hammer of choice became the convolutional neural network and many problems got recast as an image recognition problem. The software engineering, data engineering, and operations engineering that production CNNs required merited the job title of Machine Learning Engineer. Image recognition became synonymous with "modern machine learning". A data scientist's job description got blurry. Did it include ML Engineering as a subset? Should a good data scientist candidate have CNN experience? Every team took their own stance. No consensus emerged.

The age of Large Language Models

Then the spotlight abruptly shifted to Transformers and Large Language Models. With their massive scale, the engineering and operations requirements increased yet again. Only a small handful of enterprises are even capable of training such a model, and they accomplish this only with an army of engineers and an unfathomable amount of specialized computing power. LLMs are effectively black boxes. Their structure is known but their vast collection of parameters makes their behavior unpredictable and inexplicable. Using them is a matter of API calls, rather than training and evaluation. Most importantly, they are generative, rather than inductive. They don't actually answer questions, they create answer-looking things that are correct often enough to lull us into complacency.

LLMs are far enough removed from a core data scientist's skillset that their care and feeding isn't part of the data science job description. But they can still have a big effect on a data scientist's day. (More on that below.)

The next Big Thing

The next hot new trend has not yet emerged, but if the historical 5 year cycle holds true, it's due any day now.

What is "data science"?

I define data science as "answering questions using data". Also note that I am referring to lowercase-d data science here. This can include job titles of Data Scientist, Data Analyst, Quantitative Analyst, Data Engineer, and others. I'm not including data-fueled features like product recommendations or travel time estimates, which are typically the domain of Machine Learning Engineers due to their scale and latency requirements. These are typically dominated by a separate set of constraints, tools, and skills (although there is plenty of overlap).

The questions data scientsts get to wrestle with are varied, and they map closely to a company's org structure. Here are some greatest hits.

Product

• Personalization: Which one should I show you?
• Tip/donation recommendation: What suggestions should I give someone for how much to give?
• Product experience: Which pages do users visit? What buttons to they click? What features do they use? What does this tell us about how we can improve their experience?
• Demand forecasting: How many people will buy my product next year?

Operations

• Optimization: Which plan is the best? How can I minimize inventory?

Marketing

• Price optimization: How much should I charge this customer for this thing or service?
• Marketing Mix Modeling: What is the return on investment for each additional dollar spent in each of my marketing channels?

Finance

• Forecasting: What will our revenue and expenses be next quarter?

All organizations

• Decision support: Which choice should I make?
• Experimentation: Which version is better?

What about staff+ data scientists?

A well-defined technical problem, including all those listed above, are an excellent fit for the skills of a new data scientist or one with 3-5 years experience. These are quite challenging, but they are all challenges you can train for and practice on. They have fairly clear demands and success criteria.

Just like every other part of technology, the very hardest part is people. In data science, this takes the form of understanding what people want, setting their expectations, and coordinating misaligned or competing incentives. Being familiar with sophisticated tools and approaches is definitely valuable, but is often something that a data scientist with a couple years of experience can do an excellent job on. As a staff+ you can still expect to tackle some of the most challenging or time-sensitive technical tasks, but those will probably not be the toughest part of your job. It falls to staff+ data folks to navigate the uncharted and shifting terrain of stakeholder management, cross-team coordination, and communication. (I'm using "staff+" to refer to staff level and higher, typically folks who have been at it 5 to 7 years or more, although the actual time in the role varies widely.)

Most of these are struggles that data scientists have faced since the beginning. Often, one of the biggest contributions that a staff+ data scientist can make is to be a bridge between data science work and the rest of the org—including engineers, marketing, finance, product, and C-suite officers of all stripes. Staff+ data scientists are expected to take the ambiguity out of the message, for instance translating what a probability distribution means for reporting quarterly performance.

Here are the stickiest recurring topics I've been hearing about.

When stakeholders prefer a cheap, fast wrong answer to a good one

The biggest impact of AI (large language model) assistants on data science is the idea that anyone can query data with natural language Q-and-A. Sadly the reality of such systems is that they are trained on data that doesn't share the same set of quirks that your org is working with. It produces condifent, plausible answers 100% of the time, but they are accurate only 70% of the time, and the problem is that you can't know whether a particular answer is in the 30% until you dive in and re-create the analysis yourself.

There is a school of thought that being confident and fast is better than being cautious and correct. It has bled over from strategic leadership (where ambiguity is ubiquitous and one of the greatest risks is indecision) to analysis and engineering (where incaution can lead to loss of limb, life, and cash). And in most large organizations individual stakeholders don't get to feel the effects of being wrong. Those usually take time to materialize. So they can prioritize being fast and confident, which their AI-assisted analytics queries are all too happy to help them out with.

In almost every case, the overriding concern is not accuracy or rigor, but rather someone rationally pursuing what is best for them and their team. One of the hardest things a staff+ data scientist will ever have to do is build a mental model of these incentives and chart a course that successfully splits the difference between Scylla and Charybdis.

When stakeholders undervalue the skill and underestimate the time required

A closely related trend is a common assumption that data analysis has somehow gotten easier and faster, so much so that it is disposable. Randy Au calls this "data work in the fast fashion code era". It's no big deal to extract nuanced insights from your collected data, just feed it in to NotebookLM and ask, right? You should be able to have something by this afternoon right? Not the full analysis of course, but "rough numbers". Right?

I can't even come up with a rough number of the times I've had the conversation that "rough numbers" are very rough indeed. Not just off by a few percent, but maybe in the completely wrong direction. And there's no way to know for sure until you go back and do the careful numbers.

It doesn't help that questions that seem easy are actually hard to answer well. What caused this blip on this graph? Why didn't this feature boost engagement metrics? It seems natural that there should be simple answers to these questions, and obvious that someone familiar with the data should be able to pull them out quickly. So when a data scientist starts talking about the philosophical foundations of causality, and questioning whether we can really know whether anything causes anything, you can literally feel the eye-rolling of the product leader even if their camera is off.

I want it now

In a separate but related issue, stakeholders are sometimes reluctant to invest the time required to get high quality results. Bullish demands for unrealistic timelines have somehow become confused with strong leadership. Conveying the return on that time investment is a recurring challenge. How do you communicate the importance of carefully gathered data? the return on a carefully run experiment? the cost of accurately attributing a metric shift to a product change? How do you advocate for six-month-plus time investments in a company that never looks more than three months ahead? And heaven help you if you are trying to make a case for improving the robustness of your pipelines or pre-emptively cleaning your data.

Learning things from data can be expensive. A carefully constructed experiment takes time to plan and execute, particularly when data volumes are low. There is a certain philosophy amongst some leaders that everything should be experimented on, and than an experiment with a positive outcome should be required before any change is made, no matter how small. Communicating the opportunity cost of running an experiment is a regularly occurring challenge. Leaders also may be seeking to proactively defend themselves against challenges to a given decision by having receipts, a successful experiment to point back to if their judgment is questioned.

Real-time dashboards

One particular example of "I want it now" is common enough to call out on its own: The up-to-the-minute data dashboard. The sense of power it gives is intoxicating, so it's no surprise that it is such a commonly requested data product.

How do you communicate the cost of real time data availability? On the surface, it seems like a reasonable request. A leader looking at a dashboard is like a pilot in the cockpit of a fighter jet. They can see their instruments, look out the windows, and use all the information at their fingertips to make quick decisions and save the day. Of course they would want that information to be real time. If it were delayed, then they might miss critical opportunities and get shot down.

But an analytics dashboard is different than a fighter cockpit in more than one way. The world it’s representing doesn’t meaningfully change from one second to the next unless you’re doing high-speed trading. In most cases, it barely changes from one day to the next. When that’s the case, real time updates feel useful, but don’t deliver any actionable information.

More importantly, the decisions that get made based on those dashboards don’t get made minute-to-minute or even hour-to-hour. They are typically decisions that factor into quarterly planning or sprint planning—decisions that occur every few months or weeks. Maybe every few days. Because of that, having the dash more update frequently this does not drive better decisions.

The engineering effort required to go from nightly updates to few-second latency are considerable. Nightly updates, or even hourly, can be done by some DAG-based pipeline orchestror. It operates on tables and produces tables, which are easy to read and write to in code. Real time updates involve using stream technology, like Kafka or Flink. These are amazing when you need them, but they have many more moving parts, more things that can break, more things that you have to keep an eye on, more things that can cause the pipeline to go down and the dashboard to get wonky and require laborious backfills and carefully worded responses to frustrated questions from leaders about why their nerve center dashboard suite has gone down. To operate at the same reliability, it might require 3 to 5 times the effort and dollars.

This is a conversation that most staff+ data scientists end up having at least once in their careers. And they usually lose.

The perrenial promise of self-serve

Every data science organization I've worked in has gone through this cycle:

1. Data scientists generate useful results
2. Stakeholders find them valuable
3. Stakeholders ask for more such results, with increasing frequency
4. Data scientists get tired of running similar queries over and over
5. A "self-serve analytics" function is proposed

I've never seen this approach solve the original problem of getting stakeholders all the information they need without burdening the data scientists. One of these things happens instead.

• a basic self-serve system is fielded, which inevitably leads to follow-up questions outside of its scope. Data scientists are answering more questions than ever.
• a complex self-serve system is fielded, which requires stakeholders to learn a querying language, like SQL or a simplified version of it. They don't, and data scientists remain in the role of human user interface.
• data scientists get deep into building an intuitive, highly capable self-serve system. The project scope is large and occupies all of their attention and is never quite finished. They aren't available to field query questions of any sort.

Communicating uncertainty

Statistics, the native language of the data scientist, is all about distributions. But decisions get made based on concrete values. A business decision maker may have a rule of thumb in their head like “If the cost is less than three dollars, buy it, otherwise pass”. So they ask a data scientist, how much it will cost.

DS: About three and a half dollars.

BDM: What do you mean "about"? What will the actual price be?

DS: Between two and five dollars.

BDM: That’s a huge range. But you’re saying it definitely won’t be less than two dollars? And definitely not more than five?

DS: Well, no, it might be less than two or more than five. But it probably won’t.

BDM: Reaches for magic eight ball

Translating from the fuzzy smear of a probability distribution to concrete values to support decision making is one of the hardest things a data scientist has to do. The two representations are fundamentally mismatched, and the mental models required to reason about them are nearly irreconcilable.

If your audience is familiar with gambling, this gives some useful footholds like over-under and odds ratios. Even better if they are familiar with rolling 20-sided dice. You can also try using percentages, statistical significance, confidence intervals, and upper/lower bounds, and see which you have the greatest success with.

The more senior you get as a data scientist, the more of these conversations you end up having, talking with people who haven’t spent years building mental models of distributions. This conversation is a recurring one. Some version of it occurs with every analysis, and every decision. Bridging this gap well is what lets the hard work of a data science team carry maximum weight in the rest of the company. Learning to do it well is well worth it.

Building within-team consensus on how things are done

Closely related is helping data science teams communicate about uncertainty consistently within themselves. New career data scientists are usually taught frequentist null hypothesis significance testing. The p < .05 threshold is drilled into them as an axiom. As a rule of thumb it’s useful, but as an iron law, it is limiting. Another very useful thing a staff+ data scientist can do is help the rest of the team look past the p-value and consider the context of the decision. Talking through the cost of false positives and false negatives, the likely distribution of classes in practice, the opportunity cost of running long experiments in order to reach significance, alternative ways of evaluating distributions for decision-making. Helping the whole team to level up and to speak the same language naturally falls on the staff+ data scientist as a technical leader.

This work extends to coordinating definitions, tools, practices, and decision criteria more generally. There is tension between the scientific freedom of allowing everyone to use the analyses, statistical tools, modeling techniciques, presentation formats, and feature definitions that they prefer, and the chaos that brings about. The larger the data science team, the greater the chaos. Setting norms for these, particularly across multiple organizations, is a big undertaking and difficult to do in a way that doesn't feel heavy handed. Staff+ data scientists that are able to pull this off are rare but their impact is huge.

Collaboration with partner teams

Data scientists work as a specialized piece of a larger machine. They work with data analysts to define data models and metric definitions. They work with data engineers to establish availability, freshness, data types, and standard transformations. They work with infrastructure engineers to make sure the data they need is accessible at reasonable latencies. They work with frontend engineers to make sure important events are instrumented and logged. They work with backend engineers to make sure important events are logged there as well and sometimes to pre-compute expensive features.

Data scientists are part of an interconnected web and can only do what they do because of the other work that goes on around them. Coordinating with these teams, supporting them in turn, keeping those communication lines open and trust high.

Designing a data science organization: Centralized vs Distributed

At a certain level of seniority in a small company, you may be asked to help grow a data science team from scratch. Once the product team and the marketing team and the finance team all see how useful a data scientist can be, they'll each want their own team. A budding data science manager may prefer to keep the team together under one umbrella, perhaps in the same org as the analytics or the data engineering team. It is a recurring dilemma of central vs distributed data science teams.

There is no right answer. Either can work if the organization is healthy and incentives are properly aligned. If not, than neither will work well. But all else being equal, a central data science team that works closely with other teams is the easiest to pull off. It's good for getting data scientists the support they need and building capabilities that are applicable across the company. It's natural to morph into a matrixed arrangement from there, where a data scientist reports to their DS manager, but attends a lot of team meetings with the specific team they are embedded with.

Delivering Disappointing Results

Most leaders don’t like it when numbers disagree with their intuition or expectations. As much as they profess to be data-driven, more often than not they seek to be data-validated. If you get the answer they expect, they will accept and maybe celebrate it. If you come up with something surprising, disappointing, or uncomplimentary, they may push back, question assumptions, ask you to revisit it, suggest changes to the approach, or simply reject the result.

There's no one size fits all answer to this, but here are some things to watch out for.

• Don't assume that the stakeholder will be persuaded by a rigorous and watertight analysis. They are balancing a lot of antagonistic concerns. Correctness is just one of them.
• Don't assume that the stakeholder wants to know that real answer. Sometimes the incentives to believe a difference answer are just too compelling.
• Don't assume that the stakeholder doesn't want to know the real answer. Pushback and follow up questions are a natural due diligence step. When a result is surprising and will require a lot of work to acknowledge, some questioning is appropriate.
• Don't assume the stakeholder is questioning your capabilities, motives, or integrity.

Processing difficult analyses is a conversation, not a homework assignment where you get to stick it to the professor. It works best when you approach the leader as another human being who has their own biases, fears, and goals. Talk it through. Expect to walk them through the numbers, re-run them, examing your assumptions, and run some follow up analyses. More times than I care to admit, these pushback sessions have exposed my blindspots and resulted in more nuanced conclusions. At the very least, give the stakeholders space to mourn their lost hopes.

This list isn't exhaustive of course. Every company and team has its own personality. But it gives a general flavor of the the life of a staff+ data scientist in 2026. Have I sold you on it?

The good bits are very very good

With all of these sticky wickets, you might be wondering whether a career in data science is even worth it. What I haven’t mentioned yet is the upside, the marvelous feeling of being in the zone when your team is working together with other teams in synchrony, bridging the chasm between ones and zeros and the biggest decisions in the company. At the staff+ level, this is very similar to the Zen of a software engineer or a machine learning engineer who is in the zone, for all the same reasons. There’s nothing quite like it. It’s a satisfaction you feel somewhere deeply, somehwere just behind your sternum. You'll have to answer this question for yourself, but for me, yes, it’s worth it.

It covers all the basic concepts in a beginner-friendly way. The cool part is that most of them aren't too complex, once you strip away the fancy names. when you're done, you'll know the difference between strong consistency and eventual consistency, between a flat schema and a star schema, between OLAP and OLTP, and you'll be able to explain what makes a data warehouse, a data catalog, a data lake, and a data mart.

I wrote a step by step walkthrough of the re-assembly of a transformer starting from its tiniest bits. No machine learning background required. If you like looking under the hood, this is for you.

The tokenizer is the first heavy duty processing that input data gets exposed to when it churns through an LLM. The tokenizer takes the text or audio or image and breaks it down into a sequence of bite-sized pieces called tokens.

In theory, an LLM could operate on any discrete piece of information, even fine-grained ones like characters, pixels, and bytes, but in practice these are too small to carry much information individually and working with them directly puts a greater burden on the model to try to combine them and make sense of them.

To give the models a leg up, the tiny pieces of input get pre-combined into more useful bits called tokens. For instance, when processing a string of English text like How much wood could a woodchuck chuck?, it might be tokenized into How much, wood, could, a wood, chuck, chuck, ?. Tokens don't fall strictly on word boundaries and can contain more than one word. Handling 7 tokens instead of 38 characters makes life a lot easier for the stages that follow. Internally, each of these word pieces gets assigned a number for efficient handling, so this sentence ends up looking like [6387, 73, 593, 5365, 837, 837, 24].

Tokenization is an example of automated feature engineering, also known as feature learning or feature discovery. It is understood among machine learning practitioners that once you get a good set of features, you're 80% of the way to solving your problem. It is an important step to get right, and has a huge impact on the quality of results.

An advantage to an Artisanal Language Model with a curated data set is that it gives an opportunity to create a model-specific tokenizer. The representation of input data can be tailored to the needs of your specific problem, instead of having to create something that could potentially cover every conceivable question someone could ask and task they could propose.

Vocabulary

The collection of tokens that a tokenizer works with is called its vocabulary. The tokenizers used by popular LLMs have vocabularies of 100,000 or more different tokens.

The size of the vocabluary is an important design parameter that has a big effect on how big the model needs to be and how well it can perform. A larger vocabulary means that word chunks and other pieces of input can be bigger, saving the model the trouble of splicing them together. It means that a given sequence of tokens can represent a longer history of inputs. But it also means that the model has a larger number of relationships to consider. Any given input token might be predictive of any given output token, making the size of the that space O(N^2)—something that grows as the square of the the number of tokens. And when you add in the fact that it's not just the current token, but the whole context history of tokens, that complexity goes up further. (Attention is the magic trick that keeps it from exploding catastrophically. More on that later.)

The flip side of this relationship is that reducing a vocabulary to 10% of its original size means that the demands on the model are reduced by far more than that, down to something like 1% or 0.1%. The amount of training data needed, the computation required, the inference time, all get reduced. The smaller the vocabulary, the less time and expense to train and use a model.

Having an ALM focused on a single task with a curated data set makes this possible. The more targeted the data set, the fewer the tokens needed to represent it well.

All the numbers I mention above are very hand wavy because LLMs are so very expensive to train and evaluate. I'm not aware of any research published about the exact nature of the relationship between vocabulary size, number of model parameters, and model performance. A smaller ALM that is feasible to re-train a number of times make this investigation approachable. It means that the tokenizer for a given ALM can be trial-and-error optimized for its intended usage.

What if an LLM were trained to complete a specific task?

LLMs are general purpose tools, trained to be able to do many different things. As a result they are vast. And they are better at doing some things than others.

If an LLM were instead focused on a particular task, trained to perform it as well as possible and ignore everything else, then it would have a better chance at being excellent at that thing, rather than just being OK at everything.

The structure around the model, the preprocessing of the input data and postprocessing out the output data, could all be specially built to give the best results on a single task. Whether for document retrieval or spell checking or autocomplete or interactive chat, whether for text or for photos or for music, an Artisanal Language Model could be focused on doing one thing and doing it well.

What if we could train an LLM on a data set we collected ourselves?

LLMs are trained on practically the whole internet.

The downside on such a broad swath of training data is that LLMs are also trained on every stupid thing someone decided to post in a drunken rant. ML engineers try to correct for this with post-processing filter steps and prescriptive prompts, but there's no way to completely eliminate idiocy from the model once it's been trained in.

With a set of curated training data, every inclusion is a deliberate decision, and garbage-in-garbage-out becomes less of a problem. Curated training data would also avoid the many legal and ethical viiolations called out in the training sets of popular LLMs, including copyright violation, license violation, nonconsensual sharing of personal images, and inclusion of CSAM.

Another advantage of an ALM focused on a specific task is that it can limit its training data to what is relevant to the problem at hand. If building a Javascript code completion model, we wouldn't need to include large amounts of prose every language of the world, or even code in other computer languages. We wouldn't need to include catalogs of audio recordings or millions of images. We could limit the training data to Javascript, making it orders of magnitude smaller.

With smaller training data sets, training itself becomes more feasible. For LLMs we can count on one hand the companies large enough to train their own general purpose models from scratch. The investment in data engineering and computation is huge. But for a training data set that is 0.01% of the size, the training time and cost come down within the reach of many more organizations, researchers, and hobbyists.

What if an LLM could be trained on CPUs?

Training a modern LLM requires centuries of GPU time. This gets spread across many thousands of GPUs so that it completes in a reasonable amount of time, but it remains massive. And it results in huge hardware and power bills.

If a much smaller model were capable of being trained on CPU only, even if it required many of them, it would remove a huge barrier to language model training. The rate of experimentation, innovation, diversification, and specialization would increase tremendously.

What if an LLM could run on your laptop?

The current generation of LLMs require clusters of GPUs for inference. Even after they are trained, they require too much computation and hardware to sit comfortably in anything but a data center.

If there were language models small enough to run on a laptop and nimble enough to return inference results quickly on laptop hardware, their deployment could be made more robust. No network latency, no reliance on connectivity, no dependence on external services' uptime. The reliability engineering would become considerably more straightforward and the expectations of availability could be raised.

This also opens the way for proprietary models and applications requiring high security. The ability to run isolated from an external network opens up new domains.

What if an LLM could continue to learn as you use it?

Current LLMs are too unwieldy to be updated on the fly. While there are ways to refine them, such as fine tuning or reinforcement learning from human feedback, these are either tweaks to a small part of the model or post-processing steps. They aren't capable of updating the model as a whole.

If it were possible to update the model based on every interaction, every new bit of input data and user response, that would make it possible for the model to get better over time in a meaningful way. And most importantly, it would continue to adapt to the specific users, tasks, and input data it was exposed to. It would learn the parameters of its job under the continuous mentorship of its human users.

What if we had Artisanal Language Models?

I hope to find out.

Graffiti Wall ( frontend code, backend code)

It's the latest step on my journey, stitching together everything I've been learning about self-hosting, deploying a WSGI server, and backing it with a Postgres database.

Please enjoy it!

Please enjoy it!