University of Washington · eScience Institute

Generative-AI
for Software
Development

AI-in-Practice Summer Institute

Anthony Arendt & Don (Landung) Setiawan July 29, 2026

Many of us are using Generative AI for code development.

However, there are concerns about using gen-AI for scientific code. How do we ensure our results are robust and reproducible?

Today we will take a messy existing Jupyter notebook and apply some generative AI methods to demonstrate a few best practices.

Working with an agent in a codebase

We will regard the agent as a coworker who helps us review and improve existing code, rather than building from scratch.

Differences Between Chat and Agent

Chat

Answers
questions.

Agent

Reads your files,
runs your code,
edits your project.

What is a harness?

A harness is the surrounding system — the file access, tool permissions, and execution environment wrapped around the model. Saying "I'm using Claude" is under-specified: browser Claude, Claude Code, and Claude in VS Code all share the same underlying model but have completely different harnesses. The harness is what has changed most in the last two years, and it determines what AI can actually do on your behalf.

LLM backbone

The language model itself — the reasoning engine that turns prompts into text, code, and plans. Every AI tool you use is built on top of one.

Tool use

The ability to read files, edit them, and run shell commands on your machine. Without tool use the model can only talk; with it the model can act.

Agent loop

The software that decides whether to call another tool or return a result to you. It turns a single prompt into a sequence of goal-directed actions.

Project memory

A file the agent reads on startup so it knows your codebase, conventions, and working style. You write it once; every session starts with that context already loaded.

MCP servers

A standard protocol for connecting external systems — databases, calendars, APIs — directly to an agent. Any service that speaks MCP becomes immediately available to any tool that supports the standard.

Skills

Reusable procedures the agent can invoke for a specific kind of task. Instead of re-explaining how to deploy or review code each time, you capture that workflow once.

LLM backbone

The language model itself — the reasoning engine that turns prompts into text, code, and plans. Every AI tool you use is built on top of one.

Tool use

The ability to read files, edit them, and run shell commands on your machine. Without tool use the model can only talk; with it the model can act.

Agent loop

The software that decides whether to call another tool or return a result to you. It turns a single prompt into a sequence of goal-directed actions.

Project memory

A file the agent reads on startup so it knows your codebase, conventions, and working style. You write it once; every session starts with that context already loaded.

MCP servers

A standard protocol for connecting external systems — databases, calendars, APIs — directly to an agent. Any service that speaks MCP becomes immediately available to any tool that supports the standard.

Skills

Reusable procedures the agent can invoke for a specific kind of task. Instead of re-explaining how to deploy or review code each time, you capture that workflow once.

Built-in knowledge

Python and common libraries — the model has seen millions of lines of code and understands standard patterns, idioms, and APIs.
General best practices — it knows how to structure a function, write a test, or refactor a loop without being told.
When to stop — it has a sense of when a task is complete or when it needs more information before acting.

You don't control this — it is fixed at training time and has a knowledge cutoff date.

In the prompt

Your data's quirks — the model has never seen your CSV headers, your variable naming conventions, or your domain vocabulary.
Your lab's conventions — style guides, preferred libraries, and project structure exist nowhere in its training data.
Your specific goal — what "done" looks like for this task is something only you can define.

You control this!

We will be refactoring a messy notebook that processes weather station data.

Refactoring Reorganizing working code without changing what it does — same inputs, same outputs, better shape.

Step 1 · Live demo

Asking for a Plan to Improve my Code

Live demo

AGENTS.md —
ten lines that change
how the agent behaves.

What What this project is

Stack Preferred libraries

        Key
        Timestamps are UTC unless stated
      

        Key
        Don't add type hints unless asked
      

Why Version-controlled — team asset, not personal preference

Why Cross-vendor — the file travels with the code

What just happened

The agent read your files,
followed your plan,
and edited your code.

01

We switched to agent mode, prompted it to extract the cleaning logic into a module, and it did exactly that.

02

The chat tells you what the agent claims it changed. The diff is where you confirm it.

03

The code looks right. But we should verify it to be sure.

What just happened

The agent preserved
the bug faithfully.

The bug was in the original notebook. The refactor moved the pandas operations verbatim. It didn't introduce it, but it also didn't catch it.
Summary statistics were unaffected. Only your domain-informed look at the diurnal plot surfaced it.
The AGENTS.md convention we wrote earlier (timestamps are UTC) helped the agent find the fix quickly.
This is the shape of the collaboration: the agent handles the code and you handle the domain.

Act 3 — Verification

Why do we need
to verify our code?

This is not a question of whether the agent is trustworthy. It generally produces plausible, working code.
However, the agent does not know the specifics of your data: its quirks, its edge cases, its domain context.
It also lacks the scientific domain knowledge you bring as a researcher. It cannot judge whether a result is physically reasonable or scientifically meaningful.
We regard verification as a core responsibility of the human researcher.

Verification

There are multiple approaches.
Today we used two.

Read the diff, not the chat. The chat tells you what the agent claims it changed. The diff is where you confirm it.
Push back. The agent is a coworker, not an oracle. "No, you're in the wrong mode" and "no, you did more than I asked" are both valid pushback. Half of what looks like the agent going rogue is a permissions or scope problem.

There are more advanced patterns — evals, test-first workflows, structured uncertainty flagging. The handout has starting points if you want to go deeper.

Reproducibility

The conversation is scaffolding but
The artifact is the science.

LLMs are stochastic, but science demands reproducibility.
The resolution: the process is not reproducible, however the artifacts are.
Your git repo, not your chat log, is the record.
You can commit plans, review notes, and validation outputs alongside code. git log docs/ reads like a lab notebook.
Log the model, version, and date in your methods section, the way you would cite any tool.
Pin your dependencies. AI-generated code often assumes packages it did not declare.

Your next step

Write an AGENTS.md
for one of your own
projects.

Pick a project you are actively working on.
Ten lines is enough: conventions, preferences, what not to do.
Commit it to the repo so it travels with the code.
Then use an agent on that project once, and notice what changed.

Four Durable Places to Shape Agent Behavior

01

Context
documents

Always loaded on startup. Facts and conventions the agent has in hand for this project.

AGENTS.md

02

Skills

Task-triggered procedures. The agent picks them up when the work matches.

skills/

03

Rules

Path-scoped constraints that are hard limits, not helpful suggestions.

.claude/

04

Custom
agents

Distinct personas with their own tool access. For focused, narrower jobs.

agents/

Taxonomy is portable across different provider's tools. The file names differ, but the concepts don't.

Generative-AI
for Software
Development

Working with an agent in a codebase

Differences Between Chat and Agent

Every AI coding assistant has all of these six components.

Every AI coding assistant has all of these six components.

Built-in knowledge vs. In the prompt.

We will be refactoring a messy notebook that processes weather station data.

Asking for a Plan to Improve my Code

AGENTS.md —
ten lines that change
how the agent behaves.

The agent read your files,
followed your plan,
and edited your code.

The agent preserved
the bug faithfully.

Why do we need
to verify our code?

There are multiple approaches.
Today we used two.

The conversation is scaffolding but
The artifact is the science.

Write an AGENTS.md
for one of your own
projects.

Four Durable Places to Shape Agent Behavior

Generative-AIfor SoftwareDevelopment

Working with an agent in a codebase

Differences Between Chat and Agent

Every AI coding assistant has all of these six components.

Every AI coding assistant has all of these six components.

Built-in knowledge vs. In the prompt.

We will be refactoring a messy notebook that processes weather station data.

Asking for a Plan to Improve my Code

AGENTS.md —ten lines that changehow the agent behaves.

The agent read your files,followed your plan,and edited your code.

The agent preservedthe bug faithfully.

Why do we needto verify our code?

There are multiple approaches.Today we used two.

The conversation is scaffolding butThe artifact is the science.

Write an AGENTS.mdfor one of your ownprojects.

Four Durable Places to Shape Agent Behavior

Generative-AI
for Software
Development

AGENTS.md —
ten lines that change
how the agent behaves.

The agent read your files,
followed your plan,
and edited your code.

The agent preserved
the bug faithfully.

Why do we need
to verify our code?

There are multiple approaches.
Today we used two.

The conversation is scaffolding but
The artifact is the science.

Write an AGENTS.md
for one of your own
projects.