Prompt, Pray, Repeat

This post seem long? Don't worry: It's mostly code details that you can skim.

In the last post for AI Code Correct, I generated a solution for the name normalizer, which formats a person’s name as Last, First M. The set of CAX examples used to create the function:

Mononym:
Plato -> Plato

Duonym:
Clarence Ellis -> Ellis, Clarence

Trims spaces:
" \n\n Alan \n\t\r\v\fMathison Turing " -> Turing, Alan M.

Initializes middle name:
Donald Ervin Knuth -> Knuth, Donald E.

Initializes multiple middle names:
Grace Brewster Murray Hopper -> Hopper, Grace B. M.

Retains suffixes (appearing after comma)
Martin Luther King, Jr. -> King, Martin L., Jr.

The solution initially demonstrated about 83% compliance¹, failing the test for the last example (retaining suffixes). After describing the failing test to the LLM, it generated code for which all tests now passed:

export function normalizeName(name) {
  let trimmedName = name.trim();
  let hasSuffix = trimmedName.includes(',');
  let suffix = '';

  if (hasSuffix) {
    [trimmedName, suffix] = trimmedName.split(',');
    suffix = ', ' + suffix.trim();
  }

  let parts = trimmedName.split(/\s+/);

  if (parts.length === 1) {
    return parts[0] + suffix;
  }

  let lastName = parts.pop();
  let firstName = parts.shift();
  let middleInitials = parts.map(name => name.length === 1 ? name : `${name.charAt(0)}.`).join(' ');

  return `${lastName}, ${firstName} ${middleInitials}`.trim() + suffix;
}

The solution exhibits the default procedural style exemplified by LLMs.

CAX With Style

Early on in my AADV exploration (fall 2023), I started experimenting with deriving a set of guidelines to provide to the LLM. A better-designed solution might be more likely to be correctly generated, and if it wasn’t, a better design would make it easier to have a follow-up conversation.

MCP² (my custom ChatGPT) is currently configured with a set of style rules nicknamed js-jeff—probably last updated last fall. I reminded MCP to remember to adhere to js-jeff.

export const normalizeName = name =>
  formatName(splitAndClean(name))

const splitAndClean = name => {
  const [mainPart, suffix] = name.trim().split(/\s*,\s*/, 2)
  return [...mainPart.split(/\s+/), suffix].filter(Boolean)
}

const formatName = parts => {
  if (parts.length === 1) return parts[0]

  const [first, ...middleAndLast] = parts
  const suffix = middleAndLast.at(-1)?.match(/^[A-Za-z.]+$/) ? null : middleAndLast.pop()
  const last = middleAndLast.pop()
  const middleInitials = middleAndLast.map(m => `${m[0]}.`).join(' ')
  const baseName = middleInitials ? `${last}, ${first} ${middleInitials}` : `${last}, ${first}`

  return suffix ? `${baseName}, ${suffix}` : baseName
}

Nope. Failing and not at all the kind of code I wanted to see.

The formatName function contains excessive implementation details. Some folks might like the introduction of copious local variables, which do introduce some clarity about intent, but I’m not a fan—too much scanning back and forth to figure out how all the variables and statements tie together.

The Forgotten Rules

Did MCP even remember its guidelines? I couldn’t remember the specifics, so I asked.

prompt> show me your style rules

MCP regurgitated the guidelines I’d provided some time ago. Here’s its summary:

Modern ES6+ Syntax
• Use import/export, never require
• Prefer arrow functions
• No var, only const or let
• Use destructuring when applicable

Functionality-First Structure
• Small, single-purpose functions
• No local mutations (pure functions preferred)
• No unnecessary variables--prefer function calls
• Extract complex logic into separate functions

Code Formatting & Readability
• No semicolons (we’re not barbarians)
• No inline comments inside functions
• Use meaningful function & variable names
• No excessive chaining—extract readable parts
• Consistent indentation (2 spaces preferred)

I’d thought my rules were reasonable, but MCP was ignoring them. Maybe they’re just too much to hold in context. I asked MCP. It told me:

I’m trained on general code patterns. Unless your rules are very explicitly reinforced, the model’s instinct is to default to industry-standard, Stack Overflow-style code.

• Example: I may generate local variables even though your rules say to avoid them.
• Or I might throw in a helpful console.log because… that’s what average devs do.

Oh! I see. So I prompted MCP with a helpful reminder.

prompt> The code is not adhering to the style. Extract most implementation-specific logic into separate functions.

Regen (only the prod code) because the test is failing.

Sigh. The code in this solution was still not quite what I wanted:

export const normalizeName = name =>
  formatName(extractSuffix(splitAndClean(name)))

const splitAndClean = name =>
  name.trim().split(/\s+/)

const extractSuffix = parts => {
  const suffixIndex = parts.findIndex(part => part.includes(','))
  if (suffixIndex === -1) return { names: parts, suffix: null }

  const names = parts.slice(0, suffixIndex)
  const suffix = parts.slice(suffixIndex).join(' ').replace(',', '').trim()
  return { names, suffix }
}

const formatName = ({ names, suffix }) => {
  if (names.length === 1) return names[0]

  const first = names[0]
  const last = names.at(-1)
  const middleInitials = names.slice(1, -1).map(getInitial).join(' ')

  const baseName = middleInitials ? `${last}, ${first} ${middleInitials}` : `${last}, ${first}`
  return suffix ? `${baseName}, ${suffix}` : baseName
}

const getInitial = name => `${name[0]}.`

Worse, the suffix test was failing (again). I hadn’t explained things explicitly enough.

Highly Func-ified Code

I decided to be fairly explicit:

prompt> Regen, try again. When the RHS of any assignment has implementation specifics, extract a function for it.

I’m sure there’s a better way to state that rule, but I was quite happy with the result:

export const normalizeName = name =>
  formatName(processName(name))

const processName = name =>
  extractNameParts(splitWithSuffix(normalizeWhitespace(name)))

const normalizeWhitespace = name =>
  name.trim().replace(/\s+/g, ' ')

const splitWithSuffix = name => {
  const parts = name.split(/\s*,\s*/)
  return parts.length > 1
    ? { names: parts[0].split(' '), suffix: parts.slice(1).join(', ') }
    : { names: name.split(' '), suffix: null }
}

const extractNameParts = ({ names, suffix }) =>
  names.length === 1
    ? { first: names[0], middleInitials: '', last: '', suffix: null }
    : { first: names[0], middleInitials: extractMiddleInitials(names), last: names.at(-1), suffix }

const extractMiddleInitials = names =>
  names.slice(1, -1).map(name => shouldInitialize(name) ? `${name[0]}.` : name).join(' ')

const shouldInitialize = name =>
  !(name.length === 1 || (name.length === 2 && name.endsWith('.')))

const getInitial = name =>
  `${name[0]}.`

const formatName = ({ first, middleInitials, last, suffix }) =>
  last
    ? formatFullName(first, middleInitials, last, suffix)
    : first // mononym case

const formatFullName = (first, middleInitials, last, suffix) => {
  const baseName = middleInitials ? `${last}, ${first} ${middleInitials}` : `${last}, ${first}`
  return suffix ? `${baseName}, ${suffix}` : baseName
}

Short functions, each with a concise, intention-revealing name. It’s likely not your preferred style. But for me such code provides numerous benefits. One is the isolation of all the small chunks of implementation detail into separate functions or concepts. This organization allows me to rapidly gain understanding, and rapidly find the concepts I need to know more about. For most cases, I can ignore most of the code details of most of the functions.

That’s my take on short functions in human-generated code. For MCP code, having small, crystal clear functions also allows me to rapidly scan the solution and look for glaring problems. If the LLM had completely misunderstood how to solve the problem, it would be glaringly obvious with such a style. (An example: Name normalizer solutions that I generated early on introduced a small lookup table to identify suffixes—Dr., Mr., Jr., etc.—rather than looking for a comma to signal suffixes. The examples I provide to the LLM now indicate the significance of the comma.)

Still No Dice

My hypothesis that better design leads to better compliance did not hold true. The regenerated solution still failed the suffix test.

export const normalizeName = name =>
  formatName(processName(name))

const processName = name =>
  separateSuffix(splitAndClean(normalizeWhitespace(name)))

const normalizeWhitespace = name =>
  name.trim().replace(/\s+/g, ' ')

const splitAndClean = name =>
  name.includes(',') ? splitWithSuffix(name) : name.split(' ')

const splitWithSuffix = name => {
  const [mainPart, suffix] = name.split(/\s*,\s*/, 2)
  return [...mainPart.split(' '), suffix]
}

const separateSuffix = parts =>
  hasSuffix(parts)
    ? { names: extractNames(parts), suffix: extractSuffix(parts) }
    : { names: parts, suffix: null }

const hasSuffix = parts =>
  parts.at(-1)?.match(/^[A-Za-z.]+$/) === null

const extractNames = parts =>
  parts.slice(0, -1)

const extractSuffix = parts =>
  parts.at(-1)

const formatName = ({ names, suffix }) =>
  names.length === 1 ? names[0] : formatWithLastName(names, suffix)

const formatWithLastName = (names, suffix) => {
  const first = getFirstName(names)
  const last = getLastName(names)
  const middleInitials = getMiddleInitials(names)
  const baseName = middleInitials ? `${last}, ${first} ${middleInitials}` : `${last}, ${first}`
  return suffix ? appendSuffix(baseName, suffix) : baseName
}

const getFirstName = names =>
  names[0]

const getLastName = names =>
  names.at(-1)

const getMiddleInitials = names =>
  names.slice(1, -1).map(getInitial).join(' ')

const getInitial = name =>
  `${name[0]}.`

const appendSuffix = (baseName, suffix) =>
  `${baseName}, ${suffix}`

Try again, MCP. (Still failing.)

export const normalizeName = name =>
  formatName(processName(name))

const processName = name =>
  extractNameParts(splitWithSuffix(normalizeWhitespace(name)))

const normalizeWhitespace = name =>
  name.trim().replace(/\s+/g, ' ')

const splitWithSuffix = name => {
  const parts = name.split(/\s*,\s*/)
  return parts.length > 1
    ? { names: parts[0].split(' '), suffix: parts.slice(1).join(', ') }
    : { names: name.split(' '), suffix: null }
}

const extractNameParts = ({ names, suffix }) => ({
  first: names[0],
  middleInitials: extractMiddleInitials(names),
  last: names.at(-1),
  suffix
})

const extractMiddleInitials = names =>
  names.slice(1, -1).map(getInitial).join(' ')

const getInitial = name =>
  `${name[0]}.`

const formatName = ({ first, middleInitials, last, suffix }) => {
  const baseName = middleInitials ? `${last}, ${first} ${middleInitials}` : `${last}, ${first}`
  return suffix ? `${baseName}, ${suffix}` : baseName
}

Fantastic, the suffix test finally passed. But now the mononym test for Plato failed:

Expected: "Plato"
Received: "Plato, Plato"

It’s a good thing I had the LLM generate unit tests (which I assessed & blessed). No, it’s a critical thing—I can’t imagine the bad vibes that would emanate from coding without them. Fortunately, they were nearly free to generate.

One last prompt generated a solution that passes all the tests:

export const normalizeName = name =>
  formatName(processName(name))

const processName = name =>
  extractNameParts(splitWithSuffix(normalizeWhitespace(name)))

const normalizeWhitespace = name =>
  name.trim().replace(/\s+/g, ' ')

const splitWithSuffix = name => {
  const parts = name.split(/\s*,\s*/)
  return parts.length > 1
    ? { names: parts[0].split(' '), suffix: parts.slice(1).join(', ') }
    : { names: name.split(' '), suffix: null }
}

const extractNameParts = ({ names, suffix }) =>
  names.length === 1
    ? { first: names[0], middleInitials: '', last: '', suffix: null }
    : { first: names[0], middleInitials: extractMiddleInitials(names), last: names.at(-1), suffix }

const extractMiddleInitials = names =>
  names.slice(1, -1).map(getInitial).join(' ')

const getInitial = name =>
  `${name[0]}.`

const formatName = ({ first, middleInitials, last, suffix }) =>
  last
    ? formatFullName(first, middleInitials, last, suffix)
    : first // mononym case

const formatFullName = (first, middleInitials, last, suffix) => {
  const baseName = middleInitials ? `${last}, ${first} ${middleInitials}` : `${last}, ${first}`
  return suffix ? `${baseName}, ${suffix}` : baseName
}

So, for the LLM, the sixth time was the charm. As ridiculous as Vinny Gambini himself. If you only count the attempts after I established my “definitive” set of style guidelines, it took four prompts.

Still, CAX is a reasonably fast process. It would be even faster with an appropriate plugin. The above four-prompt conversation took less than five minutes. Not bad considering it might take 20 minutes plus to hand-code a solution along with tests. That’s a little disappointing to me, though.

Not Just the Rules, Ma’am

Maybe throwing a pile of rules at the problem isn’t what’s really needed. What if I were to be explicit about the way the solution is designed?

After providing the name normalizer overview, I added an overview of what the solution should look like.

prompt> Use this solution design.

* within the entry point function, normalizeName, create a functional pipeline. Its individual steps should be declared as separate top-level functions.
* it should be a true functional pipeline, with the output of each step injected into the next step
* the first step in the pipeline should be a function that parses the name. There might be multiple pipeline steps that help parse or decorate an existing parse object.

As an additional part of the prompt, I told MCP to use the pipeline operator |>, (supported here in Javascript via a Babel plugin, one of a few alternatives³). MCP did as asked:

export const normalizeName = name =>
  name
    |> trimName
    |> handleSuffix
    |> splitName
    |> initializeMiddleNames
    |> formatParsedName

const trimName = str =>
  str.trim().replace(/\s+/g, ' ')

const handleSuffix = str => {
  const [namePart, suffix] = str.split(',')
  return { name: namePart, suffix: suffix ? suffix.trim() : '' }
}

const splitName = ({ name, suffix }) => {
  const parts = name.split(' ')
  return {
    first: parts[0] || '',
    middle: parts.slice(1, -1),
    last: parts.length > 1 ? parts[parts.length - 1] : '',
    suffix
  }
}

const initializeMiddleNames = parsed =>
  ({
    ...parsed,
    middle: parsed.middle.map(n => n.length === 1 ? n : `${n[0]}.`)
  })

const formatParsedName = ({ first, middle, last, suffix }) =>
  !last
    ? first
    : `${last}, ${first}${middle.length ? ' ' + middle.join(' ') : ''}${suffix ? ', ' + suffix : ''}`

Love it. It’s a little less choppy than the earlier func-ified solution, which might appeal to some of you.

The best part: The tests pass. First time is now the charm.

Moving Forward: Tentative Recommendations

My learnings don’t provide a definitive path forward, but they do provide a basis for some deeper investigation and experimentation. I’ve started building an engine that will allow me to generate numerous solutions (like, 100 at a time) so that I can see what correlation, if any, there is between prompt guidance and compliance. (My goal is to have run at least one experiment within the next two weeks.)

For now, I believe some of these conclusions (drawn from material in the free part of this article, for those of you not on a paid subscription) might hold promise: