GO GO GOLEMS
/go-go-golems
WE ARE ROBOTIC COMPUTER SCIENTISTS WRITING ABOUT PROGRAMMING AND LARGE LANGUAGE MODELS.
GO GO GOLEMS
🚀LLMs
3y ago
Ever wrestled with an LLM error that left you scratching your head? You're not alone.
Language Models (LLMs) promise to revolutionize software development, but they also introduce new challenges due to their probabilistic nature: LLMs are much better at analogical reasoning than formal logic. In this article, we're tackling these challenges head-on. We'll provide five practical tips to navigate common LLM errors when programming. Let's dive in.
1. Variable Names are Critical
Having appropriate variable names act as reminders for the LLM, clarifying their purpose each time they are referenced (due to their attention window). For instance, using 'productId' instead of 'id' eliminates the risk of confusing it with another entity's ID, like a category. Interestingly, longer, more descriptive variable names often lead to lower token counts. For example, 'productId' is one token, while 'pId' is two.
2. Comments Rule the Land
LLMs generate a multitude of comments, some trivial at first glance. However, these comments clarify the intent of the code, irrespective of the programming language. This clarity not only helps the LLM generate accurate code but also helps identify mistakes during code review sessions. These comments can be removed in a final pass to avoid clutter.
3. Understand your language's landscape
Languages with strong opinions about their code and a widely used standard library or design patterns work best with LLMs. For instance, Golang is an ideal fit. When languages have a wide range of styles and libraries to choose from, specify those as part of the prompt to ensure consistency. This could include your preferred database driver/ORM, web framework, unit testing framework, error handling method, comment style, and import structure. Providing these guidelines to the LLM also benefits human developers.
4. Apply Software Architecture Principles
When programming with LLMs, apply traditional software architecture and design principles. Use intermediate functions, classes, structs, or types to tame complexity--remember to use readable names. This approach makes it easier for LLMs, with their analogical reasoning, to generate correct code, preventing them from spiraling into intricate, hard-to-follow code.
5. Maintain a Clean Context Window
LLMs consider everything in their context window when generating code, so any mistakes can lead to further inaccuracies. Be ruthless in correcting wrong code and restart the conversation with a fresh context when necessary. This approach ensures the LLM works with exactly the information it needs, leading to more accurate and efficient code generation.
Hopefully these tips will help you avoid errors when programming with LLMs. Start applying them today!
GO GO GOLEMS
🚀LLMs
3y ago
Unit tests enable bold refactors and protect against subtle errors introduced by Large Language Models. However, while unit testing strengthens codebases for the long haul, they drain developer bandwidth with rote work.
In this article, we'll explore tactics to leverage LLMs as a unit testing sidekick - taking care of the grunt work so you can focus on the good stuff.
Craft test blueprints
Following the funnel method, we leverage LLMs in crafting unit tests by transforming the code we want into an actionable unit testing plan.
Summarize the code as bullet points, to ensure the language model understands what needs to be tested. Experimentwith providing only your code's API (transcript) or its actual implementation (transcript).
Prompt the LLM to look for edge cases, common scenarios, complex combinations of parameters. Don't hesitate to provide your own edge cases too. Don't forget to regenerate often (transcript).
Don't have the model output any code at this point. We are actually interested in transforming the formal input into something higher up the language ladder, as it provides denser information in less tokens.
Once you have a testing plan that you are happy with, it is time to take a step back (or a short walk around the block) and make sure that you have taken everything into account.
Reviewing the test blueprints
Reviewing crafted test blueprints is pivotal to ensure they capture the essence and intricacies of your target code. This blueprint will be used for actual code generation; it is crucial to ensure it is concise, correct and exhaustive.
Begin by manually scrutinizing each test, checking that it indeed makes sense. Don't hesitate to manually edit the blueprint.
When faced with complex tests, don't hesitate to ask the LLM for clarifications on their purpose and expected outcomes; this not only aids comprehension but also doubles as invaluable documentation. It will also enhance the LLM's ability to generate proper code (transcript).
Ready to bring the blueprint to life? Dive into the next section: creating a testing skeleton.
Implementing the tests
After crafting a strong list of unit tests, it's time to let the model write the actual code.
Set a consistent style by generating a list of unit test signatures and brief descriptions (remember how self-attention works).
Guide the model by providing a template example that shows which framework, checks, and mock objects to use. This will ensure the upcoming tests align with existing code. (it's often beneficial to have the model produce table-driven tests for uniformity).
Always ask for clear error messages and reasons for test failure, making failed tests easy to decode.
Don't ask the model to implement all tests at once. Instead, iterate through the signatures, and use the same template each time to ensure consistency.
Make sure the template implementation is spotless. This will improve the quality of future implementations (transcript, note the LLM-esque off by one in the key iteration).
With straightforward tests and clear error info, there's no need to spend hours sifting through test code when issues arise.
The future of unit testing
LLMs' convenience means that even when faced with tight deadlines, there's always time for thorough tests: a short investment of 10-15 minutes can ensure foundational coverage.
This makes me pretty confident saying that large language models will revolutionize unit testing.
Remember, the aim isn't to let LLMs fully take the reins, but to utilize them as invaluable colleagues.
GO GO GOLEMS
3y ago
For too long, software documentation has been an afterthought.
We've all encountered software documentation that feels more like a last-minute addition rather than an integral part of the design process. It's no surprise, really. Crafting quality documentation requires skill, time, and a knack for writing. From high-level overviews to how-to guides and references, keeping everything updated can be a daunting task.
But what if I told that ChatGPT makes documentation work easier and more enjoyable; I think this is bound to transform software engineering.
Write documentation for the user
On the ladder of language, writing documentation is moving up from the code level to a more human level.
Documentation should anticipate questions and explain concepts clearly. GPT4 in particular is able to reason well enough to write documentation useful to the user. I ask it to skip over explaining the inner workings and instead to be exhaustive yet concise. A typical prompt might be, "Create documentation that guides a new developer on using the code, highlighting edge cases and crucial details."
Always consider your audience when crafting documentation. Then, ask ChatGPT to tailor your content to meet the reader's needs.
Targeting the different types of documentation
ChatGPT is a versatile tool that can help you create various types of documentation. Here's 4 ways how:
For tutorials and how-tos, merge existing documentation, third-party library documentation, and your shell history. Ask ChatGPT to create step-by-step guides for common operations in your codebase.
For reference material, prompt ChatGPT to produce documentation in a structured format like JSON. Repeat this for each section to ensure consistency. This also makes it easy to index and render the documentation for your website.
Use a diff and commit history to craft a compelling pull request description. You'd be surprised at how much ChatGPT can infer from a few diff lines.
When working with existing documentation and actual code, ask ChatGPT to write accompanying examples. Direct examples often prove to be the most useful part of documentation.
So, why wait? It's time to transform your documentation into something actually your colleagues actually enjoy reading!
GO GO GOLEMS
🚀LLMs
3y ago
Meet the 'Funnel' method: the Swiss Army Knife of ChatGPT prompting.
Marrying the ladder of language with the compress/expand technique, it's your go-to for every ChatGPT interaction. Dive in and master error-catching, long content management, and consistent writing—in just three steps.
1. Put text into the funnel
The Funnel method transforms verbose language into concise statements by "pressing it" into a funnel.
Ask ChatGPT to convert complex inputs—like jumbled text or meeting transcripts—into succinct bullet points, ensuring no detail is overlooked. The bullet point format strips the language bare: no more stylistic construct, implied context, messy grammar, confusing turns of phrases.
Other intermediate formats are lists of keywords and categories, sentences of the form SUBJECT VERB OBJECT, RDF triples or Prolog facts. All of these have the benefit of reducing token count, making it easier for the model to take everything into account.
2. Verify, refine and edit
Now, it's simpler to ensure your list captures all essentials. To hone it:
Validate each point.
Spot and fill any gaps.
Use ChatGPT for enhancements.
Adjust manually to your liking.
A polished list not only improves clarity but also serves as an effective reference for future notes.
3. Boldly rearrange your content
With a streamlined data format, manipulating content becomes straightforward. Your bullet list, paired with original or other sources, can be transformed into:
Topic slides.
Reports, essays, blogs, or tweet threads from rough drafts.
Flashcards.
Index entries.
A book's table of contents.
Where verbose language can muddle output, a concise format acts as a Swiss Army Knife allowing expansive text transformations.
GO GO GOLEMS
🚀LLMs
3y ago
Just because it's dusty doesn't mean it can't shine again.
Because they produce boring yet consistent results, large language models (LLMs) are the perfect cleaning crew for your legacy codebase. Primed with proper guidelines, they can defuse your ticking time bomb and turn it into a solid foundation for future work.
Here's 4 ways LLMs can help with modernizing your codebase.
Add type hints and docstrings to existing code
Who wouldn't want to have their code fully documented, in a consistent style?
To generate good docstrings, follow these steps:
Gather the code you want to document as well as relevant additional information (for example, the documentation of functions this code depends on).
Provide existing docstrings or type definitions to use as style guidelines.
Verify and edit the result (LLMs tend to be a bit rigid and verbose, not necessarily a bad thing).
As with all LLM-based coding, the trick is figuring out what to put in the context window for the model to do a good job.
Add unit tests
LLMs are a force multiplier for writing unit tests. It won't entirely remove the need to think hard about confusing legacy code, but it will do 90% of the work in most cases. As usual, priming the context window is key.
First, ask for an explanation of the code and potential edge cases (generate output that will cause the model to generate better output). Concise bullet points lists using keywords are the best way to get a lot of information into a few tokens.
Ask for a list of tests (without code). This again primes the context window without filling it with too many "low-quality" tokens just yet.
Ask the model to write each test one by one, making sure it still has the original code in its context window. If needed, paste the code and explanation again.
Special tip: build table-driven tests to save tokens.
Modernize code with intelligent refactoring
Much more flexible than traditional refactoring tools, LLMs can deal with instructions such "introduce a facade pattern", "group the arguments into a struct", "upgrade the class to use the new readonly parameter language feature" or "transform these for loops into a functional chain."
Make sure you have good unit tests in place. LLMs are fuzzy magic and can introduce bugs.
Paste your legacy code along with a complex refactoring instruction
If consistent style is needed across refactors, priming with an example of a previous refactor helps
This one is maybe my favorite. Combined with command-line tooling to automate the process, this can really slice through crusty code!
Extract OpenAPI Specifications from a Chrome HAR Recording
Thanks to a well hidden feature in Chrome, you can generate OpenAPI specifications, HTTP client code, mock REST APIs in a heartbeat. Here's how:
Record API interactions in Chrome using the Network tab in the Developer Tools.
Export as HAR format.
Extract requests (with headers, body) as JSON (for example using a tool like Google's HAR Analyzer or your own script).
Paste into ChatGPT and ask it to create OpenAPI specification (or a javascript client, etc...).
Profit!
I hope these few tips show you the incredible impact LLMs can have on modernizing your codebase!
GO GO GOLEMS
🚀LLMs
3y ago
Writing entire programs with ChatGPT is not easy.
Too high are the risks for inconsistent style, using the wrong libraries, losing track over longer conversations. So what is the secret to building high-quality, maintainable and effective software with ChatGPT?
This article serves as an introduction to a topic we will examine in depth over the next couple of days: real-world programming with ChatGPT.
Solving longer tasks with ChatGPT: 2 essential skills
Solving longer tasks with ChatGPT requires two fundamental steps:
decomposition: ChatGPT can't just solve any problem that is presented to it in 2 sentences. It is most effective at doing small moves around the ladder of language.
context information: ChatGPT also needs to have all the information required for it to do a rock-solid job, either through its training corpus or by having it explicitly provided in its context window.
This is where your knowledge as a software engineer is crucial: decomposing a problem into steps along with knowing which information is required for that step is a skill that only comes through experience and deep study.
From prototype to CLI tool: 3 steps to get high-quality results
I use the following 3 steps to create useful command-line applications (we will use "docstring", a tool that extracts structured data out of source code documentation string, as an example):
Ask ChatGPT to create a prototype: I will often ask ChatGPT to create a prototype for a small task, and give it minimal guidance. I chose a task small enough (we will come back to what is small enough): "extract docstrings out of a file using a set of regexp". Note how I already decomposed the problem within that first step.
Adapt and refine code: Once I get the first version back, I will rework the code, either by hand or by continuing to prompt ChatGPT with small tasks. When I do hand edits, I paste my changes back into ChatGPT.
Fix bugs, extract methods, clean up datastructures
Ask ChatGPT to create unit tests, docstrings
Incrementally add more functionality. In the docstring case, for example: parse attributes within docstrings, extract a "Scanner" struct, support more languages
Transform into a known pattern: When I'm satisfied with my prototype (this is the stage "docstring" is at, I will transform into a well known application pattern, usually by pasting concise instructions. I usually transform my tools into glazed applications by using these instructions (we will come back to gathering context information and the tools I use soon).
This short excursion into building real-world programs is all you need to get started: decompose and transform, by providing the right context information.
GO GO GOLEMS
🚀LLMs
3y ago
You might think that a conversation with ChatGPT is just like a normal human conversation--after all, they're both in plain English (or of course the language you speak to it).
Don't be fooled: talking to ChatGPT is really more like programming.
Introducing: the Ladder of Language
There is a widespread idea that computer code is complex and mysterious, full of inscrutable symbols and abstractions, but that's discounting human language's richness.
Programming requires everything to be stated explicitly: IF A THEN B ELSE C. It does not care much about vocabulary, and is thus replete with unnatural sounding words like networkTokenManager, ctxRef or retVal. By contrast, human language has many shades of grey: context can be implied; a sentence might mean its opposite; choice of words convey information, emotion and intent.
I like to think of this as the "Ladder of Language:"
at the top, we have the rich and dense language that humans speak,
at the bottom, the rigid and structured code that computers understand.
ChatGPT's Language: Neither Human nor Machine
ChatGPT understands language differently than most computer programs.
For example, it is not very good at working within the rigid formal logic required by programming languages: it often makes bugs or calls functions that don't exist. Similarly, it will often misunderstand something that any child would think is obvious (it often misunderstands negation, for example).
ChatGPT sits at the middle of the ladder. It is king in the land of meeting notes and TPS reports; lord in the fief of resumes and social media articles; conqueror of the steppes of domain-specific languages and structured data.
ChatGPT is Best at Moving Text Up and Down the Ladder
Because large language models are decent at parsing the complex semantics of human language (something normal programs can't do well at all), yet also able to deftly generate mostly correct computer gobbledigook (something humans are neither great at nor very fond of), ChatGPT's main strength is pushing words up and down the ladder.
It excels at tasks like:
transforming natural language into JSON (moving down the ladder)
writing comments for existing code (moving up the ladder)
transform a voice memo into meeting slides (moving down the ladder)
explain a bug based on a stacktrace (moving up the ladder)
explain how to solve a problem step by step (moving down the ladder)
Moving down the ladder keeps ChatGPT on track by constraining the meaning of words.
Moving up or down the ladder makes things easier for humans to understand: converting cryptic computer code into something more fluid; summarizing long, or just badly written human language into something more structured and clear.
When prompting, always think about how you want to move around the ladder.
GO GO GOLEMS
🚀LLMs
3y ago
Imagine transforming a simple chatbot into your personalized digital assistant. OpenAI introduced a feature that does just that: with Custom Instructions, you can now prepend custom text to every message in a conversation.
Beyond basic chat
This might at first seem like an innocuous feature: surely you can do the same with copy-paste. However, having that option preset and out of your mind transforms ChatGPT from a powerful yet generic bot into an assistant that feels deeply personal without having to constantly control its behaviour. Instead of sticking to the basic examples often cited by OpenAI, let's delve into the untapped potential of this feature.
Here are three ways to use Custom Instructions that will change how you use the tool forever.
1. Easily control the output format
Switching between verbose and concise outputs can be simplified using a custom instruction like:
If the first word is terse/concise/default/comprehensive/exhaustive, use the matching verbosity. When exhaustive, being comprehensive in depth and breadth.
Simply insert the appropriate keyword to control how chatty the answer should be.
2. Define custom slash commands
While you can't "program" ChatGPT per se, you can tell it to follow commands, like this:
Slash Commands:
/questions - Ask pertinent and interesting questions a curious user would ask
/summarize - Summarize the conversation so far
/critique - Criticize your previous answer before improving your output
3. Customize the output
You can apply our previous insights to create custom instructions. For example, in order to leverage its attention mechanism or optimize context size, we can ask it to:
Output your answers in concise bullet point form, prefixing each with a relevant emoji
Before answering, concisely list what your expertise in the domain is and how you are approaching the problem. Think step by step.
Make ChatGPT your own
We've only begun to explore the myriad ways how ChatGPT can be improved using custom instructions. ChatGPT AutoExpert offers many insights (including instructions tailored for code). Yet, the most effective instructions will be the ones tailored by you, for you.
GO GO GOLEMS
🚀LLMs
3y ago
While hands-on experience is paramount for mastering Large Language Models, a basic understanding of their internal mechanisms can have a big impact.
If you've followed this series, you are already familiar with the importance of autoregressive behavior in the attention mechanism. Today's discussion will focus on tokenization and context size: two additional implementation aspects that will show why shorter prompts are often better prompts.
1. Tokenization
In order for text to be processed by a Large Language Model, it has to be turned into numbers. This process is called tokenization: each word is broken up into smaller units known as tokens. As we will see, the amount of tokens in your prompt can have big consequences.
There are usually multiple tokens per word, which results in a token being around 3/4 of a word. To get the exact count of tokens in your prompt, you can use OpenAI's tokenizer.
2. Context Window Size
Due to memory constraints, computational costs and the use of "positional embeddings" (necessary to keep word ordering), Large Language Models have a fixed context window size: they can only look at so many words to predict the next one.
Because the model looks at input tokens as well as previously emitted output tokens, both the prompt and its answer have to fit within the context size. ChatGPT 3.5 has a context size of 32k tokens, while ChatGPT 4 has a context size of 8k.
Better Conversations with Less Tokens
Each message in a conversation adds to the context size. Exceed the limit, and ChatGPT starts forgetting earlier parts. While it can be tempting to add a lot of data to your prompts, conveying the important information concisely is often a better technique. It not only allows for longer conversations, but it tends to improve the quality of the answers as well.
Here are a few techniques to reduce the size of your prompts:
Pre-summarize data before adding it to your prompt
Request answers in bullet-point form
Condense information and often start new conversations, keeping only the key details
Understanding why token size matters is key to having conversations that make full use of the context window of the LLMs.