Jeremiah Marble

/jeremiah-marble

Jeremiah Marble

AI Product Leader ๐ŸฆŠ | Author http://Model47.com | Co-Founder #PrimaDonaStudios & #TowardsArcadia | Speaker | Writer๐Ÿšข

jeremiahmarble.com
User Avatar

Jeremiah Marble

1y ago

RAI 32/100: More agents!
Jeremiah Marble

But for those of us trying to stay abreast of the field - check out this blog on Agents from Chip Huyen (one of my favorite technical authors / GREAT communicators in AI/ML).

Link to her blog: https://huyenchip.com//2025/01/07/agents.html?utm_source=tldrai

User Avatar

Jeremiah Marble

1y ago

RAI 31/100: FACTS Grounding leaderboard (to assess LLMs' hallucination)
Jeremiah Marble

Not completely surprising that the top-scoring LLMs are all from Google, on a "truthiness" leaderboard by...Google... BUT I will say that the FACTS Grounding benchmark could be yet another useful way to assess various models as we seek to pick the one that WE will use. Will link in comment! RAI 31/100

User Avatar

Jeremiah Marble

1y ago

RAI 30/100: Agent mosaic
Jeremiah Marble

Hearing everyone else yammer on about AI Agents?
Feeling a bit lost? Like everyone else gets this?
Hint - lots of folks be TALKING about agents (cryptobros -> agent bros).

Not building.

Yesterday I was out on a run in the Seattle rain, when a buddy of mine texted me an image of agent categories and product names.

His Q: "How do I make sense of this?"

I started to text him back. Stopped.

First, it's fun to run in the rain. MUCH less fun to huddle beneath a tree and text about a complicated topic with my thumbs.
Second, I wanted to write this down.

[ MAIN PARTS OF AGENTIC AI ]

_agent_
Code that:
* Takes in some input
* Figures out how to respond to the input
* Does that
* Gives output

_agent framework aka blueprints_

* Tools & libraries to create, structure, & customize agents.
* How to reason, make decisions, remember things, use tools, execute tasks?


_agent hosting - tiny agent houses!_
* Agents need to live somewhere!
* Like the factory system to run & deliver
* Spins up agents, lets them "talk" to each other, & other parts of the system.
* LangGraph is common here. Amazon Bedrock is solid, in the AWS ecosystem. Microsoft & Google have theirs too.

_model hosting - tiny model houses!_
* Many agents are going to call models (like LLMs) to do their work
* Usual suspects - OpenAI, Microsoft, Google, etc
* Want to build / host your own model? Have seen Ollama recos

_tool libraries - doing "normal" stuff" aka "what we did before LLMs"_
* e.g. send emails, set calendar invitations, reserve rooms, book flights, etc.
* Usually call APIs, not models
* Options include Composio & Browserbase

_memory - avoiding "50 1st dates" situations_
* Plug for Armand Ruiz's free email course on AI agents (Day 6):
"remember past interactions, understand long-term objectives, and adapt to changing conditions. By getting memory right, youโ€™ll create agents that feel more coherent, purposeful, and valuable to users over time."
* link in comments

[ SUPPORTING CHARACTERS IN AGENTIC AI ]

{ storage - fka "databases" }
* My friends tell me that I can't call this "just a database for F's sake"
* Don't call it that. You'll get yelled at.
* You want to store things

{ observability - fka "logging" }
* You want to know what's going on
* Is it working? Is it failing? How long does it take? Who's yelling?


{ sandboxes - try stuff out}
* Haul out the "Breaking Prod" t-shirts

[ VERTICAL ]
This kind of combines all (or much) of the above

FIN!

User Avatar

Jeremiah Marble

1y ago

RAI 28/100: New ways of alignment
Jeremiah Marble

Quick RAI today (28/100): OpenAI announced a new method to align reasoning models, seeking to ground them with behavior goals (e.g. safety rules).

"[It] directly teaches reasoning LLMs the text of human-written and interpretable safety specifications, and trains them to reason explicitly about these specifications before answering."

Seems like a pretty good idea! Will share link in comments if you're following along at home.

Happy holidays!

User Avatar

Jeremiah Marble

1y ago

RAI 27/100: Zapier copilot
Jeremiah Marble

"Intending" to try Zapier but too busy fighting fires in the rest of your life?

Yeah, it me.

Well, it WAS me until I gave it a try today. AND THEY'RE USING AI TO MAKE IT SIMPLE. So - voila, le Zapier Copilot!!!

(For some reason, I'm picturing one of the dudes from "Flight of the Conchords" saying that.)

So I got a bit tired of manually copying over the Samcart registrations from our last workshop into Kit (our email system) - and decided finally to go for it.

It was a great experience - big props!

Building (& using) AI responsibly - FTW!

(27/100)

User Avatar

Jeremiah Marble

1y ago

RAI 26/100: Agentic course from Armand Ruiz
Jeremiah Marble

Hearing all about this "agentic" thing in AI, but have no idea where to start?

Seeing lots and lots of posts, reading release notes, playing with code snippets and tools, but feeling like it's just, well, A LOT?

Yeah, me too.

I'm going to spend a bunch of time over the next few weeks continuing my quest (just sounds cooler that way) to learn as much as I can about building (and using) AI systems responsibly.

A bit part of that is agents.

Armand Ruiz of IBM just dropped a free 10-day email course.

I INSTA signed up.

If you're curious about the space, give it a look! I'll drop the link in the comments.

Have a fun day!

RAI 26/100

User Avatar

Jeremiah Marble

1y ago

RAI 25/100: AIR-BENCH
Jeremiah Marble

This one is really exciting!

For my own RAI work (25/100!), I've been compiling my own "prompt library" - and it's not simple.

So am mad envious at their 5,694 diverse prompts across 314 granular risk categories!!

I wish we could just download their prompts - but of course that would defeat the purpose, since every model-maker would do the same & game the system.

So will try to hook up a pipeline. Will slide into their repo....

User Avatar

Jeremiah Marble

1y ago

I had so much fun speaking Saturday with Tosin & the crew at the ๐—”๐—œ & ๐—”๐—ป๐—ฎ๐—น๐˜†๐˜๐—ถ๐—ฐ๐˜€ ๐—ง๐—ฟ๐—ฎ๐—ป๐˜€๐—ณ๐—ผ๐—ฟ๐—บ๐—ฎ๐˜๐—ถ๐—ผ๐—ป ๐—–๐—ผ๐—ป๐—ณ๐—ฒ๐—ฟ๐—ฒ๐—ป๐—ฐ๐—ฒ. I delivered a new talk, called "Fellowship of the AI Rings," for those of us seeking to design and scale robust AI systems responsibly.

In the talk, I discuss five "rings" of building AI systems with RAI in mind:

* Model Creation/Training

* User Interaction

* Guardrails

* Contextual Enrichment (e.g. RAG)

* Agentification

3 things that might resonate with you if you do this kind of work:

1) We have decisions to make within each "ring." We can best make these decisions only by understanding WHO we're designing for,and WHAT problem they're facing. Turns out it's STILL important to be user-centric, even in this "3rd rodeo" of the AI era. For example, the Guardrails for an enterprise AI system for documents is going to be prettttty different than the Guardrails of a first-person car-racing AI game.

2) We can only build AI systems responsibly if we're planning for it. It has to be our choice to integrate safety measures - everything from controlling randomness (temperature) and relevance (top-n), all the way to implementing transparent feedback loops and "explainability mechanisms." RAI doesn't just happen by accident! Safety has to be planned at the design stage, NOT at the reactive last phase, 2 days before shipping! (IYKYK.)

3) Agents are exciting! If we're still figuring out the AI era, the "agentic" era is even wilder! We can make even further progress through contextual data, APIs, and multi-agent orchestration. Gradually we can transforming reactive systems into proactive agents to solve complex problems for real users. Along with the complexity of the problems Agents will be able to solve, the building of Agentic systems is even more complex! Rather than just thinking of one LLM call, we need to think about different parameters that all the agents will use. AND how to do this responsibly.

It was a fun talk, and a great conference! It was also a real treat to learn more about Nigeria. Since (for me on PT) it was an early-morning conference, I decided to brighten up my day by wearing one of our own designs, a beautiful shirt hand-made by one of our tailors in Lagos [for our Men's shirt line, Towards Arcadia].

Tosin pointed out that my shirt was made with Adire fabric, a type of beautiful batik traditionally hand-made in his home state.

Made me really miss my time in Nigeria. Hope to get back there before too long--hello my Naija friends, I think about you often!

(Ok, and suja & jollof rice; not going to lie.)

Thank you again for inviting me to talk AI with you!!

(RAI 22/100)

User Avatar

Jeremiah Marble

1y ago

I shall call #RAI 21/100 "If you give a tiny cat an orangeโ€ฆ."

So there I was. It was yesterday.

I was trying to make a specific image. A Tiny Cat Eating an Orange.

Why? The reasons have been lost to the sands of time.

Also, thanks to ChatGPT I learned that CATS CAN'T ACTUALLY EAT ORANGES.

Oranges are toxic to cats.

This is an important note to the tale I am telling. Wait for it.

Today, I'll take a chance.

I will show the world my exact process, trying to generate an image of "A Tiny Cat Eating An Orange" using ChatGPT.

The prompting process was SO PAINFUL.

Let me share the full transcript with you! Link in the comments.

I am reaching out to you, the community of AI experts and tiny cat lovers!!

How would Y'ALL have done this (what tools, different prompts, etc) to get a better result, faster?

In the comments below, show your final image; tell us what you did to create it!

Bonus points for having a look at my prompting, and letting me know what you would have done differently.

There MAY be a t-shirt in it for you (of your own design, of course).

The lawyers tell me that I can't promise this. But I am not a lawyer.

Only criteria: your design needs to be a Tiny Cat Eating an Orange.

What epic image ya got? How'd ya get it?

On y va!

User Avatar

Jeremiah Marble

1y ago

Happy Monday, friends! Here are 7 places where we need to be careful using GenAI. (#5 will shock you!!)

OK IT'S MONDAY AND I WANTED TO HAVE FUN W/ THIS ONE FOR #RAI 20/100! :)

We just sent out our weekly "3rd Rodeo AI" newsletter. I thought it was a fun one that might be worth sharing on LinkedIN. I'll drop the link in the comments if you want to read the full thing.

I mean, we're all agreed - GenAI is going to change EVERYTHING, right?

It helps me write new docs, research new topics, come up with creative ideas, and get into locked doors urgently!

(That is a story for another day.)

HOWEVER, using these tools might not always be the right call. (That's part of what I'm trying to get at, with the whole 'building & using AI responsibly' thing.)

Remember: GenAI is made to APPEAR correct to humans - which is NOT the same thing as BE CORRECT.

Important distinction, which we can often miss.

We've seen lots of examples (especially in law and medicine) where folks have gotten this REALLY WRONG.

There are a few others to call out (we list 7 of them in the newsletter, but there are more). Be careful using GenAI

1) when you NEED the CORRECT answer (more importantly than a well-written response) - GenAI is designed to be "truthy"

2) when you need to know that you'll get the same OUTPUT each time you enter the same INPUT - GenAI is designed to be "non-deterministic"

3) when the stakes are high (similar to #1), but nuanced

4) when you want to keep your data ENTIRELY safe - GenAI is designed to use ALL THE DATA

5) when you want to create original content - GenAI may slurp in & plagiarize other work

6) when it's a nuanced or "deep" field - GenAI might accidentally create errors only an expert can catch

7) when you need an exact numerical calculation - GenAI is designed for "human-like" answers, not being Data from Star Trek

Anyhoo - happy building & using AI systems responsibly!

Un abrazo fuerte mis panas!

User Avatar

Jeremiah Marble

1y ago

Hi friends! I'm excited to see some of you at the "๐—”๐—œ & ๐—”๐—ป๐—ฎ๐—น๐˜†๐˜๐—ถ๐—ฐ๐˜€ ๐—ง๐—ฟ๐—ฎ๐—ป๐˜€๐—ณ๐—ผ๐—ฟ๐—บ๐—ฎ๐˜๐—ถ๐—ผ๐—ป ๐—–๐—ผ๐—ป๐—ณ๐—ฒ๐—ฟ๐—ฒ๐—ป๐—ฐ๐—ฒ," taking place virtually December 13-14. We'll cover the future of work, business, and education - re-envisioned for the AI era (aka the 3rd Rodeo)!

This year's conference will have keynotes, fireside chats, and demos on the latest & greatest in AI.

Other speakers include Tanya Singh, Kanishk Gaur, Mo Medwani, Dorte Dopfer, Michael Glaros, Adewale Yusuf, Gamaliel Adun, Theophilus Fakiyesi, Okeowo Aderemi, Fike Adaba, Fadรจ Adeniyi, Orimolade Oluwamuyemi A., Moshood Yahaya, Kiran Kumar Jwala, Kuti Israel, and Oluwadamilare Oladele.

I'll post the registration link down below in the comments - join if it works for your schedule.

Looking forward to seeing you then!

(Since I'll be covering "building (and using) AI systems responsibly," you'd better bet that I'm including this as #RAI 19/100!! I'll go more into what I'll be talking about a bit later this week!)

Un abrazon, y pa' 'lante!

User Avatar

Jeremiah Marble

1y ago

I'm trying to write each day about "building (and using) AI systems responsibly." Today, for #RAI 17/100, I shall throwback to an earlier post I wrote, about my totally just-for-fun creative outlet of making fun t-shirts, sometimes using AI.

Since I use GenAI to make SOME of the shirts (as I described earlier), I'm counting this post as well. (HA!, I shout into the void.) Besides, I do try to be responsible as I generate images of foxen and honey badgers and murder geese.

Let me be clear. To make my t-shirts, I have the jankiest process ever.

First, I come up with the words / phrase I'd like to use. Sometimes I "borrow" a friend's phrase - with their permission, of course. More frequently, I'm seeing these phrases "in the wild."

I'm a phrase-hunter these days! I've been shocked how, now that I'm open to it, the universe has been putting LOTS of great phrases in front of me. Thanks, universe. I'm trying to pay attention, these days.

Then, I figure out what image I'd like to use. Often I'll turn first to GenAI (usually copilot.microsoft.com, but occasionally others). Sometimes, I can't quite get the image I have in my head - and I pay an artist for their own work, which also feels pretty right.

Next up is ... wait for it ... PowerPoint! Yes, ever since my strategy consulting days, I've found that I STILL think in slides. Plus, I know the tool super well, and - again - I'm not doing anything too fancy. So, shipped > perfect.

Then, I put it on my website, make a few edits, et voilร !

So - anyway, I have a few designs up. The attached image is one of them

Interested? Ping this thread! I'll DM you the link.

Un abrazo!

User Avatar

Jeremiah Marble

1y ago

It's non-trivial to build (or even use) AI systems responsibly. As far as I know, every single tech company working on AI has stumbled at some point. For #RAI 15/100, here's a recent one by OpenAI."

In their Dec 2nd article covering it, Ars Technica tagged this story as "He who must not be prompted." Cute.

Ars Technica called out four names that folks found broke ChatGPT: David Mayer, Jonathan Zittrain, Jonathan Turley, and Brian Hood.

I was curious, so I ran each of the names through ChatGPT myself. Looks like they fixed David Mayer in response to the coverage, but the other three seem still to be broken.

As a very simple next test, I ran them all through copilot.microsoft.com. They all worked! I could have done the same on some others, but - just a quick post today. Give it a try, if you like to break things like I do!

I don't work at OpenAI and have absolutely no inside knowledge.

But I'll bet I know what happened. Likely, a convo like this:

<someone in legal, or sama, etc> "Hey, these people filed suit against us. We should make sure that we keep their names out of ChatGPT's mouth."

<someone in product> ... oh wait it's a startup....

<someone in ENG> "I checked in the code. No more problems with those names."

<legal / exec> "Sweet."

I mean, yes, problem solved - AND they got some free press out of it!

So we'll tag this as a WIN.

Onwards, RAI friends! Happy Thursday.

User Avatar

Jeremiah Marble

1y ago

I made a promise to myself that I'd try to publish every single day for at least 100 days, at least one thing about #RAI. Today I've been heads down on doing actual work, but keeping my promise again today: here is #RAI 14/100.

As you know, I'm using OpenAI's Prompt Playground for my experiments. This gives me additional flexibility beyond "just" the ChatGPT interface, without making me spend time (yet) on standing up my own chat interface UX, for example through writing a web app and hosting it on Azure.

Eventually, that's exactly what I'll do - but there is a lot of good learning I can do before I take on the additional overhead of dealing with Azure quotas and instances and, you know, writing the code. That is a task for later.

But I'm making progress today. In this post, let me go a bit deeper into what I'm doing.

With the Playground, first I select the underlying LLM, which is going to give me a response to my "user" prompts. I'm currently using GPT-4-mini for cost savings, but of course for prod I'd evaluate several models to see what was best for what I was trying to do.

But - remember - I want to test MY "Boxes & Foxes" chatbot, which aims to help non-traditional founders start successful businesses. Just returning GPT-4-mini output wouldn't teach me a bunch I couldn't get from ChatGPT.

One level beyond which model I select are the modifications I make to the "plain vanilla" its responses. These are things like temperature (which I started at 0.5), top-p sampling (which I started at 0.9), and max tokens (which I started at 1000-1500).

I'll want to experiment also with these settings, to make sure that I'm optimizing my chatbot for my own specific purposes.

The next level I'm testing is a high-level Retrieval-Augmented Generation, or RAG. To do this, I enabled Code Interpreter, and provided it with a text file of "Model 47" as my grounding document.

I have LOTS of learnings & thoughts, but TBH I'm heads down in actually doing this stuff today, so I'll seek to tease them out later.

Onwards, friends! Hope that you're having a great Wednesday.

User Avatar

Jeremiah Marble

1y ago

I've been working with my chatbot (on Prompt Playground), looking at building & using AI systems responsibly. Focusing for a bit on Fairness. Currently heads down so just a quick checkin today. #RAI 13/100

For Fairness, I want to ensure my "Boxes & Foxes" chatbot gives the similar recommendations and advice to entrepreneurs with similar situations.

For example, someone who asked for advice on the "Model 47" process - going from an loose idea to a profitable biz - things like their gender, geography, and economic status shouldn't matter. [I should know, since I wrote the book.]

So literally every person should get some version "how to M47 your biz."

By design.

This is what I'll be testing!!

Wish me luck - and please send me any ideas you might have on how to make this better!

Note to my dad: I've noticed that, when I use screenshots of my chatbot (or the like), LinkedIn's algo seems to put it in front of fewer people. So I'll use a few random pictures! GrumpyFace FTW!

User Avatar

Jeremiah Marble

1y ago

The other day on #RAI LinkedIn, I came across an interesting paper about a concept called "TIBET," which talks about detecting and analyzing biases in text-to-image models. Thought it'd be fun to have a look in #RAI 12/100.

TIBET stands for "Text-to-Image Bias Evaluation Tool," which is a great name!

I think this research is pretty useful to go through, even if you're not an academic (which I'm not & never will be again)!!!

Just the way they're tackling the problems of identifying biases in these models - and offering methods that are comparable over time and across AI providers - is helpful to go through.

They take a prompt and then look for things they call "bias axes." A "bias axis" is a category representing potential biases (like gender, ethnicity, or age), which can influence which images text-to-image models generate.

These bias axes vary, according to what's in the prompt. TIBET helps suggest these, using GPT-3 and counterfactual reasoning.

What is counterfactual reasoning? - you might ask (I did).

It's taking a prompt, then coming up with alternative prompts (counterfactuals) by changing "bias axis" attributes in the original prompt. Then you run all the prompts through the model, and compare the resulting images to identify biases in the model's outputs.

The researchers also come up with quant metrics like Concept Association Score (CAS) and Mean Absolute Deviation (MAD) to quantify these biases.

It's inspiring to see frameworks like this one, helping to enhance fairness in TTI models, integrating with bias mitigation tools, and promoting ethical AI development despite challenges like inherited biases from its components.

Next up (at some point in the future) - figure out how to use it for a real project!