Making LLMs more effective by filtering haystacks for needles
ChatGPT Search can answer very specific questions. Don't be satisfied with the median response to the median question. Ask for what you want! Read: "Everything Starts Out Looking Like a Toy" #226
Hi, I’m Greg 👋! I write weekly product essays, including system “handshakes”, the expectations for workflow, and the jobs to be done for data. What is Data Operations? was the first post in the series.
This week’s toy: (this week’s toy). Edition 226 of this newsletter is here - it’s November 25, 2024.
If you have a comment or are interested in sponsoring, hit reply.
The Big Idea
A short long-form essay about data things
⚙️ Making LLMs more effective by filtering haystacks for needles
OpenAI recently introduced a new way to search using their LLM technology: ChatGPT Search. The benefit? Making it easier to add web results and the context of those results alongside your questions to ChatGPT. And no more answers like “As of my knowledge cutoff in October 2023, I don’t have any records of …”
Instead of LLM hallucination, ChatGPT Search promises flexible searching that will meet your needs by using the index to provide additional context to your requests. I called this activtiy a “multi-player” application because making ChatGPT Search works for you requires more work than just asking for a result.
It’s not an API Machine
Put on your product hat. When you’re thinking about the benefit that you want as the PM for a search product, you want to maximize the utility of having (what is essentially) a very recent copy of the Internet while limiting the number of GPUs you are keeping active by doing LLM queries.
That means you’re going to cache results aggressively and discourage searches that look like iterative list building activity and scraping.
Yes: “find me a high ranking sushi restaurant in Seattle, WA”
No: “find me the top 20 high-ranking sushi restaurants in Seattle, WA based on their Yelp ranking and filter for those that have amazing Yellowtail sashimi”
The frustrating thing about the result from question two above is that the LLM has all of the information to make this inference. The first question will yield a pretty good result (I got Sushi Kashiba, which looks amazing but is definitely not $$ out of $$$$, ChatGPT. Omikase there is $200+)
ChatGPT doesn’t have any concept of “find me a comparable restaurant that’s good for $30/person” or similar. To do this, you’d need the kind of result you get from an API, where you could specify the attributes that matter the most to you.
Something more like:
can you find me an ASIN for an otterbox iPhone 13 pro case and return all of the details as a json packet?
ASINs are the shorthand for Amazon Standard Identification Number, a unique 10 character code for products that show up on Amazon.com. Since LLMs are excellent at pattern matching, choosing a common pattern is a great way of constraining the possible answers delivered by the LLM in response.
Asking for an ASIN in my result gives me this JSON:
Go visit the URL if you like - this is a pretty solid result. It also added a lot of data attributes I didn’t know were available that I can now ask ChatGPT Search to use to give me more interesting results.
However, ask ChatGPT for a repetitive list (give me 10 like this, 50 like this), and you will likely get an uncompliant or hallucinating result.
Why?
You get the median result (which is good)
The product is oriented toward a Pretty Good Result – It gives you a conversational response to a traditional search query – and is intended to compete with Google, not with a search API. But it’s sooooo tempting to look at the way you can make this respond and wonder how to get it to react in a more structured way.
Think of this revised product goal as applying RAG to search. You want a structured query based on very specific information, returning information you could validate.
Imagine a problem statement like:
Review the top 100 Otterbox products that are ASINs on Amazon.com and produce a report on the commonalities on the highest ranking items that deliver the best value, where ranking is expressed by the category ranking and value is the price relative to the median price for those top 100 items.
All of a sudden the search index built by ChatGPT promises to be one of the most powerful analytic tools ever built. (If it’s not hallucinating.)
I’m really looking forward to the API that delivers access to ChatGPT Search.
Finding needles in haystacks
Until that API is available, there are a few techniques I’m using to make my ChatGPT Search results more effective. Think of this as finding the information needle in the search haystack. Instead of taking the default result, I’m asking for very specific outcomes that I can prove.
A JSON output is a very common structure with lots of examples that ChatGPT Search can mimic
ChatGPT is very compliant in providing structured data as an output to your search, so feel free to add a query to your JSON result like: “for each of these attributes, add an object indicating the URL where you found this information and the date of the freshness of the search.”
At worst, this gives you a reference to visually confirm the information shared by the search engine, which is pretty much what you would do with a Google Search anyway.
The difference is that if you can compose an exacting search it’s a lot easier to get a result or a series of results and then challenge the search engine to prove that it was a good result. ChatGPT Search (just like Google) will prefer to deliver a cached card of results or a highly ranking compilation post, but keep asking it questions.
You won’t be disappointed. Just be specific. Ask ChatGPT to remember the instructions you used for this search and name that search, e.g. “ASIN result” so that you can prompt in the future for a similar query.
What’s the takeaway? ChatGPTs Search widget can give you powerful results in response to your search queries, and they get even better if you ask for filtered answers. Be cautious and ask the index for relatively few things at a time (maybe only 1) and ask it to use that format for future, similar searches.
Links for Reading and Sharing
These are links that caught my 👀
1/ AI is Hungry for Attention - Ben Evans shares his summary of the future of AI.
His annual presentation, always worth a read, is particularly provocative this year. In a year or two, will everything change about most knowledge work? Or will it be kind of the same with new features? Tune in next time (same Bat-Channel).
2/ The End Of Productivity - Sari Azout makes a pretty interesting claim that writing is about associative thinking and great ideas. Specfically, she suggests that the tools we need look more like just in time coding assistants, but for writing. Still unclear - how do you identify and define your own true style if an LLM is guiding part of the search?
3/ College students no longer read books - This blurb should probably be called “Old Man Yells at Cloud”, if you think reading physical books is important for literacy. Reading a book a night – as was expected when I was a grad student – is no longer needed to advance your academic progress (and maybe, it never was.)
What to do next
Hit reply if you’ve got links to share, data stories, or want to say hello.
The next big thing always starts out being dismissed as a “toy.” - Chris Dixon