Without webhooks, Saas apps wouldn't know how to get work done on demand
"Everything Starts Out Looking Like a Toy" #106
Subscribe now to join smart, curious folks who get the Data Ops 📊 newsletter
Hi, I’m Greg 👋! I’ve written 100+ essays on product, data, and process for an idea or future product direction, often data between systems. What’s a key topic? System “handshakes”, the expectations for workflow, and the jobs we expect data to do.
Read more: What is Data Operations?
This week’s toy: build an Atari 2600 game console from LEGO. There’s only one thing missing: trying to clear the cartridge of dust when it wasn’t connecting well.
Edition 106 of this newsletter is here - it’s August 15, 2022.
The Big Idea
A short long-form essay about data things
⚙️ Webhooks let Saas apps get work done on demand
When you use an application, it’s easy to forget how work gets done behind the scenes. Clever UX designers focus your attention on Jobs To Be Done, and as a user of an application, you probably don’t even start to think about the workflow inside the application. From the front-end actions like clicking a button through the middleware that brokers that request to a database on the back end of the application, virtual handshakes happen between systems to produce the results you see on the screen.
One of the most important handshakes is when your application needs to communicate with another SaaS application. For example, your app might need to get some supplemental information by sending a value (like the email of a contact) to another system, retrieving the current company associated with the domain of that email address. Your app probably made that request in one of two ways:
by using a GET request from a webhook provided by the other application, passing the email address as a payload to that hook and receiving a response back. If the webhook request fails, you’ll need to ask again. Most of the time, this works flawlessly. (But when it fails, you might not know.)
by using an API call to do the same thing, passing a structured bit of information to a specific call and receiving a result back. When this succeeds, it works the same way, and when it fails, you get a synchronous result.
Do you always need to know in real-time if a request succeeds? Not really. There are certain requests where you must know the result, for example when you create a thing or delete a thing. But enrichment of information or other kinds of requests that are secondary don’t always need to complete immediately. However, you do need to know when they are completed. That’s why keeping track of your requests is critical.
Keeping a ledger
Workflow is messy. Typically we have a calling system sending a payload, a receiving system that authenticates and authorizes the request, an evaluation of the payload, execution of the action, and the return of a response. When it happens between systems, we need a kind of ledger to request and receive work. Applications use a system of queuing requests, where the request is registered on receipt, and then completed in the order it was received.
Many applications accept requests through “webhooks", and let’s talk about what that means. At its core, a webhook is a unique URL generated by the application that enables a calling application to POST or GET a payload of information to that application using a known schema to kick off some work. In simpler words, webhooks are like a dedicated phone number for the calling application to address with the information you want to send to the application.
What is a webhook?
A webhook is a dedicated URL created by the application that specifies:
the address to call (the URL)
the method to use (POST or GET)
the expected fields that will be included in the payload, including any specific information on data types
the format for the response that will be sent back to the calling application
If needed, filtering on the clients allowed to call this webhook URL
With a webhook, an application is able to expose more surface area than a traditional API. Because webhooks are a flexible way to call and receive information, there is less effort required to push and pull information than with an API. Webhooks have a key drawback: they take and fulfill requests only, and don’t maintain state.
Webhooks are not a state machine
Unlike an API, where transactions might have lots of states to let you know what’s happening, a webhook is a relatively simple operation. There’s a queue of information, and the submitted information is consumed from the stack of messages in the queue. When the queue is drained, the system is ready to process more information. Webhooks accept a request for work. They are not a guarantee of work completion so they need to be handled a bit differently than an API request.
Basically, this means that you need to call the webhook, and wait until it completes. If it doesn’t complete, you need to check the queue to ensure it’s been picked up, and then potentially call it again. This works well if your underlying system knows how to handle a duplicate request. It doesn’t work as well if the order of operations is super important.
To summarize, webhooks are good at being dumb pipes. This works well for one time actions that need to be completed asynchronously. Webhooks work less well to complete a real-time update where you need to know the transaction completed before continuing.
Gluing together data from multiple systems
Webhooks excel at providing data from many systems into a common format. This means that when you’re building a “golden record” combining concepts from multiple systems into a unified lead or contact or account record, or building a custom object to aggregate information, you can use webhooks to fill in the information you need.
Combining webhooks with a messaging queue gives you a flexible way to make requests, receive responses, and clear them out of the queue without building tightly coupled integrations.
On Demand Requests vs Integrations
Enabling users to do flexible things is a key goal for every product manager. Adding a webhook feature lets you delegate the integration to a moderately technical user who can “connect the dots” to transfer from or push data to other applications. With this setup, you can support many more integration scenarios than a tightly-designed one.
Integrations have their place, specifically when the connection needs to make no mistakes, to recover from errors, and to know in real-time what’s going on. That’s why certain types of connections don’t work well with webhooks.
What’s the takeaway? Webhooks are a flexible form to request and receive data. Because they use a common format, they are easy to integrate with existing applications and retrieve information from or put information to systems. Take care in using webhooks to hold state. You’ll need to handle this part of your application yourself.
Links for Reading and Sharing
These are links that caught my 👀
1/ Difference between webhooks and APIs - you might look at webhooks and think: “that’s an API”. There are some key differences between APIs and webhooks to keep in mind, writes Pragati Verma. The major difference? Webhooks send a one-way message, while APIs (application programming interfaces) maintain a promise to complete or provide an update on a transaction once it is started. Now that you’ve learned a bit about webhooks (or if you’re an experienced users or developer of webhooks), you might notice this distinction is starting to blur.
2/ If you’re starting a data team… - Need to learn more about the best practices of data-driven orgs? Emilie Schario has compiled a great resource on building data driven orgs from a variety of sources. These essays cover topics from building organizations, creating a data strategy, helping teams come up to speed, and more.
3/ Need a place to test your hooks? - If you need to know how your webhooks are coming across the wire, try webhook.site to set up an instant, temporary place to catch the information. You’ll set up a temporary hook URL, configure the payload, and get notified when your outside code (perhaps using Postman or a similar tool, perhaps using something else) sends a request to your hook.
What to do next
Hit reply if you’ve got links to share, data stories, or want to say hello.
Want more essays? Read on Data Operations or other writings at gregmeyer.com.
The next big thing always starts out being dismissed as a “toy.” - Chris Dixon