5 ways to make your low-code automation more effective

Even if you're not an engineer, use engineering practices to make low-code automation work better. Read: "Everything Starts Out Looking Like a Toy" #170

Nov 06, 2023

Hi, I’m Greg 👋! I write weekly product essays, including system “handshakes”, the expectations for workflow, and the jobs to be done for data. What is Data Operations? is a post that started this exploration.

This week’s toy: a AI-powered machine that creates videos. What’s different about this? It’s a physical box with levers and dials where you start by specifying the frames in the video storyboard (h/t to Marshall Kirkpatrick for the find). Edition 170 of this newsletter is here - it’s November 6, 2023.

If you have a comment or are interested in sponsoring, hit reply.

The Big Idea

A short long-form essay about data things

⚙️ 5 ways to make your low-code automation more effective

When I started my first software job, I remember thinking two things:

I am definitely not the smartest person in the room
The engineers who seem smarter are people who create, explain, and follow process that makes their work predictable and understandable

I seek out teams where I am not the smartest person in the room, and I’m still trying to apply software engineering principles to the work that I do so that at some point I can look back and say “I’m a professional engineer.”

Today I’m a data plumber. Diagnosing an issue, adjusting an existing automation, or creating a new workflow are all activities that benefit from software engineering practices.

Let’s not get carried away: we’re talking about the 20% of things you could do to improve your work that will provide most of the benefit.

Here are five ways to apply software engineering to your no-code and low-code workflows so that other people will see your work as predictable and understandable. These methods also make it more likely that your automation will work. 🫡

1. Write out what you want to do before you do it

Open a document in your editor of choice (I use Google Docs) and write the process from end to end. What’s the end goal, and what needs to happen to take the initial state of the process from start to finish?

Stating clearly what you’re trying to do is an underappreciated step. If it doesn’t make sense now, it will not make sense in code or in a no-code process.

2. Each piece of code has one job

It’s tempting to build everything in a monolithic function, said mostly no one ever. But that’s what you’re doing when you use a lot of no-code tools and try to do everything all at once. This practice makes it very hard to debug when something goes wrong.

What’s easier? Building a more modular process – made easier by code orchestration tools like Pipedream – where you define the event trigger that starts the work and follow it through each individual step.

Search_for_records, determine_the_update, and update_records are examples of easy-to-read pseudocode step names to transfer from your initial design to the abstraction where you describe what actually happens.

3. Build testing into the process

“What could possibly go wrong” is a good place to start your testing. When you anticipate the most common things that will break in each step, it’s easier to catch them and refactor your process.

What are the most common user errors when you’re building this sort of automation?

For me, they are:

poorly scoped variables - when you think you’re getting a JSON key and you’ve targeted the wrong part of the structure
control logic that misses a case - when you are looping a collection and haven’t determined the possible outcomes
calls to external APIs that don’t resolve in expected ways - when you call an API and have to wait for it to have completed an action, you might need an additional call to complete the task

4. Self-document the process

“What’s actually happening?” If your automation reports the current progress, it’s a lot easier to see what happened and where it went right or wrong.

Use console.log or another method to document important milestones so that you can play data detective after the fact when things don’t go as expected.

5. If possible, use source control

One of the things I really like about Pipedream is that it integrates directly into GitHub. This makes it possible to treat your low-code process much more like a software project:

You can work on a branch while the existing automation continues
It’s possible to ask other people to help in reviewing PRs
When you need to revert your change, it’s an easy process

Source control can seem like extra work, but future you will thank you. (Especially when you leave decent comments.)

What’s the takeaway? There are a few process steps you can follow to make automation easier to understand and diagnose. When you document how the process should work, log how things are working, and create a way to roll back changes, you’ve improved the odds of building successful data plumbing.

Links for Reading and Sharing

These are links that caught my 👀

1/ Turning waste into art - this story describes artists in Ohio who built a process to remove toxic iron oxide from rivers and recycle it into art pigments. Artists often see things other people do not, and this is a powerful statement about improving the environment.

2/ Make your diagrams better - If you’ve ever looked at a diagram and thought … that doesn’t look quite right, this is for you. The team at Terrastruct put together some concrete solutions to make diagrams better.

3/ The many reasons why you are not fast - Allen Downey on the distribution of running speeds. No matter how fast you are (or how good you are at something), there is someone better than you are most of the time.

What to do next

Hit reply if you’ve got a topic you’d like to see in a future edition of this newsletter or want to say hello.

The next big thing always starts out being dismissed as a “toy.” - Chris Dixon