AWS Step Functions: how we orchestrated a complete IoT pipeline without a single line of control logic

The problem we wanted to solve

Before bringing Step Functions into the pipeline, the situation was one that many teams will probably recognize:

• Glue crawlers and jobs were run manually, waiting for each one to finish before launching the next step.

• The orchestration logic lived inside the code: conditionals, sleeps, hand-rolled retries.

• If something failed midway through a pipeline spanning multiple services, catching it in time was hard.

• Monitoring the status of each execution meant jumping between multiple consoles and cross-referencing logs.

The complexity didn't come from the data itself. It came from how all the services were wired together. And that's exactly the kind of problem Step Functions solves.

The architecture we built

For this Devtalk we built a PoC of an end-to-end IoT pipeline. The goal was to have data flowing from simulated sensors all the way to a visualization dashboard, fully orchestrated and automated.

Data generation (simulated on-prem layer)

A Lambda function triggered by Amazon EventBridge generates 100 records per minute, simulating 50 industrial sensors spread across 3 plants. Data types include temperature, humidity, pressure, power consumption and vibration, all in JSON format.

Storage layer: S3 in three stages

The data moves through three S3 buckets with well-defined roles:

• devlights-datalake-raw: raw JSON exactly as it arrives from the sensors.

• devlights-datalake-processed: clean, validated and partitioned Parquet.

• devlights-datalake-curated: data aggregated by hour and day, ready for dashboards.

Processing: AWS Glue

AWS Glue handles the transformations across three layers:

Governance Layer: three crawlers that discover and catalog the data in each bucket.

Data Catalog Layer: separate catalogs for raw, processed and curated, which let you query the data with Athena with no extra configuration.

Processing Layer: two ETL jobs:

• Raw to Processed: cleaning, validation and conversion to Parquet format.

• Processed to Curated: time-based aggregations to feed the dashboard. Both with job bookmarks enabled, which ensures each run processes only new data.

Visualization: QuickSight

The final dashboard in Amazon QuickSight shows trend charts over the curated data. From there, the team can see how the sensors behave without touching the raw data.

AWS Step Functions: the glue of the pipeline

With the architecture defined, the next challenge was orchestrating all of this reliably. That's where Step Functions comes in.

What is it?

AWS Step Functions is a serverless workflow orchestration service. It lets you coordinate multiple AWS services through state machines: visual flows where each step is a state, each state has an input and an output, and the transitions between states are explicitly defined.

The features that mattered most to us:

• Automatic error handling and retries, with no hand-rolled retry code.

• Graphical visualization of each execution in real time from the AWS console.

• Native integration with Glue, Lambda, SNS, S3 and virtually any AWS service.

• Pay-per-transition pricing, with no fixed cost. You only pay for what the pipeline consumes.

The four key concepts

Before looking at how we implemented it, it's worth being clear on the basic concepts:

The State Machine we built: etl-data-team

The etl-data-team state machine controls the full sequence of Glue crawlers and jobs to move data through the three stages of the pipeline:

RAW → PROCESSED → CURATED

The flow in detail:

1. Start Raw Crawler: discovers new data in the raw bucket.

2. Wait for Raw Crawler: waits for the crawler to finish (native Step Functions polling).

3. Start Raw-to-Processed Job: launches the Glue job that cleans the data and converts it to Parquet.

4. Wait for Raw-to-Processed Job: waits for the job to finish.

5. Start Processed Crawler: catalogs the newly processed data.

6. Wait for Processed Crawler: waits for the crawler.

7. Start Processed-to-Curated Job: runs the time-based aggregations.

8. Wait for Processed-to-Curated Job: waits for it to finish.

9. Start Curated Crawler: catalogs the curated data.

10. Success / Fail: the execution ends in one of these two states.

Each step waits for the previous one before moving on. That's what used to be done manually or with ad-hoc logic in the code.

Automation and notifications: EventBridge + SNS

A State Machine you have to launch by hand doesn't solve the root problem. That's why we integrated it with two more services:

Amazon EventBridge

We set up a schedule rule in EventBridge that triggers the State Machine automatically once a week. A single rule: no servers, no cron jobs, nothing to maintain.

Simple Notification Service (SNS)

Two additional EventBridge rules monitor the outcome of each execution:

• success-email-etl-data-team → emails the team when the pipeline finishes successfully.

• error-email-etl-data-team → sends an alert if anything fails at any point in the flow.

The team doesn't need to open the console to know whether the pipeline ran fine. The information comes to them.

Before and after: the real impact

PoC costs

One of the most frequent questions when evaluating Step Functions is cost. For this PoC, the numbers are:

*Step Functions charges per state transition. For low-frequency workflows like this one (once a week), the cost is practically zero within the AWS free tier.

Conclusion

Step Functions isn't a tool for simplifying code. It's a tool for moving coordination logic out of the code and putting it where it can be seen, monitored and maintained independently.

For data pipelines with multiple services in sequence, like this ETL with Glue, the value is immediate: every step is explicit, errors report themselves, and any team member can open the console and see exactly what state each execution is or was in.

"Not all complexity comes from the data. Sometimes it comes from how you connect everything. Step Functions solves that."

If your team works with data pipelines on AWS and still handles orchestration manually or with logic scattered across the code, Step Functions is probably the lowest-friction, highest-impact change you can make today.

Want to implement something similar in your organization? At Devlights we can help you design and build the data architecture you need. Let's talk.