Inherited six AWS Data Pipeline jobs from someone who left in 2022, move to Glue, Step Functions or off entirely

On weeks: budget for the archaeology, not the writing. The rewriting of five copy jobs is a couple of days. Working out what the old definitions actually do, which of the columns are still used downstream, and why there is a mystery transformation in step three that somebody added in 2021 is where the time goes. We gave ourselves two weeks for a similar job and it took closer to six, all of it in reading and asking people what a table was for.

Split the problem by what each piece actually is. Orchestration, meaning what runs when and what happens on failure, goes to Step Functions since you already run it and your team already knows the failure modes. The Spark shaped work, meaning the EMR job, is the only piece Glue is genuinely for, and Glue is worth it if that job is real Spark rather than a shell script that happens to run on a cluster. Do not put simple S3 to Redshift copies into Glue just because it is the named alternative, you will be paying for a Spark environment to run something Redshift can do itself with a COPY and a manifest.

First thing to know is that the decision is partly made for you: AWS Data Pipeline is closed to new customers and the service is in maintenance mode with no new features or region expansion planned, and AWS publishes its own migration guidance pointing at other services. Existing pipelines keep running, so there is no fire, but you are maintaining something with no road ahead of it and no new hires who will have seen it. Your five copy jobs are the easy part and should not be treated as a migration project at all, they are a scheduled COPY statement. The EMR job is the one that deserves actual thought.

We moved a similar sized set to managed Airflow because it was the thing everyone had heard of, and the standing cost of the environment was the surprise: it bills for being available, not for the four minutes a night we used it. For six jobs that is a poor trade and we ended up back on Step Functions plus a couple of container tasks about eight months later. If your workload is bursty and small, avoid anything with a permanently running control plane you pay for.

Run the new pipeline in parallel for two weeks and diff the outputs before you switch anything off. Cheap, boring, catches everything.