User Avatar

Tim Urian

Software Engineering

4y ago

Writing about programming and learning.

If you want to change customer data for any reason a "quick script" can be disastrous.

When someone trusts you with their data they make you the keeper of their digital life. A hastily constructed bit of code can cause you to lose data, and your reputation.

Data mistakes can lead to:

• Bad data that bites you in the distant future

• Problems that affect the performance of your application

• Code being harder to ship and reason about

• Useless orphaned data that will never be cleaned up

When my script crashed halfway through the update I thought I was toast.

How did I not imagine this could happen? It wasn't even a flaw in my code, my remote session on the server dropped. How was I going to tell my manager?

After hours of tedious data manipulation my "quick fix" was finished with no loss of data or trust - but it was clear my workflow had to change.

Migrating and modifying live data is integral part of being a software engineer, and here are the two most important rules I use to do it safely:

Write scripts so that they can stop, be resumed, and be completely undone.

If I had known then what I know now I would have been disciplined in my script's ability to crash and resume.

Luckily, I had been sure to back up all critical data. Persisting your script's state, or logging work as it is done can make your script robust against interruption.

Programs like tmux and screen can keep your session alive on the remote server itself and provide another level of safety.

Have your 'one-off' scripts reviewed like any other code.

In software engineering careful beats quick every day of the year.

For non-trivial changes, create permanent, robust scripts that go through the review process. Bonus points for putting them in a shared repo so that future engineers can see what's been done.

Why forgo the benefits of peer review when faced with the high stakes changing data can represent?

With these two tweaks I've had:

• Less time spent fixing foot guns

• More trust in the integrity of the data I change

• Easier time identifying bugs

• And less stress at work!