A long time ago, in a job far, far away I made my first serious professional mistake. I deleted 10 million image files on a website that received about 4 million page views per day. And I did it in the middle of the day. Oops...

I was learning to write shell scripts for Linux, and I needed to reformat the file names of these image files to make them work consistently on Windows and Linux. Perfect. I'm learning a skill and here is a problem to solve.

My script was about 95% working in a test directory and I thought I knew the fix. I made a "small" change to the script and decided NOT to test it and just run it in production. I executed the script.

The script was rapidly deleting files and I had no idea what was going on. Part way through, I decided to cancel the execution and check the results. Almost at the same time our monitoring team called me to tell me that the images were not loading on the site. I checked and found that I had deleted about 50% of the files with my "small" change to the script.

Some mistakes are so scarring that you experience an almost physical reaction when you come close to remaking them. What I learned from this one has served me well for my entire career.

Here's what I learned:

Lesson #1: Always test after making a change to something

This was a classic trap. I only made a minor change, so it should have a minor impact. Nope.

No matter how small the change, test what you are doing every time before running it in production.

Lesson #2: Don't make major changes during peak utilization

It was in the middle of the day when people were using the site steadily. Making this change at this time was a dumb idea. I should have waited until we were out of peak traffic and the impact of any mistakes would be smaller.

Lesson #3: Always have a way to recover

I lucked out here. I had made a backup of the files a couple of days earlier to another server. Recovery was as simple as reconfiguring 6 web servers to get their files from this server. It was a high impact event, but had a fairly short duration.

Now I always make sure I have a path back to where I started.