We can sleep in a storm

Finding inner peace and strength

Image courtesy of Carlos Alexandre

"He sleeps in a storm. He will never know of this. He is completely isolated, with a gulf of storm-waters around him. And now, to-night, he will be asleep before he can even say, 'I will sleep.'”

Moby-Dick, Herman Melville.  Chapter 132,  "The Symphony”.

In the beginning.

When we first started Flask Data, I did everything myself. 

During the first 3 months,  I wrote code, managed, configured and hardened Linux servers, setup backups and monitoring, and talked to potential customers.

And I did it without ChatGPT and Github Copilot.

Within 3 months, we had 3 paying customers.  I was getting tired of late nights.   

After a few incidents, my co-founder, Jenya, put her foot down and said: 

“I’m not allowing you to touch servers any more”.

A major step for me as a manager

I started looking for an outsourcing company that would sustain our server operations on AWS.    We didn’t have the budget for a fulltime head.

I found a company that was just starting out. They were 3 partners, all experienced systems engineers.

After interviewing them, I was impressed enough to hire them and negotiate a yearly contract. The group was highly professional, provided 24x7 online support, and the price was right. Over time, we came to rely on them and enjoyed their professional attitude and work.

Surprise - we’re firing you

After 2 years, I got an email from our account manager at the outsourcing company.  Let’s all him Pavel. Pavel writes me: 

“We’re terminating your contract with us at the end of this month”.   

I called up Pavel and said: 

“a) We have 30 days, not 5 days 

b) Why are you firing us? 

c) How do you plan to transition us to someone else?”

Pavel tells me:

“OK, you have 30 days.  We decided to specialize in K8s architecture consulting, implementations and support and get out of the AWS VPC DevOps business. We’ll work with you to transition DevOps to someone else”.

They were firing us to make more money.

We were no longer a lucrative enough client for them.

Over the next 3 weeks, they wrote documentation and we searched for a DevOps engineer who would report to my CTO.

More than a revolutionary militia of barefoot children

A DevOps Systems Engineer is a professional who specializes in combining development (Dev) and operations (Ops) practices to streamline and automate the software development and deployment process. They are responsible for designing, implementing, and maintaining the infrastructure and tools that enable a DevOps culture within an organization.

By now, we had 20 customers, and over 3000 patients participating in customers’ clinical trials on our SaaS platform for collecting clinical data from patients and connected devices.

Enabling a DevOps culture was an important step in our growth as a company.

DevOps culture is a collaborative and cross-functional approach that emphasizes communication, automation, and shared responsibility between development and operations teams to streamline software development, deployment, and maintenance processes.

We needed to be more than a revolutionary militia of barefoot children, selling, writing and deploying code.

Our goal was to onboard a DevOps engineer 2 weeks before our outsourcing provider terminated. The 2 weeks would allow enough time for our outsourcing provider  to move systems, credentials, cryptographic keys and documentation to our new person.

Finding an experienced DevOps engineer, with good communication skills and ability that fit in with the team under the leadership of our CTO was not easy.

The clock was ticking.

Reut

1 week before the deadline, we interviewed a candidate.   

I’ll call her Reut.

Reut passed the interview with flying colors.   She had a confident, sassy personality and we were sure she would work well with our engineering and customer services teams. She was available to start immediately.

I ask for recommendations.

She said - “here you go”, and handed me a sheet of paper, with 1 handwritten line on it.

“She sleeps well in a storm”.

I asked - “what does that mean?”

Reut shrugged and said - “I don’t know.  I guess it means I sleep well”.

We hired her.

Reut proved to be rock-solid on Linux infrastructure.  She set up CI/CD and improved our backup and patching processes. We talked to her on Slack. Always confident and sassy with a sense of humor.

We continued to grow.

We stopped paying attention.  Things just worked.

A year later. 

It’s a Saturday night, 230 in the morning.   I’m woken up by an alert from our AWS monitoring systems.   

I leap out of bed and grab my phone from the nightstand.

I read the AWS status page:

A power line to the AWS US EAST-1 data center in Virginia was cut by a tractor.  Storage services are down.  An unknown number of disks have been fried by the power surge.

Over 20,000 servers are down.

AWS reports on their status page:

We are working to restore services.   Over 90% of the servers are still down.   We expect to restore service to 95% of the servers by Sunday afternoon Eastern.

My first thought. Holy crap. We’re f-ked. Major disaster.

I call Reut.   Her husband picks up.  “Reut is sleeping”, he hangs up and goes back to sleep himself.

So I slap some cold water on my face and start checking our services:

Clinical trial management servers.  Working fine.  Check.

Data collection servers.  Working fine.

Identity provider services. Working fine.

API services. Working fine.

All services are healthy.

And then, I understand:

“She sleeps in a storm”.

My friends.

If we take care of the important things in life.

If we are right with those we care about.

If we behave in line with our faith.

Our lives will not be cursed with the throbs of unfinished business.

Our words will always be sincere, our embraces tight.

We will never wallow in the agony of “I could have, I should have”.

We can sleep in a storm.

And when it's time, our goodbyes will be complete.

Rabbi Albert Lewis 1975