Listen to the article

Pet360 was one of our oldest clients, going back to 2014, and working with them was our first real foray into retainer-based billing. At the time, we were mostly working on projects based on ‘buckets of hours’ – a client could purchase a specific number of hours at a discounted rate of 20%-30% less per hour. If a client were to run out of hours, they could purchase more at the same discounted rate. If the hours went unused, those hours would carry over to the next month. This was a confusing and inefficient way to manage our business. We even built and maintained our own statement generation software that integrated directly with our project management tools.

A new way of working

In 2014, Pet360 approached us with a particular problem that involved site caching. We proposed a retainer rate and went to work. Team augmentation was a fairly new experience for us. Prior to this, we would normally be the only team, but Pet360 already had an incredibly talented engineering team, with great leadership. Being on retainer at a fixed rate per month allowed us to provide enough work time that we could be on equal footing with the internal team. This created a bond and made for long-lasting relationships, even to this day.  As is usually the case, we dug in to solve one problem, which eventually revealed other issues. It was an inspiring time for us – we had our first large retainer client – and a lot to prove.

Not only was this a change in the way we did business and an opportunity to work more directly with a client’s product and engineering teams, but a fundamental shift in our agile methodology of choice. Until this engagement, we were a pseudo-Scrum shop. It didn’t work well for us due to clients often changing direction and priorities, and we were now entering the realm of Kanban. Without getting too far into the details, we can say that it was one of the most enlightening opportunities in the history of our business and impacts us to this day. We now implement Kanban into every project we work on, work with clients to build their own implementations, and have built entire continuous improvement tools around it. Process maturity through Kanban is a major component of our company.

The situation

Pet360 was in a different situation than many of our clients due to their scale. It was realistic that a few lines of code could make a difference large enough to impact the bottom line in a significant way. Due to the amount of traffic and complex data flows, there was little room for error. Introducing new bugs could have deep ramifications to the overall customer journey, and because there was a pet pharmacy and the health of millions of customers’ pets was in the hands of two small engineering teams.

As we became more comfortable with their Kanban processes, we began to move quickly, and even grew into product management. This allowed us to work side-by-side with key decision-makers, which led to many big initiatives. One example was a complete catalog rebuild where we had to plan for restructuring the entire catalog along with a new ERP integration with no downtime and with active customers on the website.

Fixing the Speed Issues

Initially, the site suffered from performance issues. It was slow not only for the customers but also for the customer service team, which directly impacted how many. The client had built the site on Magento but did not have extensive experience scaling that platform.  One specific customer service issue was quantifiable by the business because it could take between 20 and 30 seconds to look up a single customer by email address on the backend. Not only did these issues cause problems at the moment by making customer service reps flustered while talking to customers, but led to massive time inefficiencies, as they could only do so many calls in a day. This was holding back the customer service team from being as efficient as possible, and every day there would be customers who couldn’t be helped.

This was the first chance we had to shine. We identified a scaling issue in the core platform due to a full table scan, fixed that, and took those 20 seconds down to two-tenths of a second. The product managers now had a fix that they could measure, and this set the stage for the types of issues we’d be working on.

Problem Solving with The Five Why’s

We grew well together, and the most significant thing we learned is to never work in isolation and ask for help when you need it. We take that with us everywhere to share responsibility. Pet360 was one of the first of our clients to have a major outage. While it’s impossible to reliably count your losses, we knew what an average day’s revenue was and it was a lot of money. This was one of our first clients where we were active participants in root cause analysis (RCA). While common among engineering teams, for anyone not familiar, this is essential when you have to dig in and provide a formal explanation of what caused an issue. The next outage-related activity we participated in was the Five Whys. The Five Whys is when you collaborate with a bunch of different groups and you ask why five times. Here’s a drastically oversimplified example:

  • Why is why did the site crash? The site crashed because it ran out of memory.
  • Why did it run out of memory? Because the settings were too low for X, Y, and Zthat.
  • Why were they too low? Because on January 2, we lowered how much memory each process could use.
  • Why did we change it? We changed it because we wanted to be able to have more concurrent processes per server. 

When you say why do you need to run more processes? You say we were getting more traffic and growth. You ask why five times leading up to the last question. 

This process shows that the breadth of any issue spans more than one group and one person, and it’s almost a defense mechanism, but it’s a good one. It helps people on different teams feel better because they don’t feel like they are at fault. It shows you sometimes that bad things come out of good situations. Sometimes you make a move that has consequences.

Lessons Learned

Pet360 taught us that it’s okay to make mistakes. Don’t beat yourself up constantly if something breaks because things are going to break. It’s where we got comfortable with deploying things, to ‘hit the button’. There’s an undo if the site goes down for 10 minutes because something broke. Undo it. You have that power, and so there’s a lot that we took with us from there.

Unfortunately, the day Pet360 signed with us was also the day PetSmart acquired them. We saw the writing on the wall, and even though there were promises from Petsmart to the contrary, we knew they’d most likely get absorbed and shut down. In the meantime, we grew some sizable retainers and even after Petsmart shut down the site, we were kept on for a few months to help with the transition. 

Even to this day, Pet360 was one of our most impactful experiences, and will forever appreciate the people we met and the relationships that we formed. We still work with or keep in touch with many of the people we worked with, and are still growing together. We consider ourselves incredibly lucky to have learned so much from one engagement and to have met so many incredible people along the way.

Photo by Andrew S on Unsplash