Getting Better Tools Through Product Experiments

By Priya Bhatia and Keilin Morrison

Product experimentation is a crucial aspect for us at 7shifts. Each test is carefully designed to learn more about the user psychology which ultimately helps us achieve our overall objectives.

Nuggets of gold can be found anywhere from user interviews to aggregate data, so it’s our responsibility to test and lean into those discoveries in order to move fast and deliver more value to our customers.

Read more to learn about how we use experimentation in our Product Growth department at 7shifts.

How does Experimentation work at 7shifts?

At 7shifts, we believe in Speed as a Habit and that, all things equal, speed is what sets you apart from your competition. To move fast, you need to release features rapidly to learn what works and what doesn't. In the software world, time and resources are limited, and we use product experimentation to get closer to discovering user psychology rapidly and at scale. These experiments guide our future investments in the product.

The Goal

As a growth product manager at 7shifts, I am constantly thinking about how we make it easier for restaurants to understand the value of 7shifts. Our company goals help us decide what metrics to focus on. From there, we use customer feedback, data, and user tests to help us identify hypotheses to test. One of our previous objectives was to increase the speed and rate of activation for restaurants trialing 7shifts. For those unfamiliar with activation, it is defined as the process of getting to your product's 'aha' moment quickly and frequently, to build a habit around it.

The Process

We got our whole squad together to brainstorm ideas and decide how we would gather more data around activation at 7shifts. That led us to reach out to our customer-facing teams like Sales, Success, and Support to see where customers were experiencing problems while on trial. At the time, 7shifts had two different ways to trial our tool.

DIY (custom) mode and demo mode. DIY mode allowed users to build their team's schedule immediately, whereas demo mode gave users access to a sample account filled with data and a pre-built schedule. Our internal teams informed us that customers choosing demo mode complained about the experience, leading to issues during the sales and onboarding process. In some cases, customers had left 7shifts after losing data created in this demo mode experience.

Next, we wanted to understand and validate the problem deeply. Why is it a problem? How big of a problem is this? How is it impacting our customers? To do this, we spoke directly to our customers. We observed their behavior while interacting with the product and learned the why behind their actions and preferences. We believe that user tests are crucial to experimentation as they help us to learn and gather insight from the people who could use the product in their everyday lives! The tests indicated that most users preferred using custom mode instead of demo mode, as they wanted to fully tailor the product to their own situation.

The Hypothesis

Additionally, we looked into the data to see if there were aggregate user findings. When observing the data, we found a clear negative correlation between the users choosing demo mode and their conversion rate compared to those that tried the custom mode. TLDR; people using DIY mode over demo mode converted much quicker than those using the demo mode. So then, after all of the research, digging in, and discovery, the question became, how can we make the demo mode experience better?

With that question in mind and our overall goal of improving activation, we took it to the team to develop different options to test our hypothesis. There were two leading solutions: improve the demo mode or eliminate it entirely. Wanting to explore both avenues, we collaborated with the design team to see how we could improve the demo mode while also talking with the developers about the requirements that would be needed to do so. During one of these brainstorming sessions, we hypothesized that if we wanted to get customers hands-on with the product, why were we even pushing them down the demo mode route, which only added to the time to value? It was also more in line with a product-led approach.

Soon after, we started exploring the idea of testing the removal of the demo mode. The entire squad was onboard because it was a quicker change to test without much development effort. Now, removing any feature is always very tricky. We presented this idea to our leadership, who were obviously hesitant to remove something we had spent time and resources building. Again, we used experimentation to build validation to back up the suggestion of a feature removal.

The Results

The results from this test were overwhelmingly positive. We saw that taking out demo mode would lead to a 44% increase in paid conversion and average revenue per trial! Once the results were in and shared with the team, we folded to the variant, confidently scrapping the demo mode after seeing such astounding results from the test. A culture of experimentation at 7shifts means continually learning, understanding what works (and what doesn't), and having the courage to act on it! That is why leaning into the data and keeping an open ear toward customers is essential for product managers — you never know where nuggets of gold are hiding! Through leveraging agile experimentation, 7shifts quickly changed directions to improve the UX, making the customer journey even more seamless.

Conclusion

Experimentation and iterative learning are continuous processes at 7shifts. We are running at least three experiments in different areas of the product at any given point. Not every experiment is successful. In the best-case scenario, only 25% of all our experiments are successful. However, our wins far outweigh the overall effort and resources spent on experimentation. Take, for example, a recent test where a slight change in the UX improved the conversion rate in the treatment group by 2.4%! With some of these phenomenal results, we don't treat experimentation as just a checkbox.

At 7shifts, we pride ourselves on being extremely intentional about experimentation. This also stems from the fact that we are deeply customer and product-driven. We formed a Product Growth department to solve the most critical customer problems that cause friction in their 7shifts journey.

Keilin Morrison

Keilin was a Growth Product Manager intern at 7shifts. She lead tests, product-led growth projects, and loved exploring the world of product. Keilin is excited to graduate from UofA!