Automated End-to-End Testing: Ensuring Quality

We take quality seriously. For both our customers and developers’ satisfaction, it is essential to provide consistent product performance and speed of development with confidence that existing use cases are not broken. To ensure agile development, here’s why quality is essential to your organization and how our strategy makes automated end-to-end testing fast, reliable, and responsive.

Importance of Quality

For Customers: Every company claims to deliver high quality to their customers, but this is especially critical for Rubrik. Our product is responsible for managing highly valuable data that powers our customers’ businesses. In the backup and recovery industry, our solution needs to be on active duty at the exact moment our customers experience trouble within systems protected by us. Given that these problems are complex, providing an extremely simple user experience alleviates troubleshooting. Of course, this simple user experience can only be simple as long as all the underlying pieces are performing reliably.

For Engineers: Engineers want to innovate without breaking existing functionality that customers depend on. If the fundamentals fail, customers cannot upgrade without losing data. It’s often difficult to innovate without affecting the interoperating pieces. In Rubrik’s case, we integrate at all levels of our internal hardware and software (hardware health, filesystem, data compaction, job scheduling, REST API, web UI) and also with multiple external products (VMware vSphere, Microsoft SQL Server, Microsoft VSS, storage partners). In today’s complicated enterprise environments, we can only build new innovations by thoroughly testing and fixing any breakages.

Our Strategy for Quality

Building quality solutions is an absolute requirement for creating value for businesses. Our approach to deliver this quality is threefold:

  1. Automation
  2. End-to-End Testing
  3. Feedback Loop


Why Automation?

Automate. Automate. Automate. We automate as much of our testing as possible. Compared to manual testing, automated testing is less expensive, more consistent, and most importantly, much faster. It allows engineers to repurpose their time on more valuable and challenging work, such as developing new test cases or debugging failures while machines perform repetitive testing.

Execution speed is where automation is a game changer. For unit tests, engineers get results almost instantaneously and for full end-to-end tests, within a couple hours. Our release qualification process runs every automated test in our suite in up to 12 hours. We can qualify a release within 24 hours with supplemental manual verification when needed. By combining engineering and release management workflows, we can not only respond to requests from the field for minor features or enhancements, but also have them fully developed, stabilized, and delivered to customers in just 1 week. Other organizations that still rely heavily on manual testing take 3 months or more to deliver on such minor requests. With automated testing, we are able to adapt much faster to customers’ needs.

Automated End-to-End Testing

For many organizations, automated testing is great in theory but not so valuable in practice. This is because their automated tests run fast without necessarily telling them enough about the product’s quality. As a result, they ultimately need to perform a lot of slow manual testing. This especially happens when most of the automated tests are unit tests. However, most bugs come not from individual pieces malfunctioning but rather from multiple pieces not working correctly together. In order to mimic what the customer sees, test cases need to cover the whole user flow from end-to-end. This includes deploying all parts of the ecosystem. We accomplish this by starting the product in a clean, unconfigured state, automating product features using the same interfaces available to the customer (no backdoors), verifying behavior, and cleaning up resources for the next test to use. Writing end-to-end automated tests is very difficult, but it’s a worthwhile investment to achieve the full speed of developing, stabilizing, and delivering quality code.

Think about it like building a house. Even if all the pieces used to build a house are fine, the completed house might be poor quality because they don’t fit well together. For instance, when building the door, you can’t only test the individual parts by themselves. If you only test the parts that make it, you won’t identify the problem. You have to construct and test the entire door as a whole.


Feedback Loop

End-to-end testing is ultimately not complete without customer input. No amount of testing we do in-house, whether automated or manual, tells us what our customers really experience when using our product. This is why we continuously take feedback from customers about the UI, issues encountered, and new features requests. Based on this data, we enhance our test suites. Through this process, many customers unknowingly benefit from other customers’ experiences as we improve the user experience before new customers encounter those problems.

Automated testing is no walk in the park, but we welcome the challenge to deliver the best quality. In my next blog post, I’ll go through the details of our support functions and suite of automated end-to-end tests. In the meantime, check out our GitHub open source project with VMware vRO workflows. Remember, software is king, but automation is queen and rules the kingdom.

Want more? Check out automated end-to-end testing part 2.