shapes/tetrist
shapes/triangle

Picking the right visual regression testing tool

A straightforward comparison of visual regression testing tools to help you find the right one for your app

If you’re looking to visual regression test your app UI you’ve probably encountered a lot of options.

From open source libraries to full-fledged products, there are plenty of choices that promise to catch UI bugs. It can be time-consuming to find the right solution for your project, so we’ve gathered some info to help you figure out which visual tool is right for you.

Read on for an overview of visual regression testing tools, including features, workflows, and pricing. We hope this helps you figure out the best testing solution for your project.

Website vs web app

Most visual regression testing tools are designed to test websites at the page-level. They’re made for marketing pages, blogs, and news sites.

The problem that app teams discover is that visual regression testing a multi-state web app is very different than testing a collection of static pages. Web apps generate countless UI permutations based on user input and context. Because managing all these permutations is hairy, modern view layers like React, Vue, and Angular exist to help you divide complex app UIs into simple modular components.

Unfortunately, testing web apps built with components using ancient page-level testing tools results in a flood of false positives that you have to dredge through to find real changes.

Tools like Applitools, Puppeteer, and BackstopJS offer features for testing pages. But the features geared to test static sites often aren’t that useful for componentized apps.

We started Chromatic because we were tired of complicated review flows (or none at all), false positives, a bunch of features we didn’t need, painful integrations, and test specs that polluted project repos.

Alt text

Chromatic is centered around tracking and visual regression testing components -the basic building blocks of apps. Testing at the component level makes it easier to scope tests and identify regressions in web apps (just like unit tests help you pinpoint functional bugs).

With Chromatic, each component in your app is tracked every commit so you can catch UI changes whenever you push code.


Paid visual regression testing tools vs open source libraries

It’s obvious, open source libraries are free. And free is pretty appealing. So what's the difference between paid visual regression testing tools and open source libraries?

Both paid and open source tools rely on the technique of visually comparing images to find differences, but that's where the similarities end.

  • Open source tools are much tighter in scope than paid professional tools. They are intended to solve specific challenges very well, but not the whole problem space.

  • Paid tools encapsulate the same basic functionality of their open source counterparts. They then layer work-saving features valuable to professional developers.

Here is a quick comparison using Chromatic, a paid visual regression testing tool.

Paid (Chromatic)Open source
PerformanceFast: Tests run in parallel in the cloud so developers can get nearly instantaneous test results.Slow: runs tests on the local development machine or requires extra work to run elsewhere.
CollaborationHelp teams review UI changes together. The online review process allows team members to review and confirm changes without pulling code.Not scoped for multiple devs. Each dev must pull code and re-run tests to review.
CI & Git workflowSeamless Git and CI workflow integration with notifications and PR statuses.NA
Branches & mergingSupports working on many branches by tracking each branch separately. Intuitive workflows for merging UI changes.Support varies
ReproductionsEasy reproductions because real code, styling, and assets for each component are indexed online. No Git required.Checkout the branch and run the code to reproduce changes.
HistoryTracks every component version so flipping through different versions is quick and easy.NA
StorageSnapshots are indexed online automatically.Snapshots live in your repository (repo bloat). Or live in parallel which requires extra hosting and keeping them in sync.
Developer experiencePolished user flows and timesaving featuresBarebones UX/UI
MaintenanceRound-the-clock support and frequent updatesYour team has to maintain
DeploymentA dedicated team constantly monitors and improves performance of the serviceYour team has to manage deploys and downtime

Which should you choose?

It depends on the project and the team. If you're a professional development team working on a critical app, the work-saving and collaborative features of a paid solution are a must-have. For instance, Chromatic pays for itself within days for small teams and hours for medium-sized organizations.

If you're the solo developer of a modest app or a seed-stage startup whose product is constantly changing then you may want to look at the open source options.


Development team vs individual developer

It takes a team to build and maintain a full-featured app. Like you, we understand that complexity grows and communication muddies the larger the team and project. Not everyone recognizes that visual testing is a team activity. You might ask a designer: “does this looks right?” or a QA person: “can you break this?”

There are plenty of choices but few are designed for the team workflow. Open source libraries like BackstopJS, Puppeteer, and Webdriver focus on the solo developer experience. They’re scoped for testing on the local machine but quickly become slow and blocking when used across a team.

Teams need collaboration

A tool like Chromatic is designed for collaboration from the start. Everyone can take part in the UI review process with the online merge manager. Multiple developers can work on multiple feature branches simultaneously. When it's time to merge, the process is painless.

Engineers, designers, and product managers can browse the component library online without needing to pull code, migrate, or maintain a dev environment. No more “this isn't working on my machine”.

Review with your team


How many components?

"What if I don't have that many components?"

A common concern with visual regression testing is that it's premature optimization. The thought process usually goes like this:

  • There aren't many components so it's easy enough for developers to confirm them manually
  • There isn't enough active development on components to warrant visual regression tests

It's easy to assume that visual regression testing is meant solely for established component libraries. Not true. A single component can realistically live in hundreds of places throughout your app. Making one tiny tweak will propagate to all the places the component is used. This is an enormous frontend risk!

Minor tweaks cause major regressions

Whether you have one or one hundred components makes no difference. When evaluating visual regression testing, first consider the number of times your components are reused not just the total component count.

Start testing UI components early

The benefit of testing UI components -even when you don't have many- is getting the assurance that your components will look as expected in all the places that they're used. Even if you rarely make changes, the changes you do make tend to have an enormous impact. The work-saving impacts of visual regression testing compound with the adoption and size of your component library.


Do you use a component explorer like Storybook?

Storybook is a component explorer that helps frontend teams ship UI components. As maintainers and sponsors of Storybook, we understand how integral Storybook is to app teams.

Chromatic's first-class integration with Storybook means developers can reuse their stories as visual regression tests without extra work writing new tests.

View your stories on Chromatic

Every story is a test

Chromatic tests map 1-to-1 with your stories. Whenever you update a story your tests are synced. This means visual regression tests are painless to keep up to date.

Your Storybook is uploaded automatically

Every time you push code, the components in your Storybook are uploaded to Chromatic. This means everyone on your team can participate in reviewing. No pulling code required.

Reproductions

Chromatic doesn’t just capture an image of each component, it’s a living re-creation of component’s DOM, styling, and assets. Whenever you’re reviewing changes, you see high-fidelity components that are fully inspectable. This makes it painless to reproduce and confirm component changes because you’re seeing exactly what your users are.

No hassle branching and merging

Work on features together or separately. With branching and merging, you can make as many feature-branches as you need and seamlessly merge them back into production. It all happens behind the scenes, and just works. No configuration needed.

Choose versions of your components


Get the free handbook

Want to learn more about visual regression testing? See why of visual regression testing is practiced by companies like Slack, Lonely Planet, and Walmart. Check out our free 31-page Visual Testing Handbook.

Visual Testing Handbook cover


Ready to try Chromatic?

There are a lot of facets to consider with visual regression testing tools. We hope this guide makes finding the right tool a bit clearer.

Trying Chromatic is easy, install the addon and setup your project in less than a minute. Sign up for your 30-day free trial now.