How we test the TeamCity UI

Developing a working piece of software is difficult. Just like building an airplane, it requires talented people, working components, and a testing framework. No plane leaves the hangar before everything is ready, checked and double-checked.

In JetBrains, we adopt the same philosophy for building our software. Vigorous testing helps us discover bugs and problems before the final product takes off. Just like building a plane, software development is a process that consists of multiple stages. Although the authors of this post are not aerospace engineers, we will use simplified aircraft analogies. There are several reasons for that: aircraft is beautiful, it is pure engineering, and it reveals that the problems we raise here are not exclusive to software engineering.

The bigger your product, the more steps and modules there are. To make sure your software is ready to lift off, every module needs to be tested and correctly integrated with everything else. CI/CD services, if set up correctly, help automate this process. Most importantly, they remove the human factor famous for one careless action being able to lead to total disaster.

Contrary to popular belief, testing is very important in front-end development. To continue the analogy, your plane is not only required to fly – it has to be comfortable inside! Moreover, its exterior affects how airplane flights (aerodynamics). Getting back to the front end, this means that you have to test the usability as well as functionality. This makes front-end testing a must. In this article, we will provide an overview of UI testing used in TeamCity. If you have any questions about the technical details – don’t hesitate to ask us.

Note:

Each section of this post contains links to useful resources. Some of them lead to public TeamCity configurations where you could examine how we test the front end. Feel free to sign in as a guest and look around.

TL;DR theses:

• UI testing is not only about the Unit tests. There are screenshots, behavior, accessibility, performance, security, and perception tests. Each of them is listed below.
• Testing Systems and CI/CD help concentrate on things that really matter.
• With the TeamCity, you can build sophisticated pipelines to test any UI issue.

Interception of issues

Let’s consider the interception of issues. In JetBrains, it works on multiple levels, and most of them are based on the CI/CD. Each test, each department, each level takes its part in the entire process of revealing and reporting problems. Here is a chart that shows all of these parts:

Every test takes care of some issue categories. Finally, it helps us to mitigate issues.

For example, a huge amount of UI problems we catch belong to the Screenshot testing stage. Fewer problems belong to the Linters / Unit / Render tests. That doesn’t make those tests meaningless. On the contrary, it could mean that we work in these areas well enough to prevent tons of issues.

The key point we would like to show here: as you can see, the Quality Assurance Department faces only one-third of the problems. This means that by using CI/CD you could help your colleagues to save time on inspecting issues that are easy to predict , which could be caught by a well-organized testing system.

The chart isn’t 100% representative and numbers float from release to release: some level can remove more problems than others. However, it shows that the test system is very important, even if a separate test covers only one case. Quantity doesn’t equal quality here: a test could find “only” one bug but that one bug could’ve crashed your whole application if left unnoticed.

Linters, typings, unit tests

Firstly, we ought to say that the code should be clean and consistent, but there’s an issue with that. Everyone has their own understanding of clean code. In JetBrains, we agreed on 200+ rules that make sure our code stays objectively clean. As a result, we get a warning from IntelliJ Idea whenever alinter detects a problem. It’s not superfluous to mention that static typings with TypeScript , Flow , Kotlin , or Reason are required for complicated applications. We, the TeamCity team, have decided to use Flow. One of the first tests we use for building the front end is just eslint/stylelint check. Its purpose not only to find code style problems but also issues with missed variables, imports, or non-cheap/safe operations (like React Hooks without dependencies).

Of course, there are also unit tests. It’s simple: we write pure atomic functions and then assert their output. If the output is OK, TeamCity marks them green and allows the pipeline to continue.

Screenshot testing

Displaying consistent UI across multiple devices and platforms is one of our main goals. We need to be certain that all the components are displayed correctly, regardless of the used browser, layout, or viewport size. This implies testing how the components are visually presented in different configurations. Screenshot testing does exactly that.

Screenshots diff

After one of the updates Comment sections have been added to the Unauthorized Agents. That’s quite ok. But imagine, that this kind of test could reveal the disappearing of elements. It happens from time to time and we find this test very useful. Although, Snapshot Testing, you’ll get familiar soon also will help in this case.

That’s how we do it:

1. Launch a server that renders the components (Storybook in our case).
2. Connect to the server using the WebDriver API; this API allows us to interact with a website in an automatic mode – without a real user.
3. WebDriver calls the relevant components.
4. Hermione, a utility tool by Yandex, connects to Storybook using WebDriver and takes multiple screenshots of the selected area in every browser.
5. Those screenshots are then put into a folder, where Hermione compares them to the default screenshots using Mocha.
6. If something has changed, we get a notification. The differences are also visually highlighted!

React and rendering

We try to improve the performance of our interfaces by minimizing the amount of unnecessary rendering. Generally, React is pretty good at that. However, there are some cases you should keep in mind: if you change something in a component, React creates a new one instead of modifying the original component.

Imagine that you pass new props into a component. In order to save the resources needed for a re-render, React checks if the passed props differ from the previous version. If they are the same, no re-rendering takes place. Unfortunately, many of us forget that in JavaScript two arrays or objects with equal content are different entities (because those objects/arrays will have different references). As a result, we end up unnecessarily rendering equal components.

React Highlights DOM Updates

You can enable React Updates Highlighting with React Developer tools. This will reveal all re-renders in your App. For example, there are rerenders, which take place during cursor moving over the Trends Page subcomponent. Fortunately, all those rerenders are intended here. But imagine, that adding to these re-renders app will fire one hundred more?

Should we re-render?

We already know about the danger. Nevertheless, we decided to check for redundant rendering using why-did-you-render . To our surprise, we discovered multiple instances of inefficiency. That’s how we did it:

1. We created a dummy action: it triggers changes to the store.
2. If we change something in the store, we will make all the components (subscribed to this store) collect data once again. It happens with mapStateToProps callback.
3. After collecting the data, we pass it to the component and launch compare function to check, whether the props had been changed or not.
4. Meanwhile, we know that the dummy action doesn’t actually change any values in the store, meaning that no new props should be passed to the component.
5. If new props lead the component to a re-render, we know that we created a new object/array somewhere where we should not. What a shame!

TeamCity reports excess re-renders

We have two tips for solving this problem:

• The reselect library
• Immutable data structures

Reselect library

Using the reselect library , you can memoize results if all the parameters for the generation function remain the same. If the passed parameters are equal to the previous ones, we will receive not the new objects, but the references to old ones. No re-rendering takes place.

Immutable data structures

You can predefine an object or array as immutable by freezing it. Next time, whenever you would like to return a fallback value, you should return this immutable object. It guarantees that reference to this object always will be the same, so the component will not be re-rendered.

Snapshot testing

Snapshot testing verifies that any changes made to the structure of important components are intentional. Once again, let’s return to our airplane analogy. Imagine that we have a snapshot of a plane’s structure: it should have a body, one wing, and four jet engines. Suddenly, we decide to remove one of the jet engines and replace it with a turbine. While this may be a great idea, it no longer fits our snapshot. Consequently, we would get a notification.

Sometimes even hardcoded items could be changed. Photo by Anthony Noble . 

In the picture above the change was intentional, with obvious ramifications. Now imagine a snapshot of our plane which consists of hundreds of components: tracking all the interactions would be very difficult. JavaScript projects are similar in this respect. A small change to one component could lead to unwanted consequences for the structure of the whole project.

You can protect structures by creating their snapshots. Whenever you misprint, change an HTML class, or add a new component, you will break the structure. A snapshot test will notify you if you do. Check the example below:

1. We create a snapshot of an important structure. We specify the IL-76LL engines:
   
2. We always want to compare future airplanes to the snapshot we made previously.
   
3. Here, we change the engine type from turbofan to propfan. Just to test how it works. Since the new engine no longer matches our snapshot, the test fails. We’ve got a report, and our engineers are on their way to investigate the problem.
   

The same to the React components:

Read more:

Storybook Structural Testing / Jest Snapshot testing guide

E2E Tests

E2E tests are very similar to test flights. Like with planes, we have to make sure that our interface is actually usable in the real world. With thousands of components interacting with each other, you never know if your plane can take off before the pilot actually takes it into the air.

E2E tests are designed to test an application flow from start to finish. Those tests emulate a real user who is going through the same specific use case over and over again.

E2E in action

This is how E2E testing looks in our case:

1. Create a list of scenarios that are critical from the user’s POV (user stories).
2. Create an automated test for every listed scenario.
3. Each of those tests should describe how Selenium is supposed to interact with the UI.
   1. Imperatively:
      1. Open the browser.
      2. Login.
      3. Go to page X.
      4. Press button Y.
      5. Make sure window Z is displayed.
   2. Declaratively:
      1. “Make sure the user gets the window Z after going to page X and start process Y.”
4. Launch the Docker container with the last TeamCity instance.
5. Launch the tests; those tests connect to Docker using Selenium and execute the algorithms.

Other tests

Writing this article took a while. During this time we had implemented some new tests, such as Dependencies Security Audit ,Accessibility tests, and we are also discussing some new categories of tests. We are looking forward to your feedback to continue writing articles and dive deeper into the front-end testing area.

TeamCity and build chains

TeamCity lets you create infinitely complex logic for launching tests and deploying builds. This is how TeamCity displays the chain/timeline of builds for its own UI.

As you can see, TeamCity runs tests in parallel, and in consequence, some builds waiting for others to been finished successfully. And if something goes wrong, it could stop the whole pipeline – just to prevent wasting resources on the tests which will definitely fail.

And this is how TeamCity visualizes a huge project like itself:

The important point here is that we could not only build complex pipelines but make them sophisticated, complicated in a good manner. For example, some parts of our pipeline must be built on OS X agents, some of them – on the Linux system, and some of them are going to be built withAmazon Cloud Agents.

Conclusion

Do you remember the first diagram? Our automated tests cover more than half of those issues, and the Quality Assurance Department covers one-third part. Nevertheless, there is always something left. In JetBrains, we widely practice “dogfooding”: we use our own products (IntelliJ, TeamCity, Space, YouTrack, and so on) to develop and build software. So, at this stage, everyone inside JetBrains could come to us and share feedback. By doing so, we get notice and fix bugs that passed all the other filters.

During our Early Access Program, you can find out more about the new features we are making right now, share your feedback, report issues, and ask us to include some new features so they are released as soon as possible.

Hopefully, we’ve made the basic principles of front-end testing clear. Summing up, you can consider it insurance: it requires some effort but can be lifesaving. To get the most profit from CI/CD for small-scale projects, you can use TeamCity free of charge. Protect yourself from costly mistakes!

To try TeamCity right now, follow thislink.

With love,

TeamCity Team