Cross-Browser Testing with Karma

We look at the unfortunately-named Testacular tool (since renamed to Karma) and how to use it. We create a simple test and run it across multiple browsers and platforms, then close with an automated script for running Testacular.

Lessons Learned: Cross-Browser Testing with Testacular

Hi, everyone. This is James Shore with my lessons learned about cross-browser testing with Testacular. I'm recording this on October 10th, 2012.

We'll start by reviewing the automated cross-browser test, then go into the details. We'll look at the unfortunately-named Testacular tool and how to use it. We'll create a simple test and run it across multiple browsers and platforms, and then we'll close by looking at an automated script for running Testacular.

The Automated Cross-Browser Test

Let's start with the complete automated test. At the beginning of our work day, we start the Testacular server... then capture each of the browsers we want to test. In this example, we're testing Safari; Chrome; and Firefox. We're also running IE 8 in a VM... and IE 9... and iOS Safari in the iOS Simulator. Once the browsers are up and running, we can just ignore them for the rest of the day.

With the setup done, we can run our tests with a single command. Every time we run the command, the tests run on all of our captured browsers.

The build also checks to make sure all of our supported browsers are tested. If we don't test against a browser—say, Chrome—then the build fails. For convenience, we can loosen that restriction.

Now let's go into more detail.

About Testacular

Testacular is a tool for client-side JavaScript testing. The great thing about Testacular is that it runs your tests in real browsers. It's very fast and it runs from the command-line, which makes it great for automation and test-driven development.

Testacular works in four parts. There's the Testacular server, which coordinates everything. There's captured browsers, which run the tests. There's the Testacular client, which triggers test runs and reports results. And then, of course, there's your tests and production code.

Here's how it works. You start up the Testacular server with testacular start. When it runs, it creates a little web server on port 8080. (It also opens a control port at 9100, but you won't need to worry about that.) Any browser you point at port 8080 will be served a chunk of Testacular JavaScript, which captures the browser for testing. You can connect any number of browsers, from anywhere on your network, even a phone or iPad.

Once the browsers are captured, you run Testacular from another command line. That tells the server to load your source files and push them to the captured browsers, which then run your tests. As they run, they report their results back to the Testacular server, which reports them back to the runner.

The net result? Your tests run on multiple real browsers, with all their real-world quirks, and you get the results, collated and summarized, with a single command. Awesome.

Now let's look at how it works in practice.

Using Testacular

We're going to look at installing and configuring Testacular, which you'll just do once per project; starting the server and capturing browsers, which you'll do about once per day; and running your tests, which you'll do every few minutes.

Configuring Testacular

Install Testacular with npm. You can use the -g option to install it globally, but as usual, I prefer to install it locally so all my dependencies are in source control. I create a small shell script to make it easier to run.

If you check Testacular into source control, as I do, be sure to update your .gitignore file (or equivalent) so that Testacular's platform-specific files aren't checked in.

At the time of this writing, Testacular's socket.io dependency has a bug that causes npm rebuild to fail on Windows if the module was installed on a Mac. To work around this bug, change into the Testacular directory... and force an install of newer socket.io code by running this command. (npm install socket.io@https://github.com/LearnBoost/socket.io/tarball/ec07fa8308da815686fcc2bc041a5d32d71c29d5) You should only do this if you're using socket.io version 0.9.10 or older and you have problems with npm rebuild.

After installing Testacular, run testacular init to have it walk you through creating a config file.

Testacular works in conjunction with a third-party testing framework. Out of the box, it supports Jasmine and Mocha. I chose Mocha.

Testacular also gives you the option of capturing browsers automatically. I've found it more convenient to capture browsers manually—it's less error-prone—so I left this option blank.

When Testacular runs, it will push your files to your captured browsers. This option tells Testacular which files to upload. Since all my source files in this example are client-side, I used src/**/*.js, to tell Testacular to load all JavaScript files in the src directory and all its subdirectories. It gives me a warning because I haven't created the files yet.

I also have the option to exclude files, but I don't have any files to exclude, so I skipped it.

The final option is to have Testacular automatically run your tests when any of them change. This can be handy, but I decided against this option because I want to use my build script, which does more than just run Testacular.

We told Testacular we were going to use Mocha for testing, so let's install it. We'll also install expect.js for assertions.

With that done, we can look at the Testacular configuration file. There's a bug in Testacular that causes it to put in a weird basePath, so we'll fix that. The basePath is used as the starting point for finding your included and excluded files. It's relative to the location of the config file, so the current directory will work for us.

We also need to update our list of files to load into the browser. We're going to use expect.js for assertions, but we didn't put that in when we ran testacular init. We'll add it now.

And that's it! Testacular is installed and configured for our project.

Capturing Browsers

Before we can run our tests, we need to start the Testacular server and capture at least one browser. That's pretty easy. From the command line, we run testacular start. It reminds us that we haven't written any code yet, then starts up and gives us the URL for capturing a browser.

Next, we capture a few browsers. We can have as many as we want—for this example, let's use Chrome and IE 8. We'll run Chrome... paste in the URL... and it's connected.

Now let's get IE 8 working. That's a little trickier because this example is running on a Mac. We'll start a Windows VM running IE 8... point it at the Testacular server running on our Mac... and now it's connected as well.

The details of how you connect across machines will depend on your VM software and your network configuration, but the same procedure applies if you're developing on Windows and running Mac OS in a VM—say, to run the iOS Simulator. You can also point real devices like an Android phone or an iPad at your Testacular server to run your tests directly on real hardware.

Running Tests

Now that our browsers are captured, we're ready to run our tests. First, let's write a simple example test.

We'll create our source directory... create the test file... and write a simple test that says our production code returns a string containing "foo." Then we create the production file... define a global... and do what we said we would.

Now let's test it. We already have our Testacular server running and the browsers are captured, so running the tests is a simple matter of opening a command prompt and typing testacular run. The code runs in our captured browsers and we can see the results in our terminal. That's it!

Automating Testacular

As I've mentioned before, I think build automation is critical. There's too many opportunities for human error, otherwise. So I've created some code for running Testacular from a Jake file. Not only does it run Testacular, it also confirms that you've tested all your supported browsers. That way you don't accidentally write code that isn't going to work on one of your production browsers. You can use the script as a starting point for your own build automation. It's available at the link on your screen.

There is a catch, unfortunately. At the time of this recording, this script does not work on Windows. There's a bug in Testacular that stops its output from being read by a script. I've got an issue open and you can follow it here.

Here's how the script works. Fill in the SUPPORTED_BROWSERS block with the names of the browsers you want to confirm, as reported by Testacular.

To use the script, run jake testacular to start the Testacular server... and capture your browsers.

Once the browsers are ready, you can run jake to run your build. It will run testacular, then check its output to confirm that each of your supported browsers was running.

If a browser isn't running... say, Chrome... the build will fail.

Sometimes you don't want it to fail. In that case, run the build with the loose=true option. The script will report which browsers weren't tested, but it won't fail.

Conclusion

So that's what I've learned so far about cross-browser testing with Testacular. To summarize, Testacular allows you to run your tests on multiple real browsers. It works by creating a little web server that you can use for capturing browsers. Then, when you run your tests, your test code is pushed to those browsers, run, and the results are displayed on the command line.

I've created a little script you can use for automating Testacular. You can find it, and the rest of this episode's example code, at the link on your screen. Thanks for watching, and I'll see you next time!