Lessons Learned: Smoke Testing a Node.js Web Server
Hi, everyone. This is James Shore with my lessons learned about smoke testing a Node.js web server. I'm recording this on September 12th, 2012.
We'll starting by showing a complete smoke test, then go into more detail. We'll look at what smoke testing is, the unique spot it occupies in the TDD ecosystem, and when—and when not—to use it. Next, we'll have a refresher on the basic Unix process model and how to work with processes in Node. Finally, we'll tie it all together by writing a basic smoke test for our WeeWikiPaint server.
A Complete Smoke Test
Let's start with a quick look at a complete smoke test before we get into the details. You can find the source code for this episode at the link on your screen.
This smoke test confirms that our WeeWikiPaint server works under real production conditions. Our server isn't very complex yet, so it just makes sure that we can serve our home page. You can build on this pattern to create more sophisticated tests.
The test launches the WeeWikiPaint process... monitors stdout to see when the server has started... then attempts to retrieve the home page. When it does, it confirms that the page contains the key phrase "WeeWikiPaint home page," which is embedded in a comment of the production home page. Then it sends the SIGTERM signal to the server... and waits for the server to die.
Now let's go into more detail.
Smoke Tests
As we discussed in Lessons Learned #3, there are three basic strategies for test-driven development: unit tests, focused integration tests, and end-to-end integration tests.
A smoke test is a particular kind of end-to-end test. The goal of a smoke test is to confirm that your application will run correctly in production. A good smoke test replicates your production environment as closely as possible, then checks that all parts of the system wire together correctly.
How much should you test? Not much! Remember, end-to-end tests are costly to create and maintain. The tests are slow, emulating your production environment is often difficult, and they're prone to false failures. A smoke test should just make sure everything's wired together. Only test the parts of your system that you can't test any other way.
For example, in WeeWikiPaint, we've thoroughly tested the behavior of our simple server with focused integration tests. However, the code that launches the server hasn't been tested. We've kept that code as small as possible, but we still need to make sure the port is read from the command line correctly, and we need to make sure that the production content is where we think it is.
So that's what our smoke test is going to check. In our test, we're going to launch WeeWikiPaint as if it had been launched in production, check that we're getting the correct content back on the right port, then shut down WeeWikiPaint.
Before we do that, though, let's review how we work with processes in Node.
The Unix Process Model
Node.js uses the Unix process model. (It emulates it on Windows.) Because our smoke test is going to launch a WeeWikiPaint process—as if it had been run from the command-line—we need to review how the Unix process model works with Node.
Don't worry—this isn't going to be comprehensive. We're going to look at launching processes, communicating with processes, and ending processes. I'm going to gloss over most of the details.
About Processes
A process is basically a running program. It exists independently of all the other processes in your system. Each process has its own memory, working directory, environment variables, and so forth.
Each process also has three standard streams: stdin, stdout, and stderr. These are just what they sound like: the standard way to input and output data to a process. Typically, when you run a program from the command-line, the standard streams are hooked up to the console—that is, your keyboard and display. Your typing is sent into stdin, and what the program writes to stdout and stderr is displayed in your command-line window.
(By the way, stderr is another output stream, just like stdout. Some programs use it for error messages, but this isn't required. By default, stdout and stderr both show up in your command-line window.)
Launching Processes in Node.js
In Unix, new processes are created by forking your current process, then exec'ing your new program. Node abstracts away those details. In Node, you launch a new process by calling child_process.spawn(). You pass in the name of the command you want to run and an array of command-line arguments. Node returns a child object that you can use to interact with the process.
Communicating with Processes
Every process in your system runs independently. When you change something in one process, no other process can see it. That's a bit of a problem for us, because our smoke test needs to know when the WeeWikiPaint server is up and running. We need some way for our server to communicate with us.
Unix offers a lot of options for inter-process communication. Pipes, signals, sockets, semaphores, yadda yadda yadda. I'm not going to go into all those details. We just want a simple communication mechanism that makes sense for our production server.
Luckily, Node makes this easy. By default, when you spawn a process, Node sets up pipes for the process's standard streams—that is, stdin, stdout, and stderr. Usually those streams are connected to your command-line console, but they can be redirected somewhere else with a pipe. That's what Node does. It pipes the child process's standard streams to the child object as... no surprise here... stdin, stdout, and stderr.
What this means is that we can have our WeeWikiPaint server use console.log, which writes to stdout, and whatever we log will show up in our test. We just have to listen for data on the child's stdout stream. It's super easy, and best of all, we don't have to munge up our server to do it. We just need to output log information, which is good practice if we didn't need it for our test.
Ending Processes
That still leaves the problem of shutting down the server. The server has a stop() function, but you can't call functions in other processes. We need some other way to tell the server to shut down.
One option would be to use our pipes again. We could write "stop" to our child.stdin stream, as if the user had typed "stop" on a keyboard, and then we could listen for that in the server. It wouldn't be too hard to implement, but it's clunky, and it's not how production servers work. They're supposed to run without keyboards or users.
Instead, the standard way for one Unix process to tell another to shut down is to send a signal. Our production environment will almost certainly do the same thing. There are a lot of different signals you can send, with names like SIGABRT, SIGALRM, and SIGHUP, but the standard signal for "it's time to shut down now" is SIGTERM, for "terminate."
Node makes it easy to send a signal: you call the kill() method. By default, kill() sends the SIGTERM signal. When the child process receives it, it has the opportunity to gracefully shut down.
Unix Processes Summary
In summary, Node makes it pretty easy to deal with child processes. We use child_process.spawn() to create a process, child.stdout to see what the process logs, and child.kill() to signal it to shut down.
Now let's put this all together and write our smoke test.
Smoke Testing WeeWikiPaint
Starting out, we have our server code, but no way to run it from the command line. We'll use our smoke test to drive the development of the command-line runner.
We'll start by using the child_process module... spawning node... with our weewikipaint launcher. For convenience, we'll have the standard streams show up in our terminal, for now.
We expect this test to fail because there's no weewikipaint module yet... and it does.
Now we create the weewikipaint module. That's pretty easy because it doesn't have to do anything... and that's enough for the test to pass.
Now let's confirm that the server started. We'll tell Node to pipe the standard streams to us... listen for data on the server's stdout... collect it as it comes in... and wait for it to say "Server started."
That fails because weewikipaint isn't doing anything.
So we'll start the server with some placeholder parameters... and log that the server's started when it's ready.
The test passes, but it hangs because we aren't shutting down the server. So let's send the SIGTERM signal... and wait for the server to exit.
Now the tests run clean.
Speaking of cleanliness, let's refactor. We'll move the server termination code to tearDown()... and we'll move the server launch code to setUp()... which passes.
That leaves our test with nothing to do. Let's get a page. We'll use the http module... to get our home page... listen for a response... and wait for it to end. That should fail... because we aren't telling the server which port to use.
So we'll add that to the command line... and modify the server to get it from the command line. When we run our tests, though, they should still fail, because we don't have any production HTML pages... and they do.
To fix that, we need to update our placeholder code in weewikipaint... and create an empty home page. Now the test should pass... and it does.
Next, we want to prove that we're getting the right home page. Let's modify our test to collect the response data... and check that it contains our home-page marker. That should fail because our home page doesn't have the marker... and it does.
Fixing the homepage is easy... and now the test passes.
Let's clean up our test by factoring out the marker checking code... and that passes.
Now we write another test to confirm that our 404 page is wired up properly... the test should fail... it's easily fixed... and now the test passes.
A bit of final refactoring to clean up the content paths... make sure it works... and the URLs... check that... and we're done!
Conclusion
So that's what I've learned so far about smoke-testing a Node.js web server. To summarize, a smoke test is an end-to-end test that makes sure your whole system is wired together correctly. A good smoke test checks the minimum necessary to ensure that the system will work correctly in production.
To write a smoke test, replicate your production environment as closely as possible. In our case, that meant launching the server as if it had been run from the command line, monitoring its log to see when the server had started, retrieving a web page, then sending a termination signal to shut down the server.
The source code for this example is available at the link on your screen. Thanks for watching, and I'll see you next time!