Testing Shiny Apps

Introduction

The goal of this vignette is to give a basic overview of how one might approach “testing” a shiny app. Shiny is a new package from RStudio that makes it dramatically easier to build interactive web applications with R. Shiny Uses a reactive programming model and has built-in widgets derived from the Bootstrap front-end framework. In this vignette we will looking at writing unit tests for a simple shiny wep app. The testing package we will use is testthat which has a short introduction here. I am using testthat version 0.8. The version on cran is version 0.7.1 and may give trouble for tests where I manipulate the test environment. You can install 0.8 from github devtools::install_github("testthat", "hadley")

Each section will be an introduction to an idea in testing shiny apps with Selenium, and point to more detailed explanation in other vignettes.

Some Thoughts on Testing

Why Test?

When faced with testing for the first time the natural reaction is to think what now? what do i test? how much/many tests do I write?

Tests need to do something useful to survive. Automated tests should help the team to make the next move by providing justified confidence a bug has been fixed, confirming refactored code still works as intended, or demonstrating that new features have been successfully implemented. There should be sufficient tests - neither more nor less: more increase the support burden, fewer leave us open to unpleasant surprises in production.

One way to create our tests is to take the view of the user. What does the user want to do?

They want to see this particular graph of a given data set. How do they do that? They select various options and input various choices. From this list of actions we can create an outline of our code for the test.

For each method, we need to work out how to implement it in code. How could an automated test select the sliderInput bar? Do alternative ways exist? An understanding of HTML, CSS, and JavaScript will help you if you plan to use browser automation tools. All the visible elements of a web application are reflected in the Document Object Model (DOM) in HTML, and they can be addressed in various ways. Some simple examples of interacting with the DOM using RSelenium are given in the Rselenium-basic vignette.

Vary the Tests

Having static tests can lead to problems. Introducing variance into the tests can help pick up unexpected errors. This can be achieved by introducing an element of randomness into automatic inputs or randomizing order of selection etc.

Vary the Browsers/OS

It can help to test against a variety of browsers and operating systems. RSelenium can interact with services like sauceLabs. sauceLabs allows one to choose the browser or operating system or the version of the selenium server to use. You can test with iOS/Android/Windows/Mac/Linux and browsers like firefox/chrome/ie/opera/safari. This can be very useful to test how your app works on a range of platforms. More detailed information and examples can be seen on the sauceLabs vignette.

Record the Tests

RSelenium has the ability to take screenshots of the browser at a particular point in time. On failure of a test a screenshot can be useful to understand what happened. If you interface RSelenium with sauceLabs you get screenshots and videos automatically. See the sauceLabs vignette for further details.

Test for Fixes

Lots of bugs are discovered by means other than automated testing - they might be reported by users, for example. Once these bugs are fixed, the fixes must be tested. The tests must establish whether the problem has been fixed and, where practical, show that the root cause has been addressed. Since we want to make sure the bug doesn’t resurface unnoticed in future releases, having automated tests for the bug seems sensible.

The Shiny Test App

Introduction

The shiny test app is composed of various widgets from the shiny package (0.8.0.99 at time of writing). We have also included the ggplot2 library as output for one of the charts adapted from a discussion on stackoverflow. The app includes examples of some of the controls included with the shiny package namely selectInput, numericInput, dateRangeInput and a sliderInput. These controls are used to produce output rendered using renderPrint, renderPlot(base) , renderPlot(ggplot2) and renderDataTable.

The app can be viewed if you have shiny installed.

An image of the app using RSelenium on a windows 8.1 machine running firefox 26.0

shinytestapp on win 8.1 firefox 26.0

The image was generated using RSelenium and the following code.

Observations

From the screenshot we retrieved from the remote Driver there are some interesting observations to make. Note that the selectInput and numericInput boxes are sticking out. This is occurring because the sidePanel is given a bootstrap span of 3. This is however fluid. The resolution on the remote machine is low so the pixel count on the span 3 is also low. On a local machine with high resolution (Nothing amazing just a laptop) we did not observe the selectInput and numericInput boxes sticking out.

We could have run with a higher resolution by passing the additional screen-resolution parameter to sauceLabs.

shinytestapp on win 8.1 firefox 26.0 res 1280x1024

We can see things look a bit better but the data-table search box is a bit compacted.

Inputs and Outputs

The app is designed to show testing of the basic shiny components. It is a bit contrived so testing it may not be as natural as testing a live working app. The outputs (charts and tables) are designed to sit side by side if possible with a maximum of 2 on a “row” then drop down to the next “row”. We can test to see if this is happening by checking the posistionof elements. We will investigate this later.

Basic Tests

Basic Functionality

The first test we will look at implementing will be basic connection to the app. Typically we would make a request for the page and then observe what status code was returned. Selenium doesn’t currently give the html status code of a navigation request so instead we will check if the title of the web page is correct. Our Shiny Test App has a title of “Shiny Test App” so we will check for this.

We create a test/ directory in our Shiny Test App folder. The first set of tests will be basic so we create a file test-basic.r. In this file we have the following code to start with:

We have a context of “basic” for the tests in this file. The test “can connect to app” simply navigates to the app URL and attempts to get the page title. If the page title is “Shiny Test App” the test is deemed successful. For testing purposes we assume the app is running locally. The easiest way to do this is open a second R session and issue the command:

The second R session will listen for connection on port 6012 and return the Shiny Test App. If we ran this basic test we would expect the following output:

[1] "Connecting to remote server"
1..1
# Context basic 
ok 1 can connect to app

So running the test we observe that we can successfully “connect” to the Shiny Test App. What other functionality can we add to our “basic” test context. We can check that the controls and the tabs are present. We can add these tests to our test-basic.r file.

When we rerun our basic test we should hopefully now see that it is checking for the presence of the controls and the tabs.

[1] "Connecting to remote server"
1..8
# Context basic 
ok 1 can connect to app 
ok 2 controls are present 
ok 3 controls are present 
ok 4 controls are present 
ok 5 controls are present 
ok 6 controls are present 
ok 7 tabs are present 
ok 8 tabs are present

That concludes our basic test of the Shiny Test App functionality. Next we look at testing the input controls.

Testing the Controls

Our first test of the controls will be the functioning of the checkbox. We open a new file in the test directory of our Shiny Test App and give it the name test-checkbox.r. We also give it a context of controls.

context("controls")

library(RSelenium)
library(testthat)

remDr <- remoteDriver()
remDr$open(silent = TRUE)
sysDetails <- remDr$getStatus()
browser <- remDr$sessionInfo$browserName
appURL <- "http://127.0.0.1:6012"

test_that("can select/deselect checkbox 1", {  
  remDr$navigate(appURL)
  webElem <- remDr$findElement("css selector", "#ctrlSelect1")
  initState <- webElem$isElementSelected()[[1]]
  # check if we can select/deselect
  if(browser == "internet explorer"){
    webElem$sendKeysToElement(list(key = "space"))
  }else{
    webElem$clickElement()
  }
  changeState <- webElem$isElementSelected()[[1]]
  expect_is(initState, "logical")  
  expect_is(changeState, "logical")  
  expect_false(initState == changeState)  
})

remDr$close()

In this case I am informed there maybe issues with Internet Explorer. Usually one would select the element for the checkbox and click it. In the case of Internet Explorer it maybe necessary to pass a space key to the element instead. Otherwise the test is straightforward. We check the initial state of the checkbox. We click the checkbox or send a keypress of space to it. We check the changed state of the checkbox. If the initial state is different to the changed state the test is deemed a success. For good measure we also check that the initial and changed states are of class “logical”. We add code for the other 3 checkboxes. We can check our test as follows:

test_dir(paste0(find.package("RSelenium"), "/apps/shinytestapp/tests/"), reporter = "Tap", filter = "checkbox")
[1] "Connecting to remote server"
1..12
# Context controls 
ok 1 can select/deselect checkbox 1 
ok 2 can select/deselect checkbox 1 
ok 3 can select/deselect checkbox 1 
ok 4 can select/deselect checkbox 2 
ok 5 can select/deselect checkbox 2 
ok 6 can select/deselect checkbox 2 
ok 7 can select/deselect checkbox 3 
ok 8 can select/deselect checkbox 3 
ok 9 can select/deselect checkbox 3 
ok 10 can select/deselect checkbox 4 
ok 11 can select/deselect checkbox 4 
ok 12 can select/deselect checkbox 4

We filter here on “checkbox” to only select this test file to run. If you watch the test running it will filter through the checkbox control checking each checkbox is functioning. The checkboxGroupInput drives the required controls which has id reqcontrols. Each of these controls is one of the building blocks of shiny and we will add a test for each.

Testing the selectInput

We write a simple test for the selectInput. It tests the options presented and the label of the control. We isolate the code in a separate file test-selectinput.r in the test folder of our Shiny Test App. It also then selects an element from the options at random. It is tested whether the output changes or not.

Running the selectInput test we get:

[1] "Connecting to remote server"
1..3
# Context controls 
ok 1 selectInput dataSet correct 
ok 2 selectInput label correct 
ok 3 selectInput selection invokes change

Note we set remDr$setImplicitWaitTimeout(3000) in this test so that we get a 3 second limit to find an element.

Testing the “numericInput”

The ideas behind testing the numericInput are similar to testing the selectInput. We test the label. We then test a random value between the allowable limits of the numericInput and check that the output changes. Finally a character string “test” is sent to the element and the appropriate error message on the output is checked. The final test can be adjusted to suit whatever bespoke error display etc is in your app. The test code is in the tests folder of the Shiny Test App in a file named test-numericinput.r. Again remDr$setImplicitWaitTimeout(3000) is called to give some leeway for element loading. Some commented out code indicates other methods one could deal with checking for element existence. Additional detail on timing races in Selenium can be found here.

Testing the “dateRangeInput”

The test on the dateRangeInput compose of two tests. We test the label and we test the two input dates. We choose two random dates from the set of allowable dates. The output is tested for change after the two dates ave been set. remDr$setImplicitWaitTimeout(3000) is set in the test to allow for race conditions on elements. The test code is in the tests folder of the Shiny Test App in a file named test-daterangeinput.r.

Testing the “sliderInput”

For the sliderInput we test the label and we test changing the controls. The test code is in the tests folder of the Shiny Test App in a file named test-sliderinput.r. The label is tested in a similar fashion as the other controls. The second test needs a bit of explaining. There are a number of ways we could interact with the slider control to change its values. Some of the easiest ways would be to execute javascript with Shiny.onInputChange("range", [2000, 10000]) or Shiny.shinyapp.sendInput({range: [6222, 9333]}). Both these methods would currently work. The Shiny server side would get the new values however the UI would show no change. The underlying sliderInput control is a jslider. Normally one can interact with the jslider thru calls similar to $(".selector").slider("value", p1, p2) as outlined here. We will use mouse movements and the buttondown buttonup methods of the remoteDriver class. Note that one may have problems forming the test in this manner, see for example here. However it is useful to illustrate mouse and keyboard interactions in RSelenium.

We get the attributes of the slider initially. We then get the dimension of the slider

This gives us the pixel width of the slider as it currently stands. This will be different across machines. We generate some random values for the two slider points and then we calculate roughly how many pixels we need to move the sliders.

The above code moves to the slider element. Pushes the left button down. Moves the mouse on the x axis in the direction calculated then releases the left mouse button. The output of the related data-table before and after the change is recorded and the test should result in the before and after not being equal.

It is interesting to note that during initial writing of this vignette a new version of firefox 27.0.1 was released. As expected native events did not work under version 2.39 of selenium server and this updated version of firefox. Subsequently our test as formulated above would fail. There is an option to pass a list rsel.opt for use with some of the tests. Using this we can set nativeEvents = FALSE and the test above will pass again. When your tests fail it is not necessarily bad. This failure indicates a problem with your test setup rather then your app however.

Testing the Output

Finally for this simple example we will look at testing the output. The test code is in the tests folder of the Shiny Test App in a file named test-output.r. The outputs should line up side by side with a maximum of 2 on a line. We can check the position of the outputs. Our first test will check whether the four outputs line up in a grid. This test will fail on low resolution setups which we will observe latter. We can check the headers on the outputs. The two chart plots are base64 encoded images which we can check in the HTML source. We can check the headers on the outputs. Finally we can check the controls on the datatable.

The first test use the getElementLocation method of the webElement class to find the location in pixels of the output objects.

webElems <- remDr$findElements("css selector", "#reqplots .span5")
out <- sapply(webElems, function(x){x$getElementLocation()})

The 1st and 2nd and the 3rd and 4th objects should share rows. The 1st and 3rd and the 2nd and 4th should share a column. This test will fail as the resolution of the app decreases and the output objects get compacted. The second test checks output labels in a similar fashion to other test.

The third test checks whether the chart output are base 64 encoded png. The final test selects the data-table output and randomly selects a column from carat or price. It then checks whether the ordering functions when the column header is clicked.

Finally running all tests with a “summary” reporter we would hope to get:

test_dir(paste0(find.package("RSelenium"), "/apps/shinytestapp/tests/"))
basic : [1] "Connecting to remote server"
........
controls : [1] "Connecting to remote server"
............
controls : [1] "Connecting to remote server"
..
controls : [1] "Connecting to remote server"
...
outputs : [1] "Connecting to remote server"
.......
controls : [1] "Connecting to remote server"
...
controls : [1] "Connecting to remote server"
..

Further Tests