Showing posts for tag "testing"

Adding Selenium Browser Tests to My Testcontainers Setup

Jul 20, 2021, 11:20 AM

Yesterday, I talked about how I dove into Testcontainers for my app-testing needs. Today, I decided to use this to close another bit of long-open business: automated browser testing. I've been very much a dilettante when it comes to that, but we have a handful of browser-ish tests just to make sure the login page, the main page, and some utility pages load up and include expected content, and those can serve as a foundation for much more.

Background

In general, when you think "automated browser testing", that means Selenium. As a toolkit, Selenium has hooks for the browsers you want and has essentially universal support, working smoothly in Java with JUnit. However, the actual act of loading a real browser is miserable, mostly on account of needing you to install the browser and point to it programmatically, which is doable but is another potential system-specific configuration that I'd much, much rather avoid in my automated builds.

Accordingly, and because my needs have been simple, I've used HtmlUnit, which is a portable Java browser-like library that does the yeoman's work of letting you perform basic Selenium tests without having to configure actual native OS installations. It's neat, imposes basically no strictures on your workflow, and I recommend it for lots of uses. Still, it's not the same as real browsers, and I had to do things like disable JavaScript processing to avoid it tripping up on some funky JS that full-grown browsers can deal with.

Enter Webdriver Containers

So, now that I had Testcontainers configured to run the web app, my eye turned to Webdriver Containers, an ancillary capability of Testcontainers that lets you run these full-fledged browsers via their Docker images, and even has cool abilities like letting you record the screen interactions over VNC. Portability and full production representation? Sign me up.

The initial setup was pretty easy, just adding some dependencies for the Selenium remote driver (replacing my HtmlUnit driver) and the Testcontainers Selenium module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-remote-driver</artifactId>
    <version>3.141.59</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>selenium</artifactId>
    <version>1.15.3</version>
    <scope>test</scope>
</dependency>

Programmatic Container Setup

After that, my next task was to configure the containers. I'll skip over some of my troubleshooting and just describe where I ended up. Basically, since both the webapp and browsers are in Docker containers, I had to coordinate how they communicate with each other. There seem to be a few ways to do this, but the route I went was to build a Docker network in my container orchestration class, bind all of the containers to it, and then reference the app via a network alias.

With that addition and some containers for Chrome and Firefox, the class looks more like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
public enum AppTestContainers {
    instance;
    
    public final Network network = Network.builder()
        .driver("bridge") //$NON-NLS-1$
        .build();
    public final GenericContainer<?> webapp;
    public final BrowserWebDriverContainer<?> chrome;
    public final BrowserWebDriverContainer<?> firefox;
    
    @SuppressWarnings("resource")
    private AppTestContainers() {
        webapp = new GenericContainer<>(DockerImageName.parse("client-webapp-test:1.0.0-SNAPSHOT")) //$NON-NLS-1$
                .withExposedPorts(8080)
                .withNetwork(network)
                .withNetworkAliases("client-webapp-test"); //$NON-NLS-1$
        
        chrome = new BrowserWebDriverContainer<>()
            .withCapabilities(new ChromeOptions())
            .withNetwork(network);
        firefox = new BrowserWebDriverContainer<>()
            .withCapabilities(new FirefoxOptions())
            .withNetwork(network);

        webapp.start();
        chrome.start();
        firefox.start();
        
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            webapp.close();
            chrome.close();
            firefox.close();
            network.close();
        }));
    }
}

Now that they're all on the same Docker network, the browser containers are able to refer to the webapp like "http://client-webapp-test:8080".

Adding Parameterized Tests

The handful of UI tests I'd set up previously had lines like WebDriver driver = new HtmlUnitDriver(BrowserVersion.FIREFOX, true) to create their WebDriver instance, but now I want to run the tests with both real Firefox and real Chrome. Since I want to test that the app works consistently, I'll want the same tests across browsers - and that's a call for parameterized tests in JUnit.

The way parameterized tests work in JUnit is that you declare a test as being parameterized, and then feed it your parameters via one of a number of mechanisms - "all values of an enum", "this array of strings", and a handful of others. The one to use here is to make a class implementing ArgumentsProvider and configure that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import java.util.stream.Stream;

import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.ArgumentsProvider;
import org.testcontainers.containers.BrowserWebDriverContainer;

public class BrowserArgumentsProvider implements ArgumentsProvider {
    @Override
    public Stream<? extends Arguments> provideArguments(ExtensionContext context) throws Exception {
        return Stream.of(
            AppTestContainers.instance.chrome,
            AppTestContainers.instance.firefox
        )
        .map(BrowserWebDriverContainer::getWebDriver)
        .map(Arguments::of);
    }
}

This class will take my configured browser containers, get the WebDriver instance for each, and provide that as parameters to a test method. In turn, the test method looks like this:

1
2
3
4
5
6
7
8
@ParameterizedTest
@ArgumentsSource(BrowserArgumentsProvider.class)
public void testDefaultLoginPage(WebDriver driver) {
    driver.get(getContainerRootUrl());
    assertEquals("Expected App Title", driver.getTitle());

    // Other tests follow
}

Now, JUnit will run the test twice, once for each browser, and I can add any other configurations I want smoothly.

Minor Gotcha: Container vs. Non-Container URLs

Though some of my tests were using Selenium already, most of them just use the JAX-RS REST client from the testing JVM directly, which is not containerized in this setup. That meant that I had to start worrying about the distinction between the URLs - the containers can't access "localhost:(some random port)", while the JUnit JVM can't access "client-webapp-test:8080".

For the most part, that's not too tough: I added some more utility methods named to suit and changed the UI tests to use those. However, there was one tricky bit: one of the UI tests uses Selenium to fetch the page and process the HTML, but then uses the JAX-RS client to make sure that a bunch of references on the page resolve to non-404 resources properly. Stuff like this:

1
2
3
4
5
driver.findElements(By.xpath("//link[@rel='stylesheet']"))
    .stream()
    .map(link -> link.getAttribute("href"))
    .map(href -> rootUri.resolve(href))
    .forEach(uri -> checkUrlWorks(uri, jaxRsClient));

(It's highly likely that there's a better way to do this in Selenium, but hey, it's still a useful example.)

The trouble with the above was that the URLs coming out of Selenium included the full container URL, not the host-accessible one.

Fortunately, that's not too tricky - it's really just string substitution, since the host and container URLs are known at runtime and won't conflict with anything. So I added a "decontainerize" method and run my URLs through it in the stream:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
public URI decontainerize(URI uri) {
    String url = uri.toString();
    if(url.startsWith(getContainerRootUrl())) {
        return URI.create(getRootUrl() + url.substring(getContainerRootUrl().length()));
    } else {
        return uri;
    }
}

// later

driver.findElements(By.xpath("//link[@rel='stylesheet']"))
    .stream()
    .map(link -> link.getAttribute("href"))
    .map(href -> rootUri.resolve(href))
    .map(this::decontainerize)
    .forEach(uri -> checkUrlWorks(uri, jaxRsClient));

With that, all the results came back green again.

Overall, this was a little fiddly, but mostly in a way that helped me learn a little bit more about how this sort of thing works, and now I'm prepped to do real, portable full test suites. Neat!

Tinkering With Testcontainers for Domino-based Web Apps

Jul 19, 2021, 12:46 PM

(Fair warning: this post is not about testing, say, a normal XPages app via Testcontainers. One could get there on this path, but this has a lot of prerequisites that are almost specific to me alone.)

For a while now, I've seen the Testcontainers project hanging around in my periphery. The idea of the project is that it uses Docker to allow you to programmatically load services needed by your automated test suites, rather than having to have the servers running separately. This is a clean match for something like a WAR-based Java webapp that uses, say, Postgres as its backend database: with this, you can spin up a Postgres image from the public repository, fill it with test data, run the suite, and tear it down cleanly.

However, this is generally not a proper match for Domino. Since the code you're testing almost always directly uses Domino API calls (from Notes.jar or another source) and that means having a local Notes runtime initialized in the test code, it's no help to have a separate container somewhere. So, instead, I've been left watching from afar, seeing all the kids having fun in a playground I never got to go to.

The Change

This situation has shifted a bit for my needs, though, thanks to secondary effects of changes I've made in one of my client projects. This is the one where I do all the bells and whistles of my tinkering over the years: XPages outside Domino, building a bunch of NSFs with Jenkins, and so forth.

For a while, I had been building test suites run using tycho-surefure-plugin, but somewhat recently moved the project to maven-bundle-plugin to reap the benefits of that. One drawback, though, was that the test suites became much more difficult to run, in large part due to the restrictions on environment propagation in macOS.

Initially, I just let them wither, but eventually I started to rebuild the test suites. The app had REST services for a while, but they've grown in prominence since we've started gradually replacing XPages-based components with Angular apps. And REST services, fortunately, are best tested at a remove.

First Pass: liberty-maven-plugin

The first way I started writing test suites for the REST services was by using liberty-maven-plugin, which is a general Swiss army knife for working with Liberty during Maven builds, but has particular support for starting a server before tests and terminating it after them. So I set up a config that boots up a Liberty server that can then initialize using a configured Notes runtime, and I started writing tests against it using the Jakarta REST client API and a bit of HtmlUnit.

To its credit, this setup did its job swimmingly. It still has the down side that you have to balance teacups to get a Notes or Domino runtime configured, but, once you do, it'll work nicely.

Next Pass: Testcontainers

Still, it'd be all the better to avoid the need to have a local Notes or Domino setup to run these tests. There's still going to be some weirdness due to things like having to have the non-public Domino Docker image pre-loaded and having an ID file and notes.ini somewhere, but that can be overcome. Plus, I've already overcome those for the CI servers I have set up with each build: I have some dev IDs in the repository and, for each build, Jenkins constructs a Docker image housing the webapp and starts a container using a technique similar to what I described a few months back to run a Liberty app with Domino stuff brought in for support.

So I decided to try adapting that to work with Testcontainers. Instead of my Maven config constructing and launching a Liberty server, I would instead build a Docker image that would then be loaded in Java with the Testcontainers library. In the case of the CI server scripts, I used Bash to copy files into a scratch directory to avoid having to include the whole repo in the Docker build context (prohibitive on the Mac particularly), and so I sought to mirror that in Maven as well.

Building the App Image in Maven

To accomplish this goal, I used maven-resources-plugin to copy the app and support files to a scratch directory, and then com.spotify:dockerfile-maven-plugin to build the Docker image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
<!-- snip -->
    <!-- Copy Docker support resources into scratch space -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-resources-plugin</artifactId>
        <version>3.2.0</version>
        <executions>
            <execution>
                <?m2e ignore?>
                <id>prepare-docker-scratch</id>
                <goals>
                    <goal>copy-resources</goal>
                </goals>
                <phase>pre-integration-test</phase>
                <configuration>
                    <outputDirectory>${project.build.directory}/dockerscratch</outputDirectory>
                    <resources>
                        <!-- Dockerfile to build -->
                        <resource>
                            <directory>${project.basedir}</directory>
                            <includes>
                                <include>testcontainer.Dockerfile</include>
                            </includes>
                        </resource>
                        <!-- The just-built WAR -->
                        <resource>
                            <directory>${project.build.directory}</directory>
                            <includes>
                                <include>client-webapp.war</include>
                            </includes>
                        </resource>
                        <!-- Support files from the main repo Docker config -->
                        <resource>
                            <directory>${project.basedir}/../../../docker/support</directory>
                            <includes>
                                <!-- Contains Liberty server.xml, etc. -->
                                <include>liberty/client-app-a/**</include>
                                <!-- Contains a Domino server.id, names.nsf, and notes.ini -->
                                <include>notesdata-ciserver/**</include>
                            </includes>
                        </resource>
                    </resources>
                </configuration>
            </execution>
        </executions>
    </plugin>
    <!-- Build a Docker image to be used by Testcontainers -->
    <plugin>
        <groupId>com.spotify</groupId>
        <artifactId>dockerfile-maven-plugin</artifactId>
        <version>1.4.13</version>
        <executions>
            <execution>
                <?m2e ignore?>
                <id>build-webapp-image</id>
                <goals>
                    <goal>build</goal>
                </goals>
                <phase>pre-integration-test</phase>
                <configuration>
                    <repository>client-webapp-test</repository>
                    <tag>${project.version}</tag>
                    <dockerfile>${project.build.directory}/dockerscratch/testcontainer.Dockerfile</dockerfile>
                    <contextDirectory>${project.build.directory}/dockerscratch</contextDirectory>
                    <!-- Don't attempt to pull Domino images -->
                    <pullNewerImage>false</pullNewerImage>
                </configuration>
            </execution>
        </executions>
    </plugin>
<!-- snip -->

The Dockerfile itself is basically what I had in the afore-linked post, minus the special ENTRYPOINT stuff.

Of note in this config is <pullNewerImage>false</pullNewerImage> in the dockerfile-maven-plugin configuration. Without that set, the plugin would attempt to look for a Domino image on the public Dockerhub and then fail because it's unavailable. With that behavior disabled, it will just use the one locally loaded.

Configuring the Tests

Now that I had that configured, it was time to adjust the tests to suit. Previously, I had been using system properties passed from the Maven environment into the test runner to identity the Liberty server, but now the container initialization will happen in code. Since this app is pretty heavyweight, I didn't want to do what most of the Testcontainers examples show, which is to let the Testcontainers JUnit hooks spawn and terminate containers for each test. Instead, I set up a centralized class to launch the container once:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
package it.com.example;

import org.testcontainers.containers.GenericContainer;
import org.testcontainers.utility.DockerImageName;

public enum AppTestContainers {
    instance;
    
    public final GenericContainer<?> webapp;
    
    @SuppressWarnings("resource")
    private AppTestContainers() {
        webapp = new GenericContainer<>(DockerImageName.parse("client-webapp-test:1.0.0-SNAPSHOT")) //$NON-NLS-1$
                .withExposedPorts(8080);
        webapp.start();
    }
}

With this setup, there will only be one instance of the container launched for the whole test suite, and then Testcontainers will shut it down for me at the end. I can also use the normal mechanisms from the Testcontainers docs to get the actual name and port it ended up mapped to:

1
2
3
4
5
6
    public String getServicesBaseUrl() {
        String host = AppTestContainers.instance.webapp.getHost();
        int port = AppTestContainers.instance.webapp.getFirstMappedPort();
        String context = "clientapp";
        return AppPathUtil.concat("http://" + host + ":" + port, context, ServicesUtil.DEFAULT_JAXRS_ROOT);
    }

Once I did that, all the tests that had previously been running against a liberty-maven-plugin-run server now worked against the Docker container, and I no longer have any dependency on the local environment actually having Notes or Domino fully installed. Neat!

A Catch: Running on my Jenkins Server

Since the whole point of Docker is to make things reproducible across environments, I was flush with confidence when I checked these changes in and pushed them up to the repo. I watched with bated breath as Jenkins picked up the change and started to build. My heart sank, though, when it got to the integration test suite and it failed with a bunch of:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
Jul 19, 2021 11:02:10 AM org.testcontainers.utility.ResourceReaper lambda$null$1
WARNING: Can not connect to Ryuk at localhost:49158
java.net.ConnectException: Connection refused (Connection refused)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.base/java.net.Socket.connect(Socket.java:609)
    at org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:163)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:159)
    at java.base/java.lang.Thread.run(Thread.java:829)

What the heck? Well, I had noticed in my prep that "Ryuk" is the name of something Testcontainers uses in its orchestration work, and is what allowed me to spawn the container manually above without explicitly terminating it. I looked around for a while and saw that a lot of people had reported similar trouble over the years, but usually it was due to some quirk in a specific version of Docker on Windows or macOS, which was not the case here. I did, though, find that Bitbucket Pipelines tripped over this at one point, and it seemed to be due to their switch of using safer user namespaces. Though it sounds like newer versions of Testcontainers fixed that, I figured it's pretty likely that I was hitting a variant of it, as I do indeed use namespace remapping.

So I tweaked my failsafe-maven-plugin configuration to set the TESTCONTAINERS_RYUK_DISABLED environment variable to false and, to be safe, added a shutdown hook at the end of my AppTestContainers init method:

1
2
3
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    webapp.close();
}));

Now, Testcontainers doesn't use its Ryuk container, but the actual app container loads up just fine and is destroyed at the end of the suite. Perfect! If all continues to go well, this will mean that it'll be one step easier for other devs to run the test suites regardless of their local setup, which is always a thorn in the side of Domino-realm testing.

Closing: What About Testing Domino Apps?

I mentioned in my disclaimer at the start that this is specifically about testing apps that use a Domino runtime, not apps on Domino. Still, I bet you could do this to test a Domino app that you deploy as an NSF and/or OSGi plugins, and I may do that myself down the line so that the test suite even-more-closely matches what is actually running in production. You could adjust the maven-resources-plugin config above (or use maven-dependency-plugin) to bring in NSFs built earlier in the build with NSF ODP Tooling as well as OSGi update sites and then have your Dockerfile copy those into the Domino data directory and the workspace/applications/eclipse directory. Similarly, if you had a Domino addin that you launch as a task and which then itself listens on a port, you could do the same there.

It's still not as convenient as being able to just easily run Domino API tests without all the scaffolding, and it implies a lot of structure that makes these more firmly "integration" than "unit" tests, but that's still a powerful capability to have.