Characterization Testing

A nice way to help you refactor legacy code is to use “Characterization Testing” or “Golden Master testing”. These approach provide a quick way to get cover legacy code with _some_ tests. There is a big catch though:

Unlike regression tests, to which they are very similar, characterization tests do not verify the correct behavior of the code, which can be impossible to determine. Instead they verify the behavior that was observed when they were written.

https://en.wikipedia.org/wiki/Characterization_test

You can use a tool like “Approval tests” to quickly set up “Characterization Testing”. Emily Bache has a nice introduction video about working with “Approval tests”.

After you’ve covered the legacy code using “Characterization Testing”, you can start refactoring the legacy code. At this point, don’t forget to add unit tests for the new code you’re writing.

For more info have a look at:

Fuzz testing

Last year at Droidcon Online Xavier Gouchet gave a talk titled: “It’s time to up your test game“. Here he talked about testing and more specifically fuzz testing.

Wikipedia describes fuzz testing as follows:

Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks.

Wikipedia

In his talk he introduced Elmyr:

A Kotlin library providing tools to generate “random” values.

Elmyr github page

Seeing how easy it was to use Elmyr, it got me inspired to start using it in my own projects as well. About a month ago I found some time to get started and I’m loving it!

Adding and using Elmyr turned out to be super easy. It took very little time to get up and running.

What surprised me most is that almost immediately I found a bug in my code due to the random test data that Elmyr produces!

Another thing I really like is that I no longer need to think about what test data to use with my subject under test. All I need to do is ask Elmyr to give me some random data.

If you need Elmyr to support some of your own classes for fuzzing, simply create a new ForgeryFactory, add it to the Forge in your test and you’re done.

Now I wish that I learned about fuzz testing ages ago!

If Elmyr doesn’t do the trick for you, Github hosts tons of other fuzzing libraries.

Cucumber & Android

A while ago, the agile/scrum team I was part of was looking for a way to record functional requirements for the app we were working on. After looking around for a bit, we came across Behaviour-Driven Development (BDD) and Cucumber.

Although documentation and automated tests are produced by a BDD team, you can think of them as nice side-effects. The real goal is valuable, working software, and the fastest way to get there is through conversations between the people who are involved in imagining and delivering that software.

Cucumber.io on BDD

Since we wanted to test the Android and iOS using the same Gherkin scenarios, we set up Cucumber together with Appium. Personally I find working with Appium too cumbersome. In a private project I’ve therefore set up Cucumber with Android and left Appium out of the equation.

In this article I’ll explain how to set up Cucumber for Android.

Set up cucumber-android

Add the “cucumber-android” library to your project:

androidTestImplementation "io.cucumber:cucumber-android:$cucumberVersion"

Custom TestRunner

To configure Cucumber we’ll create a custom “TestRunner‘:

private const val PLUGIN_KEY = "plugin"
private const val REPORTS_DIR = "reports/cucumber"

private const val REPORTER_PLUGIN_PATTERN =
    "junit:%s/cucumber-junit.xml--" +
    "html:%s/cucumber-html--" +
    "json:%s/cucumber.json"

/**
 * The CucumberOptions annotation is mandatory for exactly one of the classes in the test project.
 * Creating a custom test Runner seems the simplest way to achieve that.
 */
@CucumberOptions(
    features = ["cucumber/features"],
    glue = ["nl.ansuz.android.test"]
)
class CucumberRunner : CucumberAndroidJUnitRunner() {

    override fun onCreate(bundle: Bundle?) {
        bundle?.putString(PLUGIN_KEY, getPluginConfiguration())
        super.onCreate(bundle)
    }

    private fun getPluginConfiguration(): String {
        val path = getReportsPath()
        return REPORTER_PLUGIN_PATTERN.format(path, path, path)
    }

    private fun getReportsPath(): String =
        File(targetContext.getExternalFilesDir(null), REPORTS_DIR).absolutePath
}

Permissions

To make reporting work, we need to give our app permission to write to the external storage. Since Cucumber will run under the “androidTest” flavor, you can add the extra permissions in “src/androidTest/AndroidManifest.xml” to avoid polluting your production app with unnecessary permissions.

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

Now Cucumber can save test reports on the external storage.

Writing Gherkin

The next step is to write some scenarios so there is something for Cucumber to test.
The cucumber.io website comes with plenty of examples, so I’ll leave this as an exercise for the reader.

Note: The “cucumber-android” plugin expects all feature files to be created under “app/src/androidTest/assets“. You can tweak the base path by changing the “features” property of the “CucumberOptions” annotation in the custom “TestRunner” we created earlier.

Running tests

Now that everything is set up and we have a first test, we can run it. Because of the custom test runner, I had to create a Gradle task that will run instrumentation tests with it.

task runCucumberInstrumentationTests(type: Exec) {
     description 'Runs instrumentation tests with Cucumber'
     group 'verification'
     dependsOn 'installDebug', 'installDebugAndroidTest'
     commandLine 'adb', 'shell', 'am', 'instrument', '-w', 'nl.ansuz.android.test/nl.ansuz.android.test.CucumberRunner'
 }

When you run this new task, no tests pass, but the Cucumber reports are generated anyway. To make the tests pass,we have to create some “glue”.

Writing the glue

In the custom “TestRunner” we’ve also defined where the “cucumber-android” library can find the “glue” to match scenarios with step definitions.

The easiest way to write your step definitions is to annotate the appropriate methods in your Espresso tests with Cucumber annotations, e.g.

@When("the Maker starts a game")
fun startGame() {
  ...
}

Pulling Cucumber reports

Now that you have written some tests, can run them and they pass, you want to be able to have a look at the Cucumber report(s) as well. Pulling the reports is fairly simple, all we need to do is add a Gradle task that uses “adb pull” to do this for us.

task pullCucumberReports(type: Exec) {
    description 'Pulls the Cucumber reports from the emulator.'
    group 'verification'
    dependsOn 'runCucumberInstrumentationTests'

    workingDir buildDir
    commandLine 'mkdir', '-p', 'reports'
    commandLine 'adb', 'pull', '/mnt/sdcard/Android/data/nl.ansuz.android/files/reports/cucumber', 'reports'
}

The reports path is set in the “REPORTS_DIR” constant in the custom “TestRunner” from earlier. Make sure that the paths in the test runner and the Gradle task match.

Cleaning up

So far we are able to run Cucumber tests and pull the reports from the device. What is missing is a way to clean up after testing has finished. Again we’ll add a new Gradle task to do this for us.

task uninstallCucumberTest() {
    description 'Uninstalls the debug and debugAndroidTest apps.'
    group 'install'
    dependsOn 'uninstallDebug', 'uninstallDebugAndroidTest'
}

Tying it all together

Because we are good developers and we are lazy, we don’t want to execute three different Gradle tasks to run tests, pull reports and clean up. Let’s introduce one last Gradle task to make our lives even easier.

task cucumberCheck() {
    description 'Installs regular and instrumentation apps, runs Cucumber tests, pulls reports ' +
            'and then uninstall both apps.'
    group 'verification'
    dependsOn 'pullCucumberReports'
}

cucumberCheck.finalizedBy uninstallCucumberTest

I hope this helps you to get you started on using BDD for your Android project with Cucumber.

Bonus: IntelliJ / Android Studio plugins

To make working with Cucumber and Gherkin easier, you can install the following plugins for IntelliJ or Android Studio:

  • “Gherkin” which provides support for the Gherkin language.
  • “Cucumber for Kotlin” which enables Cucumber support with step definitions in Kotlin.
  • “Cucumber for Java” which enables Cucumber support with step definitions in Java.

Further reading

If you want to learn more about Cucumber, I can highly recommend reading “The Cucumber Book” by Matt Wynne and Aslak Hellesøy, with Steve Tooke.

For an introduction into BDD and Gherkin, the cucumber.io website offers a lot of documentation:

Notes

Software versions used at the time of writing:

  • Gradle: 5.2.1
  • cucumber-android version: 4.2.5