— Ruslan Shevchenko [iov42; codespace] // @rssh1, ruslan@shevchenko.kiev.ua

PROPERTY BASED TESTING

”

“

WHAT IS IT ?

PROPERTY BASED TESTING

Eng: property властивість свойство

Other : example приклад пример

for all x: Int x < (x+1)

3 < (3 + 1)

SHOW ME THE CODE/SCALA

import org.scalacheck.Properties import org.scalacheck.Prop.forAll

object IntSpecification extends Properties("Int") { property("x < x+1") = forAll{ (x:Int) => x < x+1 } }

SHOW ME THE CODE/SCALA

import org.scalacheck.Properties import org.scalacheck.Prop.forAll

object IntSpecification extends Properties("Int") { property(“x < x+1") = forAll{ (x:Int) => x < x+1 } }

[info] ! Int.x > x+1: Falsified after 3 passed tests.[info] > ARG_0: 2147483647[info] Failed: Total 1, Failed 1, Errors 0, Passed 0[error] Failed tests:[error] IntSpecification[error] (test:test) sbt.TestsFailedException: Tests unsuccessful

SHOW ME THE CODE/JAVA

…..

@RunWith(JUnitQuickcheck.class) public class IntPropertyTest {

@Property public void xLessIncr(int x) { assert(x < x+1); }

}

SHOW ME THE CODE/JAVA

…..

@RunWith(JUnitQuickcheck.class) public class IntPropertyTest {

@Property public void xLessIncr(int x) { assert(x < x+1); }

} [info] Running com.github.rssh.IntPropertyTestTests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.75 sec

// Problem not found. // Generation of random values are differ ;)

SHOW ME THE CODE/JAVASCRIPT

var jsc = require("jsverify")

describe('double ceiling', function() { it ('x < x + 1e-15',function() { expect( jsc.checkForall(jsc.number,function(x){ return x < x+1e-15 }) ).toBe(true) }); });

// explore the limits of the floating point ceiling.

SHOW ME THE CODE/JAVASCRIPT

var jsc = require("jsverify")

describe('double ceiling', function() { it ('x < x + 1e-15',function() { expect( jsc.checkForall(jsc.number,function(x){ return x < x+1e-15 }) ).toBe(true) }); });

// explore the limits of the floating point ceiling.

Failed after 8 tests and 1 shrinks. rngState: 0879d03f2e184b6c5a; Counterexample: 19.7693584933877; [ 19.7693584933877 ]FFailures: 1) double ceiling x < x + 1e-151.1) Expected Object({ counterexample: [ 19.7693584933877 ], counterexamplestr: '19.7693584933877', shrinks: 1, exc: false, tests: 8, rngState: '0879d03f2e184b6c5a' }) to be true.

MONKEY TESTING

PROPERTY-BASED TESTING

We generate random values. Pass one to our function or service to test Expect, that result of this function satisfy some properties

laws (classical property-based testing); crash

The Infinite Monkey Theorem

HISTORY

QuickCheck. Haskell. 1999

• https://en.wikipedia.org/wiki/QuickCheck • http://www.cse.chalmers.se/~rjmh/QuickCheck/

John Hughes founded QuivQ • commercial version of QuickCheck for Erlang • extensions/interfaces for

• C/C++ • WebServices • Industry specific standards … (appl. Volvo cars)

IMPLEMENTATIONS FOR POPULAR LANGUAGESScala: scalacheck

http://www.scalacheck.org Java: junit-quickcheck

http://pholser.github.io/junit-quickcheck/ JavaScript: jsverify

http://jsverify.github.io/ Python: Hypothesis

http://hypothesis.works/ PHP: eris; phpquickchek

https://github.com/giorgiosironi/eris https://github.com/steos/php-quickcheck

…………………… practically any language.

TYPICAL USAGE

Thinks and describe laws (dependencies between input & output)

Tune input generators Check such properties in output

Demos: Programming tasks (sorting, serialization) Form validation State checks

LET’S CHECK SORTED ALGORITHM

def sort(x:IndexedSeq[T]):IndexedSeq[T]

invariant: length

definition of sorted:

idempotency:

LET’S CHECK SORTED ALGORITHM

def sort(x:Vector[T]): Vector[T]

invariant: length

property(“sorted array must the same size of origin”) = forAll{ (x: Vector[Int]) => x.size == sort(x).size }

LET’S CHECK SORTED ALGORITHM

We need to generate: • n - max number of entries • sorted vector of the length n • i < n • j: i < j < n

CUSTOM GENERATOR

val orderedVectorGen = for { n <- Gen.choose(2,100) x <- Gen.containerOfN[Vector,Int] i <- Gen.choose(0,n-2) j <- Gen.choose(i,n-1) } yield (n, sort(x), i, j)

CUSTOM GENERATOR

val orderedVectorGen = for { n <- Gen.choose(2,100) x <- Gen.containerOfN[Vector,Int] i <- Gen.choose(0,n-2) j <- Gen.choose(i,n-1) } yield (n, sort(x), i, j)

property(“sorted array must reflect ordered indexes”) = forAll(orderedVectorGen){ case (n, x, i, j) => x(i) <= x(j) }

property test can serve as documentation for input and output.

LAWS

Ordering[T]: • reflexivity:

forall{ x:T => x <= x} • anitsimmetry:

forall{ x:T, y:T => x <= y & y <= x ==> x==y }

• transitivity: • forall{ x:T, y:T, z: T =>

x <= y & y <= z ==> x <= z }

// Laws = set of properties. // Can be used as specification for type argument.

EXAMPLE: DATA VALIDATION

case class Subsriber(id:Long, // must be unique firstName: String, // max 20 sym, no spaces. lastName:String // max 20 sym, no space )

val correctSubscriberGen = for { id <- arbitrary[Long] firstName <- Gen.alphaStr suchThat (_.length <= FIRST_NAME_LEN) lastName <- Gen.alphaStr suchThat (_.length <= LAST_NAME_LEN) } yield Subscriber(id,firstName, lastName)

val incorrectSubscriberGen = for { id <- arbitrary[Long] firstName <- incorrectNameGen(FIRST_NAME_LEN) ………

GENERAL QUESTIONS

Is property-based testing useful itself ? — definitely yes. — force to think hight-level

Are we steel need example-based testing ? — probably yes.

• What to manual construct thing, which • explore some type of bug.

• Dimension Problem. • N probes can be very tiny part.

QUESTIONS ?

THANKS FOR LISTENING

Full example code is available on github: https://github.com/rssh/property-based-testing-talk

@rssh1 Ruslan Shevchenko <ruslan@shevchenko.kiev.ua>