Saturday, 26Saturday, 26thth March, 2011 March, 2011

Reducing build timewhen patience is not a virtue

Aman KingApplication Developer

ThoughtWorks

Recognize these?

Rank these

Did you choose this?

1 32

Or did you choose this?

1 23

Patience is not always a virtue!

Fail fast!quick feedback

Rank these

High functional coverage

Short build time

Did you choose this?

①High functional coverage

②Short build time

Or did you choose this?

①Short build time

②High functional coverage

You can choose both!

①Short build time

①High functional coverage

But before we see how…

Project background

Content driven, community oriented

website

Ruby on Rails

Cucumber+

Capybara+

Selenium

4 : 1 ratio of Dev : QA

Everyone writes functional tests in a common automation

suite

Acceptance Test Driven Development (ATDD)

Our problems

Long build time~ 55 minutes

Non-deterministic failures

Manual reruns needed

Our solution:Parallelization

Basic idea

Reduced build time~ 5 minutes

Build time chart

What we did

Parallelization on a single multicore machine

used Ruby library parallel_tests

Report generation in parallelwrote custom report formatter

Isolating databaseseach process needs its own database

database.ymltest: &test adapter: mysql2 encoding: utf8 reconnect: false database: myapp_test<%= ENV['TEST_ENV_NUMBER'] %> pool: 5 username: root password:

Isolating external dependencies (Solr)

each process needs its own Solr instance

sunspot.ymltest: solr: hostname: localhost port: <%= 8982 - ENV['TEST_ENV_NUMBER'].to_i %> log_level: INFO #WARNING log_file: <%= File.join(::Rails.root, 'log', "solr_#{ENV['TEST_ENV_NUMBER'].to_i}.log") %> data_path: <%= File.join(::Rails.root, 'solr', 'data’, "#{ENV['TEST_ENV_NUMBER'].to_i}") %> pid_path: <%= File.join(::Rails.root, 'solr', 'pids', "#{ENV['TEST_ENV_NUMBER'].to_i}") %>

Handling multiple Firefox instances

and Selenium’s use of

shared ephemeral ports

Monitoring Memory and CPU usage

adjust number of parallel process accordingly

Auto rerunning failed testsnon-deterministic failures are likelier with

the CPU under stress

Beyond parallelization…

Stubbing external callsreduce dependency-related delays where

avoidable

Consolidating scenarioscombine similar test scenarios into single

runs instead of separate tests

Maintaining conventions when writing automated tests

avoid time-based wait statements, use test framework APIs that take lesser time, etc

Conclusion

Resourceshttp://cukes.info

http://rubygems.org/gems/capybara

http://seleniumhq.org/projects/remote-control

http://rubygems.org/gems/parallel_tests

http://selenium-grid.seleniumhq.org

http://test-load-balancer.github.com

http://github.com/bblimke/webmock

http://testing.thoughtworks.com

Thank you

king@thoughtworks.comhttp://www.wikyblog.com/AmanKing