JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use...
Transcript of JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use...
![Page 1: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/1.jpg)
JMeter & ELK
Search Meetup Jan. 2015
![Page 2: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/2.jpg)
2
‣ schon mehrere Lasttests durchgeführt: ‣ größere (2500 req/s)
‣ mittlere
‣ aber auch kleine (40 req/s)
‣ grober Ablauf stets:
‣ Testspezifikation
‣ Was genau soll getestet werden?
‣ Was sind die Exit Kriterien?
‣ Tests programmieren und Tests durchführen
‣ Testergebnisse auswerten und interpretieren
‣ Ggf. Anpassungen am object under test und/oder den Tests
‣ rinse and repeat
Hintergrundinfos
![Page 3: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/3.jpg)
3
‣ Was ist „schnell“? Was bedeutet „auch unter Last stabil“? ‣ Wenn möglich: sich an vorhandenen oder ähnlichen Systemen orientieren
‣ Klassischer ELK use-case: Logfiles analysieren
Problemstellung 1: Unklare Anforderungen
![Page 4: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/4.jpg)
1) fairerweise: ist auch eher fürs Entwickeln/Debuggen gedacht 2) auch im JMeter Wiki wird ein „Shell Script to Aggregate Per Minute“ beschrieben (http://wiki.apache.org/jmeter/LogAnalysis) 4
‣ 2500 req/s * 120 Minuten Laufzeit = 18.000.000 req ‣ Zu viel für Excel
‣ Einige JMeter Elemente „töten“ das GUI
‣ „View Results Tree“ ganz sicher1
‣ „Graph Results“ manchmal
‣ Headless ist generell zu bevorzugen
‣ Lösungsansatz ohne ELK war:
‣ aggregator.groovy2
‣ Excel
Problemstellung 2: Datenmenge
![Page 5: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/5.jpg)
5
Testergebnisse Ohne ELK, mit Excel
15 Webserver, überlastet
15 Webserver, überlastet
20 Webserver, stabil
15 Webserver +5 Webserver
![Page 6: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/6.jpg)
http://jmeter.apache.org/images/screenshots/graph_results.png 6
Problemstellung 3: Nachgelagerte Analysen
![Page 7: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/7.jpg)
http://jmeter.apache.org/images/screenshots/response_time_graph.png 7
Problemstellung 3: Nachgelagerte Analysen
![Page 8: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/8.jpg)
8
<...>/JMeter/config/addiontal.properties
# Timestamp format - this only affects CSV output files
# legitimate values: none, ms, or a format suitable for SimpleDateFormat
# Default:
# jmeter.save.saveservice.timestamp_format=ms
jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS
# For use with Comma-separated value (CSV) files or other formats
# where the fields' values are separated by specified delimiters.
# Default:
# jmeter.save.saveservice.default_delimiter=,
jmeter.save.saveservice.default_delimiter=|
<...>/Jmeter/logs/MasterTestPlan.jmx_foo.bar.de_20140708-145452
timeStamp|elapsed|label|responseCode|responseMessage|threadName|dataType|success|
failureMessage|bytes|grpThreads|allThreads|URL|Filename|Latency|Encoding|SampleCount|
ErrorCount|Hostname|IdleTime
2014-07-08_14:54:53.238|167|TC4-ChatPollHttpPost|200|OK|Thread Group 1-1|text|true||537|1|1|
http://foo.bar.de/index.php/?m=lite&module=chat&action=ping&hx1ylvte||167|ISO-8859-1|1|0|
barfoo|0
2014-07-08_14:54:54.074|67|TC4-ChatPollHttpPost|200|OK|Thread Group 1-1|text|true|<...>
Lösung: JMeter ELK Integration JMeter schreibt ein .csv
![Page 9: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/9.jpg)
9
/etc/logstash/conf.d/foobar_jmeter_csv.conf
# use a grok filter to match our event. For debugging use http://grokdebug.herokuapp.com/
grok {
patterns_dir => "/etc/logstash/patterns"
match => [ "message“,
"%{JMETER_TIMESTAMP:timestamp}\|%{NONNEGINT:elapsed:int}\|%{DATA:label}\|
%{POSINT_OR_DATA:responseCode}\|%{DATA:responseMessage}\|%{DATA:threadName}\|
%{DATA:dataType}\|%{DATA:success}\|%{DATA:failureMessage}\|
%{NONNEGINT:bytes:int}\|%{NONNEGINT:grpThreads:int}\|
%{NONNEGINT:allThreads:int}\|%{URI:uri}\|%{DATA:filename}\|
%{NONNEGINT:latency:int}\|%{DATA:encoding}\|%{NONNEGINT:sampleCount:int}\|
%{NONNEGINT:errorCount:int}\|%{HOSTNAME:hostname}\|
%{NONNEGINT:idleTime:int}" ]
}
# use a date filter to extract the @timestamp metadata field from the timestamp field
# remove timestamp when done
date {
match => [ "timestamp", "yyyy-MM-dd_HH:mm:ss.SSS" ]
timezone => "Europe/Berlin"
remove_field => [ "timestamp" ]
}
Lösung: JMeter ELK Integration grok und date Filter zum einlesen der JMeter Daten
![Page 10: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/10.jpg)
10
‣ idle ≈ 1 req/s ‣ elapsed time ≈ 100ms
Testergebnisse „Social Intranet“ 3 Testläufe von Köln aus
![Page 11: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/11.jpg)
11
‣ 100% Last ≈ 45 req/s ‣ elapsed time ≈ 200ms
Testergebnisse „Social Intranet“ 3 Testläufe von Köln aus
![Page 12: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/12.jpg)
12
‣ ~150% Last ≈ 70 req/s ‣ elapsed time ≈ 150ms (?!? *shrug*)
Testergebnisse „Social Intranet“ 3 Testläufe von Köln aus
![Page 13: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/13.jpg)
13
Testergebnisse „Social Intranet“ Testlauf von Hong-Kong mit 100% Last
![Page 14: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/14.jpg)
14
‣ nur zu Beginn einige erfolgreiche Requests
Testergebnisse „Social Intranet“ Zoom auf die ersten 10 Minuten
![Page 15: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/15.jpg)
15
‣ Fehler: großteils Timeouts
Testergebnisse „Social Intranet“ Zoom auf die ersten 10 Minuten
![Page 16: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/16.jpg)
16
‣ Fehler nun: UnknownHostExceptions (DNS?). Zu diesem Zeitpunkt kommen die Requests gar nicht mehr am Server an
Testergebnisse „Social Intranet“ Zoom auf die letzten 10 Minuten
![Page 17: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/17.jpg)
17
‣ cd ~/git-repos/inovex-vagrant/vagrantfiles/ubuntu12-with-jmeter-and-elk ‣ vagrant up
‣ http://localhost:13080/kibana/index.html#/dashboard/file/logstash.json
‣ load ~/git-repos/inovex-vagrant/vagrantfiles/ubuntu12-with-jmeter-and-elk/kibana-dashboard_jmeter.json
Kurze Live-Demo
![Page 18: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/18.jpg)
18
‣ ELK hilft in der Testspezifikation ‣ ELK ist sehr geeignet für die event-basierten Daten aus JMeter
‣ Datenmengen sind kein Problem mehr
‣ Nachträgliche Auswertungen inkl. Drill-down
‣ Korrelationsmöglichkeit zwischen client-seitigen Metriken aus JMeter und server-seitigen Metriken
Fazit & Ausblick
![Page 19: JMeter + ELK - inovex · jmeter.save.saveservice.timestamp_format=yyyy-MM-dd_HH:mm:ss.SSS # For use with Comma-separated value (CSV) files or other formats # where the fields' values](https://reader030.fdocument.pub/reader030/viewer/2022040405/5e9b1851ce75eb2aa916c0c6/html5/thumbnails/19.jpg)
19
Vielen Dank für Ihre Aufmerksamkeit
Kontakt Max Wippert Head of Project Management & Quality Assurance inovex GmbH Office Köln Kupferhütte 4.1 Schanzenstraße 6-20 51063, Köln Mobil: 0173-3181-048 Mail: [email protected]