CMU monitoring tool - proto
-
Upload
goutam-adwant -
Category
Documents
-
view
175 -
download
0
Transcript of CMU monitoring tool - proto
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
What does the tool do?
Purpose of this tool is to run jobs that would extract core,disk,memory & network - detail + summary data from CMU( using collect & Scripts ) for a specified time frame and then store it in MySQL DB for the purpose of reporting/graphing & analysis of the test results.
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Technologies:
Front end:•HTML•CSS•JQuery
Server Side:•PHP/PHPSECLIB – Used to exec/run monitoring dump commands on mgmt. server & connect to MySQL dB to store metadata of the test run. •Python – Used to extract & read dump file and connect to MySQL dB & insert the data to its corresponding detail & summary table.Tools:•HeidiSQL – To display mgmt. server MySQL data in a tabular form •CMU – To view the data of the defined metrics – Core,Disk,Memory & Network.
Build a simple front end screen through which user can input test details & time frame.
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Component functional view :
Test DescriptionStart TimeEnd Time
Submit
Front End Screen
PHP Script
CMU Python Script
Excel/CSVChartsReporting
Connect & Generates dump file
Returns Dump file
Triggers
Exec .py scripts
Reads the dump file & inserts into respective tables
MySQL
Entire code deployed on tomcat server of mgmt. node.AAA File
CPU & Disk Scripts
/proc/stat
Cpucore0-39
Disk1-13
Request is hit only once and the result is stored in cache so that all the core & disk data are retrieved from that.
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Schema Flow diagram:
Test Execution
Test metrics summary
cpu_util_all disk_metrics
Detail Level Data
Summary Level Data
1 row per each test execution
1 row/node during a test run
1 row/detail of node during a test run
Meta Data
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Test Execution data – captures metadata for each test execution
DB Snapshot
SchemaCREATE TABLE `test_execution` ( `test_id` VARCHAR(45) NOT NULL, `start_time` DATETIME NULL DEFAULT NULL, `end_time` DATETIME NULL DEFAULT NULL, `test_desc` VARCHAR(45) NULL DEFAULT NULL, `test_result` VARCHAR(45) NULL DEFAULT NULL, `test_user` VARCHAR(45) NULL DEFAULT NULL, `metrics_calc` TINYINT(1) NULL DEFAULT NULL, `results_captrured` TINYINT(1) NULL DEFAULT NULL, `stdout_captured` TINYINT(1) NULL DEFAULT NULL, PRIMARY KEY (`test_id`));Test id: Program generated metric. It is unique for every job executed. Acts as a foreign key to the meta & Summary table/data. start_time: User input data – start time of the data that needs to be extracted.End_time : User input data – end time of the data that needs to be extracted.Test_desc: User input data – test description of the test run that the user submits.Test_result,test_user,metrics_calc,results_captured,stdout_captured: not yet populated – to be decided.
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Test Execution data – Metric Details
Metric Name SourceTest_id Program generated metric- unique
for each test runStart_time User input data End_time User input data Test_desc User input data
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Test Metrics Summary data –
Captures node level summary data. It contains core,disk,network & memory information of each node for 5sec time span.
Schema :
CREATE TABLE `test_metrics_summary` ( `test_id` INT(11) NULL DEFAULT NULL, `servername` VARCHAR(45) NOT NULL, `date_time` VARCHAR(45) NOT NULL, `interval_time` DATETIME NOT NULL, `data_disks_read_mb` DECIMAL(10,2) NULL DEFAULT NULL, `data_disks_write_mb` DECIMAL(10,2) NULL DEFAULT NULL, `data_disks_read_iops` VARCHAR(45) NULL DEFAULT NULL, `data_disks_write_iops` VARCHAR(45) NULL DEFAULT NULL,`total_cpu` INT(11) NULL DEFAULT NULL, `sys_disks_read_mb` VARCHAR(45) NULL DEFAULT NULL,`sys_disks_write_mb` VARCHAR(45) NULL DEFAULT NULL, `sys_disks_read_iops` VARCHAR(45) NULL DEFAULT NULL,`sys_disks_write_iops` VARCHAR(45) NULL DEFAULT NULL, `nw_pkt_in` VARCHAR(45) NULL DEFAULT NULL,`nw_pkt_out` VARCHAR(45) NULL DEFAULT NULL, `memory_used` VARCHAR(45) NULL DEFAULT NULL,`total_avail_memory` VARCHAR(45) NULL DEFAULT NULL, `memory_free` VARCHAR(45) NULL DEFAULT NULL, PRIMARY KEY (`servername`, `date_time`, `interval_time`));
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Metric Name Sourcetest_id Program generated metric- unique for each test
runstart_time, end_time User input data data_disks_read_mb, data_disks_write_mb /opt/cmu/tools/cmu_get_det_disk -n disk –rs
/opt/cmu/tools/cmu_get_det_disk -n disk -wsdata_disks_read_iops, data_disks_write_iops /opt/cmu/tools/cmu_get_det_disk -n disk –rc
/opt/cmu/tools/cmu_get_det_disk -n disk -wc
system_disks_read_mb, system_disks_write_mb /opt/cmu/tools/cmu_get_det_disk -n sys –rs/opt/cmu/tools/cmu_get_det_disk -n sys -ws
system_disks_read_iops, system_disks_write_iops
/opt/cmu/tools/cmu_get_det_disk -n sys–rc/opt/cmu/tools/cmu_get_det_disk -n sys -wc
total_cpu /opt/cmu/tools/cmu_get_det_cpu -c -n cpunw_out_mbps, nw_in_mbps COLLECTL (nettotals.kbout) / 1024
COLLECTL (nettotals.kbin) / 1024total_memory COLLECTL (meminfo.tot)memory_free COLLECTL (meminfo.tot) - (meminfo.used)memory_used COLLECTL (meminfo.used) / (meminfo.tot) *
100
Test Metrics Summary data – Metric Details
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
CPU/Core - Detail Data – captures core level detail
DB Snapshot
SchemaCREATE TABLE `cpu_util_all` ( `test_id` INT(11) NOT NULL, `interval_time` INT(11) NOT NULL, `servername` VARCHAR(45) NOT NULL, `date_time` VARCHAR(45) NOT NULL, `cpu_user` VARCHAR(45) NOT NULL, `cpu_core_value` VARCHAR(45) NULL DEFAULT NULL, PRIMARY KEY (`test_id`, `interval_time`, `servername`, `date_time`, `cpu_user`));
Test id: Program generated metric. It is unique for every job executed. Acts as a foreign key to the meta & Summary table/data. Interval_time: Program generated metric. For a particular test_id, it increments for every 5seconds of test result data. ( e.g.. For 1min of test data – we have 12 interval_time id’s )Servername: Generated through the dump file.Data_time: Generated through the dump file.cpu_user: Metric data generated through the AAA file.cpu_core_value: Metric data generated through the AAA file.
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Metric Name Sourcetest_id Program generated metric- unique for each test
runInterval_time Program generated metric. For a particular
test_id, it increments for every 5seconds of test result data
Servername Generated through the dump file.date_time Generated through the dump file. Increments
with 5secs from start time to end time.cpu_user Generated through detail script in AAA.
cpucore_00 to cpucore_39cpu_core_value /opt/cmu/tools/cmu_get_det_cpu -n cpu00-39
CPU/Core data – Metric Details
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Disk – Detail Data:
DB Snapshot
SchemaCREATE TABLE `disk_stats_all` ( `test_id` INT(11) NOT NULL, `interval_time` INT(11) NOT NULL, `servername` VARCHAR(45) NOT NULL, `date_time` VARCHAR(45) NOT NULL, `disk_name` VARCHAR(45) NOT NULL, `disk_read_mbps` VARCHAR(45) NULL DEFAULT NULL, `disk_write_mbps` VARCHAR(45) NULL DEFAULT NULL, `disk_reads` VARCHAR(45) NULL DEFAULT NULL, `disk_writes` VARCHAR(45) NULL DEFAULT NULL, PRIMARY KEY (`test_id`, `interval_time`, `servername`, `date_time`, `disk_name`));Test id: Program generated metric. It is unique for every job executed. Acts as a foreign key to the meta & Summary table/data. Interval_time: Program generated metric. For a particular test_id, it increments for every 5seconds of test result data. ( e.g.. For 1min of test data – we have 12 interval_time id’s )Servername: Generated through the dump file.Data_time: Generated through the dump file.Disk_name: Inserted through the python script for the number of disks present for a particular node.disk_read_mbps, disk_write_mbps,disk_reads,disk_writes : Metric data generated through the AAA file.
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Metric Name Sourcetest_id Program generated metric- unique for each test
runInterval_time Program generated metric. For a particular
test_id, it increments for every 5seconds of test result data
Servername Generated through the dump file.date_time Generated through the dump file. Increments
with 5secs from start time to end time.disk_name Generated through detail script in AAA.disk-1 to
disk-13disk_read_mbps,disk_write_mbps /opt/cmu/tools/cmu_get_det_disk -n disk1-13 –
rs /opt/cmu/tools/cmu_get_det_disk -n disk1-13 –ws
disk_reads,disk_writes /opt/cmu/tools/cmu_get_det_disk -n disk1-13 -rc/opt/cmu/tools/cmu_get_det_disk -n disk1-13 –wc
Disk data – Metric Details
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
• Select `cmudata`.`test_metrics_summary`.`test_id` AS `test_id`,`cmudata`.`test_metrics_summary`.`interval_time` AS `interval_time`,avg(`cmudata`.`test_metrics_summary`.`total_cpu`) AS `avg(total_cpu)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_read_mb`) AS `sum(data_disks_read_mb)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_write_mb`) AS `sum(data_disks_write_mb)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_read_iops`) AS `sum(data_disks_read_iops)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_write_iops`) AS `sum(data_disks_write_iops)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_read_mb`) AS `sum(sys_disks_read_mb)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_write_mb`) AS `sum(sys_disks_write_mb)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_read_iops`) AS `sum(sys_disks_read_iops)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_write_iops`) AS `sum(sys_disks_write_iops)`,sum(`cmudata`.`test_metrics_summary`.`nw_pkt_in`) AS `sum(nw_pkt_in)`,sum(`cmudata`.`test_metrics_summary`.`nw_pkt_out`) AS `sum(nw_pkt_out)`,sum(`cmudata`.`test_metrics_summary`.`memory_used`) AS `sum(memory_used)`,sum(`cmudata`.`test_metrics_summary`.`total_avail_memory`) AS `sum(total_avail_memory)`,sum(`cmudata`.`test_metrics_summary`.`memory_free`) AS `sum(memory_free)` from `cmudata`.`test_metrics_summary` where (`cmudata`.`test_metrics_summary`.`servername` like 'f02wn%') group by `cmudata`.`test_metrics_summary`.`test_id`,`cmudata`.`test_metrics_summary`.`interval_time` order by `cmudata`.`test_metrics_summary`.`test_id`,`cmudata`.`test_metrics_summary`.`interval_time
Views:
test_metrics_north_aggregate : Retrieves the aggregate level data filtered only for worker nodes and ordered by test_id & interval id.
test_metrics_south_aggregate : Retrieves the aggregate level data filtered only for data nodes and ordered by test_id & interval id.
• Select `cmudata`.`test_metrics_summary`.`test_id` AS `test_id`,`cmudata`.`test_metrics_summary`.`interval_time` AS `interval_time`,avg(`cmudata`.`test_metrics_summary`.`total_cpu`) AS `avg(total_cpu)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_read_mb`) AS `sum(data_disks_read_mb)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_write_mb`) AS `sum(data_disks_write_mb)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_read_iops`) AS `sum(data_disks_read_iops)`,sum(`cmudata`.`test_metrics_summary`.`data_disks_write_iops`) AS `sum(data_disks_write_iops)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_read_mb`) AS `sum(sys_disks_read_mb)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_write_mb`) AS `sum(sys_disks_write_mb)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_read_iops`) AS `sum(sys_disks_read_iops)`,sum(`cmudata`.`test_metrics_summary`.`sys_disks_write_iops`) AS `sum(sys_disks_write_iops)`,sum(`cmudata`.`test_metrics_summary`.`nw_pkt_in`) AS `sum(nw_pkt_in)`,sum(`cmudata`.`test_metrics_summary`.`nw_pkt_out`) AS `sum(nw_pkt_out)`,sum(`cmudata`.`test_metrics_summary`.`memory_used`) AS `sum(memory_used)`,sum(`cmudata`.`test_metrics_summary`.`total_avail_memory`) AS `sum(total_avail_memory)`,sum(`cmudata`.`test_metrics_summary`.`memory_free`) AS `sum(memory_free)` from `cmudata`.`test_metrics_summary` where (`cmudata`.`test_metrics_summary`.`servername` like 'f02dn%') group by `cmudata`.`test_metrics_summary`.`test_id`,`cmudata`.`test_metrics_summary`.`interval_time` order by `cmudata`.`test_metrics_summary`.`test_id`,`cmudata`.`test_metrics_summary`.`interval_timetest_metrics_summary : Retrieves the aggregate level data for both
the nodes and ordered by test_id & interval id
HP Restricted
CMU Performance Monitoring tool - PrototypeIntern Name: Goutam Adwant
Final Output:Data stored in MySQL Db. is saved as excel/csv and graphs/charts can be generated.