An SNMP based failure detection service...• Consensus, Atomic broadcast, Atomic commitment •...

An SNMP based failure detection service

Matthias Wiesmann, Péter Urbán, Xavier Défago

FN NFSSwiss National Science Foundation

北陸先端科學技術大學院大學

1


Overview

• Failure Detection

• SNMP

• SNMP-FD Service

• Conclusion & Future Work

2

2


Failure detection

• Basic abstraction for distributed systems

• Needed for solving fundamental problems:

• Consensus, Atomic broadcast, Atomic commitment

• Formal definitions: P, ♢S, Ω, φ, etc.

• Also outside distributed system community

• System, Network, Cluster management, Middleware, Grid systems

• Common problem→ common solution

3

3

Failure detection service


•Factorize-out failure detection

• Shared, separate implementation

• Complex, adaptive policies

• Proposed in the literature

• Implemented by middleware systems, grid systems

• Problem: no standard

•Every system has its own service and interfaces

4

4


Standard Needed

• External standard

• Connect clients of the service to the service• Application, Middleware, Monitoring and administration tools

• Suspicion notifications, configuration

• Internal standard

• Use standard for heartbeat messages, internal queries

• We need some form of Middleware

5

5


Overview


• SNMP

• SNMP-FD Service


6

6


Why SNMP ?

• Simple Network Management Protocol

• Present in many network attached devices now

• Routers, Smart Switches, UPS, Printers…

• Simple Middleware

• Offers basic features

• Relatively lightweight

• Many applicable standard defined

• Monitoring, fault reporting.

7

7

Network

EquipmentNetwork

Management

StationSNMP

Agent

Monitored

ObjectMIB


SNMP Architecture

• Network Management Station

• Manages equipment

• Equipment’s agent

• Exposes Management Information Base (MIB)

• Sends asynchronous messages (traps) to NMS.

8

8


Overview


• SNMP

• SNMP-FD Service


9

9

!"#$%&'

!"#$%&'S

()*$'B!"#$%"&'()&"*'++,

p1B

-,."+%,&'+"/&*'+,!01

-,2"%$!*3%$"#,4,53&6'%,!01

-,.'3&%7'3%+,!01

!"#$%&'&($%)*$+,-&.%*,+/01,

!"#$%"&'()&"*'++,

p2B

!"#$%"&'()&"*'++,

p3B

-,!01,

23(4&56,%*

()*$'A

7,*

8+09:

2,*&;&7,*

<,0+*=,0*&

8+09:!"#$%&'&()%&$*"+",)-./%"012

!"34)-5"012

-,.'3&%7'3%,8#"9:'(6',!01

-,)&"*'++,8#"9:'(6',!01

-,;3$:/&',<'%'*%$"#,!01,

!"#$%&'&($%)*$+)%6&.%*,+/01,&

!"#$%"&$#6)&"*'++

pA

3$*)!10*)$%

2,*&;&7,*

6%)*7)-7"012

*/+%"=,!01

()

(*

)>?+$*3:,@$#A

!'++36'+

B/'&$'+,C,D&$%'+


SNMP-FD service

10

10


Service Architecture

• Distributed architecture

• Dæmon runs on each host

• Monitor local processes

• Exchange heartbeats

• Export MIB information

• Can interact with equipment

• Similar to other failure detection service architectures

11

!"#$%&'

!"#$%&'S

()*$'B!"#$%"&'()&"*'++,

p1B

-,."+%,&'+"/&*'+,!01

-,2"%$!*3%$"#,4,53&6'%,!01

-,.'3&%7'3%+,!01

!"#$%&'&($%)*$+,-&.%*,+/01,

!"#$%"&'()&"*'++,

p2B

!"#$%"&'()&"*'++,

p3B

-,!01,

23(4&56,%*

()*$'A

7,*

8+09:

2,*&;&7,*

<,0+*=,0*&

8+09:!"#$%&'&()%&$*"+",)-./%"012

!"34)-5"012

-,.'3&%7'3%,8#"9:'(6',!01

-,)&"*'++,8#"9:'(6',!01

-,;3$:/&',<'%'*%$"#,!01,

!"#$%&'&($%)*$+)%6&.%*,+/01,&

!"#$%"&$#6)&"*'++

pA

3$*)!10*)$%

2,*&;&7,*

6%)*7)-7"012

*/+%"=,!01

()

(*

)>?+$*3:,@$#A

!'++36'+

B/'&$'+,C,D&$%'+

11


Features

• Uses many standard MIBs

• Host Resource MIB → Process state

• Target & Notification MIB → Subscriber description

• Alarm MIB → Current suspicion description

• Lightweight messaging:

• Heartbeat are UDP traps.

• Leverages SNMP

• Configuration, Security, Interoperability with devices12

❏✔

12


New MIBs

• Heartbeat MIB

• Describes heartbeats

• Failure detection MIB

• Describes failure detector configuration

• Described using standard SNA syntax file

• Accessible using standard tools

13

SNMP table: SnmpFD::kbhbTable

index interval lastbeat lasttime count lostcount mean stddev min max kt-dhcp23.jaist.ac.jp 100 2678 0:0:04:38.49 2671 0 99.9955 28.3417 0 356

13


Implementation

• Validate the architecture

• Implemented in Java

• Open source libraries

• SNMP4J, SNMP4J Agent, Apache Libraries

• Available for download

• http://ddsg.jaist.ac.jp//en/projects/snmp-fd/

14

14

http://ddsg.jaist.ac.jp//en/projects/snmp-fd/

http://ddsg.jaist.ac.jp//en/projects/snmp-fd/

• Simple failure detector

• Stable performance

• Consistent with expectations

• Validates approach

• Issues

• High latency

• High CPU load on receiver


Performance

15

0.001

0.01

0.1

1

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Timeout (ms)

Mis

takes

per

sec

ond

99.85%

99.9%

99.95%

100.0%

Quer

y

Acc

ura

cyMistake Rate

Query Accuracy

15

• Simple failure detector

• Stable performance

• Consistent with expectations

• Validates approach

• Issues

• High latency

• High CPU load on receiver


Performance

15

0.001

0.01

0.1

1

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Timeout (ms)

Mis

takes

per

sec

ond

99.85%

99.9%

99.95%

100.0%

Quer

y

Acc

ura

cyMistake Rate

Query Accuracy

}SNMP4JIssue, code spends 95% of time there.

15


Overview


• SNMP

• SNMP-FD Service


16

16


Conclusion

• Failure detection service

• Standard based approach desirable

• No need for new standard → use existing SNMP

• Approach validated by prototype

• Can be extended with more advance FD techniques

• Hierarchical FD, adaptive FD, etc.

• Can interact with network devices

• For instance link-down traps

17

17


Future work

• Look into performance issues

• Optimizations needed in the SNMP4J library.

•Port the framework to the latest version of SNMP4J

• Support for AgentX

• Implement framework as sub-agent

• Implement more advanced failure detectors

• Handle network equipment data

18

18


Questions?

19

19


Questions?

19

Thank you…

19

An SNMP based failure detection service...• Consensus, Atomic broadcast, Atomic commitment •...

Documents

Transcript of An SNMP based failure detection service...• Consensus, Atomic broadcast, Atomic commitment •...