Evolving Architecture - Further

Evolving architecture - further

Adding the Michelin stars

Leo Lapworth @ YAPC::EU 2008http://leo.cuckoo.org/

(See seperate evolving architecture presentation for first part)

Evolving architecture - further

Adding the Michelin stars

Leo Lapworth @ YAPC::EU 2008http://leo.cuckoo.org/

(See seperate evolving architecture presentation for first part)

Covering

Server configuration

Covering

Server configurationTesting

Covering

Server configurationTestingPerformance

Covering

Server configurationTestingPerformanceWeb Frameworks

Covering

Server configurationTestingPerformanceWeb FrameworksServer maintenance

Covering

Server configurationTestingPerformanceWeb FrameworksServer maintenanceObject relational mapping

Covering

Server configurationTestingPerformanceWeb FrameworksServer maintenanceObject relational mappingCloud Computing

Covering

Server configurationTestingPerformanceWeb FrameworksServer maintenanceObject relational mappingCloud Computing

Kid splashing in puddle(representing the non-

depth nature of this talk)

Deep sea glass squid(representing depth)Kid splashing in puddle

(representing the non-depth nature of this talk)

Deep sea glass squid(representing depth)Kid splashing in puddle

(representing the non-depth nature of this talk)

How to use this talk

How this could work

for me?

How to use this talk

How this could work

for me?

Original content

0%It's about putting the pieces together

Managing many servers

Automatic server setup

Look at Puppet

Puppet & Puppetmaster

Elmo(database)

Bert(app server)

Elmo(database)

Bert(app server)

Ernie(image server)

Elmo(database)

Bert(app server)

Elmo(database)

Bert(app server)

Ernie(image server)

Kermit(Puppetmaster)

Elmo(database)

Bert(app server)

Ernie(image server)

Kermit(Puppetmaster)

Elmo(database)

Bert(app server)

Ernie(image server)

Yes, I know Kermit was Muppetsand below were Sesame Street

What to manage?

Packages (installing/removing/configuring)

What to manage?

Services (running/restarting)

What to manage?

Users (adding/setting passwords)

What to manage?

Cron (installing)

What to manage?

Cron (installing)

Files and directories (installing/removing/checking)

Manage packages

# Editors

package { emacs: ensure => latest }

package { vim: ensure => latest }

package { nano: ensure => absent }

Services

Servicesclass base_ssh { package { "openssh-client": ensure => installed } package { "openssh-server": ensure => installed }

user { sshd: home => "/var/run/sshd", shell => "/usr/sbin/nologin", allowdupe => false, }

service { ssh: ensure => running, pattern => "sshd", require => Package["openssh-server"], }}

Manage users

Manage users user { leouser: user => 'leo', fullname => 'Leo', path => $path, password => '$1$qW*&^%&^%$^&%$.' }

Manage users user { leouser: user => 'leo', fullname => 'Leo', path => $path, password => '$1$qW*&^%&^%$^&%$.' } # Setup ssh keys as well file { "$path/$user/.ssh/authorized_keys": owner => "$user", group => "$user", mode => 600, source => [ "$fileserver/default/$path/$user/ssh/authorized_keys" ], }

Croncron {

script_name:

command => "/usr/local/bin/script.pl foo",

user => "web",

hour => '4',

minute => '30',

ensure => "present";

Node setup

node ernie {

include base_ssh

include base_users

include cron::database

Installing on client

apt-get install puppet

puppetd --server=puppet.domain.com

Monitoring (simply)

Monitoring (simply)With (m)any servers you need

- Monitoring

- Alerts

Install (with puppet!):

Server:

package { "munin": ensure => latest }

Server:

Client:

Server:

Client:

package { "munin-node": ensure => latest }

Quick overview (Munin)

Quick overview (Munin)Server config:

[cuckoo;braga] address 62.222.3.44

And you get:

Quick overview (Munin)Server config:

[cuckoo;braga] address 62.222.3.44

And you get:

What's happening?

Also emails alerts:

cuckoo :: Braga :: WARNINGs:

packets is 0.00 (outside range [:1]),

CRITICALs:

sshd is 0.00 (outside range [1:]),

Custom plugins

Custom pluginsconfigure my_app

Custom pluginsconfigure my_appgraph_title My App

Custom pluginsconfigure my_appgraph_title My Appgraph_vlabel requests

Custom pluginsconfigure my_appgraph_title My Appgraph_vlabel requestsgraph_category Apps

Custom pluginsconfigure my_appgraph_title My Appgraph_vlabel requestsgraph_category Appsgraph_info Requests to my app

Custom pluginsconfigure my_appgraph_title My Appgraph_vlabel requestsgraph_category Appsgraph_info Requests to my apprequests.label Requests

Custom pluginsconfigure my_appgraph_title My Appgraph_vlabel requestsgraph_category Appsgraph_info Requests to my apprequests.label Requestsrequests.type COUNTER

fetch my_app

Custom pluginsconfigure my_appgraph_title My Appgraph_vlabel requestsgraph_category Appsgraph_info Requests to my apprequests.label Requestsrequests.type COUNTER

fetch my_apprequests.value 450424

Virtual Machines

Virtual MachinesPhysical server

Virtual machine

Virtual Machines

Xen - dom0 (Linux, NetBSD, Solaris - controller)

Physical serverVirtual machine

Virtual machine

Virtual Machines

VM - domU (*nix, even some Windoz) guest OS

Virtual machine

Virtual Machines

+ Puppet = configured VMs quickly

Virtual machine

Virtual Machines

+ Puppet = configured VMs quickly

Virtual machine

Testing & VMs

Why use mock objects / databases when you can start a whole new machine in < 10 mins?

Testing & VMs

TDD - Test Driven Development

Testing & VMs

TDD - Test Driven Deployment

Testing & VMs

TDD - Test Driven Deployment

Start with monitoring, your VM is installed when your monitoring reports OKs...

Profiling

Quote from someone talking at a Google Code Jam "Why don't people spend more time profiling so they can make their code faster?"

Profiling

Response: "Developer time is expensive... computing power is not."

Profiling

However if you do need find bottle necks Devel::NYTProf looks fantastic for Perl profiling.

Profiling

However if you do need find bottle necks Devel::NYTProf looks fantastic for Perl profiling.

Or use more computers...

Problem with more computers

More hardware... to go wrong!

Cloud Computing

Cloud storage

Cloud storageuse Net::Amazon::S3;

my $s3 = Net::Amazon::S3->new(

my $s3 = Net::Amazon::S3->new( { aws_access_key_id => $aws_access_key_id,

my $s3 = Net::Amazon::S3->new( { aws_access_key_id => $aws_access_key_id, aws_secret_access_key => $aws_secret_access_key,

retry => 1,

retry => 1, }

retry => 1, });

retry => 1, });$bucket = $s3->bucket('$name_backup_important_stuff');

# store a file in the bucket (any size - handled well)

# store a file in the bucket (any size - handled well)$bucket->add_key_filename( '1.JPG', 'DSC06256.JPG');

# get the file back

# get the file back$bucket->get_key_filename( '1.JPG', 'GET', 'backup.jpg' );

Cloud storage

• No worrying about dead disks

Cloud storage

• No worrying about RAID configuration

Cloud storage

• No worrying about backups

Cloud storage

• Cost of storing 1/2 TB =~ £38 =~ €48 per month (transfer in/out is on top of that)

Cloud storage

• Cost of storing 1/2 TB =~ £38 =~ €48 per month (transfer in/out is on top of that)

• Issue: Trusting someone else with your data

Cloud computing

Cloud computinguse Net::Amazon::EC2;

my $ec2 = Net::Amazon::EC2->new(

my $ec2 = Net::Amazon::EC2->new( AWSAccessKeyId => $aws_access_key_id,

my $ec2 = Net::Amazon::EC2->new( AWSAccessKeyId => $aws_access_key_id, SecretAccessKey => $aws_secret_access_key,

my $ec2 = Net::Amazon::EC2->new( AWSAccessKeyId => $aws_access_key_id, SecretAccessKey => $aws_secret_access_key,);

# developer-tools/Debian-Etch_Catalyst_DBIC_TT.manifest.xml

# developer-tools/Debian-Etch_Catalyst_DBIC_TT.manifest.xml my $ami = 'ami-bdbe5ad4'; # Your AMI here

my $instance = $ec2->run_instances(

my $instance = $ec2->run_instances( ImageId => $ami,

my $instance = $ec2->run_instances( ImageId => $ami, MinCount => 1,

my $instance = $ec2->run_instances( ImageId => $ami, MinCount => 1, MaxCount => 1,

my $instance = $ec2->run_instances( ImageId => $ami, MinCount => 1, MaxCount => 1, SecurityGroup => $secure_group_name,

my $instance = $ec2->run_instances( ImageId => $ami, MinCount => 1, MaxCount => 1, SecurityGroup => $secure_group_name,);

Cloud computing + Puppet

Upload your Xen instance (your own AMI)

Cloud computing + Puppet

Upload your Xen instance (your own AMI)

Can now start and manage as many or as few servers as we want.

EC2 issues

EC2 issuesWhen an instance goes down all local data is lost

S3 is only permanent storage (at the moment)

Gandi - beta

See Léon's talk after the break for more

Job queuing

Running jobs across many machines

Job queuing

- Resizing images

Job queuing

- Resizing images

- Running calculations

Job queuing

- Resizing images

- Running calculations

- Anything that doesn't have to be realtime

Jobs to do

Image resizing

Jobs to do

CalculationsImage resizing

Jobs to do

CalculationsImage resizing File conversions

Put jobs into queue

my $job = TheSchwartz::Job->new(

funcname => $funcname,

arg => $arg,

uniqkey => $uniqkey,

$the_schwartz->insert($job);

Jobs go into the queue

And on each worker

$the_schwartz->can_do($funcname_1);

$the_schwartz->can_do($funcname_2);

$the_schwartz->work_until_done;

Each computer does tasks

Worker 1 Worker 2 Worker 3

1/39/3

Tools to make development easier

DBIx::Class (ORM)

DBIx::Class (ORM)ORM = Object Relational Mapper

= AKA Database tables to objects

my $sth = $dbh->prepare("select * from $table where id = ?");

$sth->execute($id);

my $result = $sth->fetchrow_hashref();

$sth->execute($id);

my $update = $dbh->prepare("update $table set FIELD = ? where id = ?");

$sth->execute($id);

my $update = $dbh->prepare("update $table set FIELD = ? where id = ?");

$update->execute('new value',$result->{id});

DBIx::Class (ORM)

my $result =

DBIx::Class (ORM)

my $result =

$schema->resultset($table)->find($id);

DBIx::Class (ORM)

my $result =

$result->field('new value');

DBIx::Class (ORM)

my $result =

$result->field('new value');

$result->update;

The 'F' word - Frameworks

Do not keep reinventing the wheel

Web development - use tools so you do not re-implement the the basics each time (Catalyst, JIFTY, one of the many others)

Any development - see above

Catalyst

Is an MVC:

Catalyst

Is an MVC:

Model - objects representing data/logic

Catalyst

Is an MVC:

View - templates

Catalyst

Is an MVC:

View - templates

Controller - decisions of what to do based on input (can alter model and choose views)

Catalyst::Plugin::SimpleAuth

__PACKAGE__->config(

simpleauth => { class => 'Users' },

sub auto : Private {

if(!$c->user && $c->action ne 'login') {

$c->res->redirect( $c->uri_for('/login') );

sub auto : Private {

if(!$c->user && $c->action ne 'login') {

$c->res->redirect( $c->uri_for('/login') );

Catalyst::Plugin::*

Session

Catalyst::Plugin::*

Session

Authentication

Catalyst::Plugin::*

Session

Authentication

FormValidator

Catalyst::Plugin::*

Session

Authentication

FormValidator

FillInForm

Catalyst::Plugin::*

Session

Authentication

FormValidator

FillInForm

Catalyst::Plugin::*

Session

Authentication

FormValidator

FillInForm

and {flash} - a read once stash!

Catalyst::Plugin::*

Session

Authentication

FormValidator

FillInForm

and - well lots more

Catalyst::Plugin::*

Session

Authentication

FormValidator

FillInForm

and - well lots more

Exciting times

We have all these tools to play with

Exciting times

It takes time (to learn and setup)

Exciting times

Makes developing so much more fun

We covered...

Server configuration

- kitchens!

Load balancing http

- perlbal

Testing

- Test More

Performance

- mod_gzip

- gzip

- headers

- Devel::NYTProf

Image/application serving

- MogileFS

- mod_perl

Caching

- Memcache

Server maintanence

- puppet

- munin

Object relational mapping

- DBIx::Class

Cloud Computing

We covered...

QUESTIONS?

Slides: http://leo.cuckoo.org/projects/ea/

Evolving Architecture - Further

Technology

Transcript of Evolving Architecture - Further

Evolving PHR Market 20080617

Further Electrical Power

The Evolving Strategic Role of HRM - हे Buddy · PDF fileThe Evolving Strategic Role of HRM •The Role of human resource management in organization has been evolving dramatically

Commercial Banks and Microfinance: Evolving Models of Success ...

Tracking Surfaces with Evolving Topology

Studying Evolving Software Ecosystems Inspired by Ecological Models

Linux Further

ANALYTICAL REPORTS: THEIR EVOLVING

5Jan2016 RISC-V Foundation Overview · § RISC-V Foundation Overview ... , evolving and open-source licensing of the RISC-V instruction set architecture and surrounding hardware and

PDF processed with CutePDF evaluation edition platforms on social media to further our reach. We are expanding our physical branch and network presence, and constantly evolving our

Further Development

Emerging Technologies & Evolving Library

Evolving Science in Bioavailability and Bioequivalence

Evolving Research, Development and

Designer Stimulants - Evolving Abuse Patterns

Search, Social Media & The Evolving Newspaper : LPA 2012

Evolving automation

Chiu Evolving

Further Education Prospectus

The Evolving DMZ