Cannibalising Google App Engine

Kevin NoonanCalbane Ltd.IT@Cork, 30th September, 2009

Introduction

Cannibalising Google App Engine

A working definition of Cloudware

• Outsourced Virtualizationo Virtual server running on outsourced server farm

The Cloud Computing Pyramid

•Michael Sheehan of GoGrid devised this classification

What's Cloudware good for? (I)

W

• Guerilla developmento Pay-per-use model is ideal for startups

o Also good for guerillas inside the enterprise...

What's Cloudware good for? (II)

W

• Scaling upo Clone the virtual server

o pay for extra resources (bandwidth, storage, RAM, etc.)

p

What's Cloudware good for? (III)

W

• In general, outsourcing IT infrastructureo pay someone else to run your servers, network, database...

Python: The Original of the Species

Java: the All New Flavour

Languages on App Engine's JVM

• Java

• JRuby

• Jython

• BeanShell

• JavaScript

• Scala

• Groovy

• …

• Will it Play in App Engine?

http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine

What is Google App Engine?• Complete web stack

• Runs on Google's infrastructure

• Bigtable Datastore

o Like a sorted hashtable

o Not a relational database

o Use JDO or JPA (with Java) for datastore work

• Memcache

• Image API

• Mail API

• SDK for development on Windows/Mac/Linux

o Run & test locally before uploading to Cloud

Limitations of Google App Engine• Applications run in “The Sandbox”:

http://code.google.com/appengine/docs/java/runtime.html#The_Sandbox

• Many Java libraries will not work (but many have been patched or will be).

See the JRE whitelist:

http://code.google.com/appengine/docs/java/jrewhitelist.html

No “long-lived” processing (requests must return in 30 seconds).

• HTTPS only on XYZ.appspot.com

o and only with Google's certificate.

o Due to limitation of SSL protocol, as applied to Google's Cloud.

Billing on Google App Engine

• True pay-per-use

o free quotas & no bill to pay if quotas not exceeded.

o quite different model to AWS

• Developer can assign budget per resource.

• Prices

o $0.10 per CPU core-hour

o $0.15 per GB-month of storage

o $0.12 per GB outgoing bandwidth

o $0.10 per GB incoming bandwidth

o $0.0001 per email recipient for emails sent by the application

Free Quotas on App Engine

•Daily outgoing bandwidth: 10 GB

•Daily incoming bandwidth 10 GB

•Daily requests 1.3 million

• Daily CPU-hours: 46

• More quotas (mail, datastore, images, memcache)

http://code.google.com/appengine/docs/quotas.html

Other limitations on App Engine

• 1 GB of free storage per app.

•100 MB Memcache storage limit

• App Engine apps can now receive requests and send responses of up to 10MB.

• Caveat: 1 MB limit on datastore/memcache API call

• You can deploy up to 1,000 code and 1,000 static files with an app.

Workaround: use ZipImport (with Python) to store files in a Zip-archive:

http://code.google.com/appengine/articles/django10_zipimport.html

• 150MB cumulative limit on an app's code and static files

(as opposed to storage in the datastore).

JIQL for JDBC emulation Cannibalising Google App Engine

What is JIQL?o

o Abstraction layer over the Google datastore, Bigtable

o

o Bigtable is non-relational, but JIQL simulates a relational database

o

o JIQL enables the use of JDBC and SQL with Google Bigtable

How to use JIQLo Download Jiql.jar from http://jiql.googlecode.com/files/jiql.jar

o Place Jiql.jar under WEB-INFlib

o Create a connection as follows:

String url = "jdbc:jiql://local";

String password = "";

String user = "";

Properties props = new Properties();

props.put("user",user);

props.put("password",password);

Class clazz = Class.forName("org.jiql.jdbc.Driver");

Driver driver = (Driver) clazz.newInstance();

Connection Conn = driver.connect(url,props);

Supported SQL syntax on JIQLo SUPPORTED SQL SYNTAX

o INSERT

o SELECT

o DROP

o CREATE

o CONSTRAINT

o DISTINCT

o FOREIGN KEY

o PRIMARY KEY

o ALTER

o AND

o OR

o JOINS

o STRING FUNCTIONS

o ORDER

Supported JDBC statements in JIQLo * Connection

o * Statement

o * ResultSet

o * PreparedStatement

o * Driver

o * DatabaseMetaData

o * ResultSetMetaData

GaeVFS: a virtual filesystem on BigtableCastles in the Cloud

What is GaeVFS ? o Filesystem abstraction over Google BigTable

(To enable writes to the “local filesystem” in Google's Cloud)

o

o Implemented as an Apache Commons VFS plugin

o

o Uses Memcache too under the hood

o

o The URI format is

gae://path

o

o Best to use relative paths

Using GaeVFS import org.apache.commons.vfs.FileSystemManager;

import org.apache.commons.vfs.FileObject;

import com.newatlanta.commons.vfs.provider.gae.GaeVFS;

.

GaeVFS.setRootPath( getServletContext.getRealPath( "/" ) );

.

FileSystemManager fsManager = GaeVFS.getManager();

FileObject tmpDir = fsManager.resolveFile( "gae://WEB-INF/tmp" );

if ( !tmpDir.exists() ) {

tmpDir.createFolder();

}

Clear the Cache for GAEVFS Clear the cache at the end of every servlet request!

public void doGet( HttpServletRequest req, HttpServletResponse res )

throws ServletException, IOException {

try {

// ...process request...

} finally {

GaeVFS.clearFilesCache();

}

}

Close the connection to GaeVFS It's good practice to close the connection when your servlet is destroyed.

public void destroy() {

GaeVFS.close(); // this is not strictly required, but good practice

}

2/27/2009

RESTletsCastles in the Cloud

What is the RESTlet project? o An open source project to make RESTful web services easy to

implement in Java.

o

o There's a special build for App Engine, a snapshot tagged “unstable”

o

How to use RESTlets on App Engineo Create a new project with the Eclipse plugin

o

o Add "org.restlet.gae.jar" to the war/WEB-INF/lib directoy

o

o Add "org.restlet.gae.jar" to the project build path

o

o

o

Create a simple RESTful resourcepackage simpleREST;

import org.restlet.resource.Get;

import org.restlet.resource.ServerResource;

//simple resource

public class simpleResource extends ServerResource {

@Get

public String represent() {

return "A word from the Cloud gods ";

}

}

Create a parent app for resourcepackage simpleREST;

import org.restlet.Application;

import org.restlet.Restlet;

import org.restlet.routing.Router;

public class simpleRESTApplication extends Application {

@Override

public synchronized Restlet createRoot() {

Router router = new Router(getContext());

router.attachDefault(simpleResource.class);

return router;

}

}

2/27/2009

Scheduled tasksCastles in the Cloud

Scheduled tasks on App Engineo Aka “Cron jobs”

o Vital for housekeeping and maintenance tasks, especially on the

database

o Useful for various polling exercises

o Significant in a system that is single-threaded and usually activated by

incoming HTTP request.

o

o “Web hook pattern”:

Cron jobs work by invoking a URL, just like an incoming web-request

oMaximum 20 cron jobs per app

oMax one per minute (use “webcron” service to overcome limit)

cron.xml in WEB-INF directory<?xml version="1.0" encoding="UTF-8"?>

<cronentries>

<cron>

<url>/polling</url>

<description>Poll for some event every couple of minutes</description>

<schedule>every 2 minutes</schedule>

</cron>

<cron>

<url>/stats</url>

<description>Send out statistics every week</description>

<schedule>every monday 07:30</schedule>

<timezone>America/New_York</timezone>

</cron>

</cronentries>

English syntax in the cron schedule

every 5 minutes

every 12 hours

2nd,third mon,wed,thu of march 17:00

every monday 09:00

1st monday of sep,oct,nov 17:00

every day 00:00

Updating your cron jobso Update the whole “app” via Eclipse plugin or on the command-line:

appcfg update

Update just your cron-jobs

appcfg update-cron

JRuby-on-RailsCannibalising Google App Engine

Photo credit: http://www.flickr.com/photos/geoftheref/

JRubyo One of the popular new dynamic languages on the JVM

o Highest performing Ruby implementation

o Excellent compatibility with “the standard Ruby” (MRI)

o Rails runs without code-changes

o Native Rails plugins will not run on Jruby

o Core developers lately moved from SUN to Engine Yard

o

o The next four slides are a recipe for JRuby-on-Rails...

Generate a Rails appo Install the gems

$ sudo gem install google-appengine

o Assign an app identifier

Simply create an application at appspot.com

o Generate your rails app

$ rails my-app; cd my-app;

Configure Rack (middleware for Rails)o Generate a rackup file: In the same folder, save this as config.ru.

require 'appengine-rack'

AppEngine::Rack.configure_app(

:application => 'my-app',

:version => 1 )

%w{R db/ doc/ log/ script/ test/ tmp/}.each do |x|

AppEngine::Rack.app.resource_files.exclude "/#{x}**"

end

ENV['RAILS_ENV'] = AppEngine::Rack.environment

require 'config/environment'

run ActionController::Dispatcher.new

Install gems (libraries) & generate modelo Check gem sources for datamapper

$ gem sources

o Only when missing, add it like this...

$ sudo gem source -a http://gems.datamapper.org

o Install gems into your app

$ appcfg.rb gem install rails dm-appengine rails_datamapper

o #Generate a model

$ ./script/generate dm_model book title:string summary:text image_url:string \

available:boolean pages:integer -f --skip-timestamps --skip-migration

Run app locally & deploy to Cloudo Run your app locally

$ dev_appserver.rb .

o Open up the console

$ appcfg.rb run -S irb

> require 'config/environment'

o Deploy to App Engine

$ appcfg.rb update .

Task Queues for background processingCannibalising Google App Engine

:Image courtesy the SeaWiFS Project, NASA/Goddard Space Flight Center & ORBIMAGEVisible Earth Project: http://visibleearth.nasa.gov/view_rec.php?id=2600

Task Queues: Background processingoMotivation: App Engine is single-threaded.

o Place small, discrete units of work on a queue.

o App Engine monitors work-queue & executes task in background.

o

o Configure queues in web.xml (in WEB-INF directory)

oWithout any configuration, each app gets a queue named “default”

o Default rate: 5 tasks per second

o

o Tasks comprises: data payload and code to process it.

oWeb-hook pattern

o Asynchronous, parallel processing in background

Add a task to a queue

import com.google.appengine.api.labs.taskqueue.Queue;

import com.google.appengine.api.labs.taskqueue.QueueFactory;

import static com.google.appengine.api.labs.taskqueue.TaskOptions.Builder.*;

// ...

Queue queue = QueueFactory.getDefaultQueue();

queue.add(url("/worker").param("key", key))

queue.add(url("/path/to/my/worker"));

queue.add(url("/path?a=b&c=d").method("GET"));

XMPP: instant messages & peer-to-peerCannibalising Google App Engine

Photo credit: NOAA Photo Library and NSSL: http://www.photolib.noaa.gov/nssl/nssl0113.htm

XMPP – Instant message protocolo “Jabber” peer-to-peer protocol for instant messaging

o Standardised as “XML Message Passing Protocol“

o

o IM protocol in use by “Google Talk”

o Twitter uses XMPP under-the-hood

o

o An enabling technology of “The realtime Web”

oMakes peer-to-peer architectures possible with App Engine

o

o XMPP is useful for all kinds of chat / IM applications

o Also for interprocess communication.

Construct an instant message...import com.google.appengine.api.xmpp.JID;

import com.google.appengine.api.xmpp.Message;

import com.google.appengine.api.xmpp.MessageBuilder;

import com.google.appengine.api.xmpp.SendResponse;

import com.google.appengine.api.xmpp.XMPPService;

import com.google.appengine.api.xmpp.XMPPServiceFactory;

// ...

JID jid = new JID("somebody@gmail.com");

String msgBody = "Hello from the Cloud!";

Message msg = new MessageBuilder()

.withRecipientJids(jid)

.withBody(msgBody)

.build();

Check on “presence” & send messageboolean messageSent = false;

XMPPService xmpp = XMPPServiceFactory.getXMPPService();

if (xmpp.getPresence(jid).isAvailable()) {

SendResponse status = xmpp.sendMessage(msg);

messageSent = (status.getStatusMap().get(jid) ==

SendResponse.Status.SUCCESS);

}

if (!messageSent) {

// Send an email message instead...

}

Configuration to receive messagesoWeb hook pattern again

o Configuration: edit appengine-web.xml file; include these lines:

<inbound-services>

<service>xmpp_message</service>

</inbound-services>

o when App Engine receives a chat message for the app, it makes an

HTTP POST to:

/_ah/xmpp/message/chat/

Code to receive messageso import java.io.IOException;

o import javax.servlet.http.*;

o import com.google.appengine.api.xmpp.JID;

o import com.google.appengine.api.xmpp.Message;

o import com.google.appengine.api.xmpp.XMPPService;

o import com.google.appengine.api.xmpp.XMPPServiceFactory;

o @SuppressWarnings("serial")

o public class XMPPReceiverServlet extends HttpServlet {

o public void doPost(HttpServletRequest req, HttpServletResponse res)

o throws IOException {

o XMPPService xmpp = XMPPServiceFactory.getXMPPService();

o Message message = xmpp.parseMessage(req);

o JID fromJid = message.getFromJid();

o String body = message.getBody(); // .......

o }

}

Map the URL to the servletTo map the servlet to the XMPP URL path, add the following in the

web.xml file, inside the <web-app> element:

<servlet>

<servlet-name>xmppreceiver</servlet-name>

<servlet-class>XMPPReceiverServlet</servlet-class>

</servlet>

<servlet-mapping>

<servlet-name>xmppreceiver</servlet-name>

<url-pattern>/_ah/xmpp/message/chat/</url-pattern>

</servlet-mapping>

Clojure• Elegant

• Functional

• Dynamic

• Fast

• Excellent concurrency (unusable on App Engine, unfortunately)

• Direct access to Java

• Lisp Reloaded

(Feature-list above by Stuart Halloway.)

Running Clojure on App Engine

• How to run Servlets on App Engine with Clojure

http://fragmental.tw/2009/04/08/clojure-on-google-app-engine/

(“Getting Cloudy”, by Phillip Calçado of Thoughtworks)

•How to run Compojure (a web-framework for Clojure) on App Engine

http://elhumidor.blogspot.com/2009/04/clojure-on-google-appengine.html

(“Clojure on Google App Engine”, by John Hume of Thoughtworks)

Servlets with Clojure (sample-code)

Code from Phillip Calçado's BestMateServlet.clj

(ns BestMateServlet

(:gen-class

:extends javax.servlet.http.HttpServlet)

(:use mate))

(defn -doGet [_ req resp]

(process-request req resp))

Resources for Clojure

o Disclojure: http://disclojure.org/where-to-start/

– tutorial

– Book

– main Clojure site

– Plugins for IDEs

– Videos

• VimClojure: http://kotka.de/projects/clojure/vimclojure.html

Store Static Content on App EngineCannibalising Google App Engine

Photo credit: Nikolai, http://www.blogger.com/profile/13624955858359692394

Getting started with App Engine

To use App Engine, you can begin with your current Gmail address and

password or with a new Google identity you create for the purpose.

Sign up for App Engine: http://appengine.google.com

(You'll need to receive an SMS on your mobile 'phone.)

Create a project on App Engine

Log into http://appengine.google.com and create a new app. (Many

project-names will be taken already, since App Engine shares its

namespace with Gmail.)

The project name you choose will determine the default domain-name.

For example, if "blissedbride" is available,

your app's default domain will be:

http://blissedbride.appspot.com

Store static web-assets in Google's CloudInstall the plugin from e.g. (for Galileo)

http://dl.google.com/eclipse/plugin/3.5

Select the File menu > New > Web Application Project

Supposing you picked "blissedbride" as your project-name.

Navigate to this directory: workspace/blissedbride/war"

Dump all of your web-assets in this directory. Your index.html will

overwrite the placeholder file. And the rest of your HTML files can be

here too. You can put other web-assets in subdirectories.

Wrapping upCannibalising Google App Engine

Photo credit: NOAA Photo Library, NOAA Central Library; OAR/ERL/National Severe Storms Lab

Book (I)

B

Developing with Google App Engine

by Eugene Ciurana

Published:

Feb 2009

http://apress.com/

Book (II)

B

Google App Engine in Action

by Noah Gift & Michael Orr

To be published:

May 2009

http://www.manning.com/gift/

Book (III)

B

Google App Engine

by Dan Sanderson

To be published:

Sept 2009

http://oreilly.com/

• Google App Engine: http://appengine.google.com

• Videos from conference Google held in 2009:o http://sites.google.com/site/io/

• Chat with developers from the App Engine teamo first Wednesday of the month: 3am next day (Dublin/London time).

o third Wednesday of the month: 5pm (Dublin/London time).

o IRC channel: #appengine on irc.freenode.net

o http://en.wikipedia.org/wiki/List_of_IRC_clients

• Google Groupo http://groups.google.com/group/google-appengine/

• Paper on Bigtable: o http://labs.google.com/papers/bigtable.html

Resources for Google App Engine

Credits• Photos used under Creative Commons license:

• Photo Creditso http://www.flickr.com/photos/notsogoodphotography/

o http://www.flickr.com/photos/barto/

o http://www.flickr.com/photos/kitoy/

o http://www.flickr.com/photos/wolfgangstaudt/

o http://www.flickr.com/photos/motumboe/

o http://www.flickr.com/photos/ragnar1984/

o http://www.flickr.com/photos/janekm/

o http://www.flickr.com/photos/raylopez/

o http://www.flickr.com/photos/pagedooley/

o http://www.flickr.com/photos/gustty/

o http://www.flickr.com/photos/mike9alive/

o http://www.flickr.com/photos/tambako/

o http://www.flickr.com/photos/nate

o http://www.flickr.com/photos/freewine/

Contact Information

Kevin Noonan

kevin_noonan on Twitter

http://www.calbane.com