Mongodb railscamphh

on rails

railscamp hamburg, 2010

jan krutisch <jan@krutisch.de>http://jan.krutisch.de/

Samstag, 23. Oktober 2010

mongodb wtf? lol!?

document database

NoSQLincluded!

open source

http://github.com/mongodb/mongo

id title descr pos_lat pos_lng

{ "_id" : ObjectId("4c00245062610475a005afcd"), "address" : "Bernstorffstr. 174\n22767 Hamburg\nDE", "description" : null, "position" : { "lat" : 53.5600912, "lng" : 9.9596977 }, "tags" : [ "hausarzt", "naturheilverfahren", "akupunktur", "allgemeinmedizin" ], "title" : "Dr. med. Lilo Eisenbarth", "loxicon_id" : 808261 }

✗Samstag, 23. Oktober 2010

BInary Serialized jsON

http://bsonspec.org/Samstag, 23. Oktober 2010

Storage

rich queries

conceptually close to SQL

easy to grasp

flexible

language integration

on top: map/reduce

Scaling

Master/Slave replication

Replica Sets (1.6)

Primary

Member Member

Member Primary

Member

Member Primary

Autosharding (1.6)

Durability

No single server durability!

fsyncs every 60s(configurable)

Use Replication!

Use write propagation locking

Single Server Durability planned for 1.8

mongo console

$ mongo

> use testswitched to db test

db.quotes.save({ text: "You can observe a lot just by watching.", from: "Yogi Berra", created_at: new Date() });

db.quotes.save({ text: "Silence is one of the hardest arguments to refute.", from: "Josh Billings", created_at: new Date() });

> use testswitched to db test

db.quotes.save({ text: "You can observe a lot just by watching.", from: "Yogi Berra", created_at: new Date() });

db.quotes.save({ text: "Silence is one of the hardest arguments to refute.", from: "Josh Billings", created_at: new Date() });

Indexing

Same concept as with SQL databases

You want them

Same concept as with SQL databases

Sort order

Unique

Compound

Geospatial

map/reduce

we can haz it, too

function() { this.tags.forEach(function(z) { emit(z, {count: 1}); });}

function(key, values) { var total = 0; values.forEach(function(v) { total += v.count }); return {count: total}}

(it‘s not fast...)

security

simple user/password auth

per database

read only is possible

one more thing

GridFS

Binary fields in BSON < 4MB

GridFS saves files in chunks

I‘m in u‘r rubies,querying teh MongoDB!

core driver

mongo / bson_ext

ODMs / Libs

mongo_mapper

mongoid

Find examples here:http://github.com/halfbyte/mongo_ruby_examples

Basic driver usage

require 'mongo'

@connection = Mongo::Connection.new@db = @connection.db("test")

@connection = Mongo::Connection.new( 'localhost', 27017, :pool_size => 5, :timeout => 20)

@connection = Mongo::Connection.from_uri( "mongodb://localhost:27017/test")

insert/upsert

doc = { :text => "You can observe a lot just by watching.", :from => "Yogi Berra", :created_at => Time.now}@db['quotes'].insert(doc)

doc = @db['quotes'].find_one(id)

doc[:from] = "Yogi Berra, famous baseball player"

@db['quotes'].save(doc)

atomic updates

@db['quotes'].update( {"from" => "Yogi Berra"}, {"$inc" => {"reads" => 1 } })

$inc$set$unset$push$pushAll

$addToSet$pop$pull$pullAll$

getting a whole collection

@db['quotes'].find.each do |row| puts row.inspectend

exact query

@db['quotes'].find(:from => "Yogi Berra")

more queries

100.times do |i| db['numbers'].insert({"i" => i})end

db['numbers'].find("i" => {"$lt" => 2})

$lt <$gt >$lte <=$gte >=$ne !=

@db['people'].find(:tags => {"$in" => ['cool']})

obj = { "_id"=>BSON::ObjectID('4c706af16261040680000369'), "name"=>"Vernon Kreiger", "address"=>{ "street"=>"536 Haleigh Locks", "city"=>"Port Kiannahaven", "zip"=>"80730-0214", "country"=>"Fakistan" }, "tags"=>["cool", "weird"]}

$in IN (2,3,4)$nin NOT IN$all [2,3] ~ [1,2,3]

$mod yah, RLY$size okay$exists NOT NULL$type huh?

@db['people'].find("address.city" => /haven/)

Sorting

@db['people'].find().sort("address.street")

Pagination

@db['numbers'].find.sort("i").limit(10)

@db['numbers'].find.sort("i").limit(10).skip(50)

Counting

@db['numbers'].find.count

Distinct

@db['people'].distinct('tags').inspect

Poor mans map/reduce

@db['people'].group( ['created_at'], {}, {:tags => {}}, reduce, finalize)

function(doc, prev) { for(i in doc.tags) { if (doc.tags[i] in prev.tags) { prev.tags[doc.tags[i]]++ } else { prev.tags[doc.tags[i]] =1 } }}

{"created_at"=>2010-09-19 22:00:00 UTC, "tags"=>{"foo"=>11.0, "dumb"=>12.0, "stupid"=>7.0, "bar"=>7.0, "cool"=>14.0, "weird"=>17.0}}{"created_at"=>2010-09-20 22:00:00 UTC, "tags"=>{"dumb"=>11.0, "stupid"=>5.0, "foo"=>10.0, "cool"=>8.0, "weird"=>9.0, "bar"=>15.0}}{"created_at"=>2010-09-22 22:00:00 UTC, "tags"=>{"weird"=>15.0, "bar"=>9.0, "stupid"=>17.0, "cool"=>11.0, "dumb"=>10.0, "foo"=>12.0}}{"created_at"=>2010-09-15 22:00:00 UTC, "tags"=>{"foo"=>11.0, "weird"=>7.0, "stupid"=>10.0, "cool"=>11.0, "bar"=>5.0, "dumb"=>8.0}}{"created_at"=>2010-09-25 22:00:00 UTC, "tags"=>{"dumb"=>11.0, "weird"=>14.0, "cool"=>8.0, "foo"=>21.0, "bar"=>11.0, "stupid"=>13.0}}{"created_at"=>2010-09-28 22:00:00 UTC, "tags"=>{"cool"=>11.0, "dumb"=>16.0, "stupid"=>11.0, "weird"=>15.0, "foo"=>9.0, "bar"=>16.0}}{"created_at"=>2010-09-10 22:00:00 UTC, "tags"=>{"dumb"=>15.0, "weird"=>9.0, "bar"=>8.0, "cool"=>14.0, "stupid"=>11.0, "foo"=>11.0}}{"created_at"=>2010-09-03 22:00:00 UTC, "tags"=>{"weird"=>14.0, "dumb"=>15.0, "stupid"=>11.0, "foo"=>16.0, "bar"=>20.0, "cool"=>10.0}}{"created_at"=>2010-09-21 22:00:00 UTC, "tags"=>{"weird"=>15.0, "cool"=>14.0, "foo"=>13.0, "stupid"=>6.0, "bar"=>11.0, "dumb"=>9.0}}{"created_at"=>2010-09-23 22:00:00 UTC, "tags"=>{"weird"=>15.0, "stupid"=>15.0, "dumb"=>15.0, "foo"=>16.0, "cool"=>10.0, "bar"=>11.0}}{"created_at"=>2010-09-29 22:00:00 UTC, "tags"=>{"bar"=>9.0, "cool"=>14.0, "weird"=>16.0, "foo"=>8.0, "dumb"=>9.0, "stupid"=>12.0}}{"created_at"=>2010-09-27 22:00:00 UTC, "tags"=>{"cool"=>13.0, "dumb"=>10.0, "stupid"=>12.0, "bar"=>8.0, "foo"=>16.0, "weird"=>13.0}}{"created_at"=>2010-09-04 22:00:00 UTC, "tags"=>{"cool"=>11.0, "bar"=>9.0, "stupid"=>6.0, "weird"=>11.0, "dumb"=>8.0, "foo"=>11.0}}{"created_at"=>2010-09-08 22:00:00 UTC, "tags"=>{"stupid"=>12.0, "dumb"=>11.0, "cool"=>15.0, "foo"=>11.0, "bar"=>9.0, "weird"=>8.0}}{"created_at"=>2010-10-02 22:00:00 UTC, "tags"=>{"bar"=>8.0, "dumb"=>8.0, "cool"=>10.0, "foo"=>10.0, "stupid"=>8.0, "weird"=>6.0}}{"created_at"=>2010-09-24 22:00:00 UTC, "tags"=>{"foo"=>13.0, "bar"=>12.0, "stupid"=>15.0, "weird"=>17.0, "dumb"=>7.0, "cool"=>10.0}}{"created_at"=>2010-09-30 22:00:00 UTC, "tags"=>{"bar"=>10.0, "cool"=>6.0, "stupid"=>14.0, "weird"=>9.0, "dumb"=>12.0, "foo"=>19.0}}{"created_at"=>2010-09-05 22:00:00 UTC, "tags"=>{"dumb"=>12.0, "foo"=>19.0, "weird"=>8.0, "stupid"=>8.0, "bar"=>7.0, "cool"=>10.0}}{"created_at"=>2010-09-17 22:00:00 UTC, "tags"=>{"weird"=>13.0, "bar"=>14.0, "dumb"=>12.0, "foo"=>12.0, "stupid"=>10.0, "cool"=>9.0}}{"created_at"=>2010-09-18 22:00:00 UTC, "tags"=>{"dumb"=>8.0, "cool"=>11.0, "foo"=>6.0, "bar"=>12.0, "weird"=>7.0, "stupid"=>6.0}}{"created_at"=>2010-09-09 22:00:00 UTC, "tags"=>{"weird"=>11.0, "dumb"=>9.0, "foo"=>6.0, "bar"=>11.0, "cool"=>11.0, "stupid"=>6.0}}{"created_at"=>2010-09-13 22:00:00 UTC, "tags"=>{"dumb"=>19.0, "stupid"=>9.0, "weird"=>12.0, "cool"=>11.0, "bar"=>10.0, "foo"=>15.0}}{"created_at"=>2010-09-16 22:00:00 UTC, "tags"=>{"bar"=>6.0, "weird"=>8.0, "dumb"=>9.0, "cool"=>11.0, "stupid"=>17.0, "foo"=>15.0}}{"created_at"=>2010-09-11 22:00:00 UTC, "tags"=>{"foo"=>10.0, "weird"=>9.0, "bar"=>8.0, "cool"=>4.0, "dumb"=>8.0, "stupid"=>9.0}}{"created_at"=>2010-09-26 22:00:00 UTC, "tags"=>{"dumb"=>15.0, "weird"=>6.0, "stupid"=>15.0, "bar"=>10.0, "foo"=>13.0, "cool"=>15.0}}{"created_at"=>2010-10-01 22:00:00 UTC, "tags"=>{"cool"=>7.0, "weird"=>11.0, "stupid"=>11.0, "bar"=>14.0, "foo"=>12.0, "dumb"=>11.0}}{"created_at"=>2010-09-12 22:00:00 UTC, "tags"=>{"bar"=>7.0, "weird"=>12.0, "stupid"=>11.0, "cool"=>10.0, "foo"=>11.0, "dumb"=>9.0}}{"created_at"=>2010-09-14 22:00:00 UTC, "tags"=>{"dumb"=>8.0, "foo"=>15.0, "cool"=>15.0, "stupid"=>15.0, "bar"=>7.0, "weird"=>14.0}}{"created_at"=>2010-09-07 22:00:00 UTC, "tags"=>{"dumb"=>10.0, "cool"=>7.0, "foo"=>14.0, "weird"=>15.0, "bar"=>11.0, "stupid"=>7.0}}{"created_at"=>2010-09-06 22:00:00 UTC, "tags"=>{"dumb"=>7.0, "bar"=>11.0, "cool"=>16.0, "weird"=>14.0, "foo"=>12.0, "stupid"=>6.0}}

"tags" => { "foo" => 11.0, "dumb" => 12.0, "stupid" => 7.0, "bar" => 7.0, "cool" => 14.0, "weird" => 17.0}

function(prev) { var mostPopular = 0; for(i in prev.tags) { if(prev.tags[i] > mostPopular) { prev.tag = i; prev.count = prev.tags[i]; mostPopular = prev.tags[i]; } } delete prev.tags}

{"created_at"=>2010-09-27 22:00:00 UTC, "tag"=>"stupid", "count"=>18.0}{"created_at"=>2010-09-29 22:00:00 UTC, "tag"=>"stupid", "count"=>20.0}{"created_at"=>2010-09-12 22:00:00 UTC, "tag"=>"cool", "count"=>11.0}{"created_at"=>2010-09-04 22:00:00 UTC, "tag"=>"stupid", "count"=>12.0}{"created_at"=>2010-09-21 22:00:00 UTC, "tag"=>"stupid", "count"=>16.0}{"created_at"=>2010-09-03 22:00:00 UTC, "tag"=>"foo", "count"=>15.0}{"created_at"=>2010-09-26 22:00:00 UTC, "tag"=>"foo", "count"=>17.0}{"created_at"=>2010-09-18 22:00:00 UTC, "tag"=>"foo", "count"=>17.0}{"created_at"=>2010-09-24 22:00:00 UTC, "tag"=>"cool", "count"=>11.0}

Map / Reduce

map = <<-END function() { this.tags.forEach(function(z) { emit(z, {count: 1}); }); }ENDreduce = <<-END function(key, values) { var total = 0; values.forEach(function(v) { total += v.count }); return {count: total} }END

collection = @db['people'].map_reduce( map, reduce)

Indexes

db['people'].create_index("tags")

@db['people'].create_index( [["tags", Mongo::ASCENDING]])

db['people'].drop_index("tags_1")

db['people'].drop_indexes

db['people'].index_information

Geospatial stuff

@db['people'].create_index( [["latlng", Mongo::GEO2D]])

@db['people'].find( "latlng" => {"$near" => [53.593978, 10.107380]})

GridFS usage

grid = Mongo::Grid.new(@db)

id = grid.put("You can put Strings in here", :filename => 'test.txt')

file = grid.get(id)

file.filenamefile.read

grid.delete(id)

grid.put( File.open("/Users/jankrutisch/Dropbox/Photos/IMGP8989.jpg"))

fs = Mongo::GridFileSystem.new(db)

fs.open("test.txt", "w") do |f| f.write "You can put stuff in here"end

fs.open("test.txt", "r") do |f| puts f.readend

fs.delete("test.txt")

Capped collections

@db.create_collection('capped_numbers', :capped => true, :max => 50)

@db.create_collection('capped_numbers', :capped => true, :size => 1024 * 64)

explain

@db['people'].find( "address.city" => /haven/).explain

{ "cursor"=>"BasicCursor", "nscanned"=>1000, "nscannedObjects"=>1000, "n"=>39, "millis"=>2, "indexBounds"=>{}, "allPlans"=>[ {"cursor"=>"BasicCursor", "indexBounds"=>{}} ]}

{ "cursor"=>"BtreeCursor address.city_1 multi", "nscanned"=>1000, "nscannedObjects"=>39, "n"=>39, "millis"=>1, "indexBounds"=>{ "address.city"=>[["", {}], [/haven/, /haven/]] }, "allPlans"=>[ { "cursor"=>"BtreeCursor address.city_1 multi", "indexBounds"=>{ "address.city"=>[["", {}], [/haven/, /haven/]] } } ]}

mongo_mapper

John Nunemaker@jnunemaker

is in production

documentation?

how to

rails initializer

# config/initializers/mongo_mapper.rbFile.open(File.join(Rails.root, 'config/mongodb.yml'), 'r') do |f| @settings = YAML.load(f)[Rails.env]end

MongoMapper.connection = Mongo::Connection.from_uri(@settings["connection"]) if @settings["connection"]

MongoMapper.database = @settings["database"]

a simple example

MongoMapper.connection = @connectionMongoMapper.database = "test"

class Quote include MongoMapper::Document key :from key :text key :views, Integer timestamps!end

finders

Quote.where(:from => 'Yogi Berra').all

Quote.where(:from => 'Yogi Berra').limit(5).sort(:from.desc).all

embedded docs

class Person include MongoMapper::Document key :name one :address key :tags, Arrayend

class Address include MongoMapper::Document key :street key :city key :country key :zipend

person = Person.firstaddress = Person.first.address

scopes

class Person scope :tagged, lambda { |tag| where(:tags.in => [tag]) }end

puts Person.tagged('cool').first.inspect

new website coming soon

mongoid

Durran Jordan(of Hashrocket)

Two major versions

1.x (1.9.x) targeting Rails 2.3.x

2.x (2.0beta) targeting Rails 3.0

Good documentation

rails initializer

File.open(File.join(RAILS_ROOT, 'config/mongodb.yml'), 'r') do |f| @settings = YAML.load(f)[RAILS_ENV]end

Mongoid::Config.instance.from_hash(@settings)

a simple example

class Quote include Mongoid::Document include Mongoid::Timestamps field :from field :text field :views, :type => Integerend

finders

Quote.where(:from => 'Yogi Berra').all

Quote.where(:from => 'Yogi Berra').limit(5).order_by(:from.desc).all

embedded docs

class Person include Mongoid::Document field :name embeds_one :address field :tags, :type => Arrayend

class Address include Mongoid::Document field :street field :city field :country field :zipend

person = Person.firstaddress = Person.first.address

scopes

class Person scope :tagged, lambda { |tag| where(:tags.in => [tag]) }end

puts Person.tagged('cool').first.inspect

More features

atomic updates

mongoid tries to be clever

(using the „dirty“ flags)

(it‘s probably better to bypass the ODM sometimes)

GridFS

external libraries for both

mongo_mapper > grip

mongoid > mongoid_grid

Other noteworthy libraries

Other not so noteworthy libraries

thanks for listening.

‣ jan@krutisch.de‣ http://jan.krutisch.de/‣ http://github.com/halfbyte/‣ http://twitter.com/halfbyte‣ http://www.mindmatters.de/

‣ http://www.mongodb.org/‣ http://www.mongoid.org/‣ http://github.com/jnunemaker/mongo_mapper‣ http://github.com/halfbyte/mongo_ruby_examples

Mongodb railscamphh

Technology

Transcript of Mongodb railscamphh

MongoDB + Spark

MongoDB @ Globo.com

Laporan mongodb awal

Apresentação - MongoDB

Workshop MongoDB

NoSQL - MongoDB

NoSQL Concepts MongoDB Concepts MongoDB Demos Agenda.

Attacking MongoDB

Palestra mongoDB

Introduccion mongodb

The MongoDB Strikes Back / MongoDB 의 역습

Nosql & MongoDB

MongoDB Basic

ToursJUG mongoDB

MongoDB and Amazon Web Services: Storage Options for MongoDB Deployments

Mongodb @ vrt

MongoDB & Spark

MongoDB very basic (Japanese) / MongoDB基礎の基礎

MongoDB Europe 2016 - MongoDB Atlas

MongoDB Profiler Deep Dive; MongoDB Austin 2013