© Copyright IBM Corporation 2012 TrademarksUsing Hadoop with Couchbase Page 1 of 14

Using Hadoop with Couchbase

Martin BrownVP of Technical PublicationsCouchbase

Skill Level: Intermediate

Date: 11 Sep 2012

Hadoop is great for processing large quantities of data and resolving thatinformation down into a smaller set of information that you can query. However,the processing time for that process can be huge. By integrating with CouchbaseServer you can do live querying and reporting on information, while continuingto work with Hadoop for the large data set and heavy processing of the data set.Couchbase Server also uses a MapReduce querying system, which makes it easyfor you to migrate and integrate your indexing and querying system to extract andmanipulate the information effectively.

Hadoop and data processing

Hadoop combines a number of key features together that ultimately makes it veryuseful for processing a large quantity of data down into smaller, usable chunks.

The primary component is the HDFS file system, which allows for information to bedistributed across a cluster. The information stored in this distributed format can alsobe processed individually on each cluster node through a system called MapReduce.The MapReduce process converts the information stored in the HDFS file system intosmaller, processed and more manageable chunks.

Because Hadoop works on multiple nodes, it can be used to process vast quantitiesof input information and simplify it into more usable blocks of information. Thisprocessing is handled by using a simple MapReduce system.

MapReduce is a way of turning the incoming information, which may or may not bein a structured format, and converting it into a structure that can be more easily used,queried and processed.

For example, a typical usage is to process log information from hundreds of differentapplications so that you can identify specific problems, counts or other events.By using the MapReduce format, you can start to measure and look for trends —

developerWorks® ibm.com/developerWorks/

Using Hadoop with Couchbase Page 2 of 14

translating what would otherwise be a very significant quantity of information intoa smaller size. When looking at the logs of a web server, for example, you mightwant to look at the errors that occur within a specific range on specific pages. Youcan write a MapReduce function to identify specific errors on individual pages,and generate that information in the output. Using this method, you can reducemany lines of information from the log files into a much smaller collection of recordscontaining only the error information.

Understanding MapReduceMapReduce works in two phases. The map process takes the incoming informationand maps it into a standardized format. For some information types, this mappingcan be direct and explicit. For example, if you are processing input data such as aweb log, you will be extracting a single column of data from the text of the web log.For other data, the mapping might be more complex. Processing textual information,such as research papers, you might be extracting phrases or more complex blocks ofdata.

The reduce phase is used to collate and summarize the data together. The reductioncan actually take place in a number of different ways, but the typical process is toperform a basic count, sum, or other statistic based on the individual data from themap phase.

Thinking about a simple example, such as the word count used as sampleMapReduce in Hadoop, the map phase breaks apart raw text to identify individualwords, and for each word, generates a block of output data. The reduce function thentakes these blocks of mapped information, and reduces them down to incrementthe count for each unique word seen. Given a single text file of 100 words, the mapprocess would generate 100 blocks of data, but the reduce phase would summarizethis down to provide a count of each unique word into, say, 56 words, with a count ofthe number of times each word appeared.

With web logs, the map would take the input data, create a record for each errorwithin the log file, and generate a block for each error that contains the date, time,and page that caused the problem.

Within Hadoop, the MapReduce phases take place on the individual nodes on whichthe individual blocks of source information are stored. This is what enables Hadoopto work with such large data sets of information — by allowing multiple nodes to workon the data simultaneously. With 100 nodes, for example, you could process 100 logfiles simultaneously and simplify many gigabytes (or terabytes) of information muchquicker than could be achieved through a single node.

Hadoop limitationsOne of the major limitations of the core Hadoop product is that there is no way tostore and query information in the database. Data is added into the HDFS system,but you cannot ask Hadoop to return a list of all the data matching a specific data set.

ibm.com/developerWorks/ developerWorks®

Using Hadoop with Couchbase Page 3 of 14

The primary reason for this is that Hadoop doesn't store, structure, or understand thestructure of the data that is being stored within HDFS. This is why the MapReducesystem is required to parse and process the information into a more structuredformat.

However, we can combine the processing power of Hadoop with a more traditionaldatabase so that we can query the data that Hadoop has generated through it's ownMapReduce system. There are many possible solutions available, including manytraditional SQL databases, but we can keep the MapReduce theme, which is veryeffective for large data sets, by using Couchbase Server.

The basic structure of the data sharing between the systems is shown in Figure 1.

Figure 1. Basic structure of the data sharing between the systems

Installing Hadoop

If you haven't installed Hadoop already, the easiest way is to make use of one of theCloudera installations. For compatibility between Hadoop, Sqoop, and Couchbase,the best solution is to make use of the CDH3 installation (see Resources). For this,you will need to use Ubuntu 10.10 to 11.10. Later Ubuntu releases have introduced

developerWorks® ibm.com/developerWorks/

Using Hadoop with Couchbase Page 4 of 14

an incompatibility because they no longer support a package required by theCloudera Hadoop installation.

Before installation, make sure you have installed a Java™ virtual machine, andensure that you've configured the correct home directory of your JDK in theJAVA_HOME variable. Note that you must have a full Java Development Kitavailable, not just a Java Runtime Environment (JRE), as Sqoop compiles code toexport and import data between Couchbase Server and Hadoop.

To install using the CDH3 on Ubuntu and similar systems, you need to do thefollowing:

1. Download the CDH3 configuration package. This adds the configuration for theCDH3 source files to the apt repository.

2. Update your repository cache: $ apt-get update.3. Install the main Hadoop package: $ apt-get install hadoop-0.20.4. Install the Hadoop components (see Listing 1).

Listing 1. Installing the Hadoop components

$ for comp in namenode datanode secondarynamenode jobtracker tasktrackerdoapt-get install hadoop-0.20-$compdone

5. Edit the configuration files to ensure you've set up the core components.6. Edit /etc/hadoop/conf/core-site.xml to read as shown in Listing 2.

Listing 2. Edited /etc/hadoop/conf/core-site.xml file

<configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property></configuration>

This configures the default hdfs location for storing data.Edit /etc/hadoop/conf/hdfs-site.xml (see Listing 3).

Listing 3. Edited /etc/hadoop/conf/hdfs-site.xml file

<configuration> <property> <name>dfs.replication</name> <value>1</value> </property></configuration>

This enables replication of the stored data.Edit /etc/hadoop/conf/mapred-site.xml (see Listing 4).

ibm.com/developerWorks/ developerWorks®

Using Hadoop with Couchbase Page 5 of 14

Listing 4. Edited /etc/hadoop/conf/mapred-site.xml file

<configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property></configuration>

This enables the job tracker for MapReduce.7. Finally, edit the Hadoop environment to correctly point to the directory of your

JDK installation within the /usr/lib/hadoop/conf/hadoop-env.sh. There will be acommented out line for the JAVA_HOME variable. You should uncomment itand set it to your JDK location. For example: export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk.

8. Now, start up Hadoop on your system. The easiest way is to use the start-all.shscript: $ /usr/lib/hadoop/bin/start-all.sh.

Assuming everything is configured correctly, you should now have a running Hadoopsystem.

Couchbase Server overview

Couchbase Server is a clustered, document-based database system that makes useof a caching layer to provide very fast access to your data by storing the majority ofyour data in RAM. The system makes use of multiple nodes and a caching layer withautomatic sharding across the cluster. This allows for an elastic nature so that youcan grow and shrink the cluster to take advantage of more RAM or disk I/O to helpimprove performance.

All data in Couchbase Server is eventually persisted down to disk, but initially thewrites and updates operate through the caching layer, which is what provides thehigh-performance and which we can exploit when processing Hadoop data to get liveinformation and query the contents.

In its basic form, Couchbase Server remains as a basic document and key/value-based store. You can only retrieve the information from the cluster provided youknow the document ID. In Couchbase Server 2.0, you can store documentsin JSON format, and then use the view system to create a view on the storedJSON documents. A view is a MapReduce combination that is executed over thedocuments stored in the database. The output from a view is an index, matchingthe structure you've defined through the MapReduce functions. The existence of theindex provides you with the ability to query the underlying document data.

We can use this functionality to take the processed data from Hadoop, store thatinformation within Couchbase Server, and then use it as our basis for querying thatdata. Conveniently, Couchbase Server uses a MapReduce system for processing the

developerWorks® ibm.com/developerWorks/

Using Hadoop with Couchbase Page 6 of 14

documents and creating the indexes. This provides some level of compatibility andconsistency with the methods for processing the data.

Installing Couchbase ServerInstalling Couchbase Server is easy. Download the Couchbase Server 2.0 releasefrom the Couchbase website for your platform (see Resources), and install thepackage using dpkg or RPM (depending on your platform).

After installation, Couchbase Server will start automatically. To configure it, open aweb browser and point it to localhost:8091 on your machine (or access it remotelyusing the IP address of the machine).

Follow the on screen configuration instructions. You can use most of the defaultsettings as provided during the installation, but the most important settings are thelocation of the data files for the data written into the database, and the amount ofRAM you allocate to Couchbase Server.

Getting Couchbase Server to talk to the Hadoop connectorCouchbase Server uses the Sqoop connector to communicate your Hadoopcluster. Sqoop provides a connection to transfer data in bulk between Hadoop andCouchbase Server.

Technically, Sqoop is an application designed to convert information betweenstructured databases and Hadoop. The name Sqoop is actually derived from SQLand Hadoop.

Installing SqoopIf you are using the CDH3 installation, you can install sqoop by using your packagemanager: $ sudo apt-get install sqoop.

This will install sqoop in /usr/lib/sqoop.

Note: A recent bug in Sqoop means that it will sometimes try to transfer the wrongdatasets. The fix is part of Sqoop Version 1.4.2. If you experience problems, tryV1.4.2 or a later version.

Installing the Couchbase Hadoop ConnectorThe Couchbase Hadoop Connector is a collection of Java jar files that support theconnectivity between Sqoop and Couchbase. Download the Hadoop connector fromthe Couchbase website (see Resources). The file is packaged as a zip file. Unzip it,and then run the install.sh script inside, supplying the location of the Sqoop system.For example: $ sudo bash install.sh /usr/lib/sqoop.

That installs all the necessary library and configuration files. Now we can startexchanging information between the two systems.

ibm.com/developerWorks/ developerWorks®

Using Hadoop with Couchbase Page 7 of 14

Importing Data from Couchbase Server to HadoopAlthough not the scenario we will directly deal with here, it's worth noting that we canexport data from Couchbase Server into Hadoop. This could be useful if you hadloaded a large quantity of data in Couchbase Server, and wanted to take advantageof Hadoop to process and simplify it. To do this, you can load the entire data set fromthe Couchbase Server into a Hadoop file within HDFS using: $ sqoop import --connect http://192.168.0.71:8091/pools --table cbdata.

The URL provided here is the location of the Couchbase Server bucket pool. Thetable specified here is actually the name of the directory within HDFS where the datawill be stored.

The data itself is stored as a key/value dump of information from Couchbase Server.In Couchbase Server 2.0, this means that the data is written out using the uniquedocument ID, and containing the JSON value of the record.

Writing JSON data in Hadoop MapReduceFor exchanging information between Hadoop and Couchbase Server, we need tospeak a common language -- in this case the JSON (see Listing 5).

Listing 5. Outputting JSON within Hadoop MapReducepackage org.mcslp;

import java.io.IOException;import java.util.*;

import org.apache.hadoop.fs.Path;import org.apache.hadoop.conf.*;import org.apache.hadoop.io.*;import org.apache.hadoop.mapred.*;import org.apache.hadoop.util.*;import com.google.gson.*;

public class WordCount {

public static class Map extends MapReduceBase implements Mapper<LongWritable,Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text();

public void map(LongWritable key, Text value, OutputCollector<Text,IntWritable> output, Reporter reporter) throws IOException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); output.collect(word, one); } } }

public static class Reduce extends MapReduceBase implements Reducer<Text,IntWritable, Text, Text> {

class wordRecord { private String word; private int count; wordRecord() {

developerWorks® ibm.com/developerWorks/

Using Hadoop with Couchbase Page 8 of 14

} }

public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { int sum = 0; while (values.hasNext()) { sum += values.next().get(); }

wordRecord word = new wordRecord(); word.word = key.toString();; word.count = sum;

Gson json = new Gson(); System.out.println(json.toJson(word)); output.collect(key, new Text(json.toJson(word))); } }

public static void main(String[] args) throws Exception { JobConf conf = new JobConf(WordCount.class); conf.setJobName("wordcount");

conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(IntWritable.class);

conf.setMapperClass(Map.class); conf.setReducerClass(Reduce.class);

conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class);

FileInputFormat.setInputPaths(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1]));

JobClient.runJob(conf); }}

The code is a modification of the word counting sample provided with the Hadoopdistribution.

This version uses the Google Gson library for writing JSON information from thereduce phase of the processing. For convenience, a new class is used (wordRecord),which is converted by Gson into a JSON record, which is the format we require ona document by document basis for Couchbase Server to process and parse thecontents.

Note that we do not define a Combiner class for Hadoop. This will prevent Hadoopfrom trying to re-reduce the information, which with the current code will fail becauseour reduce takes in the word and a single digit and outputs a JSON value. For asecondary reduce/combine stage, we would need to parse the JSON input or definea new Combiner class that output the JSON version of the information. This simplifiesthe definition slightly.

ibm.com/developerWorks/ developerWorks®

Using Hadoop with Couchbase Page 9 of 14

To use this within Hadoop, you first need to copy the Google Gson library into theHadoop directory (/usr/lib/hadoop/lib). Then restart Hadoop to ensure that the libraryhas been correctly identified by Hadoop.

Next, compile your code into a directory: $ javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar:./google-gson-2.2.1/gson-2.2.1.jar -d

wordcount_classes WordCount.java .

Now create a jar of your library: $ jar -cvf wordcount.jar -C wordcount_classes/.

With this completed, you can copy a number of text files into a directory, and thenuse this jar to process the text files into a count of the individual words, with a JSONrecord containing the word, and the count. For example, to process this data onsome Project Gutenberg texts: $ hadoop jar wordcount.jar org.mcslp.WordCount /user/mc/gutenberg /user/mc/gutenberg-output.

This will generate a list of words in out directory having been counted by theMapReduce function within Hadoop.

Exporting the data from Hadoop to Couchbase ServerTo get the data back from Hadoop and into Couchbase Server, we need to useSqoop to export the data back: $ sqoop export --connect http://10.2.1.55:8091/pools --table ignored --export-dir gutenberg-output.

The --table argument in this example is ignored, but the --export-dir is the nameof the directory where the information to be exported is located.

Writing MapReduce in Couchbase serverWithin Hadoop, MapReduce functions are written using Java. Within CouchbaseServer, the MapReduce functionality is written in Javascript. As an interpretedlanguage, this means that you do not need to compile the view, and it allows you toedit and refine the MapReduce structure.

To create a view within Couchbase Server, open the admin console (on http://localhost:8091), and then click the View button. Views are collected within a designdocument. You can create multiple views in a single design document and createmultiple design documents. To improve the overall performance of the server, thesystem also supports a development view that can be edited, and a production viewthat cannot be edited. The production view cannot be edited because doing so wouldinvalidate the view index and cause the index to require rebuilding.

Click the Create Development View button and name your design document andview.

Within Couchbase Server, there are the same two functions: map and reduce. Themap function is used to map the input data (JSON documents) to a table. The reduce

developerWorks® ibm.com/developerWorks/

Using Hadoop with Couchbase Page 10 of 14

function is then used to summarize and reduce that. Reduce functions are optionaland are not required for the index functionality, so we'll ignore reduce functions for thepurposes of this article.

For the map function, the format of the function is shown in Listing 6.

Listing 6. Format of the map functionmap(doc) {

}

The argument doc is each stored JSON document. The storage format forCouchbase Server is a JSON document and the view is in Javascript, so we canaccess a field in JSON called count using: doc.count.

To emit information from our map function, you call the emit() function. The emit()function takes two arguments, the first is the key, which is used to select and queryinformation, and the second argument is the corresponding value. Thus we cancreate a map function that outputs the word and the count using the code in Listing 7.

Listing 7. map function that outputs the word and the countfunction (doc) { if (doc.word) { emit(doc.word,doc.count); }}

This will output a row of data for each input document, containing the document ID(actually our word), the word as a key, and the count of incidences of that word in thesource text. You can see the raw JSON output in Listing 8.

Listing 8. The raw JSON output{"total_rows":113,"rows":[{"id":"acceptance","key":"acceptance","value":2},{"id":"accompagner","key":"accompagner","value":1},{"id":"achieve","key":"achieve","value":1},{"id":"adulteration","key":"adulteration","value":1},{"id":"arsenic","key":"arsenic","value":2},{"id":"attainder","key":"attainder","value":1},{"id":"beerpull","key":"beerpull","value":2},{"id":"beware","key":"beware","value":5},{"id":"breeze","key":"breeze","value":2},{"id":"brighteyed","key":"brighteyed","value":1}]}

In the output, id is the document ID, key is the key you specified in the emitstatement, and value is the value specified in the emit statement.

Getting live data

ibm.com/developerWorks/ developerWorks®

Using Hadoop with Couchbase Page 11 of 14

Now that we have processed the information in Hadoop, imported it into CouchbaseServer, and created a view on that data within Couchbase Server, we can begin toquery the information that we have processed and stored. Views are accessed usinga REST like API, or if you are using one of the Couchbase Server SDKs, through thecorresponding view querying functions.

Querying is possible by three main selections:

• Individual key. For example, showing the information matching a specific key,such as 'unkind'.

• List of keys. You can supply an array of key values, and this will return allrecords where the view key matches one of the supplied values. For example,['unkind','kind'] would return records matching either word.

• Range of keys. You can specify a start and end key.

For example, to find the count for a specified word, you use the key argument to theview:

http://192.168.0.71:8092/words/_design/dev_words/_view/byword?connection_timeout= 60000&limit=10&skip=0&key=%22breeze%22

Couchbase Server naturally outputs the results of a MapReduce in UTF-8 orderedfashion, sorted by the specified key. This means that you can get a range of valuesby specifying the start value and end value. For example, to get all the wordsbetween 'breeze' and 'kind:

http://192.168.0.71:8092/words/_design/dev_words/_view/byword?connection_timeout= 60000&limit=10&skip=0&startkey=%22breeze%22&endkey=%22kind%22

The querying is simple, but very powerful, especially when you realize that you cancombine it with the flexible view system to generate data in the format you want.

Conclusion

Hadoop on its own provides a powerful processing platform, but there is nomethod for actually extracting useful information from the data that is processed.By connecting Hadoop to another system, you can use it to query and extractinformation. Since Hadoop uses MapReduce for processing, you can take advantageof the knowledge of MapReduce through the MapReduce system in CouchbaseServer to provide your querying platform. Using this method, you process in Hadoop,export from Hadoop into Couchbase Server as a JSON document, and then useMapReduce in Couchbase Server to query the processed information.

developerWorks® ibm.com/developerWorks/

Using Hadoop with Couchbase Page 12 of 14

ResourcesLearn

• Apache Hadoop Project• Apache Hadoop Distributed File System• HadoopDB Project website• Hadoop MapReduce tutorial: Learn more from this tutorial on Apache.org.• Using MapReduce and load balancing on the cloud (Kirpal A. Venkatesh et al.,

developerWorks, July 2010): Learn how to implement the Hadoop MapReduceframework in a cloud environment and how to use virtual load balancing toimprove the performance of both a single- and multiple-node system.

• CDH3 Installation - Cloudera Support: Find information on installing Hadoopusing CDH3.

• Big Data Glossary By Pete Warden, O'Reilly Media, ISBN: 1449314597, 2011.• Hadoop: The Definitive Guide by Tom White, O'Reilly Media, ISBN:

1449389732, 2010.• HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies

for Analytical Workloads, Azza Abouzeid et al., Proceedings of the VLDBEndowment, 2(1), 2009: This paper explores the feasibility of building a hybridsystem that takes the best features from both technologies.

• MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean andSanjay Ghemawat, OSDI, 2004

• SQL/MapReduce: A practical approach to self-describing, polymorphic, andparallelizable user-defined functions, Eric Friedman et al., Proceedings of theVLDB Endowment, 2(2), 2009: This paper describes the motivation for this newapproach to UDFs as well as the implementation within AsterData Systems'nCluster database.

• MapReduce and parallel DBMSs: friends or foes?, Michael Stonebraker et al.,Commun. ACM 53(1), 2010.

• A Survey of Large Scale Data Management Approaches in CloudEnvironments, Sherif Sakr et al., Journal of IEEE Communications Surveys andTutorials, 13(3), 2011: This paper gives a comprehensive survey of numerousapproaches and mechanisms of deploying data-intensive applications in thecloud which are gaining a lot of momentum in both research and industrialcommunities.

• James Phillips looks ahead in 2012: Tune into this podcast to hear what's goingon at CouchDB.

• Aaron Miller and Nitin Borwankar on CouchDB and the CouchOne mobileplatform: Tune into this podcast to learn more about this full-stack applicationenvironment, written in Erlang, and ported to Android.

• developerWorks Business analytics: Find more analytic technical resources fordevelopers.

ibm.com/developerWorks/ developerWorks®

Using Hadoop with Couchbase Page 13 of 14

• developerWorks Open source: Find extensive how-to information, tools, andproject updates to help you develop with open source technologies and usethem with IBM products.

• developerWorks on Twitter: Join today to follow developerWorks tweets.• developerWorks podcasts: Listen to interesting interviews and discussions for

software developers.• developerWorks technical events and webcasts: Stay current with

developerWorks technical events and webcasts.

Get products and technologies

• Couchbase Server: Download this clustered, document-based databasesystem.

• Couchbase Server Hadoop Connector: Download this collection of Java jarsthat support the connectivity between Sqoop and Couchbase.

• Hadoop 0.20.1, Hadoop MapReduce, and Hadoop HDFS: Download all fromApache.org.

• Evaluation software: Download or explore the online trials in the IBM SOASandbox and get your hands on application development tools and middlewareproducts from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

Discuss

• developerWorks community: Connect with other developerWorks users whileexploring the developer-driven blogs, forums, groups, and wikis.

• developerWorks profile: Create your profile today and set up a watchlist.

developerWorks® ibm.com/developerWorks/

Using Hadoop with Couchbase Page 14 of 14

About the author

Martin Brown

A professional writer for over 15 years, Martin 'MC' Brown is the authorand contributor to over 26 books covering an array of topics, includingthe recently published Getting Started with CouchDB. His expertisespans myriad development languages and platforms Perl, Python,Java, JavaScript, Basic, Pascal, Modula-2, C, C++, Rebol, Gawk,Shellscript, Windows, Solaris, Linux, BeOS, Microsoft® WP, Mac OSand more. He is a former LAMP Technologies Editor for LinuxWorldmagazine and is a regular contributor to ServerWatch.com, LinuxPlanet,ComputerWorld, and IBM developerWorks. He draws on a rich andvaried background as founder member of a leading UK ISP, systemsmanager and IT consultant for an advertising agency and Internetsolutions group, technical specialist for an intercontinental ISP network,and database designer and programmer and as a self-confessedcompulsive consumer of computing hardware and software. MC iscurrently the VP of Technical Publications and Education for Couchbaseand is responsible for all published documentation, training program andcontent, and the Couchbase Techzone.

© Copyright IBM Corporation 2012(www.ibm.com/legal/copytrade.shtml)Trademarks(www.ibm.com/developerworks/ibm/trademarks/)