Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
Image

Book Review: Hadoop Beginner's Guide 57

Posted by samzenpus
from the read-all-about-it dept.
First time accepted submitter sagecreek writes "Hadoop is an open-source, Java-based framework for large-scale data processing. Typically, it runs on big clusters of computers working together to crunch large chunks of data. You also can run Hadoop in "single-cluster mode" on a Linux machine, Windows PC or Mac, to learn the technology or do testing and debugging. The Hadoop framework, however, is not quickly mastered. Apache's Hadoop wiki cautions: "If you do not know about classpaths, how to compile and debug Java code, step back from Hadoop and learn a bit more about Java before proceeding." But if you are reasonably comfortable with Java, the well-written Hadoop Beginner's Guide by Garry Turkington can help you start mastering this rising star in the Big Data constellation." Read below for the rest of Si's review.
Hadoop Beginner's Guide
author Garry Turkington
pages 374
publisher Packt Publishing
rating 9/10
reviewer Si Dunn
ISBN 9781849517300
summary Explains and shows how to use Hadoop software in Big Data settings.
Dr. Turkington is vice president of data engineering and lead architect for London-based Improve Digital. He holds a doctorate in computer science from Queens University of Belfast in Northern Ireland. His Hadoop Beginner's Guide provides an effective overview of Hadoop and hands-on guidance in how to use it locally, in distributed hardware clusters, and out in the cloud.

Packt Publishing provided a review copy of the book. I have reviewed one other Packt book previously.

Much of the first chapter is devoted to "exploring the trends that led to Hadoop's creation and its enormous success." This includes brief discussions of Big Data, cloud computing, Amazon Web Services, and the differences between "scale-up" (using increasingly larger computers as data needs grow) and "scale-out" (spreading the data processing onto more and more machines as demand expands).

Dr. Turkington writes, "One of the most confusing aspects of Hadoop to a newcomer is its various components, projects, sub-projects, and their interrelationships."

His 374-page book emphasizes three major aspects of Hadoop: (1) its common projects; (2) the Hadoop Distributed File System (HDFS); and (3) MapReduce.

He explains, "Common projects comprise a set of libraries and tools that help the Hadoop product work in the real world."

The HDFS, meanwhile, "is a filesystem unlike most you may have encountered before." As a distributed filesystem, it can spread data storage across many nodes. "[I]t stores files in blocks typically at least 64 MB in size, much larger than the 4-32 KB seen in most filesystems." The book briefly describes several features, strengths, weaknesses, and other aspects of HDFS.

Finally, MapReduce is a well-known programming model for processing large data sets. Typically, MapReduce is used with clusters of computers that perform distributed computing. In the "Map" portion of the process, a single problem is split into many subtasks that are then assigned by a master computer to individual computers known as nodes (and there can be sub-nodes). During the "Reduce" part of the task, the master computer gathers up the processed data from the nodes, combines it and outputs a response to the problem that was posed to be solved. (MapReduce libraries are now available for many different computer languages, including Hadoop.)

"The developer focuses on expressing the transformation between source and result data sets, and the Hadoop framework manages all aspects of job execution, parallelization, and coordination," Dr. Turkington notes. He calls this "possibly the most important aspect of Hadoop. The platform takes responsibility for every aspect of executing the processing across the data. After the user defines the key criteria for the job, everything else becomes the responsibility of the system."

In this 11-chapter book, the first two chapters introduce Hadoop and explain how to install and run the software.

Three chapters are devoted to learning to work with MapReduce, from beginner to advanced levels. And the author stresses: "In the book, we will be learning how to write MapReduce programs to do some serious data crunching and how to run them on both locally managed and AWS-hosted Hadoop clusters." ["AWS" is "Amazon Web Services."]

Chapter 6, titled "When Things Break" zeroes in on Hadoop's "resilience to failure and an ability to survive failures when they do happen.much of the architecture and design of Hadoop is predicated on executing in an environment where failures are both frequent and expected." But node failures and numerous other problems still can arise, so the reader is given an overview of potential difficulties and how to handle them.

The next chapter, "Keeping Things Running," lays out what must be done to properly maintain a Hadoop cluster and keep it tuned and ready to crunch data.

Three of the remaining chapters show how Hadoop can be used elsewhere within an organization's systems and infrastructure, by personnel who are not trained to write MapReduce programs.

Chapter 8, for example, provides "A Relational View on Data with Hive." What Hive provides is "a data warehouse that uses MapReduce to analyze data stored on HDFS," Dr. Turkington notes. "In particular, it provides a query language called HiveQL that closely resembles the common Structured Query Language (SQL) standard."

Using Hive as an interface to Hadoop "not only accelerates the time required to produce results from data analysis, it significantly broadens who can use Hadoop and MapReduce. Instead of requiring software development skills, anyone with a familiarity with SQL can use Hive," the author states.

But, as Chapter 9 makes clear, Hive is not a relational database, and it doesn't fully implement SQL. So the text and code examples in Chapter 9 illustrate (1) how to set up MySQL to work with Hadoop and (2) how to use Sqoop to transfer bulk data between Hadoop and MySQL.

Chapter 10 shows how to set up and run Flume NG. This is a distributed service that collects, aggregates, and moves large amounts of log data from applications to Hadoop's HDFS.

The book's final chapter, "Where to Go Next," helps the newcomer see what else is available beyond the Hadoop core product. "There are," Dr. Turkington emphasizes, "a plethora of related projects and tools that build upon Hadoop and provide specific functionality or alternative approaches to existing ideas." He provides a quick tour of several of the projects and tools.

A key strength of this beginner's guide is in how its contents are structured and delivered. Four important headings appear repeatedly in most chapters. The "Time for action" heading singles out step-by-step instructions for performing a particular action. The "What just happened?" heading highlights explanations of "the working of tasks or instructions that you have just completed." The "Pop quiz" heading, meanwhile, is followed by short, multiple-choice questions that help you gauge your understanding. And the "Have a go hero" heading introduces paragraphs that "set practical challenges and give you ideas for experimenting with what you have learned."

Hadoop can be downloaded free from the Apache Software Foundation's Hadoop website.

Dr. Turkington's book does a good job of describing how to get Hadoop running on Ubuntu and other Linux distributions. But while he assures that "Hadoop does run well on other systems," he notes in his text: "Windows is supported only as a development platform, and Mac OS X is not formally supported at all." He refers users to Apache's Hadoop FAQ wiki for more information. Unfortunately, few details are offered there. So web searches become the best option for finding how-to instructions for Windows and Macs.

Running Hadoop on a Windows PC typically involves installing Cygwin and openSSH, so you can simulate using a Linux PC. But other choices can be found via sites such as Hadoop Wizard and Hadoop on Windows with Eclipse".

To install Hadoop on a Mac running OS X Mountain Lion, you will need to search for websites that offer how-to tips. Here is one example.

There are other ways get access to Hadoop on a single computer, using other operating systems or virtual machines. Again, web searches are necessary. The Cloudera Enterprise Free product is one virtual-machine option to consider.

Once you get past the hurdle of installing and running Hadoop, Garry Turkington's well-written, well-structured Hadoop Beginner's Guide can start you moving down the lengthy path to becoming an expert user.

You will have the opportunity, the book's tagline states, to "[l]earn how to crunch big data to extract meaning from the data avalanche."

Si Dunn is an author, screenwriter, and technology book reviewer.

You can purchase Hadoop Beginner's Guide from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

*

This discussion has been archived. No new comments can be posted.

Book Review: Hadoop Beginner's Guide

Comments Filter:
  • Wasn't Java being frowned upon for being insecure and so on? But at the same time, Hadoop (Java-based) is praised?
    What's the right path in this "Don't use Java!"/"DO use Java!" debacle?

    • Re:Confused! (Score:5, Insightful)

      by jandrese (485) <kensama@vt.edu> on Wednesday March 13, 2013 @01:44PM (#43162085) Homepage Journal
      Hadoop is not a browser plugin.
      • I never worked with Java, so honestly I didn't know. All I hear left and right is "Java sux because vulnerabilities". So I was wondering.
        But judging from the smug answers below, I made the impardonable mistake of not knowing EVERYTHING. Oh well.

      • by mesterha (110796)

        Why is this insightful? Shouldn't this comment have a low moderation value. A high moderation leads people into wasting their time look at this post and into a parent post who needed some help. Why should anyone else care.

        Of course this is not the fault of the parent or the poster. Is moderation really this crappy? I guess moderators want to moderate something in this article and there just aren't any good posts.

    • by admdrew (782761)

      Wasn't Java being frowned upon for being insecure and so on?

      Could you elaborate, or provide sources?

      Hadoop provides an answer to very specific questions involving large amounts of data, and isn't intended to be a database or other storage mechanism.

    • There are alternatives to Java for writing your MR programs if you have better things to do that worry about class paths etc :-) http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/ [michael-noll.com]

      And back in the 80's BT used PL/1 (plus a bit of Fortran 77) to do Map reduce as the core of a billing system for the dialcom systems -even manged ro sell it to other telcos as a replacement for the Dialcom product.
    • by roman_mir (125474)

      Well, I think [slashdot.org] that great amount of confusion results from the way Java is marketed. For a website that has a larger than normal number of technical people, /. as an aggregate still displays fundamental misunderstanding of what Java is and what it is not.

      For example Java is not Javascript. A browser sandbox that runs Java plugin has nothing to do with server side Java applications, that for example can run in servlet containers and servers like Tomcat Apache, Jetty, iPlanet, Resin (by Caucho), Enhydra, and

      • For people who apparently pride themselves being 'above marketing', developers somehow fell victims to various marketing around Java and didn't bother to check for themselves what the hell all this is about.

        As a developer who never had many reasons to do very much with (actual) Java, I have to say I lost of respect for the platform when Oracle started bundling the runtime with the Ask toolbar on Windows. Because nothing says "serious computing platform" like having to make sure it doesn't install a toolbar on every update. /rant

        • by roman_mir (125474)

          As a developer who never had many reasons to do very much with (actual) Java, I have to say I lost of respect for the platform when Oracle started bundling the runtime with the Ask toolbar on Windows.

          - I don't know, as a thinking human I have to say that what Oracle does with Java browser plugin reflects poorly on Oracle and has nothing to do with Java. As I said, the browser plugin is irrelevant for majority of what Java is actually used for.

          Personally I don't even have Java browser plugin on any of my machines because I don't have a use for it. At the same time I have developed plenty Java applications (as in Java language running on a JVM that resides on a server and has some form of application co

          • As an FYI, it's not the browser plugin that has the opt-out toolbar installation, it's the JVM itself. Do I think it's truly a sign of Java's strength/weakness as a platform? No, but as you say, it's horrible marketing.

            • by roman_mir (125474)

              You mean the JRE that is part of the plugin installation? It cannot be the JRE itself, it must the the installer of the plugin. I don't install JVM this way or Java plugin (at all), I just download the necessary installation package for the JVM/SDK as needed. If I want Oracle SDK I get it from here [oracle.com] and when it installs, it does not install any 'ask bar' or anything like that, so I am not even clear as to how people get these things. I download the SDK or JVM for a Linux distro, which is either a binary

              • No, I mean the JRE itself. (Remember, I'm talking about Windows here.) Granted, this is not the SDK I'm talking about, just the standard JRE installed in Windows, not the browser, but I just updated it on this machine, and I had to uncheck [tenthcave.com] the option to install the Ask toolbar. (link is not my blog, btw.)

                I'm fully aware of the distinctions between Java, the browser plugins, Javascript, etc., and I realize that the toolbar is not part of the install on 'real' OSs. But the Windows installer is what most us

                • by roman_mir (125474)

                  Well, I went looking at this and it's not the JRE that does this, it's the Oracle installer for Java. JRE is not the installer, it's the run time environment.

                  Here is one of these stories about this issue [java.net].

                  But yes, on Linux or Unix I don't get any of this nonsense, it's just a tarball that I untar into a directory, set the path and run the JVM for example. The binary installers for Linux or Unix are simple shell scripts with the same tarball basically as part of the script, it doesn't do any of this stuff.

                  As

                  • All due respect...but even as a developer, to say it's not the JRE but the JRE's installer that bundles the toolbar, is pretty pedantic. I really just wanted to point out this comes with standard install of Java itself, not the browser plugin.
        • by TimHunter (174406)
          I can guarantee you that developers who use Java for serious development work never worry about the Ask toolbar.
          • Agreed, but the OP's point was about Java suffering from bad marketing, and IMHO, the opt-out toolbar is a prime example.
  • by istartedi (132515) on Wednesday March 13, 2013 @01:40PM (#43162037) Journal

    How many people on the planet actually manage "Big Data"? Isn't that the kind of thing that happens as a happy accident when your humble web site becomes the next FaceBubbleSpace? You can't plan for that.

    Sure, there are other places where it happens--large corporations, governments, maybe some academic studies.

    Really though, I have a hard time imagining that there are really a lot of people who deal with BD. Does anybody have numbers on it? What's the definition, anyway? Is Slashdot's archive BD?

    • by jandrese (485)
      Anybody who is talking about data mining is already thinking about Big Data problems. These are everywhere, from correlating shopping habits based on receipts and customer loyalty cards to looking for terrorists by their travel patterns.
    • There's an entire field dedicated to Data Warehousing who's entire focus is Big Data. Large companies with auditing requirements have to keep mountains of historical data. Business Intelligence is largely based on analyzing huge segments of data.

      As storage gets cheaper and options for going through large amounts of data become more widely available, companies invariably store more data. The biggest difference is that while you previously would have simply chosen not to track certain types of data in your

    • by Sarten-X (1102295) on Wednesday March 13, 2013 @01:55PM (#43162223) Homepage

      Big Data is however big you need it to be. It's not a certain size, or speed, or software, but rather a philosophy.

      Simply put, Big Data methodologies are to gather all the data that can be gathered, and store it on a nice cheap database, without concern for storage efficiency. When a question arises, analyse the relevant data for an answer. This is in contrast to more traditional methods, where data is gathered only to solve expected questions.

      Slashdot's archives were not generated from a Big Data approach. They store only comments and a few sparse details (to my knowledge). However, they can still be used in a Big Data system to some effect, if they happen to store the information that's needed (such as IP address, timestamps, and keywords, if the question is to track political opinions by geography over time).

      It's not what you store or how you store it, but how you decide what to store. What makes Big Data approaches useful is that they store everything from the beginning, so as business needs change, the data from the past is likely just as useful as new data. What makes Big Data difficult is that the databases must be properly capable of storing all the gathered data as fast as it arrives, and must do so cheaply.

    • by admdrew (782761)

      There's actually more of a need for it than you'd think at first glance. Any business that handles upwards of a few million records of some sort of data, and then needs to *do* something with that data, could potentially benefit from this.

      I've worked at small (10,000) businesses (all tech-related), and only the smallest places didn't have the amount of data to warrant taking a look at something like this.

      Also, what about those interested in *someday* working for the Googles and the Facebooks of the world? A

      • by admdrew (782761)
        Dang, didn't look closely at the preview, some formatting killed off part of my message. The 2nd paragraph should be:

        I've worked at small (*less than* 100 employees), medium (*less than* 1000), and large (*greater than* 10,000) businesses (all tech-related), and only the smallest places didn't have the amount of data to warrant taking a look at something like this.

    • by fatphil (181876)
      > Is Slashdot's archive BD?

      43 million comments. Let's say a 1KB ballpark average size. Auxiliary data probably negligible compared to that, so let's double it and round up.

      100GB.

      That is a _puny_ database by "Big Data" standards. Every table apart from the comments themselves could be cached in RAM on a modern server, and the majority of comments would never need to be fetched off disk - a single SSD at that - so almost everything important could be cached.

      Of course, you'd never want an architecture like
  • Remember: when you need a book reviewed on Slashdot, make sure you publish it on Packt.
  • by Anonymous Coward

    It's HDFS ... apparently the book didn't teach the reviewer much. "Hadoop Distributed File System."

    • Yup, that was my bad. It IS Hadoop Distributed File System (HDFS) and NOT Hadoop File Distribution System (HFDS). I had it right in front of me and still typed it wrong from some of my notes. I'll see if I can get it fixed. Thanks.
    • It helps to have friends in high places. The HDFS correction has been made. So all who LMAO'ed because of my typo are now free to LYABO. Thanks for pointing out the mistake.
  • by Anonymous Coward

    If it's just a bleeping tool for data processing, what should the user need to be a Java developer to use it?

    • by Anonymous Coward

      Excellent point...but that is something to ask the Hadoop developers, not the author. You can use Hadoop without writing Java. You cannot use Hadoop without knowing in depth about the complete mess that is Java build, linking, etc.

      • I think the real problem with the java build is people like to use older technology. It can be as easy as:

        1) $> gradle run

        2) $> ... there is no step 2..

        3) $> ... take that configure;make;./a.out
  • Hadoop (Score:1, Troll)

    by roman_mir (125474)

    Hadoop. For when you feel like something fun to do on a friday night.

Power corrupts. And atomic power corrupts atomically.

Working...