Hadoop in Action

Front Cover......Page 1
brief contents......Page 6
contents......Page 8
preface......Page 14
acknowledgments......Page 16
about this book......Page 18
Code conventions and downloads......Page 19
Author Online......Page 20
About the author......Page 21
About the cover illustration......Page 22
Part I Hadoop–A Distributed Programming Framework......Page 24
1 Introducing Hadoop......Page 26
1.2 What is Hadoop?......Page 27
1.3 Understanding distributed systems and Hadoop......Page 29
1.4 Comparing SQL databases and Hadoop......Page 30
1.5 Understanding MapReduce......Page 31
1.5.1 Scaling a simple program manually......Page 32
1.5.2 Scaling the same program in MapReduce......Page 35
1.6 Counting words with Hadoop—running your first program......Page 37
1.7 History of Hadoop......Page 42
1.9 Resources......Page 43
2.1 The building blocks of Hadoop......Page 44
2.1.2 DataNode......Page 45
2.1.3 Secondary NameNode......Page 46
2.1.5 TaskTracker......Page 47
2.2 Setting up SSH for a Hadoop cluster......Page 48
2.2.3 Generate SSH key pair......Page 49
2.3 Running Hadoop......Page 50
2.3.1 Local (standalone) mode......Page 51
2.3.2 Pseudo-distributed mode......Page 52
2.3.3 Fully distributed mode......Page 54
2.4 Web-based cluster UI......Page 57
2.5 Summary......Page 59
3 Components of Hadoop......Page 60
3.1.1 Basic file commands......Page 61
3.1.2 Reading and writing to HDFS programmatically......Page 65
3.2 Anatomy of a MapReduce program......Page 67
3.2.1 Hadoop data types......Page 69
3.2.2 Mapper......Page 70
3.2.3 Reducer......Page 71
3.2.4 Partitioner—redirecting output from Mapper......Page 72
3.2.5 Combiner—local reduce......Page 73
3.3 Reading and writing......Page 74
3.3.1 InputFormat......Page 75
3.3.2 OutputFormat......Page 80
3.4 Summary......Page 81
Part II Hadoop in Action......Page 84
4 Writing basic MapReduce programs......Page 86
4.1.1 The patent citation data......Page 87
4.1.2 The patent description data......Page 88
4.2 Constructing the basic template of a MapReduce program......Page 90
4.3 Counting things......Page 95
4.4 Adapting for Hadoop’s API changes......Page 100
4.5 Streaming in Hadoop......Page 103
4.5.1 Streaming with Unix commands......Page 104
4.5.2 Streaming with scripts......Page 105
4.5.3 Streaming with key/value pairs......Page 109
4.5.4 Streaming with the Aggregate package......Page 113
4.6 Improving performance with combiners......Page 118
4.7 Exercising what you’ve learned......Page 121
4.8 Summary......Page 123
4.9 Further resources......Page 124
5 Advanced MapReduce......Page 125
5.1.2 Chaining MapReduce jobs with complex dependency......Page 126
5.1.3 Chaining preprocessing and postprocessing steps......Page 127
5.2 Joining data from different sources......Page 130
5.2.1 Reduce-side joining......Page 131
5.2.2 Replicated joins using DistributedCache......Page 140
5.2.3 Semijoin: reduce-side join with map-side filtering......Page 144
5.3.1 What does a Bloom filter do?......Page 145
5.3.2 Implementing a Bloom filter......Page 147
5.4 Exercising what you’ve learned......Page 154
5.5 Summary......Page 155
5.6 Further resources......Page 156
6 Programming Practices......Page 157
6.1.1 Local mode......Page 158
6.1.2 Pseudo-distributed mode......Page 163
6.2 Monitoring and debugging on a production cluster......Page 168
6.2.1 Counters......Page 169
6.2.2 Skipping bad records......Page 171
6.2.3 Rerunning failed tasks with IsolationRunner......Page 174
6.3.2 Reducing the amount of input data......Page 175
6.3.3 Using compression......Page 176
6.3.4 Reusing the JVM......Page 178
6.3.5 Running with speculative execution......Page 179
6.3.6 Refactoring code and rewriting algorithms......Page 180
6.4 Summary......Page 181
7.1 Passing job-specific parameters to your tasks......Page 183
7.2 Probing for task-specific information......Page 186
7.3 Partitioning into multiple output files......Page 187
7.4 Inputting from and outputting to a database......Page 192
7.5 Keeping all output in sorted order......Page 194
7.6 Summary......Page 195
8 Managing Hadoop......Page 196
8.1 Setting up parameter values for practical use......Page 197
8.2 Checking system’s health......Page 199
8.3 Setting permissions......Page 201
8.5 Enabling trash......Page 202
8.7 Adding DataNodes......Page 203
8.8 Managing NameNode and Secondary NameNode......Page 204
8.9 Recovering from a failed NameNode......Page 206
8.10 Designing network layout and rack awareness......Page 207
8.11.1 Multiple JobTrackers......Page 209
8.11.2 Fair Scheduler......Page 210
8.12 Summary......Page 212
Part III Hadoop Gone Wild......Page 214
9 Running Hadoop in the cloud......Page 216
9.2 Setting up AWS......Page 217
9.2.1 Getting your AWS authentication credentials......Page 218
9.2.2 Getting command line tools......Page 221
9.2.3 Preparing an SSH key pair......Page 223
9.3.1 Configuring security parameters......Page 224
9.3.2 Configuring cluster type......Page 225
9.4 Running MapReduce programs on EC2......Page 226
9.4.2 Accessing your data from the Hadoop cluster......Page 227
9.6.1 Amazon Elastic MapReduce......Page 232
9.6.2 AWS Import/Export......Page 233
9.7 Summary......Page 234
10 Programming with Pig......Page 235
10.1.2 Data types......Page 236
10.2 Installing Pig......Page 237
10.3 Running Pig......Page 238
10.3.1 Managing the Grunt shell......Page 239
10.4 Learning Pig Latin through Grunt......Page 240
10.5 Speaking Pig Latin......Page 244
10.5.1 Data types and schemas......Page 245
10.5.2 Expressions and functions......Page 246
10.5.3 Relational operators......Page 248
10.6 Working with user-defined functions......Page 256
10.6.2 Writing UDFs......Page 257
10.7.1 Comments......Page 260
10.7.2 Parameter substitution......Page 261
10.7.3 Multiquery execution......Page 262
10.8 Seeing Pig in action—example of computing similar patents......Page 263
10.9 Summary......Page 268
11 Hive and the Hadoop herd......Page 269
11.1 Hive......Page 270
11.1.1 Installing and configuring Hive......Page 271
11.1.2 Example queries......Page 273
11.1.3 HiveQL in details......Page 277
11.1.4 Hive Sum-up......Page 283
11.2.2 ZooKeeper......Page 285
11.2.5 Katta......Page 286
11.2.9 search-hadoop.com......Page 287
11.3 Summary......Page 288
12 Case studies......Page 289
12.2 Mining data at China Mobile......Page 290
12.3 Recommending the best websites at StumbleUpon......Page 295
12.3.1 Distributed beginnings at StumbleUpon......Page 296
12.3.2 HBase and StumbleUpon......Page 297
12.3.3 More Hadoop at StumbleUpon......Page 304
12.4 Building analytics for enterprise search—IBM’s Project ES2......Page 305
12.4.1 ES2 architecture......Page 308
12.4.2 ES2 crawler......Page 310
12.4.3 ES2 analytics......Page 311
12.4.4 Conclusions......Page 319
12.4.5 References......Page 320
appendix HDFS file commands......Page 321
C......Page 325
D......Page 326
G......Page 327
H......Page 328
K......Page 329
N......Page 330
P......Page 331
S......Page 332
U......Page 333
Z......Page 334
Back Cover......Page 336