資源描述：
《大規(guī)模數(shù)據(jù)處理與云計(jì)算-hadoop runtime課件.ppt》由會(huì)員上傳分享，免費(fèi)在線閱讀，更多相關(guān)內(nèi)容在教育資源-天天文庫(kù)。
1、大規(guī)模數(shù)據(jù)處理/云計(jì)算Lecture5–HadoopRuntime課程評(píng)分Project取消4次作業(yè)wordcount(不記分)co-occurrenceindexpagerank1Weekgracetime,oneday10%foreachdaydelay(60%atmost)'wordcount' Howdoesitwork?HadoopClusterdatanodedaemonLinuxfilesystem…tasktrackerslavenodedatanodedaemonLinuxfilesystem…tasktrackerslaveno
2、dedatanodedaemonLinuxfilesystem…tasktrackerslavenodenamenodenamenodedaemonjobsubmissionnodejobtracker4job提交過(guò)程AsksthejobtrackerforanewjobID(bycallinggetNewJobId()onJobTracker)(step2).Checkstheoutputspecificationofthejob.Forexample,iftheoutputdirectoryhasnotbeenspecifiedoritalre
3、adyexists,thejobisnotsubmittedandanerroristhrowntotheMapReduceprogram.Computestheinputsplitsforthejob.Ifthesplitscannotbecomputed(becausetheinputpathsdon’texist,forexample),thejobisnotsubmittedandanerroristhrowntotheMapReduceprogram.Copiestheresourcesneededtorunthejob,includin
4、gthejobJARfile,theconfigurationfile,andthecomputedinputsplits,tothejobtracker’sfilesysteminadirectorynamedafterthejobID.ThejobJARiscopiedwithahighreplicationfactor(controlledbythemapred.submit.replicationproperty,whichdefaultsto10)sothattherearelotsofcopiesacrosstheclusterfort
5、hetasktrackerstoaccesswhentheyruntasksforthejob(step3).TellsthejobtrackerthatthejobisreadyforexecutionbycallingsubmitJob()onJobTracker(step4).InputFormatClassHierarchycombinecombinecombinecombineba12c9ac52bc78partitionpartitionpartitionpartitionmapmapmapmapk1k2k3k4k5k6v1v2v3v4
6、v5v6ba12cc36ac52bc78ShuffleandSort:aggregatevaluesbykeysreducereducereducea15b27c298r1s1r2s2r3s3c23689SerializationSerializationistheprocessofturningstructuredobjectsintoabytestreamfortrans-missionoveranetworkorforwritingtopersistentstorage.Deserializationisthereverseprocessof
7、turningabytestreambackintoaseriesofstructuredobjects.InHadoop,interprocesscommunicationbetweennodesinthesystemisimplementedusingremoteprocedurecalls(RPCs).TheWritableInterfacepublicinterfaceWritable{voidwrite(DataOutputout)throwsIOException;voidreadFields(DataInputin)throwsIOE
8、xception;}publicinterfaceWritableComparableextendsWritable
