ETLÊÇʲô£¿ÎªÊ²Ã´ÒªÊ¹ÓÃETL£¿KETTLEÊÇʲô£¿ÎªÊ²Ã´ÒªÑ§KETTLE£¿
ETLÊÇÊý¾ÝµÄ³éÈ¡Çåϴת»»¼ÓÔØµÄ¹ý³Ì£¬ÊÇÊý¾Ý½øÈëÊý¾Ý²Ö¿â½øÐдóÊý¾Ý·ÖÎöµÄÔØÈë¹ý³Ì£¬Ä¿Ç°Á÷ÐеÄÊý¾Ý½øÈë²Ö¿âµÄ¹ý³ÌÓÐÁ½ÖÖÐÎʽ£¬Ò»ÖÖÊǽøÈëÊý¾Ý¿âºóÔÙ½øÐÐÇåÏ´ºÍת»»£¬ÁíÍâÒ»Ìõ·ÏßÊÇÊ×ÏȽøÐÐÇåϴת»»ÔÙ½øÈëÊý¾Ý¿â£¬ÎÒÃǵÄETLÊôÓÚºóÕß¡£
´óÊý¾ÝµÄÀûÆ÷´ó¼Ò¿ÉÄÜÆÕ±é˵ÊÇhadoop£¬µ«ÊÇ´ó¼ÒÒªÖªµÀÈç¹ûÎÒÃDz»×öÔ¤ÏȵÄÇåÏ´ºÍת»»´¦Àí£¬ÎÒÃǽøÈëhadoopºó½öͨ¹ýmapreduce½øÐÐÊý¾ÝÇåϴת»»ÔÙ½øÐзÖÎö£¬À¬»øÊý¾Ý»áµ¼ÖÂÎÒÃǵĴÅÅÌÕ¼ÓÃÁ¿»áÏ൱´ó£¬ÕâÑùÎÞÐÎÖÐÌáÉýÁËÎÒÃǵÄÓ²¼þ³É±¾£¨Ó²ÅÌ´ó£¬ÄÚ´æÐ¡´¦ÀíËÙ¶È»áºÜÂý£¬ÄÚ´æ´ócpuÐÔÄܵÍËÙ¶ÈÒ²»áÊÜÓ°Ï죩£¬Òò´ËËäÈ»hadoopÀíÂÛÉϽâ¾öÁËÀûúÆ÷Æ´ÆðÀ´½â¾ö´óÎÊÌâµÄÎÊÌ⣬µ«ÊÇÊÂʵÉÏÈç¹ûÎÒÃÇÓиüºÃµÄ½ÚµãËٶȱØÈ»ÊÇ»áÆÕ±éÌáÉýµÄ£¬Òò´ËETLÔÚ´óÊý¾Ý»·¾³ÏÂÈÔÈ»ÊDZز»¿ÉÉÙµÄÊý¾Ý½»»»¹¤¾ß¡£
Êг¡ÉÏÁ÷ÐеÄETLºÜ¶à£¬±ÈÈçinformaticaµÈ£¬µ«ÊÇ¿ªÔ´µÄ±È½ÏÍêÉÆµÄÈ´²»ÊǺܶ࣬¶øÆäÖбȽÏÓÐÃûµÄҪ˵ÊÇpentaho¿ªÔ´µÄkettleÁË£¬¸Ã¹¤¾ß±»¹ã·ºÓ㬲¢ÇÒ¿ªÔ´µÄ²úÆ·ÎÒÃÇ´ÓÖв»½ö¿ÉÒÔѧµ½ETLµÄ¼òµ¥Ó¦Ó㬲¢ÇÒ¿ÉÒÔѧϰµ½ETLµÄÔÀíÒÔ¼°Í¨¹ýÔ´Âëѧµ½¸ü¶àµÄ¶«Î÷¡£
ÁÁµãÒ»£ºKETTLEÓ¦Óù㷺£¬½ö½öѧ»áʹÓþͿÉÒÔÕÒµ½Ò»·Ý²»´íµÄ¹¤×÷¡£
ÁÁµã¶þ£º±¾¿Î³Ì²»½ö½²½â¼òµ¥ÊµÓã¬Í¬Ê±½²½â¶þ´Î¿ª·¢²¢ÇÒÅäÓпª·¢Ä£°å£¬ÌáÉý¹¤×÷ÖÊÁ¿¡£
ÁÁµãÈý£ºÉøÍ¸ÁË´óÊý¾ÝµÄһЩ´¦Àí·½·¨£¬ÓëĿǰÁ÷ÐеÄhadoopÅäºÏʹÓá£
ÁÁµãËÄ£º·ÖÎöKETTLEÔ´Â룬¼´Ê¹¶ÔETLÐËȤ²»´ó£¬ÖÁÉÙ¿ÉÒÔÁ˽â¹úÍ⿪ԴÏîÄ¿µÄһЩԴÂ룬²¢ÇÒKETTLE±¾ÉíҲʹÓÃÁ˺ܶ࿪ԴÏîÄ¿£¬Òò´Ë¿ÉÒԴӸù¤¾ßÉÏѧµ½¸ü¶à¶«Î÷¡£
ͨ¹ý¿Î³Ì¿ÉÒÔѧµ½Ê²Ã´£º
1.ETL¹ý³ÌÔÀí
2.Êý¾ÝÁ÷ÒýÇæµÄÔÀí
3.ÔªÊý¾ÝºÍÊý¾Ý½øÐж¯Ì¬Êý¾Ý½»»»µÄÉè¼Æ
4.²¢·¢ÔËËãµÄÔÀí
¿Îʱ°²ÅÅ£º£¨15¿Îʱ£©
1.ETL¼ò½é—¿ªÔ´KETTLE£¨1¿Îʱ£©
>½éÉÜKETTLEÔÚ´óÊý¾ÝÓ¦ÓõÄλÖúÍ×÷Óá£
>Ö÷Òª½²½âETLÊÇʲô£¬KETTLE½øÐмòµ¥½éÉÜ£¬²¢ÇÒʹÓÃÀý×Ó½øÐÐKETTLEµÄʹÓýéÉÜ¡£
>½éÉÜKETTLEÁ÷³ÌµÄ²¿Êð¡£
2.KETTLEʹÓã¨1¿Îʱ£©
>Ïêϸ½éÉÜKETTLEµÄspoonʹÓÃ
>KETTLEµÄtransºÍjobÈëÃÅ
>KETTLEµÄÈÕÖ¾ºÍµ÷ÊÔ¹¤¾ßʹÓÃ
3. KETTLEÖ®StepÁ÷³ÌÉè¼Æ£¨3¿Îʱ£©
>±àдÀý×Ó½éÉÜKETTLE³£ÓõÄת»»¡¢ÇåÏ´×é¼þ
>Ö÷ÒªÍê³ÉÒÔϲå¼þ£º
ÊäÈë²å¼þ£º
Îı¾ÎļþÊäÈë¡¢Éú³É¼Ç¼¡¢±íÊäÈë¡¢Fixed file input¡¢Get data from XML
Êä³ö²å¼þ£º
XMLÊä³ö¡¢É¾³ý¡¢²åÈë/¸üС¢Îı¾ÎļþÊä³ö¡¢¸üС¢±íÊä³ö
ת»»²å¼þ£º
Add a checksum¡¢Replace in string¡¢Set field value¡¢Unique rows£¨HashSet£©¡¢Ôö¼Ó³£Á¿¡¢Ôö¼ÓÐòÁС¢×Ö¶ÎÑ¡Ôñ¡¢²ð·Ö×Ö¶Î
Flow²å¼þ£º
Abort¡¢Switch/case¡¢¿Õ²Ù×÷¡¢¹ýÂ˼Ǽ
½Å±¾²å¼þ£º
Modified Java Script Value¡¢Ö´ÐÐSQL½Å±¾
²éѯ²å¼þ£º
File exists¡¢Table exists¡¢µ÷ÓÃDB´æ´¢¹ý³Ì
4. KETTLEÖ®JobÁ÷³ÌÉè¼Æ£¨2¿Îʱ£©
>±àдÀý×Ó½éÉÜKETTLE³£ÓõÄ×÷Òµ×é¼þ
>Ö÷ÒªÍê³ÉÒÔϲå¼þ£º
ͨÓòå¼þ£º
START¡¢DUMMY¡¢Transformation¡¢Success
Îļþ¹ÜÀí²å¼þ£º
Copy Files¡¢Compare folders¡¢Create a folder¡¢Create file¡¢Delete files¡¢Delete folders¡¢File Compare¡¢Move Files¡¢Wait for file¡¢Zip file¡¢Unzip file
Ìõ¼þ²å¼þ£º
Check Db connections¡¢Check files locked¡¢Check if a folder is empty¡¢Check if files exist¡¢File Exists¡¢Table exists¡¢Wait for
½Å±¾²å¼þ£º
Shell¡¢SQL
Utility²å¼þ£º
Ping a host¡¢Truncate tables
Îļþ´«Êä²å¼þ£º
Upload files to FTPS¡¢Get a file with FTPS¡¢FTP Delete
>KettleÓëHadoopµÄÁªºÏʹÓÃ
5. KETTLEÖ®Á÷³ÌÐÔÄܵ÷ÓÅÓë¼à¿Ø£¨1¿Îʱ£©
>½éÉÜKETTLEµÄÁ÷³Ì¼à¿Ø¹¦ÄÜ
>½éÉÜKETTLEµÄÐÔÄÜÓÅ»¯·½·¨
6. KETTLE֮ǶÈ뿪·¢£¨1¿Îʱ£©
>±àд³ÌÐò½éÉÜKETTLEµÄÁ÷³ÌÈçºÎǶÈëµ½ÎÒÃǵÄjavaÓ¦ÓÃÖÐ
Ö÷Òª°üÀ¨javaǶÈëtransÒÔ¼°jobÁ÷³Ì
7. KETTLEÖ®×Ô¶¨ÒåStep¡¢Job²å¼þÖÆ×÷£¨3¿Îʱ£©
>±àдStepºÍJobÄ£°å£¬²¢¸ø´ó¼Ò×÷Ϊ¶þ´Î¿ª·¢µÄ»ù´¡¹¤³ÌʹÓã¬Ìá¸ß´ó¼ÒµÄ¿ª·¢Ð§ÂÊ¡£
>±àд³ÌÐò˵Ã÷StepºÍJob²å¼þµÄ¿ª·¢·½·¨¡£
8. KETTLEÖ®Êý¾Ýͬ²½·½°¸£¨1¿Îʱ£©
>½éÉÜ5ÖÖÊý¾Ýͬ²½·½°¸£¬²¢ÇÒÕâ5ÖÖ·½°¸¶¼ÊÇÖ§³ÖÒì¹¹Êý¾Ýͬ²½µÄ¡£
°üÀ¨È«Á¿¿ìËÙͬ²½·½°¸ºÍÔöÁ¿Í¬²½·½°¸
9. KETTLEÖ®·ÖÇø¡¢¼¯ÈºÒÔ¼°ÔÀí£¨1¿Îʱ£©
>½éÉÜKETTLEµÄ·ÖÇøÔÀí£¬²¢ÇÒ½²½âÅäÖÃʹÓá£
>½éÉÜKETTLEµÄ¼¯ÈºÔÀí£¬²¢ÇÒ½²½âÅäÖÃʹÓã¬ÒÔ¼°¼à¿Ø·½·¨¡£
10. KETTLEÖ®Ô´Âë·ÖÎöÓë¶þ´Î¿ª·¢£¨1¿Îʱ£©
>½éÉÜKETTLEµÄSRCµ¼ÈëECLIPSE·½·¨£¬ÒÔ¼°´ò°üºÍÔËÐз½·¨¡£
>·ÖÎöKETTLEµÄ°ü½á¹¹ÒÔ¼°ÔËÐÐÁ÷³Ì£¬½²½âKETTLEµÄÔËÐÐÔÀí¡£