Spring Boot 配置Hadoop

Spring Boot 配置Hadoop

核心

• FsShell自动注入

项目工程

• Maven工程

Maven的setting.xml文件配置

 

。。。

    nexus
    *,!cloudera
    http://repo1.maven.org/maven2/

。。。

pom配置

 

    4.0.0

    com.peng
    hdfstest
    1.0-SNAPSHOT
    
        
            
                org.apache.maven.plugins
                maven-compiler-plugin
                
                    1.8
                    1.8
                
            
        
    


    
    
        
            repo
            http://repo1.maven.org/maven2/
        
        
            cloudera
            https://repository.cloudera.com/content/repositories/releases/
        
    


    
    
        
            org.apache.hadoop
            hadoop-client
            2.6.0-cdh5.7.0
        

        
            junit
            junit
            4.10
            test
        

        
            org.mortbay.jetty
            jetty
            6.1.26
        

        
            org.slf4j
            slf4j-log4j12
            1.7.25
            test
        

        
        
            org.springframework.data
            spring-data-hadoop-boot
            2.5.0.RELEASE
        
        

    

配置文件【resources下】

• application.properties【配置文件，映射键值对】

   

  spring.hadoop.fsUri=hdfs://centos00:8020
  

测试程序

• SpringBootHadoopTest.java

   

  package com.peng;
  
  import org.apache.hadoop.fs.FileStatus;
  import org.springframework.beans.factory.annotation.Autowired;
  import org.springframework.boot.CommandLineRunner;
  import org.springframework.boot.SpringApplication;
  import org.springframework.boot.autoconfigure.SpringBootApplication;
  import org.springframework.data.hadoop.fs.FsShell;
  
  @SpringBootApplication
  public class SpringBootHadoopTest implements CommandLineRunner {
  
      @Autowired
      FsShell fsShell;
  
      @Override
      public void run(String... strings) throws Exception {
          System.out.println("=========run start============");
          for (FileStatus fileStatus : fsShell.lsr("/data")) {
              System.out.println(">" + fileStatus.getPath());
          }
          System.out.println("===========run end===========");
      }
  
      public static void main(String[] args) {
          System.out.println("===========main start===========");
          SpringApplication.run(SpringBootHadoopTest.class, args);
          System.out.println("===========main end===========");
      }
  }
  

执行结果

 

===========main start===========
log4j:WARN No appenders could be found for logger (org.springframework.core.env.StandardEnvironment).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.4.7.RELEASE)

=========run start============
>hdfs://centos00:8020/data
>hdfs://centos00:8020/data/10000_access.log
>hdfs://centos00:8020/data/8_access.log
>hdfs://centos00:8020/data/hello.txt
===========run end===========
===========main end===========

Process finished with exit code 0