linux搭建分布式文件系统FastDFS并整合spring boot

linux搭建分布式文件系统FastDFS并整合spring boot

目录

• FastDFS是什么
• FastDFS安装
• spring boot集成FastDFS上传下载文件
• FastDFS配置nginx
• 问题&注意事项

FastDFS是什么：

是由C语言实现的开源轻量级分布式文件系统，实现了对文件管理，同步，访问，下载等功能，解决了大容量存储和负载均衡。其包含两个角色：Tracker（跟踪器）和Storage（存储节点）

• Tracker：资源调度，负载均衡，管理所有的group和storage，每一个storage启动就会链接Tracker

• Storage：存储节点，提供存储和备份服务，以group为单位，每组可以有多台Storage，数据互相备份。

  官网架构图：
  

  

FastDFS安装

多台服务器都需要安装FastDFS，为了方便测试单机安装也可以，配置相差不多。本次环境centos，如果安装过程中出现问题翻到最后查看注意事项可能会找到解决方法。

• 下载安装libfastcommon：

  cd /opt/fastdfs
  
  wget https://github.com/happyfish100/libfastcommon/archive/V1.0.7.tar.gz
  
  tar -zxvf V1.0.7.tar.gz
  
  cd libfastcommon-1.0.7
  
  ./make.sh
  
  //安装：
  ./make.sh install
  
  //设置软连接:
  ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
  ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
  ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
  ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
  

• 下载安装FastDFS，每台文件服务器都需要安装：

  wget https://github.com/happyfish100/fastdfs/archive/V5.05.tar.gz
  
  tar -zxvf V5.05.tar.gz
  
  cd fastdfs-5.05
  
  ./make.sh
  
  ./make.sh install
  

  安装成功后会在/etc/fdfs 目录下生成三个示例配置文件

• 配置Tracker服务器

  cp tracker.conf.sample tracker.conf
  
  //创建文件夹留后用
  mkdir -p /data/fastdfs/tracker
  
  vim tracker.conf
  

  编辑tracker.conf配置文件，修改：

  # the base path to store data and log files
  base_path=/data/fastdfs/tracker     
  
  http.server_port=8055
  

  保存退出，启动tracker服务：

  fdfs_trackerd /etc/fdfs/tracker.conf start
  

  启动成功，查看监听服务：

  netstat -unltp|grep fdfs
  

  可以看到tracker默认端口为22122，并且在/data/fastdfs/tracker文件夹下新增数据和日志存储文件。

• 配置Storage服务器：

  cd /etc/fdfs
  
  cp storage.conf.sample storage.conf
  
  mkdir -p /data/fastdfs/storage
  
  vim storage.conf
  

  修改storage.conf配置文件：

  # the base path to store data and log files
  base_path=/data/fastdfs/storage
  
  # store_path#, based 0, if store_path0 not exists, it's value is base_path
  # the paths must be exist
  store_path0=/data/fastdfs/storage
  
  # tracker_server can ocur more than once, and tracker_server format is
  #  "host:port", host can be hostname or ip address
  #配置tracker跟踪器ip端口
  tracker_server=192.168.5.26:22122
  

  保存退出，并启动Storage：

  fdfs_storaged /etc/fdfs/storage.conf start
  

  如果启动过慢或卡死，查看日志文件：

  /data/fastdfs/storage/logs
  

  查看端口监听，发现Storage默认端口号为23000

  netstat -unltp|grep fdfs
  

  所有节点都安装成功后可以在任意一个节点查看集群的状态：

  /usr/bin/fdfs_monitor /etc/fdfs/storage.conf
  

spring boot集成FastDFS上传下载文件：

• pom.xml引入，参考自纯洁的微笑：

  

net.oschina.zcx7878
fastdfs-client-java
1.27.0.0


• 添加FastDFS配置文件到classpath：fdfs_client.conf

  connect_timeout = 60
  network_timeout = 60
  charset = UTF-8
  http.tracker_http_port = 8055
  http.anti_steal_token = no
  http.secret_key = 123456
  
  tracker_server = 192.168.5.26:22122
  

  其中http.tracker_http_port = 8055 与tracker.conf中的http.server_port一致，并且不能与其他端口程序冲突。tracker_server修改相应ip，如果有多台，则可以写多个trackrt_server配置项。

• 封装文件基础信息类：

  @Data
  public class FastDFSFile {
  
      private String name;
  
      private byte[] content;
  
      private String ext;
  
      private String md5;
  
      private String author;
  }
  

• 封装client操作类，用于上传下载删除等操作：

  import java.io.ByteArrayInputStream;
  import java.io.IOException;
  import java.io.InputStream;
  
  import org.csource.common.NameValuePair;
  import org.csource.fastdfs.ClientGlobal;
  import org.csource.fastdfs.FileInfo;
  import org.csource.fastdfs.ServerInfo;
  import org.csource.fastdfs.StorageClient;
  import org.csource.fastdfs.StorageServer;
  import org.csource.fastdfs.TrackerClient;
  import org.csource.fastdfs.TrackerServer;
  import org.slf4j.LoggerFactory;
  import org.springframework.core.io.ClassPathResource;
  
  public class FastDFSClient {
      private static org.slf4j.Logger logger = LoggerFactory.getLogger(FastDFSClient.class);
      private static TrackerClient trackerClient;
      private static TrackerServer trackerServer;
      private static StorageClient storageClient;
      private static StorageServer storageServer;
  
      static {
          try {
              String filePath = new ClassPathResource("fdfs_client.conf").getFile().getAbsolutePath();;
              ClientGlobal.init(filePath);
              trackerClient = new TrackerClient();
              trackerServer = trackerClient.getConnection();
              storageServer = trackerClient.getStoreStorage(trackerServer);
          } catch (Exception e) {
              logger.error("FastDFS Client Init Fail!",e);
          }
      }
  
      public static String[] upload(FastDFSFile file) {
          logger.info("File Name: " + file.getName() + "File Length:" + file.getContent().length);
  
          NameValuePair[] meta_list = new NameValuePair[1];
          meta_list[0] = new NameValuePair("author", file.getAuthor());
  
          long startTime = System.currentTimeMillis();
          String[] uploadResults = null;
          try {
              storageClient = new StorageClient(trackerServer, storageServer);
              uploadResults = storageClient.upload_file(file.getContent(), file.getExt(), meta_list);
          } catch (IOException e) {
              logger.error("IO Exception when uploadind the file:" + file.getName(), e);
          } catch (Exception e) {
              logger.error("Non IO Exception when uploadind the file:" + file.getName(), e);
          }
          logger.info("upload_file time used:" + (System.currentTimeMillis() - startTime) + " ms");
  
          if (uploadResults == null) {
              logger.error("upload file fail, error code:" + storageClient.getErrorCode());
          }
          String groupName = uploadResults[0];
          String remoteFileName = uploadResults[1];
  
          logger.info("upload file successfully!!!" + "group_name:" + groupName + ", remoteFileName:" + " " + remoteFileName);
          return uploadResults;
      }
  
      public static FileInfo getFile(String groupName, String remoteFileName) {
          try {
              storageClient = new StorageClient(trackerServer, storageServer);
              return storageClient.get_file_info(groupName, remoteFileName);
          } catch (IOException e) {
              logger.error("IO Exception: Get File from Fast DFS failed", e);
          } catch (Exception e) {
              logger.error("Non IO Exception: Get File from Fast DFS failed", e);
          }
          return null;
      }
  
      public static InputStream downFile(String groupName, String remoteFileName) {
          try {
              storageClient = new StorageClient(trackerServer, storageServer);
              byte[] fileByte = storageClient.download_file(groupName, remoteFileName);
              InputStream ins = new ByteArrayInputStream(fileByte);
              return ins;
          } catch (IOException e) {
              logger.error("IO Exception: Get File from Fast DFS failed", e);
          } catch (Exception e) {
              logger.error("Non IO Exception: Get File from Fast DFS failed", e);
          }
          return null;
      }
  
      public static void deleteFile(String groupName, String remoteFileName)
              throws Exception {
          storageClient = new StorageClient(trackerServer, storageServer);
          int i = storageClient.delete_file(groupName, remoteFileName);
          logger.info("delete file successfully!!!" + i);
      }
  
      public static StorageServer[] getStoreStorages(String groupName)
              throws IOException {
          return trackerClient.getStoreStorages(trackerServer, groupName);
      }
  
      public static ServerInfo[] getFetchStorages(String groupName,
                                                  String remoteFileName) throws IOException {
          return trackerClient.getFetchStorages(trackerServer, groupName, remoteFileName);
      }
  
      public static String getTrackerUrl() {
          return "http://"+trackerServer.getInetSocketAddress().getHostString()+":"+ClientGlobal.getG_tracker_http_port()+"/";
      }
  }
  

• 编写控制器提供REST 服务器：

  @PostMapping("/upload") 
  public String singleFileUpload(@RequestParam("file") MultipartFile file,
                                 RedirectAttributes redirectAttributes) {
      if (file.isEmpty()) {
          redirectAttributes.addFlashAttribute("message", "Please select a file to upload");
          return "redirect:uploadStatus";
      }
      try {
          // Get the file and save it somewhere
          String path=saveFile(file);
          redirectAttributes.addFlashAttribute("message",
                  "You successfully uploaded '" + file.getOriginalFilename() + "'");
          redirectAttributes.addFlashAttribute("path",
                  "file path url '" + path + "'");
      } catch (Exception e) {
          logger.error("upload file failed",e);
      }
      return "redirect:/uploadStatus";
  }
  
    public String saveFile(MultipartFile multipartFile) throws IOException {
      String[] fileAbsolutePath={};
      String fileName=multipartFile.getOriginalFilename();
      String ext = fileName.substring(fileName.lastIndexOf(".") + 1);
      byte[] file_buff = null;
      InputStream inputStream=multipartFile.getInputStream();
      if(inputStream!=null){
          int len1 = inputStream.available();
          file_buff = new byte[len1];
          inputStream.read(file_buff);
      }
      inputStream.close();
      FastDFSFile file = new FastDFSFile(fileName, file_buff, ext);
      try {
          fileAbsolutePath = FastDFSClient.upload(file);  //upload to fastdfs
      } catch (Exception e) {
          logger.error("upload file Exception!",e);
      }
      if (fileAbsolutePath==null) {
          logger.error("upload file failed,please upload again!");
      }
      String path=FastDFSClient.getTrackerUrl()+fileAbsolutePath[0]+ "/"+fileAbsolutePath[1];
      return path;
  }
  

• 上传测试，如果操作成功，上传接口将返回文件上传到FastDFS集群中的位置URL信息，类似：http://192.168.5.26:8888/group1/M00/00/00/rBF80FppaNaASaqvAAvWFkcZHjA372.jpg，但该链接还并不能直接访问，需要继续配置FastDFS与nginx集成进行提供外部访问，继续。

• 示例项目位置：github

FastDFS配置nginx

• nginx 相关操作：链接

• 安装fastdfs-nginx-module,所有Storage节点都需要安装

  cd /opt/fastdfs
  
  wget http://jaist.dl.sourceforge.NET/project/fastdfs/FastDFS%20Nginx%20Module%20Source%20Code/fastdfs-nginx-module_v1.16.tar.gz
  
  tar -zxvf fastdfs-nginx-module_v1.16.tar.gz
  
  cd /opt/fastdfs/fastdfs-nginx-module/src
  
  vim config
  

  修改配置，否则nginx编译报错

  CORE_INCS="$CORE_INCS /usr/local/include/fastdfs /usr/local/include/fastcommon/"
  
  改为如下：
  
  CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"   
  

  复制fastdfs-nginx-module 的配置文件到/etc/fdfs并修改

  cd /opt/fastdfs-nginx-module/src
  
  cp mod_fastdfs.conf /etc/fdfs
  
  cd /etc/fdfs
  
  vim mod_fastdfs.conf
  

  修改如下：

  tracker_server=192.168.7.73:22122 # tracker服务IP和端口
  url_have_group_name=true # 访问链接前缀加上组名
  store_path0=/data/fastdfs/storage # 文件存储路径
  

• 重新编译安装nginx，相关操作链接在上面。

  cd /usr/local/nginx
  
  ./configure \
  --prefix=/usr/local/nginx \
  --pid-path=/var/local/nginx/nginx.pid \
  --lock-path=/var/lock/nginx/nginx.lock \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --with-http_gzip_static_module \
  --http-client-body-temp-path=/var/temp/nginx/client \
  --http-proxy-temp-path=/var/temp/nginx/proxy \
  --http-fastcgi-temp-path=/var/temp/nginx/fastcgi \
  --http-uwsgi-temp-path=/var/temp/nginx/uwsgi \
  --http-scgi-temp-path=/var/temp/nginx/scgi \
  --add-module=/opt/fastdfs/fastdfs-nginx-module/src
  
  make && make install
  
  //重启,如何添加到service查看上面链接
  service nginx restart
  

• 配置文件：

  复制FastDFS部分配置文件到/etc/fdfs

  cd /opt/fastdfs/fastdfs-5.05/conf
  
  cp http.conf mime.types /etc/fdfs
  

  修改nginx配置文件：

  vim /usr/local/nginx/conf/nginx.conf
  

  修改端口与/etc/fdfs/storage.conf 中的http.server_port=8888 相对应

    server {
      listen 8888;  
      server_name localhost;
  
     location / {
        root html;
        index index.html index.htm;
     }
     location ~/group([0-9])/M00 {
        alias /data/fastdfs/storage/;
        ngx_fastdfs_module;  
      }
      location /group1/M00 {
          root /data/fastdfs/storage/;
          ngx_fastdfs_module;
      }
    }
  

  重启nginx

  service nginx restart
  

  查看nginx状态：

  netstat -unltp|grep nginx
  

  可以看到nginx监听了8888端口。此时可以访问http://192.168.5.26:8888/group1/M00/00/00/rBF80FppaNaASaqvAAvWFkcZHjA372.jpg查看到文件

• Tracker配置nginx

  cd /usr/local/nginx
  
  ./configure \
  --prefix=/usr/local/nginx 
  
  make && make install
  

  设置group负载均衡，设置nginx端口为8000

  #设置 group1 的服务器
  upstream fdfs_group1 {
       server 192.168.5.26:8888 weight=1 max_fails=2 fail_timeout=30s;
       server 192.168.5.27:8888 weight=1 max_fails=2 fail_timeout=30s;
  }
  
  server {
      listen       8000;
      server_name  localhost;
  
      #charset koi8-r;
  
      #access_log  logs/host.access.log  main;
  
      location / {
          root   html;
          index  index.html index.htm;
      }
  
      #设置 group 的负载均衡参数
      location /group1/M00 {
          proxy_next_upstream http_502 http_504 error timeout invalid_header;
          proxy_pass http://fdfs_group1;
          expires 30d;
      }
  }
  

  此时通过Tracker和直接从Storage直接访问都可以访问到文件。

  http://tracker_server:8000/group1/M00/00/00/rBF80FppaNaASaqvAAvWFkcZHjA372.jpg
  http://storage_server:8888/group1/M00/00/00/rBF80FppaNaASaqvAAvWFkcZHjA372.jpg
  

问题&注意事项

• Tracker启动成功，Storage启动失败：

  检查防火墙配置：

  vim /etc/sysconfig/iptables
  

  有关iptables详细配置查看：iptables快速配置

  使用阿里云或其他云服务器注意平台安全组，以阿里云或腾讯云为例，进入ECS服务器–>查看安全组–>添加安全组规则，暴露出相应的端口：22122，23000，8050，8888等。

• 使用spring boot上传文件失败，发现tracker默认http端口为8080，可能与tomcat冲突，修改tracker端口，并重新配置防火墙和安全组。

• nginx安装失败，可能是缺少依赖，运行：

  yum -y install gcc
  
  yum -y install pcre pcre-devel
  
  yum -y install zlib zlib-devel
  
  yum -y install openssl openssl-devel
  

  安装相应依赖。

• 常用命令：

  • netstat -anp|grep 80
  • fdfs_trackerd /etc/fdfs/tracker.conf start
  • fdfs_storaged /etc/fdfs/storage.conf start
  • service nginx start
  • netstat -unltp|grep fdfs
  • tar zxvf nginx-1.12.1.tar.gz
  • vim /etc/sysconfig/iptables