2025年Faiss 架构（fastdfs架构）

大家好，我是讯享网，很高兴认识大家。

Tracker Server: 跟踪服务器，主要做调度工作，起到均衡的作用；负责管理所有的storage server和group，每个storage在启动后会连接 Tracker，告知自己所属 group 等信息，并保持周期性心跳。多个Tracker之间是对等关系，不存在单点故障。
Storage Server: 存储服务器，主要提供容量和备份服务；以 group 为单位，每个 group 内可以有多台 storage server，组内的storage server上的数据互为备份。
Client:客户端，上传下载数据的服务器
模块之间的主要关系如下：

1， tracker

tracker server配置:

2， Storage

3，安装nginx和fastdfs-nginx-module

在10.250.112.143和10.250.112.144上

下载nginx module

安装nginx

wget

$ tar -zxvf nginx-1.12.2.tar.gz

$ cd nginx-1.12.2/

$ https://www.ctyun.cn/zhishi/configure --prefix=/usr/local/nginx --sbin-path=/usr/local/nginx --modules-path=/usr/local/nginx/modules --conf-path=/usr/local/nginx/conf/nginx.conf --error-log-path=/usr/local/nginx/logs/error.log --http-log-path=/usr/local/nginx/logs/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-http_auth_request_module --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' --add-module=/opt/fastdfs/fastdfs-nginx-module-1.20/src/

4，配置nginx

在10.250.112.143和10.250.112.144上

将原来内容去掉，改为下面的内容

worker_processes 1;
events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;

server {
listen 80;
server_name localhost;

location / {
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}

server {
listen 8888; 该端口为storage.conf中的http.server_port相同
server_name localhost;
location ~/group[0-9]/ {
root /data/fastdfs;
ngx_fastdfs_module;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}

注意：1，启动nginx，用whereis nginx定位nginx的位置，进到sbin目录，使用 https://www.ctyun.cn/zhishi/nginx 启动。

2，将用到的端口到防火墙中打开，命令如下

vi /etc/sysconfig/iptables

添加要开放的端口，如8888，9270，22122等用到的端口

service iptables restart

讯享网

注：

在nginx的构建中会遇到不少的报错，具体如下：

https://www.ctyun.cn/zhishi/configure: error: the Google perftools module requires the Google perftools library. You can either do not enable the module or install the library.

解决方法如下：

/configure: error: the HTTP rewrite module requires the PCRE library.
解决方法如下：

https://www.ctyun.cn/zhishi/configure: error: the HTTP cache module requires md5 functions from OpenSSL library. You can either disable the module by using --without-http-cache option, or install the OpenSSL library into the system, or build the OpenSSL library statically from the source with nginx by using --with-http_ssl_module --with-openssl= options.
解决方法如下：

https://www.ctyun.cn/zhishi/configure: error: the HTTP gzip module requires the zlib library. You can either disable the module by using –without-http_gzip_module option, or install the zlib library into the system, or build the zlib library statically from the source with nginx by using –with-zlib= option.
解决方法如下：

https://www.ctyun.cn/zhishi/configure: error: the HTTP XSLT module requires the libxml2/libxslt libraries. You can either do not enable the module or install the libraries.
解决方法如下：

https://www.ctyun.cn/zhishi/configure: error: the HTTP image filter module requires the GD library. You can either do not enable the module or install the libraries.
解决方法如下：

https://www.ctyun.cn/zhishi/configure: error: perl module ExtUtils::Embed is required
解决方法如下：

https://www.ctyun.cn/zhishi/configure: error: the GeoIP module requires the GeoIP library. You can either do not enable the module or install the library.
解决方法如下：

在make过程中会出现一个报错：/usr/include/fastdfs/fdfs_define.h:15:27: fatal error: common_define.h: No such file or directory
解决方法如下：

5、配置文件访问的负载均衡和高可用

在10.250.112.145和10.250.112.146上安装nginx

wget

$ tar -zxvf nginx-1.12.2.tar.gz

$ cd nginx-1.12.2/

$ https://www.ctyun.cn/zhishi/configure

$ make

$ make install

nginx.conf配置文件如下

worker_processes 1;
events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;

upstream fdfs_group {
server 10.250.112.143:8888 weight=1 max_fails=2 fail_timeout=30s;
server 10.250.112.144:8888 weight=1 max_fails=2 fail_timeout=30s;
}
server {
listen 80;
server_name localhost;
location ~/group[0-9]/{
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_pass
expires 30d;
}
}
}

10.250.112.145 上keepalived配置文件如下：

新建一个check_nginx.sh文件，内容如下

到这里，所有的安装都完成了

在10.250.112.143上上传的图片就可以通过下面的路径访问了

后面一种是做过负载均衡的。

建立maven工程，引入

<!– 引入FastDFS –>
    <dependency>
        <groupId>net.oschina.zcx7878</groupId>
        <artifactId>fastdfs-client-java</artifactId>
        <version>1.27.0.0</version>
    </dependency>

上传文件

@Override
    public String upload(String fileName) {
        String filePath = getFilePrefix(finalFilePathPrefix);
        try {
//            读取本地conf文件配置时
            ClientGlobal.init(“fdfs_client.conf”);
//            读取apollo配置时
//            ClientGlobal.setG_connect_timeout(connectTimeout);
//            ClientGlobal.setG_network_timeout(networkTimeout);
//            ClientGlobal.setG_charset(charset);
//            ClientGlobal.setG_anti_steal_token(httpAntiStealToken);
//            ClientGlobal.setG_secret_key(httpSecretKey);
//            ClientGlobal.setG_tracker_http_port(httpTrackerPort);
//            String[] trackerServers = trackerServerIps.split(“\,”);
//            InetSocketAddress[] tracker_servers = new InetSocketAddress[trackerServers.length];
//            for (int i = 0; i < trackerServers.length; i++) {
//              tracker_servers[i] = new InetSocketAddress(trackerServers[i].trim(), trackerServerPort);
//            }
//            TrackerGroup trackerGroup = new TrackerGroup(tracker_servers);
//            ClientGlobal.setG_tracker_group(trackerGroup);
            //获取client
            TrackerClient tracker = new TrackerClient();
            TrackerServer trackerServer = tracker.getConnection();
            StorageServer storageServer = null;
            StorageClient client = new StorageClient(trackerServer, storageServer);
            //上传文件
            NameValuePair[] metaList = new NameValuePair[1];
            metaList[0] = new NameValuePair(“fileName”, fileName);
            String results[] = client.upload_file(fileName, null, metaList);
            filePath = filePath + results[0]+“/”+results[1];
        } catch (IOException e) {
            e.printStackTrace();
        } catch (MyException e) {
            e.printStackTrace();
        }
        
        return filePath;
    }

下载文件

@Override
    public String download(String url, String localName) {
        if (StringUtil.isBlank(url)) {
            return “Fail”;
        }
        String fileName = url.split(“/”+groupName+“/”)[1];
        try {
            ClientGlobal.setG_connect_timeout(connectTimeout);
            ClientGlobal.setG_network_timeout(networkTimeout);
            ClientGlobal.setG_charset(charset);
            ClientGlobal.setG_anti_steal_token(httpAntiStealToken);
            ClientGlobal.setG_secret_key(httpSecretKey);
            ClientGlobal.setG_tracker_http_port(httpTrackerPort);
            String trackerServers[] = trackerServerIps.split(“\,”);
            InetSocketAddress[] tracker_servers = new InetSocketAddress[trackerServers.length];
            for (int i = 0; i < trackerServers.length; i++) {
              tracker_servers[i] = new InetSocketAddress(trackerServers[i].trim(), trackerServerPort);
            }
            TrackerGroup trackerGroup = new TrackerGroup(tracker_servers);
            ClientGlobal.setG_tracker_group(trackerGroup);
            //获取client
            TrackerClient tracker = new TrackerClient();
            TrackerServer trackerServer = tracker.getConnection();
            StorageServer storageServer = null;
            StorageClient client = new StorageClient(trackerServer, storageServer);
            //0表示下载成功
            int count = client.download_file(groupName, fileName,localName);
            System.out.println(count);
        } catch (IOException | MyException e) {
            e.printStackTrace();
        } 
        
        return “success”;
    }

fdfs_client.conf 内容如下：

connect_timeout = 2
network_timeout = 30
charset = UTF-8
http.tracker_http_port = 8080
http.anti_steal_token = no
http.secret_key = FastDFS

tracker_server = 10.250.112.141:22122
tracker_server = 10.250.112.142:22122
 