RDMA编程实践-SEND-RECEICVE原语应用
RDMA编程实践
本文描述了RDMA编程过程中的SEND-RECEIVE双边原语的代码实现。包含多个版本，1、client向server发送消息，server回复client收到消息(ACK)，然后两边断开连接。2、server端循环等待客户端建立连接，client发送一次消息后，双方断开连接。3、server端循环等待客户端建立连接，一旦建立，client端可以一直向server端发送消息，直到发送消息为disconnect，server和client断开链接，但是server此时仍然可以等待别的client发送消息。
代码基于代码基于send-receive样例实现。关于代码注释，可以参考代码解释：
Makefile文件、会编译当前目录下的所有.c文件：
.PHONY: all clean
CC := gcc
CFLAGS := -Wall -g
LDLIBS := -lrdmacm -libverbs -lpthread -g
SRCS := $(wildcard *.c)
APPS := $(SRCS:.c=)
all: $(APPS)
%: %.c
$(CC) $(CFLAGS) $< -o $@ $(LDLIBS)
clean:
rm -f $(APPS)
version1 客户端-服务端消息一次传递
在这个阶段，我们希望能实现下面这样一个场景。client与server端相连接，client端能够发送一条消息给server，server收到该条消息之后恢复一条消息给client端表示我已经确认收到。之后两者断开连接。
代码：
// client1.c
#include server端代码
// server1.c
/*
* Copyright (c) 2005-2009 Intel Corporation. All rights reserved.
*
* This software is available to you under the OpenIB.org BSD license
* below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AWV
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include 首先make编译完之后，在server端执行 ./server1，然后在客户端执行./client1
运行结果：
可以看到 client向server发送了hello world，server收到之后打印出来并回复给client端ACK消息，client收到之后并打印。最后双方断开连接，完成！
version2-客户端发送一次，服务端循环等待
client2的代码跟上面一样，server2代码不一样。
server2的逻辑：在run函数进来之后记录一个connect点，当远程客户端发送完信息后，释放连接的资源，跳转到connect阶段准备让下一个client连接。
/*
* Copyright (c) 2005-2009 Intel Corporation. All rights reserved.
*
* This software is available to you under the OpenIB.org BSD license
* below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AWV
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include 运行结果：
可以看到客户端发送一次消息之后便结束了，服务端却一直等待连接，直到按下ctrl+c。
version3-客户端循环发送，服务端循环等待，一次连接
和上述版本2不同的时候，这里client和server只连接一次，然后可以多次发送消息。直到client发送的消息为disconnect
// client3.c
/*
* Copyright (c) 2010 Intel Corporation. All rights reserved.
*
* This software is available to you under the OpenIB.org BSD license
* below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AWV
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include // server3.c
/*
* Copyright (c) 2005-2009 Intel Corporation. All rights reserved.
*
* This software is available to you under the OpenIB.org BSD license
* below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AWV
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include 运行结果：
总结：
本文实现了rdma中send-receive双边原语的三种需求版本，从单次发送到两者都能多次发送。理解其中的代码逻辑，想要发送消息之前对端得创建一个recv队列用来接收消息。发送完了有一个发送完成队列，接收完了也有一个接收完成队列。最后双方断开连接需要一起断开，不能某一方执行disconnect另一方不执行。本次实验有一个关键点：
while ((ret = rdma_get_send_comp(id, &wc)) == 0)
这一行代码是等待发送成功，发送成功之后，对方会给一个隐式信息表示我已经收到。这里耗费得时间比较长一点，在版本3中，如果不注释掉，在server端的receive队列还没有建立好，这就导致client发送了消息，server还没有收到，双方就陷入了死循环中。