Ceph IO读写流程详解(一)——RADOSGW
1 radosgw接收用户请求过程
从今天起，我将开启一个专题，讲述一个对象是如何从上传rgw到最终落盘的整个过程，期间也会有一些扩展内容，包括一些c++、网络等基础知识，权当作为一个复习吧。
本文是以luminous版本进行讲解，虽然经过了迭代，但大体的思路是不变的。
废话不多说，现在开始。
Ceph对外提供服务是通过radosgw（存储网关）进行的，那么我们就从rgw的main函数开始进行探讨。该函数位于src/rgw/rgw_main.cc。
civetweb和beast是rgw的前端框架，用于接收来自client端的请求，启动方式如下：
...
for (multimap<string, RGWFrontendConfig *>::iterator fiter = fe_map.begin();
fiter != fe_map.end(); ++fiter, ++fe_count) {
RGWFrontendConfig *config = fiter->second;
string framework = config->get_framework();
RGWFrontend *fe = NULL;
if (framework == "civetweb" || framework == "mongoose") {
framework = "civetweb";
std::string uri_prefix;
config->get_val("prefix", "", &uri_prefix);
RGWProcessEnv env = { store, &rest, olog, 0, uri_prefix, auth_registry };
fe = new RGWCivetWebFrontend(env, config);
}
else if (framework == "loadgen") {
int port;
config->get_val("port", 80, &port);
std::string uri_prefix;
config->get_val("prefix", "", &uri_prefix);
RGWProcessEnv env = { store, &rest, olog, port, uri_prefix, auth_registry };
fe = new RGWLoadGenFrontend(env, config);
}
...
重要的就是初始化了RGWProcessEnv用于构建RGWCivetWebFrontend，L版的ceph使用的civetweb，我们也以civetweb进行讲解。
int r = fe->init();
if (r < 0) {
derr << "ERROR: failed initializing frontend" << dendl;
return -r;
}
r = fe->run();
if (r < 0) {
derr << "ERROR: failed run" << dendl;
return -r;
}
接下来的init并未做任何操作，只是return 0，run()则是真正开始运行：
int RGWCivetWebFrontend::run()
{
...
/* Initialize the CivetWeb right now. */
struct mg_callbacks cb;
memset((void *)&cb, 0, sizeof(cb));
cb.begin_request = civetweb_callback;
cb.log_message = rgw_civetweb_log_callback;
cb.log_access = rgw_civetweb_log_access_callback;
ctx = mg_start(&cb, this, options.data());
...
}
其中begin_request的回调函数如下：
static int civetweb_callback(struct mg_connection* conn)
{
const struct mg_request_info* const req_info = mg_get_request_info(conn);
return static_cast<RGWCivetWebFrontend *>(req_info->user_data)->process(conn);
}
在每次收到新请求的时候调用。
int RGWCivetWebFrontend::process(struct mg_connection* const conn)
{
/* Hold a read lock over access to env.store for reconfiguration. */
RWLock::RLocker lock(env.mutex);
RGWCivetWeb cw_client(conn);
auto real_client_io = rgw::io::add_reordering(
rgw::io::add_buffering(dout_context,
rgw::io::add_chunking(
rgw::io::add_conlen_controlling(
&cw_client))));
RGWRestfulIO client_io(dout_context, &real_client_io);
RGWRequest req(env.store->get_new_req_id());
int http_ret = 0;
int ret = process_request(env.store, env.rest, &req, env.uri_prefix,
*env.auth_registry, &client_io, env.olog, &http_ret);
if (ret < 0) {
/* We don't really care about return code. */
dout(20) << "process_request() returned " << ret << dendl;
}
if (http_ret <= 0) {
/* Mark as processed. */
return 1;
}
return http_ret;
}
也就是说每接收到一个请求，就会调用process函数进行处理，查看一下重点函数。
int process_request(RGWRados* const store,
RGWREST* const rest,
RGWRequest* const req,
const std::string& frontend_prefix,
const rgw_auth_registry_t& auth_registry,
RGWRestfulIO* const client_io,
OpsLogSocket* const olog,
int* http_ret,
void* asio_ctx)
{
...
RGWOp* op = nullptr;
int init_error = 0;
bool should_log = false;
RGWRESTMgr *mgr;
RGWHandler_REST *handler = rest->get_handler(store, s,
auth_registry,
frontend_prefix,
client_io, &mgr, &init_error);
...
op = handler->get_op(store);
if (!op) {
abort_early(s, NULL, -ERR_NOT_IMPLEMENTED, handler);
goto done;
}
req->op = op;
...
s->op_type = op->get_type();
...
ret = op->verify_requester(auth_registry);
...
ret = handler->postauth_init();
...
ret = rgw_process_authenticated(handler, op, req, s);
if (ret < 0) {
dout(0) << "fail to execute rgw_process_authenticated, ret=" << ret << dendl;
abort_early(s, op, ret, handler);
goto done;
}
done:
try {
client_io->complete_request();
} catch (rgw::io::Exception& e) {
dout(0) << __func__ << "ERROR: client_io->complete_request() returned "
<< e.what() << dendl;
}
...
if (handler)
handler->put_op(op);
rest->put_handler(handler);
...
return (ret < 0 ? ret : s->err.ret);
} /* process_request */
我将相对不太重要的语句忽略，只保留相对重要的函数调用。
借用这篇文档的一张图，很好的说明了Manager、Handler和Op之间的关系：
Ceph RGW整体结构，最全干货在这！
首先API请求打过来，会根据请求分配manager：
if (apis_map.count("admin") > 0) {
RGWRESTMgr_Admin *admin_resource = new RGWRESTMgr_Admin;
admin_resource->register_resource("usage", new RGWRESTMgr_Usage);
admin_resource->register_resource("user", new RGWRESTMgr_User);
admin_resource->register_resource("bucket", new RGWRESTMgr_Bucket);
/*Registering resource for /admin/metadata */
admin_resource->register_resource("metadata", new RGWRESTMgr_Metadata);
admin_resource->register_resource("log", new RGWRESTMgr_Log);
admin_resource->register_resource("opstate", new RGWRESTMgr_Opstate);
admin_resource->register_resource("replica_log", new RGWRESTMgr_ReplicaLog);
admin_resource->register_resource("config", new RGWRESTMgr_Config);
admin_resource->register_resource("realm", new RGWRESTMgr_Realm);
rest.register_resource(g_conf->rgw_admin_entry, admin_resource);
}
上面的这些manager是通过admin接口来实现交互的，实现的操作不多（s3请求的manager是最多的），可以当作例子来看，例如RGWRESTMgr_Config，就实现了读取zonegroup和zone信息的接口：
class RGWRESTMgr_Config : public RGWRESTMgr {
public:
RGWRESTMgr_Config() = default;
~RGWRESTMgr_Config() override = default;
RGWHandler_REST* get_handler(struct req_state*,
const rgw::auth::StrategyRegistry& auth_registry,
const std::string&) override {
return new RGWHandler_Config(auth_registry);
}
};
...
void RGWOp_ZoneGroupMap_Get::send_response() {
set_req_state_err(s, http_ret);
dump_errno(s);
end_header(s);
if (http_ret < 0)
return;
if (old_format) {
RGWRegionMap region_map;
region_map.regions = zonegroup_map.zonegroups;
region_map.master_region = zonegroup_map.master_zonegroup;
region_map.bucket_quota = zonegroup_map.bucket_quota;
region_map.user_quota = zonegroup_map.user_quota;
encode_json("region-map", region_map, s->formatter);
} else {
encode_json("zonegroup-map", zonegroup_map, s->formatter);
}
flusher.flush();
}
代码中的RGWOp_ZoneGroupMap_Get就是图中提到的Op，其他的S3请求，swift请求都与之类似，都有对应的manager，handler和执行的Op。
ret = op->verify_requester(auth_registry);这一行则是对请求进行鉴权，在这里不详细介绍了，目前使用的是aws v4鉴权：https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html
下一章节，我们将介绍最重磅的函数rgw_process_authenticated，这是处理Op最重要的函数。