[v11,06/19] eal: support attach or detach share device from secondary
Checks
Commit Message
This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process
device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.
device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.
Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
lib/librte_eal/common/eal_common_dev.c | 20 +++-
lib/librte_eal/common/hotplug_mp.c | 175 ++++++++++++++++++++++++++++++++-
2 files changed, 189 insertions(+), 6 deletions(-)
Comments
On 11-Jul-18 4:09 AM, Qi Zhang wrote:
> This patch cover the multi-process hotplug case when a device
> attach/detach request be issued from a secondary process
>
> device attach on secondary:
> a) secondary send sync request to the primary.
> b) primary receive the request and attach the new device if
> failed goto i).
> c) primary forward attach sync request to all secondary.
> d) secondary receive the request and attach the device and send a reply.
> e) primary check the reply if all success goes to j).
> f) primary send attach rollback sync request to all secondary.
> g) secondary receive the request and detach the device and send a reply.
> h) primary receive the reply and detach device as rollback action.
> i) send attach fail to secondary as a reply of step a), goto k).
> j) send attach success to secondary as a reply of step a).
> k) secondary receive reply and return.
>
> device detach on secondary:
> a) secondary send sync request to the primary.
> b) primary send detach sync request to all secondary.
> c) secondary detach the device and send a reply.
> d) primary check the reply if all success goes to g).
> e) primary send detach rollback sync request to all secondary.
> f) secondary receive the request and attach back device. goto h).
> g) primary detach the device if success goto i), else goto e).
> h) primary send detach fail to secondary as a reply of step a), goto j).
> i) primary send detach success to secondary as a reply of step a).
> j) secondary receive reply and return.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
> lib/librte_eal/common/eal_common_dev.c | 20 +++-
> lib/librte_eal/common/hotplug_mp.c | 175 ++++++++++++++++++++++++++++++++-
> 2 files changed, 189 insertions(+), 6 deletions(-)
>
> diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
> index ab2b25558..ee09720c4 100644
> --- a/lib/librte_eal/common/eal_common_dev.c
> +++ b/lib/librte_eal/common/eal_common_dev.c
> @@ -221,7 +221,7 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
> if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> /**
> * If in secondary process, just send IPC request to
> - * primary process
> + * primary process.
> */
> ret = eal_dev_hotplug_request_to_primary(&req);
> if (ret) {
> @@ -234,6 +234,7 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
> "Failed to hotplug add device\n");
> return req.result;
> }
> +
> /**
The above two changes look like unintended noise (or perhaps you meant
to merge them into one of the previous commits.
> * attach a device from primary start from here:
> *
> @@ -295,6 +296,23 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
> if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> /**
> * If in secondary process, just send IPC request to
> + * primary process.
<snip>
> + memset(&mp_req, 0, sizeof(mp_req));
> + memcpy(mp_req.param, req, sizeof(*req));
> + mp_req.len_param = sizeof(*req);
> + strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
> +
> + ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> + if (ret || mp_reply.nb_received != 1) {
> + RTE_LOG(ERR, EAL, "cannot send request to primary");
> + return ret;
> + }
In case of ret = 0 and nb_received = 0, you would be returning 0 as ret
= 0. I don't think that's what you intended here :)
> +
> + resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
> + req->result = resp->result;
> +
> + return ret;
> }
>
> int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
>
> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Wednesday, July 11, 2018 8:34 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v11 06/19] eal: support attach or detach share device from
> secondary
>
> On 11-Jul-18 4:09 AM, Qi Zhang wrote:
> > This patch cover the multi-process hotplug case when a device
> > attach/detach request be issued from a secondary process
> >
> > device attach on secondary:
> > a) secondary send sync request to the primary.
> > b) primary receive the request and attach the new device if
> > failed goto i).
> > c) primary forward attach sync request to all secondary.
> > d) secondary receive the request and attach the device and send a reply.
> > e) primary check the reply if all success goes to j).
> > f) primary send attach rollback sync request to all secondary.
> > g) secondary receive the request and detach the device and send a reply.
> > h) primary receive the reply and detach device as rollback action.
> > i) send attach fail to secondary as a reply of step a), goto k).
> > j) send attach success to secondary as a reply of step a).
> > k) secondary receive reply and return.
> >
> > device detach on secondary:
> > a) secondary send sync request to the primary.
> > b) primary send detach sync request to all secondary.
> > c) secondary detach the device and send a reply.
> > d) primary check the reply if all success goes to g).
> > e) primary send detach rollback sync request to all secondary.
> > f) secondary receive the request and attach back device. goto h).
> > g) primary detach the device if success goto i), else goto e).
> > h) primary send detach fail to secondary as a reply of step a), goto j).
> > i) primary send detach success to secondary as a reply of step a).
> > j) secondary receive reply and return.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> > lib/librte_eal/common/eal_common_dev.c | 20 +++-
> > lib/librte_eal/common/hotplug_mp.c | 175
> ++++++++++++++++++++++++++++++++-
> > 2 files changed, 189 insertions(+), 6 deletions(-)
> >
> > diff --git a/lib/librte_eal/common/eal_common_dev.c
> > b/lib/librte_eal/common/eal_common_dev.c
> > index ab2b25558..ee09720c4 100644
> > --- a/lib/librte_eal/common/eal_common_dev.c
> > +++ b/lib/librte_eal/common/eal_common_dev.c
> > @@ -221,7 +221,7 @@ rte_eal_hotplug_add(const char *busname, const
> char *devname,
> > if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> > /**
> > * If in secondary process, just send IPC request to
> > - * primary process
> > + * primary process.
> > */
> > ret = eal_dev_hotplug_request_to_primary(&req);
> > if (ret) {
> > @@ -234,6 +234,7 @@ rte_eal_hotplug_add(const char *busname, const
> char *devname,
> > "Failed to hotplug add device\n");
> > return req.result;
> > }
> > +
> > /**
>
> The above two changes look like unintended noise (or perhaps you meant to
> merge them into one of the previous commits.
>
> > * attach a device from primary start from here:
> > *
> > @@ -295,6 +296,23 @@ rte_eal_hotplug_remove(const char *busname,
> const char *devname)
> > if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> > /**
> > * If in secondary process, just send IPC request to
> > + * primary process.
>
> <snip>
>
> > + memset(&mp_req, 0, sizeof(mp_req));
> > + memcpy(mp_req.param, req, sizeof(*req));
> > + mp_req.len_param = sizeof(*req);
> > + strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST,
> > +sizeof(mp_req.name));
> > +
> > + ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> > + if (ret || mp_reply.nb_received != 1) {
> > + RTE_LOG(ERR, EAL, "cannot send request to primary");
> > + return ret;
> > + }
>
> In case of ret = 0 and nb_received = 0, you would be returning 0 as ret = 0. I
> don't think that's what you intended here :)
Ah.. yes, I missed that, thanks
>
> > +
> > + resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
> > + req->result = resp->result;
> > +
> > + return ret;
> > }
> >
> > int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
> >
>
>
> --
> Thanks,
> Anatoly
@@ -221,7 +221,7 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
/**
* If in secondary process, just send IPC request to
- * primary process
+ * primary process.
*/
ret = eal_dev_hotplug_request_to_primary(&req);
if (ret) {
@@ -234,6 +234,7 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
"Failed to hotplug add device\n");
return req.result;
}
+
/**
* attach a device from primary start from here:
*
@@ -295,6 +296,23 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
/**
* If in secondary process, just send IPC request to
+ * primary process.
+ */
+ ret = eal_dev_hotplug_request_to_primary(&req);
+ if (ret) {
+ RTE_LOG(ERR, EAL,
+ "Failed to send hotplug request to primary\n");
+ return ret;
+ }
+ if (req.result)
+ RTE_LOG(ERR, EAL,
+ "Failed to hotplug remove device\n");
+ return req.result;
+ }
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /**
+ * If in secondary process, just send IPC request to
* primary process.
*/
ret = eal_dev_hotplug_request_to_primary(&req);
@@ -17,12 +17,158 @@ struct mp_reply_bundle {
void *peer;
};
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ * failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+ int result,
+ const void *peer)
+{
+ struct rte_mp_msg mp_resp;
+ struct eal_dev_mp_req *resp =
+ (struct eal_dev_mp_req *)mp_resp.param;
+ int ret;
+
+ memset(&mp_resp, 0, sizeof(mp_resp));
+ mp_resp.len_param = sizeof(*resp);
+ strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+ memcpy(resp, req, sizeof(*req));
+ resp->result = result;
+
+ ret = rte_mp_reply(&mp_resp, peer);
+ if (ret)
+ RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+ return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+ struct mp_reply_bundle *bundle = param;
+ const struct rte_mp_msg *msg = &bundle->msg;
+ const struct eal_dev_mp_req *req =
+ (const struct eal_dev_mp_req *)msg->param;
+ struct eal_dev_mp_req tmp_req;
+ int ret = 0;
+
+ tmp_req = *req;
+
+ if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+ ret = do_dev_hotplug_add(req->busname, req->devname,
+ req->devargs);
+ if (ret) {
+ RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+ goto finish;
+ }
+ ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+ if (ret) {
+ RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+ goto finish;
+ }
+ if (tmp_req.result) {
+ ret = tmp_req.result;
+ RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+ }
+ } else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+ ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+ if (ret) {
+ RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+ goto finish;
+ }
+ if (!tmp_req.result) {
+ ret = do_dev_hotplug_remove(req->busname,
+ req->devname);
+ if (ret) {
+ RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+ goto rollback;
+ }
+ } else {
+ RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+ ret = tmp_req.result;
+ goto rollback;
+ }
+ } else {
+ RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+ ret = -ENOTSUP;
+ }
+ goto finish;
+
+rollback:
+ tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+ ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+ if (ret)
+ RTE_LOG(ERR, EAL, "Failed to send hotplug rollback request to secondary\n");
+
+finish:
+ ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+ if (ret)
+ RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+ free(bundle->peer);
+ free(bundle);
+}
+
static int
handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
{
- RTE_SET_USED(msg);
- RTE_SET_USED(peer);
- return -ENOTSUP;
+ struct mp_reply_bundle *bundle;
+ const struct eal_dev_mp_req *req =
+ (const struct eal_dev_mp_req *)msg->param;
+ int ret = 0;
+
+ bundle = malloc(sizeof(*bundle));
+ if (bundle == NULL) {
+ RTE_LOG(ERR, EAL, "not enough memory\n");
+ return send_response_to_secondary(req, -ENOMEM, peer);
+ }
+
+ bundle->msg = *msg;
+ /**
+ * We need to send reply on interrupt thread, but peer can't be
+ * parsed directly, so this is a temporal hack, need to be fixed
+ * when it is ready.
+ */
+ bundle->peer = strdup(peer);
+
+ /**
+ * We are at IPC callback thread, sync IPC is not allowed due to
+ * dead lock, so we delegate the task to interrupt thread.
+ */
+ ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+ if (ret) {
+ RTE_LOG(ERR, EAL, "failed to add mp task\n");
+ return send_response_to_secondary(req, ret, peer);
+ }
+ return 0;
}
static void __handle_primary_request(void *param)
@@ -114,8 +260,27 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
{
- RTE_SET_USED(req);
- return -ENOTSUP;
+ struct rte_mp_msg mp_req;
+ struct rte_mp_reply mp_reply;
+ struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+ struct eal_dev_mp_req *resp;
+ int ret;
+
+ memset(&mp_req, 0, sizeof(mp_req));
+ memcpy(mp_req.param, req, sizeof(*req));
+ mp_req.len_param = sizeof(*req);
+ strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+ ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+ if (ret || mp_reply.nb_received != 1) {
+ RTE_LOG(ERR, EAL, "cannot send request to primary");
+ return ret;
+ }
+
+ resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+ req->result = resp->result;
+
+ return ret;
}
int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)