[21/22] net/hns3: add multiple process support for hns3 PMD driver
diff mbox series

Message ID 1566568031-45991-22-git-send-email-xavier.huwei@huawei.com
State Changes Requested, archived
Delegated to: Ferruh Yigit
Headers show
Series
  • add hns3 ethernet PMD driver
Related show

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation fail Compilation issues

Commit Message

Wei Hu (Xavier) Aug. 23, 2019, 1:47 p.m. UTC
This patch adds multiple process support for hns3 PMD driver.
Multi-process support selection queue by configuring RSS or
flow director. The primary process supports various management
ops, and the secondary process only supports queries ops.
The primary process notifies the secondary processes to start
or stop tranceiver.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: Min Wang (Jushui) <wangmin3@huawei.com>
Signed-off-by: Wei Hu (Xavier) <xavier.huwei@huawei.com>
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Hao Chen <chenhao164@huawei.com>
Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 drivers/net/hns3/hns3_ethdev.c    |  36 ++++++-
 drivers/net/hns3/hns3_ethdev_vf.c |  29 ++++-
 drivers/net/hns3/hns3_mp.c        | 219 ++++++++++++++++++++++++++++++++++++++
 drivers/net/hns3/hns3_mp.h        |  14 +++
 4 files changed, 296 insertions(+), 2 deletions(-)
 create mode 100644 drivers/net/hns3/hns3_mp.c
 create mode 100644 drivers/net/hns3/hns3_mp.h

Comments

Ferruh Yigit Aug. 30, 2019, 3:14 p.m. UTC | #1
On 8/23/2019 2:47 PM, Wei Hu (Xavier) wrote:
> This patch adds multiple process support for hns3 PMD driver.
> Multi-process support selection queue by configuring RSS or
> flow director. The primary process supports various management
> ops, and the secondary process only supports queries ops.
> The primary process notifies the secondary processes to start
> or stop tranceiver.
> 
> Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
> Signed-off-by: Min Wang (Jushui) <wangmin3@huawei.com>
> Signed-off-by: Wei Hu (Xavier) <xavier.huwei@huawei.com>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Hao Chen <chenhao164@huawei.com>
> Signed-off-by: Huisong Li <lihuisong@huawei.com>

<...>

> @@ -1556,6 +1559,25 @@ static const struct eth_dev_ops hns3vf_eth_dev_ops = {
>  	.dev_supported_ptypes_get = hns3_dev_supported_ptypes_get,
>  };
>  
> +static const struct eth_dev_ops hns3vf_eth_dev_secondary_ops = {
> +	.stats_get          = hns3_stats_get,
> +	.stats_reset        = hns3_stats_reset,
> +	.xstats_get         = hns3_dev_xstats_get,
> +	.xstats_get_names   = hns3_dev_xstats_get_names,
> +	.xstats_reset	    = hns3_dev_xstats_reset,
> +	.xstats_get_by_id   = hns3_dev_xstats_get_by_id,
> +	.xstats_get_names_by_id = hns3_dev_xstats_get_names_by_id,
> +	.dev_infos_get      = hns3vf_dev_infos_get,
> +	.link_update        = hns3vf_dev_link_update,
> +	.rss_hash_update    = hns3_dev_rss_hash_update,
> +	.rss_hash_conf_get  = hns3_dev_rss_hash_conf_get,
> +	.reta_update        = hns3_dev_rss_reta_update,
> +	.reta_query         = hns3_dev_rss_reta_query,
> +	.filter_ctrl        = hns3_dev_filter_ctrl,
> +	.get_reg            = hns3_get_regs,
> +	.dev_supported_ptypes_get = hns3_dev_supported_ptypes_get,
> +};
> +

There shouldn't need to define separate dev_ops for the secondary processes,
what is the difference of this one used for primary process, why not use that one?

<...>

> +/*
> + * Initialize by secondary process.
> + */
> +void hns3_mp_init_secondary(void)
> +{
> +	rte_mp_action_register(HNS3_MP_NAME, mp_secondary_handle);

What is this handler for? Most of the case the MP communication is done in eal
level and nothing need to be done in the driver level.
Wei Hu (Xavier) Sept. 2, 2019, 1:41 p.m. UTC | #2
Hi, Ferruh Yigit

On 2019/8/30 23:14, Ferruh Yigit wrote:
> On 8/23/2019 2:47 PM, Wei Hu (Xavier) wrote:
>> This patch adds multiple process support for hns3 PMD driver.
>> Multi-process support selection queue by configuring RSS or
>> flow director. The primary process supports various management
>> ops, and the secondary process only supports queries ops.
>> The primary process notifies the secondary processes to start
>> or stop tranceiver.
>>
>> Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
>> Signed-off-by: Min Wang (Jushui) <wangmin3@huawei.com>
>> Signed-off-by: Wei Hu (Xavier) <xavier.huwei@huawei.com>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Hao Chen <chenhao164@huawei.com>
>> Signed-off-by: Huisong Li <lihuisong@huawei.com>
> <...>
>
>> @@ -1556,6 +1559,25 @@ static const struct eth_dev_ops hns3vf_eth_dev_ops = {
>>  	.dev_supported_ptypes_get = hns3_dev_supported_ptypes_get,
>>  };
>>  
>> +static const struct eth_dev_ops hns3vf_eth_dev_secondary_ops = {
>> +	.stats_get          = hns3_stats_get,
>> +	.stats_reset        = hns3_stats_reset,
>> +	.xstats_get         = hns3_dev_xstats_get,
>> +	.xstats_get_names   = hns3_dev_xstats_get_names,
>> +	.xstats_reset	    = hns3_dev_xstats_reset,
>> +	.xstats_get_by_id   = hns3_dev_xstats_get_by_id,
>> +	.xstats_get_names_by_id = hns3_dev_xstats_get_names_by_id,
>> +	.dev_infos_get      = hns3vf_dev_infos_get,
>> +	.link_update        = hns3vf_dev_link_update,
>> +	.rss_hash_update    = hns3_dev_rss_hash_update,
>> +	.rss_hash_conf_get  = hns3_dev_rss_hash_conf_get,
>> +	.reta_update        = hns3_dev_rss_reta_update,
>> +	.reta_query         = hns3_dev_rss_reta_query,
>> +	.filter_ctrl        = hns3_dev_filter_ctrl,
>> +	.get_reg            = hns3_get_regs,
>> +	.dev_supported_ptypes_get = hns3_dev_supported_ptypes_get,
>> +};
>> +
> There shouldn't need to define separate dev_ops for the secondary processes,
> what is the difference of this one used for primary process, why not use that one?
We limit the slave process to only perform query ops actions, and cannot
perform configuration ops actions to prevent the master and slave
processes from performing configuration ops at the same time, which may
cause conflicts.
>
> <...>
>
>> +/*
>> + * Initialize by secondary process.
>> + */
>> +void hns3_mp_init_secondary(void)
>> +{
>> +	rte_mp_action_register(HNS3_MP_NAME, mp_secondary_handle);
> What is this handler for? Most of the case the MP communication is done in eal
> level and nothing need to be done in the driver level.
>

When the primary process executes dev_stop, it will release the mbuf of
all rx queues. The secondary process may access the mbuf of the rx
queue, and a segmentation error will occur.So MP is used in
multi-process to solve this problem. When primary process executes
dev_stop, it informs the process to stop receiving, and then releases
the mbuf, so that the secondary process will not have a segmentation error.

    Regards
Xavier

Patch
diff mbox series

diff --git a/drivers/net/hns3/hns3_ethdev.c b/drivers/net/hns3/hns3_ethdev.c
index 9a4c560..28fa9a6 100644
--- a/drivers/net/hns3/hns3_ethdev.c
+++ b/drivers/net/hns3/hns3_ethdev.c
@@ -40,6 +40,7 @@ 
 #include "hns3_intr.h"
 #include "hns3_regs.h"
 #include "hns3_dcb.h"
+#include "hns3_mp.h"
 
 #define HNS3_DEFAULT_PORT_CONF_BURST_SIZE	32
 #define HNS3_DEFAULT_PORT_CONF_QUEUES_NUM	1
@@ -4078,6 +4079,10 @@  hns3_dev_stop(struct rte_eth_dev *eth_dev)
 	hw->adapter_state = HNS3_NIC_STOPPING;
 	hns3_set_rxtx_function(eth_dev);
 	rte_wmb();
+	/* Disable datapath on secondary process. */
+	hns3_mp_req_stop_rxtx(eth_dev);
+	/* Prevent crashes when queues are still in use. */
+	rte_delay_ms(hw->tqps_num);
 
 	rte_spinlock_lock(&hw->lock);
 	if (rte_atomic16_read(&hw->reset.resetting) == 0) {
@@ -4108,6 +4113,7 @@  hns3_dev_close(struct rte_eth_dev *eth_dev)
 	hns3_uninit_pf(eth_dev);
 	hns3_free_all_queues(eth_dev);
 	rte_free(hw->reset.wait_data);
+	hns3_mp_uninit_primary();
 	hns3_warn(hw, "Close port %d finished", hw->data->port_id);
 }
 
@@ -4750,6 +4756,28 @@  static const struct eth_dev_ops hns3_eth_dev_ops = {
 	.dev_supported_ptypes_get = hns3_dev_supported_ptypes_get,
 };
 
+static const struct eth_dev_ops hns3_eth_dev_secondary_ops = {
+	.stats_get          = hns3_stats_get,
+	.stats_reset        = hns3_stats_reset,
+	.xstats_get         = hns3_dev_xstats_get,
+	.xstats_get_names   = hns3_dev_xstats_get_names,
+	.xstats_reset       = hns3_dev_xstats_reset,
+	.xstats_get_by_id   = hns3_dev_xstats_get_by_id,
+	.xstats_get_names_by_id = hns3_dev_xstats_get_names_by_id,
+	.dev_infos_get          = hns3_dev_infos_get,
+	.fw_version_get         = hns3_fw_version_get,
+	.flow_ctrl_get          = hns3_flow_ctrl_get,
+	.link_update            = hns3_dev_link_update,
+	.rss_hash_update        = hns3_dev_rss_hash_update,
+	.rss_hash_conf_get      = hns3_dev_rss_hash_conf_get,
+	.reta_update            = hns3_dev_rss_reta_update,
+	.reta_query             = hns3_dev_rss_reta_query,
+	.filter_ctrl            = hns3_dev_filter_ctrl,
+	.get_reg                = hns3_get_regs,
+	.get_dcb_info           = hns3_get_dcb_info,
+	.dev_supported_ptypes_get = hns3_dev_supported_ptypes_get,
+};
+
 static const struct hns3_reset_ops hns3_reset_ops = {
 	.reset_service       = hns3_reset_service,
 	.stop_service        = hns3_stop_service,
@@ -4783,10 +4811,16 @@  hns3_dev_init(struct rte_eth_dev *eth_dev)
 	hns3_filterlist_init(eth_dev);
 
 	hns3_set_rxtx_function(eth_dev);
-	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		eth_dev->dev_ops = &hns3_eth_dev_secondary_ops;
+		hns3_mp_init_secondary();
+		hw->secondary_cnt++;
 		return 0;
+	}
 
 	eth_dev->dev_ops = &hns3_eth_dev_ops;
+	hns3_mp_init_primary();
+	hw->adapter_state = HNS3_NIC_UNINITIALIZED;
 	rte_eth_copy_pci_info(eth_dev, pci_dev);
 
 	if (device_id == HNS3_DEV_ID_25GE_RDMA ||
diff --git a/drivers/net/hns3/hns3_ethdev_vf.c b/drivers/net/hns3/hns3_ethdev_vf.c
index 45360c4..1d33779 100644
--- a/drivers/net/hns3/hns3_ethdev_vf.c
+++ b/drivers/net/hns3/hns3_ethdev_vf.c
@@ -40,6 +40,7 @@ 
 #include "hns3_regs.h"
 #include "hns3_intr.h"
 #include "hns3_dcb.h"
+#include "hns3_mp.h"
 
 #define HNS3VF_KEEP_ALIVE_INTERVAL	2000000 /* us */
 #define HNS3VF_SERVICE_INTERVAL		1000000 /* us */
@@ -1174,6 +1175,7 @@  hns3vf_dev_close(struct rte_eth_dev *eth_dev)
 	hns3vf_uninit_vf(eth_dev);
 	hns3_free_all_queues(eth_dev);
 	rte_free(hw->reset.wait_data);
+	hns3_mp_uninit_primary();
 	hns3_warn(hw, "Close port %d finished", hw->data->port_id);
 }
 
@@ -1251,6 +1253,7 @@  hns3vf_dev_start(struct rte_eth_dev *eth_dev)
 	hw->adapter_state = HNS3_NIC_STARTED;
 	rte_spinlock_unlock(&hw->lock);
 	hns3_set_rxtx_function(eth_dev);
+	hns3_mp_req_start_rxtx(eth_dev);
 	return 0;
 }
 
@@ -1556,6 +1559,25 @@  static const struct eth_dev_ops hns3vf_eth_dev_ops = {
 	.dev_supported_ptypes_get = hns3_dev_supported_ptypes_get,
 };
 
+static const struct eth_dev_ops hns3vf_eth_dev_secondary_ops = {
+	.stats_get          = hns3_stats_get,
+	.stats_reset        = hns3_stats_reset,
+	.xstats_get         = hns3_dev_xstats_get,
+	.xstats_get_names   = hns3_dev_xstats_get_names,
+	.xstats_reset	    = hns3_dev_xstats_reset,
+	.xstats_get_by_id   = hns3_dev_xstats_get_by_id,
+	.xstats_get_names_by_id = hns3_dev_xstats_get_names_by_id,
+	.dev_infos_get      = hns3vf_dev_infos_get,
+	.link_update        = hns3vf_dev_link_update,
+	.rss_hash_update    = hns3_dev_rss_hash_update,
+	.rss_hash_conf_get  = hns3_dev_rss_hash_conf_get,
+	.reta_update        = hns3_dev_rss_reta_update,
+	.reta_query         = hns3_dev_rss_reta_query,
+	.filter_ctrl        = hns3_dev_filter_ctrl,
+	.get_reg            = hns3_get_regs,
+	.dev_supported_ptypes_get = hns3_dev_supported_ptypes_get,
+};
+
 static const struct hns3_reset_ops hns3vf_reset_ops = {
 	.reset_service       = hns3vf_reset_service,
 	.stop_service        = hns3vf_stop_service,
@@ -1589,10 +1611,15 @@  hns3vf_dev_init(struct rte_eth_dev *eth_dev)
 	hns3_filterlist_init(eth_dev);
 
 	hns3_set_rxtx_function(eth_dev);
-	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		eth_dev->dev_ops = &hns3vf_eth_dev_secondary_ops;
+		hns3_mp_init_secondary();
+		hw->secondary_cnt++;
 		return 0;
+	}
 
 	eth_dev->dev_ops = &hns3vf_eth_dev_ops;
+	hns3_mp_init_primary();
 
 	hw->adapter_state = HNS3_NIC_UNINITIALIZED;
 	rte_eth_copy_pci_info(eth_dev, pci_dev);
diff --git a/drivers/net/hns3/hns3_mp.c b/drivers/net/hns3/hns3_mp.c
new file mode 100644
index 0000000..2f56d8b
--- /dev/null
+++ b/drivers/net/hns3/hns3_mp.c
@@ -0,0 +1,219 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018-2019 Hisilicon Limited.
+ */
+
+#include <stdbool.h>
+
+#include <rte_eal.h>
+#include <rte_ethdev_driver.h>
+#include <rte_string_fns.h>
+#include <rte_io.h>
+
+#include "hns3_cmd.h"
+#include "hns3_mbx.h"
+#include "hns3_rss.h"
+#include "hns3_fdir.h"
+#include "hns3_stats.h"
+#include "hns3_ethdev.h"
+#include "hns3_logs.h"
+#include "hns3_rxtx.h"
+#include "hns3_mp.h"
+
+/*
+ * Initialize IPC message.
+ *
+ * @param[in] dev
+ *   Pointer to Ethernet structure.
+ * @param[out] msg
+ *   Pointer to message to fill in.
+ * @param[in] type
+ *   Message type.
+ */
+static inline void
+mp_init_msg(struct rte_eth_dev *dev, struct rte_mp_msg *msg,
+	    enum hns3_mp_req_type type)
+{
+	struct hns3_mp_param *param = (struct hns3_mp_param *)msg->param;
+
+	memset(msg, 0, sizeof(*msg));
+	strlcpy(msg->name, HNS3_MP_NAME, sizeof(msg->name));
+	msg->len_param = sizeof(*param);
+	param->type = type;
+	param->port_id = dev->data->port_id;
+}
+
+/*
+ * IPC message handler of primary process.
+ *
+ * @param[in] dev
+ *   Pointer to Ethernet structure.
+ * @param[in] peer
+ *   Pointer to the peer socket path.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mp_primary_handle(const struct rte_mp_msg *mp_msg __rte_unused,
+		  const void *peer __rte_unused)
+{
+	return 0;
+}
+
+/*
+ * IPC message handler of a secondary process.
+ *
+ * @param[in] dev
+ *   Pointer to Ethernet structure.
+ * @param[in] peer
+ *   Pointer to the peer socket path.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mp_secondary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
+{
+	struct rte_mp_msg mp_res;
+	struct hns3_mp_param *res = (struct hns3_mp_param *)mp_res.param;
+	const struct hns3_mp_param *param =
+		(const struct hns3_mp_param *)mp_msg->param;
+	struct rte_eth_dev *dev;
+	int ret;
+
+	if (!rte_eth_dev_is_valid_port(param->port_id)) {
+		rte_errno = ENODEV;
+		PMD_INIT_LOG(ERR, "port %u invalid port ID", param->port_id);
+		return -rte_errno;
+	}
+	dev = &rte_eth_devices[param->port_id];
+	switch (param->type) {
+	case HNS3_MP_REQ_START_RXTX:
+		PMD_INIT_LOG(INFO, "port %u starting datapath",
+			     dev->data->port_id);
+		rte_mb();
+		hns3_set_rxtx_function(dev);
+		mp_init_msg(dev, &mp_res, param->type);
+		res->result = 0;
+		ret = rte_mp_reply(&mp_res, peer);
+		break;
+	case HNS3_MP_REQ_STOP_RXTX:
+		PMD_INIT_LOG(INFO, "port %u stopping datapath",
+			     dev->data->port_id);
+		hns3_set_rxtx_function(dev);
+		rte_mb();
+		mp_init_msg(dev, &mp_res, param->type);
+		res->result = 0;
+		ret = rte_mp_reply(&mp_res, peer);
+		break;
+	default:
+		rte_errno = EINVAL;
+		PMD_INIT_LOG(ERR, "port %u invalid mp request type",
+			     dev->data->port_id);
+		return -rte_errno;
+	}
+	return ret;
+}
+
+/*
+ * Broadcast request of stopping/starting data-path to secondary processes.
+ *
+ * @param[in] dev
+ *   Pointer to Ethernet structure.
+ * @param[in] type
+ *   Request type.
+ */
+static void
+mp_req_on_rxtx(struct rte_eth_dev *dev, enum hns3_mp_req_type type)
+{
+	struct hns3_hw *hw = HNS3_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct rte_mp_msg mp_req;
+	struct rte_mp_msg *mp_res;
+	struct rte_mp_reply mp_rep;
+	struct hns3_mp_param *res;
+	struct timespec ts;
+	int ret;
+	int i;
+
+	if (!hw->secondary_cnt)
+		return;
+	if (type != HNS3_MP_REQ_START_RXTX && type != HNS3_MP_REQ_STOP_RXTX) {
+		hns3_err(hw, "port %u unknown request (req_type %d)",
+			 dev->data->port_id, type);
+		return;
+	}
+	mp_init_msg(dev, &mp_req, type);
+	ts.tv_sec = HNS3_MP_REQ_TIMEOUT_SEC;
+	ts.tv_nsec = 0;
+	ret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);
+	if (ret) {
+		hns3_err(hw, "port %u failed to request stop/start Rx/Tx (%d)",
+			 dev->data->port_id, type);
+		goto exit;
+	}
+	if (mp_rep.nb_sent != mp_rep.nb_received) {
+		PMD_INIT_LOG(ERR,
+			"port %u not all secondaries responded (req_type %d)",
+			dev->data->port_id, type);
+		goto exit;
+	}
+	for (i = 0; i < mp_rep.nb_received; i++) {
+		mp_res = &mp_rep.msgs[i];
+		res = (struct hns3_mp_param *)mp_res->param;
+		if (res->result) {
+			hns3_err(hw, "port %u request failed on secondary #%d",
+				 dev->data->port_id, i);
+			goto exit;
+		}
+	}
+exit:
+	free(mp_rep.msgs);
+}
+
+/*
+ * Broadcast request of starting data-path to secondary processes. The request
+ * is synchronous.
+ *
+ * @param[in] dev
+ *   Pointer to Ethernet structure.
+ */
+void hns3_mp_req_start_rxtx(struct rte_eth_dev *dev)
+{
+	mp_req_on_rxtx(dev, HNS3_MP_REQ_START_RXTX);
+}
+
+/*
+ * Broadcast request of stopping data-path to secondary processes. The request
+ * is synchronous.
+ *
+ * @param[in] dev
+ *   Pointer to Ethernet structure.
+ */
+void hns3_mp_req_stop_rxtx(struct rte_eth_dev *dev)
+{
+	mp_req_on_rxtx(dev, HNS3_MP_REQ_STOP_RXTX);
+}
+
+/*
+ * Initialize by primary process.
+ */
+void hns3_mp_init_primary(void)
+{
+	rte_mp_action_register(HNS3_MP_NAME, mp_primary_handle);
+}
+
+/*
+ * Un-initialize by primary process.
+ */
+void hns3_mp_uninit_primary(void)
+{
+	rte_mp_action_unregister(HNS3_MP_NAME);
+}
+
+/*
+ * Initialize by secondary process.
+ */
+void hns3_mp_init_secondary(void)
+{
+	rte_mp_action_register(HNS3_MP_NAME, mp_secondary_handle);
+}
diff --git a/drivers/net/hns3/hns3_mp.h b/drivers/net/hns3/hns3_mp.h
new file mode 100644
index 0000000..aefbeb1
--- /dev/null
+++ b/drivers/net/hns3/hns3_mp.h
@@ -0,0 +1,14 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018-2019 Hisilicon Limited.
+ */
+
+#ifndef _HNS3_MP_H_
+#define _HNS3_MP_H_
+
+void hns3_mp_req_start_rxtx(struct rte_eth_dev *dev);
+void hns3_mp_req_stop_rxtx(struct rte_eth_dev *dev);
+void hns3_mp_init_primary(void);
+void hns3_mp_uninit_primary(void);
+void hns3_mp_init_secondary(void);
+
+#endif /* _HNS3_MP_H_ */