From patchwork Thu Sep 26 06:28:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59781 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 455963423; Thu, 26 Sep 2019 08:29:57 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 0A0DB324D for ; Thu, 26 Sep 2019 08:29:54 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:29:54 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlG012493; Thu, 26 Sep 2019 09:29:53 +0300 From: Ori Kam To: Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:28:57 +0000 Message-Id: <1569479349-36962-2-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 01/13] ethdev: support setup function for hairpin queue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit introduce the RX/TX hairpin setup function. Hairpin is RX/TX queue that is used by the nic in order to offload wire to wire traffic. Each hairpin queue is binded to one or more queues from other type. For example TX hairpin queue should be binded to at least 1 RX hairpin queue and vice versa. Signed-off-by: Ori Kam --- lib/librte_ethdev/rte_ethdev.c | 213 +++++++++++++++++++++++++++++++ lib/librte_ethdev/rte_ethdev.h | 145 +++++++++++++++++++++ lib/librte_ethdev/rte_ethdev_core.h | 18 +++ lib/librte_ethdev/rte_ethdev_version.map | 4 + 4 files changed, 380 insertions(+) diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c index 30b0c78..4021f38 100644 --- a/lib/librte_ethdev/rte_ethdev.c +++ b/lib/librte_ethdev/rte_ethdev.c @@ -1701,6 +1701,115 @@ struct rte_eth_dev * } int +rte_eth_rx_hairpin_queue_setup(uint16_t port_id, uint16_t rx_queue_id, + uint16_t nb_rx_desc, unsigned int socket_id, + const struct rte_eth_rxconf *rx_conf, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + int ret; + struct rte_eth_dev *dev; + struct rte_eth_dev_info dev_info; + struct rte_eth_rxconf local_conf; + void **rxq; + + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL); + + dev = &rte_eth_devices[port_id]; + if (rx_queue_id >= dev->data->nb_rx_queues) { + RTE_ETHDEV_LOG(ERR, "Invalid RX queue_id=%u\n", rx_queue_id); + return -EINVAL; + } + + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_infos_get, -ENOTSUP); + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->rx_hairpin_queue_setup, + -ENOTSUP); + + rte_eth_dev_info_get(port_id, &dev_info); + + /* Use default specified by driver, if nb_rx_desc is zero */ + if (nb_rx_desc == 0) { + nb_rx_desc = dev_info.default_rxportconf.ring_size; + /* If driver default is also zero, fall back on EAL default */ + if (nb_rx_desc == 0) + nb_rx_desc = RTE_ETH_DEV_FALLBACK_RX_RINGSIZE; + } + + if (nb_rx_desc > dev_info.rx_desc_lim.nb_max || + nb_rx_desc < dev_info.rx_desc_lim.nb_min || + nb_rx_desc % dev_info.rx_desc_lim.nb_align != 0) { + + RTE_ETHDEV_LOG(ERR, + "Invalid value for nb_rx_desc(=%hu), should be: " + "<= %hu, >= %hu, and a product of %hu\n", + nb_rx_desc, dev_info.rx_desc_lim.nb_max, + dev_info.rx_desc_lim.nb_min, + dev_info.rx_desc_lim.nb_align); + return -EINVAL; + } + + if (dev->data->dev_started && + !(dev_info.dev_capa & + RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP)) + return -EBUSY; + + if (dev->data->dev_started && + (dev->data->rx_queue_state[rx_queue_id] != + RTE_ETH_QUEUE_STATE_STOPPED)) + return -EBUSY; + + rxq = dev->data->rx_queues; + if (rxq[rx_queue_id]) { + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->rx_queue_release, + -ENOTSUP); + (*dev->dev_ops->rx_queue_release)(rxq[rx_queue_id]); + rxq[rx_queue_id] = NULL; + } + + if (rx_conf == NULL) + rx_conf = &dev_info.default_rxconf; + + local_conf = *rx_conf; + + /* + * If an offloading has already been enabled in + * rte_eth_dev_configure(), it has been enabled on all queues, + * so there is no need to enable it in this queue again. + * The local_conf.offloads input to underlying PMD only carries + * those offloadings which are only enabled on this queue and + * not enabled on all queues. + */ + local_conf.offloads &= ~dev->data->dev_conf.rxmode.offloads; + + /* + * New added offloadings for this queue are those not enabled in + * rte_eth_dev_configure() and they must be per-queue type. + * A pure per-port offloading can't be enabled on a queue while + * disabled on another queue. A pure per-port offloading can't + * be enabled for any queue as new added one if it hasn't been + * enabled in rte_eth_dev_configure(). + */ + if ((local_conf.offloads & dev_info.rx_queue_offload_capa) != + local_conf.offloads) { + RTE_ETHDEV_LOG(ERR, + "Ethdev port_id=%d rx_queue_id=%d, " + "new added offloads 0x%"PRIx64" must be " + "within per-queue offload capabilities " + "0x%"PRIx64" in %s()\n", + port_id, rx_queue_id, local_conf.offloads, + dev_info.rx_queue_offload_capa, + __func__); + return -EINVAL; + } + + ret = (*dev->dev_ops->rx_hairpin_queue_setup)(dev, rx_queue_id, + nb_rx_desc, socket_id, + &local_conf, + hairpin_conf); + + return eth_err(port_id, ret); +} + +int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id, uint16_t nb_tx_desc, unsigned int socket_id, const struct rte_eth_txconf *tx_conf) @@ -1799,6 +1908,110 @@ struct rte_eth_dev * tx_queue_id, nb_tx_desc, socket_id, &local_conf)); } +int +rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, + uint16_t nb_tx_desc, unsigned int socket_id, + const struct rte_eth_txconf *tx_conf, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + struct rte_eth_dev *dev; + struct rte_eth_dev_info dev_info; + struct rte_eth_txconf local_conf; + void **txq; + + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL); + + dev = &rte_eth_devices[port_id]; + if (tx_queue_id >= dev->data->nb_tx_queues) { + RTE_ETHDEV_LOG(ERR, "Invalid TX queue_id=%u\n", tx_queue_id); + return -EINVAL; + } + + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_infos_get, -ENOTSUP); + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->tx_hairpin_queue_setup, + -ENOTSUP); + + rte_eth_dev_info_get(port_id, &dev_info); + + /* Use default specified by driver, if nb_tx_desc is zero */ + if (nb_tx_desc == 0) { + nb_tx_desc = dev_info.default_txportconf.ring_size; + /* If driver default is zero, fall back on EAL default */ + if (nb_tx_desc == 0) + nb_tx_desc = RTE_ETH_DEV_FALLBACK_TX_RINGSIZE; + } + if (nb_tx_desc > dev_info.tx_desc_lim.nb_max || + nb_tx_desc < dev_info.tx_desc_lim.nb_min || + nb_tx_desc % dev_info.tx_desc_lim.nb_align != 0) { + RTE_ETHDEV_LOG(ERR, + "Invalid value for nb_tx_desc(=%hu), " + "should be: <= %hu, >= %hu, and a product of " + " %hu\n", + nb_tx_desc, dev_info.tx_desc_lim.nb_max, + dev_info.tx_desc_lim.nb_min, + dev_info.tx_desc_lim.nb_align); + return -EINVAL; + } + + if (dev->data->dev_started && + !(dev_info.dev_capa & + RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP)) + return -EBUSY; + + if (dev->data->dev_started && + (dev->data->tx_queue_state[tx_queue_id] != + RTE_ETH_QUEUE_STATE_STOPPED)) + return -EBUSY; + + txq = dev->data->tx_queues; + if (txq[tx_queue_id]) { + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->tx_queue_release, + -ENOTSUP); + (*dev->dev_ops->tx_queue_release)(txq[tx_queue_id]); + txq[tx_queue_id] = NULL; + } + + if (tx_conf == NULL) + tx_conf = &dev_info.default_txconf; + + local_conf = *tx_conf; + + /* + * If an offloading has already been enabled in + * rte_eth_dev_configure(), it has been enabled on all queues, + * so there is no need to enable it in this queue again. + * The local_conf.offloads input to underlying PMD only carries + * those offloadings which are only enabled on this queue and + * not enabled on all queues. + */ + local_conf.offloads &= ~dev->data->dev_conf.txmode.offloads; + + /* + * New added offloadings for this queue are those not enabled in + * rte_eth_dev_configure() and they must be per-queue type. + * A pure per-port offloading can't be enabled on a queue while + * disabled on another queue. A pure per-port offloading can't + * be enabled for any queue as new added one if it hasn't been + * enabled in rte_eth_dev_configure(). + */ + if ((local_conf.offloads & dev_info.tx_queue_offload_capa) != + local_conf.offloads) { + RTE_ETHDEV_LOG(ERR, + "Ethdev port_id=%d tx_queue_id=%d, new added " + "offloads 0x%"PRIx64" must be within " + "per-queue offload capabilities 0x%"PRIx64" " + "in %s()\n", + port_id, tx_queue_id, local_conf.offloads, + dev_info.tx_queue_offload_capa, + __func__); + return -EINVAL; + } + + return eth_err(port_id, (*dev->dev_ops->tx_hairpin_queue_setup) + (dev, tx_queue_id, nb_tx_desc, socket_id, &local_conf, + hairpin_conf)); +} + void rte_eth_tx_buffer_drop_callback(struct rte_mbuf **pkts, uint16_t unsent, void *userdata __rte_unused) diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h index 475dbda..b3b1597 100644 --- a/lib/librte_ethdev/rte_ethdev.h +++ b/lib/librte_ethdev/rte_ethdev.h @@ -803,6 +803,30 @@ struct rte_eth_txconf { uint64_t offloads; }; +#define RTE_ETH_MAX_HAIRPIN_PEERS 32 + +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * A structure used to hold hairpin peer data. + */ +struct rte_eth_hairpin_peer { + uint16_t port; /**< Peer port. */ + uint16_t queue; /**< Peer queue. */ +}; + +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * A structure used to configure hairpin binding. + */ +struct rte_eth_hairpin_conf { + uint16_t peer_n; /**< The number of peers. */ + struct rte_eth_hairpin_peer peers[RTE_ETH_MAX_HAIRPIN_PEERS]; +}; + /** * A structure contains information about HW descriptor ring limitations. */ @@ -1769,6 +1793,60 @@ int rte_eth_rx_queue_setup(uint16_t port_id, uint16_t rx_queue_id, struct rte_mempool *mb_pool); /** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Allocate and set up a hairpin receive queue for an Ethernet device. + * + * The function set up the selected queue to be used in hairpin. + * + * @param port_id + * The port identifier of the Ethernet device. + * @param rx_queue_id + * The index of the receive queue to set up. + * The value must be in the range [0, nb_rx_queue - 1] previously supplied + * to rte_eth_dev_configure(). + * @param nb_rx_desc + * The number of receive descriptors to allocate for the receive ring. + * @param socket_id + * The *socket_id* argument is the socket identifier in case of NUMA. + * The value can be *SOCKET_ID_ANY* if there is no NUMA constraint for + * the DMA memory allocated for the receive descriptors of the ring. + * @param rx_conf + * The pointer to the configuration data to be used for the receive queue. + * NULL value is allowed, in which case default RX configuration + * will be used. + * The *rx_conf* structure contains an *rx_thresh* structure with the values + * of the Prefetch, Host, and Write-Back threshold registers of the receive + * ring. + * In addition it contains the hardware offloads features to activate using + * the DEV_RX_OFFLOAD_* flags. + * If an offloading set in rx_conf->offloads + * hasn't been set in the input argument eth_conf->rxmode.offloads + * to rte_eth_dev_configure(), it is a new added offloading, it must be + * per-queue type and it is enabled for the queue. + * No need to repeat any bit in rx_conf->offloads which has already been + * enabled in rte_eth_dev_configure() at port level. An offloading enabled + * at port level can't be disabled at queue level. + * @param hairpin_conf + * The pointer to the hairpin binding configuration. + * @return + * - 0: Success, receive queue correctly set up. + * - -EINVAL: The size of network buffers which can be allocated from the + * memory pool does not fit the various buffer sizes allowed by the + * device controller. + * - -ENOMEM: Unable to allocate the receive ring descriptors or to + * allocate network memory buffers from the memory pool when + * initializing receive descriptors. + */ +__rte_experimental +int rte_eth_rx_hairpin_queue_setup + (uint16_t port_id, uint16_t rx_queue_id, + uint16_t nb_rx_desc, unsigned int socket_id, + const struct rte_eth_rxconf *rx_conf, + const struct rte_eth_hairpin_conf *hairpin_conf); + +/** * Allocate and set up a transmit queue for an Ethernet device. * * @param port_id @@ -1821,6 +1899,73 @@ int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id, const struct rte_eth_txconf *tx_conf); /** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Allocate and set up a transmit hairpin queue for an Ethernet device. + * + * @param port_id + * The port identifier of the Ethernet device. + * @param tx_queue_id + * The index of the transmit queue to set up. + * The value must be in the range [0, nb_tx_queue - 1] previously supplied + * to rte_eth_dev_configure(). + * @param nb_tx_desc + * The number of transmit descriptors to allocate for the transmit ring. + * @param socket_id + * The *socket_id* argument is the socket identifier in case of NUMA. + * Its value can be *SOCKET_ID_ANY* if there is no NUMA constraint for + * the DMA memory allocated for the transmit descriptors of the ring. + * @param tx_conf + * The pointer to the configuration data to be used for the transmit queue. + * NULL value is allowed, in which case default RX configuration + * will be used. + * The *tx_conf* structure contains the following data: + * - The *tx_thresh* structure with the values of the Prefetch, Host, and + * Write-Back threshold registers of the transmit ring. + * When setting Write-Back threshold to the value greater then zero, + * *tx_rs_thresh* value should be explicitly set to one. + * - The *tx_free_thresh* value indicates the [minimum] number of network + * buffers that must be pending in the transmit ring to trigger their + * [implicit] freeing by the driver transmit function. + * - The *tx_rs_thresh* value indicates the [minimum] number of transmit + * descriptors that must be pending in the transmit ring before setting the + * RS bit on a descriptor by the driver transmit function. + * The *tx_rs_thresh* value should be less or equal then + * *tx_free_thresh* value, and both of them should be less then + * *nb_tx_desc* - 3. + * - The *txq_flags* member contains flags to pass to the TX queue setup + * function to configure the behavior of the TX queue. This should be set + * to 0 if no special configuration is required. + * This API is obsolete and will be deprecated. Applications + * should set it to ETH_TXQ_FLAGS_IGNORE and use + * the offloads field below. + * - The *offloads* member contains Tx offloads to be enabled. + * If an offloading set in tx_conf->offloads + * hasn't been set in the input argument eth_conf->txmode.offloads + * to rte_eth_dev_configure(), it is a new added offloading, it must be + * per-queue type and it is enabled for the queue. + * No need to repeat any bit in tx_conf->offloads which has already been + * enabled in rte_eth_dev_configure() at port level. An offloading enabled + * at port level can't be disabled at queue level. + * + * Note that setting *tx_free_thresh* or *tx_rs_thresh* value to 0 forces + * the transmit function to use default values. + * @param hairpin_conf + * The hairpin binding configuration. + * + * @return + * - 0: Success, the transmit queue is correctly set up. + * - -ENOMEM: Unable to allocate the transmit ring descriptors. + */ +__rte_experimental +int rte_eth_tx_hairpin_queue_setup + (uint16_t port_id, uint16_t tx_queue_id, + uint16_t nb_tx_desc, unsigned int socket_id, + const struct rte_eth_txconf *tx_conf, + const struct rte_eth_hairpin_conf *hairpin_conf); + +/** * Return the NUMA socket to which an Ethernet device is connected * * @param port_id diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h index 2394b32..bc40708 100644 --- a/lib/librte_ethdev/rte_ethdev_core.h +++ b/lib/librte_ethdev/rte_ethdev_core.h @@ -126,6 +126,13 @@ typedef int (*eth_rx_queue_setup_t)(struct rte_eth_dev *dev, struct rte_mempool *mb_pool); /**< @internal Set up a receive queue of an Ethernet device. */ +typedef int (*eth_rx_hairpin_queue_setup_t) + (struct rte_eth_dev *dev, uint16_t rx_queue_id, + uint16_t nb_rx_desc, unsigned int socket_id, + const struct rte_eth_rxconf *rx_conf, + const struct rte_eth_hairpin_conf *hairpin_conf); +/**< @internal Set up a receive hairpin queue of an Ethernet device. */ + typedef int (*eth_tx_queue_setup_t)(struct rte_eth_dev *dev, uint16_t tx_queue_id, uint16_t nb_tx_desc, @@ -133,6 +140,13 @@ typedef int (*eth_tx_queue_setup_t)(struct rte_eth_dev *dev, const struct rte_eth_txconf *tx_conf); /**< @internal Setup a transmit queue of an Ethernet device. */ +typedef int (*eth_tx_hairpin_queue_setup_t) + (struct rte_eth_dev *dev, uint16_t tx_queue_id, + uint16_t nb_tx_desc, unsigned int socket_id, + const struct rte_eth_txconf *tx_conf, + const struct rte_eth_hairpin_conf *hairpin_conf); +/**< @internal Setup a transmit hairpin queue of an Ethernet device. */ + typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev, uint16_t rx_queue_id); /**< @internal Enable interrupt of a receive queue of an Ethernet device. */ @@ -433,6 +447,8 @@ struct eth_dev_ops { eth_queue_start_t tx_queue_start;/**< Start TX for a queue. */ eth_queue_stop_t tx_queue_stop; /**< Stop TX for a queue. */ eth_rx_queue_setup_t rx_queue_setup;/**< Set up device RX queue. */ + eth_rx_hairpin_queue_setup_t rx_hairpin_queue_setup; + /**< Set up device RX hairpin queue. */ eth_queue_release_t rx_queue_release; /**< Release RX queue. */ eth_rx_queue_count_t rx_queue_count; /**< Get the number of used RX descriptors. */ @@ -444,6 +460,8 @@ struct eth_dev_ops { eth_rx_enable_intr_t rx_queue_intr_enable; /**< Enable Rx queue interrupt. */ eth_rx_disable_intr_t rx_queue_intr_disable; /**< Disable Rx queue interrupt. */ eth_tx_queue_setup_t tx_queue_setup;/**< Set up device TX queue. */ + eth_tx_hairpin_queue_setup_t tx_hairpin_queue_setup; + /**< Set up device TX hairpin queue. */ eth_queue_release_t tx_queue_release; /**< Release TX queue. */ eth_tx_done_cleanup_t tx_done_cleanup;/**< Free tx ring mbufs */ diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map index 6df42a4..99e05fe 100644 --- a/lib/librte_ethdev/rte_ethdev_version.map +++ b/lib/librte_ethdev/rte_ethdev_version.map @@ -283,4 +283,8 @@ EXPERIMENTAL { # added in 19.08 rte_eth_read_clock; + + # added in 19.11 + rte_eth_rx_hairpin_queue_setup; + rte_eth_tx_hairpin_queue_setup; }; From patchwork Thu Sep 26 06:28:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59782 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 46601397D; Thu, 26 Sep 2019 08:30:01 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 12F08397D for ; Thu, 26 Sep 2019 08:30:00 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:29:55 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlH012493; Thu, 26 Sep 2019 09:29:55 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:28:58 +0000 Message-Id: <1569479349-36962-3-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 02/13] net/mlx5: query hca hairpin capabilities X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit query and store the hairpin capabilities from the device. Those capabilities will be used when creating the hairpin queue. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.h | 4 ++++ drivers/net/mlx5/mlx5_devx_cmds.c | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index d4d2ca8..cd896c8 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -184,6 +184,10 @@ struct mlx5_hca_attr { uint32_t tunnel_lro_vxlan:1; uint32_t lro_max_msg_sz_mode:2; uint32_t lro_timer_supported_periods[MLX5_LRO_NUM_SUPP_PERIODS]; + uint32_t hairpin:1; + uint32_t log_max_hairpin_queues:5; + uint32_t log_max_hairpin_wq_data_sz:5; + uint32_t log_max_hairpin_num_packets:5; }; /* Flow list . */ diff --git a/drivers/net/mlx5/mlx5_devx_cmds.c b/drivers/net/mlx5/mlx5_devx_cmds.c index acfe1de..b072c37 100644 --- a/drivers/net/mlx5/mlx5_devx_cmds.c +++ b/drivers/net/mlx5/mlx5_devx_cmds.c @@ -327,6 +327,13 @@ struct mlx5_devx_obj * attr->flow_counters_dump = MLX5_GET(cmd_hca_cap, hcattr, flow_counters_dump); attr->eswitch_manager = MLX5_GET(cmd_hca_cap, hcattr, eswitch_manager); + attr->hairpin = MLX5_GET(cmd_hca_cap, hcattr, hairpin); + attr->log_max_hairpin_queues = MLX5_GET(cmd_hca_cap, hcattr, + log_max_hairpin_queues); + attr->log_max_hairpin_wq_data_sz = MLX5_GET(cmd_hca_cap, hcattr, + log_max_hairpin_wq_data_sz); + attr->log_max_hairpin_num_packets = MLX5_GET + (cmd_hca_cap, hcattr, log_min_hairpin_wq_data_sz); attr->eth_net_offloads = MLX5_GET(cmd_hca_cap, hcattr, eth_net_offloads); attr->eth_virt = MLX5_GET(cmd_hca_cap, hcattr, eth_virt); From patchwork Thu Sep 26 06:28:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59784 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E25E81BE86; Thu, 26 Sep 2019 08:30:09 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 27CED4C90 for ; Thu, 26 Sep 2019 08:30:00 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:29:56 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlI012493; Thu, 26 Sep 2019 09:29:56 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:28:59 +0000 Message-Id: <1569479349-36962-4-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 03/13] net/mlx5: support Rx hairpin queues X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit adds the support for creating Rx hairpin queues. Hairpin queue is a queue that is created using DevX and only used by the HW. This results in that all the data part of the RQ is not being used. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 2 + drivers/net/mlx5/mlx5_rxq.c | 286 ++++++++++++++++++++++++++++++++++++---- drivers/net/mlx5/mlx5_rxtx.h | 17 +++ drivers/net/mlx5/mlx5_trigger.c | 7 + 4 files changed, 288 insertions(+), 24 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index be01db9..81894fb 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -974,6 +974,7 @@ struct mlx5_dev_spawn_data { .dev_supported_ptypes_get = mlx5_dev_supported_ptypes_get, .vlan_filter_set = mlx5_vlan_filter_set, .rx_queue_setup = mlx5_rx_queue_setup, + .rx_hairpin_queue_setup = mlx5_rx_hairpin_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, @@ -1040,6 +1041,7 @@ struct mlx5_dev_spawn_data { .dev_supported_ptypes_get = mlx5_dev_supported_ptypes_get, .vlan_filter_set = mlx5_vlan_filter_set, .rx_queue_setup = mlx5_rx_queue_setup, + .rx_hairpin_queue_setup = mlx5_rx_hairpin_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index a1fdeef..a673da9 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -106,21 +106,25 @@ struct mlx5_priv *priv = dev->data->dev_private; uint16_t i; uint16_t n = 0; + uint16_t n_ibv = 0; if (mlx5_check_mprq_support(dev) < 0) return 0; /* All the configured queues should be enabled. */ for (i = 0; i < priv->rxqs_n; ++i) { struct mlx5_rxq_data *rxq = (*priv->rxqs)[i]; + struct mlx5_rxq_ctrl *rxq_ctrl = container_of + (rxq, struct mlx5_rxq_ctrl, rxq); - if (!rxq) + if (rxq == NULL || rxq_ctrl->type != MLX5_RXQ_TYPE_STANDARD) continue; + n_ibv++; if (mlx5_rxq_mprq_enabled(rxq)) ++n; } /* Multi-Packet RQ can't be partially configured. */ - assert(n == 0 || n == priv->rxqs_n); - return n == priv->rxqs_n; + assert(n == 0 || n == n_ibv); + return n == n_ibv; } /** @@ -427,6 +431,7 @@ } /** + * Rx queue presetup checks. * * @param dev * Pointer to Ethernet device structure. @@ -434,25 +439,14 @@ * RX queue index. * @param desc * Number of descriptors to configure in queue. - * @param socket - * NUMA socket on which memory must be allocated. - * @param[in] conf - * Thresholds parameters. - * @param mp - * Memory pool for buffer allocations. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, - unsigned int socket, const struct rte_eth_rxconf *conf, - struct rte_mempool *mp) +static int +mlx5_rx_queue_pre_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc) { struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_rxq_data *rxq = (*priv->rxqs)[idx]; - struct mlx5_rxq_ctrl *rxq_ctrl = - container_of(rxq, struct mlx5_rxq_ctrl, rxq); if (!rte_is_power_of_2(desc)) { desc = 1 << log2above(desc); @@ -476,6 +470,41 @@ return -rte_errno; } mlx5_rxq_release(dev, idx); + return 0; +} + +/** + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * RX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * @param mp + * Memory pool for buffer allocations. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_rxconf *conf, + struct rte_mempool *mp) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_data *rxq = (*priv->rxqs)[idx]; + struct mlx5_rxq_ctrl *rxq_ctrl = + container_of(rxq, struct mlx5_rxq_ctrl, rxq); + int res; + + res = mlx5_rx_queue_pre_setup(dev, idx, desc); + if (res) + return res; rxq_ctrl = mlx5_rxq_new(dev, idx, desc, socket, conf, mp); if (!rxq_ctrl) { DRV_LOG(ERR, "port %u unable to allocate queue index %u", @@ -490,6 +519,62 @@ } /** + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * RX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * @param hairpin_conf + * Hairpin configuration parameters. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_rx_hairpin_queue_setup(struct rte_eth_dev *dev, uint16_t idx, + uint16_t desc, unsigned int socket, + const struct rte_eth_rxconf *conf, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_data *rxq = (*priv->rxqs)[idx]; + struct mlx5_rxq_ctrl *rxq_ctrl = + container_of(rxq, struct mlx5_rxq_ctrl, rxq); + int res; + + res = mlx5_rx_queue_pre_setup(dev, idx, desc); + if (res) + return res; + if (hairpin_conf->peer_n != 1 || + hairpin_conf->peers[0].port != dev->data->port_id || + hairpin_conf->peers[0].queue >= priv->txqs_n) { + DRV_LOG(ERR, "port %u unable to setup hairpin queue index %u " + " invalid hairpind configuration", dev->data->port_id, + idx); + rte_errno = EINVAL; + return -rte_errno; + } + rxq_ctrl = mlx5_rxq_hairpin_new(dev, idx, desc, socket, conf, + hairpin_conf); + if (!rxq_ctrl) { + DRV_LOG(ERR, "port %u unable to allocate queue index %u", + dev->data->port_id, idx); + rte_errno = ENOMEM; + return -rte_errno; + } + DRV_LOG(DEBUG, "port %u adding Rx queue %u to list", + dev->data->port_id, idx); + (*priv->rxqs)[idx] = &rxq_ctrl->rxq; + return 0; +} + +/** * DPDK callback to release a RX queue. * * @param dpdk_rxq @@ -561,6 +646,24 @@ } /** + * Release an Rx hairpin related resources. + * + * @param rxq_obj + * Hairpin Rx queue object. + */ +static void +rxq_obj_hairpin_release(struct mlx5_rxq_obj *rxq_obj) +{ + struct mlx5_devx_modify_rq_attr rq_attr = { 0 }; + + assert(rxq_obj); + rq_attr.state = MLX5_RQC_STATE_RST; + rq_attr.rq_state = MLX5_RQC_STATE_RDY; + mlx5_devx_cmd_modify_rq(rxq_obj->rq, &rq_attr); + claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); +} + +/** * Release an Rx verbs/DevX queue object. * * @param rxq_obj @@ -577,14 +680,22 @@ assert(rxq_obj->wq); assert(rxq_obj->cq); if (rte_atomic32_dec_and_test(&rxq_obj->refcnt)) { - rxq_free_elts(rxq_obj->rxq_ctrl); - if (rxq_obj->type == MLX5_RXQ_OBJ_TYPE_IBV) { + switch (rxq_obj->type) { + case MLX5_RXQ_OBJ_TYPE_IBV: + rxq_free_elts(rxq_obj->rxq_ctrl); claim_zero(mlx5_glue->destroy_wq(rxq_obj->wq)); - } else if (rxq_obj->type == MLX5_RXQ_OBJ_TYPE_DEVX_RQ) { + claim_zero(mlx5_glue->destroy_cq(rxq_obj->cq)); + break; + case MLX5_RXQ_OBJ_TYPE_DEVX_RQ: + rxq_free_elts(rxq_obj->rxq_ctrl); claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); rxq_release_rq_resources(rxq_obj->rxq_ctrl); + claim_zero(mlx5_glue->destroy_cq(rxq_obj->cq)); + break; + case MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN: + rxq_obj_hairpin_release(rxq_obj); + break; } - claim_zero(mlx5_glue->destroy_cq(rxq_obj->cq)); if (rxq_obj->channel) claim_zero(mlx5_glue->destroy_comp_channel (rxq_obj->channel)); @@ -1132,6 +1243,70 @@ } /** + * Create the Rx hairpin queue object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Rx queue array + * + * @return + * The hairpin DevX object initialised, NULL otherwise and rte_errno is set. + */ +static struct mlx5_rxq_obj * +mlx5_rxq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[idx]; + struct mlx5_rxq_ctrl *rxq_ctrl = + container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); + struct mlx5_devx_create_rq_attr attr = { 0 }; + struct mlx5_rxq_obj *tmpl = NULL; + int ret = 0; + + assert(rxq_data); + assert(!rxq_ctrl->obj); + tmpl = rte_calloc_socket(__func__, 1, sizeof(*tmpl), 0, + rxq_ctrl->socket); + if (!tmpl) { + DRV_LOG(ERR, + "port %u Rx queue %u cannot allocate verbs resources", + dev->data->port_id, rxq_data->idx); + rte_errno = ENOMEM; + goto error; + } + tmpl->type = MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN; + tmpl->rxq_ctrl = rxq_ctrl; + attr.hairpin = 1; + /* Workaround for hairpin startup */ + attr.wq_attr.log_hairpin_num_packets = log2above(32); + /* Workaround for packets larger than 1KB */ + attr.wq_attr.log_hairpin_data_sz = + priv->config.hca_attr.log_max_hairpin_wq_data_sz; + tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->ctx, &attr, + rxq_ctrl->socket); + if (!tmpl->rq) { + DRV_LOG(ERR, + "port %u Rx hairpin queue %u can't create rq object", + dev->data->port_id, idx); + rte_errno = errno; + goto error; + } + DRV_LOG(DEBUG, "port %u rxq %u updated with %p", dev->data->port_id, + idx, (void *)&tmpl); + rte_atomic32_inc(&tmpl->refcnt); + LIST_INSERT_HEAD(&priv->rxqsobj, tmpl, next); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; + return tmpl; +error: + ret = rte_errno; /* Save rte_errno before cleanup. */ + if (tmpl->rq) + mlx5_devx_cmd_destroy(tmpl->rq); + rte_errno = ret; /* Restore rte_errno. */ + return NULL; +} + +/** * Create the Rx queue Verbs/DevX object. * * @param dev @@ -1163,6 +1338,8 @@ struct mlx5_rxq_obj * assert(rxq_data); assert(!rxq_ctrl->obj); + if (type == MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN) + return mlx5_rxq_obj_hairpin_new(dev, idx); priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_RX_QUEUE; priv->verbs_alloc_ctx.obj = rxq_ctrl; tmpl = rte_calloc_socket(__func__, 1, sizeof(*tmpl), 0, @@ -1433,15 +1610,19 @@ struct mlx5_rxq_obj * unsigned int strd_num_n = 0; unsigned int strd_sz_n = 0; unsigned int i; + unsigned int n_ibv = 0; if (!mlx5_mprq_enabled(dev)) return 0; /* Count the total number of descriptors configured. */ for (i = 0; i != priv->rxqs_n; ++i) { struct mlx5_rxq_data *rxq = (*priv->rxqs)[i]; + struct mlx5_rxq_ctrl *rxq_ctrl = container_of + (rxq, struct mlx5_rxq_ctrl, rxq); - if (rxq == NULL) + if (rxq == NULL || rxq_ctrl->type != MLX5_RXQ_TYPE_STANDARD) continue; + n_ibv++; desc += 1 << rxq->elts_n; /* Get the max number of strides. */ if (strd_num_n < rxq->strd_num_n) @@ -1466,7 +1647,7 @@ struct mlx5_rxq_obj * * this Mempool gets available again. */ desc *= 4; - obj_num = desc + MLX5_MPRQ_MP_CACHE_SZ * priv->rxqs_n; + obj_num = desc + MLX5_MPRQ_MP_CACHE_SZ * n_ibv; /* * rte_mempool_create_empty() has sanity check to refuse large cache * size compared to the number of elements. @@ -1514,8 +1695,10 @@ struct mlx5_rxq_obj * /* Set mempool for each Rx queue. */ for (i = 0; i != priv->rxqs_n; ++i) { struct mlx5_rxq_data *rxq = (*priv->rxqs)[i]; + struct mlx5_rxq_ctrl *rxq_ctrl = container_of + (rxq, struct mlx5_rxq_ctrl, rxq); - if (rxq == NULL) + if (rxq == NULL || rxq_ctrl->type != MLX5_RXQ_TYPE_STANDARD) continue; rxq->mprq_mp = mp; } @@ -1620,6 +1803,7 @@ struct mlx5_rxq_ctrl * rte_errno = ENOMEM; return NULL; } + tmpl->type = MLX5_RXQ_TYPE_STANDARD; if (mlx5_mr_btree_init(&tmpl->rxq.mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N, socket)) { /* rte_errno is already set. */ @@ -1788,6 +1972,59 @@ struct mlx5_rxq_ctrl * } /** + * Create a DPDK Rx hairpin queue. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * RX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param conf + * The Rx configuration. + * @param hairpin_conf + * The hairpin binding configuration. + * + * @return + * A DPDK queue object on success, NULL otherwise and rte_errno is set. + */ +struct mlx5_rxq_ctrl * +mlx5_rxq_hairpin_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_rxconf *conf, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_ctrl *tmpl; + uint64_t offloads = conf->offloads | + dev->data->dev_conf.rxmode.offloads; + + tmpl = rte_calloc_socket("RXQ", 1, sizeof(*tmpl), 0, socket); + if (!tmpl) { + rte_errno = ENOMEM; + return NULL; + } + tmpl->type = MLX5_RXQ_TYPE_HAIRPIN; + tmpl->socket = socket; + /* Configure VLAN stripping. */ + tmpl->rxq.vlan_strip = !!(offloads & DEV_RX_OFFLOAD_VLAN_STRIP); + /* Save port ID. */ + tmpl->rxq.rss_hash = 0; + tmpl->rxq.port_id = dev->data->port_id; + tmpl->priv = priv; + tmpl->rxq.mp = NULL; + tmpl->rxq.elts_n = log2above(desc); + tmpl->rxq.elts = NULL; + tmpl->rxq.mr_ctrl.cache_bh = (struct mlx5_mr_btree) { 0 }; + tmpl->hairpin_conf = *hairpin_conf; + tmpl->rxq.idx = idx; + rte_atomic32_inc(&tmpl->refcnt); + LIST_INSERT_HEAD(&priv->rxqsctrl, tmpl, next); + return tmpl; +} + +/** * Get a Rx queue. * * @param dev @@ -1841,7 +2078,8 @@ struct mlx5_rxq_ctrl * if (rxq_ctrl->dbr_umem_id_valid) claim_zero(mlx5_release_dbr(dev, rxq_ctrl->dbr_umem_id, rxq_ctrl->dbr_offset)); - mlx5_mr_btree_free(&rxq_ctrl->rxq.mr_ctrl.cache_bh); + if (rxq_ctrl->type == MLX5_RXQ_TYPE_STANDARD) + mlx5_mr_btree_free(&rxq_ctrl->rxq.mr_ctrl.cache_bh); LIST_REMOVE(rxq_ctrl, next); rte_free(rxq_ctrl); (*priv->rxqs)[idx] = NULL; diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 4bb28a4..dbb616e 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -159,6 +159,13 @@ struct mlx5_rxq_data { enum mlx5_rxq_obj_type { MLX5_RXQ_OBJ_TYPE_IBV, /* mlx5_rxq_obj with ibv_wq. */ MLX5_RXQ_OBJ_TYPE_DEVX_RQ, /* mlx5_rxq_obj with mlx5_devx_rq. */ + MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN, + /* mlx5_rxq_obj with mlx5_devx_rq and hairpin support. */ +}; + +enum mlx5_rxq_type { + MLX5_RXQ_TYPE_STANDARD, /* Standard Rx queue. */ + MLX5_RXQ_TYPE_HAIRPIN, /* Hairpin Rx queue. */ }; /* Verbs/DevX Rx queue elements. */ @@ -183,6 +190,7 @@ struct mlx5_rxq_ctrl { rte_atomic32_t refcnt; /* Reference counter. */ struct mlx5_rxq_obj *obj; /* Verbs/DevX elements. */ struct mlx5_priv *priv; /* Back pointer to private data. */ + enum mlx5_rxq_type type; /* Rxq type. */ unsigned int socket; /* CPU socket ID for allocations. */ unsigned int irq:1; /* Whether IRQ is enabled. */ unsigned int dbr_umem_id_valid:1; /* dbr_umem_id holds a valid value. */ @@ -193,6 +201,7 @@ struct mlx5_rxq_ctrl { uint32_t dbr_umem_id; /* Storing door-bell information, */ uint64_t dbr_offset; /* needed when freeing door-bell. */ struct mlx5dv_devx_umem *wq_umem; /* WQ buffer registration info. */ + struct rte_eth_hairpin_conf hairpin_conf; /* Hairpin configuration. */ }; enum mlx5_ind_tbl_type { @@ -339,6 +348,10 @@ struct mlx5_txq_ctrl { int mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_rxconf *conf, struct rte_mempool *mp); +int mlx5_rx_hairpin_queue_setup + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_rxconf *conf, + const struct rte_eth_hairpin_conf *hairpin_conf); void mlx5_rx_queue_release(void *dpdk_rxq); int mlx5_rx_intr_vec_enable(struct rte_eth_dev *dev); void mlx5_rx_intr_vec_disable(struct rte_eth_dev *dev); @@ -351,6 +364,10 @@ struct mlx5_rxq_ctrl *mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_rxconf *conf, struct rte_mempool *mp); +struct mlx5_rxq_ctrl *mlx5_rxq_hairpin_new + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_rxconf *conf, + const struct rte_eth_hairpin_conf *hairpin_conf); struct mlx5_rxq_ctrl *mlx5_rxq_get(struct rte_eth_dev *dev, uint16_t idx); int mlx5_rxq_release(struct rte_eth_dev *dev, uint16_t idx); int mlx5_rxq_verify(struct rte_eth_dev *dev); diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 122f31c..cb31ae2 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -118,6 +118,13 @@ if (!rxq_ctrl) continue; + if (rxq_ctrl->type == MLX5_RXQ_TYPE_HAIRPIN) { + rxq_ctrl->obj = mlx5_rxq_obj_new + (dev, i, MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN); + if (!rxq_ctrl->obj) + goto error; + continue; + } /* Pre-register Rx mempool. */ mp = mlx5_rxq_mprq_enabled(&rxq_ctrl->rxq) ? rxq_ctrl->rxq.mprq_mp : rxq_ctrl->rxq.mp; From patchwork Thu Sep 26 06:29:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59783 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0610B1B994; Thu, 26 Sep 2019 08:30:07 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 2CC7C4C99 for ; Thu, 26 Sep 2019 08:30:00 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:29:58 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlJ012493; Thu, 26 Sep 2019 09:29:58 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:00 +0000 Message-Id: <1569479349-36962-5-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 04/13] net/mlx5: prepare txq to work with different types X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Currenlty all Tx queues are created using Verbs. This commit modify the naming so it will not include verbs, since in next commit a new type will be introduce (hairpin) Signed-off-by: Ori Kam Conflicts: drivers/net/mlx5/mlx5_txq.c Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 2 +- drivers/net/mlx5/mlx5.h | 2 +- drivers/net/mlx5/mlx5_rxtx.c | 2 +- drivers/net/mlx5/mlx5_rxtx.h | 39 +++++++++++++++++------ drivers/net/mlx5/mlx5_trigger.c | 4 +-- drivers/net/mlx5/mlx5_txq.c | 70 ++++++++++++++++++++--------------------- 6 files changed, 69 insertions(+), 50 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 81894fb..f0d122d 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -911,7 +911,7 @@ struct mlx5_dev_spawn_data { if (ret) DRV_LOG(WARNING, "port %u some Rx queues still remain", dev->data->port_id); - ret = mlx5_txq_ibv_verify(dev); + ret = mlx5_txq_obj_verify(dev); if (ret) DRV_LOG(WARNING, "port %u some Verbs Tx queue still remain", dev->data->port_id); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index cd896c8..a34972c 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -645,7 +645,7 @@ struct mlx5_priv { LIST_HEAD(rxqobj, mlx5_rxq_obj) rxqsobj; /* Verbs/DevX Rx queues. */ LIST_HEAD(hrxq, mlx5_hrxq) hrxqs; /* Verbs Hash Rx queues. */ LIST_HEAD(txq, mlx5_txq_ctrl) txqsctrl; /* DPDK Tx queues. */ - LIST_HEAD(txqibv, mlx5_txq_ibv) txqsibv; /* Verbs Tx queues. */ + LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */ /* Indirection tables. */ LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls; /* Pointer to next element. */ diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 10d0ca1..f23708c 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -863,7 +863,7 @@ enum mlx5_txcmp_code { .qp_state = IBV_QPS_RESET, .port_num = (uint8_t)priv->ibv_port, }; - struct ibv_qp *qp = txq_ctrl->ibv->qp; + struct ibv_qp *qp = txq_ctrl->obj->qp; ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); if (ret) { diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index dbb616e..2de674a 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -308,13 +308,31 @@ struct mlx5_txq_data { /* Storage for queued packets, must be the last field. */ } __rte_cache_aligned; -/* Verbs Rx queue elements. */ -struct mlx5_txq_ibv { - LIST_ENTRY(mlx5_txq_ibv) next; /* Pointer to the next element. */ +enum mlx5_txq_obj_type { + MLX5_TXQ_OBJ_TYPE_IBV, /* mlx5_txq_obj with ibv_wq. */ + MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN, + /* mlx5_txq_obj with mlx5_devx_tq and hairpin support. */ +}; + +enum mlx5_txq_type { + MLX5_TXQ_TYPE_STANDARD, /* Standard Tx queue. */ + MLX5_TXQ_TYPE_HAIRPIN, /* Hairpin Rx queue. */ +}; + +/* Verbs/DevX Tx queue elements. */ +struct mlx5_txq_obj { + LIST_ENTRY(mlx5_txq_obj) next; /* Pointer to the next element. */ rte_atomic32_t refcnt; /* Reference counter. */ struct mlx5_txq_ctrl *txq_ctrl; /* Pointer to the control queue. */ - struct ibv_cq *cq; /* Completion Queue. */ - struct ibv_qp *qp; /* Queue Pair. */ + enum mlx5_rxq_obj_type type; /* The txq object type. */ + RTE_STD_C11 + union { + struct { + struct ibv_cq *cq; /* Completion Queue. */ + struct ibv_qp *qp; /* Queue Pair. */ + }; + struct mlx5_devx_obj *sq; /* DevX object for Sx queue. */ + }; }; /* TX queue control descriptor. */ @@ -322,9 +340,10 @@ struct mlx5_txq_ctrl { LIST_ENTRY(mlx5_txq_ctrl) next; /* Pointer to the next element. */ rte_atomic32_t refcnt; /* Reference counter. */ unsigned int socket; /* CPU socket ID for allocations. */ + enum mlx5_txq_type type; /* The txq ctrl type. */ unsigned int max_inline_data; /* Max inline data. */ unsigned int max_tso_header; /* Max TSO header size. */ - struct mlx5_txq_ibv *ibv; /* Verbs queue object. */ + struct mlx5_txq_obj *obj; /* Verbs/DevX queue object. */ struct mlx5_priv *priv; /* Back pointer to private data. */ off_t uar_mmap_offset; /* UAR mmap offset for non-primary process. */ void *bf_reg; /* BlueFlame register from Verbs. */ @@ -395,10 +414,10 @@ int mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_txconf *conf); void mlx5_tx_queue_release(void *dpdk_txq); int mlx5_tx_uar_init_secondary(struct rte_eth_dev *dev, int fd); -struct mlx5_txq_ibv *mlx5_txq_ibv_new(struct rte_eth_dev *dev, uint16_t idx); -struct mlx5_txq_ibv *mlx5_txq_ibv_get(struct rte_eth_dev *dev, uint16_t idx); -int mlx5_txq_ibv_release(struct mlx5_txq_ibv *txq_ibv); -int mlx5_txq_ibv_verify(struct rte_eth_dev *dev); +struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx); +struct mlx5_txq_obj *mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t idx); +int mlx5_txq_obj_release(struct mlx5_txq_obj *txq_ibv); +int mlx5_txq_obj_verify(struct rte_eth_dev *dev); struct mlx5_txq_ctrl *mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_txconf *conf); diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index cb31ae2..50c4df5 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -52,8 +52,8 @@ if (!txq_ctrl) continue; txq_alloc_elts(txq_ctrl); - txq_ctrl->ibv = mlx5_txq_ibv_new(dev, i); - if (!txq_ctrl->ibv) { + txq_ctrl->obj = mlx5_txq_obj_new(dev, i); + if (!txq_ctrl->obj) { rte_errno = ENOMEM; goto error; } diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index d9fd143..e1ed4eb 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -375,15 +375,15 @@ * @return * The Verbs object initialised, NULL otherwise and rte_errno is set. */ -struct mlx5_txq_ibv * -mlx5_txq_ibv_new(struct rte_eth_dev *dev, uint16_t idx) +struct mlx5_txq_obj * +mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; struct mlx5_txq_ctrl *txq_ctrl = container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct mlx5_txq_ibv tmpl; - struct mlx5_txq_ibv *txq_ibv = NULL; + struct mlx5_txq_obj tmpl; + struct mlx5_txq_obj *txq_obj = NULL; union { struct ibv_qp_init_attr_ex init; struct ibv_cq_init_attr_ex cq; @@ -411,7 +411,7 @@ struct mlx5_txq_ibv * rte_errno = EINVAL; return NULL; } - memset(&tmpl, 0, sizeof(struct mlx5_txq_ibv)); + memset(&tmpl, 0, sizeof(struct mlx5_txq_obj)); attr.cq = (struct ibv_cq_init_attr_ex){ .comp_mask = 0, }; @@ -502,9 +502,9 @@ struct mlx5_txq_ibv * rte_errno = errno; goto error; } - txq_ibv = rte_calloc_socket(__func__, 1, sizeof(struct mlx5_txq_ibv), 0, + txq_obj = rte_calloc_socket(__func__, 1, sizeof(struct mlx5_txq_obj), 0, txq_ctrl->socket); - if (!txq_ibv) { + if (!txq_obj) { DRV_LOG(ERR, "port %u Tx queue %u cannot allocate memory", dev->data->port_id, idx); rte_errno = ENOMEM; @@ -568,9 +568,9 @@ struct mlx5_txq_ibv * } } #endif - txq_ibv->qp = tmpl.qp; - txq_ibv->cq = tmpl.cq; - rte_atomic32_inc(&txq_ibv->refcnt); + txq_obj->qp = tmpl.qp; + txq_obj->cq = tmpl.cq; + rte_atomic32_inc(&txq_obj->refcnt); txq_ctrl->bf_reg = qp.bf.reg; if (qp.comp_mask & MLX5DV_QP_MASK_UAR_MMAP_OFFSET) { txq_ctrl->uar_mmap_offset = qp.uar_mmap_offset; @@ -585,18 +585,18 @@ struct mlx5_txq_ibv * goto error; } txq_uar_init(txq_ctrl); - LIST_INSERT_HEAD(&priv->txqsibv, txq_ibv, next); - txq_ibv->txq_ctrl = txq_ctrl; + LIST_INSERT_HEAD(&priv->txqsobj, txq_obj, next); + txq_obj->txq_ctrl = txq_ctrl; priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; - return txq_ibv; + return txq_obj; error: ret = rte_errno; /* Save rte_errno before cleanup. */ if (tmpl.cq) claim_zero(mlx5_glue->destroy_cq(tmpl.cq)); if (tmpl.qp) claim_zero(mlx5_glue->destroy_qp(tmpl.qp)); - if (txq_ibv) - rte_free(txq_ibv); + if (txq_obj) + rte_free(txq_obj); priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; rte_errno = ret; /* Restore rte_errno. */ return NULL; @@ -613,8 +613,8 @@ struct mlx5_txq_ibv * * @return * The Verbs object if it exists. */ -struct mlx5_txq_ibv * -mlx5_txq_ibv_get(struct rte_eth_dev *dev, uint16_t idx) +struct mlx5_txq_obj * +mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_ctrl *txq_ctrl; @@ -624,29 +624,29 @@ struct mlx5_txq_ibv * if (!(*priv->txqs)[idx]) return NULL; txq_ctrl = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); - if (txq_ctrl->ibv) - rte_atomic32_inc(&txq_ctrl->ibv->refcnt); - return txq_ctrl->ibv; + if (txq_ctrl->obj) + rte_atomic32_inc(&txq_ctrl->obj->refcnt); + return txq_ctrl->obj; } /** * Release an Tx verbs queue object. * - * @param txq_ibv + * @param txq_obj * Verbs Tx queue object. * * @return * 1 while a reference on it exists, 0 when freed. */ int -mlx5_txq_ibv_release(struct mlx5_txq_ibv *txq_ibv) +mlx5_txq_obj_release(struct mlx5_txq_obj *txq_obj) { - assert(txq_ibv); - if (rte_atomic32_dec_and_test(&txq_ibv->refcnt)) { - claim_zero(mlx5_glue->destroy_qp(txq_ibv->qp)); - claim_zero(mlx5_glue->destroy_cq(txq_ibv->cq)); - LIST_REMOVE(txq_ibv, next); - rte_free(txq_ibv); + assert(txq_obj); + if (rte_atomic32_dec_and_test(&txq_obj->refcnt)) { + claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); + claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); + LIST_REMOVE(txq_obj, next); + rte_free(txq_obj); return 0; } return 1; @@ -662,15 +662,15 @@ struct mlx5_txq_ibv * * The number of object not released. */ int -mlx5_txq_ibv_verify(struct rte_eth_dev *dev) +mlx5_txq_obj_verify(struct rte_eth_dev *dev) { struct mlx5_priv *priv = dev->data->dev_private; int ret = 0; - struct mlx5_txq_ibv *txq_ibv; + struct mlx5_txq_obj *txq_obj; - LIST_FOREACH(txq_ibv, &priv->txqsibv, next) { + LIST_FOREACH(txq_obj, &priv->txqsobj, next) { DRV_LOG(DEBUG, "port %u Verbs Tx queue %u still referenced", - dev->data->port_id, txq_ibv->txq_ctrl->txq.idx); + dev->data->port_id, txq_obj->txq_ctrl->txq.idx); ++ret; } return ret; @@ -980,7 +980,7 @@ struct mlx5_txq_ctrl * if ((*priv->txqs)[idx]) { ctrl = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); - mlx5_txq_ibv_get(dev, idx); + mlx5_txq_obj_get(dev, idx); rte_atomic32_inc(&ctrl->refcnt); } return ctrl; @@ -1006,8 +1006,8 @@ struct mlx5_txq_ctrl * if (!(*priv->txqs)[idx]) return 0; txq = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); - if (txq->ibv && !mlx5_txq_ibv_release(txq->ibv)) - txq->ibv = NULL; + if (txq->obj && !mlx5_txq_obj_release(txq->obj)) + txq->obj = NULL; if (rte_atomic32_dec_and_test(&txq->refcnt)) { txq_free_elts(txq); mlx5_mr_btree_free(&txq->txq.mr_ctrl.cache_bh); From patchwork Thu Sep 26 06:29:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59785 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EAEB41BE93; Thu, 26 Sep 2019 08:30:13 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 2616D1B94D for ; Thu, 26 Sep 2019 08:30:05 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:29:59 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlK012493; Thu, 26 Sep 2019 09:29:59 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:01 +0000 Message-Id: <1569479349-36962-6-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 05/13] net/mlx5: support Tx hairpin queues X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit adds the support for creating Tx hairpin queues. Hairpin queue is a queue that is created using DevX and only used by the HW. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 26 ++++ drivers/net/mlx5/mlx5.h | 46 +++++++ drivers/net/mlx5/mlx5_devx_cmds.c | 186 +++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_prm.h | 118 ++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 20 +++- drivers/net/mlx5/mlx5_trigger.c | 10 +- drivers/net/mlx5/mlx5_txq.c | 245 +++++++++++++++++++++++++++++++++++--- 7 files changed, 631 insertions(+), 20 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index f0d122d..ad36743 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -325,6 +325,9 @@ struct mlx5_dev_spawn_data { struct mlx5_ibv_shared *sh; int err = 0; uint32_t i; +#ifdef HAVE_IBV_FLOW_DV_SUPPORT + struct mlx5_devx_tis_attr tis_attr = { 0 }; +#endif assert(spawn); /* Secondary process should not create the shared context. */ @@ -394,6 +397,19 @@ struct mlx5_dev_spawn_data { DRV_LOG(ERR, "Fail to extract pdn from PD"); goto error; } + sh->td = mlx5_devx_cmd_create_td(sh->ctx); + if (!sh->td) { + DRV_LOG(ERR, "TD allocation failure"); + err = ENOMEM; + goto error; + } + tis_attr.transport_domain = sh->td->id; + sh->tis = mlx5_devx_cmd_create_tis(sh->ctx, &tis_attr); + if (!sh->tis) { + DRV_LOG(ERR, "TIS allocation failure"); + err = ENOMEM; + goto error; + } #endif /* HAVE_IBV_FLOW_DV_SUPPORT */ /* * Once the device is added to the list of memory event @@ -425,6 +441,10 @@ struct mlx5_dev_spawn_data { error: pthread_mutex_unlock(&mlx5_ibv_list_mutex); assert(sh); + if (sh->tis) + claim_zero(mlx5_devx_cmd_destroy(sh->tis)); + if (sh->td) + claim_zero(mlx5_devx_cmd_destroy(sh->td)); if (sh->pd) claim_zero(mlx5_glue->dealloc_pd(sh->pd)); if (sh->ctx) @@ -485,6 +505,10 @@ struct mlx5_dev_spawn_data { pthread_mutex_destroy(&sh->intr_mutex); if (sh->pd) claim_zero(mlx5_glue->dealloc_pd(sh->pd)); + if (sh->tis) + claim_zero(mlx5_devx_cmd_destroy(sh->tis)); + if (sh->td) + claim_zero(mlx5_devx_cmd_destroy(sh->td)); if (sh->ctx) claim_zero(mlx5_glue->close_device(sh->ctx)); rte_free(sh); @@ -976,6 +1000,7 @@ struct mlx5_dev_spawn_data { .rx_queue_setup = mlx5_rx_queue_setup, .rx_hairpin_queue_setup = mlx5_rx_hairpin_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, + .tx_hairpin_queue_setup = mlx5_tx_hairpin_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, .flow_ctrl_get = mlx5_dev_get_flow_ctrl, @@ -1043,6 +1068,7 @@ struct mlx5_dev_spawn_data { .rx_queue_setup = mlx5_rx_queue_setup, .rx_hairpin_queue_setup = mlx5_rx_hairpin_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, + .tx_hairpin_queue_setup = mlx5_tx_hairpin_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, .flow_ctrl_get = mlx5_dev_get_flow_ctrl, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index a34972c..506920e 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -350,6 +350,43 @@ struct mlx5_devx_rqt_attr { uint32_t rq_list[]; }; +/* TIS attributes structure. */ +struct mlx5_devx_tis_attr { + uint32_t strict_lag_tx_port_affinity:1; + uint32_t tls_en:1; + uint32_t lag_tx_port_affinity:4; + uint32_t prio:4; + uint32_t transport_domain:24; +}; + +/* SQ attributes structure, used by SQ create operation. */ +struct mlx5_devx_create_sq_attr { + uint32_t rlky:1; + uint32_t cd_master:1; + uint32_t fre:1; + uint32_t flush_in_error_en:1; + uint32_t allow_multi_pkt_send_wqe:1; + uint32_t min_wqe_inline_mode:3; + uint32_t state:4; + uint32_t reg_umr:1; + uint32_t allow_swp:1; + uint32_t hairpin:1; + uint32_t user_index:24; + uint32_t cqn:24; + uint32_t packet_pacing_rate_limit_index:16; + uint32_t tis_lst_sz:16; + uint32_t tis_num:24; + struct mlx5_devx_wq_attr wq_attr; +}; + +/* SQ attributes structure, used by SQ modify operation. */ +struct mlx5_devx_modify_sq_attr { + uint32_t sq_state:4; + uint32_t state:4; + uint32_t hairpin_peer_rq:24; + uint32_t hairpin_peer_vhca:16; +}; + /** * Type of object being allocated. */ @@ -591,6 +628,8 @@ struct mlx5_ibv_shared { struct rte_intr_handle intr_handle; /* Interrupt handler for device. */ struct rte_intr_handle intr_handle_devx; /* DEVX interrupt handler. */ struct mlx5dv_devx_cmd_comp *devx_comp; /* DEVX async comp obj. */ + struct mlx5_devx_obj *tis; /* TIS object. */ + struct mlx5_devx_obj *td; /* Transport domain. */ struct mlx5_ibv_shared_port port[]; /* per device port data array. */ }; @@ -911,5 +950,12 @@ struct mlx5_devx_obj *mlx5_devx_cmd_create_tir(struct ibv_context *ctx, struct mlx5_devx_tir_attr *tir_attr); struct mlx5_devx_obj *mlx5_devx_cmd_create_rqt(struct ibv_context *ctx, struct mlx5_devx_rqt_attr *rqt_attr); +struct mlx5_devx_obj *mlx5_devx_cmd_create_sq + (struct ibv_context *ctx, struct mlx5_devx_create_sq_attr *sq_attr); +int mlx5_devx_cmd_modify_sq + (struct mlx5_devx_obj *sq, struct mlx5_devx_modify_sq_attr *sq_attr); +struct mlx5_devx_obj *mlx5_devx_cmd_create_tis + (struct ibv_context *ctx, struct mlx5_devx_tis_attr *tis_attr); +struct mlx5_devx_obj *mlx5_devx_cmd_create_td(struct ibv_context *ctx); #endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_devx_cmds.c b/drivers/net/mlx5/mlx5_devx_cmds.c index b072c37..917bbf9 100644 --- a/drivers/net/mlx5/mlx5_devx_cmds.c +++ b/drivers/net/mlx5/mlx5_devx_cmds.c @@ -709,3 +709,189 @@ struct mlx5_devx_obj * rqt->id = MLX5_GET(create_rqt_out, out, rqtn); return rqt; } + +/** + * Create SQ using DevX API. + * + * @param[in] ctx + * ibv_context returned from mlx5dv_open_device. + * @param [in] sq_attr + * Pointer to SQ attributes structure. + * @param [in] socket + * CPU socket ID for allocations. + * + * @return + * The DevX object created, NULL otherwise and rte_errno is set. + **/ +struct mlx5_devx_obj * +mlx5_devx_cmd_create_sq(struct ibv_context *ctx, + struct mlx5_devx_create_sq_attr *sq_attr) +{ + uint32_t in[MLX5_ST_SZ_DW(create_sq_in)] = {0}; + uint32_t out[MLX5_ST_SZ_DW(create_sq_out)] = {0}; + void *sq_ctx; + void *wq_ctx; + struct mlx5_devx_wq_attr *wq_attr; + struct mlx5_devx_obj *sq = NULL; + + sq = rte_calloc(__func__, 1, sizeof(*sq), 0); + if (!sq) { + DRV_LOG(ERR, "Failed to allocate SQ data"); + rte_errno = ENOMEM; + return NULL; + } + MLX5_SET(create_sq_in, in, opcode, MLX5_CMD_OP_CREATE_SQ); + sq_ctx = MLX5_ADDR_OF(create_sq_in, in, ctx); + MLX5_SET(sqc, sq_ctx, rlky, sq_attr->rlky); + MLX5_SET(sqc, sq_ctx, cd_master, sq_attr->cd_master); + MLX5_SET(sqc, sq_ctx, fre, sq_attr->fre); + MLX5_SET(sqc, sq_ctx, flush_in_error_en, sq_attr->flush_in_error_en); + MLX5_SET(sqc, sq_ctx, allow_multi_pkt_send_wqe, + sq_attr->flush_in_error_en); + MLX5_SET(sqc, sq_ctx, min_wqe_inline_mode, + sq_attr->min_wqe_inline_mode); + MLX5_SET(sqc, sq_ctx, state, sq_attr->state); + MLX5_SET(sqc, sq_ctx, reg_umr, sq_attr->reg_umr); + MLX5_SET(sqc, sq_ctx, allow_swp, sq_attr->allow_swp); + MLX5_SET(sqc, sq_ctx, hairpin, sq_attr->hairpin); + MLX5_SET(sqc, sq_ctx, user_index, sq_attr->user_index); + MLX5_SET(sqc, sq_ctx, cqn, sq_attr->cqn); + MLX5_SET(sqc, sq_ctx, packet_pacing_rate_limit_index, + sq_attr->packet_pacing_rate_limit_index); + MLX5_SET(sqc, sq_ctx, tis_lst_sz, sq_attr->tis_lst_sz); + MLX5_SET(sqc, sq_ctx, tis_num_0, sq_attr->tis_num); + wq_ctx = MLX5_ADDR_OF(sqc, sq_ctx, wq); + wq_attr = &sq_attr->wq_attr; + devx_cmd_fill_wq_data(wq_ctx, wq_attr); + sq->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in), + out, sizeof(out)); + if (!sq->obj) { + DRV_LOG(ERR, "Failed to create SQ using DevX"); + rte_errno = errno; + rte_free(sq); + return NULL; + } + sq->id = MLX5_GET(create_sq_out, out, sqn); + return sq; +} + +/** + * Modify SQ using DevX API. + * + * @param[in] sq + * Pointer to SQ object structure. + * @param [in] sq_attr + * Pointer to SQ attributes structure. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_devx_cmd_modify_sq(struct mlx5_devx_obj *sq, + struct mlx5_devx_modify_sq_attr *sq_attr) +{ + uint32_t in[MLX5_ST_SZ_DW(modify_sq_in)] = {0}; + uint32_t out[MLX5_ST_SZ_DW(modify_sq_out)] = {0}; + void *sq_ctx; + int ret; + + MLX5_SET(modify_sq_in, in, opcode, MLX5_CMD_OP_MODIFY_SQ); + MLX5_SET(modify_sq_in, in, sq_state, sq_attr->sq_state); + MLX5_SET(modify_sq_in, in, sqn, sq->id); + sq_ctx = MLX5_ADDR_OF(modify_sq_in, in, ctx); + MLX5_SET(sqc, sq_ctx, state, sq_attr->state); + MLX5_SET(sqc, sq_ctx, hairpin_peer_rq, sq_attr->hairpin_peer_rq); + MLX5_SET(sqc, sq_ctx, hairpin_peer_vhca, sq_attr->hairpin_peer_vhca); + ret = mlx5_glue->devx_obj_modify(sq->obj, in, sizeof(in), + out, sizeof(out)); + if (ret) { + DRV_LOG(ERR, "Failed to modify SQ using DevX"); + rte_errno = errno; + return -errno; + } + return ret; +} + +/** + * Create TIS using DevX API. + * + * @param[in] ctx + * ibv_context returned from mlx5dv_open_device. + * @param [in] tis_attr + * Pointer to TIS attributes structure. + * + * @return + * The DevX object created, NULL otherwise and rte_errno is set. + */ +struct mlx5_devx_obj * +mlx5_devx_cmd_create_tis(struct ibv_context *ctx, + struct mlx5_devx_tis_attr *tis_attr) +{ + uint32_t in[MLX5_ST_SZ_DW(create_tis_in)] = {0}; + uint32_t out[MLX5_ST_SZ_DW(create_tis_out)] = {0}; + struct mlx5_devx_obj *tis = NULL; + void *tis_ctx; + + tis = rte_calloc(__func__, 1, sizeof(*tis), 0); + if (!tis) { + DRV_LOG(ERR, "Failed to allocate TIS object"); + rte_errno = ENOMEM; + return NULL; + } + MLX5_SET(create_tis_in, in, opcode, MLX5_CMD_OP_CREATE_TIS); + tis_ctx = MLX5_ADDR_OF(create_tis_in, in, ctx); + MLX5_SET(tisc, tis_ctx, strict_lag_tx_port_affinity, + tis_attr->strict_lag_tx_port_affinity); + MLX5_SET(tisc, tis_ctx, strict_lag_tx_port_affinity, + tis_attr->strict_lag_tx_port_affinity); + MLX5_SET(tisc, tis_ctx, prio, tis_attr->prio); + MLX5_SET(tisc, tis_ctx, transport_domain, + tis_attr->transport_domain); + tis->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in), + out, sizeof(out)); + if (!tis->obj) { + DRV_LOG(ERR, "Failed to create TIS using DevX"); + rte_errno = errno; + rte_free(tis); + return NULL; + } + tis->id = MLX5_GET(create_tis_out, out, tisn); + return tis; +} + +/** + * Create transport domain using DevX API. + * + * @param[in] ctx + * ibv_context returned from mlx5dv_open_device. + * + * @return + * The DevX object created, NULL otherwise and rte_errno is set. + */ +struct mlx5_devx_obj * +mlx5_devx_cmd_create_td(struct ibv_context *ctx) +{ + uint32_t in[MLX5_ST_SZ_DW(alloc_transport_domain_in)] = {0}; + uint32_t out[MLX5_ST_SZ_DW(alloc_transport_domain_out)] = {0}; + struct mlx5_devx_obj *td = NULL; + + td = rte_calloc(__func__, 1, sizeof(*td), 0); + if (!td) { + DRV_LOG(ERR, "Failed to allocate TD object"); + rte_errno = ENOMEM; + return NULL; + } + MLX5_SET(alloc_transport_domain_in, in, opcode, + MLX5_CMD_OP_ALLOC_TRANSPORT_DOMAIN); + td->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in), + out, sizeof(out)); + if (!td->obj) { + DRV_LOG(ERR, "Failed to create TIS using DevX"); + rte_errno = errno; + rte_free(td); + return NULL; + } + td->id = MLX5_GET(alloc_transport_domain_out, out, + transport_domain); + return td; +} diff --git a/drivers/net/mlx5/mlx5_prm.h b/drivers/net/mlx5/mlx5_prm.h index 3765df0..faa7996 100644 --- a/drivers/net/mlx5/mlx5_prm.h +++ b/drivers/net/mlx5/mlx5_prm.h @@ -666,9 +666,13 @@ enum { MLX5_CMD_OP_QUERY_HCA_CAP = 0x100, MLX5_CMD_OP_CREATE_MKEY = 0x200, MLX5_CMD_OP_QUERY_NIC_VPORT_CONTEXT = 0x754, + MLX5_CMD_OP_ALLOC_TRANSPORT_DOMAIN = 0x816, MLX5_CMD_OP_CREATE_TIR = 0x900, + MLX5_CMD_OP_CREATE_SQ = 0X904, + MLX5_CMD_OP_MODIFY_SQ = 0X905, MLX5_CMD_OP_CREATE_RQ = 0x908, MLX5_CMD_OP_MODIFY_RQ = 0x909, + MLX5_CMD_OP_CREATE_TIS = 0x912, MLX5_CMD_OP_QUERY_TIS = 0x915, MLX5_CMD_OP_CREATE_RQT = 0x916, MLX5_CMD_OP_ALLOC_FLOW_COUNTER = 0x939, @@ -1311,6 +1315,23 @@ struct mlx5_ifc_query_tis_in_bits { u8 reserved_at_60[0x20]; }; +struct mlx5_ifc_alloc_transport_domain_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x8]; + u8 transport_domain[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_alloc_transport_domain_in_bits { + u8 opcode[0x10]; + u8 reserved_at_10[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0x40]; +}; + enum { MLX5_WQ_TYPE_LINKED_LIST = 0x0, MLX5_WQ_TYPE_CYCLIC = 0x1, @@ -1427,6 +1448,24 @@ struct mlx5_ifc_modify_rq_out_bits { u8 reserved_at_40[0x40]; }; +struct mlx5_ifc_create_tis_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x8]; + u8 tisn[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_create_tis_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0xc0]; + struct mlx5_ifc_tisc_bits ctx; +}; + enum { MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_WQ_LWM = 1ULL << 0, MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_VSD = 1ULL << 1, @@ -1572,6 +1611,85 @@ struct mlx5_ifc_create_rqt_in_bits { #pragma GCC diagnostic error "-Wpedantic" #endif +struct mlx5_ifc_sqc_bits { + u8 rlky[0x1]; + u8 cd_master[0x1]; + u8 fre[0x1]; + u8 flush_in_error_en[0x1]; + u8 allow_multi_pkt_send_wqe[0x1]; + u8 min_wqe_inline_mode[0x3]; + u8 state[0x4]; + u8 reg_umr[0x1]; + u8 allow_swp[0x1]; + u8 hairpin[0x1]; + u8 reserved_at_f[0x11]; + u8 reserved_at_20[0x8]; + u8 user_index[0x18]; + u8 reserved_at_40[0x8]; + u8 cqn[0x18]; + u8 reserved_at_60[0x8]; + u8 hairpin_peer_rq[0x18]; + u8 reserved_at_80[0x10]; + u8 hairpin_peer_vhca[0x10]; + u8 reserved_at_a0[0x50]; + u8 packet_pacing_rate_limit_index[0x10]; + u8 tis_lst_sz[0x10]; + u8 reserved_at_110[0x10]; + u8 reserved_at_120[0x40]; + u8 reserved_at_160[0x8]; + u8 tis_num_0[0x18]; + struct mlx5_ifc_wq_bits wq; +}; + +struct mlx5_ifc_query_sq_in_bits { + u8 opcode[0x10]; + u8 reserved_at_10[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0x8]; + u8 sqn[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_modify_sq_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x40]; +}; + +struct mlx5_ifc_modify_sq_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 sq_state[0x4]; + u8 reserved_at_44[0x4]; + u8 sqn[0x18]; + u8 reserved_at_60[0x20]; + u8 modify_bitmask[0x40]; + u8 reserved_at_c0[0x40]; + struct mlx5_ifc_sqc_bits ctx; +}; + +struct mlx5_ifc_create_sq_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x8]; + u8 sqn[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_create_sq_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0xc0]; + struct mlx5_ifc_sqc_bits ctx; +}; + /* CQE format mask. */ #define MLX5E_CQE_FORMAT_MASK 0xc diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 2de674a..8fa22e5 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -324,14 +324,18 @@ struct mlx5_txq_obj { LIST_ENTRY(mlx5_txq_obj) next; /* Pointer to the next element. */ rte_atomic32_t refcnt; /* Reference counter. */ struct mlx5_txq_ctrl *txq_ctrl; /* Pointer to the control queue. */ - enum mlx5_rxq_obj_type type; /* The txq object type. */ + enum mlx5_txq_obj_type type; /* The txq object type. */ RTE_STD_C11 union { struct { struct ibv_cq *cq; /* Completion Queue. */ struct ibv_qp *qp; /* Queue Pair. */ }; - struct mlx5_devx_obj *sq; /* DevX object for Sx queue. */ + struct { + struct mlx5_devx_obj *sq; + /* DevX object for Sx queue. */ + struct mlx5_devx_obj *tis; /* The TIS object. */ + }; }; }; @@ -348,6 +352,7 @@ struct mlx5_txq_ctrl { off_t uar_mmap_offset; /* UAR mmap offset for non-primary process. */ void *bf_reg; /* BlueFlame register from Verbs. */ uint16_t dump_file_n; /* Number of dump files. */ + struct rte_eth_hairpin_conf hairpin_conf; /* Hairpin configuration. */ struct mlx5_txq_data txq; /* Data path structure. */ /* Must be the last field in the structure, contains elts[]. */ }; @@ -412,15 +417,24 @@ struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev, int mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_txconf *conf); +int mlx5_tx_hairpin_queue_setup + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_txconf *conf, + const struct rte_eth_hairpin_conf *hairpin_conf); void mlx5_tx_queue_release(void *dpdk_txq); int mlx5_tx_uar_init_secondary(struct rte_eth_dev *dev, int fd); -struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx); +struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx, + enum mlx5_txq_obj_type type); struct mlx5_txq_obj *mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t idx); int mlx5_txq_obj_release(struct mlx5_txq_obj *txq_ibv); int mlx5_txq_obj_verify(struct rte_eth_dev *dev); struct mlx5_txq_ctrl *mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_txconf *conf); +struct mlx5_txq_ctrl *mlx5_txq_hairpin_new + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_txconf *conf, + const struct rte_eth_hairpin_conf *hairpin_conf); struct mlx5_txq_ctrl *mlx5_txq_get(struct rte_eth_dev *dev, uint16_t idx); int mlx5_txq_release(struct rte_eth_dev *dev, uint16_t idx); int mlx5_txq_releasable(struct rte_eth_dev *dev, uint16_t idx); diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 50c4df5..3ec86c4 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -51,8 +51,14 @@ if (!txq_ctrl) continue; - txq_alloc_elts(txq_ctrl); - txq_ctrl->obj = mlx5_txq_obj_new(dev, i); + if (txq_ctrl->type == MLX5_TXQ_TYPE_HAIRPIN) { + txq_ctrl->obj = mlx5_txq_obj_new + (dev, i, MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN); + } else { + txq_alloc_elts(txq_ctrl); + txq_ctrl->obj = mlx5_txq_obj_new + (dev, i, MLX5_TXQ_OBJ_TYPE_IBV); + } if (!txq_ctrl->obj) { rte_errno = ENOMEM; goto error; diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index e1ed4eb..44233e9 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -136,30 +136,22 @@ } /** - * DPDK callback to configure a TX queue. + * Tx queue presetup checks. * * @param dev * Pointer to Ethernet device structure. * @param idx - * TX queue index. + * Tx queue index. * @param desc * Number of descriptors to configure in queue. - * @param socket - * NUMA socket on which memory must be allocated. - * @param[in] conf - * Thresholds parameters. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, - unsigned int socket, const struct rte_eth_txconf *conf) +static int +mlx5_tx_queue_pre_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc) { struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_data *txq = (*priv->txqs)[idx]; - struct mlx5_txq_ctrl *txq_ctrl = - container_of(txq, struct mlx5_txq_ctrl, txq); if (desc <= MLX5_TX_COMP_THRESH) { DRV_LOG(WARNING, @@ -191,6 +183,38 @@ return -rte_errno; } mlx5_txq_release(dev, idx); + return 0; +} +/** + * DPDK callback to configure a TX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * TX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_txconf *conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq, struct mlx5_txq_ctrl, txq); + int res; + + res = mlx5_tx_queue_pre_setup(dev, idx, desc); + if (res) + return res; txq_ctrl = mlx5_txq_new(dev, idx, desc, socket, conf); if (!txq_ctrl) { DRV_LOG(ERR, "port %u unable to allocate queue index %u", @@ -204,6 +228,63 @@ } /** + * DPDK callback to configure a TX hairpin queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * TX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * @param[in] hairpin_conf + * The hairpin binding configuration. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_tx_hairpin_queue_setup(struct rte_eth_dev *dev, uint16_t idx, + uint16_t desc, unsigned int socket, + const struct rte_eth_txconf *conf, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq, struct mlx5_txq_ctrl, txq); + int res; + + res = mlx5_tx_queue_pre_setup(dev, idx, desc); + if (res) + return res; + if (hairpin_conf->peer_n != 1 || + hairpin_conf->peers[0].port != dev->data->port_id || + hairpin_conf->peers[0].queue >= priv->rxqs_n) { + DRV_LOG(ERR, "port %u unable to setup hairpin queue index %u " + " invalid hairpind configuration", dev->data->port_id, + idx); + rte_errno = EINVAL; + return -rte_errno; + } + txq_ctrl = mlx5_txq_hairpin_new(dev, idx, desc, socket, conf, + hairpin_conf); + if (!txq_ctrl) { + DRV_LOG(ERR, "port %u unable to allocate queue index %u", + dev->data->port_id, idx); + return -rte_errno; + } + DRV_LOG(DEBUG, "port %u adding Tx queue %u to list", + dev->data->port_id, idx); + (*priv->txqs)[idx] = &txq_ctrl->txq; + txq_ctrl->type = MLX5_TXQ_TYPE_HAIRPIN; + return 0; +} + +/** * DPDK callback to release a TX queue. * * @param dpdk_txq @@ -246,6 +327,8 @@ const size_t page_size = sysconf(_SC_PAGESIZE); #endif + if (txq_ctrl->type != MLX5_TXQ_TYPE_STANDARD) + return; assert(rte_eal_process_type() == RTE_PROC_PRIMARY); assert(ppriv); ppriv->uar_table[txq_ctrl->txq.idx] = txq_ctrl->bf_reg; @@ -282,6 +365,8 @@ uintptr_t offset; const size_t page_size = sysconf(_SC_PAGESIZE); + if (txq_ctrl->type != MLX5_TXQ_TYPE_STANDARD) + return 0; assert(ppriv); /* * As rdma-core, UARs are mapped in size of OS page @@ -316,6 +401,8 @@ const size_t page_size = sysconf(_SC_PAGESIZE); void *addr; + if (txq_ctrl->type != MLX5_TXQ_TYPE_STANDARD) + return; addr = ppriv->uar_table[txq_ctrl->txq.idx]; munmap(RTE_PTR_ALIGN_FLOOR(addr, page_size), page_size); } @@ -346,6 +433,8 @@ continue; txq = (*priv->txqs)[i]; txq_ctrl = container_of(txq, struct mlx5_txq_ctrl, txq); + if (txq_ctrl->type != MLX5_TXQ_TYPE_STANDARD) + continue; assert(txq->idx == (uint16_t)i); ret = txq_uar_init_secondary(txq_ctrl, fd); if (ret) @@ -365,18 +454,87 @@ } /** + * Create the Tx hairpin queue object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array + * + * @return + * The hairpin DevX object initialised, NULL otherwise and rte_errno is set. + */ +static struct mlx5_txq_obj * +mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + struct mlx5_devx_create_sq_attr attr = { 0 }; + struct mlx5_txq_obj *tmpl = NULL; + int ret = 0; + + assert(txq_data); + assert(!txq_ctrl->obj); + tmpl = rte_calloc_socket(__func__, 1, sizeof(*tmpl), 0, + txq_ctrl->socket); + if (!tmpl) { + DRV_LOG(ERR, + "port %u Tx queue %u cannot allocate memory resources", + dev->data->port_id, txq_data->idx); + rte_errno = ENOMEM; + goto error; + } + tmpl->type = MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN; + tmpl->txq_ctrl = txq_ctrl; + attr.hairpin = 1; + attr.tis_lst_sz = 1; + /* Workaround for hairpin startup */ + attr.wq_attr.log_hairpin_num_packets = log2above(32); + /* Workaround for packets larger than 1KB */ + attr.wq_attr.log_hairpin_data_sz = + priv->config.hca_attr.log_max_hairpin_wq_data_sz; + attr.tis_num = priv->sh->tis->id; + tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->ctx, &attr); + if (!tmpl->sq) { + DRV_LOG(ERR, + "port %u tx hairpin queue %u can't create sq object", + dev->data->port_id, idx); + rte_errno = errno; + goto error; + } + DRV_LOG(DEBUG, "port %u sxq %u updated with %p", dev->data->port_id, + idx, (void *)&tmpl); + rte_atomic32_inc(&tmpl->refcnt); + LIST_INSERT_HEAD(&priv->txqsobj, tmpl, next); + return tmpl; +error: + ret = rte_errno; /* Save rte_errno before cleanup. */ + if (tmpl->tis) + mlx5_devx_cmd_destroy(tmpl->tis); + if (tmpl->sq) + mlx5_devx_cmd_destroy(tmpl->sq); + rte_errno = ret; /* Restore rte_errno. */ + return NULL; +} + +/** * Create the Tx queue Verbs object. * * @param dev * Pointer to Ethernet device. * @param idx * Queue index in DPDK Tx queue array. + * @param type + * Type of the Tx queue object to create. * * @return * The Verbs object initialised, NULL otherwise and rte_errno is set. */ struct mlx5_txq_obj * -mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx) +mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx, + enum mlx5_txq_obj_type type) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; @@ -396,6 +554,8 @@ struct mlx5_txq_obj * const int desc = 1 << txq_data->elts_n; int ret = 0; + if (type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) + return mlx5_txq_obj_hairpin_new(dev, idx); #ifdef HAVE_IBV_FLOW_DV_SUPPORT /* If using DevX, need additional mask to read tisn value. */ if (priv->config.devx && !priv->sh->tdn) @@ -643,8 +803,13 @@ struct mlx5_txq_obj * { assert(txq_obj); if (rte_atomic32_dec_and_test(&txq_obj->refcnt)) { - claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); - claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); + if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) { + if (txq_obj->tis) + claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); + } else { + claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); + claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); + } LIST_REMOVE(txq_obj, next); rte_free(txq_obj); return 0; @@ -953,6 +1118,7 @@ struct mlx5_txq_ctrl * goto error; } rte_atomic32_inc(&tmpl->refcnt); + tmpl->type = MLX5_TXQ_TYPE_STANDARD; LIST_INSERT_HEAD(&priv->txqsctrl, tmpl, next); return tmpl; error: @@ -961,6 +1127,55 @@ struct mlx5_txq_ctrl * } /** + * Create a DPDK Tx hairpin queue. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * TX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * @param hairpin_conf + * The hairpin configuration. + * + * @return + * A DPDK queue object on success, NULL otherwise and rte_errno is set. + */ +struct mlx5_txq_ctrl * +mlx5_txq_hairpin_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_txconf *conf, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_ctrl *tmpl; + + tmpl = rte_calloc_socket("TXQ", 1, + sizeof(*tmpl), + 0, socket); + if (!tmpl) { + rte_errno = ENOMEM; + return NULL; + } + assert(desc > MLX5_TX_COMP_THRESH); + tmpl->txq.offloads = conf->offloads | + dev->data->dev_conf.txmode.offloads; + tmpl->priv = priv; + tmpl->socket = socket; + tmpl->txq.elts_n = log2above(desc); + tmpl->txq.port_id = dev->data->port_id; + tmpl->txq.idx = idx; + tmpl->hairpin_conf = *hairpin_conf; + tmpl->type = MLX5_TXQ_TYPE_HAIRPIN; + rte_atomic32_inc(&tmpl->refcnt); + LIST_INSERT_HEAD(&priv->txqsctrl, tmpl, next); + return tmpl; +} + +/** * Get a Tx queue. * * @param dev From patchwork Thu Sep 26 06:29:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59787 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 197891BEA6; Thu, 26 Sep 2019 08:30:18 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 596551B994 for ; Thu, 26 Sep 2019 08:30:05 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:30:00 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlL012493; Thu, 26 Sep 2019 09:30:00 +0300 From: Ori Kam To: Wenzhuo Lu , Jingjing Wu , Bernard Iremonger Cc: dev@dpdk.org, orika@mellanox.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:02 +0000 Message-Id: <1569479349-36962-7-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 06/13] app/testpmd: add hairpin support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit introduce the hairpin queues to the testpmd. the hairpin queue is configured using --hairpinq= the hairpin queue adds n queue objects for both the total number of TX queues and RX queues. The connection between the queues are 1 to 1, first Rx hairpin queue will be connected to the first Tx hairpin queue Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- app/test-pmd/parameters.c | 12 ++++++++++ app/test-pmd/testpmd.c | 59 +++++++++++++++++++++++++++++++++++++++++++++-- app/test-pmd/testpmd.h | 1 + 3 files changed, 70 insertions(+), 2 deletions(-) diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c index 6c78dca..16bdcc8 100644 --- a/app/test-pmd/parameters.c +++ b/app/test-pmd/parameters.c @@ -147,6 +147,8 @@ printf(" --rxd=N: set the number of descriptors in RX rings to N.\n"); printf(" --txq=N: set the number of TX queues per port to N.\n"); printf(" --txd=N: set the number of descriptors in TX rings to N.\n"); + printf(" --hairpinq=N: set the number of hairpin queues per port to " + "N.\n"); printf(" --burst=N: set the number of packets per burst to N.\n"); printf(" --mbcache=N: set the cache of mbuf memory pool to N.\n"); printf(" --rxpt=N: set prefetch threshold register of RX rings to N.\n"); @@ -618,6 +620,7 @@ { "txq", 1, 0, 0 }, { "rxd", 1, 0, 0 }, { "txd", 1, 0, 0 }, + { "hairpinq", 1, 0, 0 }, { "burst", 1, 0, 0 }, { "mbcache", 1, 0, 0 }, { "txpt", 1, 0, 0 }, @@ -1036,6 +1039,15 @@ " >= 0 && <= %u\n", n, get_allowed_max_nb_txq(&pid)); } + if (!strcmp(lgopts[opt_idx].name, "hairpinq")) { + n = atoi(optarg); + if (n >= 0 && check_nb_txq((queueid_t)n) == 0) + nb_hairpinq = (queueid_t) n; + else + rte_exit(EXIT_FAILURE, "txq %d invalid - must be" + " >= 0 && <= %u\n", n, + get_allowed_max_nb_txq(&pid)); + } if (!nb_rxq && !nb_txq) { rte_exit(EXIT_FAILURE, "Either rx or tx queues should " "be non-zero\n"); diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index de91e1b..f15a308 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -235,6 +235,7 @@ struct fwd_engine * fwd_engines[] = { /* * Configurable number of RX/TX queues. */ +queueid_t nb_hairpinq; /**< Number of hairpin queues per port. */ queueid_t nb_rxq = 1; /**< Number of RX queues per port. */ queueid_t nb_txq = 1; /**< Number of TX queues per port. */ @@ -2064,6 +2065,10 @@ struct extmem_param { queueid_t qi; struct rte_port *port; struct rte_ether_addr mac_addr; + struct rte_eth_hairpin_conf hairpin_conf = { + .peer_n = 1, + }; + int i; if (port_id_is_invalid(pid, ENABLED_WARN)) return 0; @@ -2097,8 +2102,9 @@ struct extmem_param { printf("Configuring Port %d (socket %u)\n", pi, port->socket_id); /* configure port */ - diag = rte_eth_dev_configure(pi, nb_rxq, nb_txq, - &(port->dev_conf)); + diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq, + nb_txq + nb_hairpinq, + &(port->dev_conf)); if (diag != 0) { if (rte_atomic16_cmpset(&(port->port_status), RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0) @@ -2191,6 +2197,55 @@ struct extmem_param { port->need_reconfig_queues = 1; return -1; } + /* setup hairpin queues */ + i = 0; + for (qi = nb_txq; qi < nb_hairpinq + nb_txq; qi++) { + hairpin_conf.peers[0].port = pi; + hairpin_conf.peers[0].queue = i + nb_rxq; + diag = rte_eth_tx_hairpin_queue_setup + (pi, qi, nb_txd, + port->socket_id, &(port->tx_conf[qi]), + &hairpin_conf); + i++; + if (diag == 0) + continue; + + /* Fail to setup rx queue, return */ + if (rte_atomic16_cmpset(&(port->port_status), + RTE_PORT_HANDLING, + RTE_PORT_STOPPED) == 0) + printf("Port %d can not be set back " + "to stopped\n", pi); + printf("Fail to configure port %d hairpin " + "queues\n", pi); + /* try to reconfigure queues next time */ + port->need_reconfig_queues = 1; + return -1; + } + i = 0; + for (qi = nb_rxq; qi < nb_hairpinq + nb_rxq; qi++) { + hairpin_conf.peers[0].port = pi; + hairpin_conf.peers[0].queue = i + nb_txq; + diag = rte_eth_rx_hairpin_queue_setup + (pi, qi, nb_rxd, + port->socket_id, &(port->rx_conf[qi]), + &hairpin_conf); + i++; + if (diag == 0) + continue; + + /* Fail to setup rx queue, return */ + if (rte_atomic16_cmpset(&(port->port_status), + RTE_PORT_HANDLING, + RTE_PORT_STOPPED) == 0) + printf("Port %d can not be set back " + "to stopped\n", pi); + printf("Fail to configure port %d hairpin " + "queues\n", pi); + /* try to reconfigure queues next time */ + port->need_reconfig_queues = 1; + return -1; + } } configure_rxtx_dump_callbacks(verbose_level); /* start port */ diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index d73955d..09baa72 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -383,6 +383,7 @@ struct queue_stats_mappings { extern uint64_t rss_hf; +extern queueid_t nb_hairpinq; extern queueid_t nb_rxq; extern queueid_t nb_txq; From patchwork Thu Sep 26 06:29:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59786 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D01971BE9C; Thu, 26 Sep 2019 08:30:15 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 5B7771B99D for ; Thu, 26 Sep 2019 08:30:05 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:30:01 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlM012493; Thu, 26 Sep 2019 09:30:01 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:03 +0000 Message-Id: <1569479349-36962-8-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 07/13] net/mlx5: add hairpin binding function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When starting the port, in addition to creating the queues we need to bind the hairpin queues. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_devx_cmds.c | 1 + drivers/net/mlx5/mlx5_prm.h | 6 +++ drivers/net/mlx5/mlx5_trigger.c | 97 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 105 insertions(+) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 506920e..41eb35a 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -188,6 +188,7 @@ struct mlx5_hca_attr { uint32_t log_max_hairpin_queues:5; uint32_t log_max_hairpin_wq_data_sz:5; uint32_t log_max_hairpin_num_packets:5; + uint32_t vhca_id:16; }; /* Flow list . */ diff --git a/drivers/net/mlx5/mlx5_devx_cmds.c b/drivers/net/mlx5/mlx5_devx_cmds.c index 917bbf9..0243733 100644 --- a/drivers/net/mlx5/mlx5_devx_cmds.c +++ b/drivers/net/mlx5/mlx5_devx_cmds.c @@ -334,6 +334,7 @@ struct mlx5_devx_obj * log_max_hairpin_wq_data_sz); attr->log_max_hairpin_num_packets = MLX5_GET (cmd_hca_cap, hcattr, log_min_hairpin_wq_data_sz); + attr->vhca_id = MLX5_GET(cmd_hca_cap, hcattr, vhca_id); attr->eth_net_offloads = MLX5_GET(cmd_hca_cap, hcattr, eth_net_offloads); attr->eth_virt = MLX5_GET(cmd_hca_cap, hcattr, eth_virt); diff --git a/drivers/net/mlx5/mlx5_prm.h b/drivers/net/mlx5/mlx5_prm.h index faa7996..d4084db 100644 --- a/drivers/net/mlx5/mlx5_prm.h +++ b/drivers/net/mlx5/mlx5_prm.h @@ -1611,6 +1611,12 @@ struct mlx5_ifc_create_rqt_in_bits { #pragma GCC diagnostic error "-Wpedantic" #endif +enum { + MLX5_SQC_STATE_RST = 0x0, + MLX5_SQC_STATE_RDY = 0x1, + MLX5_SQC_STATE_ERR = 0x3, +}; + struct mlx5_ifc_sqc_bits { u8 rlky[0x1]; u8 cd_master[0x1]; diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 3ec86c4..a4fcdb3 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -162,6 +162,96 @@ } /** + * Binds Tx queues to Rx queues for hairpin. + * + * Binds Tx queues to the target Rx queues. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_hairpin_bind(struct rte_eth_dev *dev) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_devx_modify_sq_attr sq_attr = { 0 }; + struct mlx5_devx_modify_rq_attr rq_attr = { 0 }; + struct mlx5_txq_ctrl *txq_ctrl; + struct mlx5_rxq_ctrl *rxq_ctrl; + struct mlx5_devx_obj *sq; + struct mlx5_devx_obj *rq; + unsigned int i; + int ret = 0; + + for (i = 0; i != priv->txqs_n; ++i) { + txq_ctrl = mlx5_txq_get(dev, i); + if (!txq_ctrl) + continue; + if (txq_ctrl->type != MLX5_TXQ_TYPE_HAIRPIN) { + mlx5_txq_release(dev, i); + continue; + } + if (!txq_ctrl->obj) { + rte_errno = ENOMEM; + DRV_LOG(ERR, "port %u no txq object found: %d", + dev->data->port_id, i); + mlx5_txq_release(dev, i); + return -rte_errno; + } + sq = txq_ctrl->obj->sq; + rxq_ctrl = mlx5_rxq_get(dev, + txq_ctrl->hairpin_conf.peers[0].queue); + if (!rxq_ctrl) { + mlx5_txq_release(dev, i); + rte_errno = EINVAL; + DRV_LOG(ERR, "port %u no rxq object found: %d", + dev->data->port_id, + txq_ctrl->hairpin_conf.peers[0].queue); + return -rte_errno; + } + if (rxq_ctrl->type != MLX5_RXQ_TYPE_HAIRPIN || + rxq_ctrl->hairpin_conf.peers[0].queue != i) { + rte_errno = ENOMEM; + DRV_LOG(ERR, "port %u Tx queue %d can't be binded to " + "Rx queue %d", dev->data->port_id, + i, txq_ctrl->hairpin_conf.peers[0].queue); + goto error; + } + rq = rxq_ctrl->obj->rq; + if (!rq) { + rte_errno = ENOMEM; + DRV_LOG(ERR, "port %u hairpin no matching rxq: %d", + dev->data->port_id, + txq_ctrl->hairpin_conf.peers[0].queue); + goto error; + } + sq_attr.state = MLX5_SQC_STATE_RDY; + sq_attr.sq_state = MLX5_SQC_STATE_RST; + sq_attr.hairpin_peer_rq = rq->id; + sq_attr.hairpin_peer_vhca = priv->config.hca_attr.vhca_id; + ret = mlx5_devx_cmd_modify_sq(sq, &sq_attr); + if (ret) + goto error; + rq_attr.state = MLX5_SQC_STATE_RDY; + rq_attr.rq_state = MLX5_SQC_STATE_RST; + rq_attr.hairpin_peer_sq = sq->id; + rq_attr.hairpin_peer_vhca = priv->config.hca_attr.vhca_id; + ret = mlx5_devx_cmd_modify_rq(rq, &rq_attr); + if (ret) + goto error; + mlx5_txq_release(dev, i); + mlx5_rxq_release(dev, txq_ctrl->hairpin_conf.peers[0].queue); + } + return 0; +error: + mlx5_txq_release(dev, i); + mlx5_rxq_release(dev, txq_ctrl->hairpin_conf.peers[0].queue); + return -rte_errno; +} + +/** * DPDK callback to start the device. * * Simulate device start by attaching all configured flows. @@ -192,6 +282,13 @@ mlx5_txq_stop(dev); return -rte_errno; } + ret = mlx5_hairpin_bind(dev); + if (ret) { + DRV_LOG(ERR, "port %u hairpin binding failed: %s", + dev->data->port_id, strerror(rte_errno)); + mlx5_txq_stop(dev); + return -rte_errno; + } dev->data->dev_started = 1; ret = mlx5_rx_intr_vec_enable(dev); if (ret) { From patchwork Thu Sep 26 06:29:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59788 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 089601BEB3; Thu, 26 Sep 2019 08:30:20 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 9B9181B9B5 for ; Thu, 26 Sep 2019 08:30:05 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:30:02 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlN012493; Thu, 26 Sep 2019 09:30:02 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:04 +0000 Message-Id: <1569479349-36962-9-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 08/13] net/mlx5: add support for hairpin hrxq X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The hairpin hrxq is based on the DevX hrxq but uses different pd. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5_rxq.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index a673da9..bf39112 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -2344,13 +2344,13 @@ struct mlx5_hrxq * struct mlx5_ind_table_obj *ind_tbl; int err; struct mlx5_devx_obj *tir = NULL; + struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[queues[0]]; + struct mlx5_rxq_ctrl *rxq_ctrl = + container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); queues_n = hash_fields ? queues_n : 1; ind_tbl = mlx5_ind_table_obj_get(dev, queues, queues_n); if (!ind_tbl) { - struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[queues[0]]; - struct mlx5_rxq_ctrl *rxq_ctrl = - container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); enum mlx5_ind_tbl_type type; type = rxq_ctrl->obj->type == MLX5_RXQ_OBJ_TYPE_IBV ? @@ -2446,7 +2446,10 @@ struct mlx5_hrxq * tir_attr.rx_hash_fn = MLX5_RX_HASH_FN_TOEPLITZ; memcpy(&tir_attr.rx_hash_field_selector_outer, &hash_fields, sizeof(uint64_t)); - tir_attr.transport_domain = priv->sh->tdn; + if (rxq_ctrl->obj->type == MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN) + tir_attr.transport_domain = priv->sh->td->id; + else + tir_attr.transport_domain = priv->sh->tdn; memcpy(tir_attr.rx_hash_toeplitz_key, rss_key, rss_key_len); tir_attr.indirect_table = ind_tbl->rqt->id; if (dev->data->dev_conf.lpbk_mode) From patchwork Thu Sep 26 06:29:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59789 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 14CB21BEBB; Thu, 26 Sep 2019 08:30:22 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id CF1B21B94D for ; Thu, 26 Sep 2019 08:30:05 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:30:03 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlO012493; Thu, 26 Sep 2019 09:30:03 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:05 +0000 Message-Id: <1569479349-36962-10-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 09/13] net/mlx5: add internal tag item and action X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit introduce the setting and matching on regiters. This item and and action will be used with number of different features like hairpin, metering, metadata. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5_flow.c | 52 +++++++++++++ drivers/net/mlx5/mlx5_flow.h | 54 ++++++++++++-- drivers/net/mlx5/mlx5_flow_dv.c | 158 +++++++++++++++++++++++++++++++++++++++- drivers/net/mlx5/mlx5_prm.h | 3 +- 4 files changed, 257 insertions(+), 10 deletions(-) diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index 482f65b..00afc18 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -316,6 +316,58 @@ struct mlx5_flow_tunnel_info { }, }; +enum mlx5_feature_name { + MLX5_HAIRPIN_RX, + MLX5_HAIRPIN_TX, + MLX5_APPLICATION, +}; + +/** + * Translate tag ID to register. + * + * @param[in] dev + * Pointer to the Ethernet device structure. + * @param[in] feature + * The feature that request the register. + * @param[in] id + * The request register ID. + * @param[out] error + * Error description in case of any. + * + * @return + * The request register on success, a negative errno + * value otherwise and rte_errno is set. + */ +__rte_unused +static enum modify_reg flow_get_reg_id(struct rte_eth_dev *dev, + enum mlx5_feature_name feature, + uint32_t id, + struct rte_flow_error *error) +{ + static enum modify_reg id2reg[] = { + [0] = REG_A, + [1] = REG_C_2, + [2] = REG_C_3, + [3] = REG_C_4, + [4] = REG_B,}; + + dev = (void *)dev; + switch (feature) { + case MLX5_HAIRPIN_RX: + return REG_B; + case MLX5_HAIRPIN_TX: + return REG_A; + case MLX5_APPLICATION: + if (id > 4) + return rte_flow_error_set(error, EINVAL, + RTE_FLOW_ERROR_TYPE_ITEM, + NULL, "invalid tag id"); + return id2reg[id]; + } + return rte_flow_error_set(error, EINVAL, RTE_FLOW_ERROR_TYPE_ITEM, + NULL, "invalid feature name"); +} + /** * Discover the maximum number of priority available. * diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 235bccd..0148c1b 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -27,6 +27,43 @@ #include "mlx5.h" #include "mlx5_prm.h" +enum modify_reg { + REG_A, + REG_B, + REG_C_0, + REG_C_1, + REG_C_2, + REG_C_3, + REG_C_4, + REG_C_5, + REG_C_6, + REG_C_7, +}; + +/* Private rte flow items. */ +enum mlx5_rte_flow_item_type { + MLX5_RTE_FLOW_ITEM_TYPE_END = INT_MIN, + MLX5_RTE_FLOW_ITEM_TYPE_TAG, +}; + +/* Private rte flow actions. */ +enum mlx5_rte_flow_action_type { + MLX5_RTE_FLOW_ACTION_TYPE_END = INT_MIN, + MLX5_RTE_FLOW_ACTION_TYPE_TAG, +}; + +/* Matches on selected register. */ +struct mlx5_rte_flow_item_tag { + uint16_t id; + rte_be32_t data; +}; + +/* Modify selected register. */ +struct mlx5_rte_flow_action_set_tag { + uint16_t id; + rte_be32_t data; +}; + /* Pattern outer Layer bits. */ #define MLX5_FLOW_LAYER_OUTER_L2 (1u << 0) #define MLX5_FLOW_LAYER_OUTER_L3_IPV4 (1u << 1) @@ -53,16 +90,17 @@ /* General pattern items bits. */ #define MLX5_FLOW_ITEM_METADATA (1u << 16) #define MLX5_FLOW_ITEM_PORT_ID (1u << 17) +#define MLX5_FLOW_ITEM_TAG (1u << 18) /* Pattern MISC bits. */ -#define MLX5_FLOW_LAYER_ICMP (1u << 18) -#define MLX5_FLOW_LAYER_ICMP6 (1u << 19) -#define MLX5_FLOW_LAYER_GRE_KEY (1u << 20) +#define MLX5_FLOW_LAYER_ICMP (1u << 19) +#define MLX5_FLOW_LAYER_ICMP6 (1u << 20) +#define MLX5_FLOW_LAYER_GRE_KEY (1u << 21) /* Pattern tunnel Layer bits (continued). */ -#define MLX5_FLOW_LAYER_IPIP (1u << 21) -#define MLX5_FLOW_LAYER_IPV6_ENCAP (1u << 22) -#define MLX5_FLOW_LAYER_NVGRE (1u << 23) +#define MLX5_FLOW_LAYER_IPIP (1u << 22) +#define MLX5_FLOW_LAYER_IPV6_ENCAP (1u << 23) +#define MLX5_FLOW_LAYER_NVGRE (1u << 24) /* Outer Masks. */ #define MLX5_FLOW_LAYER_OUTER_L3 \ @@ -139,6 +177,7 @@ #define MLX5_FLOW_ACTION_DEC_TCP_SEQ (1u << 29) #define MLX5_FLOW_ACTION_INC_TCP_ACK (1u << 30) #define MLX5_FLOW_ACTION_DEC_TCP_ACK (1u << 31) +#define MLX5_FLOW_ACTION_SET_TAG (1ull << 32) #define MLX5_FLOW_FATE_ACTIONS \ (MLX5_FLOW_ACTION_DROP | MLX5_FLOW_ACTION_QUEUE | \ @@ -172,7 +211,8 @@ MLX5_FLOW_ACTION_DEC_TCP_SEQ | \ MLX5_FLOW_ACTION_INC_TCP_ACK | \ MLX5_FLOW_ACTION_DEC_TCP_ACK | \ - MLX5_FLOW_ACTION_OF_SET_VLAN_VID) + MLX5_FLOW_ACTION_OF_SET_VLAN_VID | \ + MLX5_FLOW_ACTION_SET_TAG) #define MLX5_FLOW_VLAN_ACTIONS (MLX5_FLOW_ACTION_OF_POP_VLAN | \ MLX5_FLOW_ACTION_OF_PUSH_VLAN) diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index 2a7e3ed..dde0831 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -723,6 +723,59 @@ struct field_modify_info modify_tcp[] = { MLX5_MODIFICATION_TYPE_ADD, error); } +static enum mlx5_modification_field reg_to_field[] = { + [REG_A] = MLX5_MODI_META_DATA_REG_A, + [REG_B] = MLX5_MODI_META_DATA_REG_B, + [REG_C_0] = MLX5_MODI_META_REG_C_0, + [REG_C_1] = MLX5_MODI_META_REG_C_1, + [REG_C_2] = MLX5_MODI_META_REG_C_2, + [REG_C_3] = MLX5_MODI_META_REG_C_3, + [REG_C_4] = MLX5_MODI_META_REG_C_4, + [REG_C_5] = MLX5_MODI_META_REG_C_5, + [REG_C_6] = MLX5_MODI_META_REG_C_6, + [REG_C_7] = MLX5_MODI_META_REG_C_7, +}; + +/** + * Convert register set to DV specification. + * + * @param[in,out] resource + * Pointer to the modify-header resource. + * @param[in] action + * Pointer to action specification. + * @param[out] error + * Pointer to the error structure. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +flow_dv_convert_action_set_reg + (struct mlx5_flow_dv_modify_hdr_resource *resource, + const struct rte_flow_action *action, + struct rte_flow_error *error) +{ + const struct mlx5_rte_flow_action_set_tag *conf = (action->conf); + struct mlx5_modification_cmd *actions = resource->actions; + uint32_t i = resource->actions_num; + + if (i >= MLX5_MODIFY_NUM) + return rte_flow_error_set(error, EINVAL, + RTE_FLOW_ERROR_TYPE_ACTION, NULL, + "too many items to modify"); + actions[i].action_type = MLX5_MODIFICATION_TYPE_SET; + actions[i].field = reg_to_field[conf->id]; + actions[i].data0 = rte_cpu_to_be_32(actions[i].data0); + actions[i].data1 = conf->data; + ++i; + resource->actions_num = i; + if (!resource->actions_num) + return rte_flow_error_set(error, EINVAL, + RTE_FLOW_ERROR_TYPE_ACTION, NULL, + "invalid modification flow item"); + return 0; +} + /** * Validate META item. * @@ -4640,6 +4693,94 @@ struct field_modify_info modify_tcp[] = { } /** + * Add tag item to matcher + * + * @param[in, out] matcher + * Flow matcher. + * @param[in, out] key + * Flow matcher value. + * @param[in] item + * Flow pattern to translate. + */ +static void +flow_dv_translate_item_tag(void *matcher, void *key, + const struct rte_flow_item *item) +{ + void *misc2_m = + MLX5_ADDR_OF(fte_match_param, matcher, misc_parameters_2); + void *misc2_v = + MLX5_ADDR_OF(fte_match_param, key, misc_parameters_2); + const struct mlx5_rte_flow_item_tag *tag_v = item->spec; + const struct mlx5_rte_flow_item_tag *tag_m = item->mask; + enum modify_reg reg = tag_v->id; + rte_be32_t value = tag_v->data; + rte_be32_t mask = tag_m->data; + + switch (reg) { + case REG_A: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_a, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_a, + rte_be_to_cpu_32(value)); + break; + case REG_B: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_b, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_b, + rte_be_to_cpu_32(value)); + break; + case REG_C_0: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_0, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_0, + rte_be_to_cpu_32(value)); + break; + case REG_C_1: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_1, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_1, + rte_be_to_cpu_32(value)); + break; + case REG_C_2: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_2, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_2, + rte_be_to_cpu_32(value)); + break; + case REG_C_3: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_3, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_3, + rte_be_to_cpu_32(value)); + break; + case REG_C_4: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_4, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_4, + rte_be_to_cpu_32(value)); + break; + case REG_C_5: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_5, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_5, + rte_be_to_cpu_32(value)); + break; + case REG_C_6: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_6, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_6, + rte_be_to_cpu_32(value)); + break; + case REG_C_7: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_7, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_7, + rte_be_to_cpu_32(value)); + break; + } +} + +/** * Add source vport match to the specified matcher. * * @param[in, out] matcher @@ -5225,8 +5366,9 @@ struct field_modify_info modify_tcp[] = { struct mlx5_flow_tbl_resource *tbl; uint32_t port_id = 0; struct mlx5_flow_dv_port_id_action_resource port_id_resource; + int action_type = actions->type; - switch (actions->type) { + switch (action_type) { case RTE_FLOW_ACTION_TYPE_VOID: break; case RTE_FLOW_ACTION_TYPE_PORT_ID: @@ -5541,6 +5683,12 @@ struct field_modify_info modify_tcp[] = { MLX5_FLOW_ACTION_INC_TCP_ACK : MLX5_FLOW_ACTION_DEC_TCP_ACK; break; + case MLX5_RTE_FLOW_ACTION_TYPE_TAG: + if (flow_dv_convert_action_set_reg(&res, actions, + error)) + return -rte_errno; + action_flags |= MLX5_FLOW_ACTION_SET_TAG; + break; case RTE_FLOW_ACTION_TYPE_END: actions_end = true; if (action_flags & MLX5_FLOW_MODIFY_HDR_ACTIONS) { @@ -5565,8 +5713,9 @@ struct field_modify_info modify_tcp[] = { flow->actions = action_flags; for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) { int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); + int item_type = items->type; - switch (items->type) { + switch (item_type) { case RTE_FLOW_ITEM_TYPE_PORT_ID: flow_dv_translate_item_port_id(dev, match_mask, match_value, items); @@ -5712,6 +5861,11 @@ struct field_modify_info modify_tcp[] = { items, tunnel); last_item = MLX5_FLOW_LAYER_ICMP6; break; + case MLX5_RTE_FLOW_ITEM_TYPE_TAG: + flow_dv_translate_item_tag(match_mask, match_value, + items); + last_item = MLX5_FLOW_ITEM_TAG; + break; default: break; } diff --git a/drivers/net/mlx5/mlx5_prm.h b/drivers/net/mlx5/mlx5_prm.h index d4084db..695578f 100644 --- a/drivers/net/mlx5/mlx5_prm.h +++ b/drivers/net/mlx5/mlx5_prm.h @@ -623,7 +623,8 @@ struct mlx5_ifc_fte_match_set_misc2_bits { u8 metadata_reg_c_1[0x20]; u8 metadata_reg_c_0[0x20]; u8 metadata_reg_a[0x20]; - u8 reserved_at_1a0[0x60]; + u8 metadata_reg_b[0x20]; + u8 reserved_at_1c0[0x40]; }; struct mlx5_ifc_fte_match_set_misc3_bits { From patchwork Thu Sep 26 06:29:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59790 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 81BE41BEC4; Thu, 26 Sep 2019 08:30:24 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id CF07B4CE4 for ; Thu, 26 Sep 2019 08:30:05 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:30:04 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlP012493; Thu, 26 Sep 2019 09:30:04 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:06 +0000 Message-Id: <1569479349-36962-11-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 10/13] net/mlx5: add id generation function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When splitting flows for example in hairpin / metering, there is a need to combine the flows. This is done using ID. This commit introduce a simple way to generate such IDs. The reason why bitmap was not used is due to fact that the release and allocation are O(n) while in the chosen approch the allocation and release are O(1) Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 120 ++++++++++++++++++++++++++++++++++++++++++- drivers/net/mlx5/mlx5_flow.h | 14 +++++ 2 files changed, 133 insertions(+), 1 deletion(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index ad36743..940503d 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -179,6 +179,124 @@ struct mlx5_dev_spawn_data { static LIST_HEAD(, mlx5_ibv_shared) mlx5_ibv_list = LIST_HEAD_INITIALIZER(); static pthread_mutex_t mlx5_ibv_list_mutex = PTHREAD_MUTEX_INITIALIZER; +#define MLX5_FLOW_MIN_ID_POOL_SIZE 512 +#define MLX5_ID_GENERATION_ARRAY_FACTOR 16 + +/** + * Allocate ID pool structure. + * + * @return + * Pointer to pool object, NULL value otherwise. + */ +struct mlx5_flow_id_pool * +mlx5_flow_id_pool_alloc(void) +{ + struct mlx5_flow_id_pool *pool; + void *mem; + + pool = rte_zmalloc("id pool allocation", sizeof(*pool), + RTE_CACHE_LINE_SIZE); + if (!pool) { + DRV_LOG(ERR, "can't allocate id pool"); + rte_errno = ENOMEM; + return NULL; + } + mem = rte_zmalloc("", MLX5_FLOW_MIN_ID_POOL_SIZE * sizeof(uint32_t), + RTE_CACHE_LINE_SIZE); + if (!mem) { + DRV_LOG(ERR, "can't allocate mem for id pool"); + rte_errno = ENOMEM; + goto error; + } + pool->free_arr = mem; + pool->curr = pool->free_arr; + pool->last = pool->free_arr + MLX5_FLOW_MIN_ID_POOL_SIZE; + pool->base_index = 0; + return pool; +error: + rte_free(pool); + return NULL; +} + +/** + * Release ID pool structure. + * + * @param[in] pool + * Pointer to flow id pool object to free. + */ +void +mlx5_flow_id_pool_release(struct mlx5_flow_id_pool *pool) +{ + rte_free(pool->free_arr); + rte_free(pool); +} + +/** + * Generate ID. + * + * @param[in] pool + * Pointer to flow id pool. + * @param[out] id + * The generated ID. + * + * @return + * 0 on success, error value otherwise. + */ +uint32_t +mlx5_flow_id_get(struct mlx5_flow_id_pool *pool, uint32_t *id) +{ + if (pool->curr == pool->free_arr) { + if (pool->base_index == UINT32_MAX) { + rte_errno = ENOMEM; + DRV_LOG(ERR, "no free id"); + return -rte_errno; + } + *id = ++pool->base_index; + return 0; + } + *id = *(--pool->curr); + return 0; +} + +/** + * Release ID. + * + * @param[in] pool + * Pointer to flow id pool. + * @param[out] id + * The generated ID. + * + * @return + * 0 on success, error value otherwise. + */ +uint32_t +mlx5_flow_id_release(struct mlx5_flow_id_pool *pool, uint32_t id) +{ + uint32_t size; + uint32_t size2; + void *mem; + + if (pool->curr == pool->last) { + size = pool->curr - pool->free_arr; + size2 = size * MLX5_ID_GENERATION_ARRAY_FACTOR; + assert(size2 > size); + mem = rte_malloc("", size2 * sizeof(uint32_t), 0); + if (!mem) { + DRV_LOG(ERR, "can't allocate mem for id pool"); + rte_errno = ENOMEM; + return -rte_errno; + } + memcpy(mem, pool->free_arr, size * sizeof(uint32_t)); + rte_free(pool->free_arr); + pool->free_arr = mem; + pool->curr = pool->free_arr + size; + pool->last = pool->free_arr + size2; + } + *pool->curr = id; + pool->curr++; + return 0; +} + /** * Initialize the counters management structure. * @@ -329,7 +447,7 @@ struct mlx5_dev_spawn_data { struct mlx5_devx_tis_attr tis_attr = { 0 }; #endif - assert(spawn); +assert(spawn); /* Secondary process should not create the shared context. */ assert(rte_eal_process_type() == RTE_PROC_PRIMARY); pthread_mutex_lock(&mlx5_ibv_list_mutex); diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 0148c1b..1b14fb7 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -495,8 +495,22 @@ struct mlx5_flow_driver_ops { #define MLX5_CNT_CONTAINER_UNUSED(sh, batch, thread) (&(sh)->cmng.ccont \ [(~((sh)->cmng.mhi[batch] >> (thread)) & 0x1) * 2 + (batch)]) +/* ID generation structure. */ +struct mlx5_flow_id_pool { + uint32_t *free_arr; /**< Pointer to the a array of free values. */ + uint32_t base_index; + /**< The next index that can be used without any free elements. */ + uint32_t *curr; /**< Pointer to the index to pop. */ + uint32_t *last; /**< Pointer to the last element in the empty arrray. */ +}; + /* mlx5_flow.c */ +struct mlx5_flow_id_pool *mlx5_flow_id_pool_alloc(void); +void mlx5_flow_id_pool_release(struct mlx5_flow_id_pool *pool); +uint32_t mlx5_flow_id_get(struct mlx5_flow_id_pool *pool, uint32_t *id); +uint32_t mlx5_flow_id_release(struct mlx5_flow_id_pool *pool, + uint32_t id); int mlx5_flow_group_to_table(const struct rte_flow_attr *attributes, bool external, uint32_t group, uint32_t *table, struct rte_flow_error *error); From patchwork Thu Sep 26 06:29:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59793 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E44181BEE7; Thu, 26 Sep 2019 08:30:29 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 6C6571BE0C for ; Thu, 26 Sep 2019 08:30:09 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:30:05 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlQ012493; Thu, 26 Sep 2019 09:30:05 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:07 +0000 Message-Id: <1569479349-36962-12-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 11/13] net/mlx5: add default flows for hairpin X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When using hairpin all traffic from TX hairpin queues should jump to dedecated table where matching can be done using regesters. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.h | 2 ++ drivers/net/mlx5/mlx5_flow.c | 60 +++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 9 ++++++ drivers/net/mlx5/mlx5_flow_dv.c | 63 +++++++++++++++++++++++++++++++++++++++-- drivers/net/mlx5/mlx5_trigger.c | 18 ++++++++++++ 5 files changed, 150 insertions(+), 2 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 41eb35a..5f1a25d 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -556,6 +556,7 @@ struct mlx5_flow_tbl_resource { }; #define MLX5_MAX_TABLES UINT16_MAX +#define MLX5_HAIRPIN_TX_TABLE (UINT16_MAX - 1) #define MLX5_MAX_TABLES_FDB UINT16_MAX #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */ @@ -872,6 +873,7 @@ int mlx5_dev_filter_ctrl(struct rte_eth_dev *dev, int mlx5_flow_start(struct rte_eth_dev *dev, struct mlx5_flows *list); void mlx5_flow_stop(struct rte_eth_dev *dev, struct mlx5_flows *list); int mlx5_flow_verify(struct rte_eth_dev *dev); +int mlx5_ctrl_flow_source_queue(struct rte_eth_dev *dev, uint32_t queue); int mlx5_ctrl_flow_vlan(struct rte_eth_dev *dev, struct rte_flow_item_eth *eth_spec, struct rte_flow_item_eth *eth_mask, diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index 00afc18..33ed204 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -2712,6 +2712,66 @@ struct rte_flow * } /** + * Enable default hairpin egress flow. + * + * @param dev + * Pointer to Ethernet device. + * @param queue + * The queue index. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_ctrl_flow_source_queue(struct rte_eth_dev *dev, + uint32_t queue) +{ + struct mlx5_priv *priv = dev->data->dev_private; + const struct rte_flow_attr attr = { + .egress = 1, + .priority = 0, + }; + struct mlx5_rte_flow_item_tx_queue queue_spec = { + .queue = queue, + }; + struct mlx5_rte_flow_item_tx_queue queue_mask = { + .queue = UINT32_MAX, + }; + struct rte_flow_item items[] = { + { + .type = MLX5_RTE_FLOW_ITEM_TYPE_TX_QUEUE, + .spec = &queue_spec, + .last = NULL, + .mask = &queue_mask, + }, + { + .type = RTE_FLOW_ITEM_TYPE_END, + }, + }; + struct rte_flow_action_jump jump = { + .group = MLX5_HAIRPIN_TX_TABLE, + }; + struct rte_flow_action actions[2]; + struct rte_flow *flow; + struct rte_flow_error error; + + actions[0].type = RTE_FLOW_ACTION_TYPE_JUMP; + actions[0].conf = &jump; + actions[1].type = RTE_FLOW_ACTION_TYPE_END; + flow = flow_list_create(dev, &priv->ctrl_flows, + &attr, items, actions, false, &error); + if (!flow) { + DRV_LOG(DEBUG, + "Failed to create ctrl flow: rte_errno(%d)," + " type(%d), message(%s)\n", + rte_errno, error.type, + error.message ? error.message : " (no stated reason)"); + return -rte_errno; + } + return 0; +} + +/** * Enable a control flow configured from the control plane. * * @param dev diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 1b14fb7..bb67380 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -44,6 +44,7 @@ enum modify_reg { enum mlx5_rte_flow_item_type { MLX5_RTE_FLOW_ITEM_TYPE_END = INT_MIN, MLX5_RTE_FLOW_ITEM_TYPE_TAG, + MLX5_RTE_FLOW_ITEM_TYPE_TX_QUEUE, }; /* Private rte flow actions. */ @@ -64,6 +65,11 @@ struct mlx5_rte_flow_action_set_tag { rte_be32_t data; }; +/* Matches on source queue. */ +struct mlx5_rte_flow_item_tx_queue { + uint32_t queue; +}; + /* Pattern outer Layer bits. */ #define MLX5_FLOW_LAYER_OUTER_L2 (1u << 0) #define MLX5_FLOW_LAYER_OUTER_L3_IPV4 (1u << 1) @@ -102,6 +108,9 @@ struct mlx5_rte_flow_action_set_tag { #define MLX5_FLOW_LAYER_IPV6_ENCAP (1u << 23) #define MLX5_FLOW_LAYER_NVGRE (1u << 24) +/* Queue items. */ +#define MLX5_FLOW_ITEM_TX_QUEUE (1u << 25) + /* Outer Masks. */ #define MLX5_FLOW_LAYER_OUTER_L3 \ (MLX5_FLOW_LAYER_OUTER_L3_IPV4 | MLX5_FLOW_LAYER_OUTER_L3_IPV6) diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index dde0831..2b48680 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -3357,7 +3357,9 @@ struct field_modify_info modify_tcp[] = { return ret; for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) { int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); - switch (items->type) { + int type = items->type; + + switch (type) { case RTE_FLOW_ITEM_TYPE_VOID: break; case RTE_FLOW_ITEM_TYPE_PORT_ID: @@ -3518,6 +3520,9 @@ struct field_modify_info modify_tcp[] = { return ret; last_item = MLX5_FLOW_LAYER_ICMP6; break; + case MLX5_RTE_FLOW_ITEM_TYPE_TAG: + case MLX5_RTE_FLOW_ITEM_TYPE_TX_QUEUE: + break; default: return rte_flow_error_set(error, ENOTSUP, RTE_FLOW_ERROR_TYPE_ITEM, @@ -3526,11 +3531,12 @@ struct field_modify_info modify_tcp[] = { item_flags |= last_item; } for (; actions->type != RTE_FLOW_ACTION_TYPE_END; actions++) { + int type = actions->type; if (actions_n == MLX5_DV_MAX_NUMBER_OF_ACTIONS) return rte_flow_error_set(error, ENOTSUP, RTE_FLOW_ERROR_TYPE_ACTION, actions, "too many actions"); - switch (actions->type) { + switch (type) { case RTE_FLOW_ACTION_TYPE_VOID: break; case RTE_FLOW_ACTION_TYPE_PORT_ID: @@ -3796,6 +3802,8 @@ struct field_modify_info modify_tcp[] = { MLX5_FLOW_ACTION_INC_TCP_ACK : MLX5_FLOW_ACTION_DEC_TCP_ACK; break; + case MLX5_RTE_FLOW_ACTION_TYPE_TAG: + break; default: return rte_flow_error_set(error, ENOTSUP, RTE_FLOW_ERROR_TYPE_ACTION, @@ -5291,6 +5299,51 @@ struct field_modify_info modify_tcp[] = { } /** + * Add Tx queue matcher + * + * @param[in] dev + * Pointer to the dev struct. + * @param[in, out] matcher + * Flow matcher. + * @param[in, out] key + * Flow matcher value. + * @param[in] item + * Flow pattern to translate. + * @param[in] inner + * Item is inner pattern. + */ +static void +flow_dv_translate_item_tx_queue(struct rte_eth_dev *dev, + void *matcher, void *key, + const struct rte_flow_item *item) +{ + const struct mlx5_rte_flow_item_tx_queue *queue_m; + const struct mlx5_rte_flow_item_tx_queue *queue_v; + void *misc_m = + MLX5_ADDR_OF(fte_match_param, matcher, misc_parameters); + void *misc_v = + MLX5_ADDR_OF(fte_match_param, key, misc_parameters); + struct mlx5_txq_ctrl *txq; + uint32_t queue; + + + queue_m = (const void *)item->mask; + if (!queue_m) + return; + queue_v = (const void *)item->spec; + if (!queue_v) + return; + txq = mlx5_txq_get(dev, queue_v->queue); + if (!txq) + return; + queue = txq->obj->sq->id; + MLX5_SET(fte_match_set_misc, misc_m, source_sqn, queue_m->queue); + MLX5_SET(fte_match_set_misc, misc_v, source_sqn, + queue & queue_m->queue); + mlx5_txq_release(dev, queue_v->queue); +} + +/** * Fill the flow with DV spec. * * @param[in] dev @@ -5866,6 +5919,12 @@ struct field_modify_info modify_tcp[] = { items); last_item = MLX5_FLOW_ITEM_TAG; break; + case MLX5_RTE_FLOW_ITEM_TYPE_TX_QUEUE: + flow_dv_translate_item_tx_queue(dev, match_mask, + match_value, + items); + last_item = MLX5_FLOW_ITEM_TX_QUEUE; + break; default: break; } diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index a4fcdb3..a476cd5 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -396,6 +396,24 @@ unsigned int j; int ret; + /* + * Hairpin txq default flow should be created no matter if it is + * isolation mode. Or else all the packets to be sent will be sent + * out directly without the TX flow actions, e.g. encapsulation. + */ + for (i = 0; i != priv->txqs_n; ++i) { + struct mlx5_txq_ctrl *txq_ctrl = mlx5_txq_get(dev, i); + if (!txq_ctrl) + continue; + if (txq_ctrl->type == MLX5_TXQ_TYPE_HAIRPIN) { + ret = mlx5_ctrl_flow_source_queue(dev, i); + if (ret) { + mlx5_txq_release(dev, i); + goto error; + } + } + mlx5_txq_release(dev, i); + } if (priv->config.dv_esw_en && !priv->config.vf) if (!mlx5_flow_create_esw_table_zero_flow(dev)) goto error; From patchwork Thu Sep 26 06:29:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59791 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BBBEF1BED5; Thu, 26 Sep 2019 08:30:26 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 760731BE3D for ; Thu, 26 Sep 2019 08:30:09 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:30:06 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlR012493; Thu, 26 Sep 2019 09:30:06 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:08 +0000 Message-Id: <1569479349-36962-13-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 12/13] net/mlx5: split hairpin flows X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Since the encap action is not supported in RX, we need to split the hairpin flow into RX and TX. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 10 ++ drivers/net/mlx5/mlx5.h | 10 ++ drivers/net/mlx5/mlx5_flow.c | 281 +++++++++++++++++++++++++++++++++++-- drivers/net/mlx5/mlx5_flow.h | 14 +- drivers/net/mlx5/mlx5_flow_dv.c | 10 +- drivers/net/mlx5/mlx5_flow_verbs.c | 11 +- drivers/net/mlx5/mlx5_rxq.c | 26 ++++ drivers/net/mlx5/mlx5_rxtx.h | 2 + 8 files changed, 334 insertions(+), 30 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 940503d..2837cba 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -528,6 +528,12 @@ struct mlx5_flow_id_pool * err = ENOMEM; goto error; } + sh->flow_id_pool = mlx5_flow_id_pool_alloc(); + if (!sh->flow_id_pool) { + DRV_LOG(ERR, "can't create flow id pool"); + err = ENOMEM; + goto error; + } #endif /* HAVE_IBV_FLOW_DV_SUPPORT */ /* * Once the device is added to the list of memory event @@ -567,6 +573,8 @@ struct mlx5_flow_id_pool * claim_zero(mlx5_glue->dealloc_pd(sh->pd)); if (sh->ctx) claim_zero(mlx5_glue->close_device(sh->ctx)); + if (sh->flow_id_pool) + mlx5_flow_id_pool_release(sh->flow_id_pool); rte_free(sh); assert(err > 0); rte_errno = err; @@ -629,6 +637,8 @@ struct mlx5_flow_id_pool * claim_zero(mlx5_devx_cmd_destroy(sh->td)); if (sh->ctx) claim_zero(mlx5_glue->close_device(sh->ctx)); + if (sh->flow_id_pool) + mlx5_flow_id_pool_release(sh->flow_id_pool); rte_free(sh); exit: pthread_mutex_unlock(&mlx5_ibv_list_mutex); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 5f1a25d..5336554 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -574,6 +574,15 @@ struct mlx5_devx_dbr_page { uint64_t dbr_bitmap[MLX5_DBR_BITMAP_SIZE]; }; +/* ID generation structure. */ +struct mlx5_flow_id_pool { + uint32_t *free_arr; /**< Pointer to the a array of free values. */ + uint32_t base_index; + /**< The next index that can be used without any free elements. */ + uint32_t *curr; /**< Pointer to the index to pop. */ + uint32_t *last; /**< Pointer to the last element in the empty arrray. */ +}; + /* * Shared Infiniband device context for Master/Representors * which belong to same IB device with multiple IB ports. @@ -632,6 +641,7 @@ struct mlx5_ibv_shared { struct mlx5dv_devx_cmd_comp *devx_comp; /* DEVX async comp obj. */ struct mlx5_devx_obj *tis; /* TIS object. */ struct mlx5_devx_obj *td; /* Transport domain. */ + struct mlx5_flow_id_pool *flow_id_pool; /* Flow ID pool. */ struct mlx5_ibv_shared_port port[]; /* per device port data array. */ }; diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index 33ed204..50e1d11 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -606,7 +606,7 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, { struct mlx5_priv *priv = dev->data->dev_private; struct rte_flow *flow = dev_flow->flow; - const int mark = !!(flow->actions & + const int mark = !!(dev_flow->actions & (MLX5_FLOW_ACTION_FLAG | MLX5_FLOW_ACTION_MARK)); const int tunnel = !!(dev_flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int i; @@ -669,7 +669,7 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, { struct mlx5_priv *priv = dev->data->dev_private; struct rte_flow *flow = dev_flow->flow; - const int mark = !!(flow->actions & + const int mark = !!(dev_flow->actions & (MLX5_FLOW_ACTION_FLAG | MLX5_FLOW_ACTION_MARK)); const int tunnel = !!(dev_flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int i; @@ -2419,6 +2419,210 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, } /** + * Check if the flow should be splited due to hairpin. + * The reason for the split is that in current HW we can't + * support encap on Rx, so if a flow have encap we move it + * to Tx. + * + * @param dev + * Pointer to Ethernet device. + * @param[in] attr + * Flow rule attributes. + * @param[in] actions + * Associated actions (list terminated by the END action). + * + * @return + * > 0 the number of actions and the flow should be split, + * 0 when no split required. + */ +static int +flow_check_hairpin_split(struct rte_eth_dev *dev, + const struct rte_flow_attr *attr, + const struct rte_flow_action actions[]) +{ + int queue_action = 0; + int action_n = 0; + int encap = 0; + const struct rte_flow_action_queue *queue; + const struct rte_flow_action_rss *rss; + const struct rte_flow_action_raw_encap *raw_encap; + + if (!attr->ingress) + return 0; + for (; actions->type != RTE_FLOW_ACTION_TYPE_END; actions++) { + switch (actions->type) { + case RTE_FLOW_ACTION_TYPE_QUEUE: + queue = actions->conf; + if (mlx5_rxq_get_type(dev, queue->index) != + MLX5_RXQ_TYPE_HAIRPIN) + return 0; + queue_action = 1; + action_n++; + break; + case RTE_FLOW_ACTION_TYPE_RSS: + rss = actions->conf; + if (mlx5_rxq_get_type(dev, rss->queue[0]) != + MLX5_RXQ_TYPE_HAIRPIN) + return 0; + queue_action = 1; + action_n++; + break; + case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP: + case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP: + encap = 1; + action_n++; + break; + case RTE_FLOW_ACTION_TYPE_RAW_ENCAP: + raw_encap = actions->conf; + if (raw_encap->size > + (sizeof(struct rte_flow_item_eth) + + sizeof(struct rte_flow_item_ipv4))) + encap = 1; + action_n++; + break; + default: + action_n++; + break; + } + } + if (encap == 1 && queue_action) + return action_n; + return 0; +} + +#define MLX5_MAX_SPLIT_ACTIONS 24 +#define MLX5_MAX_SPLIT_ITEMS 24 + +/** + * Split the hairpin flow. + * Since HW can't support encap on Rx we move the encap to Tx. + * If the count action is after the encap then we also + * move the count action. in this case the count will also measure + * the outer bytes. + * + * @param dev + * Pointer to Ethernet device. + * @param[in] actions + * Associated actions (list terminated by the END action). + * @param[out] actions_rx + * Rx flow actions. + * @param[out] actions_tx + * Tx flow actions.. + * @param[out] pattern_tx + * The pattern items for the Tx flow. + * @param[out] flow_id + * The flow ID connected to this flow. + * + * @return + * 0 on success. + */ +static int +flow_hairpin_split(struct rte_eth_dev *dev, + const struct rte_flow_action actions[], + struct rte_flow_action actions_rx[], + struct rte_flow_action actions_tx[], + struct rte_flow_item pattern_tx[], + uint32_t *flow_id) +{ + struct mlx5_priv *priv = dev->data->dev_private; + const struct rte_flow_action_raw_encap *raw_encap; + const struct rte_flow_action_raw_decap *raw_decap; + struct mlx5_rte_flow_action_set_tag *set_tag; + struct rte_flow_action *tag_action; + struct mlx5_rte_flow_item_tag *tag_item; + struct rte_flow_item *item; + char *addr; + struct rte_flow_error error; + int encap = 0; + + mlx5_flow_id_get(priv->sh->flow_id_pool, flow_id); + for (; actions->type != RTE_FLOW_ACTION_TYPE_END; actions++) { + switch (actions->type) { + case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP: + case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP: + rte_memcpy(actions_tx, actions, + sizeof(struct rte_flow_action)); + actions_tx++; + break; + case RTE_FLOW_ACTION_TYPE_COUNT: + if (encap) { + rte_memcpy(actions_tx, actions, + sizeof(struct rte_flow_action)); + actions_tx++; + } else { + rte_memcpy(actions_rx, actions, + sizeof(struct rte_flow_action)); + actions_rx++; + } + break; + case RTE_FLOW_ACTION_TYPE_RAW_ENCAP: + raw_encap = actions->conf; + if (raw_encap->size > + (sizeof(struct rte_flow_item_eth) + + sizeof(struct rte_flow_item_ipv4))) { + memcpy(actions_tx, actions, + sizeof(struct rte_flow_action)); + actions_tx++; + encap = 1; + } else { + rte_memcpy(actions_rx, actions, + sizeof(struct rte_flow_action)); + actions_rx++; + } + break; + case RTE_FLOW_ACTION_TYPE_RAW_DECAP: + raw_decap = actions->conf; + if (raw_decap->size < + (sizeof(struct rte_flow_item_eth) + + sizeof(struct rte_flow_item_ipv4))) { + memcpy(actions_tx, actions, + sizeof(struct rte_flow_action)); + actions_tx++; + } else { + rte_memcpy(actions_rx, actions, + sizeof(struct rte_flow_action)); + actions_rx++; + } + break; + default: + rte_memcpy(actions_rx, actions, + sizeof(struct rte_flow_action)); + actions_rx++; + break; + } + } + /* Add set meta action and end action for the Rx flow. */ + tag_action = actions_rx; + tag_action->type = MLX5_RTE_FLOW_ACTION_TYPE_TAG; + actions_rx++; + rte_memcpy(actions_rx, actions, sizeof(struct rte_flow_action)); + actions_rx++; + set_tag = (void *)actions_rx; + set_tag->id = flow_get_reg_id(dev, MLX5_HAIRPIN_RX, 0, &error); + set_tag->data = rte_cpu_to_be_32(*flow_id); + tag_action->conf = set_tag; + /* Create Tx item list. */ + rte_memcpy(actions_tx, actions, sizeof(struct rte_flow_action)); + addr = (void *)&pattern_tx[2]; + item = pattern_tx; + item->type = MLX5_RTE_FLOW_ITEM_TYPE_TAG; + tag_item = (void *)addr; + tag_item->data = rte_cpu_to_be_32(*flow_id); + tag_item->id = flow_get_reg_id(dev, MLX5_HAIRPIN_TX, 0, &error); + item->spec = tag_item; + addr += sizeof(struct mlx5_rte_flow_item_tag); + tag_item = (void *)addr; + tag_item->data = UINT32_MAX; + tag_item->id = UINT16_MAX; + item->mask = tag_item; + addr += sizeof(struct mlx5_rte_flow_item_tag); + item->last = NULL; + item++; + item->type = RTE_FLOW_ITEM_TYPE_END; + return 0; +} + +/** * Create a flow and add it to @p list. * * @param dev @@ -2446,6 +2650,7 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, const struct rte_flow_action actions[], bool external, struct rte_flow_error *error) { + struct mlx5_priv *priv = dev->data->dev_private; struct rte_flow *flow = NULL; struct mlx5_flow *dev_flow; const struct rte_flow_action_rss *rss; @@ -2453,16 +2658,44 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, struct rte_flow_expand_rss buf; uint8_t buffer[2048]; } expand_buffer; + union { + struct rte_flow_action actions[MLX5_MAX_SPLIT_ACTIONS]; + uint8_t buffer[2048]; + } actions_rx; + union { + struct rte_flow_action actions[MLX5_MAX_SPLIT_ACTIONS]; + uint8_t buffer[2048]; + } actions_hairpin_tx; + union { + struct rte_flow_item items[MLX5_MAX_SPLIT_ITEMS]; + uint8_t buffer[2048]; + } items_tx; struct rte_flow_expand_rss *buf = &expand_buffer.buf; + const struct rte_flow_action *p_actions_rx = actions; int ret; uint32_t i; uint32_t flow_size; + int hairpin_flow = 0; + uint32_t hairpin_id = 0; + struct rte_flow_attr attr_tx = { .priority = 0 }; - ret = flow_drv_validate(dev, attr, items, actions, external, error); + hairpin_flow = flow_check_hairpin_split(dev, attr, actions); + if (hairpin_flow > 0) { + if (hairpin_flow > MLX5_MAX_SPLIT_ACTIONS) { + rte_errno = EINVAL; + return NULL; + } + flow_hairpin_split(dev, actions, actions_rx.actions, + actions_hairpin_tx.actions, items_tx.items, + &hairpin_id); + p_actions_rx = actions_rx.actions; + } + ret = flow_drv_validate(dev, attr, items, p_actions_rx, external, + error); if (ret < 0) - return NULL; + goto error_before_flow; flow_size = sizeof(struct rte_flow); - rss = flow_get_rss_action(actions); + rss = flow_get_rss_action(p_actions_rx); if (rss) flow_size += RTE_ALIGN_CEIL(rss->queue_num * sizeof(uint16_t), sizeof(void *)); @@ -2471,11 +2704,13 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, flow = rte_calloc(__func__, 1, flow_size, 0); if (!flow) { rte_errno = ENOMEM; - return NULL; + goto error_before_flow; } flow->drv_type = flow_get_drv_type(dev, attr); flow->ingress = attr->ingress; flow->transfer = attr->transfer; + if (hairpin_id != 0) + flow->hairpin_flow_id = hairpin_id; assert(flow->drv_type > MLX5_FLOW_TYPE_MIN && flow->drv_type < MLX5_FLOW_TYPE_MAX); flow->queue = (void *)(flow + 1); @@ -2496,7 +2731,7 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, } for (i = 0; i < buf->entries; ++i) { dev_flow = flow_drv_prepare(flow, attr, buf->entry[i].pattern, - actions, error); + p_actions_rx, error); if (!dev_flow) goto error; dev_flow->flow = flow; @@ -2504,7 +2739,24 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, LIST_INSERT_HEAD(&flow->dev_flows, dev_flow, next); ret = flow_drv_translate(dev, dev_flow, attr, buf->entry[i].pattern, - actions, error); + p_actions_rx, error); + if (ret < 0) + goto error; + } + /* Create the tx flow. */ + if (hairpin_flow) { + attr_tx.group = MLX5_HAIRPIN_TX_TABLE; + attr_tx.ingress = 0; + attr_tx.egress = 1; + dev_flow = flow_drv_prepare(flow, &attr_tx, items_tx.items, + actions_hairpin_tx.actions, error); + if (!dev_flow) + goto error; + dev_flow->flow = flow; + LIST_INSERT_HEAD(&flow->dev_flows, dev_flow, next); + ret = flow_drv_translate(dev, dev_flow, &attr_tx, + items_tx.items, + actions_hairpin_tx.actions, error); if (ret < 0) goto error; } @@ -2516,8 +2768,16 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, TAILQ_INSERT_TAIL(list, flow, next); flow_rxq_flags_set(dev, flow); return flow; +error_before_flow: + if (hairpin_id) + mlx5_flow_id_release(priv->sh->flow_id_pool, + hairpin_id); + return NULL; error: ret = rte_errno; /* Save rte_errno before cleanup. */ + if (flow->hairpin_flow_id) + mlx5_flow_id_release(priv->sh->flow_id_pool, + flow->hairpin_flow_id); assert(flow); flow_drv_destroy(dev, flow); rte_free(flow); @@ -2607,12 +2867,17 @@ struct rte_flow * flow_list_destroy(struct rte_eth_dev *dev, struct mlx5_flows *list, struct rte_flow *flow) { + struct mlx5_priv *priv = dev->data->dev_private; + /* * Update RX queue flags only if port is started, otherwise it is * already clean. */ if (dev->data->dev_started) flow_rxq_flags_trim(dev, flow); + if (flow->hairpin_flow_id) + mlx5_flow_id_release(priv->sh->flow_id_pool, + flow->hairpin_flow_id); flow_drv_destroy(dev, flow); TAILQ_REMOVE(list, flow, next); rte_free(flow->fdir); diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index bb67380..90a289e 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -434,6 +434,8 @@ struct mlx5_flow { struct rte_flow *flow; /**< Pointer to the main flow. */ uint64_t layers; /**< Bit-fields of present layers, see MLX5_FLOW_LAYER_*. */ + uint64_t actions; + /**< Bit-fields of detected actions, see MLX5_FLOW_ACTION_*. */ union { #ifdef HAVE_IBV_FLOW_DV_SUPPORT struct mlx5_flow_dv dv; @@ -455,12 +457,11 @@ struct rte_flow { uint16_t (*queue)[]; /**< Destination queues to redirect traffic to. */ LIST_HEAD(dev_flows, mlx5_flow) dev_flows; /**< Device flows that are part of the flow. */ - uint64_t actions; - /**< Bit-fields of detected actions, see MLX5_FLOW_ACTION_*. */ struct mlx5_fdir *fdir; /**< Pointer to associated FDIR if any. */ uint8_t ingress; /**< 1 if the flow is ingress. */ uint32_t group; /**< The group index. */ uint8_t transfer; /**< 1 if the flow is E-Switch flow. */ + uint32_t hairpin_flow_id; /**< The flow id used for hairpin. */ }; typedef int (*mlx5_flow_validate_t)(struct rte_eth_dev *dev, @@ -504,15 +505,6 @@ struct mlx5_flow_driver_ops { #define MLX5_CNT_CONTAINER_UNUSED(sh, batch, thread) (&(sh)->cmng.ccont \ [(~((sh)->cmng.mhi[batch] >> (thread)) & 0x1) * 2 + (batch)]) -/* ID generation structure. */ -struct mlx5_flow_id_pool { - uint32_t *free_arr; /**< Pointer to the a array of free values. */ - uint32_t base_index; - /**< The next index that can be used without any free elements. */ - uint32_t *curr; /**< Pointer to the index to pop. */ - uint32_t *last; /**< Pointer to the last element in the empty arrray. */ -}; - /* mlx5_flow.c */ struct mlx5_flow_id_pool *mlx5_flow_id_pool_alloc(void); diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index 2b48680..6828bd1 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -5763,7 +5763,7 @@ struct field_modify_info modify_tcp[] = { modify_action_position = actions_n++; } dev_flow->dv.actions_n = actions_n; - flow->actions = action_flags; + dev_flow->actions = action_flags; for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) { int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); int item_type = items->type; @@ -5985,7 +5985,7 @@ struct field_modify_info modify_tcp[] = { LIST_FOREACH(dev_flow, &flow->dev_flows, next) { dv = &dev_flow->dv; n = dv->actions_n; - if (flow->actions & MLX5_FLOW_ACTION_DROP) { + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) { if (flow->transfer) { dv->actions[n++] = priv->sh->esw_drop_action; } else { @@ -6000,7 +6000,7 @@ struct field_modify_info modify_tcp[] = { } dv->actions[n++] = dv->hrxq->action; } - } else if (flow->actions & + } else if (dev_flow->actions & (MLX5_FLOW_ACTION_QUEUE | MLX5_FLOW_ACTION_RSS)) { struct mlx5_hrxq *hrxq; @@ -6056,7 +6056,7 @@ struct field_modify_info modify_tcp[] = { LIST_FOREACH(dev_flow, &flow->dev_flows, next) { struct mlx5_flow_dv *dv = &dev_flow->dv; if (dv->hrxq) { - if (flow->actions & MLX5_FLOW_ACTION_DROP) + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, dv->hrxq); @@ -6290,7 +6290,7 @@ struct field_modify_info modify_tcp[] = { dv->flow = NULL; } if (dv->hrxq) { - if (flow->actions & MLX5_FLOW_ACTION_DROP) + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, dv->hrxq); diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_flow_verbs.c index 23110f2..fd27f6c 100644 --- a/drivers/net/mlx5/mlx5_flow_verbs.c +++ b/drivers/net/mlx5/mlx5_flow_verbs.c @@ -191,7 +191,7 @@ { #if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42) || \ defined(HAVE_IBV_DEVICE_COUNTERS_SET_V45) - if (flow->actions & MLX5_FLOW_ACTION_COUNT) { + if (flow->counter->cs) { struct rte_flow_query_count *qc = data; uint64_t counters[2] = {0, 0}; #if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42) @@ -1410,7 +1410,6 @@ const struct rte_flow_action actions[], struct rte_flow_error *error) { - struct rte_flow *flow = dev_flow->flow; uint64_t item_flags = 0; uint64_t action_flags = 0; uint64_t priority = attr->priority; @@ -1460,7 +1459,7 @@ "action not supported"); } } - flow->actions = action_flags; + dev_flow->actions = action_flags; for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) { int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); @@ -1592,7 +1591,7 @@ verbs->flow = NULL; } if (verbs->hrxq) { - if (flow->actions & MLX5_FLOW_ACTION_DROP) + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, verbs->hrxq); @@ -1656,7 +1655,7 @@ LIST_FOREACH(dev_flow, &flow->dev_flows, next) { verbs = &dev_flow->verbs; - if (flow->actions & MLX5_FLOW_ACTION_DROP) { + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) { verbs->hrxq = mlx5_hrxq_drop_new(dev); if (!verbs->hrxq) { rte_flow_error_set @@ -1717,7 +1716,7 @@ LIST_FOREACH(dev_flow, &flow->dev_flows, next) { verbs = &dev_flow->verbs; if (verbs->hrxq) { - if (flow->actions & MLX5_FLOW_ACTION_DROP) + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, verbs->hrxq); diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index bf39112..e51a0c6 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -2113,6 +2113,32 @@ struct mlx5_rxq_ctrl * } /** + * Get a Rx queue type. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Rx queue index. + * + * @return + * The Rx queue type. + */ +enum mlx5_rxq_type +mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_ctrl *rxq_ctrl = NULL; + + if ((*priv->rxqs)[idx]) { + rxq_ctrl = container_of((*priv->rxqs)[idx], + struct mlx5_rxq_ctrl, + rxq); + return rxq_ctrl->type; + } + return MLX5_RXQ_TYPE_UNDEFINED; +} + +/** * Create an indirection table. * * @param dev diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 8fa22e5..4707b29 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -166,6 +166,7 @@ enum mlx5_rxq_obj_type { enum mlx5_rxq_type { MLX5_RXQ_TYPE_STANDARD, /* Standard Rx queue. */ MLX5_RXQ_TYPE_HAIRPIN, /* Hairpin Rx queue. */ + MLX5_RXQ_TYPE_UNDEFINED, }; /* Verbs/DevX Rx queue elements. */ @@ -408,6 +409,7 @@ struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev, const uint16_t *queues, uint32_t queues_n); int mlx5_hrxq_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hxrq); int mlx5_hrxq_verify(struct rte_eth_dev *dev); +enum mlx5_rxq_type mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx); struct mlx5_hrxq *mlx5_hrxq_drop_new(struct rte_eth_dev *dev); void mlx5_hrxq_drop_release(struct rte_eth_dev *dev); uint64_t mlx5_get_rx_port_offloads(void); From patchwork Thu Sep 26 06:29:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 59792 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 578571BEDE; Thu, 26 Sep 2019 08:30:28 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 762651BE51 for ; Thu, 26 Sep 2019 08:30:09 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 26 Sep 2019 09:30:07 +0300 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x8Q6TmlS012493; Thu, 26 Sep 2019 09:30:07 +0300 From: Ori Kam To: John McNamara , Marko Kovacevic Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 26 Sep 2019 06:29:09 +0000 Message-Id: <1569479349-36962-14-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1569479349-36962-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH 13/13] doc: add hairpin feature X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit adds the hairpin feature to the release notes. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- doc/guides/rel_notes/release_19_11.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst index c8d97f1..a880655 100644 --- a/doc/guides/rel_notes/release_19_11.rst +++ b/doc/guides/rel_notes/release_19_11.rst @@ -56,6 +56,10 @@ New Features Also, make sure to start the actual text at the margin. ========================================================= +* **Added hairpin queue.** + + On supported nics, we can now setup haipin queue which will offload packets from the wire, + back to the wire. Removed Items ------------- @@ -234,4 +238,5 @@ Tested Platforms * Added support for VLAN push flow offload command. * Added support for VLAN set PCP offload command. * Added support for VLAN set VID offload command. + * Added hairpin support.