From patchwork Thu Oct 17 15:32:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61411 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DAAF61E97B; Thu, 17 Oct 2019 17:32:30 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 996C61E94B for ; Thu, 17 Oct 2019 17:32:25 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:22 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfI026371; Thu, 17 Oct 2019 18:32:22 +0300 From: Ori Kam To: Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:03 +0000 Message-Id: <1571326337-42692-2-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 01/15] ethdev: move queue state defines to private file X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The queue state defines are internal to the DPDK. This commit moves them to a private header file. Signed-off-by: Ori Kam Reviewed-by: Andrew Rybchenko --- V4: - new file, created due to ML comments. --- lib/librte_ethdev/rte_ethdev.h | 6 ------ lib/librte_ethdev/rte_ethdev_driver.h | 6 ++++++ 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h index d937fb4..187a2bb 100644 --- a/lib/librte_ethdev/rte_ethdev.h +++ b/lib/librte_ethdev/rte_ethdev.h @@ -1272,12 +1272,6 @@ struct rte_eth_dcb_info { struct rte_eth_dcb_tc_queue_mapping tc_queue; }; -/** - * RX/TX queue states - */ -#define RTE_ETH_QUEUE_STATE_STOPPED 0 -#define RTE_ETH_QUEUE_STATE_STARTED 1 - #define RTE_ETH_ALL RTE_MAX_ETHPORTS /* Macros to check for valid port */ diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h index 936ff8c..c404f17 100644 --- a/lib/librte_ethdev/rte_ethdev_driver.h +++ b/lib/librte_ethdev/rte_ethdev_driver.h @@ -22,6 +22,12 @@ #endif /** + * RX/TX queue states + */ +#define RTE_ETH_QUEUE_STATE_STOPPED 0 +#define RTE_ETH_QUEUE_STATE_STARTED 1 + +/** * @internal * Returns a ethdev slot specified by the unique identifier name. * From patchwork Thu Oct 17 15:32:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61413 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 215631E9AB; Thu, 17 Oct 2019 17:32:36 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 58ED11E976 for ; Thu, 17 Oct 2019 17:32:29 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:25 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfJ026371; Thu, 17 Oct 2019 18:32:25 +0300 From: Ori Kam To: Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:04 +0000 Message-Id: <1571326337-42692-3-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 02/15] ethdev: add support for hairpin queue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit introduce hairpin queue type. The hairpin queue in build from Rx queue binded to Tx queue. It is used to offload traffic coming from the wire and redirect it back to the wire. There are 3 new functions: - rte_eth_dev_hairpin_capability_get - rte_eth_rx_hairpin_queue_setup - rte_eth_tx_hairpin_queue_setup In order to use the queue, there is a need to create rte_flow with queue / RSS action that targets one or more of the Rx queues. Signed-off-by: Ori Kam --- V4: - update according to ML comments. V3: - update according to ML comments. V2: - update according to ML comments. --- lib/librte_ethdev/rte_ethdev.c | 229 +++++++++++++++++++++++++++++++ lib/librte_ethdev/rte_ethdev.h | 143 ++++++++++++++++++- lib/librte_ethdev/rte_ethdev_core.h | 91 +++++++++++- lib/librte_ethdev/rte_ethdev_driver.h | 1 + lib/librte_ethdev/rte_ethdev_version.map | 5 + 5 files changed, 461 insertions(+), 8 deletions(-) diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c index af82360..10a8bf2 100644 --- a/lib/librte_ethdev/rte_ethdev.c +++ b/lib/librte_ethdev/rte_ethdev.c @@ -904,6 +904,14 @@ struct rte_eth_dev * RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->rx_queue_start, -ENOTSUP); + if (dev->data->rx_queue_state[rx_queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + RTE_ETHDEV_LOG(INFO, + "Queue %"PRIu16" of device with port_id=%"PRIu16" is hairpin queue\n", + rx_queue_id, port_id); + return -EINVAL; + } + if (dev->data->rx_queue_state[rx_queue_id] != RTE_ETH_QUEUE_STATE_STOPPED) { RTE_ETHDEV_LOG(INFO, "Queue %"PRIu16" of device with port_id=%"PRIu16" already started\n", @@ -931,6 +939,14 @@ struct rte_eth_dev * RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->rx_queue_stop, -ENOTSUP); + if (dev->data->rx_queue_state[rx_queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + RTE_ETHDEV_LOG(INFO, + "Queue %"PRIu16" of device with port_id=%"PRIu16" is hairpin queue\n", + rx_queue_id, port_id); + return -EINVAL; + } + if (dev->data->rx_queue_state[rx_queue_id] == RTE_ETH_QUEUE_STATE_STOPPED) { RTE_ETHDEV_LOG(INFO, "Queue %"PRIu16" of device with port_id=%"PRIu16" already stopped\n", @@ -964,6 +980,14 @@ struct rte_eth_dev * RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->tx_queue_start, -ENOTSUP); + if (dev->data->tx_queue_state[tx_queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + RTE_ETHDEV_LOG(INFO, + "Queue %"PRIu16" of device with port_id=%"PRIu16" is hairpin queue\n", + tx_queue_id, port_id); + return -EINVAL; + } + if (dev->data->tx_queue_state[tx_queue_id] != RTE_ETH_QUEUE_STATE_STOPPED) { RTE_ETHDEV_LOG(INFO, "Queue %"PRIu16" of device with port_id=%"PRIu16" already started\n", @@ -989,6 +1013,14 @@ struct rte_eth_dev * RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->tx_queue_stop, -ENOTSUP); + if (dev->data->tx_queue_state[tx_queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + RTE_ETHDEV_LOG(INFO, + "Queue %"PRIu16" of device with port_id=%"PRIu16" is hairpin queue\n", + tx_queue_id, port_id); + return -EINVAL; + } + if (dev->data->tx_queue_state[tx_queue_id] == RTE_ETH_QUEUE_STATE_STOPPED) { RTE_ETHDEV_LOG(INFO, "Queue %"PRIu16" of device with port_id=%"PRIu16" already stopped\n", @@ -1758,6 +1790,81 @@ struct rte_eth_dev * } int +rte_eth_rx_hairpin_queue_setup(uint16_t port_id, uint16_t rx_queue_id, + uint16_t nb_rx_desc, + const struct rte_eth_hairpin_conf *conf) +{ + int ret; + struct rte_eth_dev *dev; + struct rte_eth_hairpin_cap cap; + void **rxq; + int i; + int count = 0; + + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL); + + dev = &rte_eth_devices[port_id]; + if (rx_queue_id >= dev->data->nb_rx_queues) { + RTE_ETHDEV_LOG(ERR, "Invalid RX queue_id=%u\n", rx_queue_id); + return -EINVAL; + } + ret = rte_eth_dev_hairpin_capability_get(port_id, &cap); + if (ret != 0) + return ret; + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->rx_hairpin_queue_setup, + -ENOTSUP); + /* Use default specified by driver, if nb_rx_desc is zero */ + if (nb_rx_desc == 0) + nb_rx_desc = cap.max_nb_desc; + if (nb_rx_desc > cap.max_nb_desc) { + RTE_ETHDEV_LOG(ERR, + "Invalid value for nb_rx_desc(=%hu), should be: " + "<= %hu", nb_rx_desc, cap.max_nb_desc); + return -EINVAL; + } + if (conf->peer_n > cap.max_rx_2_tx) { + RTE_ETHDEV_LOG(ERR, + "Invalid value for number of peers(=%hu), " + "should be: <= %hu", conf->peer_n, + cap.max_rx_2_tx); + return -EINVAL; + } + if (conf->peer_n == 0) { + RTE_ETHDEV_LOG(ERR, + "Invalid value for number of peers(=%hu), " + "should be: > 0", conf->peer_n); + return -EINVAL; + } + if (cap.max_n_queues != UINT16_MAX) { + for (i = 0; i < dev->data->nb_rx_queues; i++) { + if (dev->data->rx_queue_state[i] == + RTE_ETH_QUEUE_STATE_HAIRPIN) + count++; + } + if (count > cap.max_n_queues) { + RTE_ETHDEV_LOG(ERR, + "To many Rx hairpin queues %d", count); + return -EINVAL; + } + } + if (dev->data->dev_started) + return -EBUSY; + rxq = dev->data->rx_queues; + if (rxq[rx_queue_id] != NULL) { + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->rx_queue_release, + -ENOTSUP); + (*dev->dev_ops->rx_queue_release)(rxq[rx_queue_id]); + rxq[rx_queue_id] = NULL; + } + ret = (*dev->dev_ops->rx_hairpin_queue_setup)(dev, rx_queue_id, + nb_rx_desc, conf); + if (ret == 0) + dev->data->rx_queue_state[rx_queue_id] = + RTE_ETH_QUEUE_STATE_HAIRPIN; + return eth_err(port_id, ret); +} + +int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id, uint16_t nb_tx_desc, unsigned int socket_id, const struct rte_eth_txconf *tx_conf) @@ -1856,6 +1963,80 @@ struct rte_eth_dev * tx_queue_id, nb_tx_desc, socket_id, &local_conf)); } +int +rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, + uint16_t nb_tx_desc, + const struct rte_eth_hairpin_conf *conf) +{ + struct rte_eth_dev *dev; + struct rte_eth_hairpin_cap cap; + void **txq; + int i; + int count = 0; + int ret; + + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL); + dev = &rte_eth_devices[port_id]; + if (tx_queue_id >= dev->data->nb_tx_queues) { + RTE_ETHDEV_LOG(ERR, "Invalid TX queue_id=%u\n", tx_queue_id); + return -EINVAL; + } + ret = rte_eth_dev_hairpin_capability_get(port_id, &cap); + if (ret != 0) + return ret; + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->tx_hairpin_queue_setup, + -ENOTSUP); + /* Use default specified by driver, if nb_tx_desc is zero */ + if (nb_tx_desc == 0) + nb_tx_desc = cap.max_nb_desc; + if (nb_tx_desc > cap.max_nb_desc) { + RTE_ETHDEV_LOG(ERR, + "Invalid value for nb_tx_desc(=%hu), should be: " + "<= %hu", nb_tx_desc, cap.max_nb_desc); + return -EINVAL; + } + if (conf->peer_n > cap.max_tx_2_rx) { + RTE_ETHDEV_LOG(ERR, + "Invalid value for number of peers(=%hu), " + "should be: <= %hu", conf->peer_n, + cap.max_tx_2_rx); + return -EINVAL; + } + if (conf->peer_n == 0) { + RTE_ETHDEV_LOG(ERR, + "Invalid value for number of peers(=%hu), " + "should be: > 0", conf->peer_n); + return -EINVAL; + } + if (cap.max_n_queues != UINT16_MAX) { + for (i = 0; i < dev->data->nb_tx_queues; i++) { + if (dev->data->tx_queue_state[i] == + RTE_ETH_QUEUE_STATE_HAIRPIN) + count++; + } + if (count > cap.max_n_queues) { + RTE_ETHDEV_LOG(ERR, + "To many Rx hairpin queues %d", count); + return -EINVAL; + } + } + if (dev->data->dev_started) + return -EBUSY; + txq = dev->data->tx_queues; + if (txq[tx_queue_id] != NULL) { + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->tx_queue_release, + -ENOTSUP); + (*dev->dev_ops->tx_queue_release)(txq[tx_queue_id]); + txq[tx_queue_id] = NULL; + } + ret = (*dev->dev_ops->tx_hairpin_queue_setup) + (dev, tx_queue_id, nb_tx_desc, conf); + if (ret == 0) + dev->data->tx_queue_state[tx_queue_id] = + RTE_ETH_QUEUE_STATE_HAIRPIN; + return eth_err(port_id, ret); +} + void rte_eth_tx_buffer_drop_callback(struct rte_mbuf **pkts, uint16_t unsent, void *userdata __rte_unused) @@ -3981,12 +4162,20 @@ int rte_eth_set_queue_rate_limit(uint16_t port_id, uint16_t queue_idx, rte_errno = ENOTSUP; return NULL; #endif + struct rte_eth_dev *dev; + /* check input parameters */ if (!rte_eth_dev_is_valid_port(port_id) || fn == NULL || queue_id >= rte_eth_devices[port_id].data->nb_rx_queues) { rte_errno = EINVAL; return NULL; } + dev = &rte_eth_devices[port_id]; + if (dev->data->rx_queue_state[queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + rte_errno = EINVAL; + return NULL; + } struct rte_eth_rxtx_callback *cb = rte_zmalloc(NULL, sizeof(*cb), 0); if (cb == NULL) { @@ -4058,6 +4247,8 @@ int rte_eth_set_queue_rate_limit(uint16_t port_id, uint16_t queue_idx, rte_errno = ENOTSUP; return NULL; #endif + struct rte_eth_dev *dev; + /* check input parameters */ if (!rte_eth_dev_is_valid_port(port_id) || fn == NULL || queue_id >= rte_eth_devices[port_id].data->nb_tx_queues) { @@ -4065,6 +4256,13 @@ int rte_eth_set_queue_rate_limit(uint16_t port_id, uint16_t queue_idx, return NULL; } + dev = &rte_eth_devices[port_id]; + if (dev->data->tx_queue_state[queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + rte_errno = EINVAL; + return NULL; + } + struct rte_eth_rxtx_callback *cb = rte_zmalloc(NULL, sizeof(*cb), 0); if (cb == NULL) { @@ -4180,6 +4378,14 @@ int rte_eth_set_queue_rate_limit(uint16_t port_id, uint16_t queue_idx, RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->rxq_info_get, -ENOTSUP); + if (dev->data->rx_queue_state[queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + RTE_ETHDEV_LOG(INFO, + "Queue %"PRIu16" of device with port_id=%"PRIu16" is hairpin queue\n", + queue_id, port_id); + return -EINVAL; + } + memset(qinfo, 0, sizeof(*qinfo)); dev->dev_ops->rxq_info_get(dev, queue_id, qinfo); return 0; @@ -4202,6 +4408,14 @@ int rte_eth_set_queue_rate_limit(uint16_t port_id, uint16_t queue_idx, return -EINVAL; } + if (dev->data->tx_queue_state[queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + RTE_ETHDEV_LOG(INFO, + "Queue %"PRIu16" of device with port_id=%"PRIu16" is hairpin queue\n", + queue_id, port_id); + return -EINVAL; + } + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->txq_info_get, -ENOTSUP); memset(qinfo, 0, sizeof(*qinfo)); @@ -4510,6 +4724,21 @@ int rte_eth_set_queue_rate_limit(uint16_t port_id, uint16_t queue_idx, } int +rte_eth_dev_hairpin_capability_get(uint16_t port_id, + struct rte_eth_hairpin_cap *cap) +{ + struct rte_eth_dev *dev; + + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL); + + dev = &rte_eth_devices[port_id]; + RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->hairpin_cap_get, + -ENOTSUP); + memset(cap, 0, sizeof(*cap)); + return eth_err(port_id, (*dev->dev_ops->hairpin_cap_get)(dev, cap)); +} + +int rte_eth_dev_pool_ops_supported(uint16_t port_id, const char *pool) { struct rte_eth_dev *dev; diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h index 187a2bb..276f55f 100644 --- a/lib/librte_ethdev/rte_ethdev.h +++ b/lib/librte_ethdev/rte_ethdev.h @@ -804,6 +804,46 @@ struct rte_eth_txconf { }; /** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * A structure used to return the hairpin capabilities that are supported. + */ +struct rte_eth_hairpin_cap { + uint16_t max_n_queues; + /**< The max number of hairpin queues (different bindings). */ + uint16_t max_rx_2_tx; + /**< Max number of Rx queues to be connected to one Tx queue. */ + uint16_t max_tx_2_rx; + /**< Max number of Tx queues to be connected to one Rx queue. */ + uint16_t max_nb_desc; /**< The max num of descriptors. */ +}; + +#define RTE_ETH_MAX_HAIRPIN_PEERS 32 + +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * A structure used to hold hairpin peer data. + */ +struct rte_eth_hairpin_peer { + uint16_t port; /**< Peer port. */ + uint16_t queue; /**< Peer queue. */ +}; + +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * A structure used to configure hairpin binding. + */ +struct rte_eth_hairpin_conf { + uint16_t peer_n; /**< The number of peers. */ + struct rte_eth_hairpin_peer peers[RTE_ETH_MAX_HAIRPIN_PEERS]; +}; + +/** * A structure contains information about HW descriptor ring limitations. */ struct rte_eth_desc_lim { @@ -1765,6 +1805,37 @@ int rte_eth_rx_queue_setup(uint16_t port_id, uint16_t rx_queue_id, struct rte_mempool *mb_pool); /** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Allocate and set up a hairpin receive queue for an Ethernet device. + * + * The function set up the selected queue to be used in hairpin. + * + * @param port_id + * The port identifier of the Ethernet device. + * @param rx_queue_id + * The index of the receive queue to set up. + * The value must be in the range [0, nb_rx_queue - 1] previously supplied + * to rte_eth_dev_configure(). + * @param nb_rx_desc + * The number of receive descriptors to allocate for the receive ring. + * 0 means the PMD will use default value. + * @param conf + * The pointer to the hairpin configuration. + * + * @return + * - (0) if successful. + * - (-ENOTSUP) if hardware doesn't support. + * - (-EINVAL) if bad parameter. + * - (-ENOMEM) if unable to allocate the resources. + */ +__rte_experimental +int rte_eth_rx_hairpin_queue_setup + (uint16_t port_id, uint16_t rx_queue_id, uint16_t nb_rx_desc, + const struct rte_eth_hairpin_conf *conf); + +/** * Allocate and set up a transmit queue for an Ethernet device. * * @param port_id @@ -1817,6 +1888,35 @@ int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id, const struct rte_eth_txconf *tx_conf); /** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Allocate and set up a transmit hairpin queue for an Ethernet device. + * + * @param port_id + * The port identifier of the Ethernet device. + * @param tx_queue_id + * The index of the transmit queue to set up. + * The value must be in the range [0, nb_tx_queue - 1] previously supplied + * to rte_eth_dev_configure(). + * @param nb_tx_desc + * The number of transmit descriptors to allocate for the transmit ring. + * 0 to set default PMD value. + * @param conf + * The hairpin configuration. + * + * @return + * - (0) if successful. + * - (-ENOTSUP) if hardware doesn't support. + * - (-EINVAL) if bad parameter. + * - (-ENOMEM) if unable to allocate the resources. + */ +__rte_experimental +int rte_eth_tx_hairpin_queue_setup + (uint16_t port_id, uint16_t tx_queue_id, uint16_t nb_tx_desc, + const struct rte_eth_hairpin_conf *conf); + +/** * Return the NUMA socket to which an Ethernet device is connected * * @param port_id @@ -1851,7 +1951,7 @@ int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id, * to rte_eth_dev_configure(). * @return * - 0: Success, the receive queue is started. - * - -EINVAL: The port_id or the queue_id out of range. + * - -EINVAL: The port_id or the queue_id out of range or belong to hairpin. * - -EIO: if device is removed. * - -ENOTSUP: The function not supported in PMD driver. */ @@ -1868,7 +1968,7 @@ int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id, * to rte_eth_dev_configure(). * @return * - 0: Success, the receive queue is stopped. - * - -EINVAL: The port_id or the queue_id out of range. + * - -EINVAL: The port_id or the queue_id out of range or belong to hairpin. * - -EIO: if device is removed. * - -ENOTSUP: The function not supported in PMD driver. */ @@ -1886,7 +1986,7 @@ int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id, * to rte_eth_dev_configure(). * @return * - 0: Success, the transmit queue is started. - * - -EINVAL: The port_id or the queue_id out of range. + * - -EINVAL: The port_id or the queue_id out of range or belong to hairpin. * - -EIO: if device is removed. * - -ENOTSUP: The function not supported in PMD driver. */ @@ -1903,7 +2003,7 @@ int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id, * to rte_eth_dev_configure(). * @return * - 0: Success, the transmit queue is stopped. - * - -EINVAL: The port_id or the queue_id out of range. + * - -EINVAL: The port_id or the queue_id out of range or belong to hairpin. * - -EIO: if device is removed. * - -ENOTSUP: The function not supported in PMD driver. */ @@ -3569,7 +3669,8 @@ int rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id, * @return * - 0: Success * - -ENOTSUP: routine is not supported by the device PMD. - * - -EINVAL: The port_id or the queue_id is out of range. + * - -EINVAL: The port_id or the queue_id is out of range, or the queue + * is hairpin queue. */ int rte_eth_rx_queue_info_get(uint16_t port_id, uint16_t queue_id, struct rte_eth_rxq_info *qinfo); @@ -3589,7 +3690,8 @@ int rte_eth_rx_queue_info_get(uint16_t port_id, uint16_t queue_id, * @return * - 0: Success * - -ENOTSUP: routine is not supported by the device PMD. - * - -EINVAL: The port_id or the queue_id is out of range. + * - -EINVAL: The port_id or the queue_id is out of range, or the queue + * is hairpin queue. */ int rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id, struct rte_eth_txq_info *qinfo); @@ -4031,6 +4133,23 @@ int rte_eth_dev_adjust_nb_rx_tx_desc(uint16_t port_id, void * rte_eth_dev_get_sec_ctx(uint16_t port_id); +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Query the device hairpin capabilities. + * + * @param port_id + * The port identifier of the Ethernet device. + * @param cap + * Pointer to a structure that will hold the hairpin capabilities. + * @return + * - (0) if successful. + * - (-ENOTSUP) if hardware doesn't support. + */ +__rte_experimental +int rte_eth_dev_hairpin_capability_get(uint16_t port_id, + struct rte_eth_hairpin_cap *cap); #include @@ -4131,6 +4250,12 @@ int rte_eth_dev_adjust_nb_rx_tx_desc(uint16_t port_id, RTE_ETHDEV_LOG(ERR, "Invalid RX queue_id=%u\n", queue_id); return 0; } + if (dev->data->rx_queue_state[queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + RTE_ETHDEV_LOG(ERR, "RX queue_id=%u is hairpin queue\n", + queue_id); + return 0; + } #endif nb_rx = (*dev->rx_pkt_burst)(dev->data->rx_queues[queue_id], rx_pkts, nb_pkts); @@ -4397,6 +4522,12 @@ static inline int rte_eth_tx_descriptor_status(uint16_t port_id, RTE_ETHDEV_LOG(ERR, "Invalid TX queue_id=%u\n", queue_id); return 0; } + if (dev->data->tx_queue_state[queue_id] == + RTE_ETH_QUEUE_STATE_HAIRPIN) { + RTE_ETHDEV_LOG(ERR, "TX queue_id=%u is hairpin queue\n", + queue_id); + return 0; + } #endif #ifdef RTE_ETHDEV_RXTX_CALLBACKS diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h index dcb5ae6..6d61158 100644 --- a/lib/librte_ethdev/rte_ethdev_core.h +++ b/lib/librte_ethdev/rte_ethdev_core.h @@ -506,6 +506,86 @@ typedef int (*eth_pool_ops_supported_t)(struct rte_eth_dev *dev, /**< @internal Test if a port supports specific mempool ops */ /** + * @internal + * Get the hairpin capabilities. + * + * @param dev + * ethdev handle of port. + * @param cap + * returns the hairpin capabilities from the device. + * + * @return + * Negative errno value on error, 0 on success. + * + * @retval 0 + * Success, hairpin is supported. + * @retval -ENOTSUP + * Hairpin is not supported. + */ +typedef int (*eth_hairpin_cap_get_t)(struct rte_eth_dev *dev, + struct rte_eth_hairpin_cap *cap); + +/** + * @internal + * Setup RX hairpin queue. + * + * @param dev + * ethdev handle of port. + * @param rx_queue_id + * the selected RX queue index. + * @param nb_rx_desc + * the requested number of descriptors for this queue. 0 - use PMD default. + * @param conf + * the RX hairpin configuration structure. + * + * @return + * Negative errno value on error, 0 on success. + * + * @retval 0 + * Success, hairpin is supported. + * @retval -ENOTSUP + * Hairpin is not supported. + * @retval -EINVAL + * One of the parameters is invalid. + * @retval -ENOMEM + * Unable to allocate resources. + */ +typedef int (*eth_rx_hairpin_queue_setup_t) + (struct rte_eth_dev *dev, uint16_t rx_queue_id, + uint16_t nb_rx_desc, + const struct rte_eth_hairpin_conf *conf); + +/** + * @internal + * Setup TX hairpin queue. + * + * @param dev + * ethdev handle of port. + * @param tx_queue_id + * the selected TX queue index. + * @param nb_tx_desc + * the requested number of descriptors for this queue. 0 - use PMD default. + * @param conf + * the TX hairpin configuration structure. + * + * @return + * Negative errno value on error, 0 on success. + * + * @retval 0 + * Success, hairpin is supported. + * @retval -ENOTSUP + * Hairpin is not supported. + * @retval -EINVAL + * One of the parameters is invalid. + * @retval -ENOMEM + * Unable to allocate resources. + */ +typedef int (*eth_tx_hairpin_queue_setup_t) + (struct rte_eth_dev *dev, uint16_t tx_queue_id, + uint16_t nb_tx_desc, + const struct rte_eth_hairpin_conf *hairpin_conf); + +/** * @internal A structure containing the functions exported by an Ethernet driver. */ struct eth_dev_ops { @@ -639,6 +719,13 @@ struct eth_dev_ops { eth_pool_ops_supported_t pool_ops_supported; /**< Test if a port supports specific mempool ops */ + + eth_hairpin_cap_get_t hairpin_cap_get; + /**< Returns the hairpin capabilities. */ + eth_rx_hairpin_queue_setup_t rx_hairpin_queue_setup; + /**< Set up device RX hairpin queue. */ + eth_tx_hairpin_queue_setup_t tx_hairpin_queue_setup; + /**< Set up device TX hairpin queue. */ }; /** @@ -746,9 +833,9 @@ struct rte_eth_dev_data { dev_started : 1, /**< Device state: STARTED(1) / STOPPED(0). */ lro : 1; /**< RX LRO is ON(1) / OFF(0) */ uint8_t rx_queue_state[RTE_MAX_QUEUES_PER_PORT]; - /**< Queues state: STARTED(1) / STOPPED(0). */ + /**< Queues state: HAIRPIN(2) / STARTED(1) / STOPPED(0). */ uint8_t tx_queue_state[RTE_MAX_QUEUES_PER_PORT]; - /**< Queues state: STARTED(1) / STOPPED(0). */ + /**< Queues state: HAIRPIN(2) / STARTED(1) / STOPPED(0). */ uint32_t dev_flags; /**< Capabilities. */ enum rte_kernel_driver kdrv; /**< Kernel driver passthrough. */ int numa_node; /**< NUMA node connection. */ diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h index c404f17..59d4c01 100644 --- a/lib/librte_ethdev/rte_ethdev_driver.h +++ b/lib/librte_ethdev/rte_ethdev_driver.h @@ -26,6 +26,7 @@ */ #define RTE_ETH_QUEUE_STATE_STOPPED 0 #define RTE_ETH_QUEUE_STATE_STARTED 1 +#define RTE_ETH_QUEUE_STATE_HAIRPIN 2 /** * @internal diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map index 6df42a4..77b0a86 100644 --- a/lib/librte_ethdev/rte_ethdev_version.map +++ b/lib/librte_ethdev/rte_ethdev_version.map @@ -283,4 +283,9 @@ EXPERIMENTAL { # added in 19.08 rte_eth_read_clock; + + # added in 19.11 + rte_eth_rx_hairpin_queue_setup; + rte_eth_tx_hairpin_queue_setup; + rte_eth_dev_hairpin_capability_get; }; From patchwork Thu Oct 17 15:32:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61412 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 03A571E9A1; Thu, 17 Oct 2019 17:32:34 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 7FDED1E97B for ; Thu, 17 Oct 2019 17:32:29 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:28 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfK026371; Thu, 17 Oct 2019 18:32:27 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:05 +0000 Message-Id: <1571326337-42692-4-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 03/15] net/mlx5: query hca hairpin capabilities X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit query and store the hairpin capabilities from the device. Those capabilities will be used when creating the hairpin queue. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.h | 4 ++++ drivers/net/mlx5/mlx5_devx_cmds.c | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index baf945c..4d14e9e 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -184,6 +184,10 @@ struct mlx5_hca_attr { uint32_t tunnel_lro_vxlan:1; uint32_t lro_max_msg_sz_mode:2; uint32_t lro_timer_supported_periods[MLX5_LRO_NUM_SUPP_PERIODS]; + uint32_t hairpin:1; + uint32_t log_max_hairpin_queues:5; + uint32_t log_max_hairpin_wq_data_sz:5; + uint32_t log_max_hairpin_num_packets:5; }; /* Flow list . */ diff --git a/drivers/net/mlx5/mlx5_devx_cmds.c b/drivers/net/mlx5/mlx5_devx_cmds.c index acfe1de..b072c37 100644 --- a/drivers/net/mlx5/mlx5_devx_cmds.c +++ b/drivers/net/mlx5/mlx5_devx_cmds.c @@ -327,6 +327,13 @@ struct mlx5_devx_obj * attr->flow_counters_dump = MLX5_GET(cmd_hca_cap, hcattr, flow_counters_dump); attr->eswitch_manager = MLX5_GET(cmd_hca_cap, hcattr, eswitch_manager); + attr->hairpin = MLX5_GET(cmd_hca_cap, hcattr, hairpin); + attr->log_max_hairpin_queues = MLX5_GET(cmd_hca_cap, hcattr, + log_max_hairpin_queues); + attr->log_max_hairpin_wq_data_sz = MLX5_GET(cmd_hca_cap, hcattr, + log_max_hairpin_wq_data_sz); + attr->log_max_hairpin_num_packets = MLX5_GET + (cmd_hca_cap, hcattr, log_min_hairpin_wq_data_sz); attr->eth_net_offloads = MLX5_GET(cmd_hca_cap, hcattr, eth_net_offloads); attr->eth_virt = MLX5_GET(cmd_hca_cap, hcattr, eth_virt); From patchwork Thu Oct 17 15:32:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61414 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0A0071E9C5; Thu, 17 Oct 2019 17:32:39 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 722EF1E9A7 for ; Thu, 17 Oct 2019 17:32:34 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:32 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfL026371; Thu, 17 Oct 2019 18:32:32 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:06 +0000 Message-Id: <1571326337-42692-5-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 04/15] net/mlx5: support Rx hairpin queues X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit adds the support for creating Rx hairpin queues. Hairpin queue is a queue that is created using DevX and only used by the HW. This results in that all the data part of the RQ is not being used. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 2 + drivers/net/mlx5/mlx5_rxq.c | 270 ++++++++++++++++++++++++++++++++++++---- drivers/net/mlx5/mlx5_rxtx.h | 15 +++ drivers/net/mlx5/mlx5_trigger.c | 7 ++ 4 files changed, 270 insertions(+), 24 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 34376f6..49edb7e 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -974,6 +974,7 @@ struct mlx5_dev_spawn_data { .dev_supported_ptypes_get = mlx5_dev_supported_ptypes_get, .vlan_filter_set = mlx5_vlan_filter_set, .rx_queue_setup = mlx5_rx_queue_setup, + .rx_hairpin_queue_setup = mlx5_rx_hairpin_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, @@ -1040,6 +1041,7 @@ struct mlx5_dev_spawn_data { .dev_supported_ptypes_get = mlx5_dev_supported_ptypes_get, .vlan_filter_set = mlx5_vlan_filter_set, .rx_queue_setup = mlx5_rx_queue_setup, + .rx_hairpin_queue_setup = mlx5_rx_hairpin_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 0db065a..66596df 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -106,21 +106,25 @@ struct mlx5_priv *priv = dev->data->dev_private; uint16_t i; uint16_t n = 0; + uint16_t n_ibv = 0; if (mlx5_check_mprq_support(dev) < 0) return 0; /* All the configured queues should be enabled. */ for (i = 0; i < priv->rxqs_n; ++i) { struct mlx5_rxq_data *rxq = (*priv->rxqs)[i]; + struct mlx5_rxq_ctrl *rxq_ctrl = container_of + (rxq, struct mlx5_rxq_ctrl, rxq); - if (!rxq) + if (rxq == NULL || rxq_ctrl->type != MLX5_RXQ_TYPE_STANDARD) continue; + n_ibv++; if (mlx5_rxq_mprq_enabled(rxq)) ++n; } /* Multi-Packet RQ can't be partially configured. */ - assert(n == 0 || n == priv->rxqs_n); - return n == priv->rxqs_n; + assert(n == 0 || n == n_ibv); + return n == n_ibv; } /** @@ -427,6 +431,7 @@ } /** + * Rx queue presetup checks. * * @param dev * Pointer to Ethernet device structure. @@ -434,25 +439,14 @@ * RX queue index. * @param desc * Number of descriptors to configure in queue. - * @param socket - * NUMA socket on which memory must be allocated. - * @param[in] conf - * Thresholds parameters. - * @param mp - * Memory pool for buffer allocations. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, - unsigned int socket, const struct rte_eth_rxconf *conf, - struct rte_mempool *mp) +static int +mlx5_rx_queue_pre_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc) { struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_rxq_data *rxq = (*priv->rxqs)[idx]; - struct mlx5_rxq_ctrl *rxq_ctrl = - container_of(rxq, struct mlx5_rxq_ctrl, rxq); if (!rte_is_power_of_2(desc)) { desc = 1 << log2above(desc); @@ -476,6 +470,41 @@ return -rte_errno; } mlx5_rxq_release(dev, idx); + return 0; +} + +/** + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * RX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * @param mp + * Memory pool for buffer allocations. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_rxconf *conf, + struct rte_mempool *mp) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_data *rxq = (*priv->rxqs)[idx]; + struct mlx5_rxq_ctrl *rxq_ctrl = + container_of(rxq, struct mlx5_rxq_ctrl, rxq); + int res; + + res = mlx5_rx_queue_pre_setup(dev, idx, desc); + if (res) + return res; rxq_ctrl = mlx5_rxq_new(dev, idx, desc, socket, conf, mp); if (!rxq_ctrl) { DRV_LOG(ERR, "port %u unable to allocate queue index %u", @@ -490,6 +519,56 @@ } /** + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * RX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param hairpin_conf + * Hairpin configuration parameters. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_rx_hairpin_queue_setup(struct rte_eth_dev *dev, uint16_t idx, + uint16_t desc, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_data *rxq = (*priv->rxqs)[idx]; + struct mlx5_rxq_ctrl *rxq_ctrl = + container_of(rxq, struct mlx5_rxq_ctrl, rxq); + int res; + + res = mlx5_rx_queue_pre_setup(dev, idx, desc); + if (res) + return res; + if (hairpin_conf->peer_n != 1 || + hairpin_conf->peers[0].port != dev->data->port_id || + hairpin_conf->peers[0].queue >= priv->txqs_n) { + DRV_LOG(ERR, "port %u unable to setup hairpin queue index %u " + " invalid hairpind configuration", dev->data->port_id, + idx); + rte_errno = EINVAL; + return -rte_errno; + } + rxq_ctrl = mlx5_rxq_hairpin_new(dev, idx, desc, hairpin_conf); + if (!rxq_ctrl) { + DRV_LOG(ERR, "port %u unable to allocate queue index %u", + dev->data->port_id, idx); + rte_errno = ENOMEM; + return -rte_errno; + } + DRV_LOG(DEBUG, "port %u adding Rx queue %u to list", + dev->data->port_id, idx); + (*priv->rxqs)[idx] = &rxq_ctrl->rxq; + return 0; +} + +/** * DPDK callback to release a RX queue. * * @param dpdk_rxq @@ -561,6 +640,24 @@ } /** + * Release an Rx hairpin related resources. + * + * @param rxq_obj + * Hairpin Rx queue object. + */ +static void +rxq_obj_hairpin_release(struct mlx5_rxq_obj *rxq_obj) +{ + struct mlx5_devx_modify_rq_attr rq_attr = { 0 }; + + assert(rxq_obj); + rq_attr.state = MLX5_RQC_STATE_RST; + rq_attr.rq_state = MLX5_RQC_STATE_RDY; + mlx5_devx_cmd_modify_rq(rxq_obj->rq, &rq_attr); + claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); +} + +/** * Release an Rx verbs/DevX queue object. * * @param rxq_obj @@ -577,14 +674,22 @@ assert(rxq_obj->wq); assert(rxq_obj->cq); if (rte_atomic32_dec_and_test(&rxq_obj->refcnt)) { - rxq_free_elts(rxq_obj->rxq_ctrl); - if (rxq_obj->type == MLX5_RXQ_OBJ_TYPE_IBV) { + switch (rxq_obj->type) { + case MLX5_RXQ_OBJ_TYPE_IBV: + rxq_free_elts(rxq_obj->rxq_ctrl); claim_zero(mlx5_glue->destroy_wq(rxq_obj->wq)); - } else if (rxq_obj->type == MLX5_RXQ_OBJ_TYPE_DEVX_RQ) { + claim_zero(mlx5_glue->destroy_cq(rxq_obj->cq)); + break; + case MLX5_RXQ_OBJ_TYPE_DEVX_RQ: + rxq_free_elts(rxq_obj->rxq_ctrl); claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); rxq_release_rq_resources(rxq_obj->rxq_ctrl); + claim_zero(mlx5_glue->destroy_cq(rxq_obj->cq)); + break; + case MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN: + rxq_obj_hairpin_release(rxq_obj); + break; } - claim_zero(mlx5_glue->destroy_cq(rxq_obj->cq)); if (rxq_obj->channel) claim_zero(mlx5_glue->destroy_comp_channel (rxq_obj->channel)); @@ -1132,6 +1237,70 @@ } /** + * Create the Rx hairpin queue object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Rx queue array + * + * @return + * The hairpin DevX object initialised, NULL otherwise and rte_errno is set. + */ +static struct mlx5_rxq_obj * +mlx5_rxq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[idx]; + struct mlx5_rxq_ctrl *rxq_ctrl = + container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); + struct mlx5_devx_create_rq_attr attr = { 0 }; + struct mlx5_rxq_obj *tmpl = NULL; + int ret = 0; + + assert(rxq_data); + assert(!rxq_ctrl->obj); + tmpl = rte_calloc_socket(__func__, 1, sizeof(*tmpl), 0, + rxq_ctrl->socket); + if (!tmpl) { + DRV_LOG(ERR, + "port %u Rx queue %u cannot allocate verbs resources", + dev->data->port_id, rxq_data->idx); + rte_errno = ENOMEM; + goto error; + } + tmpl->type = MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN; + tmpl->rxq_ctrl = rxq_ctrl; + attr.hairpin = 1; + /* Workaround for hairpin startup */ + attr.wq_attr.log_hairpin_num_packets = log2above(32); + /* Workaround for packets larger than 1KB */ + attr.wq_attr.log_hairpin_data_sz = + priv->config.hca_attr.log_max_hairpin_wq_data_sz; + tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->ctx, &attr, + rxq_ctrl->socket); + if (!tmpl->rq) { + DRV_LOG(ERR, + "port %u Rx hairpin queue %u can't create rq object", + dev->data->port_id, idx); + rte_errno = errno; + goto error; + } + DRV_LOG(DEBUG, "port %u rxq %u updated with %p", dev->data->port_id, + idx, (void *)&tmpl); + rte_atomic32_inc(&tmpl->refcnt); + LIST_INSERT_HEAD(&priv->rxqsobj, tmpl, next); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; + return tmpl; +error: + ret = rte_errno; /* Save rte_errno before cleanup. */ + if (tmpl->rq) + mlx5_devx_cmd_destroy(tmpl->rq); + rte_errno = ret; /* Restore rte_errno. */ + return NULL; +} + +/** * Create the Rx queue Verbs/DevX object. * * @param dev @@ -1163,6 +1332,8 @@ struct mlx5_rxq_obj * assert(rxq_data); assert(!rxq_ctrl->obj); + if (type == MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN) + return mlx5_rxq_obj_hairpin_new(dev, idx); priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_RX_QUEUE; priv->verbs_alloc_ctx.obj = rxq_ctrl; tmpl = rte_calloc_socket(__func__, 1, sizeof(*tmpl), 0, @@ -1433,15 +1604,19 @@ struct mlx5_rxq_obj * unsigned int strd_num_n = 0; unsigned int strd_sz_n = 0; unsigned int i; + unsigned int n_ibv = 0; if (!mlx5_mprq_enabled(dev)) return 0; /* Count the total number of descriptors configured. */ for (i = 0; i != priv->rxqs_n; ++i) { struct mlx5_rxq_data *rxq = (*priv->rxqs)[i]; + struct mlx5_rxq_ctrl *rxq_ctrl = container_of + (rxq, struct mlx5_rxq_ctrl, rxq); - if (rxq == NULL) + if (rxq == NULL || rxq_ctrl->type != MLX5_RXQ_TYPE_STANDARD) continue; + n_ibv++; desc += 1 << rxq->elts_n; /* Get the max number of strides. */ if (strd_num_n < rxq->strd_num_n) @@ -1466,7 +1641,7 @@ struct mlx5_rxq_obj * * this Mempool gets available again. */ desc *= 4; - obj_num = desc + MLX5_MPRQ_MP_CACHE_SZ * priv->rxqs_n; + obj_num = desc + MLX5_MPRQ_MP_CACHE_SZ * n_ibv; /* * rte_mempool_create_empty() has sanity check to refuse large cache * size compared to the number of elements. @@ -1514,8 +1689,10 @@ struct mlx5_rxq_obj * /* Set mempool for each Rx queue. */ for (i = 0; i != priv->rxqs_n; ++i) { struct mlx5_rxq_data *rxq = (*priv->rxqs)[i]; + struct mlx5_rxq_ctrl *rxq_ctrl = container_of + (rxq, struct mlx5_rxq_ctrl, rxq); - if (rxq == NULL) + if (rxq == NULL || rxq_ctrl->type != MLX5_RXQ_TYPE_STANDARD) continue; rxq->mprq_mp = mp; } @@ -1620,6 +1797,7 @@ struct mlx5_rxq_ctrl * rte_errno = ENOMEM; return NULL; } + tmpl->type = MLX5_RXQ_TYPE_STANDARD; if (mlx5_mr_btree_init(&tmpl->rxq.mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N, socket)) { /* rte_errno is already set. */ @@ -1788,6 +1966,49 @@ struct mlx5_rxq_ctrl * } /** + * Create a DPDK Rx hairpin queue. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * RX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param hairpin_conf + * The hairpin binding configuration. + * + * @return + * A DPDK queue object on success, NULL otherwise and rte_errno is set. + */ +struct mlx5_rxq_ctrl * +mlx5_rxq_hairpin_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_ctrl *tmpl; + + tmpl = rte_calloc_socket("RXQ", 1, sizeof(*tmpl), 0, SOCKET_ID_ANY); + if (!tmpl) { + rte_errno = ENOMEM; + return NULL; + } + tmpl->type = MLX5_RXQ_TYPE_HAIRPIN; + tmpl->socket = SOCKET_ID_ANY; + tmpl->rxq.rss_hash = 0; + tmpl->rxq.port_id = dev->data->port_id; + tmpl->priv = priv; + tmpl->rxq.mp = NULL; + tmpl->rxq.elts_n = log2above(desc); + tmpl->rxq.elts = NULL; + tmpl->rxq.mr_ctrl.cache_bh = (struct mlx5_mr_btree) { 0 }; + tmpl->hairpin_conf = *hairpin_conf; + tmpl->rxq.idx = idx; + rte_atomic32_inc(&tmpl->refcnt); + LIST_INSERT_HEAD(&priv->rxqsctrl, tmpl, next); + return tmpl; +} + +/** * Get a Rx queue. * * @param dev @@ -1841,7 +2062,8 @@ struct mlx5_rxq_ctrl * if (rxq_ctrl->dbr_umem_id_valid) claim_zero(mlx5_release_dbr(dev, rxq_ctrl->dbr_umem_id, rxq_ctrl->dbr_offset)); - mlx5_mr_btree_free(&rxq_ctrl->rxq.mr_ctrl.cache_bh); + if (rxq_ctrl->type == MLX5_RXQ_TYPE_STANDARD) + mlx5_mr_btree_free(&rxq_ctrl->rxq.mr_ctrl.cache_bh); LIST_REMOVE(rxq_ctrl, next); rte_free(rxq_ctrl); (*priv->rxqs)[idx] = NULL; diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 4bb28a4..13fdc38 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -159,6 +159,13 @@ struct mlx5_rxq_data { enum mlx5_rxq_obj_type { MLX5_RXQ_OBJ_TYPE_IBV, /* mlx5_rxq_obj with ibv_wq. */ MLX5_RXQ_OBJ_TYPE_DEVX_RQ, /* mlx5_rxq_obj with mlx5_devx_rq. */ + MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN, + /* mlx5_rxq_obj with mlx5_devx_rq and hairpin support. */ +}; + +enum mlx5_rxq_type { + MLX5_RXQ_TYPE_STANDARD, /* Standard Rx queue. */ + MLX5_RXQ_TYPE_HAIRPIN, /* Hairpin Rx queue. */ }; /* Verbs/DevX Rx queue elements. */ @@ -183,6 +190,7 @@ struct mlx5_rxq_ctrl { rte_atomic32_t refcnt; /* Reference counter. */ struct mlx5_rxq_obj *obj; /* Verbs/DevX elements. */ struct mlx5_priv *priv; /* Back pointer to private data. */ + enum mlx5_rxq_type type; /* Rxq type. */ unsigned int socket; /* CPU socket ID for allocations. */ unsigned int irq:1; /* Whether IRQ is enabled. */ unsigned int dbr_umem_id_valid:1; /* dbr_umem_id holds a valid value. */ @@ -193,6 +201,7 @@ struct mlx5_rxq_ctrl { uint32_t dbr_umem_id; /* Storing door-bell information, */ uint64_t dbr_offset; /* needed when freeing door-bell. */ struct mlx5dv_devx_umem *wq_umem; /* WQ buffer registration info. */ + struct rte_eth_hairpin_conf hairpin_conf; /* Hairpin configuration. */ }; enum mlx5_ind_tbl_type { @@ -339,6 +348,9 @@ struct mlx5_txq_ctrl { int mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_rxconf *conf, struct rte_mempool *mp); +int mlx5_rx_hairpin_queue_setup + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + const struct rte_eth_hairpin_conf *hairpin_conf); void mlx5_rx_queue_release(void *dpdk_rxq); int mlx5_rx_intr_vec_enable(struct rte_eth_dev *dev); void mlx5_rx_intr_vec_disable(struct rte_eth_dev *dev); @@ -351,6 +363,9 @@ struct mlx5_rxq_ctrl *mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_rxconf *conf, struct rte_mempool *mp); +struct mlx5_rxq_ctrl *mlx5_rxq_hairpin_new + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + const struct rte_eth_hairpin_conf *hairpin_conf); struct mlx5_rxq_ctrl *mlx5_rxq_get(struct rte_eth_dev *dev, uint16_t idx); int mlx5_rxq_release(struct rte_eth_dev *dev, uint16_t idx); int mlx5_rxq_verify(struct rte_eth_dev *dev); diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 122f31c..cb31ae2 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -118,6 +118,13 @@ if (!rxq_ctrl) continue; + if (rxq_ctrl->type == MLX5_RXQ_TYPE_HAIRPIN) { + rxq_ctrl->obj = mlx5_rxq_obj_new + (dev, i, MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN); + if (!rxq_ctrl->obj) + goto error; + continue; + } /* Pre-register Rx mempool. */ mp = mlx5_rxq_mprq_enabled(&rxq_ctrl->rxq) ? rxq_ctrl->rxq.mprq_mp : rxq_ctrl->rxq.mp; From patchwork Thu Oct 17 15:32:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61415 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BEDEB1E9CF; Thu, 17 Oct 2019 17:32:40 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 6CC9C1E9CA for ; Thu, 17 Oct 2019 17:32:39 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:35 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfM026371; Thu, 17 Oct 2019 18:32:34 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:07 +0000 Message-Id: <1571326337-42692-6-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 05/15] net/mlx5: prepare txq to work with different types X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Currenlty all Tx queues are created using Verbs. This commit modify the naming so it will not include verbs, since in next commit a new type will be introduce (hairpin) Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 2 +- drivers/net/mlx5/mlx5.h | 2 +- drivers/net/mlx5/mlx5_rxtx.c | 2 +- drivers/net/mlx5/mlx5_rxtx.h | 39 +++++++++++++++++------ drivers/net/mlx5/mlx5_trigger.c | 4 +-- drivers/net/mlx5/mlx5_txq.c | 70 ++++++++++++++++++++--------------------- 6 files changed, 69 insertions(+), 50 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 49edb7e..2431a55 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -911,7 +911,7 @@ struct mlx5_dev_spawn_data { if (ret) DRV_LOG(WARNING, "port %u some Rx queues still remain", dev->data->port_id); - ret = mlx5_txq_ibv_verify(dev); + ret = mlx5_txq_obj_verify(dev); if (ret) DRV_LOG(WARNING, "port %u some Verbs Tx queue still remain", dev->data->port_id); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 4d14e9e..36cced9 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -645,7 +645,7 @@ struct mlx5_priv { LIST_HEAD(rxqobj, mlx5_rxq_obj) rxqsobj; /* Verbs/DevX Rx queues. */ LIST_HEAD(hrxq, mlx5_hrxq) hrxqs; /* Verbs Hash Rx queues. */ LIST_HEAD(txq, mlx5_txq_ctrl) txqsctrl; /* DPDK Tx queues. */ - LIST_HEAD(txqibv, mlx5_txq_ibv) txqsibv; /* Verbs Tx queues. */ + LIST_HEAD(txqobj, mlx5_txq_obj) txqsobj; /* Verbs/DevX Tx queues. */ /* Indirection tables. */ LIST_HEAD(ind_tables, mlx5_ind_table_obj) ind_tbls; /* Pointer to next element. */ diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 10d0ca1..f23708c 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -863,7 +863,7 @@ enum mlx5_txcmp_code { .qp_state = IBV_QPS_RESET, .port_num = (uint8_t)priv->ibv_port, }; - struct ibv_qp *qp = txq_ctrl->ibv->qp; + struct ibv_qp *qp = txq_ctrl->obj->qp; ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); if (ret) { diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 13fdc38..12f9bfb 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -308,13 +308,31 @@ struct mlx5_txq_data { /* Storage for queued packets, must be the last field. */ } __rte_cache_aligned; -/* Verbs Rx queue elements. */ -struct mlx5_txq_ibv { - LIST_ENTRY(mlx5_txq_ibv) next; /* Pointer to the next element. */ +enum mlx5_txq_obj_type { + MLX5_TXQ_OBJ_TYPE_IBV, /* mlx5_txq_obj with ibv_wq. */ + MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN, + /* mlx5_txq_obj with mlx5_devx_tq and hairpin support. */ +}; + +enum mlx5_txq_type { + MLX5_TXQ_TYPE_STANDARD, /* Standard Tx queue. */ + MLX5_TXQ_TYPE_HAIRPIN, /* Hairpin Rx queue. */ +}; + +/* Verbs/DevX Tx queue elements. */ +struct mlx5_txq_obj { + LIST_ENTRY(mlx5_txq_obj) next; /* Pointer to the next element. */ rte_atomic32_t refcnt; /* Reference counter. */ struct mlx5_txq_ctrl *txq_ctrl; /* Pointer to the control queue. */ - struct ibv_cq *cq; /* Completion Queue. */ - struct ibv_qp *qp; /* Queue Pair. */ + enum mlx5_rxq_obj_type type; /* The txq object type. */ + RTE_STD_C11 + union { + struct { + struct ibv_cq *cq; /* Completion Queue. */ + struct ibv_qp *qp; /* Queue Pair. */ + }; + struct mlx5_devx_obj *sq; /* DevX object for Sx queue. */ + }; }; /* TX queue control descriptor. */ @@ -322,9 +340,10 @@ struct mlx5_txq_ctrl { LIST_ENTRY(mlx5_txq_ctrl) next; /* Pointer to the next element. */ rte_atomic32_t refcnt; /* Reference counter. */ unsigned int socket; /* CPU socket ID for allocations. */ + enum mlx5_txq_type type; /* The txq ctrl type. */ unsigned int max_inline_data; /* Max inline data. */ unsigned int max_tso_header; /* Max TSO header size. */ - struct mlx5_txq_ibv *ibv; /* Verbs queue object. */ + struct mlx5_txq_obj *obj; /* Verbs/DevX queue object. */ struct mlx5_priv *priv; /* Back pointer to private data. */ off_t uar_mmap_offset; /* UAR mmap offset for non-primary process. */ void *bf_reg; /* BlueFlame register from Verbs. */ @@ -393,10 +412,10 @@ int mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_txconf *conf); void mlx5_tx_queue_release(void *dpdk_txq); int mlx5_tx_uar_init_secondary(struct rte_eth_dev *dev, int fd); -struct mlx5_txq_ibv *mlx5_txq_ibv_new(struct rte_eth_dev *dev, uint16_t idx); -struct mlx5_txq_ibv *mlx5_txq_ibv_get(struct rte_eth_dev *dev, uint16_t idx); -int mlx5_txq_ibv_release(struct mlx5_txq_ibv *txq_ibv); -int mlx5_txq_ibv_verify(struct rte_eth_dev *dev); +struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx); +struct mlx5_txq_obj *mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t idx); +int mlx5_txq_obj_release(struct mlx5_txq_obj *txq_ibv); +int mlx5_txq_obj_verify(struct rte_eth_dev *dev); struct mlx5_txq_ctrl *mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_txconf *conf); diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index cb31ae2..50c4df5 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -52,8 +52,8 @@ if (!txq_ctrl) continue; txq_alloc_elts(txq_ctrl); - txq_ctrl->ibv = mlx5_txq_ibv_new(dev, i); - if (!txq_ctrl->ibv) { + txq_ctrl->obj = mlx5_txq_obj_new(dev, i); + if (!txq_ctrl->obj) { rte_errno = ENOMEM; goto error; } diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 53d45e7..a6e2563 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -375,15 +375,15 @@ * @return * The Verbs object initialised, NULL otherwise and rte_errno is set. */ -struct mlx5_txq_ibv * -mlx5_txq_ibv_new(struct rte_eth_dev *dev, uint16_t idx) +struct mlx5_txq_obj * +mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; struct mlx5_txq_ctrl *txq_ctrl = container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct mlx5_txq_ibv tmpl; - struct mlx5_txq_ibv *txq_ibv = NULL; + struct mlx5_txq_obj tmpl; + struct mlx5_txq_obj *txq_obj = NULL; union { struct ibv_qp_init_attr_ex init; struct ibv_cq_init_attr_ex cq; @@ -411,7 +411,7 @@ struct mlx5_txq_ibv * rte_errno = EINVAL; return NULL; } - memset(&tmpl, 0, sizeof(struct mlx5_txq_ibv)); + memset(&tmpl, 0, sizeof(struct mlx5_txq_obj)); attr.cq = (struct ibv_cq_init_attr_ex){ .comp_mask = 0, }; @@ -502,9 +502,9 @@ struct mlx5_txq_ibv * rte_errno = errno; goto error; } - txq_ibv = rte_calloc_socket(__func__, 1, sizeof(struct mlx5_txq_ibv), 0, + txq_obj = rte_calloc_socket(__func__, 1, sizeof(struct mlx5_txq_obj), 0, txq_ctrl->socket); - if (!txq_ibv) { + if (!txq_obj) { DRV_LOG(ERR, "port %u Tx queue %u cannot allocate memory", dev->data->port_id, idx); rte_errno = ENOMEM; @@ -568,9 +568,9 @@ struct mlx5_txq_ibv * } } #endif - txq_ibv->qp = tmpl.qp; - txq_ibv->cq = tmpl.cq; - rte_atomic32_inc(&txq_ibv->refcnt); + txq_obj->qp = tmpl.qp; + txq_obj->cq = tmpl.cq; + rte_atomic32_inc(&txq_obj->refcnt); txq_ctrl->bf_reg = qp.bf.reg; if (qp.comp_mask & MLX5DV_QP_MASK_UAR_MMAP_OFFSET) { txq_ctrl->uar_mmap_offset = qp.uar_mmap_offset; @@ -585,18 +585,18 @@ struct mlx5_txq_ibv * goto error; } txq_uar_init(txq_ctrl); - LIST_INSERT_HEAD(&priv->txqsibv, txq_ibv, next); - txq_ibv->txq_ctrl = txq_ctrl; + LIST_INSERT_HEAD(&priv->txqsobj, txq_obj, next); + txq_obj->txq_ctrl = txq_ctrl; priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; - return txq_ibv; + return txq_obj; error: ret = rte_errno; /* Save rte_errno before cleanup. */ if (tmpl.cq) claim_zero(mlx5_glue->destroy_cq(tmpl.cq)); if (tmpl.qp) claim_zero(mlx5_glue->destroy_qp(tmpl.qp)); - if (txq_ibv) - rte_free(txq_ibv); + if (txq_obj) + rte_free(txq_obj); priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; rte_errno = ret; /* Restore rte_errno. */ return NULL; @@ -613,8 +613,8 @@ struct mlx5_txq_ibv * * @return * The Verbs object if it exists. */ -struct mlx5_txq_ibv * -mlx5_txq_ibv_get(struct rte_eth_dev *dev, uint16_t idx) +struct mlx5_txq_obj * +mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_ctrl *txq_ctrl; @@ -624,29 +624,29 @@ struct mlx5_txq_ibv * if (!(*priv->txqs)[idx]) return NULL; txq_ctrl = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); - if (txq_ctrl->ibv) - rte_atomic32_inc(&txq_ctrl->ibv->refcnt); - return txq_ctrl->ibv; + if (txq_ctrl->obj) + rte_atomic32_inc(&txq_ctrl->obj->refcnt); + return txq_ctrl->obj; } /** * Release an Tx verbs queue object. * - * @param txq_ibv + * @param txq_obj * Verbs Tx queue object. * * @return * 1 while a reference on it exists, 0 when freed. */ int -mlx5_txq_ibv_release(struct mlx5_txq_ibv *txq_ibv) +mlx5_txq_obj_release(struct mlx5_txq_obj *txq_obj) { - assert(txq_ibv); - if (rte_atomic32_dec_and_test(&txq_ibv->refcnt)) { - claim_zero(mlx5_glue->destroy_qp(txq_ibv->qp)); - claim_zero(mlx5_glue->destroy_cq(txq_ibv->cq)); - LIST_REMOVE(txq_ibv, next); - rte_free(txq_ibv); + assert(txq_obj); + if (rte_atomic32_dec_and_test(&txq_obj->refcnt)) { + claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); + claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); + LIST_REMOVE(txq_obj, next); + rte_free(txq_obj); return 0; } return 1; @@ -662,15 +662,15 @@ struct mlx5_txq_ibv * * The number of object not released. */ int -mlx5_txq_ibv_verify(struct rte_eth_dev *dev) +mlx5_txq_obj_verify(struct rte_eth_dev *dev) { struct mlx5_priv *priv = dev->data->dev_private; int ret = 0; - struct mlx5_txq_ibv *txq_ibv; + struct mlx5_txq_obj *txq_obj; - LIST_FOREACH(txq_ibv, &priv->txqsibv, next) { + LIST_FOREACH(txq_obj, &priv->txqsobj, next) { DRV_LOG(DEBUG, "port %u Verbs Tx queue %u still referenced", - dev->data->port_id, txq_ibv->txq_ctrl->txq.idx); + dev->data->port_id, txq_obj->txq_ctrl->txq.idx); ++ret; } return ret; @@ -1127,7 +1127,7 @@ struct mlx5_txq_ctrl * if ((*priv->txqs)[idx]) { ctrl = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); - mlx5_txq_ibv_get(dev, idx); + mlx5_txq_obj_get(dev, idx); rte_atomic32_inc(&ctrl->refcnt); } return ctrl; @@ -1153,8 +1153,8 @@ struct mlx5_txq_ctrl * if (!(*priv->txqs)[idx]) return 0; txq = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); - if (txq->ibv && !mlx5_txq_ibv_release(txq->ibv)) - txq->ibv = NULL; + if (txq->obj && !mlx5_txq_obj_release(txq->obj)) + txq->obj = NULL; if (rte_atomic32_dec_and_test(&txq->refcnt)) { txq_free_elts(txq); mlx5_mr_btree_free(&txq->txq.mr_ctrl.cache_bh); From patchwork Thu Oct 17 15:32:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61416 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B0F611E9DE; Thu, 17 Oct 2019 17:32:42 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 6D3CB1E9CB for ; Thu, 17 Oct 2019 17:32:39 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:37 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfN026371; Thu, 17 Oct 2019 18:32:36 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:08 +0000 Message-Id: <1571326337-42692-7-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 06/15] net/mlx5: support Tx hairpin queues X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit adds the support for creating Tx hairpin queues. Hairpin queue is a queue that is created using DevX and only used by the HW. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 36 +++++- drivers/net/mlx5/mlx5.h | 46 ++++++++ drivers/net/mlx5/mlx5_devx_cmds.c | 186 ++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_prm.h | 118 +++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 18 ++- drivers/net/mlx5/mlx5_trigger.c | 10 +- drivers/net/mlx5/mlx5_txq.c | 230 +++++++++++++++++++++++++++++++++++--- 7 files changed, 620 insertions(+), 24 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 2431a55..c53a9c6 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -325,6 +325,9 @@ struct mlx5_dev_spawn_data { struct mlx5_ibv_shared *sh; int err = 0; uint32_t i; +#ifdef HAVE_IBV_FLOW_DV_SUPPORT + struct mlx5_devx_tis_attr tis_attr = { 0 }; +#endif assert(spawn); /* Secondary process should not create the shared context. */ @@ -389,10 +392,25 @@ struct mlx5_dev_spawn_data { goto error; } #ifdef HAVE_IBV_FLOW_DV_SUPPORT - err = mlx5_get_pdn(sh->pd, &sh->pdn); - if (err) { - DRV_LOG(ERR, "Fail to extract pdn from PD"); - goto error; + if (sh->devx) { + err = mlx5_get_pdn(sh->pd, &sh->pdn); + if (err) { + DRV_LOG(ERR, "Fail to extract pdn from PD"); + goto error; + } + sh->td = mlx5_devx_cmd_create_td(sh->ctx); + if (!sh->td) { + DRV_LOG(ERR, "TD allocation failure"); + err = ENOMEM; + goto error; + } + tis_attr.transport_domain = sh->td->id; + sh->tis = mlx5_devx_cmd_create_tis(sh->ctx, &tis_attr); + if (!sh->tis) { + DRV_LOG(ERR, "TIS allocation failure"); + err = ENOMEM; + goto error; + } } #endif /* HAVE_IBV_FLOW_DV_SUPPORT */ /* @@ -425,6 +443,10 @@ struct mlx5_dev_spawn_data { error: pthread_mutex_unlock(&mlx5_ibv_list_mutex); assert(sh); + if (sh->tis) + claim_zero(mlx5_devx_cmd_destroy(sh->tis)); + if (sh->td) + claim_zero(mlx5_devx_cmd_destroy(sh->td)); if (sh->pd) claim_zero(mlx5_glue->dealloc_pd(sh->pd)); if (sh->ctx) @@ -485,6 +507,10 @@ struct mlx5_dev_spawn_data { pthread_mutex_destroy(&sh->intr_mutex); if (sh->pd) claim_zero(mlx5_glue->dealloc_pd(sh->pd)); + if (sh->tis) + claim_zero(mlx5_devx_cmd_destroy(sh->tis)); + if (sh->td) + claim_zero(mlx5_devx_cmd_destroy(sh->td)); if (sh->ctx) claim_zero(mlx5_glue->close_device(sh->ctx)); rte_free(sh); @@ -976,6 +1002,7 @@ struct mlx5_dev_spawn_data { .rx_queue_setup = mlx5_rx_queue_setup, .rx_hairpin_queue_setup = mlx5_rx_hairpin_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, + .tx_hairpin_queue_setup = mlx5_tx_hairpin_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, .flow_ctrl_get = mlx5_dev_get_flow_ctrl, @@ -1043,6 +1070,7 @@ struct mlx5_dev_spawn_data { .rx_queue_setup = mlx5_rx_queue_setup, .rx_hairpin_queue_setup = mlx5_rx_hairpin_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, + .tx_hairpin_queue_setup = mlx5_tx_hairpin_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, .flow_ctrl_get = mlx5_dev_get_flow_ctrl, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 36cced9..7ea4950 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -350,6 +350,43 @@ struct mlx5_devx_rqt_attr { uint32_t rq_list[]; }; +/* TIS attributes structure. */ +struct mlx5_devx_tis_attr { + uint32_t strict_lag_tx_port_affinity:1; + uint32_t tls_en:1; + uint32_t lag_tx_port_affinity:4; + uint32_t prio:4; + uint32_t transport_domain:24; +}; + +/* SQ attributes structure, used by SQ create operation. */ +struct mlx5_devx_create_sq_attr { + uint32_t rlky:1; + uint32_t cd_master:1; + uint32_t fre:1; + uint32_t flush_in_error_en:1; + uint32_t allow_multi_pkt_send_wqe:1; + uint32_t min_wqe_inline_mode:3; + uint32_t state:4; + uint32_t reg_umr:1; + uint32_t allow_swp:1; + uint32_t hairpin:1; + uint32_t user_index:24; + uint32_t cqn:24; + uint32_t packet_pacing_rate_limit_index:16; + uint32_t tis_lst_sz:16; + uint32_t tis_num:24; + struct mlx5_devx_wq_attr wq_attr; +}; + +/* SQ attributes structure, used by SQ modify operation. */ +struct mlx5_devx_modify_sq_attr { + uint32_t sq_state:4; + uint32_t state:4; + uint32_t hairpin_peer_rq:24; + uint32_t hairpin_peer_vhca:16; +}; + /** * Type of object being allocated. */ @@ -591,6 +628,8 @@ struct mlx5_ibv_shared { struct rte_intr_handle intr_handle; /* Interrupt handler for device. */ struct rte_intr_handle intr_handle_devx; /* DEVX interrupt handler. */ struct mlx5dv_devx_cmd_comp *devx_comp; /* DEVX async comp obj. */ + struct mlx5_devx_obj *tis; /* TIS object. */ + struct mlx5_devx_obj *td; /* Transport domain. */ struct mlx5_ibv_shared_port port[]; /* per device port data array. */ }; @@ -911,5 +950,12 @@ struct mlx5_devx_obj *mlx5_devx_cmd_create_tir(struct ibv_context *ctx, struct mlx5_devx_tir_attr *tir_attr); struct mlx5_devx_obj *mlx5_devx_cmd_create_rqt(struct ibv_context *ctx, struct mlx5_devx_rqt_attr *rqt_attr); +struct mlx5_devx_obj *mlx5_devx_cmd_create_sq + (struct ibv_context *ctx, struct mlx5_devx_create_sq_attr *sq_attr); +int mlx5_devx_cmd_modify_sq + (struct mlx5_devx_obj *sq, struct mlx5_devx_modify_sq_attr *sq_attr); +struct mlx5_devx_obj *mlx5_devx_cmd_create_tis + (struct ibv_context *ctx, struct mlx5_devx_tis_attr *tis_attr); +struct mlx5_devx_obj *mlx5_devx_cmd_create_td(struct ibv_context *ctx); #endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_devx_cmds.c b/drivers/net/mlx5/mlx5_devx_cmds.c index b072c37..917bbf9 100644 --- a/drivers/net/mlx5/mlx5_devx_cmds.c +++ b/drivers/net/mlx5/mlx5_devx_cmds.c @@ -709,3 +709,189 @@ struct mlx5_devx_obj * rqt->id = MLX5_GET(create_rqt_out, out, rqtn); return rqt; } + +/** + * Create SQ using DevX API. + * + * @param[in] ctx + * ibv_context returned from mlx5dv_open_device. + * @param [in] sq_attr + * Pointer to SQ attributes structure. + * @param [in] socket + * CPU socket ID for allocations. + * + * @return + * The DevX object created, NULL otherwise and rte_errno is set. + **/ +struct mlx5_devx_obj * +mlx5_devx_cmd_create_sq(struct ibv_context *ctx, + struct mlx5_devx_create_sq_attr *sq_attr) +{ + uint32_t in[MLX5_ST_SZ_DW(create_sq_in)] = {0}; + uint32_t out[MLX5_ST_SZ_DW(create_sq_out)] = {0}; + void *sq_ctx; + void *wq_ctx; + struct mlx5_devx_wq_attr *wq_attr; + struct mlx5_devx_obj *sq = NULL; + + sq = rte_calloc(__func__, 1, sizeof(*sq), 0); + if (!sq) { + DRV_LOG(ERR, "Failed to allocate SQ data"); + rte_errno = ENOMEM; + return NULL; + } + MLX5_SET(create_sq_in, in, opcode, MLX5_CMD_OP_CREATE_SQ); + sq_ctx = MLX5_ADDR_OF(create_sq_in, in, ctx); + MLX5_SET(sqc, sq_ctx, rlky, sq_attr->rlky); + MLX5_SET(sqc, sq_ctx, cd_master, sq_attr->cd_master); + MLX5_SET(sqc, sq_ctx, fre, sq_attr->fre); + MLX5_SET(sqc, sq_ctx, flush_in_error_en, sq_attr->flush_in_error_en); + MLX5_SET(sqc, sq_ctx, allow_multi_pkt_send_wqe, + sq_attr->flush_in_error_en); + MLX5_SET(sqc, sq_ctx, min_wqe_inline_mode, + sq_attr->min_wqe_inline_mode); + MLX5_SET(sqc, sq_ctx, state, sq_attr->state); + MLX5_SET(sqc, sq_ctx, reg_umr, sq_attr->reg_umr); + MLX5_SET(sqc, sq_ctx, allow_swp, sq_attr->allow_swp); + MLX5_SET(sqc, sq_ctx, hairpin, sq_attr->hairpin); + MLX5_SET(sqc, sq_ctx, user_index, sq_attr->user_index); + MLX5_SET(sqc, sq_ctx, cqn, sq_attr->cqn); + MLX5_SET(sqc, sq_ctx, packet_pacing_rate_limit_index, + sq_attr->packet_pacing_rate_limit_index); + MLX5_SET(sqc, sq_ctx, tis_lst_sz, sq_attr->tis_lst_sz); + MLX5_SET(sqc, sq_ctx, tis_num_0, sq_attr->tis_num); + wq_ctx = MLX5_ADDR_OF(sqc, sq_ctx, wq); + wq_attr = &sq_attr->wq_attr; + devx_cmd_fill_wq_data(wq_ctx, wq_attr); + sq->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in), + out, sizeof(out)); + if (!sq->obj) { + DRV_LOG(ERR, "Failed to create SQ using DevX"); + rte_errno = errno; + rte_free(sq); + return NULL; + } + sq->id = MLX5_GET(create_sq_out, out, sqn); + return sq; +} + +/** + * Modify SQ using DevX API. + * + * @param[in] sq + * Pointer to SQ object structure. + * @param [in] sq_attr + * Pointer to SQ attributes structure. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_devx_cmd_modify_sq(struct mlx5_devx_obj *sq, + struct mlx5_devx_modify_sq_attr *sq_attr) +{ + uint32_t in[MLX5_ST_SZ_DW(modify_sq_in)] = {0}; + uint32_t out[MLX5_ST_SZ_DW(modify_sq_out)] = {0}; + void *sq_ctx; + int ret; + + MLX5_SET(modify_sq_in, in, opcode, MLX5_CMD_OP_MODIFY_SQ); + MLX5_SET(modify_sq_in, in, sq_state, sq_attr->sq_state); + MLX5_SET(modify_sq_in, in, sqn, sq->id); + sq_ctx = MLX5_ADDR_OF(modify_sq_in, in, ctx); + MLX5_SET(sqc, sq_ctx, state, sq_attr->state); + MLX5_SET(sqc, sq_ctx, hairpin_peer_rq, sq_attr->hairpin_peer_rq); + MLX5_SET(sqc, sq_ctx, hairpin_peer_vhca, sq_attr->hairpin_peer_vhca); + ret = mlx5_glue->devx_obj_modify(sq->obj, in, sizeof(in), + out, sizeof(out)); + if (ret) { + DRV_LOG(ERR, "Failed to modify SQ using DevX"); + rte_errno = errno; + return -errno; + } + return ret; +} + +/** + * Create TIS using DevX API. + * + * @param[in] ctx + * ibv_context returned from mlx5dv_open_device. + * @param [in] tis_attr + * Pointer to TIS attributes structure. + * + * @return + * The DevX object created, NULL otherwise and rte_errno is set. + */ +struct mlx5_devx_obj * +mlx5_devx_cmd_create_tis(struct ibv_context *ctx, + struct mlx5_devx_tis_attr *tis_attr) +{ + uint32_t in[MLX5_ST_SZ_DW(create_tis_in)] = {0}; + uint32_t out[MLX5_ST_SZ_DW(create_tis_out)] = {0}; + struct mlx5_devx_obj *tis = NULL; + void *tis_ctx; + + tis = rte_calloc(__func__, 1, sizeof(*tis), 0); + if (!tis) { + DRV_LOG(ERR, "Failed to allocate TIS object"); + rte_errno = ENOMEM; + return NULL; + } + MLX5_SET(create_tis_in, in, opcode, MLX5_CMD_OP_CREATE_TIS); + tis_ctx = MLX5_ADDR_OF(create_tis_in, in, ctx); + MLX5_SET(tisc, tis_ctx, strict_lag_tx_port_affinity, + tis_attr->strict_lag_tx_port_affinity); + MLX5_SET(tisc, tis_ctx, strict_lag_tx_port_affinity, + tis_attr->strict_lag_tx_port_affinity); + MLX5_SET(tisc, tis_ctx, prio, tis_attr->prio); + MLX5_SET(tisc, tis_ctx, transport_domain, + tis_attr->transport_domain); + tis->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in), + out, sizeof(out)); + if (!tis->obj) { + DRV_LOG(ERR, "Failed to create TIS using DevX"); + rte_errno = errno; + rte_free(tis); + return NULL; + } + tis->id = MLX5_GET(create_tis_out, out, tisn); + return tis; +} + +/** + * Create transport domain using DevX API. + * + * @param[in] ctx + * ibv_context returned from mlx5dv_open_device. + * + * @return + * The DevX object created, NULL otherwise and rte_errno is set. + */ +struct mlx5_devx_obj * +mlx5_devx_cmd_create_td(struct ibv_context *ctx) +{ + uint32_t in[MLX5_ST_SZ_DW(alloc_transport_domain_in)] = {0}; + uint32_t out[MLX5_ST_SZ_DW(alloc_transport_domain_out)] = {0}; + struct mlx5_devx_obj *td = NULL; + + td = rte_calloc(__func__, 1, sizeof(*td), 0); + if (!td) { + DRV_LOG(ERR, "Failed to allocate TD object"); + rte_errno = ENOMEM; + return NULL; + } + MLX5_SET(alloc_transport_domain_in, in, opcode, + MLX5_CMD_OP_ALLOC_TRANSPORT_DOMAIN); + td->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in), + out, sizeof(out)); + if (!td->obj) { + DRV_LOG(ERR, "Failed to create TIS using DevX"); + rte_errno = errno; + rte_free(td); + return NULL; + } + td->id = MLX5_GET(alloc_transport_domain_out, out, + transport_domain); + return td; +} diff --git a/drivers/net/mlx5/mlx5_prm.h b/drivers/net/mlx5/mlx5_prm.h index 3765df0..faa7996 100644 --- a/drivers/net/mlx5/mlx5_prm.h +++ b/drivers/net/mlx5/mlx5_prm.h @@ -666,9 +666,13 @@ enum { MLX5_CMD_OP_QUERY_HCA_CAP = 0x100, MLX5_CMD_OP_CREATE_MKEY = 0x200, MLX5_CMD_OP_QUERY_NIC_VPORT_CONTEXT = 0x754, + MLX5_CMD_OP_ALLOC_TRANSPORT_DOMAIN = 0x816, MLX5_CMD_OP_CREATE_TIR = 0x900, + MLX5_CMD_OP_CREATE_SQ = 0X904, + MLX5_CMD_OP_MODIFY_SQ = 0X905, MLX5_CMD_OP_CREATE_RQ = 0x908, MLX5_CMD_OP_MODIFY_RQ = 0x909, + MLX5_CMD_OP_CREATE_TIS = 0x912, MLX5_CMD_OP_QUERY_TIS = 0x915, MLX5_CMD_OP_CREATE_RQT = 0x916, MLX5_CMD_OP_ALLOC_FLOW_COUNTER = 0x939, @@ -1311,6 +1315,23 @@ struct mlx5_ifc_query_tis_in_bits { u8 reserved_at_60[0x20]; }; +struct mlx5_ifc_alloc_transport_domain_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x8]; + u8 transport_domain[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_alloc_transport_domain_in_bits { + u8 opcode[0x10]; + u8 reserved_at_10[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0x40]; +}; + enum { MLX5_WQ_TYPE_LINKED_LIST = 0x0, MLX5_WQ_TYPE_CYCLIC = 0x1, @@ -1427,6 +1448,24 @@ struct mlx5_ifc_modify_rq_out_bits { u8 reserved_at_40[0x40]; }; +struct mlx5_ifc_create_tis_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x8]; + u8 tisn[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_create_tis_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0xc0]; + struct mlx5_ifc_tisc_bits ctx; +}; + enum { MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_WQ_LWM = 1ULL << 0, MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_VSD = 1ULL << 1, @@ -1572,6 +1611,85 @@ struct mlx5_ifc_create_rqt_in_bits { #pragma GCC diagnostic error "-Wpedantic" #endif +struct mlx5_ifc_sqc_bits { + u8 rlky[0x1]; + u8 cd_master[0x1]; + u8 fre[0x1]; + u8 flush_in_error_en[0x1]; + u8 allow_multi_pkt_send_wqe[0x1]; + u8 min_wqe_inline_mode[0x3]; + u8 state[0x4]; + u8 reg_umr[0x1]; + u8 allow_swp[0x1]; + u8 hairpin[0x1]; + u8 reserved_at_f[0x11]; + u8 reserved_at_20[0x8]; + u8 user_index[0x18]; + u8 reserved_at_40[0x8]; + u8 cqn[0x18]; + u8 reserved_at_60[0x8]; + u8 hairpin_peer_rq[0x18]; + u8 reserved_at_80[0x10]; + u8 hairpin_peer_vhca[0x10]; + u8 reserved_at_a0[0x50]; + u8 packet_pacing_rate_limit_index[0x10]; + u8 tis_lst_sz[0x10]; + u8 reserved_at_110[0x10]; + u8 reserved_at_120[0x40]; + u8 reserved_at_160[0x8]; + u8 tis_num_0[0x18]; + struct mlx5_ifc_wq_bits wq; +}; + +struct mlx5_ifc_query_sq_in_bits { + u8 opcode[0x10]; + u8 reserved_at_10[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0x8]; + u8 sqn[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_modify_sq_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x40]; +}; + +struct mlx5_ifc_modify_sq_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 sq_state[0x4]; + u8 reserved_at_44[0x4]; + u8 sqn[0x18]; + u8 reserved_at_60[0x20]; + u8 modify_bitmask[0x40]; + u8 reserved_at_c0[0x40]; + struct mlx5_ifc_sqc_bits ctx; +}; + +struct mlx5_ifc_create_sq_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x8]; + u8 sqn[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_create_sq_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0xc0]; + struct mlx5_ifc_sqc_bits ctx; +}; + /* CQE format mask. */ #define MLX5E_CQE_FORMAT_MASK 0xc diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 12f9bfb..271b648 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -324,14 +324,18 @@ struct mlx5_txq_obj { LIST_ENTRY(mlx5_txq_obj) next; /* Pointer to the next element. */ rte_atomic32_t refcnt; /* Reference counter. */ struct mlx5_txq_ctrl *txq_ctrl; /* Pointer to the control queue. */ - enum mlx5_rxq_obj_type type; /* The txq object type. */ + enum mlx5_txq_obj_type type; /* The txq object type. */ RTE_STD_C11 union { struct { struct ibv_cq *cq; /* Completion Queue. */ struct ibv_qp *qp; /* Queue Pair. */ }; - struct mlx5_devx_obj *sq; /* DevX object for Sx queue. */ + struct { + struct mlx5_devx_obj *sq; + /* DevX object for Sx queue. */ + struct mlx5_devx_obj *tis; /* The TIS object. */ + }; }; }; @@ -348,6 +352,7 @@ struct mlx5_txq_ctrl { off_t uar_mmap_offset; /* UAR mmap offset for non-primary process. */ void *bf_reg; /* BlueFlame register from Verbs. */ uint16_t dump_file_n; /* Number of dump files. */ + struct rte_eth_hairpin_conf hairpin_conf; /* Hairpin configuration. */ struct mlx5_txq_data txq; /* Data path structure. */ /* Must be the last field in the structure, contains elts[]. */ }; @@ -410,15 +415,22 @@ struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev, int mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_txconf *conf); +int mlx5_tx_hairpin_queue_setup + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + const struct rte_eth_hairpin_conf *hairpin_conf); void mlx5_tx_queue_release(void *dpdk_txq); int mlx5_tx_uar_init_secondary(struct rte_eth_dev *dev, int fd); -struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx); +struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx, + enum mlx5_txq_obj_type type); struct mlx5_txq_obj *mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t idx); int mlx5_txq_obj_release(struct mlx5_txq_obj *txq_ibv); int mlx5_txq_obj_verify(struct rte_eth_dev *dev); struct mlx5_txq_ctrl *mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_txconf *conf); +struct mlx5_txq_ctrl *mlx5_txq_hairpin_new + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + const struct rte_eth_hairpin_conf *hairpin_conf); struct mlx5_txq_ctrl *mlx5_txq_get(struct rte_eth_dev *dev, uint16_t idx); int mlx5_txq_release(struct rte_eth_dev *dev, uint16_t idx); int mlx5_txq_releasable(struct rte_eth_dev *dev, uint16_t idx); diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 50c4df5..3ec86c4 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -51,8 +51,14 @@ if (!txq_ctrl) continue; - txq_alloc_elts(txq_ctrl); - txq_ctrl->obj = mlx5_txq_obj_new(dev, i); + if (txq_ctrl->type == MLX5_TXQ_TYPE_HAIRPIN) { + txq_ctrl->obj = mlx5_txq_obj_new + (dev, i, MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN); + } else { + txq_alloc_elts(txq_ctrl); + txq_ctrl->obj = mlx5_txq_obj_new + (dev, i, MLX5_TXQ_OBJ_TYPE_IBV); + } if (!txq_ctrl->obj) { rte_errno = ENOMEM; goto error; diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index a6e2563..f9bfe31 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -136,30 +136,22 @@ } /** - * DPDK callback to configure a TX queue. + * Tx queue presetup checks. * * @param dev * Pointer to Ethernet device structure. * @param idx - * TX queue index. + * Tx queue index. * @param desc * Number of descriptors to configure in queue. - * @param socket - * NUMA socket on which memory must be allocated. - * @param[in] conf - * Thresholds parameters. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, - unsigned int socket, const struct rte_eth_txconf *conf) +static int +mlx5_tx_queue_pre_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc) { struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_data *txq = (*priv->txqs)[idx]; - struct mlx5_txq_ctrl *txq_ctrl = - container_of(txq, struct mlx5_txq_ctrl, txq); if (desc <= MLX5_TX_COMP_THRESH) { DRV_LOG(WARNING, @@ -191,6 +183,38 @@ return -rte_errno; } mlx5_txq_release(dev, idx); + return 0; +} +/** + * DPDK callback to configure a TX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * TX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_txconf *conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq, struct mlx5_txq_ctrl, txq); + int res; + + res = mlx5_tx_queue_pre_setup(dev, idx, desc); + if (res) + return res; txq_ctrl = mlx5_txq_new(dev, idx, desc, socket, conf); if (!txq_ctrl) { DRV_LOG(ERR, "port %u unable to allocate queue index %u", @@ -204,6 +228,57 @@ } /** + * DPDK callback to configure a TX hairpin queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * TX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param[in] hairpin_conf + * The hairpin binding configuration. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_tx_hairpin_queue_setup(struct rte_eth_dev *dev, uint16_t idx, + uint16_t desc, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq, struct mlx5_txq_ctrl, txq); + int res; + + res = mlx5_tx_queue_pre_setup(dev, idx, desc); + if (res) + return res; + if (hairpin_conf->peer_n != 1 || + hairpin_conf->peers[0].port != dev->data->port_id || + hairpin_conf->peers[0].queue >= priv->rxqs_n) { + DRV_LOG(ERR, "port %u unable to setup hairpin queue index %u " + " invalid hairpind configuration", dev->data->port_id, + idx); + rte_errno = EINVAL; + return -rte_errno; + } + txq_ctrl = mlx5_txq_hairpin_new(dev, idx, desc, hairpin_conf); + if (!txq_ctrl) { + DRV_LOG(ERR, "port %u unable to allocate queue index %u", + dev->data->port_id, idx); + return -rte_errno; + } + DRV_LOG(DEBUG, "port %u adding Tx queue %u to list", + dev->data->port_id, idx); + (*priv->txqs)[idx] = &txq_ctrl->txq; + txq_ctrl->type = MLX5_TXQ_TYPE_HAIRPIN; + return 0; +} + +/** * DPDK callback to release a TX queue. * * @param dpdk_txq @@ -246,6 +321,8 @@ const size_t page_size = sysconf(_SC_PAGESIZE); #endif + if (txq_ctrl->type != MLX5_TXQ_TYPE_STANDARD) + return; assert(rte_eal_process_type() == RTE_PROC_PRIMARY); assert(ppriv); ppriv->uar_table[txq_ctrl->txq.idx] = txq_ctrl->bf_reg; @@ -282,6 +359,8 @@ uintptr_t offset; const size_t page_size = sysconf(_SC_PAGESIZE); + if (txq_ctrl->type != MLX5_TXQ_TYPE_STANDARD) + return 0; assert(ppriv); /* * As rdma-core, UARs are mapped in size of OS page @@ -316,6 +395,8 @@ const size_t page_size = sysconf(_SC_PAGESIZE); void *addr; + if (txq_ctrl->type != MLX5_TXQ_TYPE_STANDARD) + return; addr = ppriv->uar_table[txq_ctrl->txq.idx]; munmap(RTE_PTR_ALIGN_FLOOR(addr, page_size), page_size); } @@ -346,6 +427,8 @@ continue; txq = (*priv->txqs)[i]; txq_ctrl = container_of(txq, struct mlx5_txq_ctrl, txq); + if (txq_ctrl->type != MLX5_TXQ_TYPE_STANDARD) + continue; assert(txq->idx == (uint16_t)i); ret = txq_uar_init_secondary(txq_ctrl, fd); if (ret) @@ -365,18 +448,87 @@ } /** + * Create the Tx hairpin queue object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array + * + * @return + * The hairpin DevX object initialised, NULL otherwise and rte_errno is set. + */ +static struct mlx5_txq_obj * +mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + struct mlx5_devx_create_sq_attr attr = { 0 }; + struct mlx5_txq_obj *tmpl = NULL; + int ret = 0; + + assert(txq_data); + assert(!txq_ctrl->obj); + tmpl = rte_calloc_socket(__func__, 1, sizeof(*tmpl), 0, + txq_ctrl->socket); + if (!tmpl) { + DRV_LOG(ERR, + "port %u Tx queue %u cannot allocate memory resources", + dev->data->port_id, txq_data->idx); + rte_errno = ENOMEM; + goto error; + } + tmpl->type = MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN; + tmpl->txq_ctrl = txq_ctrl; + attr.hairpin = 1; + attr.tis_lst_sz = 1; + /* Workaround for hairpin startup */ + attr.wq_attr.log_hairpin_num_packets = log2above(32); + /* Workaround for packets larger than 1KB */ + attr.wq_attr.log_hairpin_data_sz = + priv->config.hca_attr.log_max_hairpin_wq_data_sz; + attr.tis_num = priv->sh->tis->id; + tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->ctx, &attr); + if (!tmpl->sq) { + DRV_LOG(ERR, + "port %u tx hairpin queue %u can't create sq object", + dev->data->port_id, idx); + rte_errno = errno; + goto error; + } + DRV_LOG(DEBUG, "port %u sxq %u updated with %p", dev->data->port_id, + idx, (void *)&tmpl); + rte_atomic32_inc(&tmpl->refcnt); + LIST_INSERT_HEAD(&priv->txqsobj, tmpl, next); + return tmpl; +error: + ret = rte_errno; /* Save rte_errno before cleanup. */ + if (tmpl->tis) + mlx5_devx_cmd_destroy(tmpl->tis); + if (tmpl->sq) + mlx5_devx_cmd_destroy(tmpl->sq); + rte_errno = ret; /* Restore rte_errno. */ + return NULL; +} + +/** * Create the Tx queue Verbs object. * * @param dev * Pointer to Ethernet device. * @param idx * Queue index in DPDK Tx queue array. + * @param type + * Type of the Tx queue object to create. * * @return * The Verbs object initialised, NULL otherwise and rte_errno is set. */ struct mlx5_txq_obj * -mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx) +mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx, + enum mlx5_txq_obj_type type) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; @@ -396,6 +548,8 @@ struct mlx5_txq_obj * const int desc = 1 << txq_data->elts_n; int ret = 0; + if (type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) + return mlx5_txq_obj_hairpin_new(dev, idx); #ifdef HAVE_IBV_FLOW_DV_SUPPORT /* If using DevX, need additional mask to read tisn value. */ if (priv->config.devx && !priv->sh->tdn) @@ -643,8 +797,13 @@ struct mlx5_txq_obj * { assert(txq_obj); if (rte_atomic32_dec_and_test(&txq_obj->refcnt)) { - claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); - claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); + if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) { + if (txq_obj->tis) + claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); + } else { + claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); + claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); + } LIST_REMOVE(txq_obj, next); rte_free(txq_obj); return 0; @@ -1100,6 +1259,7 @@ struct mlx5_txq_ctrl * goto error; } rte_atomic32_inc(&tmpl->refcnt); + tmpl->type = MLX5_TXQ_TYPE_STANDARD; LIST_INSERT_HEAD(&priv->txqsctrl, tmpl, next); return tmpl; error: @@ -1108,6 +1268,46 @@ struct mlx5_txq_ctrl * } /** + * Create a DPDK Tx hairpin queue. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * TX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param hairpin_conf + * The hairpin configuration. + * + * @return + * A DPDK queue object on success, NULL otherwise and rte_errno is set. + */ +struct mlx5_txq_ctrl * +mlx5_txq_hairpin_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + const struct rte_eth_hairpin_conf *hairpin_conf) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_ctrl *tmpl; + + tmpl = rte_calloc_socket("TXQ", 1, + sizeof(*tmpl), 0, SOCKET_ID_ANY); + if (!tmpl) { + rte_errno = ENOMEM; + return NULL; + } + tmpl->priv = priv; + tmpl->socket = SOCKET_ID_ANY; + tmpl->txq.elts_n = log2above(desc); + tmpl->txq.port_id = dev->data->port_id; + tmpl->txq.idx = idx; + tmpl->hairpin_conf = *hairpin_conf; + tmpl->type = MLX5_TXQ_TYPE_HAIRPIN; + rte_atomic32_inc(&tmpl->refcnt); + LIST_INSERT_HEAD(&priv->txqsctrl, tmpl, next); + return tmpl; +} + +/** * Get a Tx queue. * * @param dev From patchwork Thu Oct 17 15:32:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61418 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 281581EA02; Thu, 17 Oct 2019 17:32:53 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id AB6EF1E9EA for ; Thu, 17 Oct 2019 17:32:44 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:39 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfO026371; Thu, 17 Oct 2019 18:32:39 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:09 +0000 Message-Id: <1571326337-42692-8-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 07/15] net/mlx5: add get hairpin capabilities X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commits adds the hairpin get capabilities function. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 2 ++ drivers/net/mlx5/mlx5.h | 3 ++- drivers/net/mlx5/mlx5_ethdev.c | 27 +++++++++++++++++++++++++++ 3 files changed, 31 insertions(+), 1 deletion(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index c53a9c6..7962936 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1028,6 +1028,7 @@ struct mlx5_dev_spawn_data { .udp_tunnel_port_add = mlx5_udp_tunnel_port_add, .get_module_info = mlx5_get_module_info, .get_module_eeprom = mlx5_get_module_eeprom, + .hairpin_cap_get = mlx5_hairpin_cap_get, }; /* Available operations from secondary process. */ @@ -1090,6 +1091,7 @@ struct mlx5_dev_spawn_data { .is_removed = mlx5_is_removed, .get_module_info = mlx5_get_module_info, .get_module_eeprom = mlx5_get_module_eeprom, + .hairpin_cap_get = mlx5_hairpin_cap_get, }; /** diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 7ea4950..ce044b9 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -782,7 +782,8 @@ int mlx5_get_module_info(struct rte_eth_dev *dev, struct rte_eth_dev_module_info *modinfo); int mlx5_get_module_eeprom(struct rte_eth_dev *dev, struct rte_dev_eeprom_info *info); - +int mlx5_hairpin_cap_get(struct rte_eth_dev *dev, + struct rte_eth_hairpin_cap *cap); /* mlx5_mac.c */ int mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN]); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index f2b1752..95c70f7 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -2028,3 +2028,30 @@ int mlx5_get_module_eeprom(struct rte_eth_dev *dev, rte_free(eeprom); return ret; } + +/** + * DPDK callback to retrieve hairpin capabilities. + * + * @param dev + * Pointer to Ethernet device structure. + * @param[out] cap + * Storage for hairpin capability data. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int mlx5_hairpin_cap_get(struct rte_eth_dev *dev, + struct rte_eth_hairpin_cap *cap) +{ + struct mlx5_priv *priv = dev->data->dev_private; + + if (priv->sh->devx == 0) { + rte_errno = ENOTSUP; + return -rte_errno; + } + cap->max_n_queues = UINT16_MAX; + cap->max_rx_2_tx = 1; + cap->max_tx_2_rx = 1; + cap->max_nb_desc = 8192; + return 0; +} From patchwork Thu Oct 17 15:32:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61417 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1B71C1E9F9; Thu, 17 Oct 2019 17:32:46 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 796A11E9E6 for ; Thu, 17 Oct 2019 17:32:44 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:43 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfP026371; Thu, 17 Oct 2019 18:32:43 +0300 From: Ori Kam To: Wenzhuo Lu , Jingjing Wu , Bernard Iremonger Cc: dev@dpdk.org, orika@mellanox.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:10 +0000 Message-Id: <1571326337-42692-9-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 08/15] app/testpmd: add hairpin support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit introduce the hairpin queues to the testpmd. the hairpin queue is configured using --hairpinq= the hairpin queue adds n queue objects for both the total number of TX queues and RX queues. The connection between the queues are 1 to 1, first Rx hairpin queue will be connected to the first Tx hairpin queue Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- app/test-pmd/parameters.c | 28 ++++++++++++ app/test-pmd/testpmd.c | 109 +++++++++++++++++++++++++++++++++++++++++++++- app/test-pmd/testpmd.h | 3 ++ 3 files changed, 138 insertions(+), 2 deletions(-) diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c index 6c78dca..6246129 100644 --- a/app/test-pmd/parameters.c +++ b/app/test-pmd/parameters.c @@ -147,6 +147,8 @@ printf(" --rxd=N: set the number of descriptors in RX rings to N.\n"); printf(" --txq=N: set the number of TX queues per port to N.\n"); printf(" --txd=N: set the number of descriptors in TX rings to N.\n"); + printf(" --hairpinq=N: set the number of hairpin queues per port to " + "N.\n"); printf(" --burst=N: set the number of packets per burst to N.\n"); printf(" --mbcache=N: set the cache of mbuf memory pool to N.\n"); printf(" --rxpt=N: set prefetch threshold register of RX rings to N.\n"); @@ -618,6 +620,7 @@ { "txq", 1, 0, 0 }, { "rxd", 1, 0, 0 }, { "txd", 1, 0, 0 }, + { "hairpinq", 1, 0, 0 }, { "burst", 1, 0, 0 }, { "mbcache", 1, 0, 0 }, { "txpt", 1, 0, 0 }, @@ -1036,6 +1039,31 @@ " >= 0 && <= %u\n", n, get_allowed_max_nb_txq(&pid)); } + if (!strcmp(lgopts[opt_idx].name, "hairpinq")) { + n = atoi(optarg); + if (n >= 0 && + check_nb_hairpinq((queueid_t)n) == 0) + nb_hairpinq = (queueid_t) n; + else + rte_exit(EXIT_FAILURE, "txq %d invalid - must be" + " >= 0 && <= %u\n", n, + get_allowed_max_nb_hairpinq + (&pid)); + if ((n + nb_txq) < 0 || + check_nb_txq((queueid_t)(n + nb_txq)) != 0) + rte_exit(EXIT_FAILURE, "txq + hairpinq " + "%d invalid - must be" + " >= 0 && <= %u\n", + n + nb_txq, + get_allowed_max_nb_txq(&pid)); + if ((n + nb_rxq) < 0 || + check_nb_rxq((queueid_t)(n + nb_rxq)) != 0) + rte_exit(EXIT_FAILURE, "rxq + hairpinq " + "%d invalid - must be" + " >= 0 && <= %u\n", + n + nb_rxq, + get_allowed_max_nb_rxq(&pid)); + } if (!nb_rxq && !nb_txq) { rte_exit(EXIT_FAILURE, "Either rx or tx queues should " "be non-zero\n"); diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 5701f31..8290e22 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -235,6 +235,7 @@ struct fwd_engine * fwd_engines[] = { /* * Configurable number of RX/TX queues. */ +queueid_t nb_hairpinq; /**< Number of hairpin queues per port. */ queueid_t nb_rxq = 1; /**< Number of RX queues per port. */ queueid_t nb_txq = 1; /**< Number of TX queues per port. */ @@ -1103,6 +1104,53 @@ struct extmem_param { return 0; } +/* + * Get the allowed maximum number of hairpin queues. + * *pid return the port id which has minimal value of + * max_hairpin_queues in all ports. + */ +queueid_t +get_allowed_max_nb_hairpinq(portid_t *pid) +{ + queueid_t allowed_max_hairpinq = MAX_QUEUE_ID; + portid_t pi; + struct rte_eth_hairpin_cap cap; + + RTE_ETH_FOREACH_DEV(pi) { + if (rte_eth_dev_hairpin_capability_get(pi, &cap) != 0) { + *pid = pi; + return 0; + } + if (cap.max_n_queues < allowed_max_hairpinq) { + allowed_max_hairpinq = cap.max_n_queues; + *pid = pi; + } + } + return allowed_max_hairpinq; +} + +/* + * Check input hairpin is valid or not. + * If input hairpin is not greater than any of maximum number + * of hairpin queues of all ports, it is valid. + * if valid, return 0, else return -1 + */ +int +check_nb_hairpinq(queueid_t hairpinq) +{ + queueid_t allowed_max_hairpinq; + portid_t pid = 0; + + allowed_max_hairpinq = get_allowed_max_nb_hairpinq(&pid); + if (hairpinq > allowed_max_hairpinq) { + printf("Fail: input hairpin (%u) can't be greater " + "than max_hairpin_queues (%u) of port %u\n", + hairpinq, allowed_max_hairpinq, pid); + return -1; + } + return 0; +} + static void init_config(void) { @@ -2064,6 +2112,11 @@ struct extmem_param { queueid_t qi; struct rte_port *port; struct rte_ether_addr mac_addr; + struct rte_eth_hairpin_conf hairpin_conf = { + .peer_n = 1, + }; + int i; + struct rte_eth_hairpin_cap cap; if (port_id_is_invalid(pid, ENABLED_WARN)) return 0; @@ -2096,9 +2149,16 @@ struct extmem_param { configure_rxtx_dump_callbacks(0); printf("Configuring Port %d (socket %u)\n", pi, port->socket_id); + if (nb_hairpinq > 0 && + rte_eth_dev_hairpin_capability_get(pi, &cap)) { + printf("Port %d doesn't support hairpin " + "queues\n", pi); + return -1; + } /* configure port */ - diag = rte_eth_dev_configure(pi, nb_rxq, nb_txq, - &(port->dev_conf)); + diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq, + nb_txq + nb_hairpinq, + &(port->dev_conf)); if (diag != 0) { if (rte_atomic16_cmpset(&(port->port_status), RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0) @@ -2191,6 +2251,51 @@ struct extmem_param { port->need_reconfig_queues = 1; return -1; } + /* setup hairpin queues */ + i = 0; + for (qi = nb_txq; qi < nb_hairpinq + nb_txq; qi++) { + hairpin_conf.peers[0].port = pi; + hairpin_conf.peers[0].queue = i + nb_rxq; + diag = rte_eth_tx_hairpin_queue_setup + (pi, qi, nb_txd, &hairpin_conf); + i++; + if (diag == 0) + continue; + + /* Fail to setup rx queue, return */ + if (rte_atomic16_cmpset(&(port->port_status), + RTE_PORT_HANDLING, + RTE_PORT_STOPPED) == 0) + printf("Port %d can not be set back " + "to stopped\n", pi); + printf("Fail to configure port %d hairpin " + "queues\n", pi); + /* try to reconfigure queues next time */ + port->need_reconfig_queues = 1; + return -1; + } + i = 0; + for (qi = nb_rxq; qi < nb_hairpinq + nb_rxq; qi++) { + hairpin_conf.peers[0].port = pi; + hairpin_conf.peers[0].queue = i + nb_txq; + diag = rte_eth_rx_hairpin_queue_setup + (pi, qi, nb_rxd, &hairpin_conf); + i++; + if (diag == 0) + continue; + + /* Fail to setup rx queue, return */ + if (rte_atomic16_cmpset(&(port->port_status), + RTE_PORT_HANDLING, + RTE_PORT_STOPPED) == 0) + printf("Port %d can not be set back " + "to stopped\n", pi); + printf("Fail to configure port %d hairpin " + "queues\n", pi); + /* try to reconfigure queues next time */ + port->need_reconfig_queues = 1; + return -1; + } } configure_rxtx_dump_callbacks(verbose_level); /* start port */ diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index f8ebe71..0682c11 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -383,6 +383,7 @@ struct queue_stats_mappings { extern uint64_t rss_hf; +extern queueid_t nb_hairpinq; extern queueid_t nb_rxq; extern queueid_t nb_txq; @@ -854,6 +855,8 @@ enum print_warning { int check_nb_rxq(queueid_t rxq); queueid_t get_allowed_max_nb_txq(portid_t *pid); int check_nb_txq(queueid_t txq); +queueid_t get_allowed_max_nb_hairpinq(portid_t *pid); +int check_nb_hairpinq(queueid_t hairpinq); uint16_t dump_rx_pkts(uint16_t port_id, uint16_t queue, struct rte_mbuf *pkts[], uint16_t nb_pkts, __rte_unused uint16_t max_pkts, From patchwork Thu Oct 17 15:32:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61419 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5AF181EA1D; Thu, 17 Oct 2019 17:32:59 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 8EDCE1E9E6 for ; Thu, 17 Oct 2019 17:32:49 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:45 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfQ026371; Thu, 17 Oct 2019 18:32:44 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:11 +0000 Message-Id: <1571326337-42692-10-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 09/15] net/mlx5: add hairpin binding function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When starting the port, in addition to creating the queues we need to bind the hairpin queues. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_devx_cmds.c | 1 + drivers/net/mlx5/mlx5_prm.h | 6 +++ drivers/net/mlx5/mlx5_trigger.c | 97 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 105 insertions(+) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index ce044b9..a43accf 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -188,6 +188,7 @@ struct mlx5_hca_attr { uint32_t log_max_hairpin_queues:5; uint32_t log_max_hairpin_wq_data_sz:5; uint32_t log_max_hairpin_num_packets:5; + uint32_t vhca_id:16; }; /* Flow list . */ diff --git a/drivers/net/mlx5/mlx5_devx_cmds.c b/drivers/net/mlx5/mlx5_devx_cmds.c index 917bbf9..0243733 100644 --- a/drivers/net/mlx5/mlx5_devx_cmds.c +++ b/drivers/net/mlx5/mlx5_devx_cmds.c @@ -334,6 +334,7 @@ struct mlx5_devx_obj * log_max_hairpin_wq_data_sz); attr->log_max_hairpin_num_packets = MLX5_GET (cmd_hca_cap, hcattr, log_min_hairpin_wq_data_sz); + attr->vhca_id = MLX5_GET(cmd_hca_cap, hcattr, vhca_id); attr->eth_net_offloads = MLX5_GET(cmd_hca_cap, hcattr, eth_net_offloads); attr->eth_virt = MLX5_GET(cmd_hca_cap, hcattr, eth_virt); diff --git a/drivers/net/mlx5/mlx5_prm.h b/drivers/net/mlx5/mlx5_prm.h index faa7996..d4084db 100644 --- a/drivers/net/mlx5/mlx5_prm.h +++ b/drivers/net/mlx5/mlx5_prm.h @@ -1611,6 +1611,12 @@ struct mlx5_ifc_create_rqt_in_bits { #pragma GCC diagnostic error "-Wpedantic" #endif +enum { + MLX5_SQC_STATE_RST = 0x0, + MLX5_SQC_STATE_RDY = 0x1, + MLX5_SQC_STATE_ERR = 0x3, +}; + struct mlx5_ifc_sqc_bits { u8 rlky[0x1]; u8 cd_master[0x1]; diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 3ec86c4..a4fcdb3 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -162,6 +162,96 @@ } /** + * Binds Tx queues to Rx queues for hairpin. + * + * Binds Tx queues to the target Rx queues. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_hairpin_bind(struct rte_eth_dev *dev) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_devx_modify_sq_attr sq_attr = { 0 }; + struct mlx5_devx_modify_rq_attr rq_attr = { 0 }; + struct mlx5_txq_ctrl *txq_ctrl; + struct mlx5_rxq_ctrl *rxq_ctrl; + struct mlx5_devx_obj *sq; + struct mlx5_devx_obj *rq; + unsigned int i; + int ret = 0; + + for (i = 0; i != priv->txqs_n; ++i) { + txq_ctrl = mlx5_txq_get(dev, i); + if (!txq_ctrl) + continue; + if (txq_ctrl->type != MLX5_TXQ_TYPE_HAIRPIN) { + mlx5_txq_release(dev, i); + continue; + } + if (!txq_ctrl->obj) { + rte_errno = ENOMEM; + DRV_LOG(ERR, "port %u no txq object found: %d", + dev->data->port_id, i); + mlx5_txq_release(dev, i); + return -rte_errno; + } + sq = txq_ctrl->obj->sq; + rxq_ctrl = mlx5_rxq_get(dev, + txq_ctrl->hairpin_conf.peers[0].queue); + if (!rxq_ctrl) { + mlx5_txq_release(dev, i); + rte_errno = EINVAL; + DRV_LOG(ERR, "port %u no rxq object found: %d", + dev->data->port_id, + txq_ctrl->hairpin_conf.peers[0].queue); + return -rte_errno; + } + if (rxq_ctrl->type != MLX5_RXQ_TYPE_HAIRPIN || + rxq_ctrl->hairpin_conf.peers[0].queue != i) { + rte_errno = ENOMEM; + DRV_LOG(ERR, "port %u Tx queue %d can't be binded to " + "Rx queue %d", dev->data->port_id, + i, txq_ctrl->hairpin_conf.peers[0].queue); + goto error; + } + rq = rxq_ctrl->obj->rq; + if (!rq) { + rte_errno = ENOMEM; + DRV_LOG(ERR, "port %u hairpin no matching rxq: %d", + dev->data->port_id, + txq_ctrl->hairpin_conf.peers[0].queue); + goto error; + } + sq_attr.state = MLX5_SQC_STATE_RDY; + sq_attr.sq_state = MLX5_SQC_STATE_RST; + sq_attr.hairpin_peer_rq = rq->id; + sq_attr.hairpin_peer_vhca = priv->config.hca_attr.vhca_id; + ret = mlx5_devx_cmd_modify_sq(sq, &sq_attr); + if (ret) + goto error; + rq_attr.state = MLX5_SQC_STATE_RDY; + rq_attr.rq_state = MLX5_SQC_STATE_RST; + rq_attr.hairpin_peer_sq = sq->id; + rq_attr.hairpin_peer_vhca = priv->config.hca_attr.vhca_id; + ret = mlx5_devx_cmd_modify_rq(rq, &rq_attr); + if (ret) + goto error; + mlx5_txq_release(dev, i); + mlx5_rxq_release(dev, txq_ctrl->hairpin_conf.peers[0].queue); + } + return 0; +error: + mlx5_txq_release(dev, i); + mlx5_rxq_release(dev, txq_ctrl->hairpin_conf.peers[0].queue); + return -rte_errno; +} + +/** * DPDK callback to start the device. * * Simulate device start by attaching all configured flows. @@ -192,6 +282,13 @@ mlx5_txq_stop(dev); return -rte_errno; } + ret = mlx5_hairpin_bind(dev); + if (ret) { + DRV_LOG(ERR, "port %u hairpin binding failed: %s", + dev->data->port_id, strerror(rte_errno)); + mlx5_txq_stop(dev); + return -rte_errno; + } dev->data->dev_started = 1; ret = mlx5_rx_intr_vec_enable(dev); if (ret) { From patchwork Thu Oct 17 15:32:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61420 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E22551EA26; Thu, 17 Oct 2019 17:33:04 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id A45381EA02 for ; Thu, 17 Oct 2019 17:32:49 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:47 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfR026371; Thu, 17 Oct 2019 18:32:47 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:12 +0000 Message-Id: <1571326337-42692-11-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 10/15] net/mlx5: add support for hairpin hrxq X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support for rss on hairpin queues. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.h | 3 ++ drivers/net/mlx5/mlx5_ethdev.c | 102 ++++++++++++++++++++++++++++++---------- drivers/net/mlx5/mlx5_rss.c | 1 + drivers/net/mlx5/mlx5_rxq.c | 22 ++++++--- drivers/net/mlx5/mlx5_trigger.c | 6 +++ 5 files changed, 104 insertions(+), 30 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index a43accf..391ae2c 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -711,6 +711,7 @@ struct mlx5_priv { rte_spinlock_t uar_lock[MLX5_UAR_PAGE_NUM_MAX]; /* UAR same-page access control required in 32bit implementations. */ #endif + uint8_t skip_default_rss_reta; /* Skip configuration of default reta. */ }; #define PORT_ID(priv) ((priv)->dev_data->port_id) @@ -785,6 +786,8 @@ int mlx5_get_module_eeprom(struct rte_eth_dev *dev, struct rte_dev_eeprom_info *info); int mlx5_hairpin_cap_get(struct rte_eth_dev *dev, struct rte_eth_hairpin_cap *cap); +int mlx5_dev_configure_rss_reta(struct rte_eth_dev *dev); + /* mlx5_mac.c */ int mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN]); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 95c70f7..5b811e8 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -383,9 +383,6 @@ struct ethtool_link_settings { struct mlx5_priv *priv = dev->data->dev_private; unsigned int rxqs_n = dev->data->nb_rx_queues; unsigned int txqs_n = dev->data->nb_tx_queues; - unsigned int i; - unsigned int j; - unsigned int reta_idx_n; const uint8_t use_app_rss_key = !!dev->data->dev_conf.rx_adv_conf.rss_conf.rss_key; int ret = 0; @@ -431,28 +428,8 @@ struct ethtool_link_settings { DRV_LOG(INFO, "port %u Rx queues number update: %u -> %u", dev->data->port_id, priv->rxqs_n, rxqs_n); priv->rxqs_n = rxqs_n; - /* - * If the requested number of RX queues is not a power of two, - * use the maximum indirection table size for better balancing. - * The result is always rounded to the next power of two. - */ - reta_idx_n = (1 << log2above((rxqs_n & (rxqs_n - 1)) ? - priv->config.ind_table_max_size : - rxqs_n)); - ret = mlx5_rss_reta_index_resize(dev, reta_idx_n); - if (ret) - return ret; - /* - * When the number of RX queues is not a power of two, - * the remaining table entries are padded with reused WQs - * and hashes are not spread uniformly. - */ - for (i = 0, j = 0; (i != reta_idx_n); ++i) { - (*priv->reta_idx)[i] = j; - if (++j == rxqs_n) - j = 0; - } } + priv->skip_default_rss_reta = 0; ret = mlx5_proc_priv_init(dev); if (ret) return ret; @@ -460,6 +437,83 @@ struct ethtool_link_settings { } /** + * Configure default RSS reta. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_dev_configure_rss_reta(struct rte_eth_dev *dev) +{ + struct mlx5_priv *priv = dev->data->dev_private; + unsigned int rxqs_n = dev->data->nb_rx_queues; + unsigned int i; + unsigned int j; + unsigned int reta_idx_n; + int ret = 0; + unsigned int *rss_queue_arr = NULL; + unsigned int rss_queue_n = 0; + + if (priv->skip_default_rss_reta) + return ret; + rss_queue_arr = rte_malloc("", rxqs_n * sizeof(unsigned int), 0); + if (!rss_queue_arr) { + DRV_LOG(ERR, "port %u cannot allocate RSS queue list (%u)", + dev->data->port_id, rxqs_n); + rte_errno = ENOMEM; + return -rte_errno; + } + for (i = 0, j = 0; i < rxqs_n; i++) { + struct mlx5_rxq_data *rxq_data; + struct mlx5_rxq_ctrl *rxq_ctrl; + + rxq_data = (*priv->rxqs)[i]; + rxq_ctrl = container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); + if (rxq_ctrl->type == MLX5_RXQ_TYPE_STANDARD) + rss_queue_arr[j++] = i; + } + rss_queue_n = j; + if (rss_queue_n > priv->config.ind_table_max_size) { + DRV_LOG(ERR, "port %u cannot handle this many Rx queues (%u)", + dev->data->port_id, rss_queue_n); + rte_errno = EINVAL; + rte_free(rss_queue_arr); + return -rte_errno; + } + DRV_LOG(INFO, "port %u Rx queues number update: %u -> %u", + dev->data->port_id, priv->rxqs_n, rxqs_n); + priv->rxqs_n = rxqs_n; + /* + * If the requested number of RX queues is not a power of two, + * use the maximum indirection table size for better balancing. + * The result is always rounded to the next power of two. + */ + reta_idx_n = (1 << log2above((rss_queue_n & (rss_queue_n - 1)) ? + priv->config.ind_table_max_size : + rss_queue_n)); + ret = mlx5_rss_reta_index_resize(dev, reta_idx_n); + if (ret) { + rte_free(rss_queue_arr); + return ret; + } + /* + * When the number of RX queues is not a power of two, + * the remaining table entries are padded with reused WQs + * and hashes are not spread uniformly. + */ + for (i = 0, j = 0; (i != reta_idx_n); ++i) { + (*priv->reta_idx)[i] = rss_queue_arr[j]; + if (++j == rss_queue_n) + j = 0; + } + rte_free(rss_queue_arr); + return ret; +} + +/** * Sets default tuning parameters. * * @param dev diff --git a/drivers/net/mlx5/mlx5_rss.c b/drivers/net/mlx5/mlx5_rss.c index 891d764..1028264 100644 --- a/drivers/net/mlx5/mlx5_rss.c +++ b/drivers/net/mlx5/mlx5_rss.c @@ -223,6 +223,7 @@ } if (dev->data->dev_started) { mlx5_dev_stop(dev); + priv->skip_default_rss_reta = 1; return mlx5_dev_start(dev); } return 0; diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 66596df..a8ff8b2 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -2156,9 +2156,13 @@ struct mlx5_rxq_ctrl * } } else { /* ind_tbl->type == MLX5_IND_TBL_TYPE_DEVX */ struct mlx5_devx_rqt_attr *rqt_attr = NULL; + const unsigned int rqt_n = + 1 << (rte_is_power_of_2(queues_n) ? + log2above(queues_n) : + log2above(priv->config.ind_table_max_size)); rqt_attr = rte_calloc(__func__, 1, sizeof(*rqt_attr) + - queues_n * sizeof(uint32_t), 0); + rqt_n * sizeof(uint32_t), 0); if (!rqt_attr) { DRV_LOG(ERR, "port %u cannot allocate RQT resources", dev->data->port_id); @@ -2166,7 +2170,7 @@ struct mlx5_rxq_ctrl * goto error; } rqt_attr->rqt_max_size = priv->config.ind_table_max_size; - rqt_attr->rqt_actual_size = queues_n; + rqt_attr->rqt_actual_size = rqt_n; for (i = 0; i != queues_n; ++i) { struct mlx5_rxq_ctrl *rxq = mlx5_rxq_get(dev, queues[i]); @@ -2175,6 +2179,9 @@ struct mlx5_rxq_ctrl * rqt_attr->rq_list[i] = rxq->obj->rq->id; ind_tbl->queues[i] = queues[i]; } + k = i; /* Retain value of i for use in error case. */ + for (j = 0; k != rqt_n; ++k, ++j) + rqt_attr->rq_list[k] = rqt_attr->rq_list[j]; ind_tbl->rqt = mlx5_devx_cmd_create_rqt(priv->sh->ctx, rqt_attr); rte_free(rqt_attr); @@ -2328,13 +2335,13 @@ struct mlx5_hrxq * struct mlx5_ind_table_obj *ind_tbl; int err; struct mlx5_devx_obj *tir = NULL; + struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[queues[0]]; + struct mlx5_rxq_ctrl *rxq_ctrl = + container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); queues_n = hash_fields ? queues_n : 1; ind_tbl = mlx5_ind_table_obj_get(dev, queues, queues_n); if (!ind_tbl) { - struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[queues[0]]; - struct mlx5_rxq_ctrl *rxq_ctrl = - container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); enum mlx5_ind_tbl_type type; type = rxq_ctrl->obj->type == MLX5_RXQ_OBJ_TYPE_IBV ? @@ -2430,7 +2437,10 @@ struct mlx5_hrxq * tir_attr.rx_hash_fn = MLX5_RX_HASH_FN_TOEPLITZ; memcpy(&tir_attr.rx_hash_field_selector_outer, &hash_fields, sizeof(uint64_t)); - tir_attr.transport_domain = priv->sh->tdn; + if (rxq_ctrl->obj->type == MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN) + tir_attr.transport_domain = priv->sh->td->id; + else + tir_attr.transport_domain = priv->sh->tdn; memcpy(tir_attr.rx_hash_toeplitz_key, rss_key, rss_key_len); tir_attr.indirect_table = ind_tbl->rqt->id; if (dev->data->dev_conf.lpbk_mode) diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index a4fcdb3..f66b6ee 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -269,6 +269,12 @@ int ret; DRV_LOG(DEBUG, "port %u starting device", dev->data->port_id); + ret = mlx5_dev_configure_rss_reta(dev); + if (ret) { + DRV_LOG(ERR, "port %u reta config failed: %s", + dev->data->port_id, strerror(rte_errno)); + return -rte_errno; + } ret = mlx5_txq_start(dev); if (ret) { DRV_LOG(ERR, "port %u Tx queue allocation failed: %s", From patchwork Thu Oct 17 15:32:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61421 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 56CE81EA3D; Thu, 17 Oct 2019 17:33:09 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 8BC6B1EA09 for ; Thu, 17 Oct 2019 17:32:54 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:49 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfS026371; Thu, 17 Oct 2019 18:32:49 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:13 +0000 Message-Id: <1571326337-42692-12-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 11/15] net/mlx5: add internal tag item and action X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit introduce the setting and matching on regiters. This item and and action will be used with number of different features like hairpin, metering, metadata. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5_flow.c | 52 +++++++++++++ drivers/net/mlx5/mlx5_flow.h | 54 ++++++++++++-- drivers/net/mlx5/mlx5_flow_dv.c | 158 +++++++++++++++++++++++++++++++++++++++- drivers/net/mlx5/mlx5_prm.h | 3 +- 4 files changed, 257 insertions(+), 10 deletions(-) diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index 578d003..b4bcd1a 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -316,6 +316,58 @@ struct mlx5_flow_tunnel_info { }, }; +enum mlx5_feature_name { + MLX5_HAIRPIN_RX, + MLX5_HAIRPIN_TX, + MLX5_APPLICATION, +}; + +/** + * Translate tag ID to register. + * + * @param[in] dev + * Pointer to the Ethernet device structure. + * @param[in] feature + * The feature that request the register. + * @param[in] id + * The request register ID. + * @param[out] error + * Error description in case of any. + * + * @return + * The request register on success, a negative errno + * value otherwise and rte_errno is set. + */ +__rte_unused +static enum modify_reg flow_get_reg_id(struct rte_eth_dev *dev, + enum mlx5_feature_name feature, + uint32_t id, + struct rte_flow_error *error) +{ + static enum modify_reg id2reg[] = { + [0] = REG_A, + [1] = REG_C_2, + [2] = REG_C_3, + [3] = REG_C_4, + [4] = REG_B,}; + + dev = (void *)dev; + switch (feature) { + case MLX5_HAIRPIN_RX: + return REG_B; + case MLX5_HAIRPIN_TX: + return REG_A; + case MLX5_APPLICATION: + if (id > 4) + return rte_flow_error_set(error, EINVAL, + RTE_FLOW_ERROR_TYPE_ITEM, + NULL, "invalid tag id"); + return id2reg[id]; + } + return rte_flow_error_set(error, EINVAL, RTE_FLOW_ERROR_TYPE_ITEM, + NULL, "invalid feature name"); +} + /** * Discover the maximum number of priority available. * diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 235bccd..0148c1b 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -27,6 +27,43 @@ #include "mlx5.h" #include "mlx5_prm.h" +enum modify_reg { + REG_A, + REG_B, + REG_C_0, + REG_C_1, + REG_C_2, + REG_C_3, + REG_C_4, + REG_C_5, + REG_C_6, + REG_C_7, +}; + +/* Private rte flow items. */ +enum mlx5_rte_flow_item_type { + MLX5_RTE_FLOW_ITEM_TYPE_END = INT_MIN, + MLX5_RTE_FLOW_ITEM_TYPE_TAG, +}; + +/* Private rte flow actions. */ +enum mlx5_rte_flow_action_type { + MLX5_RTE_FLOW_ACTION_TYPE_END = INT_MIN, + MLX5_RTE_FLOW_ACTION_TYPE_TAG, +}; + +/* Matches on selected register. */ +struct mlx5_rte_flow_item_tag { + uint16_t id; + rte_be32_t data; +}; + +/* Modify selected register. */ +struct mlx5_rte_flow_action_set_tag { + uint16_t id; + rte_be32_t data; +}; + /* Pattern outer Layer bits. */ #define MLX5_FLOW_LAYER_OUTER_L2 (1u << 0) #define MLX5_FLOW_LAYER_OUTER_L3_IPV4 (1u << 1) @@ -53,16 +90,17 @@ /* General pattern items bits. */ #define MLX5_FLOW_ITEM_METADATA (1u << 16) #define MLX5_FLOW_ITEM_PORT_ID (1u << 17) +#define MLX5_FLOW_ITEM_TAG (1u << 18) /* Pattern MISC bits. */ -#define MLX5_FLOW_LAYER_ICMP (1u << 18) -#define MLX5_FLOW_LAYER_ICMP6 (1u << 19) -#define MLX5_FLOW_LAYER_GRE_KEY (1u << 20) +#define MLX5_FLOW_LAYER_ICMP (1u << 19) +#define MLX5_FLOW_LAYER_ICMP6 (1u << 20) +#define MLX5_FLOW_LAYER_GRE_KEY (1u << 21) /* Pattern tunnel Layer bits (continued). */ -#define MLX5_FLOW_LAYER_IPIP (1u << 21) -#define MLX5_FLOW_LAYER_IPV6_ENCAP (1u << 22) -#define MLX5_FLOW_LAYER_NVGRE (1u << 23) +#define MLX5_FLOW_LAYER_IPIP (1u << 22) +#define MLX5_FLOW_LAYER_IPV6_ENCAP (1u << 23) +#define MLX5_FLOW_LAYER_NVGRE (1u << 24) /* Outer Masks. */ #define MLX5_FLOW_LAYER_OUTER_L3 \ @@ -139,6 +177,7 @@ #define MLX5_FLOW_ACTION_DEC_TCP_SEQ (1u << 29) #define MLX5_FLOW_ACTION_INC_TCP_ACK (1u << 30) #define MLX5_FLOW_ACTION_DEC_TCP_ACK (1u << 31) +#define MLX5_FLOW_ACTION_SET_TAG (1ull << 32) #define MLX5_FLOW_FATE_ACTIONS \ (MLX5_FLOW_ACTION_DROP | MLX5_FLOW_ACTION_QUEUE | \ @@ -172,7 +211,8 @@ MLX5_FLOW_ACTION_DEC_TCP_SEQ | \ MLX5_FLOW_ACTION_INC_TCP_ACK | \ MLX5_FLOW_ACTION_DEC_TCP_ACK | \ - MLX5_FLOW_ACTION_OF_SET_VLAN_VID) + MLX5_FLOW_ACTION_OF_SET_VLAN_VID | \ + MLX5_FLOW_ACTION_SET_TAG) #define MLX5_FLOW_VLAN_ACTIONS (MLX5_FLOW_ACTION_OF_POP_VLAN | \ MLX5_FLOW_ACTION_OF_PUSH_VLAN) diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index f0422dc..dde6673 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -723,6 +723,59 @@ struct field_modify_info modify_tcp[] = { MLX5_MODIFICATION_TYPE_ADD, error); } +static enum mlx5_modification_field reg_to_field[] = { + [REG_A] = MLX5_MODI_META_DATA_REG_A, + [REG_B] = MLX5_MODI_META_DATA_REG_B, + [REG_C_0] = MLX5_MODI_META_REG_C_0, + [REG_C_1] = MLX5_MODI_META_REG_C_1, + [REG_C_2] = MLX5_MODI_META_REG_C_2, + [REG_C_3] = MLX5_MODI_META_REG_C_3, + [REG_C_4] = MLX5_MODI_META_REG_C_4, + [REG_C_5] = MLX5_MODI_META_REG_C_5, + [REG_C_6] = MLX5_MODI_META_REG_C_6, + [REG_C_7] = MLX5_MODI_META_REG_C_7, +}; + +/** + * Convert register set to DV specification. + * + * @param[in,out] resource + * Pointer to the modify-header resource. + * @param[in] action + * Pointer to action specification. + * @param[out] error + * Pointer to the error structure. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +flow_dv_convert_action_set_reg + (struct mlx5_flow_dv_modify_hdr_resource *resource, + const struct rte_flow_action *action, + struct rte_flow_error *error) +{ + const struct mlx5_rte_flow_action_set_tag *conf = (action->conf); + struct mlx5_modification_cmd *actions = resource->actions; + uint32_t i = resource->actions_num; + + if (i >= MLX5_MODIFY_NUM) + return rte_flow_error_set(error, EINVAL, + RTE_FLOW_ERROR_TYPE_ACTION, NULL, + "too many items to modify"); + actions[i].action_type = MLX5_MODIFICATION_TYPE_SET; + actions[i].field = reg_to_field[conf->id]; + actions[i].data0 = rte_cpu_to_be_32(actions[i].data0); + actions[i].data1 = conf->data; + ++i; + resource->actions_num = i; + if (!resource->actions_num) + return rte_flow_error_set(error, EINVAL, + RTE_FLOW_ERROR_TYPE_ACTION, NULL, + "invalid modification flow item"); + return 0; +} + /** * Validate META item. * @@ -4640,6 +4693,94 @@ struct field_modify_info modify_tcp[] = { } /** + * Add tag item to matcher + * + * @param[in, out] matcher + * Flow matcher. + * @param[in, out] key + * Flow matcher value. + * @param[in] item + * Flow pattern to translate. + */ +static void +flow_dv_translate_item_tag(void *matcher, void *key, + const struct rte_flow_item *item) +{ + void *misc2_m = + MLX5_ADDR_OF(fte_match_param, matcher, misc_parameters_2); + void *misc2_v = + MLX5_ADDR_OF(fte_match_param, key, misc_parameters_2); + const struct mlx5_rte_flow_item_tag *tag_v = item->spec; + const struct mlx5_rte_flow_item_tag *tag_m = item->mask; + enum modify_reg reg = tag_v->id; + rte_be32_t value = tag_v->data; + rte_be32_t mask = tag_m->data; + + switch (reg) { + case REG_A: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_a, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_a, + rte_be_to_cpu_32(value)); + break; + case REG_B: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_b, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_b, + rte_be_to_cpu_32(value)); + break; + case REG_C_0: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_0, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_0, + rte_be_to_cpu_32(value)); + break; + case REG_C_1: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_1, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_1, + rte_be_to_cpu_32(value)); + break; + case REG_C_2: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_2, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_2, + rte_be_to_cpu_32(value)); + break; + case REG_C_3: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_3, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_3, + rte_be_to_cpu_32(value)); + break; + case REG_C_4: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_4, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_4, + rte_be_to_cpu_32(value)); + break; + case REG_C_5: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_5, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_5, + rte_be_to_cpu_32(value)); + break; + case REG_C_6: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_6, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_6, + rte_be_to_cpu_32(value)); + break; + case REG_C_7: + MLX5_SET(fte_match_set_misc2, misc2_m, metadata_reg_c_7, + rte_be_to_cpu_32(mask)); + MLX5_SET(fte_match_set_misc2, misc2_v, metadata_reg_c_7, + rte_be_to_cpu_32(value)); + break; + } +} + +/** * Add source vport match to the specified matcher. * * @param[in, out] matcher @@ -5225,8 +5366,9 @@ struct field_modify_info modify_tcp[] = { struct mlx5_flow_tbl_resource *tbl; uint32_t port_id = 0; struct mlx5_flow_dv_port_id_action_resource port_id_resource; + int action_type = actions->type; - switch (actions->type) { + switch (action_type) { case RTE_FLOW_ACTION_TYPE_VOID: break; case RTE_FLOW_ACTION_TYPE_PORT_ID: @@ -5541,6 +5683,12 @@ struct field_modify_info modify_tcp[] = { MLX5_FLOW_ACTION_INC_TCP_ACK : MLX5_FLOW_ACTION_DEC_TCP_ACK; break; + case MLX5_RTE_FLOW_ACTION_TYPE_TAG: + if (flow_dv_convert_action_set_reg(&res, actions, + error)) + return -rte_errno; + action_flags |= MLX5_FLOW_ACTION_SET_TAG; + break; case RTE_FLOW_ACTION_TYPE_END: actions_end = true; if (action_flags & MLX5_FLOW_MODIFY_HDR_ACTIONS) { @@ -5565,8 +5713,9 @@ struct field_modify_info modify_tcp[] = { flow->actions = action_flags; for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) { int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); + int item_type = items->type; - switch (items->type) { + switch (item_type) { case RTE_FLOW_ITEM_TYPE_PORT_ID: flow_dv_translate_item_port_id(dev, match_mask, match_value, items); @@ -5712,6 +5861,11 @@ struct field_modify_info modify_tcp[] = { items, tunnel); last_item = MLX5_FLOW_LAYER_ICMP6; break; + case MLX5_RTE_FLOW_ITEM_TYPE_TAG: + flow_dv_translate_item_tag(match_mask, match_value, + items); + last_item = MLX5_FLOW_ITEM_TAG; + break; default: break; } diff --git a/drivers/net/mlx5/mlx5_prm.h b/drivers/net/mlx5/mlx5_prm.h index d4084db..695578f 100644 --- a/drivers/net/mlx5/mlx5_prm.h +++ b/drivers/net/mlx5/mlx5_prm.h @@ -623,7 +623,8 @@ struct mlx5_ifc_fte_match_set_misc2_bits { u8 metadata_reg_c_1[0x20]; u8 metadata_reg_c_0[0x20]; u8 metadata_reg_a[0x20]; - u8 reserved_at_1a0[0x60]; + u8 metadata_reg_b[0x20]; + u8 reserved_at_1c0[0x40]; }; struct mlx5_ifc_fte_match_set_misc3_bits { From patchwork Thu Oct 17 15:32:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61423 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BA1221EA5A; Thu, 17 Oct 2019 17:33:18 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 9C2161EA29 for ; Thu, 17 Oct 2019 17:32:59 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:56 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfT026371; Thu, 17 Oct 2019 18:32:51 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:14 +0000 Message-Id: <1571326337-42692-13-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 12/15] net/mlx5: add id generation function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When splitting flows for example in hairpin / metering, there is a need to combine the flows. This is done using ID. This commit introduce a simple way to generate such IDs. The reason why bitmap was not used is due to fact that the release and allocation are O(n) while in the chosen approch the allocation and release are O(1) Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 120 ++++++++++++++++++++++++++++++++++++++++++- drivers/net/mlx5/mlx5_flow.h | 14 +++++ 2 files changed, 133 insertions(+), 1 deletion(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 7962936..0c3239c 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -179,6 +179,124 @@ struct mlx5_dev_spawn_data { static LIST_HEAD(, mlx5_ibv_shared) mlx5_ibv_list = LIST_HEAD_INITIALIZER(); static pthread_mutex_t mlx5_ibv_list_mutex = PTHREAD_MUTEX_INITIALIZER; +#define MLX5_FLOW_MIN_ID_POOL_SIZE 512 +#define MLX5_ID_GENERATION_ARRAY_FACTOR 16 + +/** + * Allocate ID pool structure. + * + * @return + * Pointer to pool object, NULL value otherwise. + */ +struct mlx5_flow_id_pool * +mlx5_flow_id_pool_alloc(void) +{ + struct mlx5_flow_id_pool *pool; + void *mem; + + pool = rte_zmalloc("id pool allocation", sizeof(*pool), + RTE_CACHE_LINE_SIZE); + if (!pool) { + DRV_LOG(ERR, "can't allocate id pool"); + rte_errno = ENOMEM; + return NULL; + } + mem = rte_zmalloc("", MLX5_FLOW_MIN_ID_POOL_SIZE * sizeof(uint32_t), + RTE_CACHE_LINE_SIZE); + if (!mem) { + DRV_LOG(ERR, "can't allocate mem for id pool"); + rte_errno = ENOMEM; + goto error; + } + pool->free_arr = mem; + pool->curr = pool->free_arr; + pool->last = pool->free_arr + MLX5_FLOW_MIN_ID_POOL_SIZE; + pool->base_index = 0; + return pool; +error: + rte_free(pool); + return NULL; +} + +/** + * Release ID pool structure. + * + * @param[in] pool + * Pointer to flow id pool object to free. + */ +void +mlx5_flow_id_pool_release(struct mlx5_flow_id_pool *pool) +{ + rte_free(pool->free_arr); + rte_free(pool); +} + +/** + * Generate ID. + * + * @param[in] pool + * Pointer to flow id pool. + * @param[out] id + * The generated ID. + * + * @return + * 0 on success, error value otherwise. + */ +uint32_t +mlx5_flow_id_get(struct mlx5_flow_id_pool *pool, uint32_t *id) +{ + if (pool->curr == pool->free_arr) { + if (pool->base_index == UINT32_MAX) { + rte_errno = ENOMEM; + DRV_LOG(ERR, "no free id"); + return -rte_errno; + } + *id = ++pool->base_index; + return 0; + } + *id = *(--pool->curr); + return 0; +} + +/** + * Release ID. + * + * @param[in] pool + * Pointer to flow id pool. + * @param[out] id + * The generated ID. + * + * @return + * 0 on success, error value otherwise. + */ +uint32_t +mlx5_flow_id_release(struct mlx5_flow_id_pool *pool, uint32_t id) +{ + uint32_t size; + uint32_t size2; + void *mem; + + if (pool->curr == pool->last) { + size = pool->curr - pool->free_arr; + size2 = size * MLX5_ID_GENERATION_ARRAY_FACTOR; + assert(size2 > size); + mem = rte_malloc("", size2 * sizeof(uint32_t), 0); + if (!mem) { + DRV_LOG(ERR, "can't allocate mem for id pool"); + rte_errno = ENOMEM; + return -rte_errno; + } + memcpy(mem, pool->free_arr, size * sizeof(uint32_t)); + rte_free(pool->free_arr); + pool->free_arr = mem; + pool->curr = pool->free_arr + size; + pool->last = pool->free_arr + size2; + } + *pool->curr = id; + pool->curr++; + return 0; +} + /** * Initialize the counters management structure. * @@ -329,7 +447,7 @@ struct mlx5_dev_spawn_data { struct mlx5_devx_tis_attr tis_attr = { 0 }; #endif - assert(spawn); +assert(spawn); /* Secondary process should not create the shared context. */ assert(rte_eal_process_type() == RTE_PROC_PRIMARY); pthread_mutex_lock(&mlx5_ibv_list_mutex); diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 0148c1b..1b14fb7 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -495,8 +495,22 @@ struct mlx5_flow_driver_ops { #define MLX5_CNT_CONTAINER_UNUSED(sh, batch, thread) (&(sh)->cmng.ccont \ [(~((sh)->cmng.mhi[batch] >> (thread)) & 0x1) * 2 + (batch)]) +/* ID generation structure. */ +struct mlx5_flow_id_pool { + uint32_t *free_arr; /**< Pointer to the a array of free values. */ + uint32_t base_index; + /**< The next index that can be used without any free elements. */ + uint32_t *curr; /**< Pointer to the index to pop. */ + uint32_t *last; /**< Pointer to the last element in the empty arrray. */ +}; + /* mlx5_flow.c */ +struct mlx5_flow_id_pool *mlx5_flow_id_pool_alloc(void); +void mlx5_flow_id_pool_release(struct mlx5_flow_id_pool *pool); +uint32_t mlx5_flow_id_get(struct mlx5_flow_id_pool *pool, uint32_t *id); +uint32_t mlx5_flow_id_release(struct mlx5_flow_id_pool *pool, + uint32_t id); int mlx5_flow_group_to_table(const struct rte_flow_attr *attributes, bool external, uint32_t group, uint32_t *table, struct rte_flow_error *error); From patchwork Thu Oct 17 15:32:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61422 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4DBC91EA3E; Thu, 17 Oct 2019 17:33:14 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 8C2C81EA0D for ; Thu, 17 Oct 2019 17:32:54 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:54 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfU026371; Thu, 17 Oct 2019 18:32:53 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:15 +0000 Message-Id: <1571326337-42692-14-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 13/15] net/mlx5: add default flows for hairpin X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When using hairpin all traffic from TX hairpin queues should jump to dedecated table where matching can be done using regesters. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.h | 2 ++ drivers/net/mlx5/mlx5_flow.c | 60 +++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_flow.h | 9 ++++++ drivers/net/mlx5/mlx5_flow_dv.c | 63 +++++++++++++++++++++++++++++++++++++++-- drivers/net/mlx5/mlx5_trigger.c | 18 ++++++++++++ 5 files changed, 150 insertions(+), 2 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 391ae2c..8e86bcf 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -556,6 +556,7 @@ struct mlx5_flow_tbl_resource { }; #define MLX5_MAX_TABLES UINT16_MAX +#define MLX5_HAIRPIN_TX_TABLE (UINT16_MAX - 1) #define MLX5_MAX_TABLES_FDB UINT16_MAX #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */ @@ -876,6 +877,7 @@ int mlx5_dev_filter_ctrl(struct rte_eth_dev *dev, int mlx5_flow_start(struct rte_eth_dev *dev, struct mlx5_flows *list); void mlx5_flow_stop(struct rte_eth_dev *dev, struct mlx5_flows *list); int mlx5_flow_verify(struct rte_eth_dev *dev); +int mlx5_ctrl_flow_source_queue(struct rte_eth_dev *dev, uint32_t queue); int mlx5_ctrl_flow_vlan(struct rte_eth_dev *dev, struct rte_flow_item_eth *eth_spec, struct rte_flow_item_eth *eth_mask, diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index b4bcd1a..b6dc105 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -2731,6 +2731,66 @@ struct rte_flow * } /** + * Enable default hairpin egress flow. + * + * @param dev + * Pointer to Ethernet device. + * @param queue + * The queue index. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_ctrl_flow_source_queue(struct rte_eth_dev *dev, + uint32_t queue) +{ + struct mlx5_priv *priv = dev->data->dev_private; + const struct rte_flow_attr attr = { + .egress = 1, + .priority = 0, + }; + struct mlx5_rte_flow_item_tx_queue queue_spec = { + .queue = queue, + }; + struct mlx5_rte_flow_item_tx_queue queue_mask = { + .queue = UINT32_MAX, + }; + struct rte_flow_item items[] = { + { + .type = MLX5_RTE_FLOW_ITEM_TYPE_TX_QUEUE, + .spec = &queue_spec, + .last = NULL, + .mask = &queue_mask, + }, + { + .type = RTE_FLOW_ITEM_TYPE_END, + }, + }; + struct rte_flow_action_jump jump = { + .group = MLX5_HAIRPIN_TX_TABLE, + }; + struct rte_flow_action actions[2]; + struct rte_flow *flow; + struct rte_flow_error error; + + actions[0].type = RTE_FLOW_ACTION_TYPE_JUMP; + actions[0].conf = &jump; + actions[1].type = RTE_FLOW_ACTION_TYPE_END; + flow = flow_list_create(dev, &priv->ctrl_flows, + &attr, items, actions, false, &error); + if (!flow) { + DRV_LOG(DEBUG, + "Failed to create ctrl flow: rte_errno(%d)," + " type(%d), message(%s)\n", + rte_errno, error.type, + error.message ? error.message : " (no stated reason)"); + return -rte_errno; + } + return 0; +} + +/** * Enable a control flow configured from the control plane. * * @param dev diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 1b14fb7..bb67380 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -44,6 +44,7 @@ enum modify_reg { enum mlx5_rte_flow_item_type { MLX5_RTE_FLOW_ITEM_TYPE_END = INT_MIN, MLX5_RTE_FLOW_ITEM_TYPE_TAG, + MLX5_RTE_FLOW_ITEM_TYPE_TX_QUEUE, }; /* Private rte flow actions. */ @@ -64,6 +65,11 @@ struct mlx5_rte_flow_action_set_tag { rte_be32_t data; }; +/* Matches on source queue. */ +struct mlx5_rte_flow_item_tx_queue { + uint32_t queue; +}; + /* Pattern outer Layer bits. */ #define MLX5_FLOW_LAYER_OUTER_L2 (1u << 0) #define MLX5_FLOW_LAYER_OUTER_L3_IPV4 (1u << 1) @@ -102,6 +108,9 @@ struct mlx5_rte_flow_action_set_tag { #define MLX5_FLOW_LAYER_IPV6_ENCAP (1u << 23) #define MLX5_FLOW_LAYER_NVGRE (1u << 24) +/* Queue items. */ +#define MLX5_FLOW_ITEM_TX_QUEUE (1u << 25) + /* Outer Masks. */ #define MLX5_FLOW_LAYER_OUTER_L3 \ (MLX5_FLOW_LAYER_OUTER_L3_IPV4 | MLX5_FLOW_LAYER_OUTER_L3_IPV6) diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index dde6673..c7a3f6b 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -3357,7 +3357,9 @@ struct field_modify_info modify_tcp[] = { return ret; for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) { int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); - switch (items->type) { + int type = items->type; + + switch (type) { case RTE_FLOW_ITEM_TYPE_VOID: break; case RTE_FLOW_ITEM_TYPE_PORT_ID: @@ -3518,6 +3520,9 @@ struct field_modify_info modify_tcp[] = { return ret; last_item = MLX5_FLOW_LAYER_ICMP6; break; + case MLX5_RTE_FLOW_ITEM_TYPE_TAG: + case MLX5_RTE_FLOW_ITEM_TYPE_TX_QUEUE: + break; default: return rte_flow_error_set(error, ENOTSUP, RTE_FLOW_ERROR_TYPE_ITEM, @@ -3526,11 +3531,12 @@ struct field_modify_info modify_tcp[] = { item_flags |= last_item; } for (; actions->type != RTE_FLOW_ACTION_TYPE_END; actions++) { + int type = actions->type; if (actions_n == MLX5_DV_MAX_NUMBER_OF_ACTIONS) return rte_flow_error_set(error, ENOTSUP, RTE_FLOW_ERROR_TYPE_ACTION, actions, "too many actions"); - switch (actions->type) { + switch (type) { case RTE_FLOW_ACTION_TYPE_VOID: break; case RTE_FLOW_ACTION_TYPE_PORT_ID: @@ -3796,6 +3802,8 @@ struct field_modify_info modify_tcp[] = { MLX5_FLOW_ACTION_INC_TCP_ACK : MLX5_FLOW_ACTION_DEC_TCP_ACK; break; + case MLX5_RTE_FLOW_ACTION_TYPE_TAG: + break; default: return rte_flow_error_set(error, ENOTSUP, RTE_FLOW_ERROR_TYPE_ACTION, @@ -5291,6 +5299,51 @@ struct field_modify_info modify_tcp[] = { } /** + * Add Tx queue matcher + * + * @param[in] dev + * Pointer to the dev struct. + * @param[in, out] matcher + * Flow matcher. + * @param[in, out] key + * Flow matcher value. + * @param[in] item + * Flow pattern to translate. + * @param[in] inner + * Item is inner pattern. + */ +static void +flow_dv_translate_item_tx_queue(struct rte_eth_dev *dev, + void *matcher, void *key, + const struct rte_flow_item *item) +{ + const struct mlx5_rte_flow_item_tx_queue *queue_m; + const struct mlx5_rte_flow_item_tx_queue *queue_v; + void *misc_m = + MLX5_ADDR_OF(fte_match_param, matcher, misc_parameters); + void *misc_v = + MLX5_ADDR_OF(fte_match_param, key, misc_parameters); + struct mlx5_txq_ctrl *txq; + uint32_t queue; + + + queue_m = (const void *)item->mask; + if (!queue_m) + return; + queue_v = (const void *)item->spec; + if (!queue_v) + return; + txq = mlx5_txq_get(dev, queue_v->queue); + if (!txq) + return; + queue = txq->obj->sq->id; + MLX5_SET(fte_match_set_misc, misc_m, source_sqn, queue_m->queue); + MLX5_SET(fte_match_set_misc, misc_v, source_sqn, + queue & queue_m->queue); + mlx5_txq_release(dev, queue_v->queue); +} + +/** * Fill the flow with DV spec. * * @param[in] dev @@ -5866,6 +5919,12 @@ struct field_modify_info modify_tcp[] = { items); last_item = MLX5_FLOW_ITEM_TAG; break; + case MLX5_RTE_FLOW_ITEM_TYPE_TX_QUEUE: + flow_dv_translate_item_tx_queue(dev, match_mask, + match_value, + items); + last_item = MLX5_FLOW_ITEM_TX_QUEUE; + break; default: break; } diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index f66b6ee..cafab25 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -402,6 +402,24 @@ unsigned int j; int ret; + /* + * Hairpin txq default flow should be created no matter if it is + * isolation mode. Or else all the packets to be sent will be sent + * out directly without the TX flow actions, e.g. encapsulation. + */ + for (i = 0; i != priv->txqs_n; ++i) { + struct mlx5_txq_ctrl *txq_ctrl = mlx5_txq_get(dev, i); + if (!txq_ctrl) + continue; + if (txq_ctrl->type == MLX5_TXQ_TYPE_HAIRPIN) { + ret = mlx5_ctrl_flow_source_queue(dev, i); + if (ret) { + mlx5_txq_release(dev, i); + goto error; + } + } + mlx5_txq_release(dev, i); + } if (priv->config.dv_esw_en && !priv->config.vf) if (!mlx5_flow_create_esw_table_zero_flow(dev)) goto error; From patchwork Thu Oct 17 15:32:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61425 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B22BF1EA76; Thu, 17 Oct 2019 17:33:23 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 9F6821EA2A for ; Thu, 17 Oct 2019 17:32:59 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:56 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfV026371; Thu, 17 Oct 2019 18:32:56 +0300 From: Ori Kam To: Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:16 +0000 Message-Id: <1571326337-42692-15-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 14/15] net/mlx5: split hairpin flows X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Since the encap action is not supported in RX, we need to split the hairpin flow into RX and TX. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 10 ++ drivers/net/mlx5/mlx5.h | 10 ++ drivers/net/mlx5/mlx5_flow.c | 281 +++++++++++++++++++++++++++++++++++-- drivers/net/mlx5/mlx5_flow.h | 14 +- drivers/net/mlx5/mlx5_flow_dv.c | 10 +- drivers/net/mlx5/mlx5_flow_verbs.c | 11 +- drivers/net/mlx5/mlx5_rxq.c | 26 ++++ drivers/net/mlx5/mlx5_rxtx.h | 2 + 8 files changed, 334 insertions(+), 30 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 0c3239c..bd9c203 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -530,6 +530,12 @@ struct mlx5_flow_id_pool * goto error; } } + sh->flow_id_pool = mlx5_flow_id_pool_alloc(); + if (!sh->flow_id_pool) { + DRV_LOG(ERR, "can't create flow id pool"); + err = ENOMEM; + goto error; + } #endif /* HAVE_IBV_FLOW_DV_SUPPORT */ /* * Once the device is added to the list of memory event @@ -569,6 +575,8 @@ struct mlx5_flow_id_pool * claim_zero(mlx5_glue->dealloc_pd(sh->pd)); if (sh->ctx) claim_zero(mlx5_glue->close_device(sh->ctx)); + if (sh->flow_id_pool) + mlx5_flow_id_pool_release(sh->flow_id_pool); rte_free(sh); assert(err > 0); rte_errno = err; @@ -631,6 +639,8 @@ struct mlx5_flow_id_pool * claim_zero(mlx5_devx_cmd_destroy(sh->td)); if (sh->ctx) claim_zero(mlx5_glue->close_device(sh->ctx)); + if (sh->flow_id_pool) + mlx5_flow_id_pool_release(sh->flow_id_pool); rte_free(sh); exit: pthread_mutex_unlock(&mlx5_ibv_list_mutex); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 8e86bcf..5f40a39 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -574,6 +574,15 @@ struct mlx5_devx_dbr_page { uint64_t dbr_bitmap[MLX5_DBR_BITMAP_SIZE]; }; +/* ID generation structure. */ +struct mlx5_flow_id_pool { + uint32_t *free_arr; /**< Pointer to the a array of free values. */ + uint32_t base_index; + /**< The next index that can be used without any free elements. */ + uint32_t *curr; /**< Pointer to the index to pop. */ + uint32_t *last; /**< Pointer to the last element in the empty arrray. */ +}; + /* * Shared Infiniband device context for Master/Representors * which belong to same IB device with multiple IB ports. @@ -632,6 +641,7 @@ struct mlx5_ibv_shared { struct mlx5dv_devx_cmd_comp *devx_comp; /* DEVX async comp obj. */ struct mlx5_devx_obj *tis; /* TIS object. */ struct mlx5_devx_obj *td; /* Transport domain. */ + struct mlx5_flow_id_pool *flow_id_pool; /* Flow ID pool. */ struct mlx5_ibv_shared_port port[]; /* per device port data array. */ }; diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index b6dc105..bb13857 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -606,7 +606,7 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, { struct mlx5_priv *priv = dev->data->dev_private; struct rte_flow *flow = dev_flow->flow; - const int mark = !!(flow->actions & + const int mark = !!(dev_flow->actions & (MLX5_FLOW_ACTION_FLAG | MLX5_FLOW_ACTION_MARK)); const int tunnel = !!(dev_flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int i; @@ -669,7 +669,7 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, { struct mlx5_priv *priv = dev->data->dev_private; struct rte_flow *flow = dev_flow->flow; - const int mark = !!(flow->actions & + const int mark = !!(dev_flow->actions & (MLX5_FLOW_ACTION_FLAG | MLX5_FLOW_ACTION_MARK)); const int tunnel = !!(dev_flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int i; @@ -2438,6 +2438,210 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, } /** + * Check if the flow should be splited due to hairpin. + * The reason for the split is that in current HW we can't + * support encap on Rx, so if a flow have encap we move it + * to Tx. + * + * @param dev + * Pointer to Ethernet device. + * @param[in] attr + * Flow rule attributes. + * @param[in] actions + * Associated actions (list terminated by the END action). + * + * @return + * > 0 the number of actions and the flow should be split, + * 0 when no split required. + */ +static int +flow_check_hairpin_split(struct rte_eth_dev *dev, + const struct rte_flow_attr *attr, + const struct rte_flow_action actions[]) +{ + int queue_action = 0; + int action_n = 0; + int encap = 0; + const struct rte_flow_action_queue *queue; + const struct rte_flow_action_rss *rss; + const struct rte_flow_action_raw_encap *raw_encap; + + if (!attr->ingress) + return 0; + for (; actions->type != RTE_FLOW_ACTION_TYPE_END; actions++) { + switch (actions->type) { + case RTE_FLOW_ACTION_TYPE_QUEUE: + queue = actions->conf; + if (mlx5_rxq_get_type(dev, queue->index) != + MLX5_RXQ_TYPE_HAIRPIN) + return 0; + queue_action = 1; + action_n++; + break; + case RTE_FLOW_ACTION_TYPE_RSS: + rss = actions->conf; + if (mlx5_rxq_get_type(dev, rss->queue[0]) != + MLX5_RXQ_TYPE_HAIRPIN) + return 0; + queue_action = 1; + action_n++; + break; + case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP: + case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP: + encap = 1; + action_n++; + break; + case RTE_FLOW_ACTION_TYPE_RAW_ENCAP: + raw_encap = actions->conf; + if (raw_encap->size > + (sizeof(struct rte_flow_item_eth) + + sizeof(struct rte_flow_item_ipv4))) + encap = 1; + action_n++; + break; + default: + action_n++; + break; + } + } + if (encap == 1 && queue_action) + return action_n; + return 0; +} + +#define MLX5_MAX_SPLIT_ACTIONS 24 +#define MLX5_MAX_SPLIT_ITEMS 24 + +/** + * Split the hairpin flow. + * Since HW can't support encap on Rx we move the encap to Tx. + * If the count action is after the encap then we also + * move the count action. in this case the count will also measure + * the outer bytes. + * + * @param dev + * Pointer to Ethernet device. + * @param[in] actions + * Associated actions (list terminated by the END action). + * @param[out] actions_rx + * Rx flow actions. + * @param[out] actions_tx + * Tx flow actions.. + * @param[out] pattern_tx + * The pattern items for the Tx flow. + * @param[out] flow_id + * The flow ID connected to this flow. + * + * @return + * 0 on success. + */ +static int +flow_hairpin_split(struct rte_eth_dev *dev, + const struct rte_flow_action actions[], + struct rte_flow_action actions_rx[], + struct rte_flow_action actions_tx[], + struct rte_flow_item pattern_tx[], + uint32_t *flow_id) +{ + struct mlx5_priv *priv = dev->data->dev_private; + const struct rte_flow_action_raw_encap *raw_encap; + const struct rte_flow_action_raw_decap *raw_decap; + struct mlx5_rte_flow_action_set_tag *set_tag; + struct rte_flow_action *tag_action; + struct mlx5_rte_flow_item_tag *tag_item; + struct rte_flow_item *item; + char *addr; + struct rte_flow_error error; + int encap = 0; + + mlx5_flow_id_get(priv->sh->flow_id_pool, flow_id); + for (; actions->type != RTE_FLOW_ACTION_TYPE_END; actions++) { + switch (actions->type) { + case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP: + case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP: + rte_memcpy(actions_tx, actions, + sizeof(struct rte_flow_action)); + actions_tx++; + break; + case RTE_FLOW_ACTION_TYPE_COUNT: + if (encap) { + rte_memcpy(actions_tx, actions, + sizeof(struct rte_flow_action)); + actions_tx++; + } else { + rte_memcpy(actions_rx, actions, + sizeof(struct rte_flow_action)); + actions_rx++; + } + break; + case RTE_FLOW_ACTION_TYPE_RAW_ENCAP: + raw_encap = actions->conf; + if (raw_encap->size > + (sizeof(struct rte_flow_item_eth) + + sizeof(struct rte_flow_item_ipv4))) { + memcpy(actions_tx, actions, + sizeof(struct rte_flow_action)); + actions_tx++; + encap = 1; + } else { + rte_memcpy(actions_rx, actions, + sizeof(struct rte_flow_action)); + actions_rx++; + } + break; + case RTE_FLOW_ACTION_TYPE_RAW_DECAP: + raw_decap = actions->conf; + if (raw_decap->size < + (sizeof(struct rte_flow_item_eth) + + sizeof(struct rte_flow_item_ipv4))) { + memcpy(actions_tx, actions, + sizeof(struct rte_flow_action)); + actions_tx++; + } else { + rte_memcpy(actions_rx, actions, + sizeof(struct rte_flow_action)); + actions_rx++; + } + break; + default: + rte_memcpy(actions_rx, actions, + sizeof(struct rte_flow_action)); + actions_rx++; + break; + } + } + /* Add set meta action and end action for the Rx flow. */ + tag_action = actions_rx; + tag_action->type = MLX5_RTE_FLOW_ACTION_TYPE_TAG; + actions_rx++; + rte_memcpy(actions_rx, actions, sizeof(struct rte_flow_action)); + actions_rx++; + set_tag = (void *)actions_rx; + set_tag->id = flow_get_reg_id(dev, MLX5_HAIRPIN_RX, 0, &error); + set_tag->data = rte_cpu_to_be_32(*flow_id); + tag_action->conf = set_tag; + /* Create Tx item list. */ + rte_memcpy(actions_tx, actions, sizeof(struct rte_flow_action)); + addr = (void *)&pattern_tx[2]; + item = pattern_tx; + item->type = MLX5_RTE_FLOW_ITEM_TYPE_TAG; + tag_item = (void *)addr; + tag_item->data = rte_cpu_to_be_32(*flow_id); + tag_item->id = flow_get_reg_id(dev, MLX5_HAIRPIN_TX, 0, &error); + item->spec = tag_item; + addr += sizeof(struct mlx5_rte_flow_item_tag); + tag_item = (void *)addr; + tag_item->data = UINT32_MAX; + tag_item->id = UINT16_MAX; + item->mask = tag_item; + addr += sizeof(struct mlx5_rte_flow_item_tag); + item->last = NULL; + item++; + item->type = RTE_FLOW_ITEM_TYPE_END; + return 0; +} + +/** * Create a flow and add it to @p list. * * @param dev @@ -2465,6 +2669,7 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, const struct rte_flow_action actions[], bool external, struct rte_flow_error *error) { + struct mlx5_priv *priv = dev->data->dev_private; struct rte_flow *flow = NULL; struct mlx5_flow *dev_flow; const struct rte_flow_action_rss *rss; @@ -2472,16 +2677,44 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, struct rte_flow_expand_rss buf; uint8_t buffer[2048]; } expand_buffer; + union { + struct rte_flow_action actions[MLX5_MAX_SPLIT_ACTIONS]; + uint8_t buffer[2048]; + } actions_rx; + union { + struct rte_flow_action actions[MLX5_MAX_SPLIT_ACTIONS]; + uint8_t buffer[2048]; + } actions_hairpin_tx; + union { + struct rte_flow_item items[MLX5_MAX_SPLIT_ITEMS]; + uint8_t buffer[2048]; + } items_tx; struct rte_flow_expand_rss *buf = &expand_buffer.buf; + const struct rte_flow_action *p_actions_rx = actions; int ret; uint32_t i; uint32_t flow_size; + int hairpin_flow = 0; + uint32_t hairpin_id = 0; + struct rte_flow_attr attr_tx = { .priority = 0 }; - ret = flow_drv_validate(dev, attr, items, actions, external, error); + hairpin_flow = flow_check_hairpin_split(dev, attr, actions); + if (hairpin_flow > 0) { + if (hairpin_flow > MLX5_MAX_SPLIT_ACTIONS) { + rte_errno = EINVAL; + return NULL; + } + flow_hairpin_split(dev, actions, actions_rx.actions, + actions_hairpin_tx.actions, items_tx.items, + &hairpin_id); + p_actions_rx = actions_rx.actions; + } + ret = flow_drv_validate(dev, attr, items, p_actions_rx, external, + error); if (ret < 0) - return NULL; + goto error_before_flow; flow_size = sizeof(struct rte_flow); - rss = flow_get_rss_action(actions); + rss = flow_get_rss_action(p_actions_rx); if (rss) flow_size += RTE_ALIGN_CEIL(rss->queue_num * sizeof(uint16_t), sizeof(void *)); @@ -2490,11 +2723,13 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, flow = rte_calloc(__func__, 1, flow_size, 0); if (!flow) { rte_errno = ENOMEM; - return NULL; + goto error_before_flow; } flow->drv_type = flow_get_drv_type(dev, attr); flow->ingress = attr->ingress; flow->transfer = attr->transfer; + if (hairpin_id != 0) + flow->hairpin_flow_id = hairpin_id; assert(flow->drv_type > MLX5_FLOW_TYPE_MIN && flow->drv_type < MLX5_FLOW_TYPE_MAX); flow->queue = (void *)(flow + 1); @@ -2515,7 +2750,7 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, } for (i = 0; i < buf->entries; ++i) { dev_flow = flow_drv_prepare(flow, attr, buf->entry[i].pattern, - actions, error); + p_actions_rx, error); if (!dev_flow) goto error; dev_flow->flow = flow; @@ -2523,7 +2758,24 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, LIST_INSERT_HEAD(&flow->dev_flows, dev_flow, next); ret = flow_drv_translate(dev, dev_flow, attr, buf->entry[i].pattern, - actions, error); + p_actions_rx, error); + if (ret < 0) + goto error; + } + /* Create the tx flow. */ + if (hairpin_flow) { + attr_tx.group = MLX5_HAIRPIN_TX_TABLE; + attr_tx.ingress = 0; + attr_tx.egress = 1; + dev_flow = flow_drv_prepare(flow, &attr_tx, items_tx.items, + actions_hairpin_tx.actions, error); + if (!dev_flow) + goto error; + dev_flow->flow = flow; + LIST_INSERT_HEAD(&flow->dev_flows, dev_flow, next); + ret = flow_drv_translate(dev, dev_flow, &attr_tx, + items_tx.items, + actions_hairpin_tx.actions, error); if (ret < 0) goto error; } @@ -2535,8 +2787,16 @@ uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t priority, TAILQ_INSERT_TAIL(list, flow, next); flow_rxq_flags_set(dev, flow); return flow; +error_before_flow: + if (hairpin_id) + mlx5_flow_id_release(priv->sh->flow_id_pool, + hairpin_id); + return NULL; error: ret = rte_errno; /* Save rte_errno before cleanup. */ + if (flow->hairpin_flow_id) + mlx5_flow_id_release(priv->sh->flow_id_pool, + flow->hairpin_flow_id); assert(flow); flow_drv_destroy(dev, flow); rte_free(flow); @@ -2626,12 +2886,17 @@ struct rte_flow * flow_list_destroy(struct rte_eth_dev *dev, struct mlx5_flows *list, struct rte_flow *flow) { + struct mlx5_priv *priv = dev->data->dev_private; + /* * Update RX queue flags only if port is started, otherwise it is * already clean. */ if (dev->data->dev_started) flow_rxq_flags_trim(dev, flow); + if (flow->hairpin_flow_id) + mlx5_flow_id_release(priv->sh->flow_id_pool, + flow->hairpin_flow_id); flow_drv_destroy(dev, flow); TAILQ_REMOVE(list, flow, next); rte_free(flow->fdir); diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index bb67380..90a289e 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -434,6 +434,8 @@ struct mlx5_flow { struct rte_flow *flow; /**< Pointer to the main flow. */ uint64_t layers; /**< Bit-fields of present layers, see MLX5_FLOW_LAYER_*. */ + uint64_t actions; + /**< Bit-fields of detected actions, see MLX5_FLOW_ACTION_*. */ union { #ifdef HAVE_IBV_FLOW_DV_SUPPORT struct mlx5_flow_dv dv; @@ -455,12 +457,11 @@ struct rte_flow { uint16_t (*queue)[]; /**< Destination queues to redirect traffic to. */ LIST_HEAD(dev_flows, mlx5_flow) dev_flows; /**< Device flows that are part of the flow. */ - uint64_t actions; - /**< Bit-fields of detected actions, see MLX5_FLOW_ACTION_*. */ struct mlx5_fdir *fdir; /**< Pointer to associated FDIR if any. */ uint8_t ingress; /**< 1 if the flow is ingress. */ uint32_t group; /**< The group index. */ uint8_t transfer; /**< 1 if the flow is E-Switch flow. */ + uint32_t hairpin_flow_id; /**< The flow id used for hairpin. */ }; typedef int (*mlx5_flow_validate_t)(struct rte_eth_dev *dev, @@ -504,15 +505,6 @@ struct mlx5_flow_driver_ops { #define MLX5_CNT_CONTAINER_UNUSED(sh, batch, thread) (&(sh)->cmng.ccont \ [(~((sh)->cmng.mhi[batch] >> (thread)) & 0x1) * 2 + (batch)]) -/* ID generation structure. */ -struct mlx5_flow_id_pool { - uint32_t *free_arr; /**< Pointer to the a array of free values. */ - uint32_t base_index; - /**< The next index that can be used without any free elements. */ - uint32_t *curr; /**< Pointer to the index to pop. */ - uint32_t *last; /**< Pointer to the last element in the empty arrray. */ -}; - /* mlx5_flow.c */ struct mlx5_flow_id_pool *mlx5_flow_id_pool_alloc(void); diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index c7a3f6b..367e632 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -5763,7 +5763,7 @@ struct field_modify_info modify_tcp[] = { modify_action_position = actions_n++; } dev_flow->dv.actions_n = actions_n; - flow->actions = action_flags; + dev_flow->actions = action_flags; for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) { int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); int item_type = items->type; @@ -5985,7 +5985,7 @@ struct field_modify_info modify_tcp[] = { LIST_FOREACH(dev_flow, &flow->dev_flows, next) { dv = &dev_flow->dv; n = dv->actions_n; - if (flow->actions & MLX5_FLOW_ACTION_DROP) { + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) { if (flow->transfer) { dv->actions[n++] = priv->sh->esw_drop_action; } else { @@ -6000,7 +6000,7 @@ struct field_modify_info modify_tcp[] = { } dv->actions[n++] = dv->hrxq->action; } - } else if (flow->actions & + } else if (dev_flow->actions & (MLX5_FLOW_ACTION_QUEUE | MLX5_FLOW_ACTION_RSS)) { struct mlx5_hrxq *hrxq; @@ -6056,7 +6056,7 @@ struct field_modify_info modify_tcp[] = { LIST_FOREACH(dev_flow, &flow->dev_flows, next) { struct mlx5_flow_dv *dv = &dev_flow->dv; if (dv->hrxq) { - if (flow->actions & MLX5_FLOW_ACTION_DROP) + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, dv->hrxq); @@ -6290,7 +6290,7 @@ struct field_modify_info modify_tcp[] = { dv->flow = NULL; } if (dv->hrxq) { - if (flow->actions & MLX5_FLOW_ACTION_DROP) + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, dv->hrxq); diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_flow_verbs.c index 23110f2..fd27f6c 100644 --- a/drivers/net/mlx5/mlx5_flow_verbs.c +++ b/drivers/net/mlx5/mlx5_flow_verbs.c @@ -191,7 +191,7 @@ { #if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42) || \ defined(HAVE_IBV_DEVICE_COUNTERS_SET_V45) - if (flow->actions & MLX5_FLOW_ACTION_COUNT) { + if (flow->counter->cs) { struct rte_flow_query_count *qc = data; uint64_t counters[2] = {0, 0}; #if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42) @@ -1410,7 +1410,6 @@ const struct rte_flow_action actions[], struct rte_flow_error *error) { - struct rte_flow *flow = dev_flow->flow; uint64_t item_flags = 0; uint64_t action_flags = 0; uint64_t priority = attr->priority; @@ -1460,7 +1459,7 @@ "action not supported"); } } - flow->actions = action_flags; + dev_flow->actions = action_flags; for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) { int tunnel = !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); @@ -1592,7 +1591,7 @@ verbs->flow = NULL; } if (verbs->hrxq) { - if (flow->actions & MLX5_FLOW_ACTION_DROP) + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, verbs->hrxq); @@ -1656,7 +1655,7 @@ LIST_FOREACH(dev_flow, &flow->dev_flows, next) { verbs = &dev_flow->verbs; - if (flow->actions & MLX5_FLOW_ACTION_DROP) { + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) { verbs->hrxq = mlx5_hrxq_drop_new(dev); if (!verbs->hrxq) { rte_flow_error_set @@ -1717,7 +1716,7 @@ LIST_FOREACH(dev_flow, &flow->dev_flows, next) { verbs = &dev_flow->verbs; if (verbs->hrxq) { - if (flow->actions & MLX5_FLOW_ACTION_DROP) + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, verbs->hrxq); diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index a8ff8b2..c39118a 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -2097,6 +2097,32 @@ struct mlx5_rxq_ctrl * } /** + * Get a Rx queue type. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Rx queue index. + * + * @return + * The Rx queue type. + */ +enum mlx5_rxq_type +mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_ctrl *rxq_ctrl = NULL; + + if ((*priv->rxqs)[idx]) { + rxq_ctrl = container_of((*priv->rxqs)[idx], + struct mlx5_rxq_ctrl, + rxq); + return rxq_ctrl->type; + } + return MLX5_RXQ_TYPE_UNDEFINED; +} + +/** * Create an indirection table. * * @param dev diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 271b648..d4ba25f 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -166,6 +166,7 @@ enum mlx5_rxq_obj_type { enum mlx5_rxq_type { MLX5_RXQ_TYPE_STANDARD, /* Standard Rx queue. */ MLX5_RXQ_TYPE_HAIRPIN, /* Hairpin Rx queue. */ + MLX5_RXQ_TYPE_UNDEFINED, }; /* Verbs/DevX Rx queue elements. */ @@ -406,6 +407,7 @@ struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev, const uint16_t *queues, uint32_t queues_n); int mlx5_hrxq_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hxrq); int mlx5_hrxq_verify(struct rte_eth_dev *dev); +enum mlx5_rxq_type mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx); struct mlx5_hrxq *mlx5_hrxq_drop_new(struct rte_eth_dev *dev); void mlx5_hrxq_drop_release(struct rte_eth_dev *dev); uint64_t mlx5_get_rx_port_offloads(void); From patchwork Thu Oct 17 15:32:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ori Kam X-Patchwork-Id: 61424 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EACB41EA69; Thu, 17 Oct 2019 17:33:20 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 9B3E11EA21 for ; Thu, 17 Oct 2019 17:32:59 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from orika@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Oct 2019 17:32:58 +0200 Received: from pegasus04.mtr.labs.mlnx. (pegasus04.mtr.labs.mlnx [10.210.16.126]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9HFWKfW026371; Thu, 17 Oct 2019 18:32:58 +0300 From: Ori Kam To: John McNamara , Marko Kovacevic Cc: dev@dpdk.org, orika@mellanox.com, jingjing.wu@intel.com, stephen@networkplumber.org Date: Thu, 17 Oct 2019 15:32:17 +0000 Message-Id: <1571326337-42692-16-git-send-email-orika@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571326337-42692-1-git-send-email-orika@mellanox.com> References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1571326337-42692-1-git-send-email-orika@mellanox.com> Subject: [dpdk-dev] [PATCH v4 15/15] doc: add hairpin feature X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit adds the hairpin feature to the release notes. Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko --- doc/guides/rel_notes/release_19_11.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst index cd4e350..2a27cb4 100644 --- a/doc/guides/rel_notes/release_19_11.rst +++ b/doc/guides/rel_notes/release_19_11.rst @@ -87,6 +87,11 @@ New Features Added support for the ``RTE_ETH_DEV_CLOSE_REMOVE`` flag. +* **Added hairpin queue.** + + On supported NICs, we can now setup haipin queue which will offload packets from the wire, + back to the wire. + Removed Items ------------- @@ -286,4 +291,5 @@ Tested Platforms * Added support for VLAN push flow offload command. * Added support for VLAN set PCP offload command. * Added support for VLAN set VID offload command. + * Added hairpin support.