From patchwork Thu May 25 09:45:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feifei Wang X-Patchwork-Id: 127385 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4A7C742B98; Thu, 25 May 2023 11:45:59 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 53F7142D31; Thu, 25 May 2023 11:45:53 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id 4963D42B7E for ; Thu, 25 May 2023 11:45:52 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D7F2B1042; Thu, 25 May 2023 02:46:36 -0700 (PDT) Received: from net-x86-dell-8268.shanghai.arm.com (net-x86-dell-8268.shanghai.arm.com [10.169.210.116]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 306C53F67D; Thu, 25 May 2023 02:45:48 -0700 (PDT) From: Feifei Wang To: Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko Cc: dev@dpdk.org, nd@arm.com, Feifei Wang , Honnappa Nagarahalli , Ruifeng Wang Subject: [PATCH v6 1/4] ethdev: add API for mbufs recycle mode Date: Thu, 25 May 2023 17:45:38 +0800 Message-Id: <20230525094541.331338-2-feifei.wang2@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230525094541.331338-1-feifei.wang2@arm.com> References: <20211224164613.32569-1-feifei.wang2@arm.com> <20230525094541.331338-1-feifei.wang2@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add 'rte_eth_recycle_rx_queue_info_get' and 'rte_eth_recycle_mbufs' APIs to recycle used mbufs from a transmit queue of an Ethernet device, and move these mbufs into a mbuf ring for a receive queue of an Ethernet device. This can bypass mempool 'put/get' operations hence saving CPU cycles. For each recycling mbufs, the rte_eth_recycle_mbufs() function performs the following operations: - Copy used *rte_mbuf* buffer pointers from Tx mbuf ring into Rx mbuf ring. - Replenish the Rx descriptors with the recycling *rte_mbuf* mbufs freed from the Tx mbuf ring. Suggested-by: Honnappa Nagarahalli Suggested-by: Ruifeng Wang Signed-off-by: Feifei Wang Reviewed-by: Ruifeng Wang Reviewed-by: Honnappa Nagarahalli Signed-off-by: Feifei Wang > Reviewed-by: Ruifeng Wang > Reviewed-by: Honnappa Nagarahalli > --- doc/guides/rel_notes/release_23_07.rst | 7 + lib/ethdev/ethdev_driver.h | 10 ++ lib/ethdev/ethdev_private.c | 2 + lib/ethdev/rte_ethdev.c | 31 +++++ lib/ethdev/rte_ethdev.h | 182 +++++++++++++++++++++++++ lib/ethdev/rte_ethdev_core.h | 15 +- lib/ethdev/version.map | 4 + 7 files changed, 249 insertions(+), 2 deletions(-) diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst index a9b1293689..f279036cb9 100644 --- a/doc/guides/rel_notes/release_23_07.rst +++ b/doc/guides/rel_notes/release_23_07.rst @@ -55,6 +55,13 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Add mbufs recycling support. ** + Added ``rte_eth_recycle_rx_queue_info_get`` and ``rte_eth_recycle_mbufs`` + APIs which allow the user to copy used mbufs from the Tx mbuf ring + into the Rx mbuf ring. This feature supports the case that the Rx Ethernet + device is different from the Tx Ethernet device with respective driver + callback functions in ``rte_eth_recycle_mbufs``. + Removed Items ------------- diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h index 2c9d615fb5..c6723d5277 100644 --- a/lib/ethdev/ethdev_driver.h +++ b/lib/ethdev/ethdev_driver.h @@ -59,6 +59,10 @@ struct rte_eth_dev { eth_rx_descriptor_status_t rx_descriptor_status; /** Check the status of a Tx descriptor */ eth_tx_descriptor_status_t tx_descriptor_status; + /** Pointer to PMD transmit mbufs reuse function */ + eth_recycle_tx_mbufs_reuse_t recycle_tx_mbufs_reuse; + /** Pointer to PMD receive descriptors refill function */ + eth_recycle_rx_descriptors_refill_t recycle_rx_descriptors_refill; /** * Device data that is shared between primary and secondary processes @@ -504,6 +508,10 @@ typedef void (*eth_rxq_info_get_t)(struct rte_eth_dev *dev, typedef void (*eth_txq_info_get_t)(struct rte_eth_dev *dev, uint16_t tx_queue_id, struct rte_eth_txq_info *qinfo); +typedef void (*eth_recycle_rxq_info_get_t)(struct rte_eth_dev *dev, + uint16_t rx_queue_id, + struct rte_eth_recycle_rxq_info *recycle_rxq_info); + typedef int (*eth_burst_mode_get_t)(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_burst_mode *mode); @@ -1247,6 +1255,8 @@ struct eth_dev_ops { eth_rxq_info_get_t rxq_info_get; /** Retrieve Tx queue information */ eth_txq_info_get_t txq_info_get; + /** Retrieve mbufs recycle Rx queue information */ + eth_recycle_rxq_info_get_t recycle_rxq_info_get; eth_burst_mode_get_t rx_burst_mode_get; /**< Get Rx burst mode */ eth_burst_mode_get_t tx_burst_mode_get; /**< Get Tx burst mode */ eth_fw_version_get_t fw_version_get; /**< Get firmware version */ diff --git a/lib/ethdev/ethdev_private.c b/lib/ethdev/ethdev_private.c index 14ec8c6ccf..f8ab64f195 100644 --- a/lib/ethdev/ethdev_private.c +++ b/lib/ethdev/ethdev_private.c @@ -277,6 +277,8 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo, fpo->rx_queue_count = dev->rx_queue_count; fpo->rx_descriptor_status = dev->rx_descriptor_status; fpo->tx_descriptor_status = dev->tx_descriptor_status; + fpo->recycle_tx_mbufs_reuse = dev->recycle_tx_mbufs_reuse; + fpo->recycle_rx_descriptors_refill = dev->recycle_rx_descriptors_refill; fpo->rxq.data = dev->data->rx_queues; fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs; diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c index 4d03255683..7c27dcfea4 100644 --- a/lib/ethdev/rte_ethdev.c +++ b/lib/ethdev/rte_ethdev.c @@ -5784,6 +5784,37 @@ rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id, return 0; } +int +rte_eth_recycle_rx_queue_info_get(uint16_t port_id, uint16_t queue_id, + struct rte_eth_recycle_rxq_info *recycle_rxq_info) +{ + struct rte_eth_dev *dev; + + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV); + dev = &rte_eth_devices[port_id]; + + if (queue_id >= dev->data->nb_rx_queues) { + RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u\n", queue_id); + return -EINVAL; + } + + if (dev->data->rx_queues == NULL || + dev->data->rx_queues[queue_id] == NULL) { + RTE_ETHDEV_LOG(ERR, + "Rx queue %"PRIu16" of device with port_id=%" + PRIu16" has not been setup\n", + queue_id, port_id); + return -EINVAL; + } + + if (*dev->dev_ops->recycle_rxq_info_get == NULL) + return -ENOTSUP; + + dev->dev_ops->recycle_rxq_info_get(dev, queue_id, recycle_rxq_info); + + return 0; +} + int rte_eth_rx_burst_mode_get(uint16_t port_id, uint16_t queue_id, struct rte_eth_burst_mode *mode) diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 99fe9e238b..7434aa2483 100644 --- a/lib/ethdev/rte_ethdev.h +++ b/lib/ethdev/rte_ethdev.h @@ -1820,6 +1820,30 @@ struct rte_eth_txq_info { uint8_t queue_state; /**< one of RTE_ETH_QUEUE_STATE_*. */ } __rte_cache_min_aligned; +/** + * @warning + * @b EXPERIMENTAL: this structure may change without prior notice. + * + * Ethernet device Rx queue information structure for recycling mbufs. + * Used to retrieve Rx queue information when Tx queue reusing mbufs and moving + * them into Rx mbuf ring. + */ +struct rte_eth_recycle_rxq_info { + struct rte_mbuf **mbuf_ring; /**< mbuf ring of Rx queue. */ + struct rte_mempool *mp; /**< mempool of Rx queue. */ + uint16_t *refill_head; /**< head of Rx queue refilling mbufs. */ + uint16_t *receive_tail; /**< tail of Rx queue receiving pkts. */ + uint16_t mbuf_ring_size; /**< configured number of mbuf ring size. */ + /** + * Requirement on mbuf refilling batch size of Rx mbuf ring. + * For some PMD drivers, the number of Rx mbuf ring refilling mbufs + * should be aligned with mbuf ring size, in order to simplify + * ring wrapping around. + * Value 0 means that PMD drivers have no requirement for this. + */ + uint16_t refill_requirement; +} __rte_cache_min_aligned; + /* Generic Burst mode flag definition, values can be ORed. */ /** @@ -4809,6 +4833,31 @@ int rte_eth_rx_queue_info_get(uint16_t port_id, uint16_t queue_id, int rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id, struct rte_eth_txq_info *qinfo); +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Retrieve information about given ports's Rx queue for recycling mbufs. + * + * @param port_id + * The port identifier of the Ethernet device. + * @param queue_id + * The Rx queue on the Ethernet devicefor which information + * will be retrieved. + * @param recycle_rxq_info + * A pointer to a structure of type *rte_eth_recycle_rxq_info* to be filled. + * + * @return + * - 0: Success + * - -ENODEV: If *port_id* is invalid. + * - -ENOTSUP: routine is not supported by the device PMD. + * - -EINVAL: The queue_id is out of range. + */ +__rte_experimental +int rte_eth_recycle_rx_queue_info_get(uint16_t port_id, + uint16_t queue_id, + struct rte_eth_recycle_rxq_info *recycle_rxq_info); + /** * Retrieve information about the Rx packet burst mode. * @@ -6483,6 +6532,139 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id, return rte_eth_tx_buffer_flush(port_id, queue_id, buffer); } +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Recycle used mbufs from a transmit queue of an Ethernet device, and move + * these mbufs into a mbuf ring for a receive queue of an Ethernet device. + * This can bypass mempool path to save CPU cycles. + * + * The rte_eth_recycle_mbufs() function loops, with rte_eth_rx_burst() and + * rte_eth_tx_burst() functions, freeing Tx used mbufs and replenishing Rx + * descriptors. The number of recycling mbufs depends on the request of Rx mbuf + * ring, with the constraint of enough used mbufs from Tx mbuf ring. + * + * For each recycling mbufs, the rte_eth_recycle_mbufs() function performs the + * following operations: + * + * - Copy used *rte_mbuf* buffer pointers from Tx mbuf ring into Rx mbuf ring. + * + * - Replenish the Rx descriptors with the recycling *rte_mbuf* mbufs freed + * from the Tx mbuf ring. + * + * This function spilts Rx and Tx path with different callback functions. The + * callback function recycle_tx_mbufs_reuse is for Tx driver. The callback + * function recycle_rx_descriptors_refill is for Rx driver. rte_eth_recycle_mbufs() + * can support the case that Rx Ethernet device is different from Tx Ethernet device. + * + * It is the responsibility of users to select the Rx/Tx queue pair to recycle + * mbufs. Before call this function, users must call rte_eth_recycle_rxq_info_get + * function to retrieve selected Rx queue information. + * @see rte_eth_recycle_rxq_info_get, struct rte_eth_recycle_rxq_info + * + * Currently, the rte_eth_recycle_mbufs() function can only support one-time pairing + * between the receive queue and transmit queue. Do not pair one receive queue with + * multiple transmit queues or pair one transmit queue with multiple receive queues, + * in order to avoid memory error rewriting. + * + * @param rx_port_id + * Port identifying the receive side. + * @param rx_queue_id + * The index of the receive queue identifying the receive side. + * The value must be in the range [0, nb_rx_queue - 1] previously supplied + * to rte_eth_dev_configure(). + * @param tx_port_id + * Port identifying the transmit side. + * @param tx_queue_id + * The index of the transmit queue identifying the transmit side. + * The value must be in the range [0, nb_tx_queue - 1] previously supplied + * to rte_eth_dev_configure(). + * @param recycle_rxq_info + * A pointer to a structure of type *rte_eth_recycle_rxq_info* which contains + * the information of the Rx queue mbuf ring. + * @return + * The number of recycling mbufs. + */ +__rte_experimental +static inline uint16_t +rte_eth_recycle_mbufs(uint16_t rx_port_id, uint16_t rx_queue_id, + uint16_t tx_port_id, uint16_t tx_queue_id, + struct rte_eth_recycle_rxq_info *recycle_rxq_info) +{ + struct rte_eth_fp_ops *p; + void *qd; + uint16_t nb_mbufs; + +#ifdef RTE_ETHDEV_DEBUG_TX + if (tx_port_id >= RTE_MAX_ETHPORTS || + tx_queue_id >= RTE_MAX_QUEUES_PER_PORT) { + RTE_ETHDEV_LOG(ERR, + "Invalid tx_port_id=%u or tx_queue_id=%u\n", + tx_port_id, tx_queue_id); + return 0; + } +#endif + + /* fetch pointer to queue data */ + p = &rte_eth_fp_ops[tx_port_id]; + qd = p->txq.data[tx_queue_id]; + +#ifdef RTE_ETHDEV_DEBUG_TX + RTE_ETH_VALID_PORTID_OR_ERR_RET(tx_port_id, 0); + + if (qd == NULL) { + RTE_ETHDEV_LOG(ERR, "Invalid Tx queue_id=%u for port_id=%u\n", + tx_queue_id, tx_port_id); + return 0; + } +#endif + if (p->recycle_tx_mbufs_reuse == NULL) + return 0; + + /* Copy used *rte_mbuf* buffer pointers from Tx mbuf ring + * into Rx mbuf ring. + */ + nb_mbufs = p->recycle_tx_mbufs_reuse(qd, recycle_rxq_info); + + /* If no recycling mbufs, return 0. */ + if (nb_mbufs == 0) + return 0; + +#ifdef RTE_ETHDEV_DEBUG_RX + if (rx_port_id >= RTE_MAX_ETHPORTS || + rx_queue_id >= RTE_MAX_QUEUES_PER_PORT) { + RTE_ETHDEV_LOG(ERR, "Invalid rx_port_id=%u or rx_queue_id=%u\n", + rx_port_id, rx_queue_id); + return 0; + } +#endif + + /* fetch pointer to queue data */ + p = &rte_eth_fp_ops[rx_port_id]; + qd = p->rxq.data[rx_queue_id]; + +#ifdef RTE_ETHDEV_DEBUG_RX + RTE_ETH_VALID_PORTID_OR_ERR_RET(rx_port_id, 0); + + if (qd == NULL) { + RTE_ETHDEV_LOG(ERR, "Invalid Rx queue_id=%u for port_id=%u\n", + rx_queue_id, rx_port_id); + return 0; + } +#endif + + if (p->recycle_rx_descriptors_refill == NULL) + return 0; + + /* Replenish the Rx descriptors with the recycling + * into Rx mbuf ring. + */ + p->recycle_rx_descriptors_refill(qd, nb_mbufs); + + return nb_mbufs; +} + /** * @warning * @b EXPERIMENTAL: this API may change without prior notice diff --git a/lib/ethdev/rte_ethdev_core.h b/lib/ethdev/rte_ethdev_core.h index dcf8adab92..a2e6ea6b6c 100644 --- a/lib/ethdev/rte_ethdev_core.h +++ b/lib/ethdev/rte_ethdev_core.h @@ -56,6 +56,13 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq, uint16_t offset); /** @internal Check the status of a Tx descriptor */ typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t offset); +/** @internal Copy used mbufs from Tx mbuf ring into Rx mbuf ring */ +typedef uint16_t (*eth_recycle_tx_mbufs_reuse_t)(void *txq, + struct rte_eth_recycle_rxq_info *recycle_rxq_info); + +/** @internal Refill Rx descriptors with the recycling mbufs */ +typedef void (*eth_recycle_rx_descriptors_refill_t)(void *rxq, uint16_t nb); + /** * @internal * Structure used to hold opaque pointers to internal ethdev Rx/Tx @@ -90,9 +97,11 @@ struct rte_eth_fp_ops { eth_rx_queue_count_t rx_queue_count; /** Check the status of a Rx descriptor. */ eth_rx_descriptor_status_t rx_descriptor_status; + /** Refill Rx descriptors with the recycling mbufs. */ + eth_recycle_rx_descriptors_refill_t recycle_rx_descriptors_refill; /** Rx queues data. */ struct rte_ethdev_qdata rxq; - uintptr_t reserved1[3]; + uintptr_t reserved1[2]; /**@}*/ /**@{*/ @@ -106,9 +115,11 @@ struct rte_eth_fp_ops { eth_tx_prep_t tx_pkt_prepare; /** Check the status of a Tx descriptor. */ eth_tx_descriptor_status_t tx_descriptor_status; + /** Copy used mbufs from Tx mbuf ring into Rx. */ + eth_recycle_tx_mbufs_reuse_t recycle_tx_mbufs_reuse; /** Tx queues data. */ struct rte_ethdev_qdata txq; - uintptr_t reserved2[3]; + uintptr_t reserved2[2]; /**@}*/ } __rte_cache_aligned; diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map index 357d1a88c0..45c417f6bd 100644 --- a/lib/ethdev/version.map +++ b/lib/ethdev/version.map @@ -299,6 +299,10 @@ EXPERIMENTAL { rte_flow_action_handle_query_update; rte_flow_async_action_handle_query_update; rte_flow_async_create_by_index; + + # added in 23.07 + rte_eth_recycle_mbufs; + rte_eth_recycle_rx_queue_info_get; }; INTERNAL { From patchwork Thu May 25 09:45:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feifei Wang X-Patchwork-Id: 127386 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D158C42B98; Thu, 25 May 2023 11:46:05 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7E01642D33; Thu, 25 May 2023 11:45:56 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id 3241342D33 for ; Thu, 25 May 2023 11:45:55 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CD5321042; Thu, 25 May 2023 02:46:39 -0700 (PDT) Received: from net-x86-dell-8268.shanghai.arm.com (net-x86-dell-8268.shanghai.arm.com [10.169.210.116]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 648683F67D; Thu, 25 May 2023 02:45:52 -0700 (PDT) From: Feifei Wang To: Yuying Zhang , Beilei Xing Cc: dev@dpdk.org, nd@arm.com, Feifei Wang , Honnappa Nagarahalli , Ruifeng Wang Subject: [PATCH v6 2/4] net/i40e: implement mbufs recycle mode Date: Thu, 25 May 2023 17:45:39 +0800 Message-Id: <20230525094541.331338-3-feifei.wang2@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230525094541.331338-1-feifei.wang2@arm.com> References: <20211224164613.32569-1-feifei.wang2@arm.com> <20230525094541.331338-1-feifei.wang2@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Define specific function implementation for i40e driver. Currently, mbufs recycle mode can support 128bit vector path and avx2 path. And can be enabled both in fast free and no fast free mode. Suggested-by: Honnappa Nagarahalli Signed-off-by: Feifei Wang Reviewed-by: Ruifeng Wang Reviewed-by: Honnappa Nagarahalli Signed-off-by: Feifei Wang > Reviewed-by: Ruifeng Wang > Reviewed-by: Honnappa Nagarahalli > --- drivers/net/i40e/i40e_ethdev.c | 1 + drivers/net/i40e/i40e_ethdev.h | 2 + .../net/i40e/i40e_recycle_mbufs_vec_common.c | 140 ++++++++++++++++++ drivers/net/i40e/i40e_rxtx.c | 32 ++++ drivers/net/i40e/i40e_rxtx.h | 4 + drivers/net/i40e/meson.build | 2 + 6 files changed, 181 insertions(+) create mode 100644 drivers/net/i40e/i40e_recycle_mbufs_vec_common.c diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c index f9d8f9791f..d4eecd16cf 100644 --- a/drivers/net/i40e/i40e_ethdev.c +++ b/drivers/net/i40e/i40e_ethdev.c @@ -496,6 +496,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = { .flow_ops_get = i40e_dev_flow_ops_get, .rxq_info_get = i40e_rxq_info_get, .txq_info_get = i40e_txq_info_get, + .recycle_rxq_info_get = i40e_recycle_rxq_info_get, .rx_burst_mode_get = i40e_rx_burst_mode_get, .tx_burst_mode_get = i40e_tx_burst_mode_get, .timesync_enable = i40e_timesync_enable, diff --git a/drivers/net/i40e/i40e_ethdev.h b/drivers/net/i40e/i40e_ethdev.h index 9b806d130e..b5b2d6cf2b 100644 --- a/drivers/net/i40e/i40e_ethdev.h +++ b/drivers/net/i40e/i40e_ethdev.h @@ -1355,6 +1355,8 @@ void i40e_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_rxq_info *qinfo); void i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_txq_info *qinfo); +void i40e_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, + struct rte_eth_recycle_rxq_info *recycle_rxq_info); int i40e_rx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_burst_mode *mode); int i40e_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id, diff --git a/drivers/net/i40e/i40e_recycle_mbufs_vec_common.c b/drivers/net/i40e/i40e_recycle_mbufs_vec_common.c new file mode 100644 index 0000000000..08d708fd7d --- /dev/null +++ b/drivers/net/i40e/i40e_recycle_mbufs_vec_common.c @@ -0,0 +1,140 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Arm Limited. + */ + +#include +#include + +#include "base/i40e_prototype.h" +#include "base/i40e_type.h" +#include "i40e_ethdev.h" +#include "i40e_rxtx.h" + +#pragma GCC diagnostic ignored "-Wcast-qual" + +void +i40e_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs) +{ + struct i40e_rx_queue *rxq = rx_queue; + struct i40e_rx_entry *rxep; + volatile union i40e_rx_desc *rxdp; + uint16_t rx_id; + uint64_t paddr; + uint64_t dma_addr; + uint16_t i; + + rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxep = &rxq->sw_ring[rxq->rxrearm_start]; + + for (i = 0; i < nb_mbufs; i++) { + /* Initialize rxdp descs. */ + paddr = (rxep[i].mbuf)->buf_iova + RTE_PKTMBUF_HEADROOM; + dma_addr = rte_cpu_to_le_64(paddr); + /* flush desc with pa dma_addr */ + rxdp[i].read.hdr_addr = 0; + rxdp[i].read.pkt_addr = dma_addr; + } + + /* Update the descriptor initializer index */ + rxq->rxrearm_start += nb_mbufs; + rx_id = rxq->rxrearm_start - 1; + + if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { + rxq->rxrearm_start = 0; + rx_id = rxq->nb_rx_desc - 1; + } + + rxq->rxrearm_nb -= nb_mbufs; + + rte_io_wmb(); + /* Update the tail pointer on the NIC */ + I40E_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id); +} + +uint16_t +i40e_recycle_tx_mbufs_reuse_vec(void *tx_queue, + struct rte_eth_recycle_rxq_info *recycle_rxq_info) +{ + struct i40e_tx_queue *txq = tx_queue; + struct i40e_tx_entry *txep; + struct rte_mbuf **rxep; + struct rte_mbuf *m[RTE_I40E_TX_MAX_FREE_BUF_SZ]; + int i, j, n; + uint16_t avail = 0; + uint16_t mbuf_ring_size = recycle_rxq_info->mbuf_ring_size; + uint16_t mask = recycle_rxq_info->mbuf_ring_size - 1; + uint16_t refill_requirement = recycle_rxq_info->refill_requirement; + uint16_t refill_head = *recycle_rxq_info->refill_head; + uint16_t receive_tail = *recycle_rxq_info->receive_tail; + + /* Get available recycling Rx buffers. */ + avail = (mbuf_ring_size - (refill_head - receive_tail)) & mask; + + /* Check Tx free thresh and Rx available space. */ + if (txq->nb_tx_free > txq->tx_free_thresh || avail <= txq->tx_rs_thresh) + return 0; + + /* check DD bits on threshold descriptor */ + if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & + rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != + rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE)) + return 0; + + n = txq->tx_rs_thresh; + + /* Mbufs recycle mode can only support no ring buffer wrapping around. + * Two case for this: + * + * case 1: The refill head of Rx buffer ring needs to be aligned with + * mbuf ring size. In this case, the number of Tx freeing buffers + * should be equal to refill_requirement. + * + * case 2: The refill head of Rx ring buffer does not need to be aligned + * with mbuf ring size. In this case, the update of refill head can not + * exceed the Rx mbuf ring size. + */ + if (refill_requirement != n || + (!refill_requirement && (refill_head + n > mbuf_ring_size))) + return 0; + + /* First buffer to free from S/W ring is at index + * tx_next_dd - (tx_rs_thresh-1). + */ + txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)]; + rxep = recycle_rxq_info->mbuf_ring; + rxep += refill_head; + + if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { + /* Directly put mbufs from Tx to Rx. */ + for (i = 0; i < n; i++, rxep++, txep++) + *rxep = txep[0].mbuf; + } else { + for (i = 0, j = 0; i < n; i++) { + /* Avoid txq contains buffers from expected mempool. */ + if (unlikely(recycle_rxq_info->mp + != txep[i].mbuf->pool)) + return 0; + + m[j] = rte_pktmbuf_prefree_seg(txep[i].mbuf); + + /* In case 1, each of Tx buffers should be the + * last reference. + */ + if (unlikely(m[j] == NULL && refill_requirement)) + return 0; + /* In case 2, the number of valid Tx free + * buffers should be recorded. + */ + j++; + } + rte_memcpy(rxep, m, sizeof(void *) * j); + } + + /* Update counters for Tx. */ + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); + if (txq->tx_next_dd >= txq->nb_tx_desc) + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + + return n; +} diff --git a/drivers/net/i40e/i40e_rxtx.c b/drivers/net/i40e/i40e_rxtx.c index 788ffb51c2..53cf787f04 100644 --- a/drivers/net/i40e/i40e_rxtx.c +++ b/drivers/net/i40e/i40e_rxtx.c @@ -3197,6 +3197,30 @@ i40e_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, qinfo->conf.offloads = txq->offloads; } +void +i40e_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, + struct rte_eth_recycle_rxq_info *recycle_rxq_info) +{ + struct i40e_rx_queue *rxq; + struct i40e_adapter *ad = + I40E_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); + + rxq = dev->data->rx_queues[queue_id]; + + recycle_rxq_info->mbuf_ring = (void *)rxq->sw_ring; + recycle_rxq_info->mp = rxq->mp; + recycle_rxq_info->mbuf_ring_size = rxq->nb_rx_desc; + recycle_rxq_info->receive_tail = &rxq->rx_tail; + + if (ad->rx_vec_allowed) { + recycle_rxq_info->refill_requirement = RTE_I40E_RXQ_REARM_THRESH; + recycle_rxq_info->refill_head = &rxq->rxrearm_start; + } else { + recycle_rxq_info->refill_requirement = rxq->rx_free_thresh; + recycle_rxq_info->refill_head = &rxq->rx_free_trigger; + } +} + #ifdef RTE_ARCH_X86 static inline bool get_avx_supported(bool request_avx512) @@ -3291,6 +3315,8 @@ i40e_set_rx_function(struct rte_eth_dev *dev) dev->rx_pkt_burst = ad->rx_use_avx2 ? i40e_recv_scattered_pkts_vec_avx2 : i40e_recv_scattered_pkts_vec; + dev->recycle_rx_descriptors_refill = + i40e_recycle_rx_descriptors_refill_vec; } } else { if (ad->rx_use_avx512) { @@ -3309,9 +3335,12 @@ i40e_set_rx_function(struct rte_eth_dev *dev) dev->rx_pkt_burst = ad->rx_use_avx2 ? i40e_recv_pkts_vec_avx2 : i40e_recv_pkts_vec; + dev->recycle_rx_descriptors_refill = + i40e_recycle_rx_descriptors_refill_vec; } } #else /* RTE_ARCH_X86 */ + dev->recycle_rx_descriptors_refill = i40e_recycle_rx_descriptors_refill_vec; if (dev->data->scattered_rx) { PMD_INIT_LOG(DEBUG, "Using Vector Scattered Rx (port %d).", @@ -3479,15 +3508,18 @@ i40e_set_tx_function(struct rte_eth_dev *dev) dev->tx_pkt_burst = ad->tx_use_avx2 ? i40e_xmit_pkts_vec_avx2 : i40e_xmit_pkts_vec; + dev->recycle_tx_mbufs_reuse = i40e_recycle_tx_mbufs_reuse_vec; } #else /* RTE_ARCH_X86 */ PMD_INIT_LOG(DEBUG, "Using Vector Tx (port %d).", dev->data->port_id); dev->tx_pkt_burst = i40e_xmit_pkts_vec; + dev->recycle_tx_mbufs_reuse = i40e_recycle_tx_mbufs_reuse_vec; #endif /* RTE_ARCH_X86 */ } else { PMD_INIT_LOG(DEBUG, "Simple tx finally be used."); dev->tx_pkt_burst = i40e_xmit_pkts_simple; + dev->recycle_tx_mbufs_reuse = i40e_recycle_tx_mbufs_reuse_vec; } dev->tx_pkt_prepare = i40e_simple_prep_pkts; } else { diff --git a/drivers/net/i40e/i40e_rxtx.h b/drivers/net/i40e/i40e_rxtx.h index 5e6eecc501..ed8921ddc0 100644 --- a/drivers/net/i40e/i40e_rxtx.h +++ b/drivers/net/i40e/i40e_rxtx.h @@ -233,6 +233,10 @@ uint32_t i40e_dev_rx_queue_count(void *rx_queue); int i40e_dev_rx_descriptor_status(void *rx_queue, uint16_t offset); int i40e_dev_tx_descriptor_status(void *tx_queue, uint16_t offset); +uint16_t i40e_recycle_tx_mbufs_reuse_vec(void *tx_queue, + struct rte_eth_recycle_rxq_info *recycle_rxq_info); +void i40e_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs); + uint16_t i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); uint16_t i40e_recv_scattered_pkts_vec(void *rx_queue, diff --git a/drivers/net/i40e/meson.build b/drivers/net/i40e/meson.build index 8e53b87a65..58eb627abc 100644 --- a/drivers/net/i40e/meson.build +++ b/drivers/net/i40e/meson.build @@ -42,6 +42,8 @@ testpmd_sources = files('i40e_testpmd.c') deps += ['hash'] includes += include_directories('base') +sources += files('i40e_recycle_mbufs_vec_common.c') + if arch_subdir == 'x86' sources += files('i40e_rxtx_vec_sse.c') From patchwork Thu May 25 09:45:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feifei Wang X-Patchwork-Id: 127388 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 10CAF42B98; Thu, 25 May 2023 11:46:12 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AF6DA42D10; Thu, 25 May 2023 11:45:59 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id 30C3A42D10 for ; Thu, 25 May 2023 11:45:58 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C36A71042; Thu, 25 May 2023 02:46:42 -0700 (PDT) Received: from net-x86-dell-8268.shanghai.arm.com (net-x86-dell-8268.shanghai.arm.com [10.169.210.116]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 59E423F67D; Thu, 25 May 2023 02:45:55 -0700 (PDT) From: Feifei Wang To: Qiming Yang , Wenjun Wu Cc: dev@dpdk.org, nd@arm.com, Feifei Wang , Honnappa Nagarahalli , Ruifeng Wang Subject: [PATCH v6 3/4] net/ixgbe: implement mbufs recycle mode Date: Thu, 25 May 2023 17:45:40 +0800 Message-Id: <20230525094541.331338-4-feifei.wang2@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230525094541.331338-1-feifei.wang2@arm.com> References: <20211224164613.32569-1-feifei.wang2@arm.com> <20230525094541.331338-1-feifei.wang2@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Define specific function implementation for ixgbe driver. Currently, recycle buffer mode can support 128bit vector path. And can be enabled both in fast free and no fast free mode. Suggested-by: Honnappa Nagarahalli Signed-off-by: Feifei Wang Reviewed-by: Ruifeng Wang Reviewed-by: Honnappa Nagarahalli --- drivers/net/ixgbe/ixgbe_ethdev.c | 1 + drivers/net/ixgbe/ixgbe_ethdev.h | 3 + .../ixgbe/ixgbe_recycle_mbufs_vec_common.c | 136 ++++++++++++++++++ drivers/net/ixgbe/ixgbe_rxtx.c | 29 ++++ drivers/net/ixgbe/ixgbe_rxtx.h | 4 + drivers/net/ixgbe/meson.build | 2 + 6 files changed, 175 insertions(+) create mode 100644 drivers/net/ixgbe/ixgbe_recycle_mbufs_vec_common.c diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c index 88118bc305..db47100a37 100644 --- a/drivers/net/ixgbe/ixgbe_ethdev.c +++ b/drivers/net/ixgbe/ixgbe_ethdev.c @@ -543,6 +543,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = { .set_mc_addr_list = ixgbe_dev_set_mc_addr_list, .rxq_info_get = ixgbe_rxq_info_get, .txq_info_get = ixgbe_txq_info_get, + .recycle_rxq_info_get = ixgbe_recycle_rxq_info_get, .timesync_enable = ixgbe_timesync_enable, .timesync_disable = ixgbe_timesync_disable, .timesync_read_rx_timestamp = ixgbe_timesync_read_rx_timestamp, diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h index 48290af512..9169930c7f 100644 --- a/drivers/net/ixgbe/ixgbe_ethdev.h +++ b/drivers/net/ixgbe/ixgbe_ethdev.h @@ -625,6 +625,9 @@ void ixgbe_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, void ixgbe_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_txq_info *qinfo); +void ixgbe_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, + struct rte_eth_recycle_rxq_info *recycle_rxq_info); + int ixgbevf_dev_rx_init(struct rte_eth_dev *dev); void ixgbevf_dev_tx_init(struct rte_eth_dev *dev); diff --git a/drivers/net/ixgbe/ixgbe_recycle_mbufs_vec_common.c b/drivers/net/ixgbe/ixgbe_recycle_mbufs_vec_common.c new file mode 100644 index 0000000000..f234e95e92 --- /dev/null +++ b/drivers/net/ixgbe/ixgbe_recycle_mbufs_vec_common.c @@ -0,0 +1,136 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Arm Limited. + */ + +#include +#include + +#include "ixgbe_ethdev.h" +#include "ixgbe_rxtx.h" + +#pragma GCC diagnostic ignored "-Wcast-qual" + +void +ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs) +{ + struct ixgbe_rx_queue *rxq = rx_queue; + struct ixgbe_rx_entry *rxep; + volatile union ixgbe_adv_rx_desc *rxdp; + uint16_t rx_id; + uint64_t paddr; + uint64_t dma_addr; + uint16_t i; + + rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxep = &rxq->sw_ring[rxq->rxrearm_start]; + + for (i = 0; i < nb_mbufs; i++) { + /* Initialize rxdp descs. */ + paddr = (rxep[i].mbuf)->buf_iova + RTE_PKTMBUF_HEADROOM; + dma_addr = rte_cpu_to_le_64(paddr); + /* Flush descriptors with pa dma_addr */ + rxdp[i].read.hdr_addr = 0; + rxdp[i].read.pkt_addr = dma_addr; + } + + /* Update the descriptor initializer index */ + rxq->rxrearm_start += nb_mbufs; + if (rxq->rxrearm_start >= rxq->nb_rx_desc) + rxq->rxrearm_start = 0; + + rxq->rxrearm_nb -= nb_mbufs; + + rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? + (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); + + /* Update the tail pointer on the NIC */ + IXGBE_PCI_REG_WRITE(rxq->rdt_reg_addr, rx_id); +} + +uint16_t +ixgbe_recycle_tx_mbufs_reuse_vec(void *tx_queue, + struct rte_eth_recycle_rxq_info *recycle_rxq_info) +{ + struct ixgbe_tx_queue *txq = tx_queue; + struct ixgbe_tx_entry *txep; + struct rte_mbuf **rxep; + struct rte_mbuf *m[RTE_IXGBE_TX_MAX_FREE_BUF_SZ]; + int i, j, n; + uint32_t status; + uint16_t avail = 0; + uint16_t mbuf_ring_size = recycle_rxq_info->mbuf_ring_size; + uint16_t mask = recycle_rxq_info->mbuf_ring_size - 1; + uint16_t refill_requirement = recycle_rxq_info->refill_requirement; + uint16_t refill_head = *recycle_rxq_info->refill_head; + uint16_t receive_tail = *recycle_rxq_info->receive_tail; + + /* Get available recycling Rx buffers. */ + avail = (mbuf_ring_size - (refill_head - receive_tail)) & mask; + + /* Check Tx free thresh and Rx available space. */ + if (txq->nb_tx_free > txq->tx_free_thresh || avail <= txq->tx_rs_thresh) + return 0; + + /* check DD bits on threshold descriptor */ + status = txq->tx_ring[txq->tx_next_dd].wb.status; + if (!(status & IXGBE_ADVTXD_STAT_DD)) + return 0; + + n = txq->tx_rs_thresh; + + /* Mbufs recycle can only support no ring buffer wrapping around. + * Two case for this: + * + * case 1: The refill head of Rx buffer ring needs to be aligned with + * buffer ring size. In this case, the number of Tx freeing buffers + * should be equal to refill_requirement. + * + * case 2: The refill head of Rx ring buffer does not need to be aligned + * with buffer ring size. In this case, the update of refill head can not + * exceed the Rx buffer ring size. + */ + if (refill_requirement != n || + (!refill_requirement && (refill_head + n > mbuf_ring_size))) + return 0; + + /* First buffer to free from S/W ring is at index + * tx_next_dd - (tx_rs_thresh-1). + */ + txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)]; + rxep = recycle_rxq_info->mbuf_ring; + rxep += refill_head; + + if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { + /* Directly put mbufs from Tx to Rx. */ + for (i = 0; i < n; i++, rxep++, txep++) + *rxep = txep[0].mbuf; + } else { + for (i = 0, j = 0; i < n; i++) { + /* Avoid txq contains buffers from expected mempool. */ + if (unlikely(recycle_rxq_info->mp + != txep[i].mbuf->pool)) + return 0; + + m[j] = rte_pktmbuf_prefree_seg(txep[i].mbuf); + + /* In case 1, each of Tx buffers should be the + * last reference. + */ + if (unlikely(m[j] == NULL && refill_requirement)) + return 0; + /* In case 2, the number of valid Tx free + * buffers should be recorded. + */ + j++; + } + rte_memcpy(rxep, m, sizeof(void *) * j); + } + + /* Update counters for Tx. */ + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); + if (txq->tx_next_dd >= txq->nb_tx_desc) + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + + return n; +} diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c index c9d6ca9efe..ef2ea84bfa 100644 --- a/drivers/net/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/ixgbe/ixgbe_rxtx.c @@ -2558,6 +2558,7 @@ ixgbe_set_tx_function(struct rte_eth_dev *dev, struct ixgbe_tx_queue *txq) (rte_eal_process_type() != RTE_PROC_PRIMARY || ixgbe_txq_vec_setup(txq) == 0)) { PMD_INIT_LOG(DEBUG, "Vector tx enabled."); + dev->recycle_tx_mbufs_reuse = ixgbe_recycle_tx_mbufs_reuse_vec; dev->tx_pkt_burst = ixgbe_xmit_pkts_vec; } else dev->tx_pkt_burst = ixgbe_xmit_pkts_simple; @@ -4823,6 +4824,8 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev) "callback (port=%d).", dev->data->port_id); + dev->recycle_rx_descriptors_refill = + ixgbe_recycle_rx_descriptors_refill_vec; dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec; } else if (adapter->rx_bulk_alloc_allowed) { PMD_INIT_LOG(DEBUG, "Using a Scattered with bulk " @@ -4852,6 +4855,7 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev) RTE_IXGBE_DESCS_PER_LOOP, dev->data->port_id); + dev->recycle_rx_descriptors_refill = ixgbe_recycle_rx_descriptors_refill_vec; dev->rx_pkt_burst = ixgbe_recv_pkts_vec; } else if (adapter->rx_bulk_alloc_allowed) { PMD_INIT_LOG(DEBUG, "Rx Burst Bulk Alloc Preconditions are " @@ -5623,6 +5627,31 @@ ixgbe_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, qinfo->conf.tx_deferred_start = txq->tx_deferred_start; } +void +ixgbe_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, + struct rte_eth_recycle_rxq_info *recycle_rxq_info) +{ + struct ixgbe_rx_queue *rxq; + struct ixgbe_adapter *adapter = dev->data->dev_private; + + rxq = dev->data->rx_queues[queue_id]; + + recycle_rxq_info->mbuf_ring = (void *)rxq->sw_ring; + recycle_rxq_info->mp = rxq->mb_pool; + recycle_rxq_info->mbuf_ring_size = rxq->nb_rx_desc; + recycle_rxq_info->receive_tail = &rxq->rx_tail; + + if (adapter->rx_vec_allowed) { +#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM) + recycle_rxq_info->refill_requirement = RTE_IXGBE_RXQ_REARM_THRESH; + recycle_rxq_info->refill_head = &rxq->rxrearm_start; +#endif + } else { + recycle_rxq_info->refill_requirement = rxq->rx_free_thresh; + recycle_rxq_info->refill_head = &rxq->rx_free_trigger; + } +} + /* * [VF] Initializes Receive Unit. */ diff --git a/drivers/net/ixgbe/ixgbe_rxtx.h b/drivers/net/ixgbe/ixgbe_rxtx.h index 668a5b9814..ee89c89929 100644 --- a/drivers/net/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/ixgbe/ixgbe_rxtx.h @@ -295,6 +295,10 @@ int ixgbe_dev_tx_done_cleanup(void *tx_queue, uint32_t free_cnt); extern const uint32_t ptype_table[IXGBE_PACKET_TYPE_MAX]; extern const uint32_t ptype_table_tn[IXGBE_PACKET_TYPE_TN_MAX]; +uint16_t ixgbe_recycle_tx_mbufs_reuse_vec(void *tx_queue, + struct rte_eth_recycle_rxq_info *recycle_rxq_info); +void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs); + uint16_t ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts); int ixgbe_txq_vec_setup(struct ixgbe_tx_queue *txq); diff --git a/drivers/net/ixgbe/meson.build b/drivers/net/ixgbe/meson.build index a18908ef7c..0ae12dd5ff 100644 --- a/drivers/net/ixgbe/meson.build +++ b/drivers/net/ixgbe/meson.build @@ -26,11 +26,13 @@ deps += ['hash', 'security'] if arch_subdir == 'x86' sources += files('ixgbe_rxtx_vec_sse.c') + sources += files('ixgbe_recycle_mbufs_vec_common.c') if is_windows and cc.get_id() != 'clang' cflags += ['-fno-asynchronous-unwind-tables'] endif elif arch_subdir == 'arm' sources += files('ixgbe_rxtx_vec_neon.c') + sources += files('ixgbe_recycle_mbufs_vec_common.c') endif includes += include_directories('base') From patchwork Thu May 25 09:45:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feifei Wang X-Patchwork-Id: 127391 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1323642B98; Thu, 25 May 2023 11:46:21 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4D9CD42D42; Thu, 25 May 2023 11:46:02 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id 1BB5542D41 for ; Thu, 25 May 2023 11:46:01 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B75371042; Thu, 25 May 2023 02:46:45 -0700 (PDT) Received: from net-x86-dell-8268.shanghai.arm.com (net-x86-dell-8268.shanghai.arm.com [10.169.210.116]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4EBD03F67D; Thu, 25 May 2023 02:45:58 -0700 (PDT) From: Feifei Wang To: Aman Singh , Yuying Zhang Cc: dev@dpdk.org, nd@arm.com, Feifei Wang , Jerin Jacob , Ruifeng Wang Subject: [PATCH v6 4/4] app/testpmd: add recycle mbufs engine Date: Thu, 25 May 2023 17:45:41 +0800 Message-Id: <20230525094541.331338-5-feifei.wang2@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230525094541.331338-1-feifei.wang2@arm.com> References: <20211224164613.32569-1-feifei.wang2@arm.com> <20230525094541.331338-1-feifei.wang2@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add recycle mbufs engine for testpmd. This engine forward pkts with I/O forward mode. But enable mbufs recycle feature to recycle used txq mbufs for rxq mbuf ring, which can bypass mempool path and save CPU cycles. Suggested-by: Jerin Jacob Signed-off-by: Feifei Wang Reviewed-by: Ruifeng Wang --- app/test-pmd/meson.build | 1 + app/test-pmd/recycle_mbufs.c | 79 +++++++++++++++++++++ app/test-pmd/testpmd.c | 1 + app/test-pmd/testpmd.h | 3 + doc/guides/testpmd_app_ug/run_app.rst | 1 + doc/guides/testpmd_app_ug/testpmd_funcs.rst | 5 +- 6 files changed, 89 insertions(+), 1 deletion(-) create mode 100644 app/test-pmd/recycle_mbufs.c diff --git a/app/test-pmd/meson.build b/app/test-pmd/meson.build index d2e3f60892..6e5f067274 100644 --- a/app/test-pmd/meson.build +++ b/app/test-pmd/meson.build @@ -22,6 +22,7 @@ sources = files( 'macswap.c', 'noisy_vnf.c', 'parameters.c', + 'recycle_mbufs.c', 'rxonly.c', 'shared_rxq_fwd.c', 'testpmd.c', diff --git a/app/test-pmd/recycle_mbufs.c b/app/test-pmd/recycle_mbufs.c new file mode 100644 index 0000000000..0c603c3ec2 --- /dev/null +++ b/app/test-pmd/recycle_mbufs.c @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Arm Limited. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "testpmd.h" + +/* + * Forwarding of packets in I/O mode. + * Enable mbufs recycle mode to recycle txq used mbufs + * for rxq mbuf ring. This can bypass mempool path and + * save CPU cycles. + */ +static bool +pkt_burst_recycle_mbufs(struct fwd_stream *fs) +{ + struct rte_mbuf *pkts_burst[MAX_PKT_BURST]; + uint16_t nb_rx; + + /* Recycle used mbufs from the txq, and move these mbufs into + * the rxq mbuf ring. + */ + rte_eth_recycle_mbufs(fs->rx_port, fs->rx_queue, + fs->tx_port, fs->tx_queue, &(fs->recycle_rxq_info)); + + /* + * Receive a burst of packets and forward them. + */ + nb_rx = common_fwd_stream_receive(fs, pkts_burst, nb_pkt_per_burst); + if (unlikely(nb_rx == 0)) + return false; + + common_fwd_stream_transmit(fs, pkts_burst, nb_rx); + + return true; +} + +static void +recycle_mbufs_stream_init(struct fwd_stream *fs) +{ + /* Retrieve information about given ports's Rx queue + * for recycling mbufs. + */ + rte_eth_recycle_rx_queue_info_get(fs->rx_port, fs->rx_queue, + &(fs->recycle_rxq_info)); + + common_fwd_stream_init(fs); +} + +struct fwd_engine recycle_mbufs_engine = { + .fwd_mode_name = "recycle_mbufs", + .stream_init = recycle_mbufs_stream_init, + .packet_fwd = pkt_burst_recycle_mbufs, +}; diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 5cb6f92523..050e48d79a 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -199,6 +199,7 @@ struct fwd_engine * fwd_engines[] = { &icmp_echo_engine, &noisy_vnf_engine, &five_tuple_swap_fwd_engine, + &recycle_mbufs_engine, #ifdef RTE_LIBRTE_IEEE1588 &ieee1588_fwd_engine, #endif diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index bdfbfd36d3..34e72fd7d5 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -179,6 +179,8 @@ struct fwd_stream { struct pkt_burst_stats rx_burst_stats; struct pkt_burst_stats tx_burst_stats; struct fwd_lcore *lcore; /**< Lcore being scheduled. */ + /**< Rx queue information for recycling mbufs */ + struct rte_eth_recycle_rxq_info recycle_rxq_info; }; /** @@ -432,6 +434,7 @@ extern struct fwd_engine csum_fwd_engine; extern struct fwd_engine icmp_echo_engine; extern struct fwd_engine noisy_vnf_engine; extern struct fwd_engine five_tuple_swap_fwd_engine; +extern struct fwd_engine recycle_mbufs_engine; #ifdef RTE_LIBRTE_IEEE1588 extern struct fwd_engine ieee1588_fwd_engine; #endif diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst index 57b23241cf..cbc68acc36 100644 --- a/doc/guides/testpmd_app_ug/run_app.rst +++ b/doc/guides/testpmd_app_ug/run_app.rst @@ -232,6 +232,7 @@ The command line options are: noisy 5tswap shared-rxq + recycle_mbufs * ``--rss-ip`` diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst index 8f23847859..482e583263 100644 --- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst +++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst @@ -318,7 +318,7 @@ set fwd Set the packet forwarding mode:: testpmd> set fwd (io|mac|macswap|flowgen| \ - rxonly|txonly|csum|icmpecho|noisy|5tswap|shared-rxq) (""|retry) + rxonly|txonly|csum|icmpecho|noisy|5tswap|shared-rxq|recycle_mbufs) (""|retry) ``retry`` can be specified for forwarding engines except ``rx_only``. @@ -364,6 +364,9 @@ The available information categories are: * ``shared-rxq``: Receive only for shared Rx queue. Resolve packet source port from mbuf and update stream statistics accordingly. +* ``recycle_mbufs``: Recycle Tx queue used mbufs for Rx queue mbuf ring. + This mode uses fast path mbuf recycle feature and forwards packets in I/O mode. + Example:: testpmd> set fwd rxonly