[dpdk-dev,v2] ixgbe: fix multi-process support
Commit Message
When using multiple processes, the TX function used in all processes
should be the same, otherwise the secondary processes cannot transmit
more than tx-ring-size - 1 packets.
To achieve this, we extract out the code to select the ixgbe TX function
to be used into a separate function inside the ixgbe driver, and call
that from a secondary process when it is attaching to an
already-configured NIC.
Testing with symmetric MP app shows that we are able to RX and TX from
both primary and secondary processes once this patch is applied.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
----
V2 Changes:
* Moved check for primary/secondary process into set_tx_function instead
of ixgbe_txq_vec_setup, which reduces scope of diff.
* Added/cleaned up some code comments for this change
---
lib/librte_pmd_ixgbe/ixgbe_ethdev.c | 12 +++++--
lib/librte_pmd_ixgbe/ixgbe_rxtx.c | 62 ++++++++++++++++++++++---------------
lib/librte_pmd_ixgbe/ixgbe_rxtx.h | 7 +++++
3 files changed, 54 insertions(+), 27 deletions(-)
Comments
2014-12-05 13:46, Bruce Richardson:
> When using multiple processes, the TX function used in all processes
> should be the same, otherwise the secondary processes cannot transmit
> more than tx-ring-size - 1 packets.
> To achieve this, we extract out the code to select the ixgbe TX function
> to be used into a separate function inside the ixgbe driver, and call
> that from a secondary process when it is attaching to an
> already-configured NIC.
>
> Testing with symmetric MP app shows that we are able to RX and TX from
> both primary and secondary processes once this patch is applied.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>
> ----
> V2 Changes:
> * Moved check for primary/secondary process into set_tx_function instead
> of ixgbe_txq_vec_setup, which reduces scope of diff.
> * Added/cleaned up some code comments for this change
Anyone to review this patch?
Thanks
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> Sent: Friday, December 5, 2014 1:46 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v2] ixgbe: fix multi-process support
>
> When using multiple processes, the TX function used in all processes should be
> the same, otherwise the secondary processes cannot transmit more than tx-
> ring-size - 1 packets.
> To achieve this, we extract out the code to select the ixgbe TX function to be
> used into a separate function inside the ixgbe driver, and call that from a
> secondary process when it is attaching to an already-configured NIC.
>
> Testing with symmetric MP app shows that we are able to RX and TX from both
> primary and secondary processes once this patch is applied.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>
> ----
> V2 Changes:
> * Moved check for primary/secondary process into set_tx_function instead
> of ixgbe_txq_vec_setup, which reduces scope of diff.
> * Added/cleaned up some code comments for this change
> ---
Acked-by: Reshma Pattan<reshma.pattan@intel.com>
With comment " checkpatch shows whitespace errors which got copied from old code"
On Thu, Dec 11, 2014 at 04:58:42PM +0000, Pattan, Reshma wrote:
>
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > Sent: Friday, December 5, 2014 1:46 PM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] [PATCH v2] ixgbe: fix multi-process support
> >
> > When using multiple processes, the TX function used in all processes should be
> > the same, otherwise the secondary processes cannot transmit more than tx-
> > ring-size - 1 packets.
> > To achieve this, we extract out the code to select the ixgbe TX function to be
> > used into a separate function inside the ixgbe driver, and call that from a
> > secondary process when it is attaching to an already-configured NIC.
> >
> > Testing with symmetric MP app shows that we are able to RX and TX from both
> > primary and secondary processes once this patch is applied.
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> >
> > ----
> > V2 Changes:
> > * Moved check for primary/secondary process into set_tx_function instead
> > of ixgbe_txq_vec_setup, which reduces scope of diff.
> > * Added/cleaned up some code comments for this change
> > ---
> Acked-by: Reshma Pattan<reshma.pattan@intel.com>
> With comment " checkpatch shows whitespace errors which got copied from old code"
Thomas, can you fix the whitespace problems when you apply, or do you want a V3?
/Bruce
2014-12-11 17:07, Bruce Richardson:
> On Thu, Dec 11, 2014 at 04:58:42PM +0000, Pattan, Reshma wrote:
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > > When using multiple processes, the TX function used in all processes should be
> > > the same, otherwise the secondary processes cannot transmit more than tx-
> > > ring-size - 1 packets.
> > > To achieve this, we extract out the code to select the ixgbe TX function to be
> > > used into a separate function inside the ixgbe driver, and call that from a
> > > secondary process when it is attaching to an already-configured NIC.
> > >
> > > Testing with symmetric MP app shows that we are able to RX and TX from both
> > > primary and secondary processes once this patch is applied.
> > >
> > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > >
> > > ----
> > > V2 Changes:
> > > * Moved check for primary/secondary process into set_tx_function instead
> > > of ixgbe_txq_vec_setup, which reduces scope of diff.
> > > * Added/cleaned up some code comments for this change
> > > ---
> > Acked-by: Reshma Pattan<reshma.pattan@intel.com>
> > With comment " checkpatch shows whitespace errors which got copied from old code"
>
> Thomas, can you fix the whitespace problems when you apply, or do you want a V3?
Applied with whitespace changes.
Thanks
@@ -68,6 +68,7 @@
#include "ixgbe/ixgbe_common.h"
#include "ixgbe_ethdev.h"
#include "ixgbe_bypass.h"
+#include "ixgbe_rxtx.h"
/*
* High threshold controlling when to start sending XOFF frames. Must be at
@@ -741,10 +742,17 @@ eth_ixgbe_dev_init(__attribute__((unused)) struct eth_driver *eth_drv,
eth_dev->rx_pkt_burst = &ixgbe_recv_pkts;
eth_dev->tx_pkt_burst = &ixgbe_xmit_pkts;
- /* for secondary processes, we don't initialise any further as primary
+ /*
+ * for secondary processes, we don't initialise any further as primary
* has already done this work. Only check we don't need a different
- * RX function */
+ * RX and TX function
+ */
if (rte_eal_process_type() != RTE_PROC_PRIMARY){
+ struct igb_tx_queue *txq;
+ /* TX queue function in primary, set by last queue initialized */
+ txq = eth_dev->data->tx_queues[eth_dev->data->nb_tx_queues-1];
+ set_tx_function(eth_dev, txq);
+
if (eth_dev->data->scattered_rx)
eth_dev->rx_pkt_burst = ixgbe_recv_scattered_pkts;
return 0;
@@ -1771,6 +1771,41 @@ static struct ixgbe_txq_ops def_txq_ops = {
.reset = ixgbe_reset_tx_queue,
};
+/* takes an ethdev and a queue and sets up the tx function to be used based on
+ * the queue parameters. Used in tx_queue_setup by primary process and then
+ * in dev_init by secondary process when attaching to an existing ethdev
+ */
+void
+set_tx_function(struct rte_eth_dev* dev, struct igb_tx_queue* txq)
+{
+ /* Use a simple Tx queue (no offloads, no multi segs) if possible */
+ if (((txq->txq_flags & IXGBE_SIMPLE_FLAGS) == IXGBE_SIMPLE_FLAGS)
+ && (txq->tx_rs_thresh >= RTE_PMD_IXGBE_TX_MAX_BURST)) {
+ PMD_INIT_LOG(INFO, "Using simple tx code path");
+#ifdef RTE_IXGBE_INC_VECTOR
+ if (txq->tx_rs_thresh <= RTE_IXGBE_TX_MAX_FREE_BUF_SZ &&
+ (rte_eal_process_type() != RTE_PROC_PRIMARY ||
+ ixgbe_txq_vec_setup(txq) == 0)) {
+ PMD_INIT_LOG(INFO, "Vector tx enabled.");
+ dev->tx_pkt_burst = ixgbe_xmit_pkts_vec;
+ }
+ else
+#endif
+ dev->tx_pkt_burst = ixgbe_xmit_pkts_simple;
+ } else {
+ PMD_INIT_LOG(INFO, "Using full-featured tx code path");
+ PMD_INIT_LOG(INFO,
+ " - txq_flags = %lx " "[IXGBE_SIMPLE_FLAGS=%lx]",
+ (long unsigned )txq->txq_flags,
+ (long unsigned)IXGBE_SIMPLE_FLAGS);
+ PMD_INIT_LOG(INFO,
+ " - tx_rs_thresh = %lu " "[RTE_PMD_IXGBE_TX_MAX_BURST=%lu]",
+ (long unsigned )txq->tx_rs_thresh,
+ (long unsigned)RTE_PMD_IXGBE_TX_MAX_BURST);
+ dev->tx_pkt_burst = ixgbe_xmit_pkts;
+ }
+}
+
int
ixgbe_dev_tx_queue_setup(struct rte_eth_dev *dev,
uint16_t queue_idx,
@@ -1933,31 +1968,8 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev *dev,
PMD_INIT_LOG(DEBUG, "sw_ring=%p hw_ring=%p dma_addr=0x%"PRIx64,
txq->sw_ring, txq->tx_ring, txq->tx_ring_phys_addr);
- /* Use a simple Tx queue (no offloads, no multi segs) if possible */
- if (((txq->txq_flags & IXGBE_SIMPLE_FLAGS) == IXGBE_SIMPLE_FLAGS) &&
- (txq->tx_rs_thresh >= RTE_PMD_IXGBE_TX_MAX_BURST)) {
- PMD_INIT_LOG(INFO, "Using simple tx code path");
-#ifdef RTE_IXGBE_INC_VECTOR
- if (txq->tx_rs_thresh <= RTE_IXGBE_TX_MAX_FREE_BUF_SZ &&
- ixgbe_txq_vec_setup(txq) == 0) {
- PMD_INIT_LOG(INFO, "Vector tx enabled.");
- dev->tx_pkt_burst = ixgbe_xmit_pkts_vec;
- }
- else
-#endif
- dev->tx_pkt_burst = ixgbe_xmit_pkts_simple;
- } else {
- PMD_INIT_LOG(INFO, "Using full-featured tx code path");
- PMD_INIT_LOG(INFO, " - txq_flags = %lx "
- "[IXGBE_SIMPLE_FLAGS=%lx]",
- (long unsigned)txq->txq_flags,
- (long unsigned)IXGBE_SIMPLE_FLAGS);
- PMD_INIT_LOG(INFO, " - tx_rs_thresh = %lu "
- "[RTE_PMD_IXGBE_TX_MAX_BURST=%lu]",
- (long unsigned)txq->tx_rs_thresh,
- (long unsigned)RTE_PMD_IXGBE_TX_MAX_BURST);
- dev->tx_pkt_burst = ixgbe_xmit_pkts;
- }
+ /* set up vector or scalar TX function as appropriate */
+ set_tx_function(dev, txq);
txq->ops->reset(txq);
@@ -248,6 +248,13 @@ struct ixgbe_txq_ops {
IXGBE_ADVTXD_DCMD_DEXT |\
IXGBE_ADVTXD_DCMD_EOP)
+
+/* takes an ethdev and a queue and sets up the tx function to be used based on
+ * the queue parameters. Used in tx_queue_setup by primary process and then
+ * in dev_init by secondary process when attaching to an existing ethdev
+ */
+void set_tx_function(struct rte_eth_dev* dev, struct igb_tx_queue* txq);
+
#ifdef RTE_IXGBE_INC_VECTOR
uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,
uint16_t nb_pkts);