From patchwork Tue Aug 1 16:54:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 27338 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 99A29A1C0; Tue, 1 Aug 2017 18:55:50 +0200 (CEST) Received: from mail-wm0-f50.google.com (mail-wm0-f50.google.com [74.125.82.50]) by dpdk.org (Postfix) with ESMTP id 027ADA108 for ; Tue, 1 Aug 2017 18:55:22 +0200 (CEST) Received: by mail-wm0-f50.google.com with SMTP id t201so20255599wmt.1 for ; Tue, 01 Aug 2017 09:55:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NipZie1IRiZT/pOVKr/tyzqG4SEK+qSAp3fYdCy7uIQ=; b=mgsj9sQ8lvMvn762K+JNzEzSiBJHht77WjPE+pIT9Z/fk7FEY4XzIgml2vtIOqJfUY +t5BYjhoi+9ztTLEdcsVDo6li39au7+nyoanAUuFcWJyCgk+S9X542XvyBHeDL1TOLIZ DZp7S20roMOb4KnaGCTGStRUbNVlAQcm4Kue4N4tGIk5LZsVetYaoSxYz4iTJcKNLTEe iXcBdzY0JhLEuBfO2ecVTFOjHzRyUQRYqbkfqyLURejNuz2SRxCOKaQ7NhlXtdQLR9ro /fY9UMmcaIJNF0ZFn3CyOfHpV+i2wRXRlEMq9uyW4cDRf4vHD1PyiMe4yWiPS1ZDjBnR fqxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NipZie1IRiZT/pOVKr/tyzqG4SEK+qSAp3fYdCy7uIQ=; b=lK1XPgIbm0Bc+DxmRn4Nmq8jDR/cbHOttJ4IvoqUpumm4KUH5z4SD8bh4ArbsRjgiJ 3986rPl2288C9ozpLFmrX2JCrgUDz7Afgzzb/4HkB2hRjIVePQCLymfSStVzexleexe7 erJiXmRyi+/E1sSmm0A9deCnmyLSjCWycDYr8rJfZiQLqMVU+4Xq/JJJsdxlwOcCdtW+ KdZ/NRtPCfU2IhWjcw4cZ+7X1SqVGH1Mql287JOdqCPwHgqgR3DrrI7s21Rt3RWEIBNJ quPd4HI0R/gmRLjU41l2hvgaYjox279nfvYoxDQaFy+OGsNy6u1rZzNrmi7VXGqEssGP G81Q== X-Gm-Message-State: AIVw113wTh54lxMokkG/yu4aN1hrld17HfckbSTuShUE2CGEOeKqvfYn ocH756CVrsq7d6a+l+Q= X-Received: by 10.28.208.78 with SMTP id h75mr1822813wmg.119.1501606522261; Tue, 01 Aug 2017 09:55:22 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id m77sm2309608wmd.21.2017.08.01.09.55.21 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Tue, 01 Aug 2017 09:55:21 -0700 (PDT) From: Adrien Mazarguil To: dev@dpdk.org Cc: Moti Haimovsky Date: Tue, 1 Aug 2017 18:54:09 +0200 Message-Id: <4d34502030061bad72efe0586139ad5df2a2120b.1501598384.git.adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: References: Subject: [dpdk-dev] [PATCH v1 22/48] net/mlx4: revert resource domain support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This reverts commit 3e49c148b715c3c0a12c1200295bb9b312f7028e. Resource domains are not part of the standard Verbs interface. The performance improvement they bring will be restored later through a different data path implementation. This commit makes the PMD not rely on the non-standard QP allocation interface. Signed-off-by: Moti Haimovsky Signed-off-by: Adrien Mazarguil --- drivers/net/mlx4/mlx4.c | 88 ++++----------------------------------- drivers/net/mlx4/mlx4.h | 2 - drivers/net/mlx4/mlx4_flow.c | 30 +++++-------- 3 files changed, 20 insertions(+), 100 deletions(-) diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c index 773ba62..144cfb0 100644 --- a/drivers/net/mlx4/mlx4.c +++ b/drivers/net/mlx4/mlx4.c @@ -697,17 +697,6 @@ txq_cleanup(struct txq *txq) claim_zero(ibv_destroy_qp(txq->qp)); if (txq->cq != NULL) claim_zero(ibv_destroy_cq(txq->cq)); - if (txq->rd != NULL) { - struct ibv_exp_destroy_res_domain_attr attr = { - .comp_mask = 0, - }; - - assert(txq->priv != NULL); - assert(txq->priv->ctx != NULL); - claim_zero(ibv_exp_destroy_res_domain(txq->priv->ctx, - txq->rd, - &attr)); - } for (i = 0; (i != elemof(txq->mp2mr)); ++i) { if (txq->mp2mr[i].mp == NULL) break; @@ -1175,9 +1164,7 @@ txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc, }; union { struct ibv_exp_query_intf_params params; - struct ibv_exp_qp_init_attr init; - struct ibv_exp_res_domain_init_attr rd; - struct ibv_exp_cq_init_attr cq; + struct ibv_qp_init_attr init; struct ibv_qp_attr mod; } attr; enum ibv_exp_query_intf_status status; @@ -1191,24 +1178,7 @@ txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc, return EINVAL; } /* MRs will be registered in mp2mr[] later. */ - attr.rd = (struct ibv_exp_res_domain_init_attr){ - .comp_mask = (IBV_EXP_RES_DOMAIN_THREAD_MODEL | - IBV_EXP_RES_DOMAIN_MSG_MODEL), - .thread_model = IBV_EXP_THREAD_SINGLE, - .msg_model = IBV_EXP_MSG_HIGH_BW, - }; - tmpl.rd = ibv_exp_create_res_domain(priv->ctx, &attr.rd); - if (tmpl.rd == NULL) { - ret = ENOMEM; - ERROR("%p: RD creation failure: %s", - (void *)dev, strerror(ret)); - goto error; - } - attr.cq = (struct ibv_exp_cq_init_attr){ - .comp_mask = IBV_EXP_CQ_INIT_ATTR_RES_DOMAIN, - .res_domain = tmpl.rd, - }; - tmpl.cq = ibv_exp_create_cq(priv->ctx, desc, NULL, NULL, 0, &attr.cq); + tmpl.cq = ibv_create_cq(priv->ctx, desc, NULL, NULL, 0); if (tmpl.cq == NULL) { ret = ENOMEM; ERROR("%p: CQ creation failure: %s", @@ -1219,7 +1189,7 @@ txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc, priv->device_attr.max_qp_wr); DEBUG("priv->device_attr.max_sge is %d", priv->device_attr.max_sge); - attr.init = (struct ibv_exp_qp_init_attr){ + attr.init = (struct ibv_qp_init_attr){ /* CQ to be associated with the send queue. */ .send_cq = tmpl.cq, /* CQ to be associated with the receive queue. */ @@ -1237,12 +1207,8 @@ txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc, /* Do *NOT* enable this, completions events are managed per * TX burst. */ .sq_sig_all = 0, - .pd = priv->pd, - .res_domain = tmpl.rd, - .comp_mask = (IBV_EXP_QP_INIT_ATTR_PD | - IBV_EXP_QP_INIT_ATTR_RES_DOMAIN), }; - tmpl.qp = ibv_exp_create_qp(priv->ctx, &attr.init); + tmpl.qp = ibv_create_qp(priv->pd, &attr.init); if (tmpl.qp == NULL) { ret = (errno ? errno : EINVAL); ERROR("%p: QP creation failure: %s", @@ -1721,17 +1687,6 @@ rxq_cleanup(struct rxq *rxq) claim_zero(ibv_destroy_cq(rxq->cq)); if (rxq->channel != NULL) claim_zero(ibv_destroy_comp_channel(rxq->channel)); - if (rxq->rd != NULL) { - struct ibv_exp_destroy_res_domain_attr attr = { - .comp_mask = 0, - }; - - assert(rxq->priv != NULL); - assert(rxq->priv->ctx != NULL); - claim_zero(ibv_exp_destroy_res_domain(rxq->priv->ctx, - rxq->rd, - &attr)); - } if (rxq->mr != NULL) claim_zero(ibv_dereg_mr(rxq->mr)); memset(rxq, 0, sizeof(*rxq)); @@ -1901,10 +1856,9 @@ mlx4_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) * QP pointer or NULL in case of error. */ static struct ibv_qp * -rxq_setup_qp(struct priv *priv, struct ibv_cq *cq, uint16_t desc, - struct ibv_exp_res_domain *rd) +rxq_setup_qp(struct priv *priv, struct ibv_cq *cq, uint16_t desc) { - struct ibv_exp_qp_init_attr attr = { + struct ibv_qp_init_attr attr = { /* CQ to be associated with the send queue. */ .send_cq = cq, /* CQ to be associated with the receive queue. */ @@ -1918,13 +1872,9 @@ rxq_setup_qp(struct priv *priv, struct ibv_cq *cq, uint16_t desc, .max_recv_sge = 1, }, .qp_type = IBV_QPT_RAW_PACKET, - .comp_mask = (IBV_EXP_QP_INIT_ATTR_PD | - IBV_EXP_QP_INIT_ATTR_RES_DOMAIN), - .pd = priv->pd, - .res_domain = rd, }; - return ibv_exp_create_qp(priv->ctx, &attr); + return ibv_create_qp(priv->pd, &attr); } /** @@ -1960,8 +1910,6 @@ rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, struct ibv_qp_attr mod; union { struct ibv_exp_query_intf_params params; - struct ibv_exp_cq_init_attr cq; - struct ibv_exp_res_domain_init_attr rd; } attr; enum ibv_exp_query_intf_status status; struct ibv_recv_wr *bad_wr; @@ -1999,19 +1947,6 @@ rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, (void *)dev, strerror(ret)); goto error; } - attr.rd = (struct ibv_exp_res_domain_init_attr){ - .comp_mask = (IBV_EXP_RES_DOMAIN_THREAD_MODEL | - IBV_EXP_RES_DOMAIN_MSG_MODEL), - .thread_model = IBV_EXP_THREAD_SINGLE, - .msg_model = IBV_EXP_MSG_HIGH_BW, - }; - tmpl.rd = ibv_exp_create_res_domain(priv->ctx, &attr.rd); - if (tmpl.rd == NULL) { - ret = ENOMEM; - ERROR("%p: RD creation failure: %s", - (void *)dev, strerror(ret)); - goto error; - } if (dev->data->dev_conf.intr_conf.rxq) { tmpl.channel = ibv_create_comp_channel(priv->ctx); if (tmpl.channel == NULL) { @@ -2022,12 +1957,7 @@ rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, goto error; } } - attr.cq = (struct ibv_exp_cq_init_attr){ - .comp_mask = IBV_EXP_CQ_INIT_ATTR_RES_DOMAIN, - .res_domain = tmpl.rd, - }; - tmpl.cq = ibv_exp_create_cq(priv->ctx, desc, NULL, tmpl.channel, 0, - &attr.cq); + tmpl.cq = ibv_create_cq(priv->ctx, desc, NULL, tmpl.channel, 0); if (tmpl.cq == NULL) { ret = ENOMEM; ERROR("%p: CQ creation failure: %s", @@ -2038,7 +1968,7 @@ rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, priv->device_attr.max_qp_wr); DEBUG("priv->device_attr.max_sge is %d", priv->device_attr.max_sge); - tmpl.qp = rxq_setup_qp(priv, tmpl.cq, desc, tmpl.rd); + tmpl.qp = rxq_setup_qp(priv, tmpl.cq, desc); if (tmpl.qp == NULL) { ret = (errno ? errno : EINVAL); ERROR("%p: QP creation failure: %s", diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h index 66efb98..6a5df5c 100644 --- a/drivers/net/mlx4/mlx4.h +++ b/drivers/net/mlx4/mlx4.h @@ -161,7 +161,6 @@ struct rxq { struct rxq_elt (*elts)[]; /* RX elements. */ struct mlx4_rxq_stats stats; /* RX queue counters. */ unsigned int socket; /* CPU socket ID for allocations. */ - struct ibv_exp_res_domain *rd; /* Resource Domain. */ }; /* TX element. */ @@ -198,7 +197,6 @@ struct txq { unsigned int elts_comp_cd_init; /* Initial value for countdown. */ struct mlx4_txq_stats stats; /* TX queue counters. */ unsigned int socket; /* CPU socket ID for allocations. */ - struct ibv_exp_res_domain *rd; /* Resource Domain. */ }; struct rte_flow; diff --git a/drivers/net/mlx4/mlx4_flow.c b/drivers/net/mlx4/mlx4_flow.c index 2c5dc3c..58d4698 100644 --- a/drivers/net/mlx4/mlx4_flow.c +++ b/drivers/net/mlx4/mlx4_flow.c @@ -752,29 +752,21 @@ mlx4_flow_create_drop_queue(struct priv *priv) ERROR("Cannot allocate memory for drop struct"); goto err; } - cq = ibv_exp_create_cq(priv->ctx, 1, NULL, NULL, 0, - &(struct ibv_exp_cq_init_attr){ - .comp_mask = 0, - }); + cq = ibv_create_cq(priv->ctx, 1, NULL, NULL, 0); if (!cq) { ERROR("Cannot create drop CQ"); goto err_create_cq; } - qp = ibv_exp_create_qp(priv->ctx, - &(struct ibv_exp_qp_init_attr){ - .send_cq = cq, - .recv_cq = cq, - .cap = { - .max_recv_wr = 1, - .max_recv_sge = 1, - }, - .qp_type = IBV_QPT_RAW_PACKET, - .comp_mask = - IBV_EXP_QP_INIT_ATTR_PD | - IBV_EXP_QP_INIT_ATTR_PORT, - .pd = priv->pd, - .port_num = priv->port, - }); + qp = ibv_create_qp(priv->pd, + &(struct ibv_qp_init_attr){ + .send_cq = cq, + .recv_cq = cq, + .cap = { + .max_recv_wr = 1, + .max_recv_sge = 1, + }, + .qp_type = IBV_QPT_RAW_PACKET, + }); if (!qp) { ERROR("Cannot create drop QP"); goto err_create_qp;