Patch Detail

HTTP 200 OK
Allow: GET, PUT, PATCH, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "id": 53903,
    "url": "http://patches.dpdk.org/api/patches/53903/?format=api",
    "web_url": "http://patches.dpdk.org/project/dpdk/patch/1559211639-17442-7-git-send-email-matan@mellanox.com/",
    "project": {
        "id": 1,
        "url": "http://patches.dpdk.org/api/projects/1/?format=api",
        "name": "DPDK",
        "link_name": "dpdk",
        "list_id": "dev.dpdk.org",
        "list_email": "dev@dpdk.org",
        "web_url": "http://core.dpdk.org",
        "scm_url": "git://dpdk.org/dpdk",
        "webscm_url": "http://git.dpdk.org/dpdk",
        "list_archive_url": "https://inbox.dpdk.org/dev",
        "list_archive_url_format": "https://inbox.dpdk.org/dev/{}",
        "commit_url_format": ""
    },
    "msgid": "<1559211639-17442-7-git-send-email-matan@mellanox.com>",
    "list_archive_url": "https://inbox.dpdk.org/dev/1559211639-17442-7-git-send-email-matan@mellanox.com",
    "date": "2019-05-30T10:20:36",
    "name": "[v1,6/9] net/mlx5: extend Rx completion with error handling",
    "commit_ref": null,
    "pull_url": null,
    "state": "accepted",
    "archived": true,
    "hash": "ec588356def4921f731ba1ef6fa6ad7d176937af",
    "submitter": {
        "id": 796,
        "url": "http://patches.dpdk.org/api/people/796/?format=api",
        "name": "Matan Azrad",
        "email": "matan@mellanox.com"
    },
    "delegate": {
        "id": 6624,
        "url": "http://patches.dpdk.org/api/users/6624/?format=api",
        "username": "shahafs",
        "first_name": "Shahaf",
        "last_name": "Shuler",
        "email": "shahafs@mellanox.com"
    },
    "mbox": "http://patches.dpdk.org/project/dpdk/patch/1559211639-17442-7-git-send-email-matan@mellanox.com/mbox/",
    "series": [
        {
            "id": 4823,
            "url": "http://patches.dpdk.org/api/series/4823/?format=api",
            "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=4823",
            "date": "2019-05-30T10:20:30",
            "name": "mlx5: Handle data-path completions with error",
            "version": 1,
            "mbox": "http://patches.dpdk.org/series/4823/mbox/"
        }
    ],
    "comments": "http://patches.dpdk.org/api/patches/53903/comments/",
    "check": "fail",
    "checks": "http://patches.dpdk.org/api/patches/53903/checks/",
    "tags": {},
    "related": [],
    "headers": {
        "Return-Path": "<dev-bounces@dpdk.org>",
        "X-Original-To": "patchwork@dpdk.org",
        "Delivered-To": "patchwork@dpdk.org",
        "Received": [
            "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 492921B9C8;\n\tThu, 30 May 2019 12:21:04 +0200 (CEST)",
            "from git-send-mailer.rdmz.labs.mlnx (unknown [37.142.13.130])\n\tby dpdk.org (Postfix) with ESMTP id 494821B953;\n\tThu, 30 May 2019 12:20:45 +0200 (CEST)"
        ],
        "From": "Matan Azrad <matan@mellanox.com>",
        "To": "Shahaf Shuler <shahafs@mellanox.com>, Yongseok Koh <yskoh@mellanox.com>",
        "Cc": "dev@dpdk.org,\n\tstable@dpdk.org",
        "Date": "Thu, 30 May 2019 10:20:36 +0000",
        "Message-Id": "<1559211639-17442-7-git-send-email-matan@mellanox.com>",
        "X-Mailer": "git-send-email 1.8.3.1",
        "In-Reply-To": "<1559211639-17442-1-git-send-email-matan@mellanox.com>",
        "References": "<1559211639-17442-1-git-send-email-matan@mellanox.com>",
        "Subject": "[dpdk-dev] [PATCH v1 6/9] net/mlx5: extend Rx completion with error\n\thandling",
        "X-BeenThere": "dev@dpdk.org",
        "X-Mailman-Version": "2.1.15",
        "Precedence": "list",
        "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
        "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>",
        "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
        "List-Post": "<mailto:dev@dpdk.org>",
        "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
        "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>",
        "Errors-To": "dev-bounces@dpdk.org",
        "Sender": "\"dev\" <dev-bounces@dpdk.org>"
    },
    "content": "When WQEs are posted to the HW to receive packets, the PMD may receive\na completion report with error from the HW, aka error CQE which is\nassociated to a bad WQE.\n\nThe error reason may be bad address, wrong lkey, small buffer size,\netc. that can wrongly be configured by the PMD or by the user.\n\nChecking all the optional mistakes to prevent error CQEs doesn't make\nsense due to performance impacts, moreover, some error CQEs can be\ntriggered because of the packets coming from the wire when the DPDK\napplication has no any control.\n\nMost of the error CQE types change the RQ state to error state what\ncauses all the next received packets to be dropped by the HW and to be\ncompleted with CQE flush error forever.\n\nThe current solution detects these error CQEs and even reports the\nerrors to the user by the statistics error counters but without\nrecovery, so if the RQ inserted to the error state it never moves to\nready state again and all the next packets ever will be dropped.\n\nExtend the error CQEs handling for recovery by moving the state to\nready again, and rearranging all the RQ WQEs and the management\nvariables appropriately.\n\nSometimes the error CQE root cause is very hard to debug and even may\nbe related to some corner cases which are not reproducible easily,\nhence a dump file with debug information will be created for the first\nnumber of error CQEs, this number can be configured by the PMD probe\nparameters.\n\nCc: stable@dpdk.org\n\nSigned-off-by: Matan Azrad <matan@mellanox.com>\n---\n drivers/net/mlx5/mlx5_rxtx.c     | 328 +++++++++++++++++++++++++++------------\n drivers/net/mlx5/mlx5_rxtx.h     | 101 ++++--------\n drivers/net/mlx5/mlx5_rxtx_vec.c |   5 +-\n 3 files changed, 266 insertions(+), 168 deletions(-)",
    "diff": "diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c\nindex aec0185..5369fc1 100644\n--- a/drivers/net/mlx5/mlx5_rxtx.c\n+++ b/drivers/net/mlx5/mlx5_rxtx.c\n@@ -25,6 +25,7 @@\n #include <rte_common.h>\n #include <rte_branch_prediction.h>\n #include <rte_ether.h>\n+#include <rte_cycles.h>\n \n #include \"mlx5.h\"\n #include \"mlx5_utils.h\"\n@@ -444,7 +445,7 @@\n \t\tcq_ci = rxq->cq_ci;\n \t}\n \tcqe = &(*rxq->cqes)[cq_ci & cqe_cnt];\n-\twhile (check_cqe(cqe, cqe_n, cq_ci) == 0) {\n+\twhile (check_cqe(cqe, cqe_n, cq_ci) != MLX5_CQE_STATUS_HW_OWN) {\n \t\tint8_t op_own;\n \t\tunsigned int n;\n \n@@ -1884,6 +1885,130 @@\n }\n \n /**\n+ * Handle a Rx error.\n+ * The function inserts the RQ state to reset when the first error CQE is\n+ * shown, then drains the CQ by the caller function loop. When the CQ is empty,\n+ * it moves the RQ state to ready and initializes the RQ.\n+ * Next CQE identification and error counting are in the caller responsibility.\n+ *\n+ * @param[in] rxq\n+ *   Pointer to RX queue structure.\n+ * @param[in] mbuf_prepare\n+ *   Whether to prepare mbufs for the RQ.\n+ *\n+ * @return\n+ *   -1 in case of recovery error, otherwise the CQE status.\n+ */\n+int\n+mlx5_rx_err_handle(struct mlx5_rxq_data *rxq, uint8_t mbuf_prepare)\n+{\n+\tconst uint16_t cqe_n = 1 << rxq->cqe_n;\n+\tconst uint16_t cqe_mask = cqe_n - 1;\n+\tconst unsigned int wqe_n = 1 << rxq->elts_n;\n+\tstruct mlx5_rxq_ctrl *rxq_ctrl =\n+\t\t\tcontainer_of(rxq, struct mlx5_rxq_ctrl, rxq);\n+\tstruct ibv_wq_attr mod = {\n+\t\t.attr_mask = IBV_WQ_ATTR_STATE,\n+\t};\n+\tunion {\n+\t\tvolatile struct mlx5_cqe *cqe;\n+\t\tvolatile struct mlx5_err_cqe *err_cqe;\n+\t} u = {\n+\t\t.cqe = &(*rxq->cqes)[rxq->cq_ci & cqe_mask],\n+\t};\n+\tint ret;\n+\n+\tswitch (rxq->err_state) {\n+\tcase MLX5_RXQ_ERR_STATE_NO_ERROR:\n+\t\trxq->err_state = MLX5_RXQ_ERR_STATE_NEED_RESET;\n+\t\t/* Fall-through */\n+\tcase MLX5_RXQ_ERR_STATE_NEED_RESET:\n+\t\tif (rte_eal_process_type() != RTE_PROC_PRIMARY)\n+\t\t\treturn -1;\n+\t\tmod.wq_state = IBV_WQS_RESET;\n+\t\tret = mlx5_glue->modify_wq(rxq_ctrl->ibv->wq, &mod);\n+\t\tif (ret) {\n+\t\t\tDRV_LOG(ERR, \"Cannot change Rx WQ state to RESET %s\\n\",\n+\t\t\t\tstrerror(errno));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (rxq_ctrl->dump_file_n <\n+\t\t    rxq_ctrl->priv->config.max_dump_files_num) {\n+\t\t\tMKSTR(err_str, \"Unexpected CQE error syndrome \"\n+\t\t\t      \"0x%02x CQN = %u RQN = %u wqe_counter = %u\"\n+\t\t\t      \" rq_ci = %u cq_ci = %u\", u.err_cqe->syndrome,\n+\t\t\t      rxq->cqn, rxq_ctrl->ibv->wq->wq_num,\n+\t\t\t      rte_be_to_cpu_16(u.err_cqe->wqe_counter),\n+\t\t\t      rxq->rq_ci << rxq->sges_n, rxq->cq_ci);\n+\t\t\tMKSTR(name, \"dpdk_mlx5_port_%u_rxq_%u_%u\",\n+\t\t\t      rxq->port_id, rxq->idx, (uint32_t)rte_rdtsc());\n+\t\t\tmlx5_dump_debug_information(name, NULL, err_str, 0);\n+\t\t\tmlx5_dump_debug_information(name, \"MLX5 Error CQ:\",\n+\t\t\t\t\t\t    (const void *)((uintptr_t)\n+\t\t\t\t\t\t\t\t    rxq->cqes),\n+\t\t\t\t\t\t    sizeof(*u.cqe) * cqe_n);\n+\t\t\tmlx5_dump_debug_information(name, \"MLX5 Error RQ:\",\n+\t\t\t\t\t\t    (const void *)((uintptr_t)\n+\t\t\t\t\t\t\t\t    rxq->wqes),\n+\t\t\t\t\t\t    16 * wqe_n);\n+\t\t\trxq_ctrl->dump_file_n++;\n+\t\t}\n+\t\trxq->err_state = MLX5_RXQ_ERR_STATE_NEED_READY;\n+\t\t/* Fall-through */\n+\tcase MLX5_RXQ_ERR_STATE_NEED_READY:\n+\t\tret = check_cqe(u.cqe, cqe_n, rxq->cq_ci);\n+\t\tif (ret == MLX5_CQE_STATUS_HW_OWN) {\n+\t\t\trte_cio_wmb();\n+\t\t\t*rxq->cq_db = rte_cpu_to_be_32(rxq->cq_ci);\n+\t\t\trte_cio_wmb();\n+\t\t\t/*\n+\t\t\t * The RQ consumer index must be zeroed while moving\n+\t\t\t * from RESET state to RDY state.\n+\t\t\t */\n+\t\t\t*rxq->rq_db = rte_cpu_to_be_32(0);\n+\t\t\trte_cio_wmb();\n+\t\t\tmod.wq_state = IBV_WQS_RDY;\n+\t\t\tret = mlx5_glue->modify_wq(rxq_ctrl->ibv->wq, &mod);\n+\t\t\tif (ret) {\n+\t\t\t\tDRV_LOG(ERR, \"Cannot change Rx WQ state to RDY\"\n+\t\t\t\t\t\" %s\\n\", strerror(errno));\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t\tif (mbuf_prepare) {\n+\t\t\t\tconst uint16_t q_mask = wqe_n - 1;\n+\t\t\t\tuint16_t elt_idx;\n+\t\t\t\tstruct rte_mbuf **elt;\n+\t\t\t\tint i;\n+\t\t\t\tunsigned int n = wqe_n - (rxq->rq_ci -\n+\t\t\t\t\t\t\t  rxq->rq_pi);\n+\n+\t\t\t\tfor (i = 0; i < (int)n; ++i) {\n+\t\t\t\t\telt_idx = (rxq->rq_ci + i) & q_mask;\n+\t\t\t\t\telt = &(*rxq->elts)[elt_idx];\n+\t\t\t\t\t*elt = rte_mbuf_raw_alloc(rxq->mp);\n+\t\t\t\t\tif (!*elt) {\n+\t\t\t\t\t\tfor (i--; i >= 0; --i) {\n+\t\t\t\t\t\t\telt_idx = (rxq->rq_ci +\n+\t\t\t\t\t\t\t\t   i) & q_mask;\n+\t\t\t\t\t\t\telt = &(*rxq->elts)\n+\t\t\t\t\t\t\t\t[elt_idx];\n+\t\t\t\t\t\t\trte_pktmbuf_free_seg\n+\t\t\t\t\t\t\t\t(*elt);\n+\t\t\t\t\t\t}\n+\t\t\t\t\t\treturn -1;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tmlx5_rxq_initialize(rxq);\n+\t\t\trxq->err_state = MLX5_RXQ_ERR_STATE_NO_ERROR;\n+\t\t}\n+\t\treturn ret;\n+\tdefault:\n+\t\treturn -1;\n+\t}\n+}\n+\n+/**\n  * Get size of the next packet for a given CQE. For compressed CQEs, the\n  * consumer index is updated only once all packets of the current one have\n  * been processed.\n@@ -1897,8 +2022,7 @@\n  *   written.\n  *\n  * @return\n- *   Packet size in bytes (0 if there is none), -1 in case of completion\n- *   with error.\n+ *   0 in case of empty CQE, otherwise the packet size in bytes.\n  */\n static inline int\n mlx5_rx_poll_len(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cqe,\n@@ -1906,98 +2030,118 @@\n {\n \tstruct rxq_zip *zip = &rxq->zip;\n \tuint16_t cqe_n = cqe_cnt + 1;\n-\tint len = 0;\n+\tint len;\n \tuint16_t idx, end;\n \n-\t/* Process compressed data in the CQE and mini arrays. */\n-\tif (zip->ai) {\n-\t\tvolatile struct mlx5_mini_cqe8 (*mc)[8] =\n-\t\t\t(volatile struct mlx5_mini_cqe8 (*)[8])\n-\t\t\t(uintptr_t)(&(*rxq->cqes)[zip->ca & cqe_cnt].pkt_info);\n-\n-\t\tlen = rte_be_to_cpu_32((*mc)[zip->ai & 7].byte_cnt);\n-\t\t*mcqe = &(*mc)[zip->ai & 7];\n-\t\tif ((++zip->ai & 7) == 0) {\n-\t\t\t/* Invalidate consumed CQEs */\n-\t\t\tidx = zip->ca;\n-\t\t\tend = zip->na;\n-\t\t\twhile (idx != end) {\n-\t\t\t\t(*rxq->cqes)[idx & cqe_cnt].op_own =\n-\t\t\t\t\tMLX5_CQE_INVALIDATE;\n-\t\t\t\t++idx;\n-\t\t\t}\n-\t\t\t/*\n-\t\t\t * Increment consumer index to skip the number of\n-\t\t\t * CQEs consumed. Hardware leaves holes in the CQ\n-\t\t\t * ring for software use.\n-\t\t\t */\n-\t\t\tzip->ca = zip->na;\n-\t\t\tzip->na += 8;\n-\t\t}\n-\t\tif (unlikely(rxq->zip.ai == rxq->zip.cqe_cnt)) {\n-\t\t\t/* Invalidate the rest */\n-\t\t\tidx = zip->ca;\n-\t\t\tend = zip->cq_ci;\n-\n-\t\t\twhile (idx != end) {\n-\t\t\t\t(*rxq->cqes)[idx & cqe_cnt].op_own =\n-\t\t\t\t\tMLX5_CQE_INVALIDATE;\n-\t\t\t\t++idx;\n-\t\t\t}\n-\t\t\trxq->cq_ci = zip->cq_ci;\n-\t\t\tzip->ai = 0;\n-\t\t}\n-\t/* No compressed data, get next CQE and verify if it is compressed. */\n-\t} else {\n-\t\tint ret;\n-\t\tint8_t op_own;\n-\n-\t\tret = check_cqe(cqe, cqe_n, rxq->cq_ci);\n-\t\tif (unlikely(ret == 1))\n-\t\t\treturn 0;\n-\t\t++rxq->cq_ci;\n-\t\top_own = cqe->op_own;\n-\t\trte_cio_rmb();\n-\t\tif (MLX5_CQE_FORMAT(op_own) == MLX5_COMPRESSED) {\n+\tdo {\n+\t\tlen = 0;\n+\t\t/* Process compressed data in the CQE and mini arrays. */\n+\t\tif (zip->ai) {\n \t\t\tvolatile struct mlx5_mini_cqe8 (*mc)[8] =\n \t\t\t\t(volatile struct mlx5_mini_cqe8 (*)[8])\n-\t\t\t\t(uintptr_t)(&(*rxq->cqes)[rxq->cq_ci &\n+\t\t\t\t(uintptr_t)(&(*rxq->cqes)[zip->ca &\n \t\t\t\t\t\t\t  cqe_cnt].pkt_info);\n \n-\t\t\t/* Fix endianness. */\n-\t\t\tzip->cqe_cnt = rte_be_to_cpu_32(cqe->byte_cnt);\n-\t\t\t/*\n-\t\t\t * Current mini array position is the one returned by\n-\t\t\t * check_cqe64().\n-\t\t\t *\n-\t\t\t * If completion comprises several mini arrays, as a\n-\t\t\t * special case the second one is located 7 CQEs after\n-\t\t\t * the initial CQE instead of 8 for subsequent ones.\n-\t\t\t */\n-\t\t\tzip->ca = rxq->cq_ci;\n-\t\t\tzip->na = zip->ca + 7;\n-\t\t\t/* Compute the next non compressed CQE. */\n-\t\t\t--rxq->cq_ci;\n-\t\t\tzip->cq_ci = rxq->cq_ci + zip->cqe_cnt;\n-\t\t\t/* Get packet size to return. */\n-\t\t\tlen = rte_be_to_cpu_32((*mc)[0].byte_cnt);\n-\t\t\t*mcqe = &(*mc)[0];\n-\t\t\tzip->ai = 1;\n-\t\t\t/* Prefetch all the entries to be invalidated */\n-\t\t\tidx = zip->ca;\n-\t\t\tend = zip->cq_ci;\n-\t\t\twhile (idx != end) {\n-\t\t\t\trte_prefetch0(&(*rxq->cqes)[(idx) & cqe_cnt]);\n-\t\t\t\t++idx;\n+\t\t\tlen = rte_be_to_cpu_32((*mc)[zip->ai & 7].byte_cnt);\n+\t\t\t*mcqe = &(*mc)[zip->ai & 7];\n+\t\t\tif ((++zip->ai & 7) == 0) {\n+\t\t\t\t/* Invalidate consumed CQEs */\n+\t\t\t\tidx = zip->ca;\n+\t\t\t\tend = zip->na;\n+\t\t\t\twhile (idx != end) {\n+\t\t\t\t\t(*rxq->cqes)[idx & cqe_cnt].op_own =\n+\t\t\t\t\t\tMLX5_CQE_INVALIDATE;\n+\t\t\t\t\t++idx;\n+\t\t\t\t}\n+\t\t\t\t/*\n+\t\t\t\t * Increment consumer index to skip the number\n+\t\t\t\t * of CQEs consumed. Hardware leaves holes in\n+\t\t\t\t * the CQ ring for software use.\n+\t\t\t\t */\n+\t\t\t\tzip->ca = zip->na;\n+\t\t\t\tzip->na += 8;\n+\t\t\t}\n+\t\t\tif (unlikely(rxq->zip.ai == rxq->zip.cqe_cnt)) {\n+\t\t\t\t/* Invalidate the rest */\n+\t\t\t\tidx = zip->ca;\n+\t\t\t\tend = zip->cq_ci;\n+\n+\t\t\t\twhile (idx != end) {\n+\t\t\t\t\t(*rxq->cqes)[idx & cqe_cnt].op_own =\n+\t\t\t\t\t\tMLX5_CQE_INVALIDATE;\n+\t\t\t\t\t++idx;\n+\t\t\t\t}\n+\t\t\t\trxq->cq_ci = zip->cq_ci;\n+\t\t\t\tzip->ai = 0;\n+\t\t\t}\n+\t\t/*\n+\t\t * No compressed data, get next CQE and verify if it is\n+\t\t * compressed.\n+\t\t */\n+\t\t} else {\n+\t\t\tint ret;\n+\t\t\tint8_t op_own;\n+\n+\t\t\tret = check_cqe(cqe, cqe_n, rxq->cq_ci);\n+\t\t\tif (unlikely(ret != MLX5_CQE_STATUS_SW_OWN)) {\n+\t\t\t\tif (unlikely(ret == MLX5_CQE_STATUS_ERR ||\n+\t\t\t\t\t     rxq->err_state)) {\n+\t\t\t\t\tret = mlx5_rx_err_handle(rxq, 0);\n+\t\t\t\t\tif (ret == MLX5_CQE_STATUS_HW_OWN ||\n+\t\t\t\t\t    ret == -1)\n+\t\t\t\t\t\treturn 0;\n+\t\t\t\t} else {\n+\t\t\t\t\treturn 0;\n+\t\t\t\t}\n \t\t\t}\n+\t\t\t++rxq->cq_ci;\n+\t\t\top_own = cqe->op_own;\n+\t\t\tif (MLX5_CQE_FORMAT(op_own) == MLX5_COMPRESSED) {\n+\t\t\t\tvolatile struct mlx5_mini_cqe8 (*mc)[8] =\n+\t\t\t\t\t(volatile struct mlx5_mini_cqe8 (*)[8])\n+\t\t\t\t\t(uintptr_t)(&(*rxq->cqes)\n+\t\t\t\t\t\t[rxq->cq_ci &\n+\t\t\t\t\t\t cqe_cnt].pkt_info);\n+\n+\t\t\t\t/* Fix endianness. */\n+\t\t\t\tzip->cqe_cnt = rte_be_to_cpu_32(cqe->byte_cnt);\n+\t\t\t\t/*\n+\t\t\t\t * Current mini array position is the one\n+\t\t\t\t * returned by check_cqe64().\n+\t\t\t\t *\n+\t\t\t\t * If completion comprises several mini arrays,\n+\t\t\t\t * as a special case the second one is located\n+\t\t\t\t * 7 CQEs after the initial CQE instead of 8\n+\t\t\t\t * for subsequent ones.\n+\t\t\t\t */\n+\t\t\t\tzip->ca = rxq->cq_ci;\n+\t\t\t\tzip->na = zip->ca + 7;\n+\t\t\t\t/* Compute the next non compressed CQE. */\n+\t\t\t\t--rxq->cq_ci;\n+\t\t\t\tzip->cq_ci = rxq->cq_ci + zip->cqe_cnt;\n+\t\t\t\t/* Get packet size to return. */\n+\t\t\t\tlen = rte_be_to_cpu_32((*mc)[0].byte_cnt);\n+\t\t\t\t*mcqe = &(*mc)[0];\n+\t\t\t\tzip->ai = 1;\n+\t\t\t\t/* Prefetch all to be invalidated */\n+\t\t\t\tidx = zip->ca;\n+\t\t\t\tend = zip->cq_ci;\n+\t\t\t\twhile (idx != end) {\n+\t\t\t\t\trte_prefetch0(&(*rxq->cqes)[(idx) &\n+\t\t\t\t\t\t\t\t    cqe_cnt]);\n+\t\t\t\t\t++idx;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tlen = rte_be_to_cpu_32(cqe->byte_cnt);\n+\t\t\t}\n+\t\t}\n+\t\tif (unlikely(rxq->err_state)) {\n+\t\t\tcqe = &(*rxq->cqes)[rxq->cq_ci & cqe_cnt];\n+\t\t\t++rxq->stats.idropped;\n \t\t} else {\n-\t\t\tlen = rte_be_to_cpu_32(cqe->byte_cnt);\n+\t\t\treturn len;\n \t\t}\n-\t\t/* Error while receiving packet. */\n-\t\tif (unlikely(MLX5_CQE_OPCODE(op_own) == MLX5_CQE_RESP_ERR))\n-\t\t\treturn -1;\n-\t}\n-\treturn len;\n+\t} while (1);\n }\n \n /**\n@@ -2140,12 +2284,6 @@\n \t\t\t\trte_mbuf_raw_free(rep);\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\tif (unlikely(len == -1)) {\n-\t\t\t\t/* RX error, packet is likely too large. */\n-\t\t\t\trte_mbuf_raw_free(rep);\n-\t\t\t\t++rxq->stats.idropped;\n-\t\t\t\tgoto skip;\n-\t\t\t}\n \t\t\tpkt = seg;\n \t\t\tassert(len >= (rxq->crc_present << 2));\n \t\t\tpkt->ol_flags = 0;\n@@ -2188,7 +2326,6 @@\n \t\tpkt = NULL;\n \t\t--pkts_n;\n \t\t++i;\n-skip:\n \t\t/* Align consumer index to the next stride. */\n \t\trq_ci >>= sges_n;\n \t\t++rq_ci;\n@@ -2321,11 +2458,6 @@\n \t\tret = mlx5_rx_poll_len(rxq, cqe, cq_mask, &mcqe);\n \t\tif (!ret)\n \t\t\tbreak;\n-\t\tif (unlikely(ret == -1)) {\n-\t\t\t/* RX error, packet is likely too large. */\n-\t\t\t++rxq->stats.idropped;\n-\t\t\tcontinue;\n-\t\t}\n \t\tbyte_cnt = ret;\n \t\tstrd_cnt = (byte_cnt & MLX5_MPRQ_STRIDE_NUM_MASK) >>\n \t\t\t   MLX5_MPRQ_STRIDE_NUM_SHIFT;\ndiff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h\nindex 35e53fc..d944fbe 100644\n--- a/drivers/net/mlx5/mlx5_rxtx.h\n+++ b/drivers/net/mlx5/mlx5_rxtx.h\n@@ -36,6 +36,7 @@\n #include \"mlx5_autoconf.h\"\n #include \"mlx5_defs.h\"\n #include \"mlx5_prm.h\"\n+#include \"mlx5_glue.h\"\n \n /* Support tunnel matching. */\n #define MLX5_FLOW_TUNNEL 5\n@@ -78,6 +79,12 @@ struct mlx5_mprq_buf {\n /* Get pointer to the first stride. */\n #define mlx5_mprq_buf_addr(ptr) ((ptr) + 1)\n \n+enum mlx5_rxq_err_state {\n+\tMLX5_RXQ_ERR_STATE_NO_ERROR = 0,\n+\tMLX5_RXQ_ERR_STATE_NEED_RESET,\n+\tMLX5_RXQ_ERR_STATE_NEED_READY,\n+};\n+\n /* RX queue descriptor. */\n struct mlx5_rxq_data {\n \tunsigned int csum:1; /* Enable checksum offloading. */\n@@ -92,7 +99,8 @@ struct mlx5_rxq_data {\n \tunsigned int strd_num_n:5; /* Log 2 of the number of stride. */\n \tunsigned int strd_sz_n:4; /* Log 2 of stride size. */\n \tunsigned int strd_shift_en:1; /* Enable 2bytes shift on a stride. */\n-\tunsigned int :6; /* Remaining bits. */\n+\tunsigned int err_state:2; /* enum mlx5_rxq_err_state. */\n+\tunsigned int :4; /* Remaining bits. */\n \tvolatile uint32_t *rq_db;\n \tvolatile uint32_t *cq_db;\n \tuint16_t port_id;\n@@ -153,6 +161,7 @@ struct mlx5_rxq_ctrl {\n \tunsigned int irq:1; /* Whether IRQ is enabled. */\n \tuint32_t flow_mark_n; /* Number of Mark/Flag flows using this Queue. */\n \tuint32_t flow_tunnels_n[MLX5_FLOW_TUNNEL]; /* Tunnels counters. */\n+\tuint16_t dump_file_n; /* Number of dump files. */\n };\n \n /* Indirection table. */\n@@ -345,6 +354,9 @@ uint16_t mlx5_tx_burst_mpw_inline(void *dpdk_txq, struct rte_mbuf **pkts,\n uint16_t mlx5_tx_burst_empw(void *dpdk_txq, struct rte_mbuf **pkts,\n \t\t\t    uint16_t pkts_n);\n uint16_t mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n);\n+void mlx5_rxq_initialize(struct mlx5_rxq_data *rxq);\n+__rte_noinline int mlx5_rx_err_handle(struct mlx5_rxq_data *rxq,\n+\t\t\t\t      uint8_t mbuf_prepare);\n void mlx5_mprq_buf_free_cb(void *addr, void *opaque);\n void mlx5_mprq_buf_free(struct mlx5_mprq_buf *buf);\n uint16_t mlx5_rx_burst_mprq(void *dpdk_rxq, struct rte_mbuf **pkts,\n@@ -439,32 +451,12 @@ int mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, uint64_t iova,\n #define mlx5_uar_write64(val, dst, lock) __mlx5_uar_write64(val, dst, lock)\n #endif\n \n-#ifndef NDEBUG\n-/**\n- * Verify or set magic value in CQE.\n- *\n- * @param cqe\n- *   Pointer to CQE.\n- *\n- * @return\n- *   0 the first time.\n- */\n-static inline int\n-check_cqe_seen(volatile struct mlx5_cqe *cqe)\n-{\n-\tstatic const uint8_t magic[] = \"seen\";\n-\tvolatile uint8_t (*buf)[sizeof(cqe->rsvd1)] = &cqe->rsvd1;\n-\tint ret = 1;\n-\tunsigned int i;\n-\n-\tfor (i = 0; i < sizeof(magic) && i < sizeof(*buf); ++i)\n-\t\tif (!ret || (*buf)[i] != magic[i]) {\n-\t\t\tret = 0;\n-\t\t\t(*buf)[i] = magic[i];\n-\t\t}\n-\treturn ret;\n-}\n-#endif /* NDEBUG */\n+/* CQE status. */\n+enum mlx5_cqe_status {\n+\tMLX5_CQE_STATUS_SW_OWN,\n+\tMLX5_CQE_STATUS_HW_OWN,\n+\tMLX5_CQE_STATUS_ERR,\n+};\n \n /**\n  * Check whether CQE is valid.\n@@ -477,51 +469,24 @@ int mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, uint64_t iova,\n  *   Consumer index.\n  *\n  * @return\n- *   0 on success, 1 on failure.\n+ *   The CQE status.\n  */\n-static __rte_always_inline int\n-check_cqe(volatile struct mlx5_cqe *cqe,\n-\t  unsigned int cqes_n, const uint16_t ci)\n+static __rte_always_inline enum mlx5_cqe_status\n+check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,\n+\t  const uint16_t ci)\n {\n-\tuint16_t idx = ci & cqes_n;\n-\tuint8_t op_own = cqe->op_own;\n-\tuint8_t op_owner = MLX5_CQE_OWNER(op_own);\n-\tuint8_t op_code = MLX5_CQE_OPCODE(op_own);\n+\tconst uint16_t idx = ci & cqes_n;\n+\tconst uint8_t op_own = cqe->op_own;\n+\tconst uint8_t op_owner = MLX5_CQE_OWNER(op_own);\n+\tconst uint8_t op_code = MLX5_CQE_OPCODE(op_own);\n \n \tif (unlikely((op_owner != (!!(idx))) || (op_code == MLX5_CQE_INVALID)))\n-\t\treturn 1; /* No CQE. */\n-#ifndef NDEBUG\n-\tif ((op_code == MLX5_CQE_RESP_ERR) ||\n-\t    (op_code == MLX5_CQE_REQ_ERR)) {\n-\t\tvolatile struct mlx5_err_cqe *err_cqe = (volatile void *)cqe;\n-\t\tuint8_t syndrome = err_cqe->syndrome;\n-\n-\t\tif ((syndrome == MLX5_CQE_SYNDROME_LOCAL_LENGTH_ERR) ||\n-\t\t    (syndrome == MLX5_CQE_SYNDROME_REMOTE_ABORTED_ERR))\n-\t\t\treturn 0;\n-\t\tif (!check_cqe_seen(cqe)) {\n-\t\t\tDRV_LOG(ERR,\n-\t\t\t\t\"unexpected CQE error %u (0x%02x) syndrome\"\n-\t\t\t\t\" 0x%02x\",\n-\t\t\t\top_code, op_code, syndrome);\n-\t\t\trte_hexdump(stderr, \"MLX5 Error CQE:\",\n-\t\t\t\t    (const void *)((uintptr_t)err_cqe),\n-\t\t\t\t    sizeof(*cqe));\n-\t\t}\n-\t\treturn 1;\n-\t} else if ((op_code != MLX5_CQE_RESP_SEND) &&\n-\t\t   (op_code != MLX5_CQE_REQ)) {\n-\t\tif (!check_cqe_seen(cqe)) {\n-\t\t\tDRV_LOG(ERR, \"unexpected CQE opcode %u (0x%02x)\",\n-\t\t\t\top_code, op_code);\n-\t\t\trte_hexdump(stderr, \"MLX5 CQE:\",\n-\t\t\t\t    (const void *)((uintptr_t)cqe),\n-\t\t\t\t    sizeof(*cqe));\n-\t\t}\n-\t\treturn 1;\n-\t}\n-#endif /* NDEBUG */\n-\treturn 0;\n+\t\treturn MLX5_CQE_STATUS_HW_OWN;\n+\trte_cio_rmb();\n+\tif (unlikely(op_code == MLX5_CQE_RESP_ERR ||\n+\t\t     op_code == MLX5_CQE_REQ_ERR))\n+\t\treturn MLX5_CQE_STATUS_ERR;\n+\treturn MLX5_CQE_STATUS_SW_OWN;\n }\n \n /**\ndiff --git a/drivers/net/mlx5/mlx5_rxtx_vec.c b/drivers/net/mlx5/mlx5_rxtx_vec.c\nindex 9a3a5ae..073044f 100644\n--- a/drivers/net/mlx5/mlx5_rxtx_vec.c\n+++ b/drivers/net/mlx5/mlx5_rxtx_vec.c\n@@ -197,7 +197,7 @@\n \tfor (i = 0; i < pkts_n; ++i) {\n \t\tstruct rte_mbuf *pkt = pkts[i];\n \n-\t\tif (pkt->packet_type == RTE_PTYPE_ALL_MASK) {\n+\t\tif (pkt->packet_type == RTE_PTYPE_ALL_MASK || rxq->err_state) {\n #ifdef MLX5_PMD_SOFT_COUNTERS\n \t\t\terr_bytes += PKT_LEN(pkt);\n #endif\n@@ -212,6 +212,7 @@\n \trxq->stats.ipackets -= (pkts_n - n);\n \trxq->stats.ibytes -= err_bytes;\n #endif\n+\tmlx5_rx_err_handle(rxq, 1);\n \treturn n;\n }\n \n@@ -236,7 +237,7 @@\n \tuint64_t err = 0;\n \n \tnb_rx = rxq_burst_v(rxq, pkts, pkts_n, &err);\n-\tif (unlikely(err))\n+\tif (unlikely(err | rxq->err_state))\n \t\tnb_rx = rxq_handle_pending_error(rxq, pkts, nb_rx);\n \treturn nb_rx;\n }\n",
    "prefixes": [
        "v1",
        "6/9"
    ]
}