From patchwork Mon Feb 3 13:32:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: bugzilla@dpdk.org X-Patchwork-Id: 65484 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C0470A052E; Mon, 3 Feb 2020 14:32:06 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 10FFB1BFB5; Mon, 3 Feb 2020 14:32:06 +0100 (CET) Received: from inbox.dpdk.org (xvm-172-178.dc0.ghst.net [95.142.172.178]) by dpdk.org (Postfix) with ESMTP id 080891BFB1 for ; Mon, 3 Feb 2020 14:32:05 +0100 (CET) Received: by inbox.dpdk.org (Postfix, from userid 33) id D186FA0530; Mon, 3 Feb 2020 14:32:04 +0100 (CET) From: bugzilla@dpdk.org To: dev@dpdk.org Date: Mon, 03 Feb 2020 13:32:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: DPDK X-Bugzilla-Component: core X-Bugzilla-Version: 18.11 X-Bugzilla-Keywords: X-Bugzilla-Severity: critical X-Bugzilla-Who: scott_wasson@affirmednetworks.com X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: dev@dpdk.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter target_milestone Message-ID: X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All MIME-Version: 1.0 Subject: [dpdk-dev] [Bug 389] Crash in librte_kni driver due to noncontiguous pages X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" https://bugs.dpdk.org/show_bug.cgi?id=389 Bug ID: 389 Summary: Crash in librte_kni driver due to noncontiguous pages Product: DPDK Version: 18.11 Hardware: All OS: All Status: UNCONFIRMED Severity: critical Priority: Normal Component: core Assignee: dev@dpdk.org Reporter: scott_wasson@affirmednetworks.com Target Milestone: --- We’re seeing a continuous crash since upgrading to 18.11, the kni FIFO’s apparently aren’t contiguous. From user-space’s perspective, the kni’s tx_q straddles the 2MB page boundary at 0x17a600000. The mbuf pointers in the ring prior to this address are valid. The tx_q’s write pointer is indicating there are mbufs at 0x17a600000 and beyond, but the pointers are all NULL. Because the rte_kni kernel module is loaded: In eal.c: /* Workaround for KNI which requires physical address to work */ if (iova_mode == RTE_IOVA_VA && rte_eal_check_module("rte_kni") == 1) { if (phys_addrs) { iova_mode = RTE_IOVA_PA; Iova_mode is automatically forced to PA. We determined that enabling --legacy-mem caused the problem to go away. But this caused the locations of the kni’s data structures to move, so they no longer straddled a hugepages boundary. Our concern is that the furniture may move around again and bring us back to where we were. Being tied to using --legacy-mem is undesirable in the long-term, anyway. We also found that the following code patch helps (even without --legacy-mem): index 3d2ffb2..5cc9d69 100644 snprintf(mz_name, RTE_MEMZONE_NAMESIZE, KNI_RESP_Q_MZ_NAME_FMT, kni->name); - kni->m_resp_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, 0); + kni->m_resp_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, RTE_MEMZONE_IOVA_CONTIG); KNI_MEM_CHECK(kni->m_resp_q == NULL, resp_q_fail); snprintf(mz_name, RTE_MEMZONE_NAMESIZE, KNI_SYNC_ADDR_MZ_NAME_FMT, kni->name); - kni->m_sync_addr = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, 0); + kni->m_sync_addr = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, RTE_MEMZONE_IOVA_CONTIG); KNI_MEM_CHECK(kni->m_sync_addr == NULL, sync_addr_fail); return 0; I removed --legacy-mem, the tx_q still straddles the same 2MB page boundary, yet now everything seems OK. This would seem to follow precedent in rte_mempool.c: /* if we're trying to reserve contiguous memory, add appropriate * memzone flag. */ if (try_contig) flags |= RTE_MEMZONE_IOVA_CONTIG; which I think explains why our mbufs haven’t seen data truncation issues. Could you please why RTE_MEMZONE_IOVA_CONTIG is necessary in PA mode? Isn’t contiguousness a fundamental property of physical addressing? Are we still potentially vulnerable with --legacy-mem and without the above code change? Did we just get lucky because the furniture moved and doesn’t straddle a page boundary at the moment? We also tested with 19.11 and did not see the crash. However the 19.11 release notes say: +* Changed mempool allocation behavior. Changed the mempool allocation behaviour so that objects no longer cross pages by default. Note, this may consume more memory when using small memory pages. Thanks! -Scott --- a/lib/librte_kni/rte_kni.c +++ b/lib/librte_kni/rte_kni.c @@ -143,31 +143,31 @@ kni_reserve_mz(struct rte_kni *kni) char mz_name[RTE_MEMZONE_NAMESIZE]; snprintf(mz_name, RTE_MEMZONE_NAMESIZE, KNI_TX_Q_MZ_NAME_FMT, kni->name); - kni->m_tx_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, 0); + kni->m_tx_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, RTE_MEMZONE_IOVA_CONTIG); KNI_MEM_CHECK(kni->m_tx_q == NULL, tx_q_fail); snprintf(mz_name, RTE_MEMZONE_NAMESIZE, KNI_RX_Q_MZ_NAME_FMT, kni->name); - kni->m_rx_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, 0); + kni->m_rx_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, RTE_MEMZONE_IOVA_CONTIG); KNI_MEM_CHECK(kni->m_rx_q == NULL, rx_q_fail); snprintf(mz_name, RTE_MEMZONE_NAMESIZE, KNI_ALLOC_Q_MZ_NAME_FMT, kni->name); - kni->m_alloc_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, 0); + kni->m_alloc_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, RTE_MEMZONE_IOVA_CONTIG); KNI_MEM_CHECK(kni->m_alloc_q == NULL, alloc_q_fail); snprintf(mz_name, RTE_MEMZONE_NAMESIZE, KNI_FREE_Q_MZ_NAME_FMT, kni->name); - kni->m_free_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, 0); + kni->m_free_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, RTE_MEMZONE_IOVA_CONTIG); KNI_MEM_CHECK(kni->m_free_q == NULL, free_q_fail); snprintf(mz_name, RTE_MEMZONE_NAMESIZE, KNI_REQ_Q_MZ_NAME_FMT, kni->name); - kni->m_req_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, 0); + kni->m_req_q = rte_memzone_reserve(mz_name, KNI_FIFO_SIZE, SOCKET_ID_ANY, RTE_MEMZONE_IOVA_CONTIG); KNI_MEM_CHECK(kni->m_req_q == NULL, req_q_fail);