Message ID | 20190816061252.17214-1-vattunuru@marvell.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 30B891BEC6; Fri, 16 Aug 2019 08:13:19 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id ECD501BE8E for <dev@dpdk.org>; Fri, 16 Aug 2019 08:13:17 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x7G6AF8a008656; Thu, 15 Aug 2019 23:13:16 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0818; bh=jL1jj2ojdsTH0/bfy61qMqm2t5fxX7BkMD7po5iRSE0=; b=Xbv5i3cMCygeOZDQMDwRgNFWEadvp47SoMwFIrQ+ChP/PbAk1FX294DGoiVYvoIe++q0 jbviBxbAXl8NxUJt4m3AiQhtfAU6n+U656PrbBvdwxyLoxfvzvJ2KY87T5tqZevuZk3R An6d07F77WJescwNxce4oTH0SdgUgKuOm+xo8gEIUIfVDTRHKpZ8HjSzK/46rTMdwV0w l/cqnqUAB4uLjApyQvt1UTToSgkkOKz/4kF47e4cNyK09L2wr26IjOir+ey1Jrg12Dgt 4JlHhvajnCE3Vu/6kD2NKeGqqUllsqkmVpAKwlFSBL9x8WvkZhy089L0nh6bHFCPG01w Rg== Received: from sc-exch01.marvell.com ([199.233.58.181]) by mx0b-0016f401.pphosted.com with ESMTP id 2udefw1md4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 15 Aug 2019 23:13:16 -0700 Received: from SC-EXCH03.marvell.com (10.93.176.83) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Thu, 15 Aug 2019 23:13:14 -0700 Received: from maili.marvell.com (10.93.176.43) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Thu, 15 Aug 2019 23:13:14 -0700 Received: from hyd1vattunuru-dt.caveonetworks.com (unknown [10.29.52.72]) by maili.marvell.com (Postfix) with ESMTP id B06413F7041; Thu, 15 Aug 2019 23:13:11 -0700 (PDT) From: <vattunuru@marvell.com> To: <dev@dpdk.org> CC: <thomas@monjalon.net>, <jerinj@marvell.com>, <olivier.matz@6wind.com>, <ferruh.yigit@intel.com>, <anatoly.burakov@intel.com>, <arybchenko@solarflare.com>, <kirankumark@marvell.com>, Vamsi Attunuru <vattunuru@marvell.com> Date: Fri, 16 Aug 2019 11:42:47 +0530 Message-ID: <20190816061252.17214-1-vattunuru@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20190729121313.30639-2-vattunuru@marvell.com> References: <20190729121313.30639-2-vattunuru@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:5.22.84,1.0.8 definitions=2019-08-16_03:2019-08-14,2019-08-16 signatures=0 Subject: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
kni: add IOVA=VA support
|
|
Message
Vamsi Krishna Attunuru
Aug. 16, 2019, 6:12 a.m. UTC
From: Vamsi Attunuru <vattunuru@marvell.com>
---
V10 Changes:
* Fixed function return code on failure when min_chunk_size > pg_sz.
* Marked new mempool populate routine as EXPERIMENTAL.
V9 Changes:
* Used rte_mempool_ops_calc_mem_size() instead of default handler in
the new mempool populate routine.
* Check min_chunk_size and return values.
* Removed ethdev_info memset to '0' and moved pci dev_info populate into
kni_dev_pci_addr_get() routine.
* Addressed misc. review comments.
V8 Changes:
* Remove default mempool populate() routine changes.
* Add kni app specific mempool create & free routines.
* Add new mempool populate routine to allocate page-aligned memzones
with page size to make sure all mempool objects reside on a page.
* Update release notes and map files.
V7 Changes:
* Removed previously proposed mempool flag and made those page boundary
checks default in mempool populate() except for the objects size bigger
than the size of page.
* Removed KNI example application related changes since pool related
requirement is taken care in mempool lib.
* All PCI dev related info is moved under rte_eal_iova_mode() == VA check.
* Added wrapper functions in KNI module to hide IOVA checks and make
address translation routines more readable.
* Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA mode
is enabled.
V6 Changes:
* Added new mempool flag to ensure mbuf memory is not scattered
across page boundaries.
* Added KNI kernel module required PCI device information.
* Modified KNI example application to create mempool with new
mempool flag.
V5 changes:
* Fixed build issue with 32b build
V4 changes:
* Fixed build issues with older kernel versions
* This approach will only work with kernel above 4.4.0
V3 Changes:
* Add new approach to work kni with IOVA=VA mode using
iommu_iova_to_phys API.
Kiran Kumar K (1):
kni: add IOVA=VA support in KNI module
Vamsi Attunuru (4):
mempool: populate mempool with the page sized chunks
kni: add IOVA=VA support in KNI lib
kni: add app specific mempool create and free routines
kni: modify IOVA mode checks to support VA
doc/guides/prog_guide/kernel_nic_interface.rst | 8 ++
doc/guides/rel_notes/release_19_11.rst | 5 ++
examples/kni/main.c | 5 +-
kernel/linux/kni/compat.h | 4 +
kernel/linux/kni/kni_dev.h | 4 +
kernel/linux/kni/kni_misc.c | 71 ++++++++++++++---
kernel/linux/kni/kni_net.c | 59 ++++++++++----
lib/librte_eal/linux/eal/eal.c | 4 +-
lib/librte_eal/linux/eal/include/rte_kni_common.h | 8 ++
lib/librte_kni/Makefile | 2 +
lib/librte_kni/meson.build | 2 +
lib/librte_kni/rte_kni.c | 95 +++++++++++++++++++++--
lib/librte_kni/rte_kni.h | 48 ++++++++++++
lib/librte_kni/rte_kni_version.map | 2 +
lib/librte_mempool/rte_mempool.c | 69 ++++++++++++++++
lib/librte_mempool/rte_mempool.h | 20 +++++
lib/librte_mempool/rte_mempool_version.map | 1 +
17 files changed, 378 insertions(+), 29 deletions(-)
Comments
PING. -----Original Message----- From: vattunuru@marvell.com <vattunuru@marvell.com> Sent: Friday, August 16, 2019 11:43 AM To: dev@dpdk.org Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; olivier.matz@6wind.com; ferruh.yigit@intel.com; anatoly.burakov@intel.com; arybchenko@solarflare.com; Kiran Kumar Kokkilagadda <kirankumark@marvell.com>; Vamsi Krishna Attunuru <vattunuru@marvell.com> Subject: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support From: Vamsi Attunuru <vattunuru@marvell.com> --- V10 Changes: * Fixed function return code on failure when min_chunk_size > pg_sz. * Marked new mempool populate routine as EXPERIMENTAL. V9 Changes: * Used rte_mempool_ops_calc_mem_size() instead of default handler in the new mempool populate routine. * Check min_chunk_size and return values. * Removed ethdev_info memset to '0' and moved pci dev_info populate into kni_dev_pci_addr_get() routine. * Addressed misc. review comments. V8 Changes: * Remove default mempool populate() routine changes. * Add kni app specific mempool create & free routines. * Add new mempool populate routine to allocate page-aligned memzones with page size to make sure all mempool objects reside on a page. * Update release notes and map files. V7 Changes: * Removed previously proposed mempool flag and made those page boundary checks default in mempool populate() except for the objects size bigger than the size of page. * Removed KNI example application related changes since pool related requirement is taken care in mempool lib. * All PCI dev related info is moved under rte_eal_iova_mode() == VA check. * Added wrapper functions in KNI module to hide IOVA checks and make address translation routines more readable. * Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA mode is enabled. V6 Changes: * Added new mempool flag to ensure mbuf memory is not scattered across page boundaries. * Added KNI kernel module required PCI device information. * Modified KNI example application to create mempool with new mempool flag. V5 changes: * Fixed build issue with 32b build V4 changes: * Fixed build issues with older kernel versions * This approach will only work with kernel above 4.4.0 V3 Changes: * Add new approach to work kni with IOVA=VA mode using iommu_iova_to_phys API. Kiran Kumar K (1): kni: add IOVA=VA support in KNI module Vamsi Attunuru (4): mempool: populate mempool with the page sized chunks kni: add IOVA=VA support in KNI lib kni: add app specific mempool create and free routines kni: modify IOVA mode checks to support VA doc/guides/prog_guide/kernel_nic_interface.rst | 8 ++ doc/guides/rel_notes/release_19_11.rst | 5 ++ examples/kni/main.c | 5 +- kernel/linux/kni/compat.h | 4 + kernel/linux/kni/kni_dev.h | 4 + kernel/linux/kni/kni_misc.c | 71 ++++++++++++++--- kernel/linux/kni/kni_net.c | 59 ++++++++++---- lib/librte_eal/linux/eal/eal.c | 4 +- lib/librte_eal/linux/eal/include/rte_kni_common.h | 8 ++ lib/librte_kni/Makefile | 2 + lib/librte_kni/meson.build | 2 + lib/librte_kni/rte_kni.c | 95 +++++++++++++++++++++-- lib/librte_kni/rte_kni.h | 48 ++++++++++++ lib/librte_kni/rte_kni_version.map | 2 + lib/librte_mempool/rte_mempool.c | 69 ++++++++++++++++ lib/librte_mempool/rte_mempool.h | 20 +++++ lib/librte_mempool/rte_mempool_version.map | 1 + 17 files changed, 378 insertions(+), 29 deletions(-) -- 2.8.4
@All, we are expecting to merge this in 19.11 release and if any one have comments please respond. > -----Original Message----- > From: Vamsi Krishna Attunuru > Sent: Wednesday, September 25, 2019 9:30 AM > To: vattunuru@marvell.com; dev@dpdk.org > Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; > olivier.matz@6wind.com; ferruh.yigit@intel.com; > anatoly.burakov@intel.com; arybchenko@solarflare.com; Kiran Kumar > Kokkilagadda <kirankumark@marvell.com> > Subject: RE: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support > > PING. > > -----Original Message----- > From: vattunuru@marvell.com <vattunuru@marvell.com> > Sent: Friday, August 16, 2019 11:43 AM > To: dev@dpdk.org > Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; > olivier.matz@6wind.com; ferruh.yigit@intel.com; > anatoly.burakov@intel.com; arybchenko@solarflare.com; Kiran Kumar > Kokkilagadda <kirankumark@marvell.com>; Vamsi Krishna Attunuru > <vattunuru@marvell.com> > Subject: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support > > From: Vamsi Attunuru <vattunuru@marvell.com> > > --- > V10 Changes: > * Fixed function return code on failure when min_chunk_size > pg_sz. > * Marked new mempool populate routine as EXPERIMENTAL. > > V9 Changes: > * Used rte_mempool_ops_calc_mem_size() instead of default handler in the > new mempool populate routine. > * Check min_chunk_size and return values. > * Removed ethdev_info memset to '0' and moved pci dev_info populate into > kni_dev_pci_addr_get() routine. > * Addressed misc. review comments. > > V8 Changes: > * Remove default mempool populate() routine changes. > * Add kni app specific mempool create & free routines. > * Add new mempool populate routine to allocate page-aligned memzones > with page size to make sure all mempool objects reside on a page. > * Update release notes and map files. > > V7 Changes: > * Removed previously proposed mempool flag and made those page > boundary checks default in mempool populate() except for the objects size > bigger than the size of page. > * Removed KNI example application related changes since pool related > requirement is taken care in mempool lib. > * All PCI dev related info is moved under rte_eal_iova_mode() == VA check. > * Added wrapper functions in KNI module to hide IOVA checks and make > address translation routines more readable. > * Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA > mode is enabled. > > V6 Changes: > * Added new mempool flag to ensure mbuf memory is not scattered across > page boundaries. > * Added KNI kernel module required PCI device information. > * Modified KNI example application to create mempool with new mempool > flag. > > V5 changes: > * Fixed build issue with 32b build > > V4 changes: > * Fixed build issues with older kernel versions > * This approach will only work with kernel above 4.4.0 > > V3 Changes: > * Add new approach to work kni with IOVA=VA mode using > iommu_iova_to_phys API. > > Kiran Kumar K (1): > kni: add IOVA=VA support in KNI module > > Vamsi Attunuru (4): > mempool: populate mempool with the page sized chunks > kni: add IOVA=VA support in KNI lib > kni: add app specific mempool create and free routines > kni: modify IOVA mode checks to support VA > > doc/guides/prog_guide/kernel_nic_interface.rst | 8 ++ > doc/guides/rel_notes/release_19_11.rst | 5 ++ > examples/kni/main.c | 5 +- > kernel/linux/kni/compat.h | 4 + > kernel/linux/kni/kni_dev.h | 4 + > kernel/linux/kni/kni_misc.c | 71 ++++++++++++++--- > kernel/linux/kni/kni_net.c | 59 ++++++++++---- > lib/librte_eal/linux/eal/eal.c | 4 +- > lib/librte_eal/linux/eal/include/rte_kni_common.h | 8 ++ > lib/librte_kni/Makefile | 2 + > lib/librte_kni/meson.build | 2 + > lib/librte_kni/rte_kni.c | 95 +++++++++++++++++++++-- > lib/librte_kni/rte_kni.h | 48 ++++++++++++ > lib/librte_kni/rte_kni_version.map | 2 + > lib/librte_mempool/rte_mempool.c | 69 ++++++++++++++++ > lib/librte_mempool/rte_mempool.h | 20 +++++ > lib/librte_mempool/rte_mempool_version.map | 1 + > 17 files changed, 378 insertions(+), 29 deletions(-) > > -- > 2.8.4
Hi Ferruh, Mempool related patch has been acked by Olivier. Could you please review the kni related patches and make sure that it will be merged in 19.11 release. Regards, Vamsi > -----Original Message----- > From: Vamsi Krishna Attunuru <vattunuru@marvell.com> > Sent: Tuesday, October 8, 2019 10:39 AM > To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org > Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; > olivier.matz@6wind.com; ferruh.yigit@intel.com; anatoly.burakov@intel.com; > arybchenko@solarflare.com; Kiran Kumar Kokkilagadda > <kirankumark@marvell.com> > Subject: RE: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support > > @All, we are expecting to merge this in 19.11 release and if any one have > comments please respond. > > > -----Original Message----- > > From: Vamsi Krishna Attunuru > > Sent: Wednesday, September 25, 2019 9:30 AM > > To: vattunuru@marvell.com; dev@dpdk.org > > Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran > > <jerinj@marvell.com>; olivier.matz@6wind.com; ferruh.yigit@intel.com; > > anatoly.burakov@intel.com; arybchenko@solarflare.com; Kiran Kumar > > Kokkilagadda <kirankumark@marvell.com> > > Subject: RE: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support > > > > PING. > > > > -----Original Message----- > > From: vattunuru@marvell.com <vattunuru@marvell.com> > > Sent: Friday, August 16, 2019 11:43 AM > > To: dev@dpdk.org > > Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran > > <jerinj@marvell.com>; olivier.matz@6wind.com; ferruh.yigit@intel.com; > > anatoly.burakov@intel.com; arybchenko@solarflare.com; Kiran Kumar > > Kokkilagadda <kirankumark@marvell.com>; Vamsi Krishna Attunuru > > <vattunuru@marvell.com> > > Subject: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support > > > > From: Vamsi Attunuru <vattunuru@marvell.com> > > > > --- > > V10 Changes: > > * Fixed function return code on failure when min_chunk_size > pg_sz. > > * Marked new mempool populate routine as EXPERIMENTAL. > > > > V9 Changes: > > * Used rte_mempool_ops_calc_mem_size() instead of default handler in > > the new mempool populate routine. > > * Check min_chunk_size and return values. > > * Removed ethdev_info memset to '0' and moved pci dev_info populate > > into > > kni_dev_pci_addr_get() routine. > > * Addressed misc. review comments. > > > > V8 Changes: > > * Remove default mempool populate() routine changes. > > * Add kni app specific mempool create & free routines. > > * Add new mempool populate routine to allocate page-aligned memzones > > with page size to make sure all mempool objects reside on a page. > > * Update release notes and map files. > > > > V7 Changes: > > * Removed previously proposed mempool flag and made those page > > boundary checks default in mempool populate() except for the objects > > size bigger than the size of page. > > * Removed KNI example application related changes since pool related > > requirement is taken care in mempool lib. > > * All PCI dev related info is moved under rte_eal_iova_mode() == VA check. > > * Added wrapper functions in KNI module to hide IOVA checks and make > > address translation routines more readable. > > * Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA > > mode is enabled. > > > > V6 Changes: > > * Added new mempool flag to ensure mbuf memory is not scattered across > > page boundaries. > > * Added KNI kernel module required PCI device information. > > * Modified KNI example application to create mempool with new mempool > > flag. > > > > V5 changes: > > * Fixed build issue with 32b build > > > > V4 changes: > > * Fixed build issues with older kernel versions > > * This approach will only work with kernel above 4.4.0 > > > > V3 Changes: > > * Add new approach to work kni with IOVA=VA mode using > > iommu_iova_to_phys API. > > > > Kiran Kumar K (1): > > kni: add IOVA=VA support in KNI module > > > > Vamsi Attunuru (4): > > mempool: populate mempool with the page sized chunks > > kni: add IOVA=VA support in KNI lib > > kni: add app specific mempool create and free routines > > kni: modify IOVA mode checks to support VA > > > > doc/guides/prog_guide/kernel_nic_interface.rst | 8 ++ > > doc/guides/rel_notes/release_19_11.rst | 5 ++ > > examples/kni/main.c | 5 +- > > kernel/linux/kni/compat.h | 4 + > > kernel/linux/kni/kni_dev.h | 4 + > > kernel/linux/kni/kni_misc.c | 71 ++++++++++++++--- > > kernel/linux/kni/kni_net.c | 59 ++++++++++---- > > lib/librte_eal/linux/eal/eal.c | 4 +- > > lib/librte_eal/linux/eal/include/rte_kni_common.h | 8 ++ > > lib/librte_kni/Makefile | 2 + > > lib/librte_kni/meson.build | 2 + > > lib/librte_kni/rte_kni.c | 95 +++++++++++++++++++++-- > > lib/librte_kni/rte_kni.h | 48 ++++++++++++ > > lib/librte_kni/rte_kni_version.map | 2 + > > lib/librte_mempool/rte_mempool.c | 69 ++++++++++++++++ > > lib/librte_mempool/rte_mempool.h | 20 +++++ > > lib/librte_mempool/rte_mempool_version.map | 1 + > > 17 files changed, 378 insertions(+), 29 deletions(-) > > > > -- > > 2.8.4
On 8/16/2019 7:12 AM, vattunuru@marvell.com wrote: > From: Vamsi Attunuru <vattunuru@marvell.com> > > --- > V10 Changes: > * Fixed function return code on failure when min_chunk_size > pg_sz. > * Marked new mempool populate routine as EXPERIMENTAL. > > V9 Changes: > * Used rte_mempool_ops_calc_mem_size() instead of default handler in > the new mempool populate routine. > * Check min_chunk_size and return values. > * Removed ethdev_info memset to '0' and moved pci dev_info populate into > kni_dev_pci_addr_get() routine. > * Addressed misc. review comments. > > V8 Changes: > * Remove default mempool populate() routine changes. > * Add kni app specific mempool create & free routines. > * Add new mempool populate routine to allocate page-aligned memzones > with page size to make sure all mempool objects reside on a page. > * Update release notes and map files. > > V7 Changes: > * Removed previously proposed mempool flag and made those page boundary > checks default in mempool populate() except for the objects size bigger > than the size of page. > * Removed KNI example application related changes since pool related > requirement is taken care in mempool lib. > * All PCI dev related info is moved under rte_eal_iova_mode() == VA check. > * Added wrapper functions in KNI module to hide IOVA checks and make > address translation routines more readable. > * Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA mode > is enabled. > > V6 Changes: > * Added new mempool flag to ensure mbuf memory is not scattered > across page boundaries. > * Added KNI kernel module required PCI device information. > * Modified KNI example application to create mempool with new > mempool flag. > > V5 changes: > * Fixed build issue with 32b build > > V4 changes: > * Fixed build issues with older kernel versions > * This approach will only work with kernel above 4.4.0 > > V3 Changes: > * Add new approach to work kni with IOVA=VA mode using > iommu_iova_to_phys API. > > Kiran Kumar K (1): > kni: add IOVA=VA support in KNI module > > Vamsi Attunuru (4): > mempool: populate mempool with the page sized chunks > kni: add IOVA=VA support in KNI lib > kni: add app specific mempool create and free routines > kni: modify IOVA mode checks to support VA Hi Vamsi, I am aware that this patchset is around for a long time, and I have seen your request to merge in 19.11, but as you can understand the concern I have is to break KNI or existing KNI applications while trying to add this new feature. In high level, there are two issues, 1) kernel modules updates expect there will be a backed device of the KNI which is not always true: if (dev_info.iova_mode) { #ifdef HAVE_IOVA_AS_VA_SUPPORT pci = pci_get_device(dev_info.vendor_id, dev_info.device_id, NULL); if (pci == NULL) { pr_err("pci dev does not exist\n"); return -ENODEV; } For example this breaks: ./build/app/testpmd -w0:0.0 --vdev net_kni0 --vdev net_kni1 -- -i 2) Applications will have to change the API to allocate the mempool. If the user upgraded to new version of DPDK, now it is possible to have iova=va mode and application should use new KNI API 'rte_kni_pktmbuf_pool_create()' to allocate mempool. And most probably application will have datapath and will use the KNI only for exception path, will there be any affect using KNI version of mempool alloc? I would like to see KNI is enabled via iova=va mode, but can we have it limited to a specific command line argument or config option? This increases the test surface but at least old application can continue to work by default, what do you think? And I will put a few minor comments to the patches...
> -----Original Message----- > From: dev <dev-bounces@dpdk.org> On Behalf Of Yigit, Ferruh > Sent: Tuesday, October 15, 2019 9:05 PM > To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org > Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; > olivier.matz@6wind.com; ferruh.yigit@intel.com; anatoly.burakov@intel.com; > arybchenko@solarflare.com; Kiran Kumar Kokkilagadda > <kirankumark@marvell.com> > Subject: Re: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support > > On 8/16/2019 7:12 AM, vattunuru@marvell.com wrote: > > From: Vamsi Attunuru <vattunuru@marvell.com> > > > > --- > > V10 Changes: > > * Fixed function return code on failure when min_chunk_size > pg_sz. > > * Marked new mempool populate routine as EXPERIMENTAL. > > > > V9 Changes: > > * Used rte_mempool_ops_calc_mem_size() instead of default handler in > > the new mempool populate routine. > > * Check min_chunk_size and return values. > > * Removed ethdev_info memset to '0' and moved pci dev_info populate > > into > > kni_dev_pci_addr_get() routine. > > * Addressed misc. review comments. > > > > V8 Changes: > > * Remove default mempool populate() routine changes. > > * Add kni app specific mempool create & free routines. > > * Add new mempool populate routine to allocate page-aligned memzones > > with page size to make sure all mempool objects reside on a page. > > * Update release notes and map files. > > > > V7 Changes: > > * Removed previously proposed mempool flag and made those page > > boundary checks default in mempool populate() except for the objects > > size bigger than the size of page. > > * Removed KNI example application related changes since pool related > > requirement is taken care in mempool lib. > > * All PCI dev related info is moved under rte_eal_iova_mode() == VA check. > > * Added wrapper functions in KNI module to hide IOVA checks and make > > address translation routines more readable. > > * Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA > > mode is enabled. > > > > V6 Changes: > > * Added new mempool flag to ensure mbuf memory is not scattered across > > page boundaries. > > * Added KNI kernel module required PCI device information. > > * Modified KNI example application to create mempool with new mempool > > flag. > > > > V5 changes: > > * Fixed build issue with 32b build > > > > V4 changes: > > * Fixed build issues with older kernel versions > > * This approach will only work with kernel above 4.4.0 > > > > V3 Changes: > > * Add new approach to work kni with IOVA=VA mode using > > iommu_iova_to_phys API. > > > > Kiran Kumar K (1): > > kni: add IOVA=VA support in KNI module > > > > Vamsi Attunuru (4): > > mempool: populate mempool with the page sized chunks > > kni: add IOVA=VA support in KNI lib > > kni: add app specific mempool create and free routines > > kni: modify IOVA mode checks to support VA > > Hi Vamsi, > > I am aware that this patchset is around for a long time, and I have seen your > request to merge in 19.11, but as you can understand the concern I have is to > break KNI or existing KNI applications while trying to add this new feature. > > In high level, there are two issues, > > 1) kernel modules updates expect there will be a backed device of the KNI which > is not always true: > > if (dev_info.iova_mode) { > #ifdef HAVE_IOVA_AS_VA_SUPPORT > pci = pci_get_device(dev_info.vendor_id, > dev_info.device_id, NULL); > if (pci == NULL) { > pr_err("pci dev does not exist\n"); > return -ENODEV; > } > > For example this breaks: > ./build/app/testpmd -w0:0.0 --vdev net_kni0 --vdev net_kni1 -- -i Vamsi> Yes, these can be fixed by forcing iommu_mode to PA for vdev or vdev&pdev based KNI usecases. > > > 2) Applications will have to change the API to allocate the mempool. > If the user upgraded to new version of DPDK, now it is possible to have iova=va > mode and application should use new KNI API 'rte_kni_pktmbuf_pool_create()' > to allocate mempool. And most probably application will have datapath and will > use the KNI only for exception path, will there be any affect using KNI version of > mempool alloc? Vamsi> There would not be any affect in using KNI version of mempool. > > > I would like to see KNI is enabled via iova=va mode, but can we have it limited to > a specific command line argument or config option? This increases the test > surface but at least old application can continue to work by default, what do you > think? Vamsi> Yes, it's appropriate to control the mode to ensure old apps work by default. We are fine with having a command line arg or config option to enable KNI in iova=va mode. Earlier we thought of having similar approach that also controls mempool allocation using a newer mempool flag. After multiple reviews, flag has been discard and added a separate mempool populate routine for these usecase. When command line arg/config option is introduced, functionality will be as below. Please correct me if any cases are missed or not considered. Without command: Existing KNI is intact, iommu mode will be PA. With command: Pdev/vdev's iommu mode is considered and accordingly iova=va/pa is enabled. Application is supposed to use KNI version of mempool alloc. I think these mempool quirk will go away when Olivier's mempool patchset(RFC) is merged. > > And I will put a few minor comments to the patches... >
On 10/16/2019 1:17 PM, Vamsi Krishna Attunuru wrote: > > >> -----Original Message----- >> From: dev <dev-bounces@dpdk.org> On Behalf Of Yigit, Ferruh >> Sent: Tuesday, October 15, 2019 9:05 PM >> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org >> Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; >> olivier.matz@6wind.com; ferruh.yigit@intel.com; anatoly.burakov@intel.com; >> arybchenko@solarflare.com; Kiran Kumar Kokkilagadda >> <kirankumark@marvell.com> >> Subject: Re: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support >> >> On 8/16/2019 7:12 AM, vattunuru@marvell.com wrote: >>> From: Vamsi Attunuru <vattunuru@marvell.com> >>> >>> --- >>> V10 Changes: >>> * Fixed function return code on failure when min_chunk_size > pg_sz. >>> * Marked new mempool populate routine as EXPERIMENTAL. >>> >>> V9 Changes: >>> * Used rte_mempool_ops_calc_mem_size() instead of default handler in >>> the new mempool populate routine. >>> * Check min_chunk_size and return values. >>> * Removed ethdev_info memset to '0' and moved pci dev_info populate >>> into >>> kni_dev_pci_addr_get() routine. >>> * Addressed misc. review comments. >>> >>> V8 Changes: >>> * Remove default mempool populate() routine changes. >>> * Add kni app specific mempool create & free routines. >>> * Add new mempool populate routine to allocate page-aligned memzones >>> with page size to make sure all mempool objects reside on a page. >>> * Update release notes and map files. >>> >>> V7 Changes: >>> * Removed previously proposed mempool flag and made those page >>> boundary checks default in mempool populate() except for the objects >>> size bigger than the size of page. >>> * Removed KNI example application related changes since pool related >>> requirement is taken care in mempool lib. >>> * All PCI dev related info is moved under rte_eal_iova_mode() == VA check. >>> * Added wrapper functions in KNI module to hide IOVA checks and make >>> address translation routines more readable. >>> * Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA >>> mode is enabled. >>> >>> V6 Changes: >>> * Added new mempool flag to ensure mbuf memory is not scattered across >>> page boundaries. >>> * Added KNI kernel module required PCI device information. >>> * Modified KNI example application to create mempool with new mempool >>> flag. >>> >>> V5 changes: >>> * Fixed build issue with 32b build >>> >>> V4 changes: >>> * Fixed build issues with older kernel versions >>> * This approach will only work with kernel above 4.4.0 >>> >>> V3 Changes: >>> * Add new approach to work kni with IOVA=VA mode using >>> iommu_iova_to_phys API. >>> >>> Kiran Kumar K (1): >>> kni: add IOVA=VA support in KNI module >>> >>> Vamsi Attunuru (4): >>> mempool: populate mempool with the page sized chunks >>> kni: add IOVA=VA support in KNI lib >>> kni: add app specific mempool create and free routines >>> kni: modify IOVA mode checks to support VA >> >> Hi Vamsi, >> >> I am aware that this patchset is around for a long time, and I have seen your >> request to merge in 19.11, but as you can understand the concern I have is to >> break KNI or existing KNI applications while trying to add this new feature. >> >> In high level, there are two issues, >> >> 1) kernel modules updates expect there will be a backed device of the KNI which >> is not always true: >> >> if (dev_info.iova_mode) { >> #ifdef HAVE_IOVA_AS_VA_SUPPORT >> pci = pci_get_device(dev_info.vendor_id, >> dev_info.device_id, NULL); >> if (pci == NULL) { >> pr_err("pci dev does not exist\n"); >> return -ENODEV; >> } >> >> For example this breaks: >> ./build/app/testpmd -w0:0.0 --vdev net_kni0 --vdev net_kni1 -- -i > > Vamsi> Yes, these can be fixed by forcing iommu_mode to PA for > vdev or vdev&pdev based KNI usecases. > >> >> >> 2) Applications will have to change the API to allocate the mempool. >> If the user upgraded to new version of DPDK, now it is possible to have iova=va >> mode and application should use new KNI API 'rte_kni_pktmbuf_pool_create()' >> to allocate mempool. And most probably application will have datapath and will >> use the KNI only for exception path, will there be any affect using KNI version of >> mempool alloc? > > Vamsi> There would not be any affect in using KNI version of mempool. If you were developing a product, would you rely on a KNI API for pkt_mbuf allocation? What if there is a problem, will it be as easy as mempool/mbuf API to figure out and fix? I am not sure about pushing users to this direction? > >> >> >> I would like to see KNI is enabled via iova=va mode, but can we have it limited to >> a specific command line argument or config option? This increases the test >> surface but at least old application can continue to work by default, what do you >> think? > > Vamsi> Yes, it's appropriate to control the mode to ensure old apps work by default. > We are fine with having a command line arg or config option to enable KNI in iova=va mode. > Earlier we thought of having similar approach that also controls mempool allocation using > a newer mempool flag. After multiple reviews, flag has been discard and added a separate > mempool populate routine for these usecase. I didn't like the idea of having a flag in mempool library, but perhaps we can have it in KNI scope. > > When command line arg/config option is introduced, functionality will be as below. > Please correct me if any cases are missed or not considered. > Without command: > Existing KNI is intact, iommu mode will be PA. +1 > With command: > Pdev/vdev's iommu mode is considered and accordingly iova=va/pa is enabled. Application is > supposed to use KNI version of mempool alloc. +1 > I think these mempool quirk will go away when > Olivier's mempool patchset(RFC) is merged. > >> >> And I will put a few minor comments to the patches... >> >
> -----Original Message----- > From: Ferruh Yigit <ferruh.yigit@intel.com> > Sent: Wednesday, October 16, 2019 9:52 PM > To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; Yigit, Ferruh > <ferruh.yigit@linux.intel.com> > Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; > olivier.matz@6wind.com; anatoly.burakov@intel.com; > arybchenko@solarflare.com; Kiran Kumar Kokkilagadda > <kirankumark@marvell.com>; dev@dpdk.org > Subject: [EXT] Re: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support > > External Email > > ---------------------------------------------------------------------- > On 10/16/2019 1:17 PM, Vamsi Krishna Attunuru wrote: > > > > > >> -----Original Message----- > >> From: dev <dev-bounces@dpdk.org> On Behalf Of Yigit, Ferruh > >> Sent: Tuesday, October 15, 2019 9:05 PM > >> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org > >> Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran > >> <jerinj@marvell.com>; olivier.matz@6wind.com; ferruh.yigit@intel.com; > >> anatoly.burakov@intel.com; arybchenko@solarflare.com; Kiran Kumar > >> Kokkilagadda <kirankumark@marvell.com> > >> Subject: Re: [dpdk-dev] [PATCH v10 0/5] kni: add IOVA=VA support > >> > >> On 8/16/2019 7:12 AM, vattunuru@marvell.com wrote: > >>> From: Vamsi Attunuru <vattunuru@marvell.com> > >>> > >>> --- > >>> V10 Changes: > >>> * Fixed function return code on failure when min_chunk_size > pg_sz. > >>> * Marked new mempool populate routine as EXPERIMENTAL. > >>> > >>> V9 Changes: > >>> * Used rte_mempool_ops_calc_mem_size() instead of default handler in > >>> the new mempool populate routine. > >>> * Check min_chunk_size and return values. > >>> * Removed ethdev_info memset to '0' and moved pci dev_info populate > >>> into > >>> kni_dev_pci_addr_get() routine. > >>> * Addressed misc. review comments. > >>> > >>> V8 Changes: > >>> * Remove default mempool populate() routine changes. > >>> * Add kni app specific mempool create & free routines. > >>> * Add new mempool populate routine to allocate page-aligned > memzones > >>> with page size to make sure all mempool objects reside on a page. > >>> * Update release notes and map files. > >>> > >>> V7 Changes: > >>> * Removed previously proposed mempool flag and made those page > >>> boundary checks default in mempool populate() except for the objects > >>> size bigger than the size of page. > >>> * Removed KNI example application related changes since pool related > >>> requirement is taken care in mempool lib. > >>> * All PCI dev related info is moved under rte_eal_iova_mode() == VA > check. > >>> * Added wrapper functions in KNI module to hide IOVA checks and make > >>> address translation routines more readable. > >>> * Updated IOVA mode checks that enforcing IOVA=PA mode when > IOVA=VA > >>> mode is enabled. > >>> > >>> V6 Changes: > >>> * Added new mempool flag to ensure mbuf memory is not scattered > >>> across page boundaries. > >>> * Added KNI kernel module required PCI device information. > >>> * Modified KNI example application to create mempool with new > >>> mempool flag. > >>> > >>> V5 changes: > >>> * Fixed build issue with 32b build > >>> > >>> V4 changes: > >>> * Fixed build issues with older kernel versions > >>> * This approach will only work with kernel above 4.4.0 > >>> > >>> V3 Changes: > >>> * Add new approach to work kni with IOVA=VA mode using > >>> iommu_iova_to_phys API. > >>> > >>> Kiran Kumar K (1): > >>> kni: add IOVA=VA support in KNI module > >>> > >>> Vamsi Attunuru (4): > >>> mempool: populate mempool with the page sized chunks > >>> kni: add IOVA=VA support in KNI lib > >>> kni: add app specific mempool create and free routines > >>> kni: modify IOVA mode checks to support VA > >> > >> Hi Vamsi, > >> > >> I am aware that this patchset is around for a long time, and I have > >> seen your request to merge in 19.11, but as you can understand the > >> concern I have is to break KNI or existing KNI applications while trying to > add this new feature. > >> > >> In high level, there are two issues, > >> > >> 1) kernel modules updates expect there will be a backed device of the > >> KNI which is not always true: > >> > >> if (dev_info.iova_mode) { > >> #ifdef HAVE_IOVA_AS_VA_SUPPORT > >> pci = pci_get_device(dev_info.vendor_id, > >> dev_info.device_id, NULL); > >> if (pci == NULL) { > >> pr_err("pci dev does not exist\n"); > >> return -ENODEV; > >> } > >> > >> For example this breaks: > >> ./build/app/testpmd -w0:0.0 --vdev net_kni0 --vdev net_kni1 -- -i > > > > Vamsi> Yes, these can be fixed by forcing iommu_mode to PA for > > vdev or vdev&pdev based KNI usecases. > > > >> > >> > >> 2) Applications will have to change the API to allocate the mempool. > >> If the user upgraded to new version of DPDK, now it is possible to > >> have iova=va mode and application should use new KNI API > 'rte_kni_pktmbuf_pool_create()' > >> to allocate mempool. And most probably application will have datapath > >> and will use the KNI only for exception path, will there be any > >> affect using KNI version of mempool alloc? > > > > Vamsi> There would not be any affect in using KNI version of mempool. > > If you were developing a product, would you rely on a KNI API for pkt_mbuf > allocation? What if there is a problem, will it be as easy as mempool/mbuf > API to figure out and fix? I am not sure about pushing users to this direction? Vamsi > If user wants to run KNI app in iova=va mode, mempool populate needs to ensure mbuf would not be allocated from a page boundary. IMO after having config option/cmd line parameter to enable KNI iova=va mode, existing KNI will be intact and these mempool API only be called upon request. I will document these details in KNI document. Based on earlier comments and discussions with Olivier and Andrew on the mempool populate patch, we arrived at these solution. Hi Olivier, Please let us know your thoughts on these patch. I am open to any of your suggestions/solution to fix mempool populate issue for KNI iova=va use case. > > > > >> > >> > >> I would like to see KNI is enabled via iova=va mode, but can we have > >> it limited to a specific command line argument or config option? This > >> increases the test surface but at least old application can continue > >> to work by default, what do you think? > > > > Vamsi> Yes, it's appropriate to control the mode to ensure old apps work > by default. > > We are fine with having a command line arg or config option to enable KNI > in iova=va mode. > > Earlier we thought of having similar approach that also controls > > mempool allocation using a newer mempool flag. After multiple reviews, > > flag has been discard and added a separate mempool populate routine for > these usecase. > > I didn't like the idea of having a flag in mempool library, but perhaps we can > have it in KNI scope. Vamsi> I failed to understand KNI scope here, IMO, though we have flag in KNI scope, it needs to be affected in mempool lib right finally. > > > > > When command line arg/config option is introduced, functionality will be > as below. > > Please correct me if any cases are missed or not considered. > > Without command: > > Existing KNI is intact, iommu mode will be PA. > > +1 > > > With command: > > Pdev/vdev's iommu mode is considered and accordingly iova=va/pa is > > enabled. Application is supposed to use KNI version of mempool alloc. > > +1 > > > I think these mempool quirk will go away when Olivier's mempool > > patchset(RFC) is merged. > > > >> > >> And I will put a few minor comments to the patches... > >> > >