Message ID | 20191105110416.8955-1-vattunuru@marvell.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E6D66A0353; Tue, 5 Nov 2019 12:04:47 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0A4612C02; Tue, 5 Nov 2019 12:04:47 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 09B102BCE for <dev@dpdk.org>; Tue, 5 Nov 2019 12:04:45 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id xA5B3Ell019613; Tue, 5 Nov 2019 03:04:45 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0818; bh=TmZR7Ce1979Eg/x/Ltr988b23kNZCPm/bu0GYXv093g=; b=lcQrwOq0vSoD1ruAavgru/0nw989HerWhQBH+dBW83LPLXNLeo3NNwnIDyySNpEJNsLD c7LbYK4CrHTrLsxU/GdR8u+GV926XRPFPzAHaqAFfKgG+BhWKVdkMH0xcRo+VSpErTaw SzuF9vvE2R+PyqkO4ba/HKPahxXQz8tCmzyjVwuTNnv/0H9pD+twuPhE5XGdxEeXqMpb /gXSXYFNpz736+gig8cZtjz5BieUj55HOwZP3u5rHl6nsaspWFPIHfZfLLyggnDRM+jJ 1kZ3cuseCQWgh3vslPlnXZkOX6fa/cVs18L2jMGRgEgXcWkzRUOp8bAH4TO46QP4brXi sQ== Received: from sc-exch04.marvell.com ([199.233.58.184]) by mx0b-0016f401.pphosted.com with ESMTP id 2w19amsw0r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Tue, 05 Nov 2019 03:04:44 -0800 Received: from SC-EXCH01.marvell.com (10.93.176.81) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 5 Nov 2019 03:04:42 -0800 Received: from maili.marvell.com (10.93.176.43) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 5 Nov 2019 03:04:42 -0800 Received: from hyd1vattunuru-dt.caveonetworks.com (unknown [10.29.52.72]) by maili.marvell.com (Postfix) with ESMTP id D6B133F703F; Tue, 5 Nov 2019 03:04:38 -0800 (PST) From: <vattunuru@marvell.com> To: <dev@dpdk.org> CC: <thomas@monjalon.net>, <jerinj@marvell.com>, <kirankumark@marvell.com>, <olivier.matz@6wind.com>, <ferruh.yigit@intel.com>, <anatoly.burakov@intel.com>, <arybchenko@solarflare.com>, <stephen@networkplumber.org>, Vamsi Attunuru <vattunuru@marvell.com> Date: Tue, 5 Nov 2019 16:34:14 +0530 Message-ID: <20191105110416.8955-1-vattunuru@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20191021080324.10659-1-vattunuru@marvell.com> References: <20191021080324.10659-1-vattunuru@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,1.0.8 definitions=2019-11-05_04:2019-11-05,2019-11-05 signatures=0 Subject: [dpdk-dev] [PATCH v12 0/2] add IOVA=VA mode support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
add IOVA=VA mode support
|
|
Message
Vamsi Krishna Attunuru
Nov. 5, 2019, 11:04 a.m. UTC
From: Vamsi Attunuru <vattunuru@marvell.com>
---
V12 Changes:
* Removed previously added `--legacy-kni` eal option.
* Removed previously added kni specific mempool create routines
and mempool populate routines.
This patch set(V12) is dependent on following patch set, since the mempool
related support to enable KNI in IOVA=VA mode is taken care in below
patchset.
https://patchwork.dpdk.org/cover/62376/
V11 Changes:
* Added iova to kva address translation routines in kernel module to
make it work in iova=va mode which enables DPDK to create kni devices
on any kind of backed device/memory.
* Added ``--legacy-kni`` eal option to make existing KNI applications
work with DPDK 19.11 and later versions.
* Removed previously added pci device info from kni device info struct.
V10 Changes:
* Fixed function return code on failure when min_chunk_size > pg_sz.
* Marked new mempool populate routine as EXPERIMENTAL.
V9 Changes:
* Used rte_mempool_ops_calc_mem_size() instead of default handler in the
new mempool populate routine.
* Check min_chunk_size and return values.
* Removed ethdev_info memset to '0' and moved pci dev_info populate into
kni_dev_pci_addr_get() routine.
* Addressed misc. review comments.
V8 Changes:
* Remove default mempool populate() routine changes.
* Add kni app specific mempool create & free routines.
* Add new mempool populate routine to allocate page-aligned memzones
with page size to make sure all mempool objects reside on a page.
* Update release notes and map files.
V7 Changes:
* Removed previously proposed mempool flag and made those page
boundary checks default in mempool populate() except for the objects size
bigger than the size of page.
* Removed KNI example application related changes since pool related
requirement is taken care in mempool lib.
* All PCI dev related info is moved under rte_eal_iova_mode() == VA check.
* Added wrapper functions in KNI module to hide IOVA checks and make
address translation routines more readable.
* Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA
mode is enabled.
V6 Changes:
* Added new mempool flag to ensure mbuf memory is not scattered across
page boundaries.
* Added KNI kernel module required PCI device information.
* Modified KNI example application to create mempool with new mempool
flag.
V5 changes:
* Fixed build issue with 32b build
V4 changes:
* Fixed build issues with older kernel versions
* This approach will only work with kernel above 4.4.0
V3 Changes:
* Add new approach to work kni with IOVA=VA mode using
iommu_iova_to_phys API.
Vamsi Attunuru (2):
kni: add IOVA=VA mode support
kni: add IOVA=VA support in kernel module
doc/guides/prog_guide/kernel_nic_interface.rst | 9 ++++
doc/guides/rel_notes/release_19_11.rst | 5 ++
kernel/linux/kni/compat.h | 15 ++++++
kernel/linux/kni/kni_dev.h | 42 +++++++++++++++
kernel/linux/kni/kni_misc.c | 39 ++++++++++----
kernel/linux/kni/kni_net.c | 62 ++++++++++++++++++-----
lib/librte_eal/linux/eal/eal.c | 29 ++++++-----
lib/librte_eal/linux/eal/include/rte_kni_common.h | 1 +
lib/librte_kni/rte_kni.c | 7 +--
9 files changed, 170 insertions(+), 39 deletions(-)
Comments
For info, the mempool patches are merged now. 05/11/2019 12:04, vattunuru@marvell.com: > Vamsi Attunuru (2): > kni: add IOVA=VA mode support > kni: add IOVA=VA support in kernel module Should the kernel support be the first patch? About the titles, can it be simply "support IOVA mode"?
> -----Original Message----- > From: Thomas Monjalon <thomas@monjalon.net> > Sent: Wednesday, November 6, 2019 4:19 PM > To: Vamsi Krishna Attunuru <vattunuru@marvell.com> > Cc: dev@dpdk.org; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Kiran > Kumar Kokkilagadda <kirankumark@marvell.com>; olivier.matz@6wind.com; > ferruh.yigit@intel.com; anatoly.burakov@intel.com; > arybchenko@solarflare.com; stephen@networkplumber.org > Subject: [EXT] Re: [dpdk-dev] [PATCH v12 0/2] add IOVA=VA mode support > > External Email > > ---------------------------------------------------------------------- > For info, the mempool patches are merged now. > > 05/11/2019 12:04, vattunuru@marvell.com: > > Vamsi Attunuru (2): > > kni: add IOVA=VA mode support > > kni: add IOVA=VA support in kernel module > > Should the kernel support be the first patch? But required variable `iova_mode` is defined in first patch currently. Either one is fine to me. Please let me know if want to put kernel support patch at first, I will send next version accordingly. > > About the titles, can it be simply "support IOVA mode"? Yes, "support IOVA_VA mode" should be fine. >
06/11/2019 12:09, Vamsi Krishna Attunuru: > From: Thomas Monjalon <thomas@monjalon.net> > > 05/11/2019 12:04, vattunuru@marvell.com: > > > Vamsi Attunuru (2): > > > kni: add IOVA=VA mode support > > > kni: add IOVA=VA support in kernel module > > > > Should the kernel support be the first patch? > > But required variable `iova_mode` is defined in first patch currently. > Either one is fine to me. Please let me know if want to put kernel support patch at first, I will send next version accordingly. > > > > > About the titles, can it be simply "support IOVA mode"? > > Yes, "support IOVA_VA mode" should be fine. I really dislike the name IOVA_VA. You mean support virtual IO addressing?
> -----Original Message----- > From: Thomas Monjalon <thomas@monjalon.net> > Sent: Wednesday, November 6, 2019 5:23 PM > To: Vamsi Krishna Attunuru <vattunuru@marvell.com> > Cc: dev@dpdk.org; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Kiran > Kumar Kokkilagadda <kirankumark@marvell.com>; olivier.matz@6wind.com; > ferruh.yigit@intel.com; anatoly.burakov@intel.com; > arybchenko@solarflare.com; stephen@networkplumber.org > Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v12 0/2] add IOVA=VA mode support > > 06/11/2019 12:09, Vamsi Krishna Attunuru: > > From: Thomas Monjalon <thomas@monjalon.net> > > > 05/11/2019 12:04, vattunuru@marvell.com: > > > > Vamsi Attunuru (2): > > > > kni: add IOVA=VA mode support > > > > kni: add IOVA=VA support in kernel module > > > > > > Should the kernel support be the first patch? > > > > But required variable `iova_mode` is defined in first patch currently. > > Either one is fine to me. Please let me know if want to put kernel support > patch at first, I will send next version accordingly. > > > > > > > > About the titles, can it be simply "support IOVA mode"? > > > > Yes, "support IOVA_VA mode" should be fine. > > I really dislike the name IOVA_VA. > > You mean support virtual IO addressing? > Yes.
Hi Ferruh, Could you please review v12. Regards, A Vamsi > -----Original Message----- > From: Vamsi Krishna Attunuru > Sent: Wednesday, November 6, 2019 4:39 PM > To: Thomas Monjalon <thomas@monjalon.net> > Cc: dev@dpdk.org; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Kiran > Kumar Kokkilagadda <kirankumark@marvell.com>; olivier.matz@6wind.com; > ferruh.yigit@intel.com; anatoly.burakov@intel.com; > arybchenko@solarflare.com; stephen@networkplumber.org > Subject: RE: [EXT] Re: [dpdk-dev] [PATCH v12 0/2] add IOVA=VA mode support > > > > > -----Original Message----- > > From: Thomas Monjalon <thomas@monjalon.net> > > Sent: Wednesday, November 6, 2019 4:19 PM > > To: Vamsi Krishna Attunuru <vattunuru@marvell.com> > > Cc: dev@dpdk.org; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; > > Kiran Kumar Kokkilagadda <kirankumark@marvell.com>; > > olivier.matz@6wind.com; ferruh.yigit@intel.com; > > anatoly.burakov@intel.com; arybchenko@solarflare.com; > > stephen@networkplumber.org > > Subject: [EXT] Re: [dpdk-dev] [PATCH v12 0/2] add IOVA=VA mode support > > > > External Email > > > > ---------------------------------------------------------------------- > > For info, the mempool patches are merged now. > > > > 05/11/2019 12:04, vattunuru@marvell.com: > > > Vamsi Attunuru (2): > > > kni: add IOVA=VA mode support > > > kni: add IOVA=VA support in kernel module > > > > Should the kernel support be the first patch? > > But required variable `iova_mode` is defined in first patch currently. > Either one is fine to me. Please let me know if want to put kernel support patch > at first, I will send next version accordingly. > > > > > About the titles, can it be simply "support IOVA mode"? > > Yes, "support IOVA_VA mode" should be fine. > >
On 11/5/2019 11:04 AM, vattunuru@marvell.com wrote: > From: Vamsi Attunuru <vattunuru@marvell.com> > > --- > V12 Changes: > * Removed previously added `--legacy-kni` eal option. > * Removed previously added kni specific mempool create routines > and mempool populate routines. > > This patch set(V12) is dependent on following patch set, since the mempool > related support to enable KNI in IOVA=VA mode is taken care in below > patchset. > > https://patchwork.dpdk.org/cover/62376/ Hi Vasim, Jerin, Overall looks good and I not getting any functional error but I am observing a huge performance drop with this update, 3.8Mpps to 0.7Mpps [1]. I don't know really what to do, I think we need to give a decision as community, and even we go with the patch we should document this performance drop clearly and document how to mitigate it. [1] This is with kni sample application, a) IOVA=VA mode selected ./examples/kni/build/kni -l0,40-47 --log-level=*:debug -- -p 0x3 -P --config "(0,44,45,40),(1,46,47,41)" forwarding performance is around 0.7Mpps and 'kni_single' kernel thread consumes all cpu. b) IOVA=PA mode forced ./examples/kni/build/kni -l0,40-47 --log-level=*:debug --iova=pa -- -p 0x3 -P --config "(0,44,45,40),(1,46,47,41)" forwarding performance is around 3.8Mpps and 'kni_single' core utilization is ~80%. I am on 5.1.20-300.fc30.x86_64 kernel. kni module inserted as: "insmod ./build/kmod/rte_kni.ko lo_mode=lo_mode_fifo" > > V11 Changes: > * Added iova to kva address translation routines in kernel module to > make it work in iova=va mode which enables DPDK to create kni devices > on any kind of backed device/memory. > * Added ``--legacy-kni`` eal option to make existing KNI applications > work with DPDK 19.11 and later versions. > * Removed previously added pci device info from kni device info struct. > > V10 Changes: > * Fixed function return code on failure when min_chunk_size > pg_sz. > * Marked new mempool populate routine as EXPERIMENTAL. > > V9 Changes: > * Used rte_mempool_ops_calc_mem_size() instead of default handler in the > new mempool populate routine. > * Check min_chunk_size and return values. > * Removed ethdev_info memset to '0' and moved pci dev_info populate into > kni_dev_pci_addr_get() routine. > * Addressed misc. review comments. > > V8 Changes: > * Remove default mempool populate() routine changes. > * Add kni app specific mempool create & free routines. > * Add new mempool populate routine to allocate page-aligned memzones > with page size to make sure all mempool objects reside on a page. > * Update release notes and map files. > > V7 Changes: > * Removed previously proposed mempool flag and made those page > boundary checks default in mempool populate() except for the objects size > bigger than the size of page. > * Removed KNI example application related changes since pool related > requirement is taken care in mempool lib. > * All PCI dev related info is moved under rte_eal_iova_mode() == VA check. > * Added wrapper functions in KNI module to hide IOVA checks and make > address translation routines more readable. > * Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA > mode is enabled. > > V6 Changes: > * Added new mempool flag to ensure mbuf memory is not scattered across > page boundaries. > * Added KNI kernel module required PCI device information. > * Modified KNI example application to create mempool with new mempool > flag. > > V5 changes: > * Fixed build issue with 32b build > > V4 changes: > * Fixed build issues with older kernel versions > * This approach will only work with kernel above 4.4.0 > > V3 Changes: > * Add new approach to work kni with IOVA=VA mode using > iommu_iova_to_phys API. > > Vamsi Attunuru (2): > kni: add IOVA=VA mode support > kni: add IOVA=VA support in kernel module > > doc/guides/prog_guide/kernel_nic_interface.rst | 9 ++++ > doc/guides/rel_notes/release_19_11.rst | 5 ++ > kernel/linux/kni/compat.h | 15 ++++++ > kernel/linux/kni/kni_dev.h | 42 +++++++++++++++ > kernel/linux/kni/kni_misc.c | 39 ++++++++++---- > kernel/linux/kni/kni_net.c | 62 ++++++++++++++++++----- > lib/librte_eal/linux/eal/eal.c | 29 ++++++----- > lib/librte_eal/linux/eal/include/rte_kni_common.h | 1 + > lib/librte_kni/rte_kni.c | 7 +-- > 9 files changed, 170 insertions(+), 39 deletions(-) >
> -----Original Message----- > From: Ferruh Yigit <ferruh.yigit@intel.com> > Sent: Friday, November 8, 2019 1:23 AM > To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org > Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; > Kiran Kumar Kokkilagadda <kirankumark@marvell.com>; > olivier.matz@6wind.com; anatoly.burakov@intel.com; > arybchenko@solarflare.com; stephen@networkplumber.org > Subject: [EXT] Re: [dpdk-dev] [PATCH v12 0/2] add IOVA=VA mode support > > External Email > > ---------------------------------------------------------------------- > On 11/5/2019 11:04 AM, vattunuru@marvell.com wrote: > > > From: Vamsi Attunuru <vattunuru@marvell.com> > > > > > > --- > > > V12 Changes: > > > * Removed previously added `--legacy-kni` eal option. > > > * Removed previously added kni specific mempool create routines > > > and mempool populate routines. > > > > > > This patch set(V12) is dependent on following patch set, since the mempool > > > related support to enable KNI in IOVA=VA mode is taken care in below > > > patchset. > > > > > > https://urldefense.proofpoint.com/v2/url?u=https- > 3A__patchwork.dpdk.org_cover_62376_&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7 > xtfQ&r=WllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=sEREtIZWQwtHJ > CxXRRv8euBGdr5q6K4L8Kz7PI25QcY&s=5Rz1dbSeIWt56cCtiwR6pfGprEFUumtcd > 34TmF3sjs4&e= > > > > Hi Vasim, Jerin, > > > > Overall looks good and I not getting any functional error but I am observing a > > huge performance drop with this update, 3.8Mpps to 0.7Mpps [1]. Hi Ferruh, When it comes to actual kernel netdev test cases like iperf or any other use cases, there would not be any impact on performance. I think synthetic test case like loopback mode might not be the actual test case alone to depend on when the kernel module is featured to work with kind of devices(pdev or vdev). Users can always fallback to pa mode with cmd line option. Please suggest your thoughts on considering what test case to use & evaluate the performance difference. > > > > I don't know really what to do, I think we need to give a decision as community, > > and even we go with the patch we should document this performance drop > clearly > > and document how to mitigate it. > > > > > > > > [1] > > This is with kni sample application, > > a) IOVA=VA mode selected > > ./examples/kni/build/kni -l0,40-47 --log-level=*:debug -- -p 0x3 -P --config > > "(0,44,45,40),(1,46,47,41)" > > > > forwarding performance is around 0.7Mpps and 'kni_single' kernel thread > consumes > > all cpu. > > > > b) IOVA=PA mode forced > > ./examples/kni/build/kni -l0,40-47 --log-level=*:debug --iova=pa -- -p 0x3 -P > > --config "(0,44,45,40),(1,46,47,41)" > > > > forwarding performance is around 3.8Mpps and 'kni_single' core utilization is > ~80%. > > > > I am on 5.1.20-300.fc30.x86_64 kernel. > > kni module inserted as: "insmod ./build/kmod/rte_kni.ko > lo_mode=lo_mode_fifo" > > > > > > > > V11 Changes: > > > * Added iova to kva address translation routines in kernel module to > > > make it work in iova=va mode which enables DPDK to create kni devices > > > on any kind of backed device/memory. > > > * Added ``--legacy-kni`` eal option to make existing KNI applications > > > work with DPDK 19.11 and later versions. > > > * Removed previously added pci device info from kni device info struct. > > > > > > V10 Changes: > > > * Fixed function return code on failure when min_chunk_size > pg_sz. > > > * Marked new mempool populate routine as EXPERIMENTAL. > > > > > > V9 Changes: > > > * Used rte_mempool_ops_calc_mem_size() instead of default handler in the > > > new mempool populate routine. > > > * Check min_chunk_size and return values. > > > * Removed ethdev_info memset to '0' and moved pci dev_info populate into > > > kni_dev_pci_addr_get() routine. > > > * Addressed misc. review comments. > > > > > > V8 Changes: > > > * Remove default mempool populate() routine changes. > > > * Add kni app specific mempool create & free routines. > > > * Add new mempool populate routine to allocate page-aligned memzones > > > with page size to make sure all mempool objects reside on a page. > > > * Update release notes and map files. > > > > > > V7 Changes: > > > * Removed previously proposed mempool flag and made those page > > > boundary checks default in mempool populate() except for the objects size > > > bigger than the size of page. > > > * Removed KNI example application related changes since pool related > > > requirement is taken care in mempool lib. > > > * All PCI dev related info is moved under rte_eal_iova_mode() == VA check. > > > * Added wrapper functions in KNI module to hide IOVA checks and make > > > address translation routines more readable. > > > * Updated IOVA mode checks that enforcing IOVA=PA mode when IOVA=VA > > > mode is enabled. > > > > > > V6 Changes: > > > * Added new mempool flag to ensure mbuf memory is not scattered across > > > page boundaries. > > > * Added KNI kernel module required PCI device information. > > > * Modified KNI example application to create mempool with new mempool > > > flag. > > > > > > V5 changes: > > > * Fixed build issue with 32b build > > > > > > V4 changes: > > > * Fixed build issues with older kernel versions > > > * This approach will only work with kernel above 4.4.0 > > > > > > V3 Changes: > > > * Add new approach to work kni with IOVA=VA mode using > > > iommu_iova_to_phys API. > > > > > > Vamsi Attunuru (2): > > > kni: add IOVA=VA mode support > > > kni: add IOVA=VA support in kernel module > > > > > > doc/guides/prog_guide/kernel_nic_interface.rst | 9 ++++ > > > doc/guides/rel_notes/release_19_11.rst | 5 ++ > > > kernel/linux/kni/compat.h | 15 ++++++ > > > kernel/linux/kni/kni_dev.h | 42 +++++++++++++++ > > > kernel/linux/kni/kni_misc.c | 39 ++++++++++---- > > > kernel/linux/kni/kni_net.c | 62 ++++++++++++++++++----- > > > lib/librte_eal/linux/eal/eal.c | 29 ++++++----- > > > lib/librte_eal/linux/eal/include/rte_kni_common.h | 1 + > > > lib/librte_kni/rte_kni.c | 7 +-- > > > 9 files changed, 170 insertions(+), 39 deletions(-) > > > > >
>> Hi Vasim, Jerin, >> >> Overall looks good and I not getting any functional error but I am observing a >> huge performance drop with this update, 3.8Mpps to 0.7Mpps [1]. > > Hi Ferruh, > When it comes to actual kernel netdev test cases like iperf or any other use cases, there would not be any impact on performance. I think synthetic test case like loopback mode might not be the actual test case alone to depend on when the kernel module is featured to work with kind of devices(pdev or vdev). Users can always fallback to pa mode with cmd line option. > > Please suggest your thoughts on considering what test case to use & evaluate the performance difference. Hi Vasim, I also assume the real life test cases will be affected less, but the loopback performance testing is good to show performance impact of the change. (Stephen's predictions that KNI is not as fast as tun/tap are getting more real by time J) At least I think the possible performance drop and how to mitigate it should be documented both in release notes and kni documentation. For the final decision, I am not objecting it but I would like to see more ack from community to confirm that we trade off iova=va functionality against performance. @Jerin, @Thomas, should we conclude this in techboard? Perhaps we can get it for rc2 and drop it back if rejected in techboard? Regards, ferruh
On Fri, Nov 8, 2019 at 7:56 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote: > > > >> Hi Vasim, Jerin, > >> > >> Overall looks good and I not getting any functional error but I am observing a > >> huge performance drop with this update, 3.8Mpps to 0.7Mpps [1]. > > > > Hi Ferruh, > > When it comes to actual kernel netdev test cases like iperf or any other use cases, there would not be any impact on performance. I think synthetic test case like loopback mode might not be the actual test case alone to depend on when the kernel module is featured to work with kind of devices(pdev or vdev). Users can always fallback to pa mode with cmd line option. > > > > Please suggest your thoughts on considering what test case to use & evaluate the performance difference. > > Hi Vasim, > > I also assume the real life test cases will be affected less, but the loopback > performance testing is good to show performance impact of the change. Yes. real-world case Linux kernel stack will be the bottleneck. > > (Stephen's predictions that KNI is not as fast as tun/tap are getting more real > by time J) > > At least I think the possible performance drop and how to mitigate it should be > documented both in release notes and kni documentation. +1 for adding documentation. Setting iova-mode=pa will be mitigation if the application does not care about iova-mode. > > For the final decision, I am not objecting it but I would like to see more ack > from community to confirm that we trade off iova=va functionality against > performance. In my view, IOVA as VA mode case, translation cannot be avoided and we have the requirement where it needs to work with vdev(where is not backed by any IOMMU context) so I am not sure how to avoid the translation cost. Since we have support for both modes,i.e existing IOVA as PA path still exists, I don't think, we are losing anything. > @Jerin, @Thomas, should we conclude this in techboard? Perhaps we can get it for > rc2 and drop it back if rejected in techboard? > > Regards, > ferruh
> -----Original Message----- > From: Jerin Jacob <jerinjacobk@gmail.com> > Sent: Friday, November 8, 2019 8:24 PM > To: Ferruh Yigit <ferruh.yigit@intel.com> > Cc: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org; > thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Kiran > Kumar Kokkilagadda <kirankumark@marvell.com>; olivier.matz@6wind.com; > anatoly.burakov@intel.com; arybchenko@solarflare.com; > stephen@networkplumber.org > Subject: Re: [dpdk-dev] [EXT] Re: [PATCH v12 0/2] add IOVA=VA mode support > > On Fri, Nov 8, 2019 at 7:56 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote: > > > > > > >> Hi Vasim, Jerin, > > >> > > >> Overall looks good and I not getting any functional error but I am > > >> observing a huge performance drop with this update, 3.8Mpps to 0.7Mpps > [1]. > > > > > > Hi Ferruh, > > > When it comes to actual kernel netdev test cases like iperf or any other use > cases, there would not be any impact on performance. I think synthetic test case > like loopback mode might not be the actual test case alone to depend on when > the kernel module is featured to work with kind of devices(pdev or vdev). Users > can always fallback to pa mode with cmd line option. > > > > > > Please suggest your thoughts on considering what test case to use & > evaluate the performance difference. > > > > Hi Vasim, > > > > I also assume the real life test cases will be affected less, but the > > loopback performance testing is good to show performance impact of the > change. > > Yes. real-world case Linux kernel stack will be the bottleneck. > > > > > (Stephen's predictions that KNI is not as fast as tun/tap are getting > > more real by time J) > > > > At least I think the possible performance drop and how to mitigate it > > should be documented both in release notes and kni documentation. > > +1 for adding documentation. Setting iova-mode=pa will be mitigation > if the application does > not care about iova-mode. > > > > > For the final decision, I am not objecting it but I would like to see > > more ack from community to confirm that we trade off iova=va > > functionality against performance. > > In my view, IOVA as VA mode case, translation cannot be avoided and we have > the requirement where it needs to work with vdev(where is not backed by any > IOMMU context) so I am not sure how to avoid the translation cost. Since we > have support for both modes,i.e existing IOVA as PA path still exists, I don't > think, we are losing anything. > > > @Jerin, @Thomas, should we conclude this in techboard? Perhaps we can > > get it for > > rc2 and drop it back if rejected in techboard? Hi Ferruh, Any update on the conclusion about the acceptance of patch set. I will send the next version of patch set with updated documentation part on performance impact if there are no other concerns. Regards A Vamsi > > > > Regards, > > ferruh
On 11/13/2019 6:33 AM, Vamsi Krishna Attunuru wrote: > >> -----Original Message----- >> From: Jerin Jacob <jerinjacobk@gmail.com> >> Sent: Friday, November 8, 2019 8:24 PM >> To: Ferruh Yigit <ferruh.yigit@intel.com> >> Cc: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org; >> thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Kiran >> Kumar Kokkilagadda <kirankumark@marvell.com>; olivier.matz@6wind.com; >> anatoly.burakov@intel.com; arybchenko@solarflare.com; >> stephen@networkplumber.org >> Subject: Re: [dpdk-dev] [EXT] Re: [PATCH v12 0/2] add IOVA=VA mode support >> >> On Fri, Nov 8, 2019 at 7:56 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote: >>> >>> >>>>> Hi Vasim, Jerin, >>>>> >>>>> Overall looks good and I not getting any functional error but I am >>>>> observing a huge performance drop with this update, 3.8Mpps to 0.7Mpps >> [1]. >>>> >>>> Hi Ferruh, >>>> When it comes to actual kernel netdev test cases like iperf or any other use >> cases, there would not be any impact on performance. I think synthetic test case >> like loopback mode might not be the actual test case alone to depend on when >> the kernel module is featured to work with kind of devices(pdev or vdev). Users >> can always fallback to pa mode with cmd line option. >>>> >>>> Please suggest your thoughts on considering what test case to use & >> evaluate the performance difference. >>> >>> Hi Vasim, >>> >>> I also assume the real life test cases will be affected less, but the >>> loopback performance testing is good to show performance impact of the >> change. >> >> Yes. real-world case Linux kernel stack will be the bottleneck. >> >>> >>> (Stephen's predictions that KNI is not as fast as tun/tap are getting >>> more real by time J) >>> >>> At least I think the possible performance drop and how to mitigate it >>> should be documented both in release notes and kni documentation. >> >> +1 for adding documentation. Setting iova-mode=pa will be mitigation >> if the application does >> not care about iova-mode. >> >>> >>> For the final decision, I am not objecting it but I would like to see >>> more ack from community to confirm that we trade off iova=va >>> functionality against performance. >> >> In my view, IOVA as VA mode case, translation cannot be avoided and we have >> the requirement where it needs to work with vdev(where is not backed by any >> IOMMU context) so I am not sure how to avoid the translation cost. Since we >> have support for both modes,i.e existing IOVA as PA path still exists, I don't >> think, we are losing anything. >> >>> @Jerin, @Thomas, should we conclude this in techboard? Perhaps we can >>> get it for >>> rc2 and drop it back if rejected in techboard? > > Hi Ferruh, > > Any update on the conclusion about the acceptance of patch set. I will send the next version of patch set with updated documentation part on performance impact if there are no other concerns. > Hi Vamsi, From my perspective there is no other change request than the documentation update, I suggest sending new version. For the final decision, it can be in technical board but there is no meeting this week, so I will start an offline discussion. Thanks, ferruh