[dpdk-dev,PATCHv3,1/2] config/armv8a: disable igb_uio

  IGB_UIO not supported for arm64 arch in kernel so disable.

Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
---
 config/defconfig_arm64-armv8a-linuxapp-gcc | 2 ++
 1 file changed, 2 insertions(+)
  

On Wed, 11 May 2016 19:17:58 +0530
Hemant Agrawal <hemant.agrawal@nxp.com> wrote:

> IGB_UIO not supported for arm64 arch in kernel so disable.
> 
> Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>

Really, I have use IGB_UIO on ARM64
  

On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
> On Wed, 11 May 2016 19:17:58 +0530
> Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
> 
> > IGB_UIO not supported for arm64 arch in kernel so disable.
> > 
> > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> 
> Really, I have use IGB_UIO on ARM64

upstream kernel doesn't support pci mmap for arm64. pl. refer [1], [2]. so I
assume
- your using out of tree patch something like below for igb_uio on arm64.  also
  arm-linux community encouraging vfio use for such use-cases. 

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2015-July/359435.html
[2] http://www.spinics.net/lists/arm-kernel/msg498005.html
  

On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
> On Wed, 11 May 2016 19:17:58 +0530
> Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
> 
> > IGB_UIO not supported for arm64 arch in kernel so disable.
> > 
> > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> 
> Really, I have use IGB_UIO on ARM64

May I know what is the technical use case for igb_uio on arm64
which cannot be addressed through vfio or vfioionommu.
  

On Wed, 11 May 2016 22:32:16 +0530
Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:

> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
> > On Wed, 11 May 2016 19:17:58 +0530
> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
> > 
> > > IGB_UIO not supported for arm64 arch in kernel so disable.
> > > 
> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> > 
> > Really, I have use IGB_UIO on ARM64
> 
> May I know what is the technical use case for igb_uio on arm64
> which cannot be addressed through vfio or vfioionommu.

I was running on older kernel which did not support vfioionommu mode.
  

On 12 May 2016 at 02:25, Stephen Hemminger <stephen@networkplumber.org> wrote:
> On Wed, 11 May 2016 22:32:16 +0530
> Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
>
>> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
>> > On Wed, 11 May 2016 19:17:58 +0530
>> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
>> >
>> > > IGB_UIO not supported for arm64 arch in kernel so disable.
>> > >
>> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
>> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
>> >
>> > Really, I have use IGB_UIO on ARM64
>>
>> May I know what is the technical use case for igb_uio on arm64
>> which cannot be addressed through vfio or vfioionommu.
>
> I was running on older kernel which did not support vfioionommu mode.

As I said, most of DPDK developers are not kernel developers. They may
have their own kernel tree, and couldn't like to upgrade to latest
kernel.
They can choose to use or not use igb_uio when binding the driver. But
blindly disabling it in the base config seems unreasonable.
  

On Wed, May 11, 2016 at 11:25:59AM -0700, Stephen Hemminger wrote:
> On Wed, 11 May 2016 22:32:16 +0530
> Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
> 
> > On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
> > > On Wed, 11 May 2016 19:17:58 +0530
> > > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
> > > 
> > > > IGB_UIO not supported for arm64 arch in kernel so disable.
> > > > 
> > > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> > > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> > > 
> > > Really, I have use IGB_UIO on ARM64
> > 
> > May I know what is the technical use case for igb_uio on arm64
> > which cannot be addressed through vfio or vfioionommu.
> 
> I was running on older kernel which did not support vfioionommu mode.

That way if we see older and latest kernel does not have ibg_uio(due to
sysfs mmap issue) support .If you are back-porting the changes
I recommend to back port vfioionommu changes to old kernel.

If it comes to out of tree then dpdk out of tree configuration can also set
CONFIG_RTE_EAL_IGB_UIO or even while configuring dpdk.

IMO, It is better to keep arm64 dpdk.org changes inline with
upstream arm64 linux kernel changes.

What do you think?
  

On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
> On 12 May 2016 at 02:25, Stephen Hemminger <stephen@networkplumber.org> wrote:
> > On Wed, 11 May 2016 22:32:16 +0530
> > Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
> >
> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
> >> > On Wed, 11 May 2016 19:17:58 +0530
> >> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
> >> >
> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
> >> > >
> >> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> >> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> >> >
> >> > Really, I have use IGB_UIO on ARM64
> >>
> >> May I know what is the technical use case for igb_uio on arm64
> >> which cannot be addressed through vfio or vfioionommu.
> >
> > I was running on older kernel which did not support vfioionommu mode.
> 
> As I said, most of DPDK developers are not kernel developers. They may
> have their own kernel tree, and couldn't like to upgrade to latest
> kernel.
> They can choose to use or not use igb_uio when binding the driver. But
> blindly disabling it in the base config seems unreasonable.

if user keeping his own kernel so they could also keep IGB_UIO=y in their local
dpdk tree. Why are you imposing user-x custome depedancy on upstream dpdk base
config. Is it not enough for explanation that - Base config ie.. armv8 doesn;t
support pci mmap, so igb_uio is n/a. New user wont able to build/run dpdk/arm64
in igb_uio-way, He'll prefer to use upstream stuff. I think, you are not making
sense.
  

On 12 May 2016 at 11:17, Santosh Shukla
<santosh.shukla@caviumnetworks.com> wrote:
> On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
>> On 12 May 2016 at 02:25, Stephen Hemminger <stephen@networkplumber.org> wrote:
>> > On Wed, 11 May 2016 22:32:16 +0530
>> > Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
>> >
>> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
>> >> > On Wed, 11 May 2016 19:17:58 +0530
>> >> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
>> >> >
>> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
>> >> > >
>> >> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
>> >> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
>> >> >
>> >> > Really, I have use IGB_UIO on ARM64
>> >>
>> >> May I know what is the technical use case for igb_uio on arm64
>> >> which cannot be addressed through vfio or vfioionommu.
>> >
>> > I was running on older kernel which did not support vfioionommu mode.
>>
>> As I said, most of DPDK developers are not kernel developers. They may
>> have their own kernel tree, and couldn't like to upgrade to latest
>> kernel.
>> They can choose to use or not use igb_uio when binding the driver. But
>> blindly disabling it in the base config seems unreasonable.
>
> if user keeping his own kernel so they could also keep IGB_UIO=y in their local
Most likely they don't have local dpdk tree. They write their own
applications, complie and link to dpdk lib, then done.

> dpdk tree. Why are you imposing user-x custome depedancy on upstream dpdk base
Customer requiremnts is important. I want they can choose the way they like.

> config. Is it not enough for explanation that - Base config ie.. armv8 doesn;t
> support pci mmap, so igb_uio is n/a. New user wont able to build/run dpdk/arm64
> in igb_uio-way, He'll prefer to use upstream stuff. I think, you are not making
You are wrong, he can build dpdk. If he like to use upstream without
patching, he can use vfio.
But you can't ignore the need from old user which is more comfortable
with older kernel.

> sense.
>
  

On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:
> On 12 May 2016 at 11:17, Santosh Shukla
> <santosh.shukla@caviumnetworks.com> wrote:
> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
> >> On 12 May 2016 at 02:25, Stephen Hemminger <stephen@networkplumber.org> wrote:
> >> > On Wed, 11 May 2016 22:32:16 +0530
> >> > Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
> >> >
> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
> >> >> > On Wed, 11 May 2016 19:17:58 +0530
> >> >> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
> >> >> >
> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
> >> >> > >
> >> >> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> >> >> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> >> >> >
> >> >> > Really, I have use IGB_UIO on ARM64
> >> >>
> >> >> May I know what is the technical use case for igb_uio on arm64
> >> >> which cannot be addressed through vfio or vfioionommu.
> >> >
> >> > I was running on older kernel which did not support vfioionommu mode.
> >>
> >> As I said, most of DPDK developers are not kernel developers. They may
> >> have their own kernel tree, and couldn't like to upgrade to latest
> >> kernel.
> >> They can choose to use or not use igb_uio when binding the driver. But
> >> blindly disabling it in the base config seems unreasonable.
> >
> > if user keeping his own kernel so they could also keep IGB_UIO=y in their local
> Most likely they don't have local dpdk tree. They write their own
> applications, complie and link to dpdk lib, then done.
> 
> > dpdk tree. Why are you imposing user-x custome depedancy on upstream dpdk base
> Customer requiremnts is important. I want they can choose the way they like.
>

so you choose to keep igb_uio option, provided arch doesn't support?
new user did reported issues with igb_uio for arm64, refer this thread [1], as
well hemanth too faced issues. we want to avoid that. 

If customer maintaing out-of-tree kernel then he can also switch to vfio-way.
isn;t it?

> > config. Is it not enough for explanation that - Base config ie.. armv8 doesn;t
> > support pci mmap, so igb_uio is n/a. New user wont able to build/run dpdk/arm64
> > in igb_uio-way, He'll prefer to use upstream stuff. I think, you are not making
> You are wrong, he can build dpdk. If he like to use upstream without
> patching, he can use vfio.

I disagree, we want to avoid [1] for new user.

> But you can't ignore the need from old user which is more comfortable
> with older kernel.
> 
arm/arm64 dpdk support recently added and I am guessing, most likely customer
using near latest kernel, switching to vfio won't be so difficult.

Or can you take up responsibility of upstreaming pci mmap patch, then we don't
need this patch.

[1] http://dpdk.org/ml/archives/dev/2016-January/031313.html
> > sense.
> >
  

On 12 May 2016 at 13:06, Santosh Shukla
<santosh.shukla@caviumnetworks.com> wrote:
> On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:
>> On 12 May 2016 at 11:17, Santosh Shukla
>> <santosh.shukla@caviumnetworks.com> wrote:
>> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
>> >> On 12 May 2016 at 02:25, Stephen Hemminger <stephen@networkplumber.org> wrote:
>> >> > On Wed, 11 May 2016 22:32:16 +0530
>> >> > Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
>> >> >
>> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
>> >> >> > On Wed, 11 May 2016 19:17:58 +0530
>> >> >> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
>> >> >> >
>> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
>> >> >> > >
>> >> >> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
>> >> >> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
>> >> >> >
>> >> >> > Really, I have use IGB_UIO on ARM64
>> >> >>
>> >> >> May I know what is the technical use case for igb_uio on arm64
>> >> >> which cannot be addressed through vfio or vfioionommu.
>> >> >
>> >> > I was running on older kernel which did not support vfioionommu mode.
>> >>
>> >> As I said, most of DPDK developers are not kernel developers. They may
>> >> have their own kernel tree, and couldn't like to upgrade to latest
>> >> kernel.
>> >> They can choose to use or not use igb_uio when binding the driver. But
>> >> blindly disabling it in the base config seems unreasonable.
>> >
>> > if user keeping his own kernel so they could also keep IGB_UIO=y in their local
>> Most likely they don't have local dpdk tree. They write their own
>> applications, complie and link to dpdk lib, then done.
>>
>> > dpdk tree. Why are you imposing user-x custome depedancy on upstream dpdk base
>> Customer requiremnts is important. I want they can choose the way they like.
>>
>
> so you choose to keep igb_uio option, provided arch doesn't support?
> new user did reported issues with igb_uio for arm64, refer this thread [1], as
> well hemanth too faced issues. we want to avoid that.
>
> If customer maintaing out-of-tree kernel then he can also switch to vfio-way.
> isn;t it?
>
>> > config. Is it not enough for explanation that - Base config ie.. armv8 doesn;t
>> > support pci mmap, so igb_uio is n/a. New user wont able to build/run dpdk/arm64
>> > in igb_uio-way, He'll prefer to use upstream stuff. I think, you are not making
>> You are wrong, he can build dpdk. If he like to use upstream without
>> patching, he can use vfio.
>
> I disagree, we want to avoid [1] for new user.
>
>> But you can't ignore the need from old user which is more comfortable
>> with older kernel.
>>
> arm/arm64 dpdk support recently added and I am guessing, most likely customer
> using near latest kernel, switching to vfio won't be so difficult.
>
> Or can you take up responsibility of upstreaming pci mmap patch, then we don't
> need this patch.
>
> [1] http://dpdk.org/ml/archives/dev/2016-January/031313.html

Can you read carefully about the guide at
http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html? It says to use
uio_pci_generic, igb_uio or vfio-pci.
Could it be possible that the user in that thread has already read and
tried them all and found that he can't enable vifo with his kernel,
and igb_uio is the easy way for him and asked for help from community?
If so, we have no choice but keeping igb_uio enabled.

He use lsmod to show us the modules, most likely he know vifo-pci.

Below are the details on modules, hugepages and device binding.
root at arm64:~# lsmod
Module                  Size  Used by
rte_kni               292795  0
igb_uio                 4338  0
ixgbe                 184456  0
  

On Thu, May 12, 2016 at 01:54:13PM +0800, Jianbo Liu wrote:
> On 12 May 2016 at 13:06, Santosh Shukla
> <santosh.shukla@caviumnetworks.com> wrote:
> > On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:
> >> On 12 May 2016 at 11:17, Santosh Shukla
> >> <santosh.shukla@caviumnetworks.com> wrote:
> >> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
> >> >> On 12 May 2016 at 02:25, Stephen Hemminger <stephen@networkplumber.org> wrote:
> >> >> > On Wed, 11 May 2016 22:32:16 +0530
> >> >> > Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
> >> >> >
> >> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
> >> >> >> > On Wed, 11 May 2016 19:17:58 +0530
> >> >> >> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
> >> >> >> >
> >> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
> >> >> >> > >
> >> >> >> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> >> >> >> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> >> >> >> >
> >> >> >> > Really, I have use IGB_UIO on ARM64
> >> >> >>
> >> >> >> May I know what is the technical use case for igb_uio on arm64
> >> >> >> which cannot be addressed through vfio or vfioionommu.
> >> >> >
> >> >> > I was running on older kernel which did not support vfioionommu mode.
> >> >>
> >> >> As I said, most of DPDK developers are not kernel developers. They may
> >> >> have their own kernel tree, and couldn't like to upgrade to latest
> >> >> kernel.
> >> >> They can choose to use or not use igb_uio when binding the driver. But
> >> >> blindly disabling it in the base config seems unreasonable.
> >> >
> >> > if user keeping his own kernel so they could also keep IGB_UIO=y in their local
> >> Most likely they don't have local dpdk tree. They write their own
> >> applications, complie and link to dpdk lib, then done.
> >>
> >> > dpdk tree. Why are you imposing user-x custome depedancy on upstream dpdk base
> >> Customer requiremnts is important. I want they can choose the way they like.
> >>
> >
> > so you choose to keep igb_uio option, provided arch doesn't support?
> > new user did reported issues with igb_uio for arm64, refer this thread [1], as
> > well hemanth too faced issues. we want to avoid that.
> >
> > If customer maintaing out-of-tree kernel then he can also switch to vfio-way.
> > isn;t it?
> >
> >> > config. Is it not enough for explanation that - Base config ie.. armv8 doesn;t
> >> > support pci mmap, so igb_uio is n/a. New user wont able to build/run dpdk/arm64
> >> > in igb_uio-way, He'll prefer to use upstream stuff. I think, you are not making
> >> You are wrong, he can build dpdk. If he like to use upstream without
> >> patching, he can use vfio.
> >
> > I disagree, we want to avoid [1] for new user.
> >
> >> But you can't ignore the need from old user which is more comfortable
> >> with older kernel.
> >>
> > arm/arm64 dpdk support recently added and I am guessing, most likely customer
> > using near latest kernel, switching to vfio won't be so difficult.
> >
> > Or can you take up responsibility of upstreaming pci mmap patch, then we don't
> > need this patch.
> >
> > [1] http://dpdk.org/ml/archives/dev/2016-January/031313.html
> 
> Can you read carefully about the guide at
> http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html? It says to use
> uio_pci_generic, igb_uio or vfio-pci.

*** applicable and works for x86 only, not for arm64: because pci mmap support
not present for arm64, in that case we should update the doc.

> Could it be possible that the user in that thread has already read and
> tried them all and found that he can't enable vifo with his kernel,
> and igb_uio is the easy way for him and asked for help from community?
> If so, we have no choice but keeping igb_uio enabled.

By then vfionoiommu support was wip progress in dpdk/linux. but now it merged
and it works. So no need to retain igb_uio in base config for which to work -
user need to use mmap patch at linux side.

Or can you maintain out-of-tree pci mmap patch/ kerne source and make it
explicit somewhere in dpdk build doc that - if user want igb_uio way then
use kernel/mmap patch from x location.

> He use lsmod to show us the modules, most likely he know vifo-pci.
> 
> Below are the details on modules, hugepages and device binding.
> root at arm64:~# lsmod
> Module                  Size  Used by
> rte_kni               292795  0
> igb_uio                 4338  0
> ixgbe                 184456  0
  

On 12 May 2016 at 16:57, Santosh Shukla
<santosh.shukla@caviumnetworks.com> wrote:
> On Thu, May 12, 2016 at 01:54:13PM +0800, Jianbo Liu wrote:
>> On 12 May 2016 at 13:06, Santosh Shukla
>> <santosh.shukla@caviumnetworks.com> wrote:
>> > On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:
>> >> On 12 May 2016 at 11:17, Santosh Shukla
>> >> <santosh.shukla@caviumnetworks.com> wrote:
>> >> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
>> >> >> On 12 May 2016 at 02:25, Stephen Hemminger <stephen@networkplumber.org> wrote:
>> >> >> > On Wed, 11 May 2016 22:32:16 +0530
>> >> >> > Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
>> >> >> >
>> >> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
>> >> >> >> > On Wed, 11 May 2016 19:17:58 +0530
>> >> >> >> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
>> >> >> >> >
>> >> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
>> >> >> >> > >
>> >> >> >> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
>> >> >> >> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
>> >> >> >> >
>> >> >> >> > Really, I have use IGB_UIO on ARM64
>> >> >> >>
>> >> >> >> May I know what is the technical use case for igb_uio on arm64
>> >> >> >> which cannot be addressed through vfio or vfioionommu.
>> >> >> >
>> >> >> > I was running on older kernel which did not support vfioionommu mode.
>> >> >>
>> >> >> As I said, most of DPDK developers are not kernel developers. They may
>> >> >> have their own kernel tree, and couldn't like to upgrade to latest
>> >> >> kernel.
>> >> >> They can choose to use or not use igb_uio when binding the driver. But
>> >> >> blindly disabling it in the base config seems unreasonable.
>> >> >
>> >> > if user keeping his own kernel so they could also keep IGB_UIO=y in their local
>> >> Most likely they don't have local dpdk tree. They write their own
>> >> applications, complie and link to dpdk lib, then done.
>> >>
>> >> > dpdk tree. Why are you imposing user-x custome depedancy on upstream dpdk base
>> >> Customer requiremnts is important. I want they can choose the way they like.
>> >>
>> >
>> > so you choose to keep igb_uio option, provided arch doesn't support?
>> > new user did reported issues with igb_uio for arm64, refer this thread [1], as
>> > well hemanth too faced issues. we want to avoid that.
>> >
>> > If customer maintaing out-of-tree kernel then he can also switch to vfio-way.
>> > isn;t it?
>> >
>> >> > config. Is it not enough for explanation that - Base config ie.. armv8 doesn;t
>> >> > support pci mmap, so igb_uio is n/a. New user wont able to build/run dpdk/arm64
>> >> > in igb_uio-way, He'll prefer to use upstream stuff. I think, you are not making
>> >> You are wrong, he can build dpdk. If he like to use upstream without
>> >> patching, he can use vfio.
>> >
>> > I disagree, we want to avoid [1] for new user.
>> >
>> >> But you can't ignore the need from old user which is more comfortable
>> >> with older kernel.
>> >>
>> > arm/arm64 dpdk support recently added and I am guessing, most likely customer
>> > using near latest kernel, switching to vfio won't be so difficult.
>> >
>> > Or can you take up responsibility of upstreaming pci mmap patch, then we don't
>> > need this patch.
>> >
>> > [1] http://dpdk.org/ml/archives/dev/2016-January/031313.html
>>
>> Can you read carefully about the guide at
>> http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html? It says to use
>> uio_pci_generic, igb_uio or vfio-pci.
>
> *** applicable and works for x86 only, not for arm64: because pci mmap support
> not present for arm64, in that case we should update the doc.
>
>> Could it be possible that the user in that thread has already read and
>> tried them all and found that he can't enable vifo with his kernel,
>> and igb_uio is the easy way for him and asked for help from community?
>> If so, we have no choice but keeping igb_uio enabled.
>
> By then vfionoiommu support was wip progress in dpdk/linux. but now it merged
> and it works. So no need to retain igb_uio in base config for which to work -
> user need to use mmap patch at linux side.

We can't decide which kernel user will use.

>
> Or can you maintain out-of-tree pci mmap patch/ kerne source and make it
> explicit somewhere in dpdk build doc that - if user want igb_uio way then
> use kernel/mmap patch from x location.

The patch is in the kernel maillist, and user google it.
And isn't funny to ask someone to do something again and again (3
times) in this thread?

>
>> He use lsmod to show us the modules, most likely he know vifo-pci.
>>
>> Below are the details on modules, hugepages and device binding.
>> root at arm64:~# lsmod
>> Module                  Size  Used by
>> rte_kni               292795  0
>> igb_uio                 4338  0
>> ixgbe                 184456  0
  

On Thu, May 12, 2016 at 05:52:54PM +0800, Jianbo Liu wrote:
> On 12 May 2016 at 16:57, Santosh Shukla
> <santosh.shukla@caviumnetworks.com> wrote:
> > On Thu, May 12, 2016 at 01:54:13PM +0800, Jianbo Liu wrote:
> >> On 12 May 2016 at 13:06, Santosh Shukla
> >> <santosh.shukla@caviumnetworks.com> wrote:
> >> > On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:
> >> >> On 12 May 2016 at 11:17, Santosh Shukla
> >> >> <santosh.shukla@caviumnetworks.com> wrote:
> >> >> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
> >> >> >> On 12 May 2016 at 02:25, Stephen Hemminger <stephen@networkplumber.org> wrote:
> >> >> >> > On Wed, 11 May 2016 22:32:16 +0530
> >> >> >> > Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
> >> >> >> >
> >> >> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
> >> >> >> >> > On Wed, 11 May 2016 19:17:58 +0530
> >> >> >> >> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
> >> >> >> >> >
> >> >> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
> >> >> >> >> > >
> >> >> >> >> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> >> >> >> >> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> >> >> >> >> >
> >> >> >> >> > Really, I have use IGB_UIO on ARM64
> >> >> >> >>
> >> >> >> >> May I know what is the technical use case for igb_uio on arm64
> >> >> >> >> which cannot be addressed through vfio or vfioionommu.
> >> >> >> >
> >> >> >> > I was running on older kernel which did not support vfioionommu mode.
> >> >> >>
> >> >> >> As I said, most of DPDK developers are not kernel developers. They may
> >> >> >> have their own kernel tree, and couldn't like to upgrade to latest
> >> >> >> kernel.
> >> >> >> They can choose to use or not use igb_uio when binding the driver. But
> >> >> >> blindly disabling it in the base config seems unreasonable.
> >> >> >
> >> >> > if user keeping his own kernel so they could also keep IGB_UIO=y in their local
> >> >> Most likely they don't have local dpdk tree. They write their own
> >> >> applications, complie and link to dpdk lib, then done.
> >> >>
> >> >> > dpdk tree. Why are you imposing user-x custome depedancy on upstream dpdk base
> >> >> Customer requiremnts is important. I want they can choose the way they like.
> >> >>
> >> >
> >> > so you choose to keep igb_uio option, provided arch doesn't support?
> >> > new user did reported issues with igb_uio for arm64, refer this thread [1], as
> >> > well hemanth too faced issues. we want to avoid that.
> >> >
> >> > If customer maintaing out-of-tree kernel then he can also switch to vfio-way.
> >> > isn;t it?
> >> >
> >> >> > config. Is it not enough for explanation that - Base config ie.. armv8 doesn;t
> >> >> > support pci mmap, so igb_uio is n/a. New user wont able to build/run dpdk/arm64
> >> >> > in igb_uio-way, He'll prefer to use upstream stuff. I think, you are not making
> >> >> You are wrong, he can build dpdk. If he like to use upstream without
> >> >> patching, he can use vfio.
> >> >
> >> > I disagree, we want to avoid [1] for new user.
> >> >
> >> >> But you can't ignore the need from old user which is more comfortable
> >> >> with older kernel.
> >> >>
> >> > arm/arm64 dpdk support recently added and I am guessing, most likely customer
> >> > using near latest kernel, switching to vfio won't be so difficult.
> >> >
> >> > Or can you take up responsibility of upstreaming pci mmap patch, then we don't
> >> > need this patch.
> >> >
> >> > [1] http://dpdk.org/ml/archives/dev/2016-January/031313.html
> >>
> >> Can you read carefully about the guide at
> >> http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html? It says to use
> >> uio_pci_generic, igb_uio or vfio-pci.
> >
> > *** applicable and works for x86 only, not for arm64: because pci mmap support
> > not present for arm64, in that case we should update the doc.
> >
> >> Could it be possible that the user in that thread has already read and
> >> tried them all and found that he can't enable vifo with his kernel,
> >> and igb_uio is the easy way for him and asked for help from community?
> >> If so, we have no choice but keeping igb_uio enabled.
> >
> > By then vfionoiommu support was wip progress in dpdk/linux. but now it merged
> > and it works. So no need to retain igb_uio in base config for which to work -
> > user need to use mmap patch at linux side.
> 
> We can't decide which kernel user will use.
>

yes, we can't decide kernel for user but we should be explicit to user on - what
works for dpdk/linux out-of-box vs what could work with use of out-of-tree
patch/RFC's.. example igb_uio.

> >
> > Or can you maintain out-of-tree pci mmap patch/ kerne source and make it
> > explicit somewhere in dpdk build doc that - if user want igb_uio way then
> > use kernel/mmap patch from x location.
> 
> The patch is in the kernel maillist, and user google it.

there are feature specific rfc's in plenty in lkml/qemu mailing list,  and you
suggest- user to hunt for all those information. Is this how we;re officially
supporting igb_uio for arm64.. that let user to google?

> And isn't funny to ask someone to do something again and again (3
> times) in this thread?
>

I am asking becasue your in favour of keeping igb_uio for arm64 but not
agreeing to streamline (writing a note in dpdk doc for igb_uio for arm64 or
pointing to working tree).. so that user don;t need to grep or google for known
findings. 

I find discussion going in circle and nothing will conclude, So given up.
> >
> >> He use lsmod to show us the modules, most likely he know vifo-pci.
> >>
> >> Below are the details on modules, hugepages and device binding.
> >> root at arm64:~# lsmod
> >> Module                  Size  Used by
> >> rte_kni               292795  0
> >> igb_uio                 4338  0
> >> ixgbe                 184456  0
  

On 12 May 2016 at 18:31, Santosh Shukla
<santosh.shukla@caviumnetworks.com> wrote:
> On Thu, May 12, 2016 at 05:52:54PM +0800, Jianbo Liu wrote:
>> On 12 May 2016 at 16:57, Santosh Shukla
>> <santosh.shukla@caviumnetworks.com> wrote:
>> > On Thu, May 12, 2016 at 01:54:13PM +0800, Jianbo Liu wrote:
>> >> On 12 May 2016 at 13:06, Santosh Shukla
>> >> <santosh.shukla@caviumnetworks.com> wrote:
>> >> > On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:
>> >> >> On 12 May 2016 at 11:17, Santosh Shukla
>> >> >> <santosh.shukla@caviumnetworks.com> wrote:
>> >> >> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
>> >> >> >> On 12 May 2016 at 02:25, Stephen Hemminger <stephen@networkplumber.org> wrote:
>> >> >> >> > On Wed, 11 May 2016 22:32:16 +0530
>> >> >> >> > Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
>> >> >> >> >
>> >> >> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen Hemminger wrote:
>> >> >> >> >> > On Wed, 11 May 2016 19:17:58 +0530
>> >> >> >> >> > Hemant Agrawal <hemant.agrawal@nxp.com> wrote:
>> >> >> >> >> >
>> >> >> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
>> >> >> >> >> > >
>> >> >> >> >> > > Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
>> >> >> >> >> > > Reviewed-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
>> >> >> >> >> >
>> >> >> >> >> > Really, I have use IGB_UIO on ARM64
>> >> >> >> >>
>> >> >> >> >> May I know what is the technical use case for igb_uio on arm64
>> >> >> >> >> which cannot be addressed through vfio or vfioionommu.
>> >> >> >> >
>> >> >> >> > I was running on older kernel which did not support vfioionommu mode.
>> >> >> >>
>> >> >> >> As I said, most of DPDK developers are not kernel developers. They may
>> >> >> >> have their own kernel tree, and couldn't like to upgrade to latest
>> >> >> >> kernel.
>> >> >> >> They can choose to use or not use igb_uio when binding the driver. But
>> >> >> >> blindly disabling it in the base config seems unreasonable.
>> >> >> >
>> >> >> > if user keeping his own kernel so they could also keep IGB_UIO=y in their local
>> >> >> Most likely they don't have local dpdk tree. They write their own
>> >> >> applications, complie and link to dpdk lib, then done.
>> >> >>
>> >> >> > dpdk tree. Why are you imposing user-x custome depedancy on upstream dpdk base
>> >> >> Customer requiremnts is important. I want they can choose the way they like.
>> >> >>
>> >> >
>> >> > so you choose to keep igb_uio option, provided arch doesn't support?
>> >> > new user did reported issues with igb_uio for arm64, refer this thread [1], as
>> >> > well hemanth too faced issues. we want to avoid that.
>> >> >
>> >> > If customer maintaing out-of-tree kernel then he can also switch to vfio-way.
>> >> > isn;t it?
>> >> >
>> >> >> > config. Is it not enough for explanation that - Base config ie.. armv8 doesn;t
>> >> >> > support pci mmap, so igb_uio is n/a. New user wont able to build/run dpdk/arm64
>> >> >> > in igb_uio-way, He'll prefer to use upstream stuff. I think, you are not making
>> >> >> You are wrong, he can build dpdk. If he like to use upstream without
>> >> >> patching, he can use vfio.
>> >> >
>> >> > I disagree, we want to avoid [1] for new user.
>> >> >
>> >> >> But you can't ignore the need from old user which is more comfortable
>> >> >> with older kernel.
>> >> >>
>> >> > arm/arm64 dpdk support recently added and I am guessing, most likely customer
>> >> > using near latest kernel, switching to vfio won't be so difficult.
>> >> >
>> >> > Or can you take up responsibility of upstreaming pci mmap patch, then we don't
>> >> > need this patch.
>> >> >
>> >> > [1] http://dpdk.org/ml/archives/dev/2016-January/031313.html
>> >>
>> >> Can you read carefully about the guide at
>> >> http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html? It says to use
>> >> uio_pci_generic, igb_uio or vfio-pci.
>> >
>> > *** applicable and works for x86 only, not for arm64: because pci mmap support
>> > not present for arm64, in that case we should update the doc.
>> >
>> >> Could it be possible that the user in that thread has already read and
>> >> tried them all and found that he can't enable vifo with his kernel,
>> >> and igb_uio is the easy way for him and asked for help from community?
>> >> If so, we have no choice but keeping igb_uio enabled.
>> >
>> > By then vfionoiommu support was wip progress in dpdk/linux. but now it merged
>> > and it works. So no need to retain igb_uio in base config for which to work -
>> > user need to use mmap patch at linux side.
>>
>> We can't decide which kernel user will use.
>>
>
> yes, we can't decide kernel for user but we should be explicit to user on - what
> works for dpdk/linux out-of-box vs what could work with use of out-of-tree
> patch/RFC's.. example igb_uio.
>

OK, please persuade Stephen Hemminger and the other guy to use
upstream kernel first!

>> >
>> > Or can you maintain out-of-tree pci mmap patch/ kerne source and make it
>> > explicit somewhere in dpdk build doc that - if user want igb_uio way then
>> > use kernel/mmap patch from x location.
>>
>> The patch is in the kernel maillist, and user google it.
>
> there are feature specific rfc's in plenty in lkml/qemu mailing list,  and you
> suggest- user to hunt for all those information. Is this how we;re officially
> supporting igb_uio for arm64.. that let user to google?
>

Sorry I don't know you are offically support users here.
And you also don't know what they really want.

>> And isn't funny to ask someone to do something again and again (3
>> times) in this thread?
>>
>
> I am asking becasue your in favour of keeping igb_uio for arm64 but not
> agreeing to streamline (writing a note in dpdk doc for igb_uio for arm64 or
> pointing to working tree).. so that user don;t need to grep or google for known
> findings.
>
> I find discussion going in circle and nothing will conclude, So given up.
>> >
>> >> He use lsmod to show us the modules, most likely he know vifo-pci.
>> >>
>> >> Below are the details on modules, hugepages and device binding.
>> >> root at arm64:~# lsmod
>> >> Module                  Size  Used by
>> >> rte_kni               292795  0
>> >> igb_uio                 4338  0
>> >> ixgbe                 184456  0
  

> -----Original Message-----

> From: Jianbo Liu [mailto:jianbo.liu@linaro.org]

> Sent: Friday, May 13, 2016 7:13 AM

> To: Santosh Shukla <santosh.shukla@caviumnetworks.com>

> Cc: Stephen Hemminger <stephen@networkplumber.org>; Jerin Jacob

> <jerin.jacob@caviumnetworks.com>; Hemant Agrawal

> <hemant.agrawal@nxp.com>; dev@dpdk.org; Thomas Monjalon

> <thomas.monjalon@6wind.com>

> Subject: Re: [dpdk-dev] [PATCHv3 1/2] config/armv8a: disable igb_uio

> 

> On 12 May 2016 at 18:31, Santosh Shukla

> <santosh.shukla@caviumnetworks.com> wrote:

> > On Thu, May 12, 2016 at 05:52:54PM +0800, Jianbo Liu wrote:

> >> On 12 May 2016 at 16:57, Santosh Shukla

> >> <santosh.shukla@caviumnetworks.com> wrote:

> >> > On Thu, May 12, 2016 at 01:54:13PM +0800, Jianbo Liu wrote:

> >> >> On 12 May 2016 at 13:06, Santosh Shukla

> >> >> <santosh.shukla@caviumnetworks.com> wrote:

> >> >> > On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:

> >> >> >> On 12 May 2016 at 11:17, Santosh Shukla

> >> >> >> <santosh.shukla@caviumnetworks.com> wrote:

> >> >> >> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:

> >> >> >> >> On 12 May 2016 at 02:25, Stephen Hemminger

> <stephen@networkplumber.org> wrote:

> >> >> >> >> > On Wed, 11 May 2016 22:32:16 +0530 Jerin Jacob

> >> >> >> >> > <jerin.jacob@caviumnetworks.com> wrote:

> >> >> >> >> >

> >> >> >> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen

> Hemminger wrote:

> >> >> >> >> >> > On Wed, 11 May 2016 19:17:58 +0530 Hemant Agrawal

> >> >> >> >> >> > <hemant.agrawal@nxp.com> wrote:

> >> >> >> >> >> >

> >> >> >> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.

> >> >> >> >> >> > >

> >> >> >> >> >> > > Signed-off-by: Hemant Agrawal

> >> >> >> >> >> > > <hemant.agrawal@nxp.com>

> >> >> >> >> >> > > Reviewed-by: Santosh Shukla

> >> >> >> >> >> > > <santosh.shukla@caviumnetworks.com>

> >> >> >> >> >> >

> >> >> >> >> >> > Really, I have use IGB_UIO on ARM64

> >> >> >> >> >>

> >> >> >> >> >> May I know what is the technical use case for igb_uio on

> >> >> >> >> >> arm64 which cannot be addressed through vfio or vfioionommu.

> >> >> >> >> >

> >> >> >> >> > I was running on older kernel which did not support vfioionommu

> mode.

> >> >> >> >>

> >> >> >> >> As I said, most of DPDK developers are not kernel

> >> >> >> >> developers. They may have their own kernel tree, and

> >> >> >> >> couldn't like to upgrade to latest kernel.

> >> >> >> >> They can choose to use or not use igb_uio when binding the

> >> >> >> >> driver. But blindly disabling it in the base config seems unreasonable.

> >> >> >> >

> >> >> >> > if user keeping his own kernel so they could also keep

> >> >> >> > IGB_UIO=y in their local

> >> >> >> Most likely they don't have local dpdk tree. They write their

> >> >> >> own applications, complie and link to dpdk lib, then done.

> >> >> >>

> >> >> >> > dpdk tree. Why are you imposing user-x custome depedancy on

> >> >> >> > upstream dpdk base

> >> >> >> Customer requiremnts is important. I want they can choose the way

> they like.

> >> >> >>

> >> >> >

> >> >> > so you choose to keep igb_uio option, provided arch doesn't support?

> >> >> > new user did reported issues with igb_uio for arm64, refer this

> >> >> > thread [1], as well hemanth too faced issues. we want to avoid that.

> >> >> >

> >> >> > If customer maintaing out-of-tree kernel then he can also switch to vfio-

> way.

> >> >> > isn;t it?

> >> >> >

> >> >> >> > config. Is it not enough for explanation that - Base config

> >> >> >> > ie.. armv8 doesn;t support pci mmap, so igb_uio is n/a. New

> >> >> >> > user wont able to build/run dpdk/arm64 in igb_uio-way, He'll

> >> >> >> > prefer to use upstream stuff. I think, you are not making

> >> >> >> You are wrong, he can build dpdk. If he like to use upstream

> >> >> >> without patching, he can use vfio.

> >> >> >

> >> >> > I disagree, we want to avoid [1] for new user.

> >> >> >

> >> >> >> But you can't ignore the need from old user which is more

> >> >> >> comfortable with older kernel.

> >> >> >>

> >> >> > arm/arm64 dpdk support recently added and I am guessing, most

> >> >> > likely customer using near latest kernel, switching to vfio won't be so

> difficult.

> >> >> >

> >> >> > Or can you take up responsibility of upstreaming pci mmap patch,

> >> >> > then we don't need this patch.

> >> >> >

> >> >> > [1] http://dpdk.org/ml/archives/dev/2016-January/031313.html

> >> >>

> >> >> Can you read carefully about the guide at

> >> >> http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html? It says to

> >> >> use uio_pci_generic, igb_uio or vfio-pci.

> >> >

> >> > *** applicable and works for x86 only, not for arm64: because pci

> >> > mmap support not present for arm64, in that case we should update the

> doc.

> >> >

> >> >> Could it be possible that the user in that thread has already read

> >> >> and tried them all and found that he can't enable vifo with his

> >> >> kernel, and igb_uio is the easy way for him and asked for help from

> community?

> >> >> If so, we have no choice but keeping igb_uio enabled.

> >> >

> >> > By then vfionoiommu support was wip progress in dpdk/linux. but now

> >> > it merged and it works. So no need to retain igb_uio in base config

> >> > for which to work - user need to use mmap patch at linux side.

> >>

> >> We can't decide which kernel user will use.

> >>

> >

> > yes, we can't decide kernel for user but we should be explicit to user

> > on - what works for dpdk/linux out-of-box vs what could work with use

> > of out-of-tree patch/RFC's.. example igb_uio.

> >

> 

> OK, please persuade Stephen Hemminger and the other guy to use upstream

> kernel first!


[Hemant] It is just a matter for choice for arm64 maintainers.  You only have two choices
1. with the default arm64 support in DPDK, you allow it to be used with any kernel seamlessly. If they need any specific function, they can enable/disable it. 
2. Or, you maintain specific support in the default config, and force all others to manually disable it/or, disable it via their specific configuration.

I was not intending to change the default arm64 config in my original patch, I rather used it to disable it via NXP specific platform config.
I still agree with Santosh that a new arm64 user should not be worry about searching  & patching the kernel. 


In any case, who is going to take a call here?  
> 

> >> >

> >> > Or can you maintain out-of-tree pci mmap patch/ kerne source and

> >> > make it explicit somewhere in dpdk build doc that - if user want

> >> > igb_uio way then use kernel/mmap patch from x location.

> >>

> >> The patch is in the kernel maillist, and user google it.

> >

> > there are feature specific rfc's in plenty in lkml/qemu mailing list,

> > and you

> > suggest- user to hunt for all those information. Is this how we;re

> > officially supporting igb_uio for arm64.. that let user to google?

> >

> 

> Sorry I don't know you are offically support users here.

> And you also don't know what they really want.

> 

> >> And isn't funny to ask someone to do something again and again (3

> >> times) in this thread?

> >>

> >

> > I am asking becasue your in favour of keeping igb_uio for arm64 but

> > not agreeing to streamline (writing a note in dpdk doc for igb_uio for

> > arm64 or pointing to working tree).. so that user don;t need to grep

> > or google for known findings.

> >

> > I find discussion going in circle and nothing will conclude, So given up.

> >> >

> >> >> He use lsmod to show us the modules, most likely he know vifo-pci.

> >> >>

> >> >> Below are the details on modules, hugepages and device binding.

> >> >> root at arm64:~# lsmod

> >> >> Module                  Size  Used by

> >> >> rte_kni               292795  0

> >> >> igb_uio                 4338  0

> >> >> ixgbe                 184456  0
  

On Fri, May 13, 2016 at 03:37:01AM +0000, Hemant Agrawal wrote:
> 
> 
> > -----Original Message-----
> > From: Jianbo Liu [mailto:jianbo.liu@linaro.org]
> > Sent: Friday, May 13, 2016 7:13 AM
> > To: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> > Cc: Stephen Hemminger <stephen@networkplumber.org>; Jerin Jacob
> > <jerin.jacob@caviumnetworks.com>; Hemant Agrawal
> > <hemant.agrawal@nxp.com>; dev@dpdk.org; Thomas Monjalon
> > <thomas.monjalon@6wind.com>
> > Subject: Re: [dpdk-dev] [PATCHv3 1/2] config/armv8a: disable igb_uio
> > 
> > On 12 May 2016 at 18:31, Santosh Shukla
> > <santosh.shukla@caviumnetworks.com> wrote:
> > > On Thu, May 12, 2016 at 05:52:54PM +0800, Jianbo Liu wrote:
> > >> On 12 May 2016 at 16:57, Santosh Shukla
> > >> <santosh.shukla@caviumnetworks.com> wrote:
> > >> > On Thu, May 12, 2016 at 01:54:13PM +0800, Jianbo Liu wrote:
> > >> >> On 12 May 2016 at 13:06, Santosh Shukla
> > >> >> <santosh.shukla@caviumnetworks.com> wrote:
> > >> >> > On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:
> > >> >> >> On 12 May 2016 at 11:17, Santosh Shukla
> > >> >> >> <santosh.shukla@caviumnetworks.com> wrote:
> > >> >> >> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
> > >> >> >> >> On 12 May 2016 at 02:25, Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >> >> >> >> > On Wed, 11 May 2016 22:32:16 +0530 Jerin Jacob
> > >> >> >> >> > <jerin.jacob@caviumnetworks.com> wrote:
> > >> >> >> >> >
> > >> >> >> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen
> > Hemminger wrote:
> > >> >> >> >> >> > On Wed, 11 May 2016 19:17:58 +0530 Hemant Agrawal
> > >> >> >> >> >> > <hemant.agrawal@nxp.com> wrote:
> > >> >> >> >> >> >
> > >> >> >> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
> > >> >> >> >> >> > >
> > >> >> >> >> >> > > Signed-off-by: Hemant Agrawal
> > >> >> >> >> >> > > <hemant.agrawal@nxp.com>
> > >> >> >> >> >> > > Reviewed-by: Santosh Shukla
> > >> >> >> >> >> > > <santosh.shukla@caviumnetworks.com>
> > >> >> >> >> >> >
> > >> >> >> >> >> > Really, I have use IGB_UIO on ARM64
> > >> >> >> >> >>
> > >> >> >> >> >> May I know what is the technical use case for igb_uio on
> > >> >> >> >> >> arm64 which cannot be addressed through vfio or vfioionommu.
> > >> >> >> >> >
> > >> >> >> >> > I was running on older kernel which did not support vfioionommu
> > mode.
> > >> >> >> >>
> > >> >> >> >> As I said, most of DPDK developers are not kernel
> > >> >> >> >> developers. They may have their own kernel tree, and
> > >> >> >> >> couldn't like to upgrade to latest kernel.
> > >> >> >> >> They can choose to use or not use igb_uio when binding the
> > >> >> >> >> driver. But blindly disabling it in the base config seems unreasonable.
> > >> >> >> >
> > >> >> >> > if user keeping his own kernel so they could also keep
> > >> >> >> > IGB_UIO=y in their local
> > >> >> >> Most likely they don't have local dpdk tree. They write their
> > >> >> >> own applications, complie and link to dpdk lib, then done.
> > >> >> >>
> > >> >> >> > dpdk tree. Why are you imposing user-x custome depedancy on
> > >> >> >> > upstream dpdk base
> > >> >> >> Customer requiremnts is important. I want they can choose the way
> > they like.
> > >> >> >>
> > >> >> >
> > >> >> > so you choose to keep igb_uio option, provided arch doesn't support?
> > >> >> > new user did reported issues with igb_uio for arm64, refer this
> > >> >> > thread [1], as well hemanth too faced issues. we want to avoid that.
> > >> >> >
> > >> >> > If customer maintaing out-of-tree kernel then he can also switch to vfio-
> > way.
> > >> >> > isn;t it?
> > >> >> >
> > >> >> >> > config. Is it not enough for explanation that - Base config
> > >> >> >> > ie.. armv8 doesn;t support pci mmap, so igb_uio is n/a. New
> > >> >> >> > user wont able to build/run dpdk/arm64 in igb_uio-way, He'll
> > >> >> >> > prefer to use upstream stuff. I think, you are not making
> > >> >> >> You are wrong, he can build dpdk. If he like to use upstream
> > >> >> >> without patching, he can use vfio.
> > >> >> >
> > >> >> > I disagree, we want to avoid [1] for new user.
> > >> >> >
> > >> >> >> But you can't ignore the need from old user which is more
> > >> >> >> comfortable with older kernel.
> > >> >> >>
> > >> >> > arm/arm64 dpdk support recently added and I am guessing, most
> > >> >> > likely customer using near latest kernel, switching to vfio won't be so
> > difficult.
> > >> >> >
> > >> >> > Or can you take up responsibility of upstreaming pci mmap patch,
> > >> >> > then we don't need this patch.
> > >> >> >
> > >> >> > [1] http://dpdk.org/ml/archives/dev/2016-January/031313.html
> > >> >>
> > >> >> Can you read carefully about the guide at
> > >> >> http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html? It says to
> > >> >> use uio_pci_generic, igb_uio or vfio-pci.
> > >> >
> > >> > *** applicable and works for x86 only, not for arm64: because pci
> > >> > mmap support not present for arm64, in that case we should update the
> > doc.
> > >> >
> > >> >> Could it be possible that the user in that thread has already read
> > >> >> and tried them all and found that he can't enable vifo with his
> > >> >> kernel, and igb_uio is the easy way for him and asked for help from
> > community?
> > >> >> If so, we have no choice but keeping igb_uio enabled.
> > >> >
> > >> > By then vfionoiommu support was wip progress in dpdk/linux. but now
> > >> > it merged and it works. So no need to retain igb_uio in base config
> > >> > for which to work - user need to use mmap patch at linux side.
> > >>
> > >> We can't decide which kernel user will use.
> > >>
> > >
> > > yes, we can't decide kernel for user but we should be explicit to user
> > > on - what works for dpdk/linux out-of-box vs what could work with use
> > > of out-of-tree patch/RFC's.. example igb_uio.
> > >
> > 
> > OK, please persuade Stephen Hemminger and the other guy to use upstream
> > kernel first!
> 
> [Hemant] It is just a matter for choice for arm64 maintainers.  You only have two choices
> 1. with the default arm64 support in DPDK, you allow it to be used with any kernel seamlessly. If they need any specific function, they can enable/disable it. 
> 2. Or, you maintain specific support in the default config, and force all others to manually disable it/or, disable it via their specific configuration.
> 
> I was not intending to change the default arm64 config in my original patch, I rather used it to disable it via NXP specific platform config.
> I still agree with Santosh that a new arm64 user should not be worry about searching  & patching the kernel. 
> 
> 
> In any case, who is going to take a call here?  

My take is to disable for arm64 to make sync with upstream kernel.
If Jianbo still feel it is better to keep for arm64 then lets disable it for
NXP and ThunderX and keep it enable for arm64 def config.

Thanks,
Jerin

> > 
> > >> >
> > >> > Or can you maintain out-of-tree pci mmap patch/ kerne source and
> > >> > make it explicit somewhere in dpdk build doc that - if user want
> > >> > igb_uio way then use kernel/mmap patch from x location.
> > >>
> > >> The patch is in the kernel maillist, and user google it.
> > >
> > > there are feature specific rfc's in plenty in lkml/qemu mailing list,
> > > and you
> > > suggest- user to hunt for all those information. Is this how we;re
> > > officially supporting igb_uio for arm64.. that let user to google?
> > >
> > 
> > Sorry I don't know you are offically support users here.
> > And you also don't know what they really want.
> > 
> > >> And isn't funny to ask someone to do something again and again (3
> > >> times) in this thread?
> > >>
> > >
> > > I am asking becasue your in favour of keeping igb_uio for arm64 but
> > > not agreeing to streamline (writing a note in dpdk doc for igb_uio for
> > > arm64 or pointing to working tree).. so that user don;t need to grep
> > > or google for known findings.
> > >
> > > I find discussion going in circle and nothing will conclude, So given up.
> > >> >
> > >> >> He use lsmod to show us the modules, most likely he know vifo-pci.
> > >> >>
> > >> >> Below are the details on modules, hugepages and device binding.
> > >> >> root at arm64:~# lsmod
> > >> >> Module                  Size  Used by
> > >> >> rte_kni               292795  0
> > >> >> igb_uio                 4338  0
> > >> >> ixgbe                 184456  0
  

On 13 May 2016 at 15:47, Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
> On Fri, May 13, 2016 at 03:37:01AM +0000, Hemant Agrawal wrote:
>>
>>
>> > -----Original Message-----
>> > From: Jianbo Liu [mailto:jianbo.liu@linaro.org]
>> > Sent: Friday, May 13, 2016 7:13 AM
>> > To: Santosh Shukla <santosh.shukla@caviumnetworks.com>
>> > Cc: Stephen Hemminger <stephen@networkplumber.org>; Jerin Jacob
>> > <jerin.jacob@caviumnetworks.com>; Hemant Agrawal
>> > <hemant.agrawal@nxp.com>; dev@dpdk.org; Thomas Monjalon
>> > <thomas.monjalon@6wind.com>
>> > Subject: Re: [dpdk-dev] [PATCHv3 1/2] config/armv8a: disable igb_uio
>> >
>> > On 12 May 2016 at 18:31, Santosh Shukla
>> > <santosh.shukla@caviumnetworks.com> wrote:
>> > > On Thu, May 12, 2016 at 05:52:54PM +0800, Jianbo Liu wrote:
>> > >> On 12 May 2016 at 16:57, Santosh Shukla
>> > >> <santosh.shukla@caviumnetworks.com> wrote:
>> > >> > On Thu, May 12, 2016 at 01:54:13PM +0800, Jianbo Liu wrote:
>> > >> >> On 12 May 2016 at 13:06, Santosh Shukla
>> > >> >> <santosh.shukla@caviumnetworks.com> wrote:
>> > >> >> > On Thu, May 12, 2016 at 11:42:26AM +0800, Jianbo Liu wrote:
>> > >> >> >> On 12 May 2016 at 11:17, Santosh Shukla
>> > >> >> >> <santosh.shukla@caviumnetworks.com> wrote:
>> > >> >> >> > On Thu, May 12, 2016 at 10:01:05AM +0800, Jianbo Liu wrote:
>> > >> >> >> >> On 12 May 2016 at 02:25, Stephen Hemminger
>> > <stephen@networkplumber.org> wrote:
>> > >> >> >> >> > On Wed, 11 May 2016 22:32:16 +0530 Jerin Jacob
>> > >> >> >> >> > <jerin.jacob@caviumnetworks.com> wrote:
>> > >> >> >> >> >
>> > >> >> >> >> >> On Wed, May 11, 2016 at 08:22:59AM -0700, Stephen
>> > Hemminger wrote:
>> > >> >> >> >> >> > On Wed, 11 May 2016 19:17:58 +0530 Hemant Agrawal
>> > >> >> >> >> >> > <hemant.agrawal@nxp.com> wrote:
>> > >> >> >> >> >> >
>> > >> >> >> >> >> > > IGB_UIO not supported for arm64 arch in kernel so disable.
>> > >> >> >> >> >> > >
>> > >> >> >> >> >> > > Signed-off-by: Hemant Agrawal
>> > >> >> >> >> >> > > <hemant.agrawal@nxp.com>
>> > >> >> >> >> >> > > Reviewed-by: Santosh Shukla
>> > >> >> >> >> >> > > <santosh.shukla@caviumnetworks.com>
>> > >> >> >> >> >> >
>> > >> >> >> >> >> > Really, I have use IGB_UIO on ARM64
>> > >> >> >> >> >>
>> > >> >> >> >> >> May I know what is the technical use case for igb_uio on
>> > >> >> >> >> >> arm64 which cannot be addressed through vfio or vfioionommu.
>> > >> >> >> >> >
>> > >> >> >> >> > I was running on older kernel which did not support vfioionommu
>> > mode.
>> > >> >> >> >>
>> > >> >> >> >> As I said, most of DPDK developers are not kernel
>> > >> >> >> >> developers. They may have their own kernel tree, and
>> > >> >> >> >> couldn't like to upgrade to latest kernel.
>> > >> >> >> >> They can choose to use or not use igb_uio when binding the
>> > >> >> >> >> driver. But blindly disabling it in the base config seems unreasonable.
>> > >> >> >> >
>> > >> >> >> > if user keeping his own kernel so they could also keep
>> > >> >> >> > IGB_UIO=y in their local
>> > >> >> >> Most likely they don't have local dpdk tree. They write their
>> > >> >> >> own applications, complie and link to dpdk lib, then done.
>> > >> >> >>
>> > >> >> >> > dpdk tree. Why are you imposing user-x custome depedancy on
>> > >> >> >> > upstream dpdk base
>> > >> >> >> Customer requiremnts is important. I want they can choose the way
>> > they like.
>> > >> >> >>
>> > >> >> >
>> > >> >> > so you choose to keep igb_uio option, provided arch doesn't support?
>> > >> >> > new user did reported issues with igb_uio for arm64, refer this
>> > >> >> > thread [1], as well hemanth too faced issues. we want to avoid that.
>> > >> >> >
>> > >> >> > If customer maintaing out-of-tree kernel then he can also switch to vfio-
>> > way.
>> > >> >> > isn;t it?
>> > >> >> >
>> > >> >> >> > config. Is it not enough for explanation that - Base config
>> > >> >> >> > ie.. armv8 doesn;t support pci mmap, so igb_uio is n/a. New
>> > >> >> >> > user wont able to build/run dpdk/arm64 in igb_uio-way, He'll
>> > >> >> >> > prefer to use upstream stuff. I think, you are not making
>> > >> >> >> You are wrong, he can build dpdk. If he like to use upstream
>> > >> >> >> without patching, he can use vfio.
>> > >> >> >
>> > >> >> > I disagree, we want to avoid [1] for new user.
>> > >> >> >
>> > >> >> >> But you can't ignore the need from old user which is more
>> > >> >> >> comfortable with older kernel.
>> > >> >> >>
>> > >> >> > arm/arm64 dpdk support recently added and I am guessing, most
>> > >> >> > likely customer using near latest kernel, switching to vfio won't be so
>> > difficult.
>> > >> >> >
>> > >> >> > Or can you take up responsibility of upstreaming pci mmap patch,
>> > >> >> > then we don't need this patch.
>> > >> >> >
>> > >> >> > [1] http://dpdk.org/ml/archives/dev/2016-January/031313.html
>> > >> >>
>> > >> >> Can you read carefully about the guide at
>> > >> >> http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html? It says to
>> > >> >> use uio_pci_generic, igb_uio or vfio-pci.
>> > >> >
>> > >> > *** applicable and works for x86 only, not for arm64: because pci
>> > >> > mmap support not present for arm64, in that case we should update the
>> > doc.
>> > >> >
>> > >> >> Could it be possible that the user in that thread has already read
>> > >> >> and tried them all and found that he can't enable vifo with his
>> > >> >> kernel, and igb_uio is the easy way for him and asked for help from
>> > community?
>> > >> >> If so, we have no choice but keeping igb_uio enabled.
>> > >> >
>> > >> > By then vfionoiommu support was wip progress in dpdk/linux. but now
>> > >> > it merged and it works. So no need to retain igb_uio in base config
>> > >> > for which to work - user need to use mmap patch at linux side.
>> > >>
>> > >> We can't decide which kernel user will use.
>> > >>
>> > >
>> > > yes, we can't decide kernel for user but we should be explicit to user
>> > > on - what works for dpdk/linux out-of-box vs what could work with use
>> > > of out-of-tree patch/RFC's.. example igb_uio.
>> > >
>> >
>> > OK, please persuade Stephen Hemminger and the other guy to use upstream
>> > kernel first!
>>
>> [Hemant] It is just a matter for choice for arm64 maintainers.  You only have two choices
>> 1. with the default arm64 support in DPDK, you allow it to be used with any kernel seamlessly. If they need any specific function, they can enable/disable it.
>> 2. Or, you maintain specific support in the default config, and force all others to manually disable it/or, disable it via their specific configuration.
>>
>> I was not intending to change the default arm64 config in my original patch, I rather used it to disable it via NXP specific platform config.
>> I still agree with Santosh that a new arm64 user should not be worry about searching  & patching the kernel.
>>
>>
>> In any case, who is going to take a call here?
>
> My take is to disable for arm64 to make sync with upstream kernel.
> If Jianbo still feel it is better to keep for arm64 then lets disable it for
> NXP and ThunderX and keep it enable for arm64 def config.
>

Thanks, Jerin.
The reason why I want to keep base config untouched is that users
often update the DPDK more frequently than kernel, and some of them
are still using older kernel.
For new users, we can encourage them to use vfio-pci. We will remove
igb_uio on ARM finally, and hope this transition period is not long.
  

2016-05-11 19:17, Hemant Agrawal:
> IGB_UIO not supported for arm64 arch in kernel so disable.

If I understand well, a patch is needed in the kernel to make
igb_uio works? Please confirm.

In that case, yes, the default configuration should be to disable
igb_uio.
Please note it's just a default to make it work in most common cases.
And yes, the default should focus on recent kernels and future directions.

> +CONFIG_RTE_EAL_IGB_UIO=n
  

On Fri, May 13, 2016 at 02:50:48PM +0200, Thomas Monjalon wrote:
> 2016-05-11 19:17, Hemant Agrawal:
> > IGB_UIO not supported for arm64 arch in kernel so disable.
> 
> If I understand well, a patch is needed in the kernel to make
> igb_uio works? Please confirm.
>

Yes. User need this [1] out-of-tree patch for igb_uio.

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2015-July/359435.html

> In that case, yes, the default configuration should be to disable
> igb_uio.
> Please note it's just a default to make it work in most common cases.
> And yes, the default should focus on recent kernels and future directions.
> 
> > +CONFIG_RTE_EAL_IGB_UIO=n
>
  

2016-05-13 18:41, Santosh Shukla:
> On Fri, May 13, 2016 at 02:50:48PM +0200, Thomas Monjalon wrote:
> > 2016-05-11 19:17, Hemant Agrawal:
> > > IGB_UIO not supported for arm64 arch in kernel so disable.
> > 
> > If I understand well, a patch is needed in the kernel to make
> > igb_uio works? Please confirm.
> >
> 
> Yes. User need this [1] out-of-tree patch for igb_uio.
> 
> [1] http://lists.infradead.org/pipermail/linux-arm-kernel/2015-July/359435.html
> 
> > In that case, yes, the default configuration should be to disable
> > igb_uio.
> > Please note it's just a default to make it work in most common cases.
> > And yes, the default should focus on recent kernels and future directions.
> > 
> > > +CONFIG_RTE_EAL_IGB_UIO=n

Series applied, thanks