[dpdk-dev,v6,1/8] eal: pci: add api to rd/wr pci bar region

  Introducing below api for pci bar region rd/wr.
Api's are:
- rte_eal_pci_read_bar
- rte_eal_pci_write_bar

Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
v5-->v6:
- update api infor in rte_eal_version.map file
  suggested by david manchand.

 lib/librte_eal/bsdapp/eal/eal_pci.c             |   19 ++++++++++++
 lib/librte_eal/bsdapp/eal/rte_eal_version.map   |    3 ++
 lib/librte_eal/common/include/rte_pci.h         |   38 +++++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_pci.c           |   34 ++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_pci_init.h      |    6 ++++
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c      |   28 +++++++++++++++++
 lib/librte_eal/linuxapp/eal/rte_eal_version.map |    3 ++
 7 files changed, 131 insertions(+)
  

On Fri, Jan 29, 2016 at 11:51:50PM +0530, Santosh Shukla wrote:
> Introducing below api for pci bar region rd/wr.
> Api's are:
> - rte_eal_pci_read_bar
> - rte_eal_pci_write_bar
> 
> Signed-off-by: Santosh Shukla <sshukla@mvista.com>
> ---
> v5-->v6:
> - update api infor in rte_eal_version.map file
>   suggested by david manchand.
> 
>  lib/librte_eal/bsdapp/eal/eal_pci.c             |   19 ++++++++++++
>  lib/librte_eal/bsdapp/eal/rte_eal_version.map   |    3 ++
>  lib/librte_eal/common/include/rte_pci.h         |   38 +++++++++++++++++++++++
>  lib/librte_eal/linuxapp/eal/eal_pci.c           |   34 ++++++++++++++++++++
>  lib/librte_eal/linuxapp/eal/eal_pci_init.h      |    6 ++++
>  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c      |   28 +++++++++++++++++
>  lib/librte_eal/linuxapp/eal/rte_eal_version.map |    3 ++
>  7 files changed, 131 insertions(+)
> 
> diff --git a/lib/librte_eal/bsdapp/eal/eal_pci.c b/lib/librte_eal/bsdapp/eal/eal_pci.c
> index 95c32c1..2e535ea 100644
> --- a/lib/librte_eal/bsdapp/eal/eal_pci.c
> +++ b/lib/librte_eal/bsdapp/eal/eal_pci.c
...
> +int rte_eal_pci_read_bar(const struct rte_pci_device *device,
> +			 void *buf, size_t len, off_t offset,
> +			 int bar_idx)
> +
> +{
> +	const struct rte_intr_handle *intr_handle = &device->intr_handle;

I'd suggest to reference this var inside pci_vfio_read/write_bar(), and
pass device as the parmater instead.

> +
> +	switch (device->kdrv) {
> +	case RTE_KDRV_VFIO:
> +		return pci_vfio_read_bar(intr_handle, buf, len,
> +					 offset, bar_idx);
> +	default:
> +		RTE_LOG(ERR, EAL, "write bar not supported by driver\n");
                                   ^^^^^
typo.


BTW, I have a question about this API. Obviously, reading/writing bar
space is supported with UIO (when memory resource is mmapped). And I
know why you introduced such 2 APIs, for reading IO bar.

So, here is the question: what are the 2 APIs for, for being gerneric
APIs to read/write bar spaces, or just to read IO bar spaces? If it's
former, the message is wrong; if it's later, you may better rename it
to rte_eal_pci_read/write_io_bar()?

David, what do you think of that?


> +		return -1;
> +	}
> +}
> +
...
> +int
> +pci_vfio_read_bar(const struct rte_intr_handle *intr_handle,
> +		  void *buf, size_t len, off_t offs, int bar_idx)
> +{
> +	if (bar_idx < VFIO_PCI_BAR0_REGION_INDEX
> +	   || bar_idx > VFIO_PCI_BAR5_REGION_INDEX) {

A minor nit: it's more nature to put the '||' at the end of expression,
instead of at the front:

    if (bar_idx < VFIO_PCI_BAR0_REGION_INDEX ||
        bar_idx > VFIO_PCI_BAR5_REGION_INDEX) {


	--yliu
  

On Mon, Feb 1, 2016 at 7:18 PM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
> On Fri, Jan 29, 2016 at 11:51:50PM +0530, Santosh Shukla wrote:
>> Introducing below api for pci bar region rd/wr.
>> Api's are:
>> - rte_eal_pci_read_bar
>> - rte_eal_pci_write_bar
>>
>> Signed-off-by: Santosh Shukla <sshukla@mvista.com>
>> ---
>> v5-->v6:
>> - update api infor in rte_eal_version.map file
>>   suggested by david manchand.
>>
>>  lib/librte_eal/bsdapp/eal/eal_pci.c             |   19 ++++++++++++
>>  lib/librte_eal/bsdapp/eal/rte_eal_version.map   |    3 ++
>>  lib/librte_eal/common/include/rte_pci.h         |   38 +++++++++++++++++++++++
>>  lib/librte_eal/linuxapp/eal/eal_pci.c           |   34 ++++++++++++++++++++
>>  lib/librte_eal/linuxapp/eal/eal_pci_init.h      |    6 ++++
>>  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c      |   28 +++++++++++++++++
>>  lib/librte_eal/linuxapp/eal/rte_eal_version.map |    3 ++
>>  7 files changed, 131 insertions(+)
>>
>> diff --git a/lib/librte_eal/bsdapp/eal/eal_pci.c b/lib/librte_eal/bsdapp/eal/eal_pci.c
>> index 95c32c1..2e535ea 100644
>> --- a/lib/librte_eal/bsdapp/eal/eal_pci.c
>> +++ b/lib/librte_eal/bsdapp/eal/eal_pci.c
> ...
>> +int rte_eal_pci_read_bar(const struct rte_pci_device *device,
>> +                      void *buf, size_t len, off_t offset,
>> +                      int bar_idx)
>> +
>> +{
>> +     const struct rte_intr_handle *intr_handle = &device->intr_handle;
>
> I'd suggest to reference this var inside pci_vfio_read/write_bar(), and
> pass device as the parmater instead.
>

(Sorry for late reply, I was travelling on Monday.)
Make sense.

>> +
>> +     switch (device->kdrv) {
>> +     case RTE_KDRV_VFIO:
>> +             return pci_vfio_read_bar(intr_handle, buf, len,
>> +                                      offset, bar_idx);
>> +     default:
>> +             RTE_LOG(ERR, EAL, "write bar not supported by driver\n");
>                                    ^^^^^
> typo.
>

Oh, r / write / read, right? sorry for typo error (:-

>
> BTW, I have a question about this API. Obviously, reading/writing bar
> space is supported with UIO (when memory resource is mmapped). And I
> know why you introduced such 2 APIs, for reading IO bar.
>
> So, here is the question: what are the 2 APIs for, for being gerneric
> APIs to read/write bar spaces, or just to read IO bar spaces? If it's
> former, the message is wrong; if it's later, you may better rename it
> to rte_eal_pci_read/write_io_bar()?
>

Current use-case is virtio: It is used as io_bar which is first
bar[1]. But implementation is generic, can be used to do rd/wr for
other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
w/o mapping that bar, So apis will be useful for such cases in future.

AFAIU: uio has read/write_config api only and Yes if bar region mapped
then no need to do rd/wr, user can directly access the pci_memory. But
use-case of this api entirely different: unmapped memory by
application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.

Is above explanation convincing? Pl. let me know.

[1] https://en.wikipedia.org/wiki/PCI_configuration_space (first bar
offset 0x010)

> David, what do you think of that?
>
>
>> +             return -1;
>> +     }
>> +}
>> +
> ...
>> +int
>> +pci_vfio_read_bar(const struct rte_intr_handle *intr_handle,
>> +               void *buf, size_t len, off_t offs, int bar_idx)
>> +{
>> +     if (bar_idx < VFIO_PCI_BAR0_REGION_INDEX
>> +        || bar_idx > VFIO_PCI_BAR5_REGION_INDEX) {
>
> A minor nit: it's more nature to put the '||' at the end of expression,
> instead of at the front:
>
>     if (bar_idx < VFIO_PCI_BAR0_REGION_INDEX ||
>         bar_idx > VFIO_PCI_BAR5_REGION_INDEX) {
>
>
>         --yliu
  

On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
> >> +int rte_eal_pci_read_bar(const struct rte_pci_device *device,
> >> +                      void *buf, size_t len, off_t offset,
> >> +                      int bar_idx)
> >> +
> >> +{
> >> +     const struct rte_intr_handle *intr_handle = &device->intr_handle;
> >
> > I'd suggest to reference this var inside pci_vfio_read/write_bar(), and
> > pass device as the parmater instead.
> >
> 
> (Sorry for late reply, I was travelling on Monday.)
> Make sense.
> 
> >> +
> >> +     switch (device->kdrv) {
> >> +     case RTE_KDRV_VFIO:
> >> +             return pci_vfio_read_bar(intr_handle, buf, len,
> >> +                                      offset, bar_idx);
> >> +     default:
> >> +             RTE_LOG(ERR, EAL, "write bar not supported by driver\n");
> >                                    ^^^^^
> > typo.
> >
> 
> Oh, r / write / read, right? sorry for typo error (:-

Right.

> 
> >
> > BTW, I have a question about this API. Obviously, reading/writing bar
> > space is supported with UIO (when memory resource is mmapped). And I
> > know why you introduced such 2 APIs, for reading IO bar.
> >
> > So, here is the question: what are the 2 APIs for, for being gerneric
> > APIs to read/write bar spaces, or just to read IO bar spaces? If it's
> > former, the message is wrong; if it's later, you may better rename it
> > to rte_eal_pci_read/write_io_bar()?
> >
> 
> Current use-case is virtio: It is used as io_bar which is first
> bar[1]. But implementation is generic, can be used to do rd/wr for
> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
> w/o mapping that bar, So apis will be useful for such cases in future.
> 
> AFAIU: uio has read/write_config api only and Yes if bar region mapped
> then no need to do rd/wr, user can directly access the pci_memory. But
> use-case of this api entirely different: unmapped memory by
> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
> 
> Is above explanation convincing? Pl. let me know.

TBH, not really. So, as you stated, it should be generic APIs to
read/write bar space, but limiting it to VFIO only and claiming
that read/write bar space is not support by other drivers (such
as UIO) while in fact it can (in some ways) doesn't seem right
to me.

Anyway, it's just some thoughts from me. David, comments?

	--yliu
  

On Tue, Feb 2, 2016 at 6:43 AM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
> On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
>> Current use-case is virtio: It is used as io_bar which is first
>> bar[1]. But implementation is generic, can be used to do rd/wr for
>> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
>> w/o mapping that bar, So apis will be useful for such cases in future.
>>
>> AFAIU: uio has read/write_config api only and Yes if bar region mapped
>> then no need to do rd/wr, user can directly access the pci_memory. But
>> use-case of this api entirely different: unmapped memory by
>> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
>>
>> Is above explanation convincing? Pl. let me know.
>
> TBH, not really. So, as you stated, it should be generic APIs to
> read/write bar space, but limiting it to VFIO only and claiming
> that read/write bar space is not support by other drivers (such
> as UIO) while in fact it can (in some ways) doesn't seem right
> to me.
>
> Anyway, it's just some thoughts from me. David, comments?

From the very start, same opinion.
We should have a unique api to access those, and eal should hide
details like kernel drivers (uio, vfio, whatever) to the pmd.

Now the thing is, how to do this in an elegant and efficient way.


Regard,
  

On Tue, Feb 2, 2016 at 11:13 AM, Yuanhan Liu
<yuanhan.liu@linux.intel.com> wrote:
> On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
>> >> +int rte_eal_pci_read_bar(const struct rte_pci_device *device,
>> >> +                      void *buf, size_t len, off_t offset,
>> >> +                      int bar_idx)
>> >> +
>> >> +{
>> >> +     const struct rte_intr_handle *intr_handle = &device->intr_handle;
>> >
>> > I'd suggest to reference this var inside pci_vfio_read/write_bar(), and
>> > pass device as the parmater instead.
>> >
>>
>> (Sorry for late reply, I was travelling on Monday.)
>> Make sense.
>>
>> >> +
>> >> +     switch (device->kdrv) {
>> >> +     case RTE_KDRV_VFIO:
>> >> +             return pci_vfio_read_bar(intr_handle, buf, len,
>> >> +                                      offset, bar_idx);
>> >> +     default:
>> >> +             RTE_LOG(ERR, EAL, "write bar not supported by driver\n");
>> >                                    ^^^^^
>> > typo.
>> >
>>
>> Oh, r / write / read, right? sorry for typo error (:-
>
> Right.
>
>>
>> >
>> > BTW, I have a question about this API. Obviously, reading/writing bar
>> > space is supported with UIO (when memory resource is mmapped). And I
>> > know why you introduced such 2 APIs, for reading IO bar.
>> >
>> > So, here is the question: what are the 2 APIs for, for being gerneric
>> > APIs to read/write bar spaces, or just to read IO bar spaces? If it's
>> > former, the message is wrong; if it's later, you may better rename it
>> > to rte_eal_pci_read/write_io_bar()?
>> >
>>
>> Current use-case is virtio: It is used as io_bar which is first
>> bar[1]. But implementation is generic, can be used to do rd/wr for
>> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
>> w/o mapping that bar, So apis will be useful for such cases in future.
>>
>> AFAIU: uio has read/write_config api only and Yes if bar region mapped
>> then no need to do rd/wr, user can directly access the pci_memory. But
>> use-case of this api entirely different: unmapped memory by
>> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
>>
>> Is above explanation convincing? Pl. let me know.
>
> TBH, not really. So, as you stated, it should be generic APIs to
> read/write bar space, but limiting it to VFIO only and claiming
> that read/write bar space is not support by other drivers (such
> as UIO) while in fact it can (in some ways) doesn't seem right
> to me.
>

I agree.. But if UIO doesn't and need could, then I am confused what
can be done? However we have a use-case for vfio so It make sense to
me use this api.  Or else If we all agree then I can export api only
for VFIO.. but it will violate EAL abstraction.


> Anyway, it's just some thoughts from me. David, comments?
>
>         --yliu
  

On Tue, Feb 2, 2016 at 12:30 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> On Tue, Feb 2, 2016 at 11:13 AM, Yuanhan Liu
> <yuanhan.liu@linux.intel.com> wrote:
>> On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
>>> >> +int rte_eal_pci_read_bar(const struct rte_pci_device *device,
>>> >> +                      void *buf, size_t len, off_t offset,
>>> >> +                      int bar_idx)
>>> >> +
>>> >> +{
>>> >> +     const struct rte_intr_handle *intr_handle = &device->intr_handle;
>>> >
>>> > I'd suggest to reference this var inside pci_vfio_read/write_bar(), and
>>> > pass device as the parmater instead.
>>> >
>>>
>>> (Sorry for late reply, I was travelling on Monday.)
>>> Make sense.
>>>
>>> >> +
>>> >> +     switch (device->kdrv) {
>>> >> +     case RTE_KDRV_VFIO:
>>> >> +             return pci_vfio_read_bar(intr_handle, buf, len,
>>> >> +                                      offset, bar_idx);
>>> >> +     default:
>>> >> +             RTE_LOG(ERR, EAL, "write bar not supported by driver\n");
>>> >                                    ^^^^^
>>> > typo.
>>> >
>>>
>>> Oh, r / write / read, right? sorry for typo error (:-
>>
>> Right.
>>
>>>
>>> >
>>> > BTW, I have a question about this API. Obviously, reading/writing bar
>>> > space is supported with UIO (when memory resource is mmapped). And I
>>> > know why you introduced such 2 APIs, for reading IO bar.
>>> >
>>> > So, here is the question: what are the 2 APIs for, for being gerneric
>>> > APIs to read/write bar spaces, or just to read IO bar spaces? If it's
>>> > former, the message is wrong; if it's later, you may better rename it
>>> > to rte_eal_pci_read/write_io_bar()?
>>> >
>>>
>>> Current use-case is virtio: It is used as io_bar which is first
>>> bar[1]. But implementation is generic, can be used to do rd/wr for
>>> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
>>> w/o mapping that bar, So apis will be useful for such cases in future.
>>>
>>> AFAIU: uio has read/write_config api only and Yes if bar region mapped
>>> then no need to do rd/wr, user can directly access the pci_memory. But
>>> use-case of this api entirely different: unmapped memory by
>>> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
>>>
>>> Is above explanation convincing? Pl. let me know.
>>
>> TBH, not really. So, as you stated, it should be generic APIs to
>> read/write bar space, but limiting it to VFIO only and claiming
>> that read/write bar space is not support by other drivers (such
>> as UIO) while in fact it can (in some ways) doesn't seem right
>> to me.
>>
>

Sorry typo
> I agree.. But if UIO doesn't and need could, then I am confused what

r / But if UIO doesn't and need could / But if UIO doesn't and vfio could

> can be done? However we have a use-case for vfio so It make sense to
> me use this api.  Or else If we all agree then I can export api only
> for VFIO.. but it will violate EAL abstraction.
>
>
>> Anyway, it's just some thoughts from me. David, comments?
>>
>>         --yliu
  

On Tue, Feb 02, 2016 at 06:50:18AM +0100, David Marchand wrote:
> On Tue, Feb 2, 2016 at 6:43 AM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
> > On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
> >> Current use-case is virtio: It is used as io_bar which is first
> >> bar[1]. But implementation is generic, can be used to do rd/wr for
> >> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
> >> w/o mapping that bar, So apis will be useful for such cases in future.
> >>
> >> AFAIU: uio has read/write_config api only and Yes if bar region mapped
> >> then no need to do rd/wr, user can directly access the pci_memory. But
> >> use-case of this api entirely different: unmapped memory by
> >> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
> >>
> >> Is above explanation convincing? Pl. let me know.
> >
> > TBH, not really. So, as you stated, it should be generic APIs to
> > read/write bar space, but limiting it to VFIO only and claiming
> > that read/write bar space is not support by other drivers (such
> > as UIO) while in fact it can (in some ways) doesn't seem right
> > to me.
> >
> > Anyway, it's just some thoughts from me. David, comments?
> 
> >From the very start, same opinion.
> We should have a unique api to access those, and eal should hide
> details like kernel drivers (uio, vfio, whatever) to the pmd.
> 
> Now the thing is, how to do this in an elegant and efficient way.

I was thinking that we may just make it be IO port specific read/
write functions:

	rte_eal_pci_ioport_read(dev, bar, buf, size)
	{

		return if not an IO bar;

		if (has io)
			return inb/w/l();

		if (vfio)
			return vfio_ioport_read();

		else, claim aloud that io port read is not allowed
	}

Let us not handle memory bar resource here: in such case, you should
go with rte_eal_pci_map_device() and do it with memory mapped io.

Does that make any sense?

	--yliu
  

On Tue, Feb 2, 2016 at 2:19 PM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
> On Tue, Feb 02, 2016 at 06:50:18AM +0100, David Marchand wrote:
>> On Tue, Feb 2, 2016 at 6:43 AM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
>> > On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
>> >> Current use-case is virtio: It is used as io_bar which is first
>> >> bar[1]. But implementation is generic, can be used to do rd/wr for
>> >> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
>> >> w/o mapping that bar, So apis will be useful for such cases in future.
>> >>
>> >> AFAIU: uio has read/write_config api only and Yes if bar region mapped
>> >> then no need to do rd/wr, user can directly access the pci_memory. But
>> >> use-case of this api entirely different: unmapped memory by
>> >> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
>> >>
>> >> Is above explanation convincing? Pl. let me know.
>> >
>> > TBH, not really. So, as you stated, it should be generic APIs to
>> > read/write bar space, but limiting it to VFIO only and claiming
>> > that read/write bar space is not support by other drivers (such
>> > as UIO) while in fact it can (in some ways) doesn't seem right
>> > to me.
>> >
>> > Anyway, it's just some thoughts from me. David, comments?
>>
>> >From the very start, same opinion.
>> We should have a unique api to access those, and eal should hide
>> details like kernel drivers (uio, vfio, whatever) to the pmd.
>>
>> Now the thing is, how to do this in an elegant and efficient way.
>
> I was thinking that we may just make it be IO port specific read/
> write functions:
>

Ok,

>         rte_eal_pci_ioport_read(dev, bar, buf, size)
>         {
>
>                 return if not an IO bar;
>
>                 if (has io)
>                         return inb/w/l();
>

In that case, It may be r / if (has io) / if (drv->kdrv == UIO)

>                 if (vfio)
>                         return vfio_ioport_read();
>
>                 else, claim aloud that io port read is not allowed
>         }
>
> Let us not handle memory bar resource here: in such case, you should
> go with rte_eal_pci_map_device() and do it with memory mapped io.
>
> Does that make any sense?
>
I am not entirely sure.
Are you considering IGB_UIO, UIO_GENERIC and NIC_UIO: all the cases ?


>         --yliu
  

On Tue, Feb 2, 2016 at 9:21 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> On Tue, Feb 2, 2016 at 2:19 PM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
>> On Tue, Feb 02, 2016 at 06:50:18AM +0100, David Marchand wrote:
>>> On Tue, Feb 2, 2016 at 6:43 AM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
>>> > On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
>>> >> Current use-case is virtio: It is used as io_bar which is first
>>> >> bar[1]. But implementation is generic, can be used to do rd/wr for
>>> >> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
>>> >> w/o mapping that bar, So apis will be useful for such cases in future.
>>> >>
>>> >> AFAIU: uio has read/write_config api only and Yes if bar region mapped
>>> >> then no need to do rd/wr, user can directly access the pci_memory. But
>>> >> use-case of this api entirely different: unmapped memory by
>>> >> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
>>> >>
>>> >> Is above explanation convincing? Pl. let me know.
>>> >
>>> > TBH, not really. So, as you stated, it should be generic APIs to
>>> > read/write bar space, but limiting it to VFIO only and claiming
>>> > that read/write bar space is not support by other drivers (such
>>> > as UIO) while in fact it can (in some ways) doesn't seem right
>>> > to me.
>>> >
>>> > Anyway, it's just some thoughts from me. David, comments?
>>>
>>> >From the very start, same opinion.
>>> We should have a unique api to access those, and eal should hide
>>> details like kernel drivers (uio, vfio, whatever) to the pmd.
>>>
>>> Now the thing is, how to do this in an elegant and efficient way.
>>
>> I was thinking that we may just make it be IO port specific read/
>> write functions:
>>
>
> Ok,
>
>>         rte_eal_pci_ioport_read(dev, bar, buf, size)
>>         {
>>
>>                 return if not an IO bar;
>>
>>                 if (has io)
>>                         return inb/w/l();
>>
>
> In that case, It may be r / if (has io) / if (drv->kdrv == UIO)
>
>>                 if (vfio)
>>                         return vfio_ioport_read();
>>
>>                 else, claim aloud that io port read is not allowed
>>         }
>>
>> Let us not handle memory bar resource here: in such case, you should
>> go with rte_eal_pci_map_device() and do it with memory mapped io.
>>
>> Does that make any sense?
>>
> I am not entirely sure.
> Are you considering IGB_UIO, UIO_GENERIC and NIC_UIO: all the cases ?
>

Just came-up something below what Yuanhan has proposed, Does this look okay?

int rte_eal_pci_ioport_read(const struct rte_pci_device *device,
                                              void *buf, size_t len,
off_t offset,
                                              int bar_idx)
{
      if (bar_idx != 0) {
              RTE_LOG(ERR, EAL, "not a ioport bar\n");
              return -1;
       }

     switch (device->kdrv) {
     case RTE_KDRV_VFIO:
               return pci_vfio_ioport_read(device, buf, len, offset, bar_idx);
     case RTE_KDRV_IGB_UIO:
     case RTE_KDRV_UIO_GENERIC:
     case RTE_KDRV_NIC_UIO:
         {
              switch (size)
                      case 1: return inb(buf /*ioport address*/);
                      case 2: return inw(buf /* ioport address*/);
                      case 4: return inl(buf /* ioport address*/);
                      default:
                             RTE_LOG(ERR, EAL, "invalid size\n");
          }

       default:
                 RTE_LOG(ERR, EAL, "read bar not supported by driver\n");
                 return -1;
      }
}

>
>>         --yliu
  

On Tue, Feb 2, 2016 at 9:48 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> On Tue, Feb 2, 2016 at 9:21 PM, Santosh Shukla <sshukla@mvista.com> wrote:
>> On Tue, Feb 2, 2016 at 2:19 PM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
>>> On Tue, Feb 02, 2016 at 06:50:18AM +0100, David Marchand wrote:
>>>> On Tue, Feb 2, 2016 at 6:43 AM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
>>>> > On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
>>>> >> Current use-case is virtio: It is used as io_bar which is first
>>>> >> bar[1]. But implementation is generic, can be used to do rd/wr for
>>>> >> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
>>>> >> w/o mapping that bar, So apis will be useful for such cases in future.
>>>> >>
>>>> >> AFAIU: uio has read/write_config api only and Yes if bar region mapped
>>>> >> then no need to do rd/wr, user can directly access the pci_memory. But
>>>> >> use-case of this api entirely different: unmapped memory by
>>>> >> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
>>>> >>
>>>> >> Is above explanation convincing? Pl. let me know.
>>>> >
>>>> > TBH, not really. So, as you stated, it should be generic APIs to
>>>> > read/write bar space, but limiting it to VFIO only and claiming
>>>> > that read/write bar space is not support by other drivers (such
>>>> > as UIO) while in fact it can (in some ways) doesn't seem right
>>>> > to me.
>>>> >
>>>> > Anyway, it's just some thoughts from me. David, comments?
>>>>
>>>> >From the very start, same opinion.
>>>> We should have a unique api to access those, and eal should hide
>>>> details like kernel drivers (uio, vfio, whatever) to the pmd.
>>>>
>>>> Now the thing is, how to do this in an elegant and efficient way.
>>>
>>> I was thinking that we may just make it be IO port specific read/
>>> write functions:
>>>
>>
>> Ok,
>>
>>>         rte_eal_pci_ioport_read(dev, bar, buf, size)
>>>         {
>>>
>>>                 return if not an IO bar;
>>>
>>>                 if (has io)
>>>                         return inb/w/l();
>>>
>>
>> In that case, It may be r / if (has io) / if (drv->kdrv == UIO)
>>
>>>                 if (vfio)
>>>                         return vfio_ioport_read();
>>>
>>>                 else, claim aloud that io port read is not allowed
>>>         }
>>>
>>> Let us not handle memory bar resource here: in such case, you should
>>> go with rte_eal_pci_map_device() and do it with memory mapped io.
>>>
>>> Does that make any sense?
>>>
>> I am not entirely sure.
>> Are you considering IGB_UIO, UIO_GENERIC and NIC_UIO: all the cases ?
>>
>
> Just came-up something below what Yuanhan has proposed, Does this look okay?
>
> int rte_eal_pci_ioport_read(const struct rte_pci_device *device,
>                                               void *buf, size_t len,
> off_t offset,
>                                               int bar_idx)
> {
>       if (bar_idx != 0) {
>               RTE_LOG(ERR, EAL, "not a ioport bar\n");
>               return -1;
>        }
>
>      switch (device->kdrv) {
>      case RTE_KDRV_VFIO:
>                return pci_vfio_ioport_read(device, buf, len, offset, bar_idx);
>      case RTE_KDRV_IGB_UIO:
>      case RTE_KDRV_UIO_GENERIC:
>      case RTE_KDRV_NIC_UIO:
>          {
>               switch (size)
>                       case 1: return inb(buf /*ioport address*/);
>                       case 2: return inw(buf /* ioport address*/);
>                       case 4: return inl(buf /* ioport address*/);
>                       default:
>                              RTE_LOG(ERR, EAL, "invalid size\n");
>           }
>
>        default:
>                  RTE_LOG(ERR, EAL, "read bar not supported by driver\n");
>                  return -1;
>       }
> }
>

Ping?

Also can someone please review rest of series. This patchset going
through multiple revision, Each revision get one / two comment, It
would help if I get review comment for each patch.

>>
>>>         --yliu
  

On Tue, Feb 02, 2016 at 09:48:44PM +0530, Santosh Shukla wrote:
> On Tue, Feb 2, 2016 at 9:21 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> > On Tue, Feb 2, 2016 at 2:19 PM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
> >> On Tue, Feb 02, 2016 at 06:50:18AM +0100, David Marchand wrote:
> >>> On Tue, Feb 2, 2016 at 6:43 AM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
> >>> > On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
> >>> >> Current use-case is virtio: It is used as io_bar which is first
> >>> >> bar[1]. But implementation is generic, can be used to do rd/wr for
> >>> >> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
> >>> >> w/o mapping that bar, So apis will be useful for such cases in future.
> >>> >>
> >>> >> AFAIU: uio has read/write_config api only and Yes if bar region mapped
> >>> >> then no need to do rd/wr, user can directly access the pci_memory. But
> >>> >> use-case of this api entirely different: unmapped memory by
> >>> >> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
> >>> >>
> >>> >> Is above explanation convincing? Pl. let me know.
> >>> >
> >>> > TBH, not really. So, as you stated, it should be generic APIs to
> >>> > read/write bar space, but limiting it to VFIO only and claiming
> >>> > that read/write bar space is not support by other drivers (such
> >>> > as UIO) while in fact it can (in some ways) doesn't seem right
> >>> > to me.
> >>> >
> >>> > Anyway, it's just some thoughts from me. David, comments?
> >>>
> >>> >From the very start, same opinion.
> >>> We should have a unique api to access those, and eal should hide
> >>> details like kernel drivers (uio, vfio, whatever) to the pmd.
> >>>
> >>> Now the thing is, how to do this in an elegant and efficient way.
> >>
> >> I was thinking that we may just make it be IO port specific read/
> >> write functions:
> >>
> >
> > Ok,
> >
> >>         rte_eal_pci_ioport_read(dev, bar, buf, size)
> >>         {
> >>
> >>                 return if not an IO bar;
> >>
> >>                 if (has io)
> >>                         return inb/w/l();
> >>
> >
> > In that case, It may be r / if (has io) / if (drv->kdrv == UIO)

Nope, I meant platform supports inb/w/l() command.

> >
> >>                 if (vfio)
> >>                         return vfio_ioport_read();
> >>
> >>                 else, claim aloud that io port read is not allowed
> >>         }
> >>
> >> Let us not handle memory bar resource here: in such case, you should
> >> go with rte_eal_pci_map_device() and do it with memory mapped io.
> >>
> >> Does that make any sense?
> >>
> > I am not entirely sure.
> > Are you considering IGB_UIO, UIO_GENERIC and NIC_UIO: all the cases ?
> >
> 
> Just came-up something below what Yuanhan has proposed, Does this look okay?
> 
> int rte_eal_pci_ioport_read(const struct rte_pci_device *device,
>                                               void *buf, size_t len,
> off_t offset,
>                                               int bar_idx)

Your implementation doesn't look right to me. But anyway, that's not
important so far; the feasibility is.

David, would you please comment on this?

	--yliu
  

On Wed, Feb 03, 2016 at 03:20:09PM +0530, Santosh Shukla wrote:
> On Tue, Feb 2, 2016 at 9:48 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> > On Tue, Feb 2, 2016 at 9:21 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> >> On Tue, Feb 2, 2016 at 2:19 PM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
> >>> On Tue, Feb 02, 2016 at 06:50:18AM +0100, David Marchand wrote:
> >>>> On Tue, Feb 2, 2016 at 6:43 AM, Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
> >>>> > On Tue, Feb 02, 2016 at 09:44:14AM +0530, Santosh Shukla wrote:
> >>>> >> Current use-case is virtio: It is used as io_bar which is first
> >>>> >> bar[1]. But implementation is generic, can be used to do rd/wr for
> >>>> >> other bar index too. Also vfio facilitate user to do rd/wr to pci_bars
> >>>> >> w/o mapping that bar, So apis will be useful for such cases in future.
> >>>> >>
> >>>> >> AFAIU: uio has read/write_config api only and Yes if bar region mapped
> >>>> >> then no need to do rd/wr, user can directly access the pci_memory. But
> >>>> >> use-case of this api entirely different: unmapped memory by
> >>>> >> application context i.e.. vfio_rd/wr-way {pread/pwrite-way}.
> >>>> >>
> >>>> >> Is above explanation convincing? Pl. let me know.
> >>>> >
> >>>> > TBH, not really. So, as you stated, it should be generic APIs to
> >>>> > read/write bar space, but limiting it to VFIO only and claiming
> >>>> > that read/write bar space is not support by other drivers (such
> >>>> > as UIO) while in fact it can (in some ways) doesn't seem right
> >>>> > to me.
> >>>> >
> >>>> > Anyway, it's just some thoughts from me. David, comments?
> >>>>
> >>>> >From the very start, same opinion.
> >>>> We should have a unique api to access those, and eal should hide
> >>>> details like kernel drivers (uio, vfio, whatever) to the pmd.
> >>>>
> >>>> Now the thing is, how to do this in an elegant and efficient way.
> >>>
> >>> I was thinking that we may just make it be IO port specific read/
> >>> write functions:
> >>>
> >>
> >> Ok,
> >>
> >>>         rte_eal_pci_ioport_read(dev, bar, buf, size)
> >>>         {
> >>>
> >>>                 return if not an IO bar;
> >>>
> >>>                 if (has io)
> >>>                         return inb/w/l();
> >>>
> >>
> >> In that case, It may be r / if (has io) / if (drv->kdrv == UIO)
> >>
> >>>                 if (vfio)
> >>>                         return vfio_ioport_read();
> >>>
> >>>                 else, claim aloud that io port read is not allowed
> >>>         }
> >>>
> >>> Let us not handle memory bar resource here: in such case, you should
> >>> go with rte_eal_pci_map_device() and do it with memory mapped io.
> >>>
> >>> Does that make any sense?
> >>>
> >> I am not entirely sure.
> >> Are you considering IGB_UIO, UIO_GENERIC and NIC_UIO: all the cases ?
> >>
> >
> > Just came-up something below what Yuanhan has proposed, Does this look okay?
> >
> > int rte_eal_pci_ioport_read(const struct rte_pci_device *device,
> >                                               void *buf, size_t len,
> > off_t offset,
> >                                               int bar_idx)
> > {
> >       if (bar_idx != 0) {
> >               RTE_LOG(ERR, EAL, "not a ioport bar\n");
> >               return -1;
> >        }
> >
> >      switch (device->kdrv) {
> >      case RTE_KDRV_VFIO:
> >                return pci_vfio_ioport_read(device, buf, len, offset, bar_idx);
> >      case RTE_KDRV_IGB_UIO:
> >      case RTE_KDRV_UIO_GENERIC:
> >      case RTE_KDRV_NIC_UIO:
> >          {
> >               switch (size)
> >                       case 1: return inb(buf /*ioport address*/);
> >                       case 2: return inw(buf /* ioport address*/);
> >                       case 4: return inl(buf /* ioport address*/);
> >                       default:
> >                              RTE_LOG(ERR, EAL, "invalid size\n");
> >           }
> >
> >        default:
> >                  RTE_LOG(ERR, EAL, "read bar not supported by driver\n");
> >                  return -1;
> >       }
> > }
> >
> 
> Ping?

Please be a bit more patient. Everybody has work got to do; and today I was
out for personal affairs the whole day.

> 
> Also can someone please review rest of series. This patchset going
> through multiple revision, Each revision get one / two comment, It
> would help if I get review comment for each patch.

The others looks good to me; if you have the EAL pci API resolved
properly, I guess I could give my ACK.

	--yliu
  

On Wed, Feb 3, 2016 at 12:50 PM, Yuanhan Liu
<yuanhan.liu@linux.intel.com> wrote:
> On Wed, Feb 03, 2016 at 03:20:09PM +0530, Santosh Shukla wrote:
>> Also can someone please review rest of series. This patchset going
>> through multiple revision, Each revision get one / two comment, It
>> would help if I get review comment for each patch.
>
> The others looks good to me; if you have the EAL pci API resolved
> properly, I guess I could give my ACK.

As discussed offlist, I just sent a patchset [1] that proposes a new api.
This is not perfect, but I think this should be fine for x86 / arm.
Santosh should be able to rebase his code on top of it.

[1] http://dpdk.org/ml/archives/dev/2016-February/032778.html


Regards,