[v4,00/10] ethdev: datapath-focused flow rules management

  Three major changes to a generic RTE Flow API were implemented in order
to speed up flow rule insertion/destruction and adapt the API to the
needs of a datapath-focused flow rules management applications:

1. Pre-configuration hints.
Application may give us some hints on what type of resources are needed.
Introduce the configuration routine to prepare all the needed resources
inside a PMD/HW before any flow rules are created at the init stage.

2. Flow grouping using templates.
Use the knowledge about which flow rules are to be used in an application
and prepare item and action templates for them in advance. Group flow rules
with common patterns and actions together for better resource management.

3. Queue-based flow management.
Perform flow rule insertion/destruction asynchronously to spare the datapath
from blocking on RTE Flow API and allow it to continue with packet processing.
Enqueue flow rules operations and poll for the results later.

testpmd examples are part of the patch series. PMD changes will follow.

RFC: https://patchwork.dpdk.org/project/dpdk/cover/20211006044835.3936226-1-akozyrev@nvidia.com/

Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>

---
v4: 
- removed structures versioning
- introduced new rte_flow_port_info structure for rte_flow_info_get API
- renamed rte_flow_table_create to rte_flow_template_table_create

v3: addressed review comments and updated documentation
- added API to get info about pre-configurable resources
- renamed rte_flow_item_template to rte_flow_pattern_template
- renamed drain operation attribute to postpone
- renamed rte_flow_q_drain to rte_flow_q_push
- renamed rte_flow_q_dequeue to rte_flow_q_pull

v2: fixed patch series thread

Alexander Kozyrev (10):
  ethdev: introduce flow pre-configuration hints
  ethdev: add flow item/action templates
  ethdev: bring in async queue-based flow rules operations
  app/testpmd: implement rte flow configuration
  app/testpmd: implement rte flow template management
  app/testpmd: implement rte flow table management
  app/testpmd: implement rte flow queue flow operations
  app/testpmd: implement rte flow push operations
  app/testpmd: implement rte flow pull operations
  app/testpmd: implement rte flow queue indirect actions

 app/test-pmd/cmdline_flow.c                   | 1496 ++++++++++++++++-
 app/test-pmd/config.c                         |  771 +++++++++
 app/test-pmd/testpmd.h                        |   66 +
 doc/guides/prog_guide/img/rte_flow_q_init.svg |  205 +++
 .../prog_guide/img/rte_flow_q_usage.svg       |  351 ++++
 doc/guides/prog_guide/rte_flow.rst            |  326 ++++
 doc/guides/rel_notes/release_22_03.rst        |   22 +
 doc/guides/testpmd_app_ug/testpmd_funcs.rst   |  378 ++++-
 lib/ethdev/rte_flow.c                         |  359 ++++
 lib/ethdev/rte_flow.h                         |  702 ++++++++
 lib/ethdev/rte_flow_driver.h                  |  102 ++
 lib/ethdev/version.map                        |   15 +
 12 files changed, 4773 insertions(+), 20 deletions(-)
 create mode 100644 doc/guides/prog_guide/img/rte_flow_q_init.svg
 create mode 100644 doc/guides/prog_guide/img/rte_flow_q_usage.svg
  

On 2/9/2022 9:37 PM, Alexander Kozyrev wrote:
> Three major changes to a generic RTE Flow API were implemented in order
> to speed up flow rule insertion/destruction and adapt the API to the
> needs of a datapath-focused flow rules management applications:
> 
> 1. Pre-configuration hints.
> Application may give us some hints on what type of resources are needed.
> Introduce the configuration routine to prepare all the needed resources
> inside a PMD/HW before any flow rules are created at the init stage.
> 
> 2. Flow grouping using templates.
> Use the knowledge about which flow rules are to be used in an application
> and prepare item and action templates for them in advance. Group flow rules
> with common patterns and actions together for better resource management.
> 
> 3. Queue-based flow management.
> Perform flow rule insertion/destruction asynchronously to spare the datapath
> from blocking on RTE Flow API and allow it to continue with packet processing.
> Enqueue flow rules operations and poll for the results later.
> 
> testpmd examples are part of the patch series. PMD changes will follow.
> 
> RFC: https://patchwork.dpdk.org/project/dpdk/cover/20211006044835.3936226-1-akozyrev@nvidia.com/
> 
> Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
> Acked-by: Ori Kam <orika@nvidia.com>
> 
> ---
> v4:
> - removed structures versioning
> - introduced new rte_flow_port_info structure for rte_flow_info_get API
> - renamed rte_flow_table_create to rte_flow_template_table_create
> 
> v3: addressed review comments and updated documentation
> - added API to get info about pre-configurable resources
> - renamed rte_flow_item_template to rte_flow_pattern_template
> - renamed drain operation attribute to postpone
> - renamed rte_flow_q_drain to rte_flow_q_push
> - renamed rte_flow_q_dequeue to rte_flow_q_pull
> 
> v2: fixed patch series thread
> 
> Alexander Kozyrev (10):
>    ethdev: introduce flow pre-configuration hints
>    ethdev: add flow item/action templates
>    ethdev: bring in async queue-based flow rules operations
>    app/testpmd: implement rte flow configuration
>    app/testpmd: implement rte flow template management
>    app/testpmd: implement rte flow table management
>    app/testpmd: implement rte flow queue flow operations
>    app/testpmd: implement rte flow push operations
>    app/testpmd: implement rte flow pull operations
>    app/testpmd: implement rte flow queue indirect actions
> 

Hi Jerin, Ajit, Ivan,

As far as I can see you did some reviews in the previous versions,
but not ack the patch.
Is there any objection to last version of the patch, if not I will
proceed with it.


Hi Alex,

As process we require at least one PMD implementation (it can be draft)
to justify the API design.

If there is no objection from above reviewers and PMD implementation
exists before end of the week, I think we can get the set for -rc1.

Thanks,
ferruh
  

Thanks, Ferruh.
The pmd part is being updated according to the previous API comments.
@Suanming Mou is working on it and will send it once ready, before the weekend.

Regards,
Asaf Penso

>-----Original Message-----
>From: Ferruh Yigit <ferruh.yigit@intel.com>
>Sent: Thursday, February 10, 2022 6:00 PM
>To: Alexander Kozyrev <akozyrev@nvidia.com>; dev@dpdk.org
>Cc: Ori Kam <orika@nvidia.com>; NBU-Contact-Thomas Monjalon (EXTERNAL)
><thomas@monjalon.net>; ivan.malov@oktetlabs.ru;
>andrew.rybchenko@oktetlabs.ru; mohammad.abdul.awal@intel.com;
>qi.z.zhang@intel.com; jerinj@marvell.com; ajit.khaparde@broadcom.com;
>bruce.richardson@intel.com
>Subject: Re: [PATCH v4 00/10] ethdev: datapath-focused flow rules
>management
>
>On 2/9/2022 9:37 PM, Alexander Kozyrev wrote:
>> Three major changes to a generic RTE Flow API were implemented in
>> order to speed up flow rule insertion/destruction and adapt the API to
>> the needs of a datapath-focused flow rules management applications:
>>
>> 1. Pre-configuration hints.
>> Application may give us some hints on what type of resources are needed.
>> Introduce the configuration routine to prepare all the needed
>> resources inside a PMD/HW before any flow rules are created at the init
>stage.
>>
>> 2. Flow grouping using templates.
>> Use the knowledge about which flow rules are to be used in an
>> application and prepare item and action templates for them in advance.
>> Group flow rules with common patterns and actions together for better
>resource management.
>>
>> 3. Queue-based flow management.
>> Perform flow rule insertion/destruction asynchronously to spare the
>> datapath from blocking on RTE Flow API and allow it to continue with packet
>processing.
>> Enqueue flow rules operations and poll for the results later.
>>
>> testpmd examples are part of the patch series. PMD changes will follow.
>>
>> RFC:
>> https://patchwork.dpdk.org/project/dpdk/cover/20211006044835.3936226-
>1
>> -akozyrev@nvidia.com/
>>
>> Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
>> Acked-by: Ori Kam <orika@nvidia.com>
>>
>> ---
>> v4:
>> - removed structures versioning
>> - introduced new rte_flow_port_info structure for rte_flow_info_get
>> API
>> - renamed rte_flow_table_create to rte_flow_template_table_create
>>
>> v3: addressed review comments and updated documentation
>> - added API to get info about pre-configurable resources
>> - renamed rte_flow_item_template to rte_flow_pattern_template
>> - renamed drain operation attribute to postpone
>> - renamed rte_flow_q_drain to rte_flow_q_push
>> - renamed rte_flow_q_dequeue to rte_flow_q_pull
>>
>> v2: fixed patch series thread
>>
>> Alexander Kozyrev (10):
>>    ethdev: introduce flow pre-configuration hints
>>    ethdev: add flow item/action templates
>>    ethdev: bring in async queue-based flow rules operations
>>    app/testpmd: implement rte flow configuration
>>    app/testpmd: implement rte flow template management
>>    app/testpmd: implement rte flow table management
>>    app/testpmd: implement rte flow queue flow operations
>>    app/testpmd: implement rte flow push operations
>>    app/testpmd: implement rte flow pull operations
>>    app/testpmd: implement rte flow queue indirect actions
>>
>
>Hi Jerin, Ajit, Ivan,
>
>As far as I can see you did some reviews in the previous versions, but not ack
>the patch.
>Is there any objection to last version of the patch, if not I will proceed with it.
>
>
>Hi Alex,
>
>As process we require at least one PMD implementation (it can be draft) to
>justify the API design.
>
>If there is no objection from above reviewers and PMD implementation exists
>before end of the week, I think we can get the set for -rc1.
>
>Thanks,
>ferruh
  

Hi,

I wish the PMD part is not too late.  You can find the series here:
https://patches.dpdk.org/project/dpdk/cover/20220210162926.20436-1-suanmingm@nvidia.com/

Thanks,
Suanming Mou

> -----Original Message-----
> From: Asaf Penso <asafp@nvidia.com>
> Sent: Friday, February 11, 2022 12:12 AM
> To: Ferruh Yigit <ferruh.yigit@intel.com>; Alexander Kozyrev
> <akozyrev@nvidia.com>; dev@dpdk.org; Suanming Mou
> <suanmingm@nvidia.com>
> Cc: Ori Kam <orika@nvidia.com>; NBU-Contact-Thomas Monjalon (EXTERNAL)
> <thomas@monjalon.net>; ivan.malov@oktetlabs.ru;
> andrew.rybchenko@oktetlabs.ru; mohammad.abdul.awal@intel.com;
> qi.z.zhang@intel.com; jerinj@marvell.com; ajit.khaparde@broadcom.com;
> bruce.richardson@intel.com
> Subject: RE: [PATCH v4 00/10] ethdev: datapath-focused flow rules
> management
> 
> Thanks, Ferruh.
> The pmd part is being updated according to the previous API comments.
> @Suanming Mou is working on it and will send it once ready, before the
> weekend.
> 
> Regards,
> Asaf Penso
> 
> >-----Original Message-----
> >From: Ferruh Yigit <ferruh.yigit@intel.com>
> >Sent: Thursday, February 10, 2022 6:00 PM
> >To: Alexander Kozyrev <akozyrev@nvidia.com>; dev@dpdk.org
> >Cc: Ori Kam <orika@nvidia.com>; NBU-Contact-Thomas Monjalon (EXTERNAL)
> ><thomas@monjalon.net>; ivan.malov@oktetlabs.ru;
> >andrew.rybchenko@oktetlabs.ru; mohammad.abdul.awal@intel.com;
> >qi.z.zhang@intel.com; jerinj@marvell.com; ajit.khaparde@broadcom.com;
> >bruce.richardson@intel.com
> >Subject: Re: [PATCH v4 00/10] ethdev: datapath-focused flow rules
> >management
> >
> >On 2/9/2022 9:37 PM, Alexander Kozyrev wrote:
> >> Three major changes to a generic RTE Flow API were implemented in
> >> order to speed up flow rule insertion/destruction and adapt the API
> >> to the needs of a datapath-focused flow rules management applications:
> >>
> >> 1. Pre-configuration hints.
> >> Application may give us some hints on what type of resources are needed.
> >> Introduce the configuration routine to prepare all the needed
> >> resources inside a PMD/HW before any flow rules are created at the
> >> init
> >stage.
> >>
> >> 2. Flow grouping using templates.
> >> Use the knowledge about which flow rules are to be used in an
> >> application and prepare item and action templates for them in advance.
> >> Group flow rules with common patterns and actions together for better
> >resource management.
> >>
> >> 3. Queue-based flow management.
> >> Perform flow rule insertion/destruction asynchronously to spare the
> >> datapath from blocking on RTE Flow API and allow it to continue with
> >> packet
> >processing.
> >> Enqueue flow rules operations and poll for the results later.
> >>
> >> testpmd examples are part of the patch series. PMD changes will follow.
> >>
> >> RFC:
> >> https://patchwork.dpdk.org/project/dpdk/cover/20211006044835.3936226-
> >1
> >> -akozyrev@nvidia.com/
> >>
> >> Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
> >> Acked-by: Ori Kam <orika@nvidia.com>
> >>
> >> ---
> >> v4:
> >> - removed structures versioning
> >> - introduced new rte_flow_port_info structure for rte_flow_info_get
> >> API
> >> - renamed rte_flow_table_create to rte_flow_template_table_create
> >>
> >> v3: addressed review comments and updated documentation
> >> - added API to get info about pre-configurable resources
> >> - renamed rte_flow_item_template to rte_flow_pattern_template
> >> - renamed drain operation attribute to postpone
> >> - renamed rte_flow_q_drain to rte_flow_q_push
> >> - renamed rte_flow_q_dequeue to rte_flow_q_pull
> >>
> >> v2: fixed patch series thread
> >>
> >> Alexander Kozyrev (10):
> >>    ethdev: introduce flow pre-configuration hints
> >>    ethdev: add flow item/action templates
> >>    ethdev: bring in async queue-based flow rules operations
> >>    app/testpmd: implement rte flow configuration
> >>    app/testpmd: implement rte flow template management
> >>    app/testpmd: implement rte flow table management
> >>    app/testpmd: implement rte flow queue flow operations
> >>    app/testpmd: implement rte flow push operations
> >>    app/testpmd: implement rte flow pull operations
> >>    app/testpmd: implement rte flow queue indirect actions
> >>
> >
> >Hi Jerin, Ajit, Ivan,
> >
> >As far as I can see you did some reviews in the previous versions, but
> >not ack the patch.
> >Is there any objection to last version of the patch, if not I will proceed with it.
> >
> >
> >Hi Alex,
> >
> >As process we require at least one PMD implementation (it can be draft)
> >to justify the API design.
> >
> >If there is no objection from above reviewers and PMD implementation
> >exists before end of the week, I think we can get the set for -rc1.
> >
> >Thanks,
> >ferruh
  

On Thu, Feb 10, 2022 at 8:00 AM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> On 2/9/2022 9:37 PM, Alexander Kozyrev wrote:
> > Three major changes to a generic RTE Flow API were implemented in order
> > to speed up flow rule insertion/destruction and adapt the API to the
> > needs of a datapath-focused flow rules management applications:
> >
> > 1. Pre-configuration hints.
> > Application may give us some hints on what type of resources are needed.
> > Introduce the configuration routine to prepare all the needed resources
> > inside a PMD/HW before any flow rules are created at the init stage.
> >
> > 2. Flow grouping using templates.
> > Use the knowledge about which flow rules are to be used in an application
> > and prepare item and action templates for them in advance. Group flow rules
> > with common patterns and actions together for better resource management.
> >
> > 3. Queue-based flow management.
> > Perform flow rule insertion/destruction asynchronously to spare the datapath
> > from blocking on RTE Flow API and allow it to continue with packet processing.
> > Enqueue flow rules operations and poll for the results later.
> >
> > testpmd examples are part of the patch series. PMD changes will follow.
> >
> > RFC: https://patchwork.dpdk.org/project/dpdk/cover/20211006044835.3936226-1-akozyrev@nvidia.com/
> >
> > Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
> > Acked-by: Ori Kam <orika@nvidia.com>
> >
> > ---
> > v4:
> > - removed structures versioning
> > - introduced new rte_flow_port_info structure for rte_flow_info_get API
> > - renamed rte_flow_table_create to rte_flow_template_table_create
> >
> > v3: addressed review comments and updated documentation
> > - added API to get info about pre-configurable resources
> > - renamed rte_flow_item_template to rte_flow_pattern_template
> > - renamed drain operation attribute to postpone
> > - renamed rte_flow_q_drain to rte_flow_q_push
> > - renamed rte_flow_q_dequeue to rte_flow_q_pull
> >
> > v2: fixed patch series thread
> >
> > Alexander Kozyrev (10):
> >    ethdev: introduce flow pre-configuration hints
> >    ethdev: add flow item/action templates
> >    ethdev: bring in async queue-based flow rules operations
> >    app/testpmd: implement rte flow configuration
> >    app/testpmd: implement rte flow template management
> >    app/testpmd: implement rte flow table management
> >    app/testpmd: implement rte flow queue flow operations
> >    app/testpmd: implement rte flow push operations
> >    app/testpmd: implement rte flow pull operations
> >    app/testpmd: implement rte flow queue indirect actions
> >
>
> Hi Jerin, Ajit, Ivan,
>
> As far as I can see you did some reviews in the previous versions,
> but not ack the patch.
> Is there any objection to last version of the patch, if not I will
> proceed with it.
The latest set is looking good. There are some places where we could
cleanup or rephrase the text. But that need not block the series.
So for the series
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>

Thanks for checking.

>
>
> Hi Alex,
>
> As process we require at least one PMD implementation (it can be draft)
> to justify the API design.
>
> If there is no objection from above reviewers and PMD implementation
> exists before end of the week, I think we can get the set for -rc1.
>
> Thanks,
> ferruh
  

Hi Ferruh,

On Thu, 10 Feb 2022, Ferruh Yigit wrote:

> On 2/9/2022 9:37 PM, Alexander Kozyrev wrote:
>> Three major changes to a generic RTE Flow API were implemented in order
>> to speed up flow rule insertion/destruction and adapt the API to the
>> needs of a datapath-focused flow rules management applications:
>> 
>> 1. Pre-configuration hints.
>> Application may give us some hints on what type of resources are needed.
>> Introduce the configuration routine to prepare all the needed resources
>> inside a PMD/HW before any flow rules are created at the init stage.
>> 
>> 2. Flow grouping using templates.
>> Use the knowledge about which flow rules are to be used in an application
>> and prepare item and action templates for them in advance. Group flow rules
>> with common patterns and actions together for better resource management.
>> 
>> 3. Queue-based flow management.
>> Perform flow rule insertion/destruction asynchronously to spare the 
>> datapath
>> from blocking on RTE Flow API and allow it to continue with packet 
>> processing.
>> Enqueue flow rules operations and poll for the results later.
>> 
>> testpmd examples are part of the patch series. PMD changes will follow.
>> 
>> RFC: 
>> https://patchwork.dpdk.org/project/dpdk/cover/20211006044835.3936226-1-akozyrev@nvidia.com/
>> 
>> Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
>> Acked-by: Ori Kam <orika@nvidia.com>
>> 
>> ---
>> v4:
>> - removed structures versioning
>> - introduced new rte_flow_port_info structure for rte_flow_info_get API
>> - renamed rte_flow_table_create to rte_flow_template_table_create
>> 
>> v3: addressed review comments and updated documentation
>> - added API to get info about pre-configurable resources
>> - renamed rte_flow_item_template to rte_flow_pattern_template
>> - renamed drain operation attribute to postpone
>> - renamed rte_flow_q_drain to rte_flow_q_push
>> - renamed rte_flow_q_dequeue to rte_flow_q_pull
>> 
>> v2: fixed patch series thread
>> 
>> Alexander Kozyrev (10):
>>    ethdev: introduce flow pre-configuration hints
>>    ethdev: add flow item/action templates
>>    ethdev: bring in async queue-based flow rules operations
>>    app/testpmd: implement rte flow configuration
>>    app/testpmd: implement rte flow template management
>>    app/testpmd: implement rte flow table management
>>    app/testpmd: implement rte flow queue flow operations
>>    app/testpmd: implement rte flow push operations
>>    app/testpmd: implement rte flow pull operations
>>    app/testpmd: implement rte flow queue indirect actions
>> 
>
> Hi Jerin, Ajit, Ivan,
>
> As far as I can see you did some reviews in the previous versions,
> but not ack the patch.

Thanks for sending the reminder. Yes, I did review the series.
During the review, we did not find a common ground with regard
to possibly having a universal "task enqueue" method. However,
I was assured that such design would affect performance.

> Is there any objection to last version of the patch, if not I will
> proceed with it.

Personally, I have no strong objections. The v5 series seems a lot
clearer in a number of ways, yet, it is going to be experimental,
so I believe that if we run into some issues with this deisgn,
we will still have a chance to improve it to some extent.
In general, the author did a very good job applying that
many review notes. Thanks to Alexander for perseverance.

Please feel free to proceed with the series as you see fit.

>
>
> Hi Alex,
>
> As process we require at least one PMD implementation (it can be draft)
> to justify the API design.
>
> If there is no objection from above reviewers and PMD implementation
> exists before end of the week, I think we can get the set for -rc1.
>
> Thanks,
> ferruh
>

--
Ivan M
  

On Thu, Feb 10, 2022 at 9:30 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> On 2/9/2022 9:37 PM, Alexander Kozyrev wrote:
> > Three major changes to a generic RTE Flow API were implemented in order
> > to speed up flow rule insertion/destruction and adapt the API to the
> > needs of a datapath-focused flow rules management applications:
> >
> > 1. Pre-configuration hints.
> > Application may give us some hints on what type of resources are needed.
> > Introduce the configuration routine to prepare all the needed resources
> > inside a PMD/HW before any flow rules are created at the init stage.
> >
> > 2. Flow grouping using templates.
> > Use the knowledge about which flow rules are to be used in an application
> > and prepare item and action templates for them in advance. Group flow rules
> > with common patterns and actions together for better resource management.
> >
> > 3. Queue-based flow management.
> > Perform flow rule insertion/destruction asynchronously to spare the datapath
> > from blocking on RTE Flow API and allow it to continue with packet processing.
> > Enqueue flow rules operations and poll for the results later.
> >
> > testpmd examples are part of the patch series. PMD changes will follow.
> >
> > RFC: https://patchwork.dpdk.org/project/dpdk/cover/20211006044835.3936226-1-akozyrev@nvidia.com/
> >
> > Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
> > Acked-by: Ori Kam <orika@nvidia.com>
> >
> > ---
> > v4:
> > - removed structures versioning
> > - introduced new rte_flow_port_info structure for rte_flow_info_get API
> > - renamed rte_flow_table_create to rte_flow_template_table_create
> >
> > v3: addressed review comments and updated documentation
> > - added API to get info about pre-configurable resources
> > - renamed rte_flow_item_template to rte_flow_pattern_template
> > - renamed drain operation attribute to postpone
> > - renamed rte_flow_q_drain to rte_flow_q_push
> > - renamed rte_flow_q_dequeue to rte_flow_q_pull
> >
> > v2: fixed patch series thread
> >
> > Alexander Kozyrev (10):
> >    ethdev: introduce flow pre-configuration hints
> >    ethdev: add flow item/action templates
> >    ethdev: bring in async queue-based flow rules operations
> >    app/testpmd: implement rte flow configuration
> >    app/testpmd: implement rte flow template management
> >    app/testpmd: implement rte flow table management
> >    app/testpmd: implement rte flow queue flow operations
> >    app/testpmd: implement rte flow push operations
> >    app/testpmd: implement rte flow pull operations
> >    app/testpmd: implement rte flow queue indirect actions
> >
>
> Hi Jerin, Ajit, Ivan,
>
> As far as I can see you did some reviews in the previous versions,
> but not ack the patch.
> Is there any objection to last version of the patch, if not I will
> proceed with it.


Personally, I have no strong objections.  Based on the top level review,
It looks good to me on the application API side.
Please feel free to proceed with the series as you see fit.


>
> Hi Alex,
>
> As process we require at least one PMD implementation (it can be draft)
> to justify the API design.
>
> If there is no objection from above reviewers and PMD implementation
> exists before end of the week, I think we can get the set for -rc1.
>
> Thanks,
> ferruh