[v2,1/5] ethdev: add capability to keep flow rules on restart

Message ID 20211015161822.3099818-2-dkozlyuk@nvidia.com (mailing list archive)
State Superseded, archived
Delegated to: Ferruh Yigit
Headers
Series Flow entites behavior on port restart |

Checks

Context Check Description
ci/checkpatch warning coding style issues

Commit Message

Dmitry Kozlyuk Oct. 15, 2021, 4:18 p.m. UTC
  Currently, it is not specified what happens to the flow rules when
the device is stopped, possibly reconfigured, then started.
If flow rules were kept, it could be convenient for application
developers, because they wouldn't need to save and restore them.
However, due to the number of flows and possible creation rate it is
impractical to save all flow rules in DPDK layer. This means that flow
rules persistence really depends on whether PMD and HW can implement it
efficiently. It can also be limited by the rule item and action types,
and its attributes transfer bit, which together comprise the rule kind.

Add a device capability bit for PMDs that can keep at least some
of the flow rules across restart. Without this capability behavior
is still unspecified, which is now explicitly stated.
Declare that the application can test for persitence of flow rules
of a particular kind by attempting to create a rule of that kind
when the device is stopped and checking for the specific error.
This is logical because if the PMD can to create the flow rule
when the device is not started and use it after the start happens,
it is natural that it can move its internal flow rule object
to the same state when the device is stopped and restore the state
when the device is started.

If the device is being reconfigured in a way that is incompatible with
existing flow rules, PMD is required to report an error.
This is mandatory, because flow API does not supply users with
capabilities, so this is the only way for a user to learn that
configuration is invalid. For example, if queue count changes and the
action of a flow rule specifies queues that are going away, the user
must update or remove the flow rule before removing the queues.

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
---
 doc/guides/prog_guide/rte_flow.rst | 27 +++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h            |  7 +++++++
 lib/ethdev/rte_flow.h              |  1 +
 3 files changed, 35 insertions(+)
  

Comments

Andrew Rybchenko Oct. 18, 2021, 8:56 a.m. UTC | #1
On 10/15/21 7:18 PM, Dmitry Kozlyuk wrote:
> Currently, it is not specified what happens to the flow rules when
> the device is stopped, possibly reconfigured, then started.
> If flow rules were kept, it could be convenient for application
> developers, because they wouldn't need to save and restore them.
> However, due to the number of flows and possible creation rate it is
> impractical to save all flow rules in DPDK layer. This means that flow
> rules persistence really depends on whether PMD and HW can implement it
> efficiently. It can also be limited by the rule item and action types,
> and its attributes transfer bit, which together comprise the rule kind.
> 
> Add a device capability bit for PMDs that can keep at least some
> of the flow rules across restart. Without this capability behavior
> is still unspecified, which is now explicitly stated.
> Declare that the application can test for persitence of flow rules
> of a particular kind by attempting to create a rule of that kind
> when the device is stopped and checking for the specific error.

stopped -> configured but not yet started

> This is logical because if the PMD can to create the flow rule

can to -> can

> when the device is not started and use it after the start happens,
> it is natural that it can move its internal flow rule object
> to the same state when the device is stopped and restore the state
> when the device is started.
> 
> If the device is being reconfigured in a way that is incompatible with
> existing flow rules, PMD is required to report an error.
> This is mandatory, because flow API does not supply users with
> capabilities, so this is the only way for a user to learn that
> configuration is invalid. For example, if queue count changes and the
> action of a flow rule specifies queues that are going away, the user
> must update or remove the flow rule before removing the queues.
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> ---
>  doc/guides/prog_guide/rte_flow.rst | 27 +++++++++++++++++++++++++++
>  lib/ethdev/rte_ethdev.h            |  7 +++++++
>  lib/ethdev/rte_flow.h              |  1 +
>  3 files changed, 35 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/rte_flow.rst b/doc/guides/prog_guide/rte_flow.rst
> index 2b42d5ec8c..b0ced4209b 100644
> --- a/doc/guides/prog_guide/rte_flow.rst
> +++ b/doc/guides/prog_guide/rte_flow.rst
> @@ -87,6 +87,33 @@ To avoid resource leaks on the PMD side, handles must be explicitly
>  destroyed by the application before releasing associated resources such as
>  queues and ports.
>  
> +By default it is unspecified if the flow rules persist after the device stop.

or can be created before the first device start

> +If ``RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP`` is not advertised,
> +then rules must be explicitly flushed before stopping the device
> +if the application needs to ensure they are removed.
> +If it is advertised, this means the PMD can keep at least some rules
> +across the device stop and start with possible reconfiguration in between.
> +However, it may be only supported for some kinds of rules.
> +The kind is a combination of the following rule properties:
> +
> +- the sequence of item types;
> +- the sequence of action types;
> +- the value of the transfer attribute.
> +
> +To test if a particular kind of rules is kept, the application must try
> +to create a valid rule of that kind when the device is stopped
> +(after it has been configured or started previously).
> +If it succeeds, all rules of the same kind are kept at the device stop.
> +If it fails with an error of type ``RTE_FLOW_ERROR_TYPE_STATE``,
> +rules of this kind are flushed when the device is stopped.
> +Rules of a kept kind that are created when the device is stopped, including
> +the rules created for the test, will be kept after the device is started.

It must be defined what application should expect for
not tested rule kinds.

For me about check sounds extremely complicated and hardly
doable. Yes, some applications know kinds of rule it would
like to create, but some, like OvS, do not. Please, correct
me if I'm wrong. OvS knows which types of actions and even
possible combinations of actions (harder, but still possible)
it would like to install. But all possible combinations of
items together with all possible combinations of actions
could be very-very big.

May be I still misunderstand the above idea.

> +Some configuration changes may be incompatible with existing rules.
> +In this case ``rte_eth_dev_configure()``, ``rte_eth_rx/tx_queue_setup()``,
> +and/or ``rte_eth_dev_start()`` will fail with a log message from the PMD that
> +should be similar to the one that would be emitted by ``rte_flow_create()``
> +if an attempt was made to create the offending rule with the new configuration.
> +
>  The following sections cover:
>  
>  - **Attributes** (represented by ``struct rte_flow_attr``): properties of a
> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
> index 6d80514ba7..a0b388bb25 100644
> --- a/lib/ethdev/rte_ethdev.h
> +++ b/lib/ethdev/rte_ethdev.h
> @@ -90,6 +90,11 @@
>   *     - flow director filtering mode (but not filtering rules)
>   *     - NIC queue statistics mappings
>   *
> + * The following configuration may be retained or not
> + * depending on the device capabilities:
> + *
> + *     - flow rules
> + *
>   * Any other configuration will not be stored and will need to be re-entered
>   * before a call to rte_eth_dev_start().
>   *
> @@ -1445,6 +1450,8 @@ struct rte_eth_conf {
>  #define RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP 0x00000001
>  /** Device supports Tx queue setup after device started. */
>  #define RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP 0x00000002
> +/** Device supports keeping flow rules across restart. */
> +#define RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP 0x00000004
>  /**@}*/
>  
>  /*
> diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
> index a89945061a..aa0182d021 100644
> --- a/lib/ethdev/rte_flow.h
> +++ b/lib/ethdev/rte_flow.h
> @@ -3344,6 +3344,7 @@ enum rte_flow_error_type {
>  	RTE_FLOW_ERROR_TYPE_ACTION_NUM, /**< Number of actions. */
>  	RTE_FLOW_ERROR_TYPE_ACTION_CONF, /**< Action configuration. */
>  	RTE_FLOW_ERROR_TYPE_ACTION, /**< Specific action. */
> +	RTE_FLOW_ERROR_TYPE_STATE, /**< Current device state. */
>  };
>  
>  /**
>
  
Qi Zhang Oct. 18, 2021, 1:06 p.m. UTC | #2
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Dmitry Kozlyuk
> Sent: Saturday, October 16, 2021 12:18 AM
> To: dev@dpdk.org
> Cc: Ori Kam <orika@oss.nvidia.com>; Thomas Monjalon
> <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Andrew
> Rybchenko <andrew.rybchenko@oktetlabs.ru>
> Subject: [dpdk-dev] [PATCH v2 1/5] ethdev: add capability to keep flow rules
> on restart
> 
> Currently, it is not specified what happens to the flow rules when the device is
> stopped, possibly reconfigured, then started.
> If flow rules were kept, it could be convenient for application developers,
> because they wouldn't need to save and restore them.
> However, due to the number of flows and possible creation rate it is
> impractical to save all flow rules in DPDK layer. This means that flow rules
> persistence really depends on whether PMD and HW can implement it
> efficiently. It can also be limited by the rule item and action types, and its
> attributes transfer bit, which together comprise the rule kind.
> 
> Add a device capability bit for PMDs that can keep at least some of the flow
> rules across restart. Without this capability behavior is still unspecified, which
> is now explicitly stated.
> Declare that the application can test for persitence of flow rules of a particular
> kind by attempting to create a rule of that kind when the device is stopped
> and checking for the specific error.
> This is logical because if the PMD can to create the flow rule when the device
> is not started and use it after the start happens, it is natural that it can move
> its internal flow rule object to the same state when the device is stopped and
> restore the state when the device is started.
> 
> If the device is being reconfigured in a way that is incompatible with existing
> flow rules, PMD is required to report an error.
> This is mandatory, because flow API does not supply users with capabilities, so
> this is the only way for a user to learn that configuration is invalid.

What if a PMD does not flush rules during start /stop cycle, but just want to simply flush rules during dev_config? 
Is it reasonable to take above as an typical implementation to avoid all the complexity for handling the conflicts?

1. queues are destroyed and re-created with a different number which may impact "to queue" action.
2. hash key may be overwritten which impact RSS result.
3. offload flags changes may impact data path selection which cause mark action does not work.
....





> example, if queue count changes and the action of a flow rule specifies queues
> that are going away, the user must update or remove the flow rule before
> removing the queues.
> 
> Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> ---
>  doc/guides/prog_guide/rte_flow.rst | 27 +++++++++++++++++++++++++++
>  lib/ethdev/rte_ethdev.h            |  7 +++++++
>  lib/ethdev/rte_flow.h              |  1 +
>  3 files changed, 35 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/rte_flow.rst
> b/doc/guides/prog_guide/rte_flow.rst
> index 2b42d5ec8c..b0ced4209b 100644
> --- a/doc/guides/prog_guide/rte_flow.rst
> +++ b/doc/guides/prog_guide/rte_flow.rst
> @@ -87,6 +87,33 @@ To avoid resource leaks on the PMD side, handles must
> be explicitly  destroyed by the application before releasing associated
> resources such as  queues and ports.
> 
> +By default it is unspecified if the flow rules persist after the device stop.
> +If ``RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP`` is not advertised, then rules
> +must be explicitly flushed before stopping the device if the
> +application needs to ensure they are removed.
> +If it is advertised, this means the PMD can keep at least some rules
> +across the device stop and start with possible reconfiguration in between.
> +However, it may be only supported for some kinds of rules.
> +The kind is a combination of the following rule properties:
> +
> +- the sequence of item types;
> +- the sequence of action types;
> +- the value of the transfer attribute.
> +
> +To test if a particular kind of rules is kept, the application must try
> +to create a valid rule of that kind when the device is stopped (after
> +it has been configured or started previously).
> +If it succeeds, all rules of the same kind are kept at the device stop.
> +If it fails with an error of type ``RTE_FLOW_ERROR_TYPE_STATE``, rules
> +of this kind are flushed when the device is stopped.
> +Rules of a kept kind that are created when the device is stopped,
> +including the rules created for the test, will be kept after the device is started.
> +Some configuration changes may be incompatible with existing rules.
> +In this case ``rte_eth_dev_configure()``,
> +``rte_eth_rx/tx_queue_setup()``, and/or ``rte_eth_dev_start()`` will
> +fail with a log message from the PMD that should be similar to the one
> +that would be emitted by ``rte_flow_create()`` if an attempt was made to
> create the offending rule with the new configuration.
> +
>  The following sections cover:
> 
>  - **Attributes** (represented by ``struct rte_flow_attr``): properties of a diff
> --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index
> 6d80514ba7..a0b388bb25 100644
> --- a/lib/ethdev/rte_ethdev.h
> +++ b/lib/ethdev/rte_ethdev.h
> @@ -90,6 +90,11 @@
>   *     - flow director filtering mode (but not filtering rules)
>   *     - NIC queue statistics mappings
>   *
> + * The following configuration may be retained or not
> + * depending on the device capabilities:
> + *
> + *     - flow rules
> + *
>   * Any other configuration will not be stored and will need to be re-entered
>   * before a call to rte_eth_dev_start().
>   *
> @@ -1445,6 +1450,8 @@ struct rte_eth_conf {  #define
> RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP 0x00000001
>  /** Device supports Tx queue setup after device started. */  #define
> RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP 0x00000002
> +/** Device supports keeping flow rules across restart. */ #define
> +RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP 0x00000004
>  /**@}*/
> 
>  /*
> diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h index
> a89945061a..aa0182d021 100644
> --- a/lib/ethdev/rte_flow.h
> +++ b/lib/ethdev/rte_flow.h
> @@ -3344,6 +3344,7 @@ enum rte_flow_error_type {
>  	RTE_FLOW_ERROR_TYPE_ACTION_NUM, /**< Number of actions. */
>  	RTE_FLOW_ERROR_TYPE_ACTION_CONF, /**< Action configuration. */
>  	RTE_FLOW_ERROR_TYPE_ACTION, /**< Specific action. */
> +	RTE_FLOW_ERROR_TYPE_STATE, /**< Current device state. */
>  };
> 
>  /**
> --
> 2.25.1
  
Dmitry Kozlyuk Oct. 18, 2021, 10:51 p.m. UTC | #3
> -----Original Message-----
> From: Zhang, Qi Z <qi.z.zhang@intel.com>
> Sent: 18 октября 2021 г. 16:06
> To: Dmitry Kozlyuk <dkozlyuk@nvidia.com>; dev@dpdk.org
> Cc: Ori Kam <orika@nvidia.com>; NBU-Contact-Thomas Monjalon
> <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Andrew
> Rybchenko <andrew.rybchenko@oktetlabs.ru>
> Subject: RE: [dpdk-dev] [PATCH v2 1/5] ethdev: add capability to keep flow
> rules on restart
> 
> External email: Use caution opening links or attachments
> 
> 
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Dmitry Kozlyuk
> > Sent: Saturday, October 16, 2021 12:18 AM
> > To: dev@dpdk.org
> > Cc: Ori Kam <orika@oss.nvidia.com>; Thomas Monjalon
> > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Andrew
> > Rybchenko <andrew.rybchenko@oktetlabs.ru>
> > Subject: [dpdk-dev] [PATCH v2 1/5] ethdev: add capability to keep flow
> rules
> > on restart
> >
> > Currently, it is not specified what happens to the flow rules when the
> device is
> > stopped, possibly reconfigured, then started.
> > If flow rules were kept, it could be convenient for application
> developers,
> > because they wouldn't need to save and restore them.
> > However, due to the number of flows and possible creation rate it is
> > impractical to save all flow rules in DPDK layer. This means that flow
> rules
> > persistence really depends on whether PMD and HW can implement it
> > efficiently. It can also be limited by the rule item and action types,
> and its
> > attributes transfer bit, which together comprise the rule kind.
> >
> > Add a device capability bit for PMDs that can keep at least some of the
> flow
> > rules across restart. Without this capability behavior is still
> unspecified, which
> > is now explicitly stated.
> > Declare that the application can test for persitence of flow rules of a
> particular
> > kind by attempting to create a rule of that kind when the device is
> stopped
> > and checking for the specific error.
> > This is logical because if the PMD can to create the flow rule when the
> device
> > is not started and use it after the start happens, it is natural that it
> can move
> > its internal flow rule object to the same state when the device is
> stopped and
> > restore the state when the device is started.
> >
> > If the device is being reconfigured in a way that is incompatible with
> existing
> > flow rules, PMD is required to report an error.
> > This is mandatory, because flow API does not supply users with
> capabilities, so
> > this is the only way for a user to learn that configuration is invalid.
> 
> What if a PMD does not flush rules during start /stop cycle, but just want
> to simply flush rules during dev_config?
> Is it reasonable to take above as an typical implementation to avoid all
> the complexity for handling the conflicts?
> 
> 1. queues are destroyed and re-created with a different number which may
> impact "to queue" action.
> 2. hash key may be overwritten which impact RSS result.
> 3. offload flags changes may impact data path selection which cause mark
> action does not work.
> ....

Hello Qi,

Yes, it sounds reasonable that rules do need not to persist across reconfiguration.
Unlike indirect actions, they are too numerous for PMD to track and check.
I'm not sure rte_eth_dev_configure() should be specified to implicitly flush them.
Some PMDs may wish to preserve the rules even then in the future,
so we don't want applications to rely on configure flushing the rules.
It can be specified that applications should flush the rules themselves before.
  
Qi Zhang Oct. 19, 2021, 1 a.m. UTC | #4
> -----Original Message-----
> From: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> Sent: Tuesday, October 19, 2021 6:51 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org
> Cc: Ori Kam <orika@nvidia.com>; NBU-Contact-Thomas Monjalon
> <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Andrew
> Rybchenko <andrew.rybchenko@oktetlabs.ru>
> Subject: RE: [dpdk-dev] [PATCH v2 1/5] ethdev: add capability to keep flow
> rules on restart
> 
> > -----Original Message-----
> > From: Zhang, Qi Z <qi.z.zhang@intel.com>
> > Sent: 18 октября 2021 г. 16:06
> > To: Dmitry Kozlyuk <dkozlyuk@nvidia.com>; dev@dpdk.org
> > Cc: Ori Kam <orika@nvidia.com>; NBU-Contact-Thomas Monjalon
> > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Andrew
> > Rybchenko <andrew.rybchenko@oktetlabs.ru>
> > Subject: RE: [dpdk-dev] [PATCH v2 1/5] ethdev: add capability to keep
> > flow rules on restart
> >
> > External email: Use caution opening links or attachments
> >
> >
> > > -----Original Message-----
> > > From: dev <dev-bounces@dpdk.org> On Behalf Of Dmitry Kozlyuk
> > > Sent: Saturday, October 16, 2021 12:18 AM
> > > To: dev@dpdk.org
> > > Cc: Ori Kam <orika@oss.nvidia.com>; Thomas Monjalon
> > > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>;
> > > Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> > > Subject: [dpdk-dev] [PATCH v2 1/5] ethdev: add capability to keep
> > > flow
> > rules
> > > on restart
> > >
> > > Currently, it is not specified what happens to the flow rules when
> > > the
> > device is
> > > stopped, possibly reconfigured, then started.
> > > If flow rules were kept, it could be convenient for application
> > developers,
> > > because they wouldn't need to save and restore them.
> > > However, due to the number of flows and possible creation rate it is
> > > impractical to save all flow rules in DPDK layer. This means that
> > > flow
> > rules
> > > persistence really depends on whether PMD and HW can implement it
> > > efficiently. It can also be limited by the rule item and action
> > > types,
> > and its
> > > attributes transfer bit, which together comprise the rule kind.
> > >
> > > Add a device capability bit for PMDs that can keep at least some of
> > > the
> > flow
> > > rules across restart. Without this capability behavior is still
> > unspecified, which
> > > is now explicitly stated.
> > > Declare that the application can test for persitence of flow rules
> > > of a
> > particular
> > > kind by attempting to create a rule of that kind when the device is
> > stopped
> > > and checking for the specific error.
> > > This is logical because if the PMD can to create the flow rule when
> > > the
> > device
> > > is not started and use it after the start happens, it is natural
> > > that it
> > can move
> > > its internal flow rule object to the same state when the device is
> > stopped and
> > > restore the state when the device is started.
> > >
> > > If the device is being reconfigured in a way that is incompatible
> > > with
> > existing
> > > flow rules, PMD is required to report an error.
> > > This is mandatory, because flow API does not supply users with
> > capabilities, so
> > > this is the only way for a user to learn that configuration is invalid.
> >
> > What if a PMD does not flush rules during start /stop cycle, but just
> > want to simply flush rules during dev_config?
> > Is it reasonable to take above as an typical implementation to avoid
> > all the complexity for handling the conflicts?
> >
> > 1. queues are destroyed and re-created with a different number which
> > may impact "to queue" action.
> > 2. hash key may be overwritten which impact RSS result.
> > 3. offload flags changes may impact data path selection which cause
> > mark action does not work.
> > ....
> 
> Hello Qi,
> 
> Yes, it sounds reasonable that rules do need not to persist across
> reconfiguration.
> Unlike indirect actions, they are too numerous for PMD to track and check.
> I'm not sure rte_eth_dev_configure() should be specified to implicitly flush
> them.
> Some PMDs may wish to preserve the rules even then in the future, so we
> don't want applications to rely on configure flushing the rules.
> It can be specified that applications should flush the rules themselves before.

OK, I'm trying to figure out how to set this "keep" capability for a PMD that don't want application to re-create rules after dev_stop, but still want rules be flushed before reconfigure.
I think the answer is it should expose the "keep" capability and simply return error in dev_configure if any rules exists, thanks
  
Dmitry Kozlyuk Oct. 19, 2021, 12:38 p.m. UTC | #5
> > [...]
> > Add a device capability bit for PMDs that can keep at least some
> > of the flow rules across restart. Without this capability behavior
> > is still unspecified, which is now explicitly stated.
> > Declare that the application can test for persitence of flow rules
> > of a particular kind by attempting to create a rule of that kind
> > when the device is stopped and checking for the specific error.
> 
> stopped -> configured but not yet started

Correct, fixed in v3.

> > diff --git a/doc/guides/prog_guide/rte_flow.rst
> b/doc/guides/prog_guide/rte_flow.rst
> > index 2b42d5ec8c..b0ced4209b 100644
> > --- a/doc/guides/prog_guide/rte_flow.rst
> > +++ b/doc/guides/prog_guide/rte_flow.rst
> > @@ -87,6 +87,33 @@ To avoid resource leaks on the PMD side, handles must
> be explicitly
> >  destroyed by the application before releasing associated resources such
> as
> >  queues and ports.
> >
> > +By default it is unspecified if the flow rules persist after the device
> stop.
> 
> or can be created before the first device start

Correct, fixed in v3.

> 
> > +If ``RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP`` is not advertised,
> > +then rules must be explicitly flushed before stopping the device
> > +if the application needs to ensure they are removed.
> > +If it is advertised, this means the PMD can keep at least some rules
> > +across the device stop and start with possible reconfiguration in
> between.
> > +However, it may be only supported for some kinds of rules.
> > +The kind is a combination of the following rule properties:
> > +
> > +- the sequence of item types;
> > +- the sequence of action types;
> > +- the value of the transfer attribute.
> > +
> > +To test if a particular kind of rules is kept, the application must try
> > +to create a valid rule of that kind when the device is stopped
> > +(after it has been configured or started previously).
> > +If it succeeds, all rules of the same kind are kept at the device stop.
> > +If it fails with an error of type ``RTE_FLOW_ERROR_TYPE_STATE``,
> > +rules of this kind are flushed when the device is stopped.
> > +Rules of a kept kind that are created when the device is stopped,
> including
> > +the rules created for the test, will be kept after the device is
> started.
> 
> It must be defined what application should expect for
> not tested rule kinds.
> 
> For me about check sounds extremely complicated and hardly
> doable. Yes, some applications know kinds of rule it would
> like to create, but some, like OvS, do not. Please, correct
> me if I'm wrong. OvS knows which types of actions and even
> possible combinations of actions (harder, but still possible)
> it would like to install. But all possible combinations of
> items together with all possible combinations of actions
> could be very-very big.
> 
> May be I still misunderstand the above idea.

This is a very valid concern.
After an offline consideration me and Ori concluded
that an item/action type + transfer bit ("a feature") is enough.
That is, if some feature cannot be kept, no rules using it can be kept.
  

Patch

diff --git a/doc/guides/prog_guide/rte_flow.rst b/doc/guides/prog_guide/rte_flow.rst
index 2b42d5ec8c..b0ced4209b 100644
--- a/doc/guides/prog_guide/rte_flow.rst
+++ b/doc/guides/prog_guide/rte_flow.rst
@@ -87,6 +87,33 @@  To avoid resource leaks on the PMD side, handles must be explicitly
 destroyed by the application before releasing associated resources such as
 queues and ports.
 
+By default it is unspecified if the flow rules persist after the device stop.
+If ``RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP`` is not advertised,
+then rules must be explicitly flushed before stopping the device
+if the application needs to ensure they are removed.
+If it is advertised, this means the PMD can keep at least some rules
+across the device stop and start with possible reconfiguration in between.
+However, it may be only supported for some kinds of rules.
+The kind is a combination of the following rule properties:
+
+- the sequence of item types;
+- the sequence of action types;
+- the value of the transfer attribute.
+
+To test if a particular kind of rules is kept, the application must try
+to create a valid rule of that kind when the device is stopped
+(after it has been configured or started previously).
+If it succeeds, all rules of the same kind are kept at the device stop.
+If it fails with an error of type ``RTE_FLOW_ERROR_TYPE_STATE``,
+rules of this kind are flushed when the device is stopped.
+Rules of a kept kind that are created when the device is stopped, including
+the rules created for the test, will be kept after the device is started.
+Some configuration changes may be incompatible with existing rules.
+In this case ``rte_eth_dev_configure()``, ``rte_eth_rx/tx_queue_setup()``,
+and/or ``rte_eth_dev_start()`` will fail with a log message from the PMD that
+should be similar to the one that would be emitted by ``rte_flow_create()``
+if an attempt was made to create the offending rule with the new configuration.
+
 The following sections cover:
 
 - **Attributes** (represented by ``struct rte_flow_attr``): properties of a
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index 6d80514ba7..a0b388bb25 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -90,6 +90,11 @@ 
  *     - flow director filtering mode (but not filtering rules)
  *     - NIC queue statistics mappings
  *
+ * The following configuration may be retained or not
+ * depending on the device capabilities:
+ *
+ *     - flow rules
+ *
  * Any other configuration will not be stored and will need to be re-entered
  * before a call to rte_eth_dev_start().
  *
@@ -1445,6 +1450,8 @@  struct rte_eth_conf {
 #define RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP 0x00000001
 /** Device supports Tx queue setup after device started. */
 #define RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP 0x00000002
+/** Device supports keeping flow rules across restart. */
+#define RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP 0x00000004
 /**@}*/
 
 /*
diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
index a89945061a..aa0182d021 100644
--- a/lib/ethdev/rte_flow.h
+++ b/lib/ethdev/rte_flow.h
@@ -3344,6 +3344,7 @@  enum rte_flow_error_type {
 	RTE_FLOW_ERROR_TYPE_ACTION_NUM, /**< Number of actions. */
 	RTE_FLOW_ERROR_TYPE_ACTION_CONF, /**< Action configuration. */
 	RTE_FLOW_ERROR_TYPE_ACTION, /**< Specific action. */
+	RTE_FLOW_ERROR_TYPE_STATE, /**< Current device state. */
 };
 
 /**