[v3] examples/eventdev: refactor ethdev port stop
Checks
Commit Message
Move eth stop code from "signal_handler" function to the end of "main"
function. There are two reasons for this:
First, this improves code maintenance and makes code look simple and
clear. Based on this change, after receiving the interrupt signal,
"fdata->done" is set as 1. Then the main thread will wait all worker
lcores to jump out of the loop. Finally, the main thread will stop and
then close eth dev port.
Second, for older version, the main thread first stops eth dev port and
then waits the end of worker lcore. This may cause errors because it may
stop the eth dev port which worker lcores are using. This moving change
can fix this by waiting all worker threads to exit and then stop the
eth dev port.
In the meanwhile, remove wmb in signal_handler.
This is because when the main lcore receive the stop signal, it stores 1
into fdata->done. And then the worker lcores load "fdata->done" and jump
out of the loop to stop running. Nothing should be stored after updating
fdata->done, so the wmb is unnecessary.
Fixes: 085edac2ca38 ("examples/eventdev_pipeline: support Tx adapter")
Cc: pbhagavatula@marvell.com
Cc: stable@dpdk.org
Suggested-by: Ruifeng Wang <ruifeng.wang@arm.com>
Signed-off-by: Feifei Wang <feifei.wang2@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Harry van Haaren <harry.van.haaren@intel.com>
---
v2:
1. Delete unused variable to fix build error
2. Reduce commit message length to fix coding style issues
v3:
1. Title improvement (Van Haaren)
examples/eventdev_pipeline/main.c | 16 ++++------------
1 file changed, 4 insertions(+), 12 deletions(-)
Comments
Hi Feifei,
>Move eth stop code from "signal_handler" function to the end of
>"main"
>function. There are two reasons for this:
>
>First, this improves code maintenance and makes code look simple and
>clear. Based on this change, after receiving the interrupt signal,
>"fdata->done" is set as 1. Then the main thread will wait all worker
>lcores to jump out of the loop. Finally, the main thread will stop and
>then close eth dev port.
>
>Second, for older version, the main thread first stops eth dev port and
>then waits the end of worker lcore. This may cause errors because it
>may
>stop the eth dev port which worker lcores are using. This moving
>change
>can fix this by waiting all worker threads to exit and then stop the
>eth dev port.
Apologies for the delayed reply,
In case of event dev the workers don't interact with eth device directly,
Instead eth device "injects" packets into event device and event device
is responsible for scheduling them to the workers.
If the producer is not stopped i.e. in this case eth device then the worker
threads might never exit and the main core would wait indefinitely for
workers to exit. This will be predominantly seen in cases where there are
only a few flows and large number of workers causing a lot of intra thread
dependency.
Regards,
Pavan.
>
>In the meanwhile, remove wmb in signal_handler.
>
>This is because when the main lcore receive the stop signal, it stores 1
>into fdata->done. And then the worker lcores load "fdata->done" and
>jump
>out of the loop to stop running. Nothing should be stored after
>updating
>fdata->done, so the wmb is unnecessary.
>
>Fixes: 085edac2ca38 ("examples/eventdev_pipeline: support Tx
>adapter")
>Cc: pbhagavatula@marvell.com
>Cc: stable@dpdk.org
>
>Suggested-by: Ruifeng Wang <ruifeng.wang@arm.com>
>Signed-off-by: Feifei Wang <feifei.wang2@arm.com>
>Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
>Reviewed-by: Honnappa Nagarahalli
><honnappa.nagarahalli@arm.com>
>Acked-by: Harry van Haaren <harry.van.haaren@intel.com>
>---
>
>v2:
>1. Delete unused variable to fix build error
>2. Reduce commit message length to fix coding style issues
>
>v3:
>1. Title improvement (Van Haaren)
>
> examples/eventdev_pipeline/main.c | 16 ++++------------
> 1 file changed, 4 insertions(+), 12 deletions(-)
>
>diff --git a/examples/eventdev_pipeline/main.c
>b/examples/eventdev_pipeline/main.c
>index 823f8b51c..fdbaf667b 100644
>--- a/examples/eventdev_pipeline/main.c
>+++ b/examples/eventdev_pipeline/main.c
>@@ -280,7 +280,6 @@ static void
> signal_handler(int signum)
> {
> static uint8_t once;
>- uint16_t portid;
>
> if (fdata->done)
> rte_exit(1, "Exiting on signal %d\n", signum);
>@@ -291,17 +290,6 @@ signal_handler(int signum)
> rte_event_dev_dump(0, stdout);
> once = 1;
> fdata->done = 1;
>- rte_smp_wmb();
>-
>- RTE_ETH_FOREACH_DEV(portid) {
>- rte_event_eth_rx_adapter_stop(portid);
>- rte_event_eth_tx_adapter_stop(portid);
>- if (rte_eth_dev_stop(portid) < 0)
>- printf("Failed to stop port %u", portid);
>- }
>-
>- rte_eal_mp_wait_lcore();
>-
> }
> if (signum == SIGTSTP)
> rte_event_dev_dump(0, stdout);
>@@ -465,6 +453,10 @@ main(int argc, char **argv)
> }
>
> RTE_ETH_FOREACH_DEV(portid) {
>+ rte_event_eth_rx_adapter_stop(portid);
>+ rte_event_eth_tx_adapter_stop(portid);
>+ if (rte_eth_dev_stop(portid) < 0)
>+ printf("Failed to stop port %u", portid);
> rte_eth_dev_close(portid);
> }
>
>--
>2.25.1
> -----邮件原件-----
> 发件人: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>
> 发送时间: 2021年1月5日 18:09
> 收件人: Feifei Wang <Feifei.Wang2@arm.com>; Harry van Haaren
> <harry.van.haaren@intel.com>; Nikhil Rao <nikhil.rao@intel.com>; Pavan
> Nikhilesh <pbhagavatula@caviumnetworks.com>
> 抄送: dev@dpdk.org; jerinj@marvell.com; nd <nd@arm.com>;
> stable@dpdk.org; Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa
> Nagarahalli <Honnappa.Nagarahalli@arm.com>
> 主题: RE: [EXT] [PATCH v3] examples/eventdev: refactor ethdev port stop
>
Hi, Pavan
> Hi Feifei,
>
> >Move eth stop code from "signal_handler" function to the end of "main"
> >function. There are two reasons for this:
> >
> >First, this improves code maintenance and makes code look simple and
> >clear. Based on this change, after receiving the interrupt signal,
> >"fdata->done" is set as 1. Then the main thread will wait all worker
> >lcores to jump out of the loop. Finally, the main thread will stop and
> >then close eth dev port.
> >
> >Second, for older version, the main thread first stops eth dev port and
> >then waits the end of worker lcore. This may cause errors because it
> >may stop the eth dev port which worker lcores are using. This moving
> >change can fix this by waiting all worker threads to exit and then stop
> >the eth dev port.
>
> Apologies for the delayed reply,
>
> In case of event dev the workers don't interact with eth device directly,
> Instead eth device "injects" packets into event device and event device is
> responsible for scheduling them to the workers.
>
> If the producer is not stopped i.e. in this case eth device then the worker
> threads might never exit and the main core would wait indefinitely for
> workers to exit. This will be predominantly seen in cases where there are
> only a few flows and large number of workers causing a lot of intra thread
> dependency.
For the case that the event device scheduling packets to the workers. Though the producer
is not stopped (eth device), when the main core receive the interrupt signal, the "fdata->done"
will be set as 1. Then all the workers load the value 1 of "fdata->done" and jump out of the loop
to finish their thread.
Best Regards
Feifei
>
> Regards,
> Pavan.
>
> >
> >In the meanwhile, remove wmb in signal_handler.
> >
> >This is because when the main lcore receive the stop signal, it stores
> >1 into fdata->done. And then the worker lcores load "fdata->done" and
> >jump out of the loop to stop running. Nothing should be stored after
> >updating
> >fdata->done, so the wmb is unnecessary.
> >
> >Fixes: 085edac2ca38 ("examples/eventdev_pipeline: support Tx
> >adapter")
> >Cc: pbhagavatula@marvell.com
> >Cc: stable@dpdk.org
> >
> >Suggested-by: Ruifeng Wang <ruifeng.wang@arm.com>
> >Signed-off-by: Feifei Wang <feifei.wang2@arm.com>
> >Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> >Reviewed-by: Honnappa Nagarahalli
> ><honnappa.nagarahalli@arm.com>
> >Acked-by: Harry van Haaren <harry.van.haaren@intel.com>
> >---
> >
> >v2:
> >1. Delete unused variable to fix build error 2. Reduce commit message
> >length to fix coding style issues
> >
> >v3:
> >1. Title improvement (Van Haaren)
> >
> > examples/eventdev_pipeline/main.c | 16 ++++------------
> > 1 file changed, 4 insertions(+), 12 deletions(-)
> >
> >diff --git a/examples/eventdev_pipeline/main.c
> >b/examples/eventdev_pipeline/main.c
> >index 823f8b51c..fdbaf667b 100644
> >--- a/examples/eventdev_pipeline/main.c
> >+++ b/examples/eventdev_pipeline/main.c
> >@@ -280,7 +280,6 @@ static void
> > signal_handler(int signum)
> > {
> > static uint8_t once;
> >- uint16_t portid;
> >
> > if (fdata->done)
> > rte_exit(1, "Exiting on signal %d\n", signum); @@ -291,17
> +290,6 @@
> >signal_handler(int signum)
> > rte_event_dev_dump(0, stdout);
> > once = 1;
> > fdata->done = 1;
> >- rte_smp_wmb();
> >-
> >- RTE_ETH_FOREACH_DEV(portid) {
> >- rte_event_eth_rx_adapter_stop(portid);
> >- rte_event_eth_tx_adapter_stop(portid);
> >- if (rte_eth_dev_stop(portid) < 0)
> >- printf("Failed to stop port %u", portid);
> >- }
> >-
> >- rte_eal_mp_wait_lcore();
> >-
> > }
> > if (signum == SIGTSTP)
> > rte_event_dev_dump(0, stdout);
> >@@ -465,6 +453,10 @@ main(int argc, char **argv)
> > }
> >
> > RTE_ETH_FOREACH_DEV(portid) {
> >+ rte_event_eth_rx_adapter_stop(portid);
> >+ rte_event_eth_tx_adapter_stop(portid);
> >+ if (rte_eth_dev_stop(portid) < 0)
> >+ printf("Failed to stop port %u", portid);
> > rte_eth_dev_close(portid);
> > }
> >
> >--
> >2.25.1
> -----邮件原件-----
> 发件人: Feifei Wang
> 发送时间: 2021年1月14日 14:24
> 收件人: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Harry
> van Haaren <harry.van.haaren@intel.com>; Nikhil Rao
> <nikhil.rao@intel.com>; Pavan Nikhilesh
> <pbhagavatula@caviumnetworks.com>
> 抄送: dev@dpdk.org; jerinj@marvell.com; nd <nd@arm.com>;
> stable@dpdk.org; Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa
> Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> 主题: 回复: [EXT] [PATCH v3] examples/eventdev: refactor ethdev port stop
>
> > -----邮件原件-----
> > 发件人: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>
> > 发送时间: 2021年1月5日 18:09
> > 收件人: Feifei Wang <Feifei.Wang2@arm.com>; Harry van Haaren
> > <harry.van.haaren@intel.com>; Nikhil Rao <nikhil.rao@intel.com>; Pavan
> > Nikhilesh <pbhagavatula@caviumnetworks.com>
> > 抄送: dev@dpdk.org; jerinj@marvell.com; nd <nd@arm.com>;
> > stable@dpdk.org; Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa
> > Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > 主题: RE: [EXT] [PATCH v3] examples/eventdev: refactor ethdev port stop
> >
> Hi, Pavan
>
> > Hi Feifei,
> >
> > >Move eth stop code from "signal_handler" function to the end of "main"
> > >function. There are two reasons for this:
> > >
> > >First, this improves code maintenance and makes code look simple and
> > >clear. Based on this change, after receiving the interrupt signal,
> > >"fdata->done" is set as 1. Then the main thread will wait all worker
> > >lcores to jump out of the loop. Finally, the main thread will stop
> > >and then close eth dev port.
> > >
> > >Second, for older version, the main thread first stops eth dev port
> > >and then waits the end of worker lcore. This may cause errors because
> > >it may stop the eth dev port which worker lcores are using. This
> > >moving change can fix this by waiting all worker threads to exit and
> > >then stop the eth dev port.
> >
> > Apologies for the delayed reply,
> >
> > In case of event dev the workers don't interact with eth device
> > directly, Instead eth device "injects" packets into event device and
> > event device is responsible for scheduling them to the workers.
> >
> > If the producer is not stopped i.e. in this case eth device then the
> > worker threads might never exit and the main core would wait
> > indefinitely for workers to exit. This will be predominantly seen in
> > cases where there are only a few flows and large number of workers
> > causing a lot of intra thread dependency.
>
> For the case that the event device scheduling packets to the workers.
> Though the producer is not stopped (eth device), when the main core
> receive the interrupt signal, the "fdata->done"
> will be set as 1. Then all the workers load the value 1 of "fdata->done" and
> jump out of the loop to finish their thread.
>
And I also test the case you said above with this patch, when I send interrupt signal(ctrl + c), IXIA
is running and eth device is also working, then the program can exit normally.
HW:
Traffic generator: IXIA
Nics: ixgbe 82599ES 10-Gigabit
Architecture: aarch64
CPU: Cortex-A72
With this patch:
@@ -280,7 +280,6 @@ static void
signal_handler(int signum)
{
static uint8_t once;
- uint16_t portid;
if (fdata->done)
rte_exit(1, "Exiting on signal %d\n", signum);
@@ -291,17 +290,6 @@ signal_handler(int signum)
rte_event_dev_dump(0, stdout);
once = 1;
fdata->done = 1;
- rte_smp_wmb();
-
- RTE_ETH_FOREACH_DEV(portid) {
- rte_event_eth_rx_adapter_stop(portid);
- rte_event_eth_tx_adapter_stop(portid);
- if (rte_eth_dev_stop(portid) < 0)
- printf("Failed to stop port %u", portid);
- }
-
- rte_eal_mp_wait_lcore();
-
}
if (signum == SIGTSTP)
rte_event_dev_dump(0, stdout);
@@ -465,6 +453,10 @@ main(int argc, char **argv)
}
RTE_ETH_FOREACH_DEV(portid) {
+ rte_event_eth_rx_adapter_stop(portid);
+ rte_event_eth_tx_adapter_stop(portid);
+ if (rte_eth_dev_stop(portid) < 0)
+ printf("Failed to stop port %u", portid);
rte_eth_dev_close(portid);
}