Message ID | 20230424130208.9517-1-fengchengwen@huawei.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E3360429DB; Mon, 24 Apr 2023 15:09:14 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D32C6410D0; Mon, 24 Apr 2023 15:09:14 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id A22DE40FAE for <dev@dpdk.org>; Mon, 24 Apr 2023 15:09:13 +0200 (CEST) Received: from dggpeml500024.china.huawei.com (unknown [172.30.72.56]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4Q4lj52RmJzSv6C; Mon, 24 Apr 2023 21:04:57 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by dggpeml500024.china.huawei.com (7.185.36.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Mon, 24 Apr 2023 21:09:11 +0800 From: Chengwen Feng <fengchengwen@huawei.com> To: <thomas@monjalon.net>, <ferruh.yigit@amd.com> CC: <dev@dpdk.org> Subject: [RFC 0/3] introduce coroutine library Date: Mon, 24 Apr 2023 13:02:05 +0000 Message-ID: <20230424130208.9517-1-fengchengwen@huawei.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpeml500024.china.huawei.com (7.185.36.10) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org |
Series |
introduce coroutine library
|
|
Message
fengchengwen
April 24, 2023, 1:02 p.m. UTC
This patchset introduces the coroutine library which will help refactor the hns3 PMD's reset process. The hns3 single function reset process consists of the following steps: 1.stop_service(); 2.prepare_reset(); 3.delay(100ms); 4.notify_hw(); 5.wait_hw_reset_done(); // multiple sleep waits are involved. 6.reinit(); 7.restore_conf(); If the DPDK process take over multiple hns3 functions (e.g. 100), it's impractical to reset and restore functions in sequence: 1.proc_func(001); // will completed in 100+ms range. 2.proc_func(002); // will completed in 100~200+ms range. ... x.proc_func(100); // will completed in 9900~10000+ms range. The later functions will process fail because it's too late to deal with. One solution is that create a reset thread for each function, and it will lead to large number of threads if the DPDK process take over multiple hns3 functions. So the current hns3 driver uses asynchronous mechanism, for examples, it use rte_eal_alarm_set() when process delay(100ms), it splits a serial process into multiple asynchronous processes, and the code is complex and difficult to understand. The coroutine is a good mechanism to provide programmers with the simplicity of keeping serial processes within a limited number of threads. This patchset use <ucontext.h> to build the coroutine framework, and it just provides a demo. More APIs maybe added in the future. In addition, we would like to ask the community whether it it possible to accept the library. If not, whether it is allowed to provide the library in hns3 PMD. Chengwen Feng (3): lib/coroutine: add coroutine library examples/coroutine: support coroutine examples net/hns3: refactor reset process with coroutine drivers/net/hns3/hns3_ethdev.c | 217 ++++++++++++++++++++++++++++++ drivers/net/hns3/hns3_ethdev.h | 3 + drivers/net/hns3/hns3_intr.c | 38 ++++++ drivers/net/hns3/meson.build | 2 +- examples/coroutine/main.c | 153 +++++++++++++++++++++ examples/coroutine/meson.build | 10 ++ examples/meson.build | 1 + lib/coroutine/meson.build | 8 ++ lib/coroutine/rte_coroutine.c | 190 ++++++++++++++++++++++++++ lib/coroutine/rte_coroutine.h | 110 +++++++++++++++ lib/coroutine/rte_coroutine_imp.h | 46 +++++++ lib/coroutine/version.map | 11 ++ lib/meson.build | 1 + 13 files changed, 789 insertions(+), 1 deletion(-) create mode 100644 examples/coroutine/main.c create mode 100644 examples/coroutine/meson.build create mode 100644 lib/coroutine/meson.build create mode 100644 lib/coroutine/rte_coroutine.c create mode 100644 lib/coroutine/rte_coroutine.h create mode 100644 lib/coroutine/rte_coroutine_imp.h create mode 100644 lib/coroutine/version.map
Comments
On Mon, 24 Apr 2023 13:02:05 +0000 Chengwen Feng <fengchengwen@huawei.com> wrote: > This patchset introduces the coroutine library which will help refactor > the hns3 PMD's reset process. > > The hns3 single function reset process consists of the following steps: > 1.stop_service(); > 2.prepare_reset(); > 3.delay(100ms); > 4.notify_hw(); > 5.wait_hw_reset_done(); // multiple sleep waits are involved. > 6.reinit(); > 7.restore_conf(); > > If the DPDK process take over multiple hns3 functions (e.g. 100), > it's impractical to reset and restore functions in sequence: > 1.proc_func(001); // will completed in 100+ms range. > 2.proc_func(002); // will completed in 100~200+ms range. > ... > x.proc_func(100); // will completed in 9900~10000+ms range. > The later functions will process fail because it's too late to deal with. > > One solution is that create a reset thread for each function, and it > will lead to large number of threads if the DPDK process take over > multiple hns3 functions. > > So the current hns3 driver uses asynchronous mechanism, for examples, it > use rte_eal_alarm_set() when process delay(100ms), it splits a serial > process into multiple asynchronous processes, and the code is complex > and difficult to understand. > > The coroutine is a good mechanism to provide programmers with the > simplicity of keeping serial processes within a limited number of > threads. > > This patchset use <ucontext.h> to build the coroutine framework, and it > just provides a demo. More APIs maybe added in the future. > > In addition, we would like to ask the community whether it it possible > to accept the library. If not, whether it is allowed to provide the > library in hns3 PMD. > > Chengwen Feng (3): > lib/coroutine: add coroutine library > examples/coroutine: support coroutine examples > net/hns3: refactor reset process with coroutine Interesting, but the DPDK really is not the right place for this. Also, why so much sleeping. Can't this device be handled with an event based model. Plus any complexity like this introduces more bugs into already fragile interaction of DPDK userspace applications and threads. Not only that, coroutines add to the pre-existing problems with locking. If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself. And someone will spend days figuring that out. And the existing analyzer tools will not know about the magic coroutine library. Bottom line: please no
On 2023/4/25 0:08, Stephen Hemminger wrote: > On Mon, 24 Apr 2023 13:02:05 +0000 > Chengwen Feng <fengchengwen@huawei.com> wrote: > >> This patchset introduces the coroutine library which will help refactor >> the hns3 PMD's reset process. >> >> The hns3 single function reset process consists of the following steps: >> 1.stop_service(); >> 2.prepare_reset(); >> 3.delay(100ms); >> 4.notify_hw(); >> 5.wait_hw_reset_done(); // multiple sleep waits are involved. >> 6.reinit(); >> 7.restore_conf(); >> >> If the DPDK process take over multiple hns3 functions (e.g. 100), >> it's impractical to reset and restore functions in sequence: >> 1.proc_func(001); // will completed in 100+ms range. >> 2.proc_func(002); // will completed in 100~200+ms range. >> ... >> x.proc_func(100); // will completed in 9900~10000+ms range. >> The later functions will process fail because it's too late to deal with. >> >> One solution is that create a reset thread for each function, and it >> will lead to large number of threads if the DPDK process take over >> multiple hns3 functions. >> >> So the current hns3 driver uses asynchronous mechanism, for examples, it >> use rte_eal_alarm_set() when process delay(100ms), it splits a serial >> process into multiple asynchronous processes, and the code is complex >> and difficult to understand. >> >> The coroutine is a good mechanism to provide programmers with the >> simplicity of keeping serial processes within a limited number of >> threads. >> >> This patchset use <ucontext.h> to build the coroutine framework, and it >> just provides a demo. More APIs maybe added in the future. >> >> In addition, we would like to ask the community whether it it possible >> to accept the library. If not, whether it is allowed to provide the >> library in hns3 PMD. >> >> Chengwen Feng (3): >> lib/coroutine: add coroutine library >> examples/coroutine: support coroutine examples >> net/hns3: refactor reset process with coroutine > > Interesting, but the DPDK really is not the right place for this. > Also, why so much sleeping. Can't this device be handled with an event based > model. Plus any complexity like this introduces more bugs into already fragile > interaction of DPDK userspace applications and threads. A event base model will function as: event-handler() { for (...) { event = get_next_event(); proc_event(); } } The root cause is that the proc_event() take too many time, and it will lead to other function can't be processed timely. For which proc_event() may wait a lot of time, the coroutine could also used to optimize it. > > Not only that, coroutines add to the pre-existing problems with locking. > If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself. > And someone will spend days figuring that out. And the existing analyzer > tools will not know about the magic coroutine library. Analyzer tools like lock annotations maybe a problem. Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API. In addition, any technology has two sides, the greatest advantage of coroutine I think is removes a large number of callbacks in asychronous programming. And also high-level languages generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe evolved to support detect. And one more, if not acceptable as public library, whether it is allowed intergration of this library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible, but the code needs to be upstream, hope to listen to community's comments. > > Bottom line: please no > > . >
On Tue, 25 Apr 2023 10:11:43 +0800 fengchengwen <fengchengwen@huawei.com> wrote: > On 2023/4/25 0:08, Stephen Hemminger wrote: > > On Mon, 24 Apr 2023 13:02:05 +0000 > > Chengwen Feng <fengchengwen@huawei.com> wrote: > > > >> This patchset introduces the coroutine library which will help refactor > >> the hns3 PMD's reset process. > >> > >> The hns3 single function reset process consists of the following steps: > >> 1.stop_service(); > >> 2.prepare_reset(); > >> 3.delay(100ms); > >> 4.notify_hw(); > >> 5.wait_hw_reset_done(); // multiple sleep waits are involved. > >> 6.reinit(); > >> 7.restore_conf(); > >> > >> If the DPDK process take over multiple hns3 functions (e.g. 100), > >> it's impractical to reset and restore functions in sequence: > >> 1.proc_func(001); // will completed in 100+ms range. > >> 2.proc_func(002); // will completed in 100~200+ms range. > >> ... > >> x.proc_func(100); // will completed in 9900~10000+ms range. > >> The later functions will process fail because it's too late to deal with. > >> > >> One solution is that create a reset thread for each function, and it > >> will lead to large number of threads if the DPDK process take over > >> multiple hns3 functions. > >> > >> So the current hns3 driver uses asynchronous mechanism, for examples, it > >> use rte_eal_alarm_set() when process delay(100ms), it splits a serial > >> process into multiple asynchronous processes, and the code is complex > >> and difficult to understand. > >> > >> The coroutine is a good mechanism to provide programmers with the > >> simplicity of keeping serial processes within a limited number of > >> threads. > >> > >> This patchset use <ucontext.h> to build the coroutine framework, and it > >> just provides a demo. More APIs maybe added in the future. > >> > >> In addition, we would like to ask the community whether it it possible > >> to accept the library. If not, whether it is allowed to provide the > >> library in hns3 PMD. > >> > >> Chengwen Feng (3): > >> lib/coroutine: add coroutine library > >> examples/coroutine: support coroutine examples > >> net/hns3: refactor reset process with coroutine > > > > Interesting, but the DPDK really is not the right place for this. > > Also, why so much sleeping. Can't this device be handled with an event based > > model. Plus any complexity like this introduces more bugs into already fragile > > interaction of DPDK userspace applications and threads. > > A event base model will function as: > event-handler() { > for (...) { > event = get_next_event(); > proc_event(); > } > } > The root cause is that the proc_event() take too many time, and it will lead to other > function can't be processed timely. > > For which proc_event() may wait a lot of time, the coroutine could also used to optimize > it. > > > > > Not only that, coroutines add to the pre-existing problems with locking. > > If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself. > > And someone will spend days figuring that out. And the existing analyzer > > tools will not know about the magic coroutine library. > > Analyzer tools like lock annotations maybe a problem. > > Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such > as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API. > In addition, any technology has two sides, the greatest advantage of coroutine I think is > removes a large number of callbacks in asychronous programming. And also high-level languages > generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe > evolved to support detect. > > > And one more, if not acceptable as public library, whether it is allowed intergration of this > library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible, > but the code needs to be upstream, hope to listen to community's comments. The standard DPDK architecture is to have dedicated threads. Unless you convert the user application to some other model, there really is no other useful work that can be done while waiting for your driver. There was a previous DPDK library for lightweight threading, but it never got any usage and was abandoned and dropped. Why is this better?
On 2023/4/25 10:16, Stephen Hemminger wrote: > On Tue, 25 Apr 2023 10:11:43 +0800 > fengchengwen <fengchengwen@huawei.com> wrote: > >> On 2023/4/25 0:08, Stephen Hemminger wrote: >>> On Mon, 24 Apr 2023 13:02:05 +0000 >>> Chengwen Feng <fengchengwen@huawei.com> wrote: >>> >>>> This patchset introduces the coroutine library which will help refactor >>>> the hns3 PMD's reset process. >>>> >>>> The hns3 single function reset process consists of the following steps: >>>> 1.stop_service(); >>>> 2.prepare_reset(); >>>> 3.delay(100ms); >>>> 4.notify_hw(); >>>> 5.wait_hw_reset_done(); // multiple sleep waits are involved. >>>> 6.reinit(); >>>> 7.restore_conf(); >>>> >>>> If the DPDK process take over multiple hns3 functions (e.g. 100), >>>> it's impractical to reset and restore functions in sequence: >>>> 1.proc_func(001); // will completed in 100+ms range. >>>> 2.proc_func(002); // will completed in 100~200+ms range. >>>> ... >>>> x.proc_func(100); // will completed in 9900~10000+ms range. >>>> The later functions will process fail because it's too late to deal with. >>>> >>>> One solution is that create a reset thread for each function, and it >>>> will lead to large number of threads if the DPDK process take over >>>> multiple hns3 functions. >>>> >>>> So the current hns3 driver uses asynchronous mechanism, for examples, it >>>> use rte_eal_alarm_set() when process delay(100ms), it splits a serial >>>> process into multiple asynchronous processes, and the code is complex >>>> and difficult to understand. >>>> >>>> The coroutine is a good mechanism to provide programmers with the >>>> simplicity of keeping serial processes within a limited number of >>>> threads. >>>> >>>> This patchset use <ucontext.h> to build the coroutine framework, and it >>>> just provides a demo. More APIs maybe added in the future. >>>> >>>> In addition, we would like to ask the community whether it it possible >>>> to accept the library. If not, whether it is allowed to provide the >>>> library in hns3 PMD. >>>> >>>> Chengwen Feng (3): >>>> lib/coroutine: add coroutine library >>>> examples/coroutine: support coroutine examples >>>> net/hns3: refactor reset process with coroutine >>> >>> Interesting, but the DPDK really is not the right place for this. >>> Also, why so much sleeping. Can't this device be handled with an event based >>> model. Plus any complexity like this introduces more bugs into already fragile >>> interaction of DPDK userspace applications and threads. >> >> A event base model will function as: >> event-handler() { >> for (...) { >> event = get_next_event(); >> proc_event(); >> } >> } >> The root cause is that the proc_event() take too many time, and it will lead to other >> function can't be processed timely. >> >> For which proc_event() may wait a lot of time, the coroutine could also used to optimize >> it. >> >>> >>> Not only that, coroutines add to the pre-existing problems with locking. >>> If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself. >>> And someone will spend days figuring that out. And the existing analyzer >>> tools will not know about the magic coroutine library. >> >> Analyzer tools like lock annotations maybe a problem. >> >> Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such >> as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API. > >> In addition, any technology has two sides, the greatest advantage of coroutine I think is >> removes a large number of callbacks in asychronous programming. And also high-level languages >> generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe >> evolved to support detect. >> >> >> And one more, if not acceptable as public library, whether it is allowed intergration of this >> library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible, >> but the code needs to be upstream, hope to listen to community's comments. > > > The standard DPDK architecture is to have dedicated threads. > Unless you convert the user application to some other model, there really is no other Instead of adding a new running model, this coroutine library just adapts to the current DPDK framework. My visions: 1. DPDK launch a default thread run coroutine service, this thread just like interrupt thread, it could provide server for PMD drivers. 2. Application could launch coroutine scheduler in lcore (just like 2/3 commit) if it want to use this library. > useful work that can be done while waiting for your driver. > > There was a previous DPDK library for lightweight threading, but it never got any > usage and was abandoned and dropped. Why is this better? DPDK lightweight threading ? I didn't know before, but will take a closer look at. This patchset is caused by a problem in the our driver reset process. The reset process is complex and difficult to understand. We think the coroutine implementation is practical to solve this problem, so we try to send this RFC. > . >
First time poster here: I worry a bit about a coroutine approach as it may be challenging for some uses like ours. We have a purely event driven loop with a Reactor model written in D. The details are not specifically needed here, except to point out that an approach based on ucontext.h or something like that would very likely be utterly incompatible in our environment. While we don’t currently plan to integrate support for your hns3 device, I would have grave reservations about a general coroutine library making it’s way into DPDK drivers — it would almost certainly cause no end of grief for us at Weka. I’m doubtful that we’re the only DPDK users in this situation. • Garrett On Apr 24, 2023 at 7:50 PM -0700, fengchengwen <fengchengwen@huawei.com>, wrote: > On 2023/4/25 10:16, Stephen Hemminger wrote: > > On Tue, 25 Apr 2023 10:11:43 +0800 > > fengchengwen <fengchengwen@huawei.com> wrote: > > > > > On 2023/4/25 0:08, Stephen Hemminger wrote: > > > > On Mon, 24 Apr 2023 13:02:05 +0000 > > > > Chengwen Feng <fengchengwen@huawei.com> wrote: > > > > > > > > > This patchset introduces the coroutine library which will help refactor > > > > > the hns3 PMD's reset process. > > > > > > > > > > The hns3 single function reset process consists of the following steps: > > > > > 1.stop_service(); > > > > > 2.prepare_reset(); > > > > > 3.delay(100ms); > > > > > 4.notify_hw(); > > > > > 5.wait_hw_reset_done(); // multiple sleep waits are involved. > > > > > 6.reinit(); > > > > > 7.restore_conf(); > > > > > > > > > > If the DPDK process take over multiple hns3 functions (e.g. 100), > > > > > it's impractical to reset and restore functions in sequence: > > > > > 1.proc_func(001); // will completed in 100+ms range. > > > > > 2.proc_func(002); // will completed in 100~200+ms range. > > > > > ... > > > > > x.proc_func(100); // will completed in 9900~10000+ms range. > > > > > The later functions will process fail because it's too late to deal with. > > > > > > > > > > One solution is that create a reset thread for each function, and it > > > > > will lead to large number of threads if the DPDK process take over > > > > > multiple hns3 functions. > > > > > > > > > > So the current hns3 driver uses asynchronous mechanism, for examples, it > > > > > use rte_eal_alarm_set() when process delay(100ms), it splits a serial > > > > > process into multiple asynchronous processes, and the code is complex > > > > > and difficult to understand. > > > > > > > > > > The coroutine is a good mechanism to provide programmers with the > > > > > simplicity of keeping serial processes within a limited number of > > > > > threads. > > > > > > > > > > This patchset use <ucontext.h> to build the coroutine framework, and it > > > > > just provides a demo. More APIs maybe added in the future. > > > > > > > > > > In addition, we would like to ask the community whether it it possible > > > > > to accept the library. If not, whether it is allowed to provide the > > > > > library in hns3 PMD. > > > > > > > > > > Chengwen Feng (3): > > > > > lib/coroutine: add coroutine library > > > > > examples/coroutine: support coroutine examples > > > > > net/hns3: refactor reset process with coroutine > > > > > > > > Interesting, but the DPDK really is not the right place for this. > > > > Also, why so much sleeping. Can't this device be handled with an event based > > > > model. Plus any complexity like this introduces more bugs into already fragile > > > > interaction of DPDK userspace applications and threads. > > > > > > A event base model will function as: > > > event-handler() { > > > for (...) { > > > event = get_next_event(); > > > proc_event(); > > > } > > > } > > > The root cause is that the proc_event() take too many time, and it will lead to other > > > function can't be processed timely. > > > > > > For which proc_event() may wait a lot of time, the coroutine could also used to optimize > > > it. > > > > > > > > > > > Not only that, coroutines add to the pre-existing problems with locking. > > > > If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself. > > > > And someone will spend days figuring that out. And the existing analyzer > > > > tools will not know about the magic coroutine library. > > > > > > Analyzer tools like lock annotations maybe a problem. > > > > > > Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such > > > as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API. > > > > > In addition, any technology has two sides, the greatest advantage of coroutine I think is > > > removes a large number of callbacks in asychronous programming. And also high-level languages > > > generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe > > > evolved to support detect. > > > > > > > > > And one more, if not acceptable as public library, whether it is allowed intergration of this > > > library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible, > > > but the code needs to be upstream, hope to listen to community's comments. > > > > > > The standard DPDK architecture is to have dedicated threads. > > Unless you convert the user application to some other model, there really is no other > > Instead of adding a new running model, this coroutine library just adapts to the current > DPDK framework. > My visions: > 1. DPDK launch a default thread run coroutine service, this thread just like interrupt thread, > it could provide server for PMD drivers. > 2. Application could launch coroutine scheduler in lcore (just like 2/3 commit) if it want to > use this library. > > > useful work that can be done while waiting for your driver. > > > > There was a previous DPDK library for lightweight threading, but it never got any > > usage and was abandoned and dropped. Why is this better? > > DPDK lightweight threading ? I didn't know before, but will take a closer look at. > > This patchset is caused by a problem in the our driver reset process. The reset process is > complex and difficult to understand. We think the coroutine implementation is practical to > solve this problem, so we try to send this RFC. > > > . > >
On 2023-04-24 15:02, Chengwen Feng wrote: > This patchset introduces the coroutine library which will help refactor > the hns3 PMD's reset process. > > The hns3 single function reset process consists of the following steps: > 1.stop_service(); > 2.prepare_reset(); > 3.delay(100ms); > 4.notify_hw(); > 5.wait_hw_reset_done(); // multiple sleep waits are involved. > 6.reinit(); > 7.restore_conf(); > > If the DPDK process take over multiple hns3 functions (e.g. 100), > it's impractical to reset and restore functions in sequence: > 1.proc_func(001); // will completed in 100+ms range. > 2.proc_func(002); // will completed in 100~200+ms range. > ... > x.proc_func(100); // will completed in 9900~10000+ms range. > The later functions will process fail because it's too late to deal with. > > One solution is that create a reset thread for each function, and it > will lead to large number of threads if the DPDK process take over > multiple hns3 functions. > > So the current hns3 driver uses asynchronous mechanism, for examples, it > use rte_eal_alarm_set() when process delay(100ms), it splits a serial > process into multiple asynchronous processes, and the code is complex > and difficult to understand. > > The coroutine is a good mechanism to provide programmers with the > simplicity of keeping serial processes within a limited number of > threads. > Coroutines (or anything with a stack, really) are generally too slow as a vehicle for concurrency in data plane applications, I would argue. They might help you solve this particular slow path task, but that alone doesn't seem like anything close to a rationale why a new concurrency library should be accepted. DPDK does need a deferred work mechanism, but I don't think coroutines are the answer. Currently, RTE timer is the closest thing you have in DPDK. To solve your issue (and many other, including things in the fast path), I would rather look in that direction, maybe extending to something Linux' tasklets. > This patchset use <ucontext.h> to build the coroutine framework, and it > just provides a demo. More APIs maybe added in the future. > > In addition, we would like to ask the community whether it it possible > to accept the library. If not, whether it is allowed to provide the > library in hns3 PMD. > > Chengwen Feng (3): > lib/coroutine: add coroutine library > examples/coroutine: support coroutine examples > net/hns3: refactor reset process with coroutine > > drivers/net/hns3/hns3_ethdev.c | 217 ++++++++++++++++++++++++++++++ > drivers/net/hns3/hns3_ethdev.h | 3 + > drivers/net/hns3/hns3_intr.c | 38 ++++++ > drivers/net/hns3/meson.build | 2 +- > examples/coroutine/main.c | 153 +++++++++++++++++++++ > examples/coroutine/meson.build | 10 ++ > examples/meson.build | 1 + > lib/coroutine/meson.build | 8 ++ > lib/coroutine/rte_coroutine.c | 190 ++++++++++++++++++++++++++ > lib/coroutine/rte_coroutine.h | 110 +++++++++++++++ > lib/coroutine/rte_coroutine_imp.h | 46 +++++++ > lib/coroutine/version.map | 11 ++ > lib/meson.build | 1 + > 13 files changed, 789 insertions(+), 1 deletion(-) > create mode 100644 examples/coroutine/main.c > create mode 100644 examples/coroutine/meson.build > create mode 100644 lib/coroutine/meson.build > create mode 100644 lib/coroutine/rte_coroutine.c > create mode 100644 lib/coroutine/rte_coroutine.h > create mode 100644 lib/coroutine/rte_coroutine_imp.h > create mode 100644 lib/coroutine/version.map >
On Mon, 24 Apr 2023 19:59:27 -0700 Garrett D'Amore <garrett@damore.org> wrote: > > > There was a previous DPDK library for lightweight threading, but it never got any > > > usage and was abandoned and dropped. Why is this better? > > > > DPDK lightweight threading ? I didn't know before, but will take a closer look at. https://doc.dpdk.org/guides-21.11/sample_app_ug/performance_thread.html
On 4/25/2023 3:11 AM, fengchengwen wrote: > On 2023/4/25 0:08, Stephen Hemminger wrote: >> On Mon, 24 Apr 2023 13:02:05 +0000 >> Chengwen Feng <fengchengwen@huawei.com> wrote: >> >>> This patchset introduces the coroutine library which will help refactor >>> the hns3 PMD's reset process. >>> >>> The hns3 single function reset process consists of the following steps: >>> 1.stop_service(); >>> 2.prepare_reset(); >>> 3.delay(100ms); >>> 4.notify_hw(); >>> 5.wait_hw_reset_done(); // multiple sleep waits are involved. >>> 6.reinit(); >>> 7.restore_conf(); >>> >>> If the DPDK process take over multiple hns3 functions (e.g. 100), >>> it's impractical to reset and restore functions in sequence: >>> 1.proc_func(001); // will completed in 100+ms range. >>> 2.proc_func(002); // will completed in 100~200+ms range. >>> ... >>> x.proc_func(100); // will completed in 9900~10000+ms range. >>> The later functions will process fail because it's too late to deal with. >>> >>> One solution is that create a reset thread for each function, and it >>> will lead to large number of threads if the DPDK process take over >>> multiple hns3 functions. >>> >>> So the current hns3 driver uses asynchronous mechanism, for examples, it >>> use rte_eal_alarm_set() when process delay(100ms), it splits a serial >>> process into multiple asynchronous processes, and the code is complex >>> and difficult to understand. >>> >>> The coroutine is a good mechanism to provide programmers with the >>> simplicity of keeping serial processes within a limited number of >>> threads. >>> >>> This patchset use <ucontext.h> to build the coroutine framework, and it >>> just provides a demo. More APIs maybe added in the future. >>> >>> In addition, we would like to ask the community whether it it possible >>> to accept the library. If not, whether it is allowed to provide the >>> library in hns3 PMD. >>> >>> Chengwen Feng (3): >>> lib/coroutine: add coroutine library >>> examples/coroutine: support coroutine examples >>> net/hns3: refactor reset process with coroutine >> >> Interesting, but the DPDK really is not the right place for this. >> Also, why so much sleeping. Can't this device be handled with an event based >> model. Plus any complexity like this introduces more bugs into already fragile >> interaction of DPDK userspace applications and threads. > > A event base model will function as: > event-handler() { > for (...) { > event = get_next_event(); > proc_event(); > } > } > The root cause is that the proc_event() take too many time, and it will lead to other > function can't be processed timely. > > For which proc_event() may wait a lot of time, the coroutine could also used to optimize > it. > >> >> Not only that, coroutines add to the pre-existing problems with locking. >> If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself. >> And someone will spend days figuring that out. And the existing analyzer >> tools will not know about the magic coroutine library. > > Analyzer tools like lock annotations maybe a problem. > > Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such > as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API. > > > In addition, any technology has two sides, the greatest advantage of coroutine I think is > removes a large number of callbacks in asychronous programming. And also high-level languages > generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe > evolved to support detect. > > > And one more, if not acceptable as public library, whether it is allowed intergration of this > library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible, > but the code needs to be upstream, hope to listen to community's comments. > Hi Chengwen, For having library in hns3 PMD, my concern is ucontext.h portability. If this breaks the build in a specific platform, a user even doesn't have an intention to use hns3 driver will be impacted and will need to debug/fix this issue. If the dependency properly managed in the meson and for unsupported platforms coroutine can be disabled, I don't see any reason to not have this in the driver (except than the reasons already mentioned to not have coroutine as a public library applies to here). But this means you will need to maintain and support reset method without coroutine, or disable driver on the platform that ucontext.h not supported, are you OK with this? And if you continue continue to have coroutine in the driver can you please group into a subfolder etc in the driver and document it (again in a subsection of the driver) to help users that found it useful and try for themselves. I put a few comments to the implementation as well. Thanks, ferruh
On 2023/4/26 19:27, Ferruh Yigit wrote: > On 4/25/2023 3:11 AM, fengchengwen wrote: >> On 2023/4/25 0:08, Stephen Hemminger wrote: >>> On Mon, 24 Apr 2023 13:02:05 +0000 >>> Chengwen Feng <fengchengwen@huawei.com> wrote: >>> >>>> This patchset introduces the coroutine library which will help refactor >>>> the hns3 PMD's reset process. >>>> >>>> The hns3 single function reset process consists of the following steps: >>>> 1.stop_service(); >>>> 2.prepare_reset(); >>>> 3.delay(100ms); >>>> 4.notify_hw(); >>>> 5.wait_hw_reset_done(); // multiple sleep waits are involved. >>>> 6.reinit(); >>>> 7.restore_conf(); >>>> >>>> If the DPDK process take over multiple hns3 functions (e.g. 100), >>>> it's impractical to reset and restore functions in sequence: >>>> 1.proc_func(001); // will completed in 100+ms range. >>>> 2.proc_func(002); // will completed in 100~200+ms range. >>>> ... >>>> x.proc_func(100); // will completed in 9900~10000+ms range. >>>> The later functions will process fail because it's too late to deal with. >>>> >>>> One solution is that create a reset thread for each function, and it >>>> will lead to large number of threads if the DPDK process take over >>>> multiple hns3 functions. >>>> >>>> So the current hns3 driver uses asynchronous mechanism, for examples, it >>>> use rte_eal_alarm_set() when process delay(100ms), it splits a serial >>>> process into multiple asynchronous processes, and the code is complex >>>> and difficult to understand. >>>> >>>> The coroutine is a good mechanism to provide programmers with the >>>> simplicity of keeping serial processes within a limited number of >>>> threads. >>>> >>>> This patchset use <ucontext.h> to build the coroutine framework, and it >>>> just provides a demo. More APIs maybe added in the future. >>>> >>>> In addition, we would like to ask the community whether it it possible >>>> to accept the library. If not, whether it is allowed to provide the >>>> library in hns3 PMD. >>>> >>>> Chengwen Feng (3): >>>> lib/coroutine: add coroutine library >>>> examples/coroutine: support coroutine examples >>>> net/hns3: refactor reset process with coroutine >>> >>> Interesting, but the DPDK really is not the right place for this. >>> Also, why so much sleeping. Can't this device be handled with an event based >>> model. Plus any complexity like this introduces more bugs into already fragile >>> interaction of DPDK userspace applications and threads. >> >> A event base model will function as: >> event-handler() { >> for (...) { >> event = get_next_event(); >> proc_event(); >> } >> } >> The root cause is that the proc_event() take too many time, and it will lead to other >> function can't be processed timely. >> >> For which proc_event() may wait a lot of time, the coroutine could also used to optimize >> it. >> >>> >>> Not only that, coroutines add to the pre-existing problems with locking. >>> If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself. >>> And someone will spend days figuring that out. And the existing analyzer >>> tools will not know about the magic coroutine library. >> >> Analyzer tools like lock annotations maybe a problem. >> >> Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such >> as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API. >> >> >> In addition, any technology has two sides, the greatest advantage of coroutine I think is >> removes a large number of callbacks in asychronous programming. And also high-level languages >> generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe >> evolved to support detect. >> >> >> And one more, if not acceptable as public library, whether it is allowed intergration of this >> library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible, >> but the code needs to be upstream, hope to listen to community's comments. >> > > Hi Chengwen, > > For having library in hns3 PMD, my concern is ucontext.h portability. Yes, after in-depth evaluation, it was found that this was indeed a problem. And ucontext_t were deprecated in Posix v6 and removed in v7. See: https://stackoverflow.com/questions/33331894/why-does-ucontext-have-such-high-overhead Although it is possible to implement an assembly version (which replace ucontext_t), but maintaining assembly code in DPDK is difficult. > > If this breaks the build in a specific platform, a user even doesn't > have an intention to use hns3 driver will be impacted and will need to > debug/fix this issue. > > If the dependency properly managed in the meson and for unsupported > platforms coroutine can be disabled, I don't see any reason to not have > this in the driver (except than the reasons already mentioned to not > have coroutine as a public library applies to here). > > But this means you will need to maintain and support reset method > without coroutine, or disable driver on the platform that ucontext.h not > supported, are you OK with this? Yes, if two set are maintained at the same time, the complexity is increased. Combine the opinions of all reviewer, the introduction of coroutine will cause a lot of problems, and I will *stop* this patchset. @Ferruh @Setphen @Garrett @Mattias Thanks for your review and feedback. > > > And if you continue continue to have coroutine in the driver can you > please group into a subfolder etc in the driver and document it (again > in a subsection of the driver) to help users that found it useful and > try for themselves. > > > I put a few comments to the implementation as well. @Ferruh, thanks for the review. > > > Thanks, > ferruh > > . >