Message ID | cover.1528404133.git.anatoly.burakov@intel.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D5B93D4A0; Thu, 7 Jun 2018 23:02:08 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 317A8D4A8 for <dev@dpdk.org>; Thu, 7 Jun 2018 23:02:06 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 Jun 2018 14:02:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,488,1520924400"; d="scan'208";a="47294703" Received: from irvmail001.ir.intel.com ([163.33.26.43]) by orsmga007.jf.intel.com with ESMTP; 07 Jun 2018 14:02:02 -0700 Received: from sivswdev01.ir.intel.com (sivswdev01.ir.intel.com [10.237.217.45]) by irvmail001.ir.intel.com (8.14.3/8.13.6/MailSET/Hub) with ESMTP id w57L21rH023015; Thu, 7 Jun 2018 22:02:01 +0100 Received: from sivswdev01.ir.intel.com (localhost [127.0.0.1]) by sivswdev01.ir.intel.com with ESMTP id w57L21Xe004346; Thu, 7 Jun 2018 22:02:01 +0100 Received: (from aburakov@localhost) by sivswdev01.ir.intel.com with LOCAL id w57L213j004342; Thu, 7 Jun 2018 22:02:01 +0100 From: Anatoly Burakov <anatoly.burakov@intel.com> To: dev@dpdk.org Cc: john.mcnamara@intel.com, reshma.pattan@intel.com, bruce.richardson@intel.com Date: Thu, 7 Jun 2018 22:01:54 +0100 Message-Id: <cover.1528404133.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 1.7.0.7 Subject: [dpdk-dev] [PATCH 0/7] Make unit tests great again X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
Make unit tests great again
|
|
Message
Burakov, Anatoly
June 7, 2018, 9:01 p.m. UTC
Previously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script. In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that. This patchset conflicts with some of the earlier work on autotests [1] [2] [3], but i think it presents a cleaner solution for some of the problems highlighted by those patch series. I can integrate those patches into this series if need be. [1] http://dpdk.org/dev/patchwork/patch/40370/ [2] http://dpdk.org/dev/patchwork/patch/40371/ [3] http://dpdk.org/dev/patchwork/patch/40372/ Anatoly Burakov (7): autotest: fix printing autotest: fix invalid code on reports autotest: make autotest runner python 2/3 compliant autotest: visually separate different test categories autotest: improve filtering autotest: remove autotest grouping autotest: properly parallelize unit tests test/test/autotest.py | 13 +- test/test/autotest_data.py | 749 ++++++++++++++--------------------- test/test/autotest_runner.py | 519 ++++++++++++------------ 3 files changed, 586 insertions(+), 695 deletions(-)
Comments
+Cc Jananee 07/06/2018 23:01, Anatoly Burakov: > Previously, unit tests were running in groups. There were > technical reasons why that was the case (mostly having to do > with limiting memory), but it was hard to maintain and update > the autotest script. > > In 18.05, limiting of memory at DPDK startup was no longer > necessary, as DPDK allocates memory at runtime as needed. This > has the implication that the old test grouping can now be > retired and replaced with a more sensible way of running unit > tests (using multiprocessing pool of workers and a queue of > tests). This patchset accomplishes exactly that. > > This patchset conflicts with some of the earlier work on > autotests [1] [2] [3], but i think it presents a cleaner > solution for some of the problems highlighted by those patch > series. I can integrate those patches into this series if > need be. > > [1] http://dpdk.org/dev/patchwork/patch/40370/ > [2] http://dpdk.org/dev/patchwork/patch/40371/ > [3] http://dpdk.org/dev/patchwork/patch/40372/ It may be interesting to work on lists of tests as done in the following patch: http://dpdk.org/dev/patchwork/patch/40373/ The idea is to split tests in several categories: - basic and short test - longer and lower priority - performance test As a long term solution, we can think about making category an attribute inside the test itself?
On 12-Jun-18 2:07 PM, Thomas Monjalon wrote: > +Cc Jananee > > 07/06/2018 23:01, Anatoly Burakov: >> Previously, unit tests were running in groups. There were >> technical reasons why that was the case (mostly having to do >> with limiting memory), but it was hard to maintain and update >> the autotest script. >> >> In 18.05, limiting of memory at DPDK startup was no longer >> necessary, as DPDK allocates memory at runtime as needed. This >> has the implication that the old test grouping can now be >> retired and replaced with a more sensible way of running unit >> tests (using multiprocessing pool of workers and a queue of >> tests). This patchset accomplishes exactly that. >> >> This patchset conflicts with some of the earlier work on >> autotests [1] [2] [3], but i think it presents a cleaner >> solution for some of the problems highlighted by those patch >> series. I can integrate those patches into this series if >> need be. >> >> [1] http://dpdk.org/dev/patchwork/patch/40370/ >> [2] http://dpdk.org/dev/patchwork/patch/40371/ >> [3] http://dpdk.org/dev/patchwork/patch/40372/ > > It may be interesting to work on lists of tests as done > in the following patch: > http://dpdk.org/dev/patchwork/patch/40373/ > > The idea is to split tests in several categories: > - basic and short test > - longer and lower priority > - performance test > As a long term solution, we can think about making category an attribute > inside the test itself? > These test categories do not conflict with my patchset as they ultimately rely on white/blacklisting, which will continue to work as before. In my view, it really boils down to two things - either tests can be run in parallel with others (i.e. their result won't be affected by another independent DPDK test app instance), or not. On top of that, we can use blacklisting or whitelisting to define which tests will actually be run (i.e. define any "categories" we want), but their (non-)parallelism must always be respected to get good test results.
On Wed, Jun 13, 2018 at 09:38:32AM +0100, Burakov, Anatoly wrote: > On 12-Jun-18 2:07 PM, Thomas Monjalon wrote: > > +Cc Jananee > > > > 07/06/2018 23:01, Anatoly Burakov: > > > Previously, unit tests were running in groups. There were > > > technical reasons why that was the case (mostly having to do > > > with limiting memory), but it was hard to maintain and update > > > the autotest script. > > > > > > In 18.05, limiting of memory at DPDK startup was no longer > > > necessary, as DPDK allocates memory at runtime as needed. This > > > has the implication that the old test grouping can now be > > > retired and replaced with a more sensible way of running unit > > > tests (using multiprocessing pool of workers and a queue of > > > tests). This patchset accomplishes exactly that. > > > > > > This patchset conflicts with some of the earlier work on > > > autotests [1] [2] [3], but i think it presents a cleaner > > > solution for some of the problems highlighted by those patch > > > series. I can integrate those patches into this series if > > > need be. > > > > > > [1] http://dpdk.org/dev/patchwork/patch/40370/ > > > [2] http://dpdk.org/dev/patchwork/patch/40371/ > > > [3] http://dpdk.org/dev/patchwork/patch/40372/ > > > > It may be interesting to work on lists of tests as done > > in the following patch: > > http://dpdk.org/dev/patchwork/patch/40373/ > > > > The idea is to split tests in several categories: > > - basic and short test > > - longer and lower priority > > - performance test > > As a long term solution, we can think about making category an attribute > > inside the test itself? > > > > These test categories do not conflict with my patchset as they ultimately > rely on white/blacklisting, which will continue to work as before. > > In my view, it really boils down to two things - either tests can be run in > parallel with others (i.e. their result won't be affected by another > independent DPDK test app instance), or not. On top of that, we can use > blacklisting or whitelisting to define which tests will actually be run > (i.e. define any "categories" we want), but their (non-)parallelism must > always be respected to get good test results. > Have you looked at: http://mesonbuild.com/Unit-tests.html, at it would be good to transition away from our own custom script infrastructure for the tests. There is already some support for running the unit tests using "meson test", but it could do with some more cleanup e.g. to move the tests into suitable suites (corresponding to the categories Thomas has suggested). We could also do with specifying properly what tests are parallel-safe and what aren't.