test/compress: replace test vector

Message ID 20200114115656.14611-1-arturx.trybula@intel.com (mailing list archive)
State Rejected, archived
Delegated to: akhil goyal
Headers
Series test/compress: replace test vector |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation fail Compilation issues
ci/iol-testing success Testing PASS
ci/travis-robot success Travis build: passed
ci/iol-nxp-Performance success Performance Testing PASS
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-mellanox-Performance success Performance Testing PASS

Commit Message

Artur Trybula Jan. 14, 2020, 11:56 a.m. UTC
  This patch replaces existing test vector with a new
one containing C code to fix license issue.

Signed-off-by: Artur Trybula <arturx.trybula@intel.com>
---
 app/test/test_compressdev_test_buffer.h | 228 ++++++++++++++----------
 1 file changed, 132 insertions(+), 96 deletions(-)
  

Comments

Fiona Trahe Jan. 14, 2020, 12:33 p.m. UTC | #1
> -----Original Message-----
> From: Trybula, ArturX <arturx.trybula@intel.com>
> Sent: Tuesday, January 14, 2020 11:57 AM
> To: dev@dpdk.org; stable@dpdk.org; Trahe, Fiona <fiona.trahe@intel.com>; shallyv@marvell.com;
> Dybkowski, AdamX <adamx.dybkowski@intel.com>; Danilewicz, MarcinX
> <marcinx.danilewicz@intel.com>; Trybula, ArturX <arturx.trybula@intel.com>; akhil.goyal@nxp.com
> Subject: [PATCH] test/compress: replace test vector
> 
> This patch replaces existing test vector with a new
> one containing C code to fix license issue.
> 
> Signed-off-by: Artur Trybula <arturx.trybula@intel.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>
  
Akhil Goyal Jan. 15, 2020, 3:54 p.m. UTC | #2
> >
> > This patch replaces existing test vector with a new
> > one containing C code to fix license issue.
> >
> > Signed-off-by: Artur Trybula <arturx.trybula@intel.com>
> Acked-by: Fiona Trahe <fiona.trahe@intel.com>
Applied to dpdk-next-crypto

Thanks.
  
Thomas Monjalon Jan. 17, 2020, 9:51 p.m. UTC | #3
15/01/2020 16:54, Akhil Goyal:
> > >
> > > This patch replaces existing test vector with a new
> > > one containing C code to fix license issue.
> > >
> > > Signed-off-by: Artur Trybula <arturx.trybula@intel.com>
> > Acked-by: Fiona Trahe <fiona.trahe@intel.com>
> Applied to dpdk-next-crypto

This is a terrible idea.
Adding C code in a C project as text, and worst, its own code,
will make grep matching on fake text.

Why do you need so much text to compress?
Why not just opening our own code files as text?

Why stable@dpdk.org is Cc'ed in this email?
What is the license issue?
Should it be backported? If yes, the stable tag is missing.

I drop this patch from -rc1.
  
John McNamara Jan. 20, 2020, 1:54 p.m. UTC | #4
> -----Original Message-----
> From: stable <stable-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> Sent: Friday, January 17, 2020 9:52 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Trybula, ArturX
> <arturx.trybula@intel.com>
> Cc: dev@dpdk.org; stable@dpdk.org; shallyv@marvell.com; Dybkowski, AdamX
> <adamx.dybkowski@intel.com>; Danilewicz, MarcinX
> <marcinx.danilewicz@intel.com>; Akhil Goyal <akhil.goyal@nxp.com>
> Subject: Re: [dpdk-stable] [PATCH] test/compress: replace test vector
> 
> 15/01/2020 16:54, Akhil Goyal:
> > > >
> > > > This patch replaces existing test vector with a new one containing
> > > > C code to fix license issue.
> > > >
> > > > Signed-off-by: Artur Trybula <arturx.trybula@intel.com>
> > > Acked-by: Fiona Trahe <fiona.trahe@intel.com>
> > Applied to dpdk-next-crypto
> 
> This is a terrible idea.
> Adding C code in a C project as text, and worst, its own code, will make
> grep matching on fake text.

Hi,

I don't think this is a valid criticism. The patch replace code in Pascal with some code in C as a compression test. The grep part isn't really relevant, it should be clear to anyone who finds a match.

If you have a strong objection we will replace it with some more public domain text.

> Why do you need so much text to compress?
> Why not just opening our own code files as text?

That probably isn't worth the effort.

> Why stable@dpdk.org is Cc'ed in this email?

Ok. We'll do that.


> What is the license issue?

The issue is that the Pascal code from the Calgary Corpus (a standard body of compression test input) doesn't have a license or explicitly say that it is in the public domain so it isn't clear if we can relicense  it as BSD-3. We are being overly cautious here but we didn't want to run into any potential license issues. 
 
John
  
Thomas Monjalon Jan. 20, 2020, 2:33 p.m. UTC | #5
20/01/2020 14:54, Mcnamara, John:
> From: stable <stable-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> > 15/01/2020 16:54, Akhil Goyal:
> > > > >
> > > > > This patch replaces existing test vector with a new one containing
> > > > > C code to fix license issue.
> > > > >
> > > > > Signed-off-by: Artur Trybula <arturx.trybula@intel.com>
> > > > Acked-by: Fiona Trahe <fiona.trahe@intel.com>
> > > Applied to dpdk-next-crypto
> > 
> > This is a terrible idea.
> > Adding C code in a C project as text, and worst, its own code, will make
> > grep matching on fake text.
> 
> Hi,
> 
> I don't think this is a valid criticism. The patch replace code in Pascal with some code in C as a compression test. The grep part isn't really relevant, it should be clear to anyone who finds a match.
> 
> If you have a strong objection we will replace it with some more public domain text.

Yes please, it will be less confusing.

> > Why do you need so much text to compress?

Please why do we need some code at all?

> > Why not just opening our own code files as text?
> 
> That probably isn't worth the effort.
> 
> > Why stable@dpdk.org is Cc'ed in this email?
> 
> Ok. We'll do that.

I say Why? You say Ok :-)

> > What is the license issue?
> 
> The issue is that the Pascal code from the Calgary Corpus (a standard body of compression test input) doesn't have a license or explicitly say that it is in the public domain so it isn't clear if we can relicense  it as BSD-3. We are being overly cautious here but we didn't want to run into any potential license issues.

OK
Please explain in the commit log.
  
Fiona Trahe Jan. 20, 2020, 3 p.m. UTC | #6
Hi Thomas

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, January 20, 2020 2:33 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Trybula, ArturX <arturx.trybula@intel.com>; Mcnamara,
> John <john.mcnamara@intel.com>
> Cc: dev@dpdk.org; stable@dpdk.org; shallyv@marvell.com; Dybkowski, AdamX
> <adamx.dybkowski@intel.com>; Danilewicz, MarcinX <marcinx.danilewicz@intel.com>; Akhil Goyal
> <akhil.goyal@nxp.com>
> Subject: Re: [dpdk-stable] [PATCH] test/compress: replace test vector
> 
> 20/01/2020 14:54, Mcnamara, John:
> > From: stable <stable-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> > > 15/01/2020 16:54, Akhil Goyal:
> > > > > >
> > > > > > This patch replaces existing test vector with a new one containing
> > > > > > C code to fix license issue.
> > > > > >
> > > > > > Signed-off-by: Artur Trybula <arturx.trybula@intel.com>
> > > > > Acked-by: Fiona Trahe <fiona.trahe@intel.com>
> > > > Applied to dpdk-next-crypto
> > >
> > > This is a terrible idea.
> > > Adding C code in a C project as text, and worst, its own code, will make
> > > grep matching on fake text.
> >
> > Hi,
> >
> > I don't think this is a valid criticism. The patch replace code in Pascal with some code in C as a
> compression test. The grep part isn't really relevant, it should be clear to anyone who finds a match.
> >
> > If you have a strong objection we will replace it with some more public domain text.
> 
> Yes please, it will be less confusing.
> 
> > > Why do you need so much text to compress?
> 
> Please why do we need some code at all?
[Fiona] compression engines are usually tested with different types of data to represent various use-cases.
So include for example ASCII text, binary chunks, excel files, etc. Compression of code is a typical compression workload - and
as code has lots of patterns may be quite compressible, at different ratios to standard English language text.
That said, it's not so important in DPDK to cover all data types as our focus is not on verifying the compression
engines, but just the path through the API and PMD to them. So we can replace this text if it's less confusing.

> 
> > > Why not just opening our own code files as text?
> >
> > That probably isn't worth the effort.
> >
> > > Why stable@dpdk.org is Cc'ed in this email?
> >
> > Ok. We'll do that.
> 
> I say Why? You say Ok :-)
[Fiona] Simple mistake - sorry 😊
> 
> > > What is the license issue?
> >
> > The issue is that the Pascal code from the Calgary Corpus (a standard body of compression test
> input) doesn't have a license or explicitly say that it is in the public domain so it isn't clear if we can
> relicense  it as BSD-3. We are being overly cautious here but we didn't want to run into any potential
> license issues.
> 
> OK
> Please explain in the commit log.
> 
[Fiona] ok
  
John McNamara Jan. 20, 2020, 6 p.m. UTC | #7
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, January 20, 2020 2:33 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>; Trybula, ArturX
> <arturx.trybula@intel.com>; Mcnamara, John <john.mcnamara@intel.com>
> Cc: dev@dpdk.org; stable@dpdk.org; shallyv@marvell.com; Dybkowski, AdamX
> <adamx.dybkowski@intel.com>; Danilewicz, MarcinX
> <marcinx.danilewicz@intel.com>; Akhil Goyal <akhil.goyal@nxp.com>
> Subject: Re: [dpdk-stable] [PATCH] test/compress: replace test vector
> 
> 20/01/2020 14:54, Mcnamara, John:
> > From: stable <stable-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> > > 15/01/2020 16:54, Akhil Goyal:
> > > > > >
> > > > > > This patch replaces existing test vector with a new one
> > > > > > containing C code to fix license issue.
> > > > > >
> > > > > > Signed-off-by: Artur Trybula <arturx.trybula@intel.com>
> > > > > Acked-by: Fiona Trahe <fiona.trahe@intel.com>
> > > > Applied to dpdk-next-crypto
> > >
> > > This is a terrible idea.
> > > Adding C code in a C project as text, and worst, its own code, will
> > > make grep matching on fake text.
> >
> > Hi,
> >
> > I don't think this is a valid criticism. The patch replace code in
> Pascal with some code in C as a compression test. The grep part isn't
> really relevant, it should be clear to anyone who finds a match.
> >
> > If you have a strong objection we will replace it with some more public
> domain text.
> 
> Yes please, it will be less confusing.
> 
> > > Why do you need so much text to compress?
> 
> Please why do we need some code at all?
> 
> > > Why not just opening our own code files as text?
> >
> > That probably isn't worth the effort.
> >
> > > Why stable@dpdk.org is Cc'ed in this email?
> >
> > Ok. We'll do that.
> 
> I say Why? You say Ok :-)

I mis-read it. I thought you said it wasn't CCed.

John
  
John McNamara Jan. 27, 2020, 10:59 a.m. UTC | #8
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Mcnamara, John
> Sent: Monday, January 20, 2020 6:00 PM
> To: Thomas Monjalon <thomas@monjalon.net>; Trahe, Fiona
> <fiona.trahe@intel.com>; Trybula, ArturX <arturx.trybula@intel.com>
> Cc: dev@dpdk.org; stable@dpdk.org; shallyv@marvell.com; Dybkowski, AdamX
> <adamx.dybkowski@intel.com>; Danilewicz, MarcinX
> <marcinx.danilewicz@intel.com>; Akhil Goyal <akhil.goyal@nxp.com>
> Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH] test/compress: replace test
> vector


This patch also needs a BSD SPDX header. Please add that.

John
  

Patch

diff --git a/app/test/test_compressdev_test_buffer.h b/app/test/test_compressdev_test_buffer.h
index c0492f89a..4e8bd9eb2 100644
--- a/app/test/test_compressdev_test_buffer.h
+++ b/app/test/test_compressdev_test_buffer.h
@@ -190,106 +190,142 @@  static const char test_buf_shakespeare[] =
 	"\n"
 	"ORLANDO	Go apart, Adam, and thou shalt hear how he will\n";
 
-/* Snippet of source code in Pascal */
-static const char test_buf_pascal[] =
-	"	Ptr    = 1..DMem;\n"
-	"	Loc    = 1..IMem;\n"
-	"	Loc0   = 0..IMem;\n"
-	"	EdgeT  = (hout,lin,hin,lout); {Warning this order is important in}\n"
-	"				      {predicates such as gtS,geS}\n"
-	"	CardT  = (finite,infinite);\n"
-	"	ExpT   = Minexp..Maxexp;\n"
-	"	ManT   = Mininf..Maxinf; \n"
-	"	Pflag  = (PNull,PSoln,PTrace,PPrint);\n"
-	"	Sreal  = record\n"
-	"		    edge:EdgeT;\n"
-	"		    cardinality:CardT;\n"
-	"		    exp:ExpT; {exponent}\n"
-	"		    mantissa:ManT;\n"
-	"		 end;\n"
-	"	Int    = record\n"
-	"		    hi:Sreal;\n"
-	"		    lo:Sreal;\n"
-	"	 end;\n"
-	"	Instr  = record\n"
-	"		    Code:OpType;\n"
-	"		    Pars: array[0..Par] of 0..DMem;\n"
-	"		 end;\n"
-	"	DataMem= record\n"
-	"		    D        :array [Ptr] of Int;\n"
-	"		    S        :array [Loc] of State;\n"
-	"		    LastHalve:Loc;\n"
-	"		    RHalve   :array [Loc] of real;\n"
-	"		 end;\n"
-	"	DataFlags=record\n"
-	"		    PF	     :array [Ptr] of Pflag;\n"
-	"		 end;\n"
-	"var\n"
-	"	Debug  : (none,activity,post,trace,dump);\n"
-	"	Cut    : (once,all);\n"
-	"	GlobalEnd,Verifiable:boolean;\n"
-	"	HalveThreshold:real;\n"
-	"	I      : array [Loc] of Instr; {Memory holding instructions}\n"
-	"	End    : Loc; {last instruction in I}\n"
-	"	ParN   : array [OpType] of -1..Par; {number of parameters for each \n"
-	"			opcode. -1 means no result}\n"
-	"        ParIntersect : array [OpType] of boolean ;\n"
-	"	DInit  : DataMem; {initial memory which is cleared and \n"
-	"				used in first call}\n"
-	"	DF     : DataFlags; {hold flags for variables, e.g. print/trace}\n"
-	"	MaxDMem:0..DMem;\n"
-	"	Shift  : array[0..Digits] of 1..maxint;{array of constant multipliers}\n"
-	"						{used for alignment etc.}\n"
-	"	Dummy  :Positive;\n"
-	"	{constant intervals and Sreals}\n"
-	"	PlusInfS,MinusInfS,PlusSmallS,MinusSmallS,ZeroS,\n"
-	"	PlusFiniteS,MinusFiniteS:Sreal;\n"
-	"	Zero,All,AllFinite:Int;\n"
-	"\n"
-	"procedure deblank;\n"
-	"var Ch:char;\n"
-	"begin\n"
-	"   while (not eof) and (input^ in [' ','	']) do read(Ch);\n"
-	"end;\n"
-	"\n"
-	"procedure InitialOptions;\n"
-	"\n"
-	"#include '/user/profs/cleary/bin/options.i';\n"
-	"\n"
-	"   procedure Option;\n"
-	"   begin\n"
-	"      case Opt of\n"
-	"      'a','A':Debug:=activity;\n"
-	"      'd','D':Debug:=dump;\n"
-	"      'h','H':HalveThreshold:=StringNum/100;\n"
-	"      'n','N':Debug:=none;\n"
-	"      'p','P':Debug:=post;\n"
-	"      't','T':Debug:=trace;\n"
-	"      'v','V':Verifiable:=true;\n"
-	"      end;\n"
-	"   end;\n"
-	"\n"
-	"begin\n"
-	"   Debug:=trace;\n"
-	"   Verifiable:=false;\n"
-	"   HalveThreshold:=67/100;\n"
-	"   Options;\n"
-	"   writeln(Debug);\n"
-	"   writeln('Verifiable:',Verifiable);\n"
-	"   writeln('Halve threshold',HalveThreshold);\n"
-	"end;{InitialOptions}\n"
-	"\n"
-	"procedure NormalizeUp(E,M:integer;var S:Sreal;var Closed:boolean);\n"
-	"begin\n"
-	"with S do\n"
-	"begin\n"
-	"   if M=0 then S:=ZeroS else\n"
-	"   if M>0 then\n";
+/* Snippet of source code in C */
+static const char test_buf_c_code[] =
+	"/* SPDX-License-Identifier: BSD-3-Clause\n"
+	" * Copyright(c) 2010-2014 Intel Corporation\n"
+	" */\n"
+	"\n"
+	"#include <stdio.h>\n"
+	"#include <inttypes.h>\n"
+	"#include <string.h>\n"
+	"#include <math.h>\n"
+	"#include <rte_common.h>\n"
+	"#include <rte_hexdump.h>\n"
+	"#include <rte_pause.h>\n"
+	"\n"
+	"#include test.h\n"
+	"\n"
+	"#define MAX_NUM 1 << 20\n"
+	"\n"
+	"#define FAIL(x)\\n"
+	"	{printf(x '() test failed!\n');\\n"
+	"	return -1;}\n"
+	"\n"
+	"/* this is really a sanity check */\n"
+	"static int\n"
+	"test_macros(int __rte_unused unused_parm)\n"
+	"{\n"
+	"#define SMALLER 0x1000U\n"
+	"#define BIGGER 0x2000U\n"
+	"#define PTR_DIFF BIGGER - SMALLER\n"
+	"#define FAIL_MACRO(x)\\n"
+	"	{printf(#x '() test failed!\n');\\n"
+	"	return -1;}\n"
+	"\n"
+	"	uintptr_t unused = 0;\n"
+	"\n"
+	"	RTE_SET_USED(unused);\n"
+	"\n"
+	"if ((uintptr_t)RTE_PTR_ADD(SMALLER, PTR_DIFF) != BIGGER)\n"
+	"		FAIL_MACRO(RTE_PTR_ADD);\n"
+	"	if ((uintptr_t)RTE_PTR_SUB(BIGGER, PTR_DIFF) != SMALLER)\n"
+	"		FAIL_MACRO(RTE_PTR_SUB);\n"
+	"	if (RTE_PTR_DIFF(BIGGER, SMALLER) != PTR_DIFF)\n"
+	"		FAIL_MACRO(RTE_PTR_DIFF);\n"
+	"	if (RTE_MAX(SMALLER, BIGGER) != BIGGER)\n"
+	"		FAIL_MACRO(RTE_MAX);\n"
+	"	if (RTE_MIN(SMALLER, BIGGER) != SMALLER)\n"
+	"		FAIL_MACRO(RTE_MIN);\n"
+	"\n"
+	"	if (strncmp(RTE_STR(test), 'test', sizeof('test')))\n"
+	"		FAIL_MACRO(RTE_STR);\n"
+	"\n"
+	"	return 0;\n"
+	"}\n"
+	"\n"
+	"static int\n"
+	"test_bsf(void)\n"
+	"{\n"
+	"	uint32_t shift, pos;\n"
+	"\n"
+	"	/* safe versions should be able to handle 0 */\n"
+	"	if (rte_bsf32_safe(0, &pos) != 0)\n"
+	"		FAIL('rte_bsf32_safe');\n"
+	"	if (rte_bsf64_safe(0, &pos) != 0)\n"
+	"		FAIL('rte_bsf64_safe');\n"
+	"\n"
+	"	for (shift = 0; shift < 63; shift++) {\n"
+	"		uint32_t val32;\n"
+	"		uint64_t val64;\n"
+	"\n"
+	"		val64 = 1ULL << shift;\n"
+	"		if ((uint32_t)rte_bsf64(val64) != shift)\n"
+	"			FAIL('rte_bsf64');\n"
+	"		if (rte_bsf64_safe(val64, &pos) != 1)\n"
+	"			FAIL('rte_bsf64_safe');\n"
+	"		if (pos != shift)\n"
+	"			FAIL('rte_bsf64_safe');\n"
+	"\n"
+	"		if (shift > 31)\n"
+	"			continue;\n"
+	"\n"
+	"		val32 = 1U << shift;\n"
+	"		if ((uint32_t)rte_bsf32(val32) != shift)\n"
+	"			FAIL('rte_bsf32');\n"
+	"		if (rte_bsf32_safe(val32, &pos) != 1)\n"
+	"			FAIL('rte_bsf32_safe');\n"
+	"		if (pos != shift)\n"
+	"			FAIL('rte_bsf32_safe');\n"
+	"	}\n"
+	"\n"
+	"	return 0;\n"
+	"}\n"
+	"\n"
+	"static int\n"
+	"test_log2(void)\n"
+	"{\n"
+	"	uint32_t i, base, compare;\n"
+	"	const uint32_t max = 0x10000;\n"
+	"	const uint32_t step = 1;\n"
+	"\n"
+	"	for (i = 0; i < max; i = i + step) {\n"
+	"		uint64_t i64;\n"
+	"\n"
+	"		/* extend range for 64-bit */\n"
+	"		i64 = (uint64_t)i << 32;\n"
+	"		base = (uint32_t)ceilf(log2(i64));\n"
+	"		compare = rte_log2_u64(i64);\n"
+	"		if (base != compare) {\n"
+	"			printf('Wrong rte_log2_u64(%' PRIx64 ') val %x,"
+		" expected %x\n',\n"
+	"				i64, compare, base);\n"
+	"			return TEST_FAILED;\n"
+	"		}\n"
+	"\n"
+	"		base = (uint32_t)ceilf(log2((uint32_t)i));\n"
+	"		compare = rte_log2_u32((uint32_t)i);\n"
+	"		if (base != compare) {\n"
+	"			printf('Wrong rte_log2_u32(%x) val %x, expected"
+		" %x\n',\n"
+	"				i, compare, base);\n"
+	"			return TEST_FAILED;\n"
+	"		}\n"
+	"		compare = rte_log2_u64((uint64_t)i);\n"
+	"		if (base != compare) {\n"
+	"			printf('Wrong rte_log2_u64(%x) val %x, expected"
+		" %x\n',\n"
+	"				i, compare, base);\n"
+	"			return TEST_FAILED;\n"
+	"		}\n"
+	"	}\n"
+	"	return 0;\n"
+	"}\n";
 
 static const char * const compress_test_bufs[] = {
 	test_buf_alice,
 	test_buf_shakespeare,
-	test_buf_pascal
+	test_buf_c_code
 };
 
 #endif /* TEST_COMPRESSDEV_TEST_BUFFERS_H_ */