[v2] tools: check for pending test status when parsing emails

Today, the community CI infrastructure only uses post-result reporting,
such as "SUCCESS", "FAILED", and "WARNING".  These results get reported
only after a test finishes.  This creates some confusion about whether a
test might have been started for the series in question.  It isn't easy
to tell at-a-glance which tests are currently running for a given patch
or series.

This patch aims to introduce support for a "PENDING" state in the CI
infrastructure.  This allows labs to indicate which tests have started
and are awaiting results.  That means test writers should now send a
"PENDING" status for tests as they start, and then update with a
post-test result after.  With this change, understanding which tests ran
at-a-glance is something we can achieve.

This change should have no affect on the actual tests being run.

Signed-off-by: Patrick Robb <probb@iol.unh.edu>
---
 tools/update-pw.sh | 1 +
 1 file changed, 1 insertion(+)
  

> -----Original Message-----
> From: Patrick Robb <probb@iol.unh.edu>
> Sent: Friday, May 24, 2024 1:00 AM
> To: ci@dpdk.org
> Cc: aconole@redhat.com; ahassick@iol.unh.edu; NBU-Contact-Thomas
> Monjalon (EXTERNAL) <thomas@monjalon.net>; Ali Alnubani
> <alialnu@nvidia.com>; Patrick Robb <probb@iol.unh.edu>
> Subject: [PATCH v2] tools: check for pending test status when parsing emails
> 
> Today, the community CI infrastructure only uses post-result reporting,
> such as "SUCCESS", "FAILED", and "WARNING".  These results get reported
> only after a test finishes.  This creates some confusion about whether a
> test might have been started for the series in question.  It isn't easy
> to tell at-a-glance which tests are currently running for a given patch
> or series.
> 
> This patch aims to introduce support for a "PENDING" state in the CI
> infrastructure.  This allows labs to indicate which tests have started
> and are awaiting results.  That means test writers should now send a
> "PENDING" status for tests as they start, and then update with a
> post-test result after.  With this change, understanding which tests ran
> at-a-glance is something we can achieve.
> 
> This change should have no affect on the actual tests being run.
> 
> Signed-off-by: Patrick Robb <probb@iol.unh.edu>
> ---

Acked-by: Ali Alnubani <alialnu@nvidia.com>

Thanks,
Ali
  

Patrick Robb <probb@iol.unh.edu> writes:

> Today, the community CI infrastructure only uses post-result reporting,
> such as "SUCCESS", "FAILED", and "WARNING".  These results get reported
> only after a test finishes.  This creates some confusion about whether a
> test might have been started for the series in question.  It isn't easy
> to tell at-a-glance which tests are currently running for a given patch
> or series.
>
> This patch aims to introduce support for a "PENDING" state in the CI
> infrastructure.  This allows labs to indicate which tests have started
> and are awaiting results.  That means test writers should now send a
> "PENDING" status for tests as they start, and then update with a
> post-test result after.  With this change, understanding which tests ran
> at-a-glance is something we can achieve.
>
> This change should have no affect on the actual tests being run.
>
> Signed-off-by: Patrick Robb <probb@iol.unh.edu>
> ---

Thanks Patrick and Ali - applied.
  

Thanks Aaron for merging and Ali for updating on the PW side.

UNH is going to start reporting with the PENDING status today for just
the ABI patchwork context. If all looks good, we will enable for the
entire set of contexts we are responsible for tomorrow.
  

Ali,

I've been monitoring Patchwork today after rolling out the feature on
just the ABI context, and I've noticed that checks with "pending"
status do not appear in the table or in the Patchwork API. I only see
the check appear in the table/API once the final "success" check is
sent.

Here's a thread that I've been tracking for example:
http://inbox.dpdk.org/test-report/667c6064.050a0220.f0848.4472SMTPIN_ADDED_MISSING@mx.google.com/T/#u
It looks to me that the "Test-Status" tag is set correctly.

Is it possible there are errors when adding the checks on the Patchwork end?

Thanks,
Adam


On Tue, Jun 25, 2024 at 3:34 PM Patrick Robb <probb@iol.unh.edu> wrote:
>
> Thanks Aaron for merging and Ali for updating on the PW side.
>
> UNH is going to start reporting with the PENDING status today for just
> the ABI patchwork context. If all looks good, we will enable for the
> entire set of contexts we are responsible for tomorrow.
  

> -----Original Message-----
> From: Adam Hassick <ahassick@iol.unh.edu>
> Sent: Wednesday, June 26, 2024 10:32 PM
> To: Patrick Robb <probb@iol.unh.edu>
> Cc: Aaron Conole <aconole@redhat.com>; ci@dpdk.org; NBU-Contact-Thomas
> Monjalon (EXTERNAL) <thomas@monjalon.net>; Ali Alnubani
> <alialnu@nvidia.com>
> Subject: Re: [PATCH v2] tools: check for pending test status when parsing emails
> 
> Ali,
> 
> I've been monitoring Patchwork today after rolling out the feature on
> just the ABI context, and I've noticed that checks with "pending"
> status do not appear in the table or in the Patchwork API. I only see
> the check appear in the table/API once the final "success" check is
> sent.
> 
> Here's a thread that I've been tracking for example:
> http://inbox.dpdk.org/test-
> report/667c6064.050a0220.f0848.4472SMTPIN_ADDED_MISSING@mx.google.c
> om/T/#u
> It looks to me that the "Test-Status" tag is set correctly.
> 
> Is it possible there are errors when adding the checks on the Patchwork end?
> 

Hi Adam,

I just realized that I pulled the new change in the wrong path. Can you please try now?

Thanks,
Ali
  

https://patchwork.dpdk.org/project/dpdk/patch/20240627075755.66386-1-sthotton@marvell.com/

Thanks Ali. I see pending for this patch for ABI testing. We will look
in a couple of hours and it should by then have updated to pass/fail.
Assuming that we don't notice any weird behavior, we will merge our PR
tomorrow morning for the remaining PW contexts we report (currently we
still only have ABI pending enabled).

One thing we didn't think about when we set this up was the "S/W/F"
counter that can be seen from
https://patchwork.dpdk.org/project/dpdk/list/. That does not include
pending. I guess that will be something to discuss at our next CI
meeting!

Thanks.

On Thu, Jun 27, 2024 at 12:48 PM Ali Alnubani <alialnu@nvidia.com> wrote:
>
> > -----Original Message-----
> > From: Adam Hassick <ahassick@iol.unh.edu>
> > Sent: Wednesday, June 26, 2024 10:32 PM
> > To: Patrick Robb <probb@iol.unh.edu>
> > Cc: Aaron Conole <aconole@redhat.com>; ci@dpdk.org; NBU-Contact-Thomas
> > Monjalon (EXTERNAL) <thomas@monjalon.net>; Ali Alnubani
> > <alialnu@nvidia.com>
> > Subject: Re: [PATCH v2] tools: check for pending test status when parsing emails
> >
> > Ali,
> >
> > I've been monitoring Patchwork today after rolling out the feature on
> > just the ABI context, and I've noticed that checks with "pending"
> > status do not appear in the table or in the Patchwork API. I only see
> > the check appear in the table/API once the final "success" check is
> > sent.
> >
> > Here's a thread that I've been tracking for example:
> > http://inbox.dpdk.org/test-
> > report/667c6064.050a0220.f0848.4472SMTPIN_ADDED_MISSING@mx.google.c
> > om/T/#u
> > It looks to me that the "Test-Status" tag is set correctly.
> >
> > Is it possible there are errors when adding the checks on the Patchwork end?
> >
>
> Hi Adam,
>
> I just realized that I pulled the new change in the wrong path. Can you please try now?
>
> Thanks,
> Ali