From patchwork Tue Nov 7 20:31:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Conole X-Patchwork-Id: 133950 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F1711432CA; Tue, 7 Nov 2023 21:32:02 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EB94142D2E; Tue, 7 Nov 2023 21:32:02 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id E55E942D2E for ; Tue, 7 Nov 2023 21:32:01 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699389121; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R2RFGZkSy2M4Gb34kLHOqTLK9NspfeUeenjvSQm/V2Y=; b=gv+VX7m4zQ5atRXTggQGWd/IaDwNGY/95pPLc3lEDH4EmCy5AL/Q/4WXj6ix0HW+VK1Gg6 PtbOIzjFfqkUzXwwg072eMT31hjItHmzFwVfa+AP5/BZ/DBz67Z/RgpkeLYdXAQs8v4Ul+ ClDz/nkCtRCi1V0mjbiCt18/NMGV1fg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-596-HIqCJg-hNraetBJk-oQwVQ-1; Tue, 07 Nov 2023 15:31:59 -0500 X-MC-Unique: HIqCJg-hNraetBJk-oQwVQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4444E101A53B; Tue, 7 Nov 2023 20:31:59 +0000 (UTC) Received: from RHTPC1VM0NT.redhat.com (unknown [10.22.33.168]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1180119E93; Tue, 7 Nov 2023 20:31:59 +0000 (UTC) From: Aaron Conole To: ci@dpdk.org Cc: Michael Santana , David Marchand , Thomas Monjalon , Patrick Robb , Dumitru Ceara Subject: [RFC v2 1/3] pw_mon: improve command line options Date: Tue, 7 Nov 2023 15:31:56 -0500 Message-ID: <20231107203158.1261199-2-aconole@redhat.com> In-Reply-To: <20231107203158.1261199-1-aconole@redhat.com> References: <20231107203158.1261199-1-aconole@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.5 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org In the future, we'll use this to add support for passing opts into some parts of pw_mon. Signed-off-by: Aaron Conole Acked-by: Michael Santana --- pw_mon | 65 +++++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 51 insertions(+), 14 deletions(-) diff --git a/pw_mon b/pw_mon index 28feb8b..da4b9a9 100755 --- a/pw_mon +++ b/pw_mon @@ -21,34 +21,71 @@ [ -f "${HOME}/.pwmon-rc" ] && source "${HOME}/.pwmon-rc" -if [ "$1" != "" ]; then - pw_project="$1" - shift +if [ "$1" != "" ]; then + if ! echo "$1" | grep -q -s -E ^-- ; then + pw_project="$1" + shift + fi fi if [ "$1" != "" ]; then - pw_instance="$1" - shift -fi - -if [ "X$pw_instance" == "X" -o "X$pw_project" == "X" ]; then - echo "ERROR: Patchwork instance and project are unset." - echo "Please setup ${HOME}/.pwmon-rc and set pw_project " - echo "(or pass it as an argument)." - echo "Also either setup pw_instance or pass it as an argument." - exit 1 + if ! echo "$1" | grep -q -s -E ^-- ; then + pw_instance="$1" + shift + fi fi userpw="" if [ "$1" != "" ]; then - pw_credential="$1" + if ! echo "$1" | grep -q -s -E ^-- ; then + pw_credential="$1" + shift + fi fi + +while [ "$1" != "" ]; do + if echo "$1" | grep -q -s -E ^--pw-project= ; then + pw_project=$(echo "$1" | sed s/^--pw-project=//) + shift + elif echo "$1" | grep -q -s -E ^--pw-instance= ; then + pw_instance=$(echo "$1" | sed s/^--pw-instance=//) + shift + elif echo "$1" | grep -q -s -E ^--pw-credentials= ; then + pw_credential=$(echo "$1" | sed s/^--pw-credentials=//) + shift + elif echo "$1" | grep -E ^--help >/dev/null 2>&1; then + echo "patchwork monitor script" + echo "$0: args" + echo "Required if not set in ~/.pwmon-rc file:" + echo " proj|--pw-project= Project name" + echo " instance|--pw-instance= URL for pw" + echo "" + echo "Options:" + echo " --pw-credentials=u:p Sets user / password for web client" + echo " --add-filter-recheck=filter Adds a filter to flag that a recheck needs to be done" + echo "" + exit 0 + else + echo "Unknown option: '$1'" + echo "Rerun with --help for details" + exit 1 + fi +done + if [ "X$pw_credential" != "X" ]; then userpw="-u \"${pw_credential}\"" fi +if [ "X$pw_instance" == "X" -o "X$pw_project" == "X" ]; then + echo "ERROR: Patchwork instance and project are unset." + echo "Please setup ${HOME}/.pwmon-rc and set pw_project " + echo "(or pass it as an argument)." + echo "Also either setup pw_instance or pass it as an argument." + exit 1 +fi + source $(dirname $0)/series_db_lib.sh function emit_series() { From patchwork Tue Nov 7 20:31:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Conole X-Patchwork-Id: 133952 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2FF3A432CA; Tue, 7 Nov 2023 21:32:05 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2933140698; Tue, 7 Nov 2023 21:32:05 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 5CE254025E for ; Tue, 7 Nov 2023 21:32:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699389122; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zZiNvzrLQcJUzoWyUYa6+tbUy87KQew8wlxhjXcMRcY=; b=bXgkgh4pIKJOHk5JLEiCV4DNSOyPxR9O8PmPUZX4r+8a0bhfjqpv8C8bkXSj1W2WrWOt9Z n7ai4tUD/Ts9zfSvDVN39hDySIt/dzM9cB7LT7Isp+Li8Bnh1U5Cswbt00eqiGwuG0dORC IcDotd5LObmlf2mzoXn3e5QZEkbBnjs= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-373-STaBDw0MObCopNCKFbOmFA-1; Tue, 07 Nov 2023 15:31:59 -0500 X-MC-Unique: STaBDw0MObCopNCKFbOmFA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 88EF13C100A8; Tue, 7 Nov 2023 20:31:59 +0000 (UTC) Received: from RHTPC1VM0NT.redhat.com (unknown [10.22.33.168]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4F45719E93; Tue, 7 Nov 2023 20:31:59 +0000 (UTC) From: Aaron Conole To: ci@dpdk.org Cc: Michael Santana , David Marchand , Thomas Monjalon , Patrick Robb , Dumitru Ceara Subject: [RFC v2 2/3] recheck: Add a recheck parser for patchwork comments Date: Tue, 7 Nov 2023 15:31:57 -0500 Message-ID: <20231107203158.1261199-3-aconole@redhat.com> In-Reply-To: <20231107203158.1261199-1-aconole@redhat.com> References: <20231107203158.1261199-1-aconole@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.5 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Add a recheck parsing tool that will allow for labs to build a recheck workflow based on specific recheck labels and projects, with an associated state machine and querying interface. The output of the recheck parsing tool is json and can be fed to jq or other json parsing utilities for better field support. Signed-off-by: Aaron Conole Acked-by: Michael Santana --- pw_mon | 58 ++++++++++++++++++++++++++- recheck_tool | 100 +++++++++++++++++++++++++++++++++++++++++++++++ series_db_lib.sh | 64 +++++++++++++++++++++++++++++- 3 files changed, 220 insertions(+), 2 deletions(-) create mode 100755 recheck_tool diff --git a/pw_mon b/pw_mon index da4b9a9..29a9cb4 100755 --- a/pw_mon +++ b/pw_mon @@ -1,6 +1,6 @@ #!/bin/sh # SPDX-Identifier: gpl-2.0-or-later -# Copyright (C) 2018, Red Hat, Inc. +# Copyright (C) 2018-2023 Red Hat, Inc. # # Monitors a project on a patchwork instance for new series submissions # Records the submissions in the series database (and emits them on the @@ -44,6 +44,7 @@ if [ "$1" != "" ]; then fi fi +recheck_filter="" while [ "$1" != "" ]; do if echo "$1" | grep -q -s -E ^--pw-project= ; then @@ -67,6 +68,10 @@ while [ "$1" != "" ]; do echo " --add-filter-recheck=filter Adds a filter to flag that a recheck needs to be done" echo "" exit 0 + elif echo "$1" | grep -q -s -E ^--add-filter-recheck=; then + filter_str=$(echo "$1" | sed s/--add-filter-recheck=//) + recheck_filter="$filter_str $recheck_filter" + shift else echo "Unknown option: '$1'" echo "Rerun with --help for details" @@ -191,7 +196,58 @@ function check_superseded_series() { done } +function run_recheck() { + local recheck_list=$(echo "$7" | sed -e 's/^Recheck-request: // ' -e 's/,/ /g') + + for filter in $recheck_filter; do + for check in $recheck_list; do + if [ "$filter" == "$check" ]; then + insert_recheck_request_if_needed "$1" "$3" "$8" "$check" "$2" "$9" + fi + done + done +} + +function check_patch_for_retest_request() { + local patch_url="$1" + + local patch_json=$(curl -s "$userpw" "$patch_url") + local patch_comments_url=$(echo "$patch_json" | jq -rc '.comments') + local patch_id=$(echo "$patch_json" | jq -rc '.id') + if [ "Xnull" != "X$patch_comments_url" ]; then + local comments_json=$(curl -s "$userpw" "$patch_comments_url") + + local seq_end=$(echo "$comments_json" | jq -rc 'length') + if [ "$seq_end" -a $seq_end -gt 0 ]; then + seq_end=$((seq_end-1)) + for comment_id in $(seq 0 $seq_end); do + local recheck_requested=$(echo "$comments_json" | jq -rc ".[$comment_id].content" | grep "^Recheck-request: ") + if [ "X$recheck_requested" != "X" ]; then + local msgid=$(echo "$comments_json" | jq -rc ".[$comment_id].msgid") + run_recheck "$pw_instance" "$series_id" "$project" "$url" "$repo" "$branchname" "$recheck_requested" "$msgid" "$patch_id" + fi + done + fi + fi +} + +function check_series_needs_retest() { + local pw_instance="$1" + + series_get_active_branches "$pw_instance" | while IFS=\| read -r series_id project url repo branchname; do + local patch_comments_url=$(curl -s "$userpw" "$url" | jq -rc '.patches[] | .url') + + for patch in $patch_comments_url; do + check_patch_for_retest_request $patch + done + done +} + check_undownloaded_series "$pw_instance" "$pw_project" check_completed_series "$pw_instance" "$pw_project" check_new_series "$pw_instance" "$pw_project" check_superseded_series "$pw_instance" + +# check for retest requests after a series is still passing all the +# checks above +check_series_needs_retest "$pw_instance" diff --git a/recheck_tool b/recheck_tool new file mode 100755 index 0000000..98031a9 --- /dev/null +++ b/recheck_tool @@ -0,0 +1,100 @@ +#!/bin/sh +# SPDX-Identifier: gpl-2.0-or-later +# Copyright (C) 2023 Red Hat, Inc. +# +# Licensed under the terms of the GNU General Public License as published +# by the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. You may obtain a copy of the +# license at +# +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +mode="select" + +while [ "$1" != "" ]; do + if echo "$1" | grep -q -s -E ^--help ; then + echo "recheck / retest state machine script" + echo "" + echo "$0:" + echo " --pw-project=: Patchwork project." + echo " --pw-instance=: Patchwork instance." + echo " --filter=: Job / request for recheck." + echo " --state=<0..>: Resync state ID." + echo " --msgid=: Message ID to select." + echo " --update: Set tool in update mode" + echo " --new-state=<0..>: New state ID to set" + echo " --series-id=<..>: Series ID" + echo "" + echo "Will spit out a parsable json for each db line when selecting" + exit 0 + elif echo "$1" | grep -q -s -E ^--pw-project= ; then + pw_project=$(echo "$1" | sed s/^--pw-project=//) + elif echo "$1" | grep -q -s -E ^--pw-instance= ; then + pw_instance=$(echo "$1" | sed s/^--pw-instance=//) + elif echo "$1" | grep -q -s -E ^--filter= ; then + filter=$(echo "$1" | sed s/^--filter=//) + elif echo "$1" | grep -q -s -E ^--state= ; then + recheck_state=$(echo "$1" | sed s/^--state=//) + elif echo "$1" | grep -q -s -E ^--msgid= ; then + message_id=$(echo "$1" | sed s/^--msgid=//) + elif echo "$1" | grep -q -s -E ^--update ; then + mode="update" + elif echo "$1" | grep -q -s -E ^--new-state= ; then + new_recheck_state=$(echo "$1" | sed s/^--new-state=//) + elif echo "$1" | grep -q -s -E ^--series-id= ; then + series_id=$(echo "$1" | sed s/^--series-id=//) + else + echo "Unknown option: '$1'" + echo "Rerun with --help for details" + exit 1 + fi + shift +done + +source $(dirname $0)/series_db_lib.sh + +if [ "$mode" == "select" ]; then + echo -n "{\"rechecks\":[" + printed="" + for request in $(get_recheck_requests_by_project "$pw_instance" \ + "$pw_project" \ + "$recheck_state" \ + "$filter"); do + message_id=$(echo $request | cut -d\| -f1) + series_id=$(echo $request | cut -d\| -f2) + patch_id=$(echo $request | cut -d\| -f3) + + sha=$(get_sha_for_series_id_and_patch "$series_id" "$patch_id" \ + "$pw_instance") + echo -n "$printed{\"pw_instance\":\"$pw_instance\",\"series_id\":$series_id,\"patch_id\":$patch_id,\"sha\":\"$sha\",\"msg_id\":\"$message_id\",\"state\":\"$recheck_state\",\"requested\":\"$filter\"}" + printed="," + done + echo "]}" +elif [ "$mode" == "update" ]; then + if [ "X$new_recheck_state" == "X" -o "X$series_id" == "X" ]; then + echo "Need to set a series-id and a new recheck state when updating." + exit 1 + fi + + request=$(get_recheck_request "$pw_instance" "$pw_project" "$message_id" \ + "$filter" "$series_id" "$recheck_state") + if [ "X$request" == "X" ]; then + echo "{\"result\":\"notfound\"}" + exit 0 + fi + + set_recheck_request_state "$pw_instance" "$pw_project" "$message_id" \ + "$filter" "$series_id" "$new_recheck_state" + + echo "{\"result\":\"executed\",\"recheck\":{\"pw_instance\": \"$pw_instance\", \"series_id\":$series_id, \"msg_id\":\"$message_id\", \"state\":\"$new_recheck_state\", \"requested\": \"$filter\"}}" +else + echo "Unknown state: $mode" + exit 1 +fi + diff --git a/series_db_lib.sh b/series_db_lib.sh index 6c2d98e..ca33c1f 100644 --- a/series_db_lib.sh +++ b/series_db_lib.sh @@ -1,6 +1,6 @@ #!/bin/sh # SPDX-Identifier: gpl-2.0-or-later -# Copyright (C) 2018,2019 Red Hat, Inc. +# Copyright (C) 2018-2023 Red Hat, Inc. # # Licensed under the terms of the GNU General Public License as published # by the Free Software Foundation; either version 2 of the License, or @@ -114,6 +114,22 @@ EOF run_db_command "INSERT INTO series_schema_version(id) values (7);" fi + run_db_command "select * from series_schema_version;" | egrep '^8$' > /dev/null 2>&1 + if [ $? -eq 1 ]; then + sqlite3 ${HOME}/.series-db </dev/null 2>&1; then + echo "INSERT INTO recheck_requests (recheck_message_id, recheck_requested_by, recheck_series, recheck_patch, patchwork_instance, patchwork_project, recheck_sync) values (\"$recheck_msgid\", \"$recheck_requested_by\", \"$recheck_series\", $recheck_patch, \"$recheck_instance\", \"$recheck_project\", 0);" | series_db_execute + fi +} + +function get_recheck_request() { + local recheck_instance="$1" + local recheck_project="$2" + local recheck_msgid="$3" + local recheck_requested_by="$4" + local recheck_series="$5" + local recheck_state="$6" + + echo "select * from recheck_requests where patchwork_instance=\"$recheck_instance\" and patchwork_project=\"$recheck_project\" and recheck_requested_by=\"$recheck_requested_by\" and recheck_series=\"$recheck_series\" and recheck_message_id=\"$recheck_msgid\" and recheck_sync=$recheck_state;" | series_db_execute +} + +function set_recheck_request_state() { + local recheck_instance="$1" + local recheck_project="$2" + local recheck_msgid="$3" + local recheck_requested_by="$4" + local recheck_series="$5" + local recheck_state="$6" + + echo "UPDATE recheck_requests set recheck_sync=$recheck_state where patchwork_instance=\"$recheck_instance\" and patchwork_project=\"$recheck_project\" and recheck_requested_by=\"$recheck_requested_by\" and recheck_series=\"$recheck_series\";" | series_db_execute +} From patchwork Tue Nov 7 20:31:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Conole X-Patchwork-Id: 133951 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 17D60432CA; Tue, 7 Nov 2023 21:32:04 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 11FFC40698; Tue, 7 Nov 2023 21:32:04 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id CAD674025E for ; Tue, 7 Nov 2023 21:32:02 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699389122; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gykds1XSdqvJ2Oye6cSilZN506KLY6/+NXz51yROpZg=; b=LOTXeczHYNLmVWQ/EX5ApZSDle5xf3i7T/LdsWp1E4G4dLnadpr/QoNBQzBm/lNEyvnDa3 gXLcXpnXSC6lG381KwSNtRhhiCDVFr5a/8x9EzzhL1AaGo/qzDKqa+zWDPzyJ/d61Fc5L/ 3qi2egOjfzd+pxwAonSQGVjyv9KItD0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-670-l6ilWbjAN2uEKLdlR1ss4w-1; Tue, 07 Nov 2023 15:32:00 -0500 X-MC-Unique: l6ilWbjAN2uEKLdlR1ss4w-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C81BE2810D41; Tue, 7 Nov 2023 20:31:59 +0000 (UTC) Received: from RHTPC1VM0NT.redhat.com (unknown [10.22.33.168]) by smtp.corp.redhat.com (Postfix) with ESMTP id 947A419E93; Tue, 7 Nov 2023 20:31:59 +0000 (UTC) From: Aaron Conole To: ci@dpdk.org Cc: Michael Santana , David Marchand , Thomas Monjalon , Patrick Robb , Dumitru Ceara Subject: [RFC v2 3/3] github: add a tool for restarting checks Date: Tue, 7 Nov 2023 15:31:58 -0500 Message-ID: <20231107203158.1261199-4-aconole@redhat.com> In-Reply-To: <20231107203158.1261199-1-aconole@redhat.com> References: <20231107203158.1261199-1-aconole@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.5 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org The recheck framework can track specific recheck labels and track a state for them. Add a tool that will restart github workflow runs and mark the git build table to check for a repoll. This will allow existing monitor tools to re-poll the workflow and report rerun results. Signed-off-by: Aaron Conole Acked-by: Michael Santana --- github_restart | 141 +++++++++++++++++++++++++++++++++++++++++++++++ series_db_lib.sh | 16 ++++++ 2 files changed, 157 insertions(+) create mode 100755 github_restart diff --git a/github_restart b/github_restart new file mode 100755 index 0000000..5ba9677 --- /dev/null +++ b/github_restart @@ -0,0 +1,141 @@ +#!/bin/bash +# SPDX-Identifier: gpl-2.0-or-later +# Copyright (C) 2023, Red Hat, Inc. +# +# Restarts a github job run. This can be used in conjunction with +# the recheck requests to provide the ability for a user to restart +# a test - in case the workflow is suspected of having a spurious run. +# +# Licensed under the terms of the GNU General Public License as published +# by the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. You may obtain a copy of the +# license at +# +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +wait_start="no" + +while [ "$1" != "" ]; do + if echo "$1" | grep -q -s -E ^--pw-project= ; then + pw_project=$(echo "$1" | sed s/^--pw-project=//) + shift + elif echo "$1" | grep -q -s -E ^--pw-instance= ; then + pw_instance=$(echo "$1" | sed s/^--pw-instance=//) + shift + elif echo "$1" | grep -q -s -E ^--pw-credentials= ; then + pw_credential=$(echo "$1" | sed s/^--pw-credentials=//) + shift + elif echo "$1" | grep -q -s -E ^--series-id= ; then + series_id=$(echo "$1" | sed s/^--series-id=//) + shift + elif echo "$1" | grep -q -s -E ^--wait-start ; then + wait_start="yes" + if echo "$1" | grep -q -s -E ^--wait-start= ; then + wait_start=$(echo "$1" | sed s/^--wait-start=//) + fi + shift + elif echo "$1" | grep -q -s -E ^--github-token= ; then + github_token=$(echo "$1" | sed s/^--github-token=//) + shift + elif echo "$1" | grep -q -s -E ^--repository= ; then + reponame=$(echo "$1" | sed s/^--repository=//) + shift + elif echo "$1" | grep -q -s -E ^--run-id= ; then + runid=$(echo "$1" | sed s/^--repository=//) + shift + elif echo "$1" | grep -q -s -E ^--sha= ; then + sha=$(echo "$1" | sed s/^--sha=//) + shift + elif echo "$1" | grep -q -s -E ^--workflow= ; then + workflow=$(echo "$1" | sed s/^--workflow=//) + shift + elif echo "$1" | grep -q -s -E ^--help ; then + echo "github restarting script" + echo "$0: args" + echo " --pw-project= Project name" + echo " --pw-instance= URL for pw" + echo " --series-id=id Series ID for reference" + echo " --github-token=token Token for github" + echo " --wait-start[=yes|no] Wait for the remote to start" + echo " --repository=repo Repository name (ex foo/bar)" + echo " --run-id=id run ID" + echo " --workflow=workflow Workflow name" + echo " --sha=commit Commit sha" + echo "" + exit 0 + else + echo "Unknown option: '$1'" >&2 + echo "Rerun with --help for details" >&2 + exit 1 + fi +done + +source $(dirname $0)/series_db_lib.sh + +if [ "X$wait_start" != "Xno" -a "X$wait_start" != "Xyes" ]; then + echo "Unrecognized '$wait_start' argument to --wait-start=" >&2 + echo "valid values are 'yes' or 'no'." >&2 + exit 1 +fi + +if [ "X$github_token" == "X" ]; then + echo "Please set a github API token." >&2 + echo "Use --help for more info." >&2 + exit 1 +fi + +if [ "X$reponame" == "X" ]; then + echo "Please set a repository (ie: '--repository=owner/repo')." >&2 + echo "Use --help for more info." >&2 + exit 1 +fi + +AUTH="Authorization: token ${github_token}" +APP="Accept: application/vnd.github.v3+json" + +if [ "X$runid" == "X" ]; then + + # lookup the runs based on the shasum + + if [ "X$sha" == "X" ]; then + echo "Need a runid or shasum to key off." >&2 + echo "See --help for more details." >&2 + exit 1 + fi + + comma="" + for job in $(curl -s -S -H "${AUTH}" -H "${APP}" \ + "https://api.github.com/repos/${reponame}/actions/runs?head_sha=${sha}" | \ + jq -rc '.workflow_runs[].id') + do + runid="${comma}${job}" + comma="," + done +fi + +echo -n "{\"results\":[" +comma="" +for job in $(echo "$runid" | sed 's/,/ /'); do + result=$(curl -s -X POST -L -S -H "${AUTH}" -H "${APP}" \ + "https://api.github.com/repos/${reponame}/actions/runs/$job/rerun") + msg=$(echo "$result" | jq -rc '.message') + + echo -n "$comma{\"run\":$job,\"result\":" + if [ "X$msg" == "Xnull" ]; then + echo -n "\"sent\"" + if [ "X$series_id" != "X" ]; then + echo -n ",\"gap_sync\":\"reset\"" + set_unsynced_for_series "$series_id" "$pw_instance" "gap_sync" + fi + else + echo -n "\"err\",\"error\":\"$msg\"" + fi + echo -n "}" +done +echo "]}" diff --git a/series_db_lib.sh b/series_db_lib.sh index ca33c1f..3f052ad 100644 --- a/series_db_lib.sh +++ b/series_db_lib.sh @@ -382,6 +382,14 @@ function set_synced_for_series() { echo "update git_builds set gap_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute } +function set_unsynced_for_series() { + local series_id="$1" + local instance="$2" + local ci_instance="$3" + + echo "update git_builds set $ci_instance=0 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute +} + function insert_commit() { local series_id="$1" local patch_id="$2" @@ -407,6 +415,14 @@ function get_patch_id_by_series_id_and_sha() { echo "select patch_id from git_builds where patchwork_instance=\"$instance\" and series_id=$series_id and sha=\"$sha\";" | series_db_execute } +function get_sha_for_series_id_and_patch() { + local series_id="$1" + local patch_id="$2" + local instance="$3" + + echo "select sha from git_builds where patchwork_instance=\"$instance\" and series_id=\"$series_id\" and patch_id=\"$patch_id\"" | series_db_execute +} + function get_recheck_requests_by_project() { local recheck_instance="$1" local recheck_project="$2"