From patchwork Mon Jul 15 15:58:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Hyong Youb Kim (hyonkim)" X-Patchwork-Id: 56463 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7E7062C18; Mon, 15 Jul 2019 17:58:33 +0200 (CEST) Received: from rcdn-iport-4.cisco.com (rcdn-iport-4.cisco.com [173.37.86.75]) by dpdk.org (Postfix) with ESMTP id 43B142BF5 for ; Mon, 15 Jul 2019 17:58:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=5355; q=dns/txt; s=iport; t=1563206312; x=1564415912; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=z/F8TCOwVVn8gl0v2waqs8j49GavoUwRqNL+0CvZmOs=; b=K0Rh7lTNOG583+AqJECPlmBjZQmhsVLDlzHXYeq55JcyTdIu/4/XvDSH bV1mbmFLHzmrHlpc1i1eK49okiuLMMZmzELi7hOLxeA+m4tc7kHIVRHHy TfSmc3Rm4s8SuOc+xZTKezhHXJne9xZK28meLEOPj35eBFuGSqWRYGudl o=; X-IronPort-AV: E=Sophos;i="5.63,493,1557187200"; d="scan'208";a="596907986" Received: from rcdn-core-8.cisco.com ([173.37.93.144]) by rcdn-iport-4.cisco.com with ESMTP/TLS/DHE-RSA-SEED-SHA; 15 Jul 2019 15:58:30 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by rcdn-core-8.cisco.com (8.15.2/8.15.2) with ESMTP id x6FFwUIC021548; Mon, 15 Jul 2019 15:58:30 GMT Received: by cisco.com (Postfix, from userid 508933) id EFBFE20F2003; Mon, 15 Jul 2019 08:58:29 -0700 (PDT) From: Hyong Youb Kim To: David Marchand , Thomas Monjalon , Jerin Jacob Kollanukkaran , Ferruh Yigit Cc: dev@dpdk.org, John Daley , Hyong Youb Kim Date: Mon, 15 Jul 2019 08:58:12 -0700 Message-Id: <20190715155812.10124-1-hyonkim@cisco.com> X-Mailer: git-send-email 2.22.0 MIME-Version: 1.0 X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: rcdn-core-8.cisco.com Subject: [dpdk-dev] [RFC PATCH] vfio: avoid re-installing irq handler X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" A rough patch for the approach mentioned earlier. It is only for discussion. http://mails.dpdk.org/archives/dev/2019-July/138113.html To try this out, first revert the following then apply. commit 89aac60e0be9 ("vfio: fix interrupts race condition") Signed-off-by: Hyong Youb Kim --- .../common/include/rte_eal_interrupts.h | 1 + lib/librte_eal/linux/eal/eal_interrupts.c | 68 ++++++++++++++++++- 2 files changed, 67 insertions(+), 2 deletions(-) diff --git a/lib/librte_eal/common/include/rte_eal_interrupts.h b/lib/librte_eal/common/include/rte_eal_interrupts.h index b370c0d26..d509967d3 100644 --- a/lib/librte_eal/common/include/rte_eal_interrupts.h +++ b/lib/librte_eal/common/include/rte_eal_interrupts.h @@ -76,6 +76,7 @@ struct rte_intr_handle { enum rte_intr_handle_type type; /**< handle type */ uint32_t max_intr; /**< max interrupt requested */ uint32_t nb_efd; /**< number of available efd(event fd) */ + uint8_t vfio_efd_assigned; uint8_t efd_counter_size; /**< size of efd counter, used for vdev */ int efds[RTE_MAX_RXTX_INTR_VEC_ID]; /**< intr vectors/efds mapping */ struct rte_epoll_event elist[RTE_MAX_RXTX_INTR_VEC_ID]; diff --git a/lib/librte_eal/linux/eal/eal_interrupts.c b/lib/librte_eal/linux/eal/eal_interrupts.c index 79ad5e8d7..5b03388dd 100644 --- a/lib/librte_eal/linux/eal/eal_interrupts.c +++ b/lib/librte_eal/linux/eal/eal_interrupts.c @@ -107,9 +107,21 @@ static pthread_t intr_thread; #define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \ sizeof(int) * (RTE_MAX_RXTX_INTR_VEC_ID + 1)) -/* enable legacy (INTx) interrupts */ +#ifndef __INTEL_COMPILER +#pragma GCC diagnostic ignored "-Wcast-qual" +#endif + +static void +set_vfio_efd_assigned(const struct rte_intr_handle *intr_handle, uint8_t v) { + ((struct rte_intr_handle *)intr_handle)->vfio_efd_assigned = v; +} + +#ifndef __INTEL_COMPILER +#pragma GCC diagnostic error "-Wcast-qual" +#endif + static int -vfio_enable_intx(const struct rte_intr_handle *intr_handle) { +vfio_assign_efd_intx(const struct rte_intr_handle *intr_handle) { struct vfio_irq_set *irq_set; char irq_set_buf[IRQ_SET_BUF_LEN]; int len, ret; @@ -134,8 +146,36 @@ vfio_enable_intx(const struct rte_intr_handle *intr_handle) { intr_handle->fd); return -1; } + /* + * In the kernel, irq has been set up, its handler installed (request_irq), + * and efd is assigned to it. Remember this so we can avoid + * re-doing irq setup later. + */ + set_vfio_efd_assigned(intr_handle, 1); + return 0; +} + +/* enable legacy (INTx) interrupts */ +static int +vfio_enable_intx(const struct rte_intr_handle *intr_handle) { + struct vfio_irq_set *irq_set; + char irq_set_buf[IRQ_SET_BUF_LEN]; + int len, ret; + + /* + * Assign efd to irq only if it's not been done, to avoid + * re-installing irq handler in the kernel. + */ + if (!intr_handle->vfio_efd_assigned) { + ret = vfio_assign_efd_intx(intr_handle); + if (ret) + return ret; + } + + len = sizeof(irq_set_buf); /* unmask INTx after enabling */ + irq_set = (struct vfio_irq_set *) irq_set_buf; memset(irq_set, 0, len); len = sizeof(struct vfio_irq_set); irq_set->argsz = len; @@ -194,6 +234,12 @@ vfio_disable_intx(const struct rte_intr_handle *intr_handle) { "Error disabling INTx interrupts for fd %d\n", intr_handle->fd); return -1; } + /* + * Irq handler's been removed in the kernel. The next time we + * enable intr_handle, we need to assign efd to irq. Remember + * that. + */ + set_vfio_efd_assigned(intr_handle, 0); return 0; } @@ -205,6 +251,13 @@ vfio_enable_msi(const struct rte_intr_handle *intr_handle) { struct vfio_irq_set *irq_set; int *fd_ptr; + /* + * vfio does not support MSI masking/unmaking. So if irq has + * been set up previously, nothing to do. + */ + if (intr_handle->vfio_efd_assigned) + return 0; + len = sizeof(irq_set_buf); irq_set = (struct vfio_irq_set *) irq_set_buf; @@ -223,6 +276,7 @@ vfio_enable_msi(const struct rte_intr_handle *intr_handle) { intr_handle->fd); return -1; } + set_vfio_efd_assigned(intr_handle, 1); return 0; } @@ -248,6 +302,7 @@ vfio_disable_msi(const struct rte_intr_handle *intr_handle) { RTE_LOG(ERR, EAL, "Error disabling MSI interrupts for fd %d\n", intr_handle->fd); + set_vfio_efd_assigned(intr_handle, 0); return ret; } @@ -259,6 +314,13 @@ vfio_enable_msix(const struct rte_intr_handle *intr_handle) { struct vfio_irq_set *irq_set; int *fd_ptr; + /* + * vfio does not support MSI-X masking/unmaking. So if irq has + * been set up previously, nothing to do. + */ + if (intr_handle->vfio_efd_assigned) + return 0; + len = sizeof(irq_set_buf); irq_set = (struct vfio_irq_set *) irq_set_buf; @@ -284,6 +346,7 @@ vfio_enable_msix(const struct rte_intr_handle *intr_handle) { return -1; } + set_vfio_efd_assigned(intr_handle, 1); return 0; } @@ -309,6 +372,7 @@ vfio_disable_msix(const struct rte_intr_handle *intr_handle) { RTE_LOG(ERR, EAL, "Error disabling MSI-X interrupts for fd %d\n", intr_handle->fd); + set_vfio_efd_assigned(intr_handle, 0); return ret; }