Message ID | 20230407060152.16335-1-nipun.gupta@amd.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A70B0428E2; Fri, 7 Apr 2023 08:02:31 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 33BD140DDB; Fri, 7 Apr 2023 08:02:31 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2072.outbound.protection.outlook.com [40.107.93.72]) by mails.dpdk.org (Postfix) with ESMTP id 8677940041 for <dev@dpdk.org>; Fri, 7 Apr 2023 08:02:29 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Orp5UOSt6P7HcycobLdMLuNuGwjgBS/7fJX6BvF0Nx3gOdLlrG13uvyZQZPWh6RcOahHDPR0VRZ3RxE7hu3kyOuda3QA53gx0oXzSAwIjz7BncaY5gyK59moolK2czoFZJAgChNfgsW3q0baE5sxzlwC2eJeWAa8Gvs8wm2GSsLYvw+8KVEEiSz4zJ63uAMWlk9znFlh83B+3bYSM9LZ7gfwCm5eABaucToMfVobEfugyP/EztUMV/X0fdw9zRUslbwnH9J+6pgSFQ/04CS0KWv8tY7k3xec2xu2d52uhTVo+RxtZmK1BMAhJgH7mT/N4/Dmnafi1dQjIhnSC1CSHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nQeD00S0MQALn8D5TvwshOHXCtVGeHBcUxqPdf5zxPo=; b=HdQNklfYgOjsoPW69C+/3iL8GdbWrpikLj5IcaS50WFDpOw4QxQgRfFFKgmmKLjGlHmoYIZw/LK2MUyIlCG2b9Vrc9qbNMVaHniR+rBdVKCM8d4icLmQxw6sQCDlELMKZRxz656xu0sBjkh8Xbuwi7ahefBdki1PLgtmzT+XmLJy+m2T5lM70EdGAY1KR0yzjwAkFA7M8NvSxb9GscJd34U3kFfRUd5BvOvFlttRClI6XTXp9UDnrXPzZTknaqK96kJpYo+eQSgb4M9GVup0HQgM0Xsf2WzVM6LMJAhZH1/pYZMRA0J9Eyy/Ya9tNHGz9w2goGevbmuAFisogjwLxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=dpdk.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nQeD00S0MQALn8D5TvwshOHXCtVGeHBcUxqPdf5zxPo=; b=sfQUxnNSAg+wKwQlFtYXYuTNvL4t0U+5Zb6wEjAh6wjpIM32IvjL6PHu704OvHM+exd3u3ZyUZLUZxk4ZR7S3NZ1c77HwOtWpHXrsAKYv7D23EkQAuRubdyIFubPUADyNuG6B/g11C8WIGP1eg6lqzHdxT08JX/LWr+f2AHuhPo= Received: from BN0PR03CA0009.namprd03.prod.outlook.com (2603:10b6:408:e6::14) by CY8PR12MB7612.namprd12.prod.outlook.com (2603:10b6:930:9c::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Fri, 7 Apr 2023 06:02:25 +0000 Received: from BN8NAM11FT087.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e6:cafe::ad) by BN0PR03CA0009.outlook.office365.com (2603:10b6:408:e6::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6277.31 via Frontend Transport; Fri, 7 Apr 2023 06:02:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT087.mail.protection.outlook.com (10.13.177.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6298.19 via Frontend Transport; Fri, 7 Apr 2023 06:02:25 +0000 Received: from SATLEXMB07.amd.com (10.181.41.45) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Fri, 7 Apr 2023 01:02:24 -0500 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB07.amd.com (10.181.41.45) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Thu, 6 Apr 2023 23:02:18 -0700 Received: from xhdipdslab41.xilinx.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2375.34 via Frontend Transport; Fri, 7 Apr 2023 01:02:16 -0500 From: Nipun Gupta <nipun.gupta@amd.com> To: <dev@dpdk.org>, <thomas@monjalon.net>, <david.marchand@redhat.com> CC: <ferruh.yigit@amd.com>, <harpreet.anand@amd.com>, <nikhil.agarwal@amd.com>, Nipun Gupta <nipun.gupta@amd.com> Subject: [PATCH 0/6] add support for CDX bus Date: Fri, 7 Apr 2023 11:31:46 +0530 Message-ID: <20230407060152.16335-1-nipun.gupta@amd.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230124140746.594066-1-nipun.gupta@amd.com> References: <20230124140746.594066-1-nipun.gupta@amd.com> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT087:EE_|CY8PR12MB7612:EE_ X-MS-Office365-Filtering-Correlation-Id: ade43303-ab3b-4bc9-abd1-08db372da932 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: T4T2YScOBwqZqQnZN6NUT1PiLPBMhvzrSEQfoF9e40wu6bqEfv7uO9N+GPd7janNZiUZQlDdqshIRLKBC2Rxc1RvlKzkZs8HWSFoNp1H22Dv5QxhGIWBYJ9es8HO/KKT0CCsi98MRf9wrvR9UoevQzj/aPjU4SH7vlci3i8qfuJhAUAm6GSs+Ikq1tT7Nz3IbFi3nVtG/uo6MpjGqDFW7LsqKxL/uFfre2Gfo2dspQk89KA7VYZ3wrrCiaKfw1vzBDFnx2/dOFFe48BzAzpNxFnWQL8OpkWZUtPHlORnXVPzElgj6QMWrJlCNBTiIuDy82/57I6v6ok2vmQvqFw0x88oGqKacDfu53jFGQadBQFo3x8bd8PYKLgKokHGA/FZXnWuRmqoqhHADpSAsNUeUjFe6UH5HUQDThwtu6gNi+B6x006PNyoI+OYBzY4YSoYSAP/OL7B4reSckg4MX8GNraZkRYtx/jbbo3OzdihcwCWOrJlFyZaRPZtBYEPRYSYwECvrWMu7lEfYC4Re56WGoKOnrv6RK4M7XANFOgC3hkBsAcocWP/Ho5GCKbXKctBqh3l6u+P8ISOBkdqDJTPqLOfbjkklNvRh8ygT2W+YwXvSCj5zIc6N/aPoMIKnfWG8bYjOUSuEbP8rgqtG4J9g8s5JG0qqU/adw24bogII0qJHJLPtI4jWMZ5yTkySuTpgyYzVTlkApm7MnQmmd/DDcvdMIp5CqPHCx512JdikvK1KUl+vypeOsbnvzMsguYIJhWyY5BrlvWjBtcYaBSHXA== X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230028)(4636009)(136003)(396003)(39860400002)(346002)(376002)(451199021)(36840700001)(40470700004)(46966006)(2616005)(86362001)(36756003)(82310400005)(2906002)(8936002)(40480700001)(186003)(426003)(966005)(6666004)(26005)(1076003)(82740400003)(70586007)(8676002)(36860700001)(336012)(40460700003)(478600001)(70206006)(4326008)(54906003)(41300700001)(44832011)(81166007)(47076005)(5660300002)(356005)(316002)(110136005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Apr 2023 06:02:25.4314 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ade43303-ab3b-4bc9-abd1-08db372da932 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT087.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7612 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org |
Series | add support for CDX bus | |
Message
Gupta, Nipun
April 7, 2023, 6:01 a.m. UTC
Support AMD CDX bus, for FPGA based CDX devices. The CDX devices are memory mapped on system bus for embedded CPUs. It uses sysfs interface and the vfio-cdx driver to discover and initialize the CDX devices. The patches are intended for DPDK 23.07 release, and have been sent as an RFC as patches are yet to be merged in Linux. Linux CDX bus patches has been added into linux next: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx VFIO patches are also submitted in upstream: https://www.spinics.net/lists/kvm/msg310623.html CDX is a Hardware Architecture designed for AMD FPGA devices. It consists of mechanism for interaction between FPGA, Firmware and the APUs (Application CPUs). Firmware resides on RPU (Realtime CPUs) which interacts with the FPGA program manager and the APUs. The RPU provides memory-mapped interface (RPU if) which is used to communicate with APUs. VFIO CDX driver provides the CDX device resources like MMIO and interrupts to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx driver to discover and initialize the CDX devices for user-space applications. RFC patches in DPDK were submitted at: http://patches.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/ Changes RFC->v1: - Marked few API's as internal which were not required to be provided to user. Nipun Gupta (6): bus/cdx: introduce cdx bus bus/cdx: add dma map and unmap support bus/cdx: add support for MSI bus/cdx: support plug unplug and dev iterator bus: enable cdx bus config/arm: add AMD CDX MAINTAINERS | 5 + config/arm/arm64_cdx_linux_gcc | 17 + config/arm/meson.build | 14 + drivers/bus/cdx/cdx.c | 743 +++++++++++++++++++++++++ drivers/bus/cdx/cdx.h | 54 ++ drivers/bus/cdx/cdx_logs.h | 37 ++ drivers/bus/cdx/cdx_vfio.c | 598 ++++++++++++++++++++ drivers/bus/cdx/meson.build | 13 + drivers/bus/cdx/rte_bus_cdx.h | 245 ++++++++ drivers/bus/cdx/version.map | 21 + drivers/bus/meson.build | 1 + lib/eal/common/eal_common_interrupts.c | 21 + lib/eal/common/eal_interrupts.h | 1 + lib/eal/include/rte_interrupts.h | 32 ++ lib/eal/version.map | 2 + 15 files changed, 1804 insertions(+) create mode 100644 config/arm/arm64_cdx_linux_gcc create mode 100644 drivers/bus/cdx/cdx.c create mode 100644 drivers/bus/cdx/cdx.h create mode 100644 drivers/bus/cdx/cdx_logs.h create mode 100644 drivers/bus/cdx/cdx_vfio.c create mode 100644 drivers/bus/cdx/meson.build create mode 100644 drivers/bus/cdx/rte_bus_cdx.h create mode 100644 drivers/bus/cdx/version.map
Comments
Hello, On Fri, Apr 7, 2023 at 8:02 AM Nipun Gupta <nipun.gupta@amd.com> wrote: > > Support AMD CDX bus, for FPGA based CDX devices. The CDX > devices are memory mapped on system bus for embedded CPUs. > > It uses sysfs interface and the vfio-cdx driver to discover > and initialize the CDX devices. > > The patches are intended for DPDK 23.07 release, and have been sent > as an RFC as patches are yet to be merged in Linux. > > Linux CDX bus patches has been added into linux next: > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx > > VFIO patches are also submitted in upstream: > https://www.spinics.net/lists/kvm/msg310623.html Hard to tell just from this link what the status is. Has it been reviewed? When are you expecting this to get merged? > > CDX is a Hardware Architecture designed for AMD FPGA devices. It > consists of mechanism for interaction between FPGA, Firmware and > the APUs (Application CPUs). > Firmware resides on RPU (Realtime CPUs) which interacts with > the FPGA program manager and the APUs. The RPU provides memory-mapped > interface (RPU if) which is used to communicate with APUs. > > VFIO CDX driver provides the CDX device resources like MMIO and interrupts > to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx > driver to discover and initialize the CDX devices for user-space > applications. > > RFC patches in DPDK were submitted at: > http://patches.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/ > > Changes RFC->v1: > - Marked few API's as internal which were not required > to be provided to user. Just a quick pass on this series. - There is an issue with the exported header rte_bus_cdx.h. All the bus specific / driver only parts must be moved to a dedicated bus_cdx_driver.h header (which must be listed in driver_sdk_headers). - To whom is the rte_cdx_dump symbol intended to? Is this for use by user applications? If so, the rte_bus_cdx.h can be kept, to export this symbol. But otherwise, everything in rte_bus_cdx.h can be moved to bus_cdx_driver.h. - Please remove the rte_ prefix for static symbols in *.c.
On 4/7/2023 12:48 PM, David Marchand wrote: > Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding. > > > Hello, > > On Fri, Apr 7, 2023 at 8:02 AM Nipun Gupta <nipun.gupta@amd.com> wrote: >> >> Support AMD CDX bus, for FPGA based CDX devices. The CDX >> devices are memory mapped on system bus for embedded CPUs. >> >> It uses sysfs interface and the vfio-cdx driver to discover >> and initialize the CDX devices. >> >> The patches are intended for DPDK 23.07 release, and have been sent >> as an RFC as patches are yet to be merged in Linux. >> >> Linux CDX bus patches has been added into linux next: >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/cdx >> >> VFIO patches are also submitted in upstream: >> https://www.spinics.net/lists/kvm/msg310623.html > > Hard to tell just from this link what the status is. > Has it been reviewed? > When are you expecting this to get merged? The CDX bus code has been merged and VFIO code is under review. Apart from this, we will soon have Open source Linux from AMD which has all the required patches for CDX in a week or so (I will provide the link once that is available). > > >> >> CDX is a Hardware Architecture designed for AMD FPGA devices. It >> consists of mechanism for interaction between FPGA, Firmware and >> the APUs (Application CPUs). >> Firmware resides on RPU (Realtime CPUs) which interacts with >> the FPGA program manager and the APUs. The RPU provides memory-mapped >> interface (RPU if) which is used to communicate with APUs. >> >> VFIO CDX driver provides the CDX device resources like MMIO and interrupts >> to map to user-space. DPDK CDX bus uses sysfs interface and the vfio-cdx >> driver to discover and initialize the CDX devices for user-space >> applications. >> >> RFC patches in DPDK were submitted at: >> http://patches.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/ >> >> Changes RFC->v1: >> - Marked few API's as internal which were not required >> to be provided to user. > > Just a quick pass on this series. > > - There is an issue with the exported header rte_bus_cdx.h. > All the bus specific / driver only parts must be moved to a dedicated > bus_cdx_driver.h header (which must be listed in driver_sdk_headers). > > - To whom is the rte_cdx_dump symbol intended to? Is this for use by > user applications? > If so, the rte_bus_cdx.h can be kept, to export this symbol. > But otherwise, everything in rte_bus_cdx.h can be moved to bus_cdx_driver.h. > > - Please remove the rte_ prefix for static symbols in *.c. Agree to this. Regards, Nipun > > > -- > David Marchand >
> -----Original Message----- > From: Gupta, Nipun > Sent: Friday, April 7, 2023 1:00 PM > To: David Marchand <david.marchand@redhat.com> > Cc: dev@dpdk.org; thomas@monjalon.net; Yigit, Ferruh > <Ferruh.Yigit@amd.com>; Anand, Harpreet <harpreet.anand@amd.com>; > Agarwal, Nikhil <nikhil.agarwal@amd.com> > Subject: Re: [PATCH 0/6] add support for CDX bus > > > > On 4/7/2023 12:48 PM, David Marchand wrote: > > Caution: This message originated from an External Source. Use proper caution > when opening attachments, clicking links, or responding. > > > > > > Hello, > > > > On Fri, Apr 7, 2023 at 8:02 AM Nipun Gupta <nipun.gupta@amd.com> wrote: > >> > >> Support AMD CDX bus, for FPGA based CDX devices. The CDX > >> devices are memory mapped on system bus for embedded CPUs. > >> > >> It uses sysfs interface and the vfio-cdx driver to discover > >> and initialize the CDX devices. > >> > >> The patches are intended for DPDK 23.07 release, and have been sent > >> as an RFC as patches are yet to be merged in Linux. > >> > >> Linux CDX bus patches has been added into linux next: > >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux- > next.git/tree/drivers/cdx > >> > >> VFIO patches are also submitted in upstream: > >> https://www.spinics.net/lists/kvm/msg310623.html > > > > Hard to tell just from this link what the status is. > > Has it been reviewed? > > When are you expecting this to get merged? > > The CDX bus code has been merged and VFIO code is under review. Apart > from this, we will soon have Open source Linux from AMD which has all > the required patches for CDX in a week or so (I will provide the link > once that is available). The CDX bus and VFIO support is now available at Xilinx open-source tree: https://github.com/Xilinx/linux-xlnx (drivers/cdx/ and drivers/vfio/cdx) Regards, Nipun