From patchwork Tue Oct 17 05:45:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chaoyong He X-Patchwork-Id: 132711 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 62C5243186; Tue, 17 Oct 2023 07:48:53 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1634542D6A; Tue, 17 Oct 2023 07:47:18 +0200 (CEST) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2103.outbound.protection.outlook.com [40.107.244.103]) by mails.dpdk.org (Postfix) with ESMTP id 79413427D9 for ; Tue, 17 Oct 2023 07:47:16 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZzqY9YdbQPvEzYP1B4B5aPSGNYrbgM1+1Xv/zFRYgi4UIXJESq/XQXh/E0kS3BWnKYH19AVSwBxI5bRBfi002n7BIEV07lRlW6e3Jp1eVdWN24z8powznLP1P102ZGJ/LUYmWMNbBAr2JOyoC22jfcq67lLYqkgcGhkfIfEnc4cJjkgLrCRzoTFvpIfj0TwrV4wKNP/CbunPq94yS9lR41cpXDMkfmRnnWpihrB1ccCUvQJDM/uBH6+PMAxEcgt3/7sdqbugR0dpyo9NASDR1JmQvHBIwLLoaZMOoBqt7FYHi0mhsGHun/jAbgdDqud8PCJul8PrdAWOA34SGN1P3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ueD306333BC6aL/GrzldxBPxOnMsxU/MlgBa+o8r1+8=; b=Ke5ZoaZpfFbixm2JYn9r28Bbut1QCLujlOnt30z7a9dof6CA4KacfTtKL/asaHoO0nM40ydpUz23pxtDXniMQNix4UPn0uYCB5eNowjDBjh6kEWF4cy3oVj6SvdHi3uUzaz5oBII5xWUmlHytAA/PNO5cKnb0oq8NNYUMvHpHM/iCdj4nqZBJf6nAv4oJtPexxMtclF7vCu9IsEQRKPSym6SKYOiAQhtzceEjpEjU60fYCQdmW4m/Zv/w4mtG0Z1O8OGUxddD+McO95ohALRjOI8WBwl1HOAW0ZjFvwd3py1Xb6VZXSHewbSTGpuncmUnOm+Oa8Jm8J9LbsZpSZbbw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ueD306333BC6aL/GrzldxBPxOnMsxU/MlgBa+o8r1+8=; b=fLrelO66Z6t54091NFxw6npdacbcnC7ToD7koEFBrDsPJM3yzFZIoAtSTWuPw0k833ydhblYzx+9U3XeKbahh9Ep2Ifhdk7/Lp9ZlcOKWAnJXs0jy+NVC/fzBQdsNddm/pMIAr1i3GsuWGWMuD1bi1uq9P0U9yncfWuAcyeaqUs= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from PH0PR13MB5568.namprd13.prod.outlook.com (2603:10b6:510:12b::16) by MW5PR13MB5904.namprd13.prod.outlook.com (2603:10b6:303:1ce::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.45; Tue, 17 Oct 2023 05:47:14 +0000 Received: from PH0PR13MB5568.namprd13.prod.outlook.com ([fe80::b070:92e1:931e:fee7]) by PH0PR13MB5568.namprd13.prod.outlook.com ([fe80::b070:92e1:931e:fee7%4]) with mapi id 15.20.6863.047; Tue, 17 Oct 2023 05:47:14 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH 21/25] vdpa/nfp: add the hardware init logic Date: Tue, 17 Oct 2023 13:45:41 +0800 Message-Id: <20231017054545.1692509-22-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231017054545.1692509-1-chaoyong.he@corigine.com> References: <20231017054545.1692509-1-chaoyong.he@corigine.com> X-ClientProxiedBy: PH7PR17CA0069.namprd17.prod.outlook.com (2603:10b6:510:325::29) To PH0PR13MB5568.namprd13.prod.outlook.com (2603:10b6:510:12b::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR13MB5568:EE_|MW5PR13MB5904:EE_ X-MS-Office365-Filtering-Correlation-Id: 255db3ee-cb26-4b56-6a5f-08dbced483ed X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: yyqt2qI5oIhQUimWOKIMJOi9kBl2rlM9nM/mB4N9jwn2NOPxgNWN/hBZ10cKKvKSv7QkJYSE/KkLHZTMdFhH5ZvzDS1As0gto0ZZVbZFEcZ1Hm+CuoQGIxBrJmYK3OkpEOgsXbpnPhrIuJwGM5oFQHV0TkudiLYh+/zCUQ+4FZn0PmsoXjJ+s3IZ0H2mWr3g0f1qhKyKloIcsJ3sGn2dfYITd1Qk31h+M9fusw5DQO4xyXFsRJFEbXw6HpsWbFaRKlGYm7CjoWeINKCJSfmngckuBZxldcIsCfrki/dfo3A3d7kSSCoFx6yHD+qPbHvt6H2Z9J6lzTZyi+FHHT5qzdAvqwcasFaprVv7MQDhWoBvtWtCFMMVbNJH330snqHNaHgC/g1K4KKUt0qIL6Kb2eRlPw/ZF8RCYxVeOeJrwxTxo44abPDUnRtQT8O1KzPSGBsSQwZqcbIMV0lNyVxJBPz9dolkG4xyGPApXrVHtUYoT4kH3tUZ1mEqcgjRJbbJxQ2CHDrc3FoGD2/ncInlnXH1qRldxUJ3D0h6gKhyE4DKoPhcyfDAW2utWwNDq2RVAlA5KAT3mRHKZcAZwbDGgBiqkZ2hcapUmgT5cag1xHx+w3wE9Ppsw5Y/55Bm7hxO157uFr6szbecmvSFzyhlx2wsNdGDnkDfrkHZlk6wHVQ= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR13MB5568.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(366004)(376002)(39830400003)(346002)(396003)(230922051799003)(186009)(1800799009)(451199024)(64100799003)(2906002)(5660300002)(8936002)(8676002)(83380400001)(66556008)(66476007)(6916009)(316002)(66946007)(54906003)(41300700001)(478600001)(4326008)(6486002)(44832011)(6666004)(6512007)(6506007)(107886003)(38350700005)(2616005)(1076003)(26005)(36756003)(38100700002)(52116002)(86362001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Aigx6828HqY8KNf7LH3tlUoA8HuGc+SBsJZioBkJzYpSbseY9fC4QkixBUP0954Yx1a1b5mbs1Ofv8nMuxrfq3Eb8hhC1b4hYnc202ZbBVguWdgfRocxOxZVnLDiP94et1Xx9j9C9d6b0YCFSy8+RXoYAfX6Eg+ATl/zGYmHjXTQpzWcH6oRsmU6T4oiZ2hQ3cM+0GAjhY+zPVjTrhreg7Cd3+2sHv5dCN+j4RjELpSoZHCqgEiluT+cjKBS3GnHYfI8a1Ye1aTwKolBoZ3CWOfXif8liz+T2Rw1Nbj9IC14Bs/9WFqUPOT6WR5CKR5lSqYcjt+XxY3HzOzt1w5mStMbI/+mX6k/nwB7mY3RpxCkqCoB4b8WOBUZ7NHot91ebaGdr9/gDepDILRS9A8YkDefmDNamLW12/NxBeQ0HD+0YH7UcKa35y8ZAaoIm2m9o6xz5NiIM17F2v+PXC03COxSPL51fF0PKiDqE4Uq1lFm5SibrVmrUXbIEZuytg6+PmytSxKa1dlpYt/GW3SsfJbHypQzPkEbOIITRTVMTCvPQWY4N5K0vaWJYwC9IvcVa7nEcXMsbwXGNXPU6BoKrZI8mEVDNqXce6yR4vp9zJqziZUph++XsWAFt122WDPfon4NQfz5NoSuPvrFA+J4X/SZENDW9GC9pvhp4Eg2Vt2UGQA5xghqmbPAiWJNcrNs4XtpPg2eCTsaWQs0/16kIEAbEjYuD7HqwG9HQsamWBypyJ6MSacTnu81/z/Tg1h478U9mBc2sdFeONd5PMc51vMIN1WQmuEeJ0SUXzyg/UI2Z+WPcgaTUEQcDo2z2sPuKamVhrhpo5hBBzU9AdZBa4qutoSIs05Rsd9ZAWsGZy0s+UdaKW+lvVHqQo0K/Xqb+lb5lWJhD81hnXbUctxrCNkb4NqA9ec0zDe1J8fEbi96+L/UF/MZg0i9HnDwdMiWlkfpuUGnXfH3I//vm6ffuIQjEKGxo6Nm4CyM0IQ1/ruQDvvJJDxwRASVFdG7Q0oAC+AGlLJHoPYzNbtLKiDUJ8iDqWJKd8Qk3RRgcCSf8F2AgZ0qMkxJ7arBVhPdAy2UsZ+UFlx/groKlnp11QwA2759hrnnFn7ATy1hKLycgZGN2pT5YuyaovJOBMsKqCcR/y0EAuwbiAcw0wc9HAsgtbZetvLNBn2fHE1hjcyyrW47dlvriTErKvQwMd4AwYjMBMJiyQZyG2oSYqZC3hJEh7SLJTtpL7V9R31BQO8p3eXGSQoVT9yV210fl5R4MTPu7USl9JkM+2JekXL3InJ91IpWq11q4mknUWcc3G2z8GW9YAcEZk3aOYri1LUZOmQEgOn9eo3fvpDD9b55eHAGrGvcaqBuxXqtUBm9gePbpXJYgXrpYuc11zR9ZEmi6+0bcD0UWrQZKUSUMTsp97iNai1d/V3Y1P3dYEsAW52lNoCT9q5311RFQMrwyA4cBsVSvGazgdcEmbUbnr+A5eBECjtplTRbKs2uxxydRZ9Y4fXlSRMxGFD+sXz2xBEqNCXlQz5x/v1c3sQE3kHRLMOx7HJQml2tpL2dDiavSU68QlCyD/4wdge0aoKUdxVDZ2Gv01yqEqjF3AUgMgFaYI0xNg== X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 255db3ee-cb26-4b56-6a5f-08dbced483ed X-MS-Exchange-CrossTenant-AuthSource: PH0PR13MB5568.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 05:47:14.8481 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WYMvT1Ir9jZSXIjDUaUL0ZPfF4SUKC7dpIcr1WFNAnQf2D6FSynNaYPOdpOocNjh9XZ0HqWrb4/pIW5pplFZl8DTYAiB6rVQMhVKggY+SFM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR13MB5904 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add the hardware initialize logic. Signed-off-by: Chaoyong He Signed-off-by: Shujing Dong Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/vdpa/nfp/meson.build | 1 + drivers/vdpa/nfp/nfp_vdpa.c | 10 ++++++ drivers/vdpa/nfp/nfp_vdpa_core.c | 54 ++++++++++++++++++++++++++++++++ drivers/vdpa/nfp/nfp_vdpa_core.h | 32 +++++++++++++++++++ 4 files changed, 97 insertions(+) create mode 100644 drivers/vdpa/nfp/nfp_vdpa_core.c create mode 100644 drivers/vdpa/nfp/nfp_vdpa_core.h diff --git a/drivers/vdpa/nfp/meson.build b/drivers/vdpa/nfp/meson.build index e1cc244578..27eb1a6276 100644 --- a/drivers/vdpa/nfp/meson.build +++ b/drivers/vdpa/nfp/meson.build @@ -13,6 +13,7 @@ endif sources = files( 'nfp_vdpa.c', + 'nfp_vdpa_core.c', 'nfp_vdpa_log.c', ) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index 824b0d681a..00d8f7e007 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -10,6 +10,7 @@ #include #include +#include "nfp_vdpa_core.h" #include "nfp_vdpa_log.h" #define NFP_VDPA_DRIVER_NAME nfp_vdpa @@ -17,11 +18,14 @@ struct nfp_vdpa_dev { struct rte_pci_device *pci_dev; struct rte_vdpa_device *vdev; + struct nfp_vdpa_hw hw; int vfio_container_fd; int vfio_group_fd; int vfio_dev_fd; int iommu_group; + + uint16_t max_queues; }; struct nfp_vdpa_dev_node { @@ -135,6 +139,12 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) if (ret != 0) goto free_device; + ret = nfp_vdpa_hw_init(&device->hw, pci_dev); + if (ret != 0) + goto vfio_teardown; + + device->max_queues = NFP_VDPA_MAX_QUEUES; + device->vdev = rte_vdpa_register_device(&pci_dev->device, &nfp_vdpa_ops); if (device->vdev == NULL) { DRV_VDPA_LOG(ERR, "Failed to register device %s", pci_dev->name); diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c new file mode 100644 index 0000000000..a7e15fa88a --- /dev/null +++ b/drivers/vdpa/nfp/nfp_vdpa_core.c @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Corigine, Inc. + * All rights reserved. + */ + +#include "nfp_vdpa_core.h" + +#include + +#include "nfp_vdpa_log.h" + +/* Available and used descs are in same order */ +#ifndef VIRTIO_F_IN_ORDER +#define VIRTIO_F_IN_ORDER 35 +#endif + +int +nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, + struct rte_pci_device *pci_dev) +{ + uint32_t queue; + struct nfp_hw *hw; + uint8_t *notify_base; + + hw = &vdpa_hw->super; + hw->ctrl_bar = pci_dev->mem_resource[0].addr; + if (hw->ctrl_bar == NULL) { + DRV_CORE_LOG(ERR, "hw->ctrl_bar is NULL. BAR0 not configured."); + return -ENODEV; + } + + notify_base = hw->ctrl_bar + NFP_VDPA_NOTIFY_ADDR_BASE; + for (queue = 0; queue < NFP_VDPA_MAX_QUEUES; queue++) { + uint32_t idx = queue * 2; + + /* RX */ + vdpa_hw->notify_addr[idx] = notify_base; + notify_base += NFP_VDPA_NOTIFY_ADDR_INTERVAL; + /* TX */ + vdpa_hw->notify_addr[idx + 1] = notify_base; + notify_base += NFP_VDPA_NOTIFY_ADDR_INTERVAL; + + vdpa_hw->notify_region = queue; + DRV_CORE_LOG(DEBUG, "notify_addr[%d] at %p, notify_addr[%d] at %p", + idx, vdpa_hw->notify_addr[idx], + idx + 1, vdpa_hw->notify_addr[idx + 1]); + } + + vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) | + (1ULL << VIRTIO_F_IN_ORDER) | + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES); + + return 0; +} diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h new file mode 100644 index 0000000000..c9403e0ea4 --- /dev/null +++ b/drivers/vdpa/nfp/nfp_vdpa_core.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Corigine, Inc. + * All rights reserved. + */ + +#ifndef __NFP_VDPA_CORE_H__ +#define __NFP_VDPA_CORE_H__ + +#include +#include +#include + +#define NFP_VDPA_MAX_QUEUES 1 + +#define NFP_VDPA_NOTIFY_ADDR_BASE 0x4000 +#define NFP_VDPA_NOTIFY_ADDR_INTERVAL 0x1000 + +struct nfp_vdpa_hw { + struct nfp_hw super; + + uint64_t features; + uint64_t req_features; + + uint8_t *notify_addr[NFP_VDPA_MAX_QUEUES * 2]; + + uint8_t mac_addr[RTE_ETHER_ADDR_LEN]; + uint8_t notify_region; +}; + +int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev); + +#endif /* __NFP_VDPA_CORE_H__ */