From patchwork Fri Apr 19 19:53:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Boyer X-Patchwork-Id: 139556 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 796AB43EB5; Fri, 19 Apr 2024 21:54:15 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 31D4C40A7F; Fri, 19 Apr 2024 21:53:57 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2076.outbound.protection.outlook.com [40.107.220.76]) by mails.dpdk.org (Postfix) with ESMTP id DAEF740A72 for ; Fri, 19 Apr 2024 21:53:55 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gG94afX2thagsOb36zpq93YqHtXfIltX/TT9CUxobiU+61AkAekU/nLViiCBIG70cI38M/t7/bUWrYRkpoFRqvmLrNqEyG6fXscPHnxKfFO5KG4ToP5VEEIUpBR2H2q9WWlSlGvWcl38HrqeaVKOAhzqeGfopQYHDi2+wCs0ljpJ25vV1Ivlpx8NQwqLbXthNH7trXxS8T+9myhAwrt8U017oybRPi9Gq/Yr3nKkmCYaCsRbx45ZF8G8sYPlkPV4vrR81VhJRHDQzd3Req/hN0XG5vXcVhy0m4VVbd83Oy8oN/9GEFtpET4VIITWSBXNvw7QTI4WoJc5fVuZPC1+Mg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rds9JQkFmRDQ/DfGbhd9V9UNm0zC7RNm1Oar/8Jf3dY=; b=oarRNQ29kwfHBfw2w6KRbU8TsY19X3h+K84AiVJnRL93fs2bZviHCGrSwbiLhimU1vOiRhYkzEWuNNl5oQooFJFQ0NWY85ARXp4Jr9GAberLvzjUOqzTidb9vSZnHOvzx06EiOTCgGQHWdVO8SK3uh8nVIzSA7YJ/piyq8ydBTtZPmYvNF0yvWIPWQ0n9n6rx+RXI44NlYKL/ZBNd/h+qPu0pzv8Am4D6ZxX3IvWFIJedCfRAvZSvLDqeUw4mgqQ0nHOm1aNAhTk+fmRhnhR0aBiDENN8Lxi4mJzXyVud8fz/ylzBbYshgsGKTCii2EsXp00ii0k9/vjcx3IaZ/BFw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=dpdk.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rds9JQkFmRDQ/DfGbhd9V9UNm0zC7RNm1Oar/8Jf3dY=; b=0xz4e9LE2Oe3ipKO/XhIyIHyuA3Xyi1oExEBR4jss4jYr8RWEwamCfEo7BofW4jn2DuhdXIrFN5ezJtmQF3GB1C2ZtAbjAB73NGxn7oZujPK4d4Vtvu8IcJ1jd+enjP2ogVi2rak0VTC4B/W2c9vZKeR/TvCMloy3a5xjz/SsAc= Received: from SA0PR12CA0019.namprd12.prod.outlook.com (2603:10b6:806:6f::24) by SJ2PR12MB8806.namprd12.prod.outlook.com (2603:10b6:a03:4d0::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.41; Fri, 19 Apr 2024 19:53:50 +0000 Received: from SA2PEPF00001506.namprd04.prod.outlook.com (2603:10b6:806:6f:cafe::32) by SA0PR12CA0019.outlook.office365.com (2603:10b6:806:6f::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7495.31 via Frontend Transport; Fri, 19 Apr 2024 19:53:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SA2PEPF00001506.mail.protection.outlook.com (10.167.242.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7452.22 via Frontend Transport; Fri, 19 Apr 2024 19:53:50 +0000 Received: from driver-dev1.pensando.io (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 19 Apr 2024 14:53:48 -0500 From: Andrew Boyer To: CC: Akhil Goyal , Andrew Boyer Subject: [PATCH 4/6] crypto/ionic: add device object and vdev support Date: Fri, 19 Apr 2024 12:53:08 -0700 Message-ID: <20240419195310.21432-5-andrew.boyer@amd.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240419195310.21432-1-andrew.boyer@amd.com> References: <20240419195310.21432-1-andrew.boyer@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001506:EE_|SJ2PR12MB8806:EE_ X-MS-Office365-Filtering-Correlation-Id: 2117f918-2203-46a0-3a3f-08dc60aa6ee9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +rkCfdV30IXJQAZP8INX/WOATJJArlF+4Zx9mQs+LR/VD964FHgGKVTFJuh4DmjxvTH3zMVdtaBNRYWH+fQFGhEnrDqjReHLVfh0p7TrnHFP32I9oYhIGjp0ArPvzh5pBaGe1DCp/mJgdG2E1n24h7H/tN/S1d2/jDQ0vzfWvlACl+NZ/0LHkU8uNfTQaQQTHtj10pFNl5qUE+d6LZD+71PD/1FNccdhgRAOV+FHLtXEkCMC290KFX/ctJddmU+5RlVEYZVgdCm9LIApqYULubTEfJivS7id2JzLNx+2TAzkwIu/MTBWwl0cB6ivjqIoyW9HnCvha7kRUEIj3fFGxH/E8o87ljxvWfvd8dtoGU/2/vu47PoK2aQmbkoByUhgTzmGf8dLRRabx0BSMSOxI2mzr7YRO6hDlvYOD8SqFwLTcj6AIYVXWG0w4bxI6p7TsgIJ2H8z2/6c+sNydyDTveVpOiW+FHxGAr1jG0NuuSagzo7G8gC0kN+hJSpgwqsb/Are7b62LSMsqMocElpGopuGiyYvNDTe9aq+eONAzb5l+xekmFDUVKzJtxtX6NtyOXs4/cnMr0tICKrIp5tRfXNMc+dDB5nKdGvfXCCFdUayw+ji9c43AAMBXrB0CpYGc3Hb4lSzrngaKCZ+g8D/YcSV1xjRw9DccwNe8haxczJInqXQp7coA9rPE08Mrb6o/8qizZDnKi1DtDXQ9ELdj6OLdLBf7TeoT9uZ6Or31Vpq2/WmrnNn9bZkFx27Yt7SFj4vB2KwFmofGJ3b/tw9zDdj0umOLCQkW9XCCtRFoG95xwoKvU2SCxuwXYH78d3ayPUOODbGmM/jqXqqItViGQL3tz0awzdrbL2A6zO3FfRvx7a7YIUGAJDwC+qz0VB880Gekyqrr09ZieJD8PWaBahBd1Kf+FzQKy2EfTswido5oqVqs2BKEVfGKz8bhooQ6K39LNBVm/G27wZoqlQcjOYJKN2rUz7sNDNNGgOixE3XJty0K4lh1/+nEm5pYrserQ1KEX14MmQHYh0QHpr+loDdzShG0+a1Ak9nvNVeqIFTh46/XWH2HMzLgZRAPltgWDdIXChzMynZ6OdpDz6c7Cwo43vXI2nVbnkyPNUbzuHMs8USlC1FTvAne1J7S+py3n461jE4j9GBt02mpzfvxceFFZd4ShT/Xlnftrp36AIbzWfYm2i1YZiFbIwfV7jEp8YxDIJs5g/A2xjt5ePeAGwOASWySO/1cTeMzXhJtGS6zonEwtxVbpCys/LBolxpmYs9BzxB79uiL99XRhlxfjVp9vZdkm1YV3FkLZfRrPCs9aYcJU6vI6Et2NvMTo+0BVxckONtBBfOspFYcylu2RsyxZkt3teMvTFBobu5eAo= X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230031)(376005)(1800799015)(82310400014)(36860700004); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Apr 2024 19:53:50.0628 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2117f918-2203-46a0-3a3f-08dc60aa6ee9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001506.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8806 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This defines the main device object routines and the vdev support code. The vdev code uses the common library. Signed-off-by: Andrew Boyer --- drivers/crypto/ionic/ionic_crypto.h | 89 ++++ drivers/crypto/ionic/ionic_crypto_main.c | 538 +++++++++++++++++++++++ drivers/crypto/ionic/ionic_crypto_vdev.c | 128 ++++++ drivers/crypto/ionic/meson.build | 2 + 4 files changed, 757 insertions(+) create mode 100644 drivers/crypto/ionic/ionic_crypto_vdev.c diff --git a/drivers/crypto/ionic/ionic_crypto.h b/drivers/crypto/ionic/ionic_crypto.h index 958e611337..d048f7aa51 100644 --- a/drivers/crypto/ionic/ionic_crypto.h +++ b/drivers/crypto/ionic/ionic_crypto.h @@ -20,6 +20,11 @@ #include "ionic_crypto_if.h" #include "ionic_regs.h" +/* Devargs */ +/* NONE */ + +#define IOCPT_MAX_RING_DESC 32768 +#define IOCPT_MIN_RING_DESC 16 #define IOCPT_ADMINQ_LENGTH 16 /* must be a power of two */ #define IOCPT_CRYPTOQ_WAIT 10 /* 1s */ @@ -32,6 +37,64 @@ extern int iocpt_logtype; #define IOCPT_PRINT_CALL() IOCPT_PRINT(DEBUG, " >>") +static inline void iocpt_struct_size_checks(void) +{ + RTE_BUILD_BUG_ON(sizeof(struct ionic_doorbell) != 8); + RTE_BUILD_BUG_ON(sizeof(struct ionic_intr) != 32); + RTE_BUILD_BUG_ON(sizeof(struct ionic_intr_status) != 8); + + RTE_BUILD_BUG_ON(sizeof(union iocpt_dev_regs) != 4096); + RTE_BUILD_BUG_ON(sizeof(union iocpt_dev_info_regs) != 2048); + RTE_BUILD_BUG_ON(sizeof(union iocpt_dev_cmd_regs) != 2048); + + RTE_BUILD_BUG_ON(sizeof(struct iocpt_admin_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_admin_comp) != 16); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_nop_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_nop_comp) != 16); + + /* Device commands */ + RTE_BUILD_BUG_ON(sizeof(struct iocpt_dev_identify_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_dev_identify_comp) != 16); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_dev_reset_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_dev_reset_comp) != 16); + + /* LIF commands */ + RTE_BUILD_BUG_ON(sizeof(struct iocpt_lif_identify_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_lif_identify_comp) != 16); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_lif_init_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_lif_init_comp) != 16); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_lif_reset_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_lif_getattr_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_lif_getattr_comp) != 16); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_lif_setattr_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_lif_setattr_comp) != 16); + + /* Queue commands */ + RTE_BUILD_BUG_ON(sizeof(struct iocpt_q_identify_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_q_identify_comp) != 16); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_q_init_cmd) != 64); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_q_init_comp) != 16); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_q_control_cmd) != 64); + + /* Crypto */ + RTE_BUILD_BUG_ON(sizeof(struct iocpt_crypto_desc) != 32); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_crypto_sg_desc) != 256); + RTE_BUILD_BUG_ON(sizeof(struct iocpt_crypto_comp) != 16); +} + +struct iocpt_dev_bars { + struct ionic_dev_bar bar[IONIC_BARS_MAX]; + uint32_t num_bars; +}; + +/* Queue watchdog */ +#define IOCPT_Q_WDOG_SESS_IDX 0 +#define IOCPT_Q_WDOG_KEY_LEN 16 +#define IOCPT_Q_WDOG_IV_LEN 12 +#define IOCPT_Q_WDOG_PLD_LEN 4 +#define IOCPT_Q_WDOG_TAG_LEN 16 +#define IOCPT_Q_WDOG_OP_TYPE RTE_CRYPTO_OP_TYPE_UNDEFINED + struct iocpt_qtype_info { uint8_t version; uint8_t supported; @@ -108,8 +171,10 @@ struct iocpt_admin_q { struct iocpt_dev { const char *name; char fw_version[IOCPT_FWVERS_BUFLEN]; + struct iocpt_dev_bars bars; struct iocpt_identity ident; + const struct iocpt_dev_intf *intf; void *bus_dev; struct rte_cryptodev *crypto_dev; @@ -130,6 +195,8 @@ struct iocpt_dev { struct iocpt_admin_q *adminq; + struct rte_bitmap *sess_bm; /* SET bit indicates index is free */ + uint64_t features; uint32_t hw_features; @@ -144,6 +211,20 @@ struct iocpt_dev { struct rte_cryptodev_stats stats_base; }; +struct iocpt_dev_intf { + int (*setup_bars)(struct iocpt_dev *dev); + void (*unmap_bars)(struct iocpt_dev *dev); +}; + +static inline int +iocpt_setup_bars(struct iocpt_dev *dev) +{ + if (dev->intf->setup_bars == NULL) + return -EINVAL; + + return (*dev->intf->setup_bars)(dev); +} + /** iocpt_admin_ctx - Admin command context. * @pending_work: Flag that indicates a completion. * @cmd: Admin command (64B) to be copied to the queue. @@ -155,6 +236,14 @@ struct iocpt_admin_ctx { union iocpt_adminq_comp comp; }; +int iocpt_probe(void *bus_dev, struct rte_device *rte_dev, + struct iocpt_dev_bars *bars, const struct iocpt_dev_intf *intf, + uint8_t driver_id, uint8_t socket_id); +int iocpt_remove(struct rte_device *rte_dev); + +void iocpt_configure(struct iocpt_dev *dev); +void iocpt_deinit(struct iocpt_dev *dev); + int iocpt_dev_identify(struct iocpt_dev *dev); int iocpt_dev_init(struct iocpt_dev *dev, rte_iova_t info_pa); int iocpt_dev_adminq_init(struct iocpt_dev *dev); diff --git a/drivers/crypto/ionic/ionic_crypto_main.c b/drivers/crypto/ionic/ionic_crypto_main.c index 7b26080bd1..84aff65f22 100644 --- a/drivers/crypto/ionic/ionic_crypto_main.c +++ b/drivers/crypto/ionic/ionic_crypto_main.c @@ -12,6 +12,32 @@ int iocpt_logtype; +static int +iocpt_cq_init(struct iocpt_cq *cq, uint16_t num_descs) +{ + if (!rte_is_power_of_2(num_descs) || + num_descs < IOCPT_MIN_RING_DESC || + num_descs > IOCPT_MAX_RING_DESC) { + IOCPT_PRINT(ERR, "%u descriptors (min: %u max: %u)", + num_descs, IOCPT_MIN_RING_DESC, IOCPT_MAX_RING_DESC); + return -EINVAL; + } + + cq->num_descs = num_descs; + cq->size_mask = num_descs - 1; + cq->tail_idx = 0; + cq->done_color = 1; + + return 0; +} + +static void +iocpt_cq_map(struct iocpt_cq *cq, void *base, rte_iova_t base_pa) +{ + cq->base = base; + cq->base_pa = base_pa; +} + uint32_t iocpt_cq_service(struct iocpt_cq *cq, uint32_t work_to_do, iocpt_cq_cb cb, void *cb_arg) @@ -33,10 +59,522 @@ iocpt_cq_service(struct iocpt_cq *cq, uint32_t work_to_do, return work_done; } +static int +iocpt_q_init(struct iocpt_queue *q, uint8_t type, uint32_t index, + uint16_t num_descs, uint16_t num_segs, uint32_t socket_id) +{ + uint32_t ring_size; + + if (!rte_is_power_of_2(num_descs)) + return -EINVAL; + + ring_size = rte_log2_u32(num_descs); + if (ring_size < 2 || ring_size > 16) + return -EINVAL; + + q->type = type; + q->index = index; + q->num_descs = num_descs; + q->num_segs = num_segs; + q->size_mask = num_descs - 1; + q->head_idx = 0; + q->tail_idx = 0; + + q->info = rte_calloc_socket("iocpt", + num_descs * num_segs, sizeof(void *), + rte_mem_page_size(), socket_id); + if (q->info == NULL) { + IOCPT_PRINT(ERR, "Cannot allocate queue info"); + return -ENOMEM; + } + + return 0; +} + +static void +iocpt_q_map(struct iocpt_queue *q, void *base, rte_iova_t base_pa) +{ + q->base = base; + q->base_pa = base_pa; +} + +static void +iocpt_q_sg_map(struct iocpt_queue *q, void *base, rte_iova_t base_pa) +{ + q->sg_base = base; + q->sg_base_pa = base_pa; +} + +static void +iocpt_q_free(struct iocpt_queue *q) +{ + if (q->info != NULL) { + rte_free(q->info); + q->info = NULL; + } +} + +static const struct rte_memzone * +iocpt_dma_zone_reserve(const char *type_name, uint16_t qid, size_t size, + unsigned int align, int socket_id) +{ + char zone_name[RTE_MEMZONE_NAMESIZE]; + const struct rte_memzone *mz; + int err; + + err = snprintf(zone_name, sizeof(zone_name), + "iocpt_%s_%u", type_name, qid); + if (err >= RTE_MEMZONE_NAMESIZE) { + IOCPT_PRINT(ERR, "Name %s too long", type_name); + return NULL; + } + + mz = rte_memzone_lookup(zone_name); + if (mz != NULL) + return mz; + + return rte_memzone_reserve_aligned(zone_name, size, socket_id, + RTE_MEMZONE_IOVA_CONTIG, align); +} + +static int +iocpt_commonq_alloc(struct iocpt_dev *dev, + uint8_t type, + size_t struct_size, + uint32_t socket_id, + uint32_t index, + const char *type_name, + uint16_t flags, + uint16_t num_descs, + uint16_t num_segs, + uint16_t desc_size, + uint16_t cq_desc_size, + uint16_t sg_desc_size, + struct iocpt_common_q **comq) +{ + struct iocpt_common_q *new; + uint32_t q_size, cq_size, sg_size, total_size; + void *q_base, *cq_base, *sg_base; + rte_iova_t q_base_pa = 0; + rte_iova_t cq_base_pa = 0; + rte_iova_t sg_base_pa = 0; + size_t page_size = rte_mem_page_size(); + int err; + + *comq = NULL; + + q_size = num_descs * desc_size; + cq_size = num_descs * cq_desc_size; + sg_size = num_descs * sg_desc_size; + + /* + * Note: aligning q_size/cq_size is not enough due to cq_base address + * aligning as q_base could be not aligned to the page. + * Adding page_size. + */ + total_size = RTE_ALIGN(q_size, page_size) + + RTE_ALIGN(cq_size, page_size) + page_size; + if (flags & IOCPT_Q_F_SG) + total_size += RTE_ALIGN(sg_size, page_size) + page_size; + + new = rte_zmalloc_socket("iocpt", struct_size, + RTE_CACHE_LINE_SIZE, socket_id); + if (new == NULL) { + IOCPT_PRINT(ERR, "Cannot allocate queue structure"); + return -ENOMEM; + } + + new->dev = dev; + + err = iocpt_q_init(&new->q, type, index, num_descs, num_segs, + socket_id); + if (err != 0) { + IOCPT_PRINT(ERR, "Queue initialization failed"); + goto err_free_q; + } + + err = iocpt_cq_init(&new->cq, num_descs); + if (err != 0) { + IOCPT_PRINT(ERR, "Completion queue initialization failed"); + goto err_deinit_q; + } + + new->base_z = iocpt_dma_zone_reserve(type_name, index, total_size, + IONIC_ALIGN, socket_id); + if (new->base_z == NULL) { + IOCPT_PRINT(ERR, "Cannot reserve queue DMA memory"); + err = -ENOMEM; + goto err_deinit_cq; + } + + new->base = new->base_z->addr; + new->base_pa = new->base_z->iova; + + q_base = new->base; + q_base_pa = new->base_pa; + iocpt_q_map(&new->q, q_base, q_base_pa); + + cq_base = (void *)RTE_ALIGN((uintptr_t)q_base + q_size, page_size); + cq_base_pa = RTE_ALIGN(q_base_pa + q_size, page_size); + iocpt_cq_map(&new->cq, cq_base, cq_base_pa); + + if (flags & IOCPT_Q_F_SG) { + sg_base = (void *)RTE_ALIGN((uintptr_t)cq_base + cq_size, + page_size); + sg_base_pa = RTE_ALIGN(cq_base_pa + cq_size, page_size); + iocpt_q_sg_map(&new->q, sg_base, sg_base_pa); + } + + IOCPT_PRINT(DEBUG, "q_base_pa %#jx cq_base_pa %#jx sg_base_pa %#jx", + q_base_pa, cq_base_pa, sg_base_pa); + + *comq = new; + + return 0; + +err_deinit_cq: +err_deinit_q: + iocpt_q_free(&new->q); +err_free_q: + rte_free(new); + return err; +} + struct ionic_doorbell * iocpt_db_map(struct iocpt_dev *dev, struct iocpt_queue *q) { return dev->db_pages + q->hw_type; } +static int +iocpt_adminq_alloc(struct iocpt_dev *dev) +{ + struct iocpt_admin_q *aq; + uint16_t num_descs = IOCPT_ADMINQ_LENGTH; + uint16_t flags = 0; + int err; + + err = iocpt_commonq_alloc(dev, + IOCPT_QTYPE_ADMINQ, + sizeof(struct iocpt_admin_q), + rte_socket_id(), + 0, + "admin", + flags, + num_descs, + 1, + sizeof(struct iocpt_admin_cmd), + sizeof(struct iocpt_admin_comp), + 0, + (struct iocpt_common_q **)&aq); + if (err != 0) + return err; + + aq->flags = flags; + + dev->adminq = aq; + + return 0; +} + +static int +iocpt_adminq_init(struct iocpt_dev *dev) +{ + return iocpt_dev_adminq_init(dev); +} + +static void +iocpt_adminq_deinit(struct iocpt_dev *dev) +{ + dev->adminq->flags &= ~IOCPT_Q_F_INITED; +} + +static void +iocpt_adminq_free(struct iocpt_admin_q *aq) +{ + if (aq->base_z != NULL) { + rte_memzone_free(aq->base_z); + aq->base_z = NULL; + aq->base = NULL; + aq->base_pa = 0; + } + + iocpt_q_free(&aq->q); + + rte_free(aq); +} + +static int +iocpt_alloc_objs(struct iocpt_dev *dev) +{ + uint32_t bmsize, i; + uint8_t *bm; + int err; + + IOCPT_PRINT(DEBUG, "Crypto: %s", dev->name); + + rte_spinlock_init(&dev->adminq_lock); + rte_spinlock_init(&dev->adminq_service_lock); + + err = iocpt_adminq_alloc(dev); + if (err != 0) { + IOCPT_PRINT(ERR, "Cannot allocate admin queue"); + err = -ENOMEM; + goto err_out; + } + + dev->info_sz = RTE_ALIGN(sizeof(*dev->info), rte_mem_page_size()); + dev->info_z = iocpt_dma_zone_reserve("info", 0, dev->info_sz, + IONIC_ALIGN, dev->socket_id); + if (dev->info_z == NULL) { + IOCPT_PRINT(ERR, "Cannot allocate dev info memory"); + err = -ENOMEM; + goto err_free_adminq; + } + + dev->info = dev->info_z->addr; + dev->info_pa = dev->info_z->iova; + + bmsize = rte_bitmap_get_memory_footprint(dev->max_sessions); + bm = rte_malloc_socket("iocpt", bmsize, + RTE_CACHE_LINE_SIZE, dev->socket_id); + if (bm == NULL) { + IOCPT_PRINT(ERR, "Cannot allocate %uB bitmap memory", bmsize); + err = -ENOMEM; + goto err_free_dmazone; + } + + dev->sess_bm = rte_bitmap_init(dev->max_sessions, bm, bmsize); + if (dev->sess_bm == NULL) { + IOCPT_PRINT(ERR, "Cannot initialize bitmap"); + err = -EFAULT; + goto err_free_bm; + } + for (i = 0; i < dev->max_sessions; i++) + rte_bitmap_set(dev->sess_bm, i); + + return 0; + +err_free_bm: + rte_free(bm); +err_free_dmazone: + rte_memzone_free(dev->info_z); + dev->info_z = NULL; + dev->info = NULL; + dev->info_pa = 0; +err_free_adminq: + iocpt_adminq_free(dev->adminq); + dev->adminq = NULL; +err_out: + return err; +} + +static int +iocpt_init(struct iocpt_dev *dev) +{ + int err; + + memset(&dev->stats_base, 0, sizeof(dev->stats_base)); + + /* Uses dev_cmds */ + err = iocpt_dev_init(dev, dev->info_pa); + if (err != 0) + return err; + + err = iocpt_adminq_init(dev); + if (err != 0) + return err; + + dev->state |= IOCPT_DEV_F_INITED; + + return 0; +} + +void +iocpt_configure(struct iocpt_dev *dev) +{ + RTE_SET_USED(dev); +} + +void +iocpt_deinit(struct iocpt_dev *dev) +{ + IOCPT_PRINT_CALL(); + + if (!(dev->state & IOCPT_DEV_F_INITED)) + return; + + iocpt_adminq_deinit(dev); + + dev->state &= ~IOCPT_DEV_F_INITED; +} + +static void +iocpt_free_objs(struct iocpt_dev *dev) +{ + IOCPT_PRINT_CALL(); + + if (dev->sess_bm != NULL) { + rte_bitmap_free(dev->sess_bm); + rte_free(dev->sess_bm); + dev->sess_bm = NULL; + } + + if (dev->adminq != NULL) { + iocpt_adminq_free(dev->adminq); + dev->adminq = NULL; + } + + if (dev->info != NULL) { + rte_memzone_free(dev->info_z); + dev->info_z = NULL; + dev->info = NULL; + dev->info_pa = 0; + } +} + +static int +iocpt_devargs(struct rte_devargs *devargs, struct iocpt_dev *dev) +{ + RTE_SET_USED(devargs); + RTE_SET_USED(dev); + + return 0; +} + +int +iocpt_probe(void *bus_dev, struct rte_device *rte_dev, + struct iocpt_dev_bars *bars, const struct iocpt_dev_intf *intf, + uint8_t driver_id, uint8_t socket_id) +{ + struct rte_cryptodev_pmd_init_params init_params = { + "iocpt", + sizeof(struct iocpt_dev), + socket_id, + RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS + }; + struct rte_cryptodev *cdev; + struct iocpt_dev *dev; + uint32_t i, sig; + int err; + + /* Check structs (trigger error at compilation time) */ + iocpt_struct_size_checks(); + + /* Multi-process not supported */ + if (rte_eal_process_type() != RTE_PROC_PRIMARY) { + err = -EPERM; + goto err; + } + + cdev = rte_cryptodev_pmd_create(rte_dev->name, rte_dev, &init_params); + if (cdev == NULL) { + IOCPT_PRINT(ERR, "OOM"); + err = -ENOMEM; + goto err; + } + + dev = cdev->data->dev_private; + dev->crypto_dev = cdev; + dev->bus_dev = bus_dev; + dev->intf = intf; + dev->driver_id = driver_id; + dev->socket_id = socket_id; + + for (i = 0; i < bars->num_bars; i++) { + struct ionic_dev_bar *bar = &bars->bar[i]; + + IOCPT_PRINT(DEBUG, + "bar[%u] = { .va = %p, .pa = %#jx, .len = %lu }", + i, bar->vaddr, bar->bus_addr, bar->len); + if (bar->vaddr == NULL) { + IOCPT_PRINT(ERR, "Null bar found, aborting"); + err = -EFAULT; + goto err_destroy_crypto_dev; + } + + dev->bars.bar[i].vaddr = bar->vaddr; + dev->bars.bar[i].bus_addr = bar->bus_addr; + dev->bars.bar[i].len = bar->len; + } + dev->bars.num_bars = bars->num_bars; + + err = iocpt_devargs(rte_dev->devargs, dev); + if (err != 0) { + IOCPT_PRINT(ERR, "Cannot parse device arguments"); + goto err_destroy_crypto_dev; + } + + err = iocpt_setup_bars(dev); + if (err != 0) { + IOCPT_PRINT(ERR, "Cannot setup BARs: %d, aborting", err); + goto err_destroy_crypto_dev; + } + + sig = ioread32(&dev->dev_info->signature); + if (sig != IOCPT_DEV_INFO_SIGNATURE) { + IOCPT_PRINT(ERR, "Incompatible firmware signature %#x", sig); + err = -EFAULT; + goto err_destroy_crypto_dev; + } + + for (i = 0; i < IOCPT_FWVERS_BUFLEN; i++) + dev->fw_version[i] = ioread8(&dev->dev_info->fw_version[i]); + dev->fw_version[IOCPT_FWVERS_BUFLEN - 1] = '\0'; + IOCPT_PRINT(DEBUG, "%s firmware: %s", dev->name, dev->fw_version); + + err = iocpt_dev_identify(dev); + if (err != 0) { + IOCPT_PRINT(ERR, "Cannot identify device: %d, aborting", + err); + goto err_destroy_crypto_dev; + } + + err = iocpt_alloc_objs(dev); + if (err != 0) { + IOCPT_PRINT(ERR, "Cannot alloc device objects: %d", err); + goto err_destroy_crypto_dev; + } + + err = iocpt_init(dev); + if (err != 0) { + IOCPT_PRINT(ERR, "Cannot init device: %d, aborting", err); + goto err_free_objs; + } + + return 0; + +err_free_objs: + iocpt_free_objs(dev); +err_destroy_crypto_dev: + rte_cryptodev_pmd_destroy(cdev); +err: + return err; +} + +int +iocpt_remove(struct rte_device *rte_dev) +{ + struct rte_cryptodev *cdev; + struct iocpt_dev *dev; + + cdev = rte_cryptodev_pmd_get_named_dev(rte_dev->name); + if (cdev == NULL) { + IOCPT_PRINT(DEBUG, "Cannot find device %s", rte_dev->name); + return -ENODEV; + } + + dev = cdev->data->dev_private; + + iocpt_deinit(dev); + + iocpt_dev_reset(dev); + + iocpt_free_objs(dev); + + rte_cryptodev_pmd_destroy(cdev); + + return 0; +} + RTE_LOG_REGISTER_DEFAULT(iocpt_logtype, NOTICE); diff --git a/drivers/crypto/ionic/ionic_crypto_vdev.c b/drivers/crypto/ionic/ionic_crypto_vdev.c new file mode 100644 index 0000000000..d15acf660a --- /dev/null +++ b/drivers/crypto/ionic/ionic_crypto_vdev.c @@ -0,0 +1,128 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021-2024 Advanced Micro Devices, Inc. + */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "ionic_crypto.h" + +#define IOCPT_VDEV_DEV_BAR 0 +#define IOCPT_VDEV_INTR_CTL_BAR 1 +#define IOCPT_VDEV_INTR_CFG_BAR 2 +#define IOCPT_VDEV_DB_BAR 3 +#define IOCPT_VDEV_BARS_MAX 4 + +#define IOCPT_VDEV_DEV_INFO_REGS_OFFSET 0x0000 +#define IOCPT_VDEV_DEV_CMD_REGS_OFFSET 0x0800 + +#define IOCPT_VDEV_FW_WAIT_US 1000 /* 1ms */ +#define IOCPT_VDEV_FW_WAIT_MAX 5000 /* 5s */ + +static int +iocpt_vdev_setup_bars(struct iocpt_dev *dev) +{ + struct iocpt_dev_bars *bars = &dev->bars; + uint8_t *bar0_base; + uint32_t fw_waits = 0; + uint8_t fw; + + IOCPT_PRINT_CALL(); + + /* BAR0: dev_cmd */ + bar0_base = bars->bar[IOCPT_VDEV_DEV_BAR].vaddr; + dev->dev_info = (union iocpt_dev_info_regs *) + &bar0_base[IOCPT_VDEV_DEV_INFO_REGS_OFFSET]; + dev->dev_cmd = (union iocpt_dev_cmd_regs *) + &bar0_base[IOCPT_VDEV_DEV_CMD_REGS_OFFSET]; + + /* BAR1: interrupts */ + dev->intr_ctrl = (void *)bars->bar[IOCPT_VDEV_INTR_CTL_BAR].vaddr; + + /* BAR3: doorbells */ + dev->db_pages = (void *)bars->bar[IOCPT_VDEV_DB_BAR].vaddr; + + /* Wait for the FW to indicate readiness */ + while (1) { + fw = ioread8(&dev->dev_info->fw_status); + if ((fw & IOCPT_FW_STS_F_RUNNING) != 0) + break; + + if (fw_waits > IOCPT_VDEV_FW_WAIT_MAX) { + IOCPT_PRINT(ERR, "Firmware readiness bit not set"); + return -ETIMEDOUT; + } + + fw_waits++; + rte_delay_us_block(IOCPT_VDEV_FW_WAIT_US); + } + IOCPT_PRINT(DEBUG, "Firmware ready (%u waits)", fw_waits); + + dev->name = rte_vdev_device_name(dev->bus_dev); + + return 0; +} + +static void +iocpt_vdev_unmap_bars(struct iocpt_dev *dev) +{ + struct iocpt_dev_bars *bars = &dev->bars; + uint32_t i; + + for (i = 0; i < IOCPT_VDEV_BARS_MAX; i++) + ionic_uio_rel_rsrc(dev->name, i, &bars->bar[i]); +} + +static uint8_t iocpt_vdev_driver_id; +static const struct iocpt_dev_intf iocpt_vdev_intf = { + .setup_bars = iocpt_vdev_setup_bars, + .unmap_bars = iocpt_vdev_unmap_bars, +}; + +static int +iocpt_vdev_probe(struct rte_vdev_device *vdev) +{ + struct iocpt_dev_bars bars = {}; + const char *name = rte_vdev_device_name(vdev); + unsigned int i; + + IOCPT_PRINT(NOTICE, "Initializing device %s%s", name, + rte_eal_process_type() == RTE_PROC_SECONDARY ? + " [SECONDARY]" : ""); + + ionic_uio_scan_mcrypt_devices(); + + for (i = 0; i < IOCPT_VDEV_BARS_MAX; i++) + ionic_uio_get_rsrc(name, i, &bars.bar[i]); + + bars.num_bars = IOCPT_VDEV_BARS_MAX; + + return iocpt_probe((void *)vdev, &vdev->device, + &bars, &iocpt_vdev_intf, + iocpt_vdev_driver_id, rte_socket_id()); +} + +static int +iocpt_vdev_remove(struct rte_vdev_device *vdev) +{ + return iocpt_remove(&vdev->device); +} + +static struct rte_vdev_driver rte_vdev_iocpt_pmd = { + .probe = iocpt_vdev_probe, + .remove = iocpt_vdev_remove, +}; + +static struct cryptodev_driver rte_vdev_iocpt_drv; + +RTE_PMD_REGISTER_VDEV(crypto_ionic, rte_vdev_iocpt_pmd); +RTE_PMD_REGISTER_CRYPTO_DRIVER(rte_vdev_iocpt_drv, rte_vdev_iocpt_pmd.driver, + iocpt_vdev_driver_id); diff --git a/drivers/crypto/ionic/meson.build b/drivers/crypto/ionic/meson.build index 6eaef41196..a6e0a1d415 100644 --- a/drivers/crypto/ionic/meson.build +++ b/drivers/crypto/ionic/meson.build @@ -1,11 +1,13 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2021-2024 Advanced Micro Devices, Inc. +deps += ['bus_vdev'] deps += ['common_ionic'] sources = files( 'ionic_crypto_cmds.c', 'ionic_crypto_main.c', + 'ionic_crypto_vdev.c', ) name = 'ionic_crypto'