From patchwork Fri Apr 19 19:53:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Boyer X-Patchwork-Id: 139554 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3206543EB5; Fri, 19 Apr 2024 21:53:57 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DD76640A7A; Fri, 19 Apr 2024 21:53:51 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2061.outbound.protection.outlook.com [40.107.236.61]) by mails.dpdk.org (Postfix) with ESMTP id 660C940A6D for ; Fri, 19 Apr 2024 21:53:49 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hJG/ryhXY0DZcP2MV4Vr7mO2069cXCBGCAZpFKerXgCQGNyi9Oc76JH+v3JPF6/gsCQg4fdcDpX29bXIUjhs6aY/qf9n1X+jlsO1GjjeESjbrDOOdu9D8/If+fA7VBfImf5bcrxLAUnpWTYSIWb5hwvBudirCLMpXBTM0s/NEzR70u9LiCjbpLd2vCmYoPMNSHOEzYgSrlxovkhsUBvR06vknRsxPuMc85rHZcU88XkG9foRRx6H7r3ieZ5CtV4an9m+RifqFcMSYwCCJnkylZkbhSVXb92Wlt6A+1BJFM0bzk9IAo7sCrfTjrEpn2/EOIjfkoO0mR2ojr9qyNxN1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TsjSooobmVsTZ3rPwl1nLqX/InbUALcS5+58hLgizY8=; b=F15QbsaX18LMHaNvdBI2L+02SwMT+4QjfdsWywoy20A546JA+WQLz8KQcIxbH10x6m+ISn/wujVm9oDfuOL5Q7UmoMMW0r7bM91xKKLOpUoCO/0r+21OUkZn2ZUSxKRjLTiEleqemSYLlZ7gZAZZFpa/rL350j906VFC5NXFHtfEvY7QuD756Mkdz19aFD6mvG6jIaropjMoNcBOIEB5ZE9ozfDKI3yKJjgY5044/VfrqATq5BDiXVgXcM1QRMXDRpZwQ5dQCdnj4ap9Fgjc5/bg5r2xkpiKYGS9zQjMQVQZ32ksG6d/xcudSGD7hogjg8o8+WyApSXyWZwBHJU6XA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=dpdk.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TsjSooobmVsTZ3rPwl1nLqX/InbUALcS5+58hLgizY8=; b=p4Zu10FNFCoPYbj+XUMZy4OXcl0oF7ReUB21nQI2jzyvOR5ILG4PkX7exPhHLtl57NP0sLttTtBhRT8euZEw0J7Imc+RFvUHdvtbhe1Fs9MZKWDsNzaJDDz3BCrE5Cs31qZMZj2VoaVhdsG2yMzwvRRsH7cqoDG0RwB/oXKwjXs= Received: from SA9PR13CA0144.namprd13.prod.outlook.com (2603:10b6:806:27::29) by SN7PR12MB8131.namprd12.prod.outlook.com (2603:10b6:806:32d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.39; Fri, 19 Apr 2024 19:53:45 +0000 Received: from SA2PEPF0000150B.namprd04.prod.outlook.com (2603:10b6:806:27:cafe::1) by SA9PR13CA0144.outlook.office365.com (2603:10b6:806:27::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.11 via Frontend Transport; Fri, 19 Apr 2024 19:53:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SA2PEPF0000150B.mail.protection.outlook.com (10.167.242.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7452.22 via Frontend Transport; Fri, 19 Apr 2024 19:53:44 +0000 Received: from driver-dev1.pensando.io (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 19 Apr 2024 14:53:44 -0500 From: Andrew Boyer To: CC: Akhil Goyal , Andrew Boyer Subject: [PATCH 2/6] crypto/ionic: add device and admin command handlers Date: Fri, 19 Apr 2024 12:53:06 -0700 Message-ID: <20240419195310.21432-3-andrew.boyer@amd.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240419195310.21432-1-andrew.boyer@amd.com> References: <20240419195310.21432-1-andrew.boyer@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF0000150B:EE_|SN7PR12MB8131:EE_ X-MS-Office365-Filtering-Correlation-Id: 7c1d268b-6255-4ae0-369b-08dc60aa6be0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /jg6v/SG8W6+ImCK0kqVDvyrmRsJwYsKfE46eSvMXEXbJNExtUoMWtBzZ1LdcjFOxwd93RrWuOwWyuPXn/AVmf6Fxb1HnKQ5SsAtJtP68IZhS6CPI+vU2Qk+xPO62NhvIzM/I1hdFJVv5vGpjs1DahjdIyn2hHgCVQXX2DlIrTZbYCUakAiT/1tjDRXS7KVNzut8m54IRzCRabsoQjqDoLncFe98Mt2D3xkcXvQrqLOpnAOXUpXgUz+vkjJuitsyk9oNptT47kNgr5D1VQ4BG2PMWywdHSb6G17atTYixX++pcKlnSmcw9as1eOvzXrUl1turCdzCpeSAQmcwnMnQ1j9IL4jXmR9jaQuUv4rEh53HNgBaLdZ4IpJCPdJwawULnbwuxm7Ebt8UmQm8n1vgiJ+bH3KOcY0j4L2wXGg7McX2dgHB8ywlmZSqC4M1OeK2oiYAU5cdE8aLCckxgmhU67aBjDQHbDzq/sdUIXWMdxwKhbJOFPWsY+z3VKnKMncdnFgp6PLkhNhoIHTdDjUEGodwcjs5zsPAVhaRG9TpwzlRFqgQ0txAqSCM51rVouomvaGHJPubbI6dUDz//cYjCp//ZQrGM2ecxUatzXu/TXJ0B6JNBzh3/BspCoc44iBWHQa4c3SpiF2B3WmlMc8DKj9B2CXZg1584qUfqUPswc+6DOCG7Iz21IY6TIIk4/gBs6aXdIoK0upFEoqA08BgsSP/QaBLh/y1Qdw//OJXsbaVTztPCMhNdGgrz3Z2giGeN8QjtVAsmfWffJiw0WbGTWbPJyv4k5nQFIz1BH0hew05jyi96cTUOHa0+/k9D6Vr/X4Rpl9Kjgufahy/bQDMewXLjG5WK4phP3348MPh48aVnSBEWVTw6hb6CbgPO6d4M63K4embosz2SJhK+JG3+lsmRmco+on9uoF1xQgSQJLN6jKMhzDxywLIOdGmKGjwFD7nVV+VM+omoz6ppzLSEm5op70BOQSKf+4Lru7nfRVfA6zbYhDQdT+z+VUgYbo0YKFWRJMcPJzVKKNYKSZABpFmkn7zKK55tnL2kbYxdK/dTBVXVaUEbIU0rIOiha6JI5vbeSXZpq+wMSFv7Zgu9fEBoITsfZFS83UUEgvM2A+Px2ePaQqQCdLQyIHW6u4uRY1nNTulzYmxrNqFRHdTLjn3LEa5c6io9RWM4w1xBSAlkBoinr/z++/CYr2RxnPeTUfGyzvdPQy/RV2tY2pmSmSskIL6SKNJ10WW9ApnocPirlTm3sbbbuGGIPvmF8Hud7MqGcgCqbyR0BxB3jYOocKKPDE2Q+caXy0w4EXCUYqAT9OEFfVO93cKelJnw7APcLyCm0ODCmV+SYnGS/8qgJSDmWIxfVbueAsHOi4iDxya7mhDMq+KOEB/UGTLzV7 X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230031)(376005)(82310400014)(36860700004)(1800799015); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Apr 2024 19:53:44.9705 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7c1d268b-6255-4ae0-369b-08dc60aa6be0 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF0000150B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8131 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This defines the handlers used for device (register-based) and admin (adminq-based) commands. Signed-off-by: Andrew Boyer --- drivers/crypto/ionic/ionic_crypto.h | 210 ++++++++ drivers/crypto/ionic/ionic_crypto_cmds.c | 651 +++++++++++++++++++++++ drivers/crypto/ionic/ionic_crypto_main.c | 42 ++ drivers/crypto/ionic/meson.build | 12 + 4 files changed, 915 insertions(+) create mode 100644 drivers/crypto/ionic/ionic_crypto.h create mode 100644 drivers/crypto/ionic/ionic_crypto_cmds.c create mode 100644 drivers/crypto/ionic/ionic_crypto_main.c create mode 100644 drivers/crypto/ionic/meson.build diff --git a/drivers/crypto/ionic/ionic_crypto.h b/drivers/crypto/ionic/ionic_crypto.h new file mode 100644 index 0000000000..958e611337 --- /dev/null +++ b/drivers/crypto/ionic/ionic_crypto.h @@ -0,0 +1,210 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021-2024 Advanced Micro Devices, Inc. + */ + +#ifndef _IONIC_CRYPTO_H_ +#define _IONIC_CRYPTO_H_ + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "ionic_common.h" +#include "ionic_crypto_if.h" +#include "ionic_regs.h" + +#define IOCPT_ADMINQ_LENGTH 16 /* must be a power of two */ + +#define IOCPT_CRYPTOQ_WAIT 10 /* 1s */ + +extern int iocpt_logtype; +#define RTE_LOGTYPE_IOCPT iocpt_logtype + +#define IOCPT_PRINT(level, ...) \ + RTE_LOG_LINE_PREFIX(level, IOCPT, "%s(): ", __func__, __VA_ARGS__) + +#define IOCPT_PRINT_CALL() IOCPT_PRINT(DEBUG, " >>") + +struct iocpt_qtype_info { + uint8_t version; + uint8_t supported; + uint64_t features; + uint16_t desc_sz; + uint16_t comp_sz; + uint16_t sg_desc_sz; + uint16_t max_sg_elems; + uint16_t sg_desc_stride; +}; + +#define IOCPT_Q_F_INITED BIT(0) +#define IOCPT_Q_F_DEFERRED BIT(1) +#define IOCPT_Q_F_SG BIT(2) + +#define Q_NEXT_TO_POST(_q, _n) (((_q)->head_idx + (_n)) & ((_q)->size_mask)) +#define Q_NEXT_TO_SRVC(_q, _n) (((_q)->tail_idx + (_n)) & ((_q)->size_mask)) + +#define IOCPT_INFO_SZ(_q) ((_q)->num_segs * sizeof(void *)) +#define IOCPT_INFO_IDX(_q, _i) ((_i) * (_q)->num_segs) +#define IOCPT_INFO_PTR(_q, _i) (&(_q)->info[IOCPT_INFO_IDX((_q), _i)]) + +struct iocpt_queue { + uint16_t num_descs; + uint16_t num_segs; + uint16_t head_idx; + uint16_t tail_idx; + uint16_t size_mask; + uint8_t type; + uint8_t hw_type; + void *base; + void *sg_base; + struct ionic_doorbell __iomem *db; + void **info; + + uint32_t index; + uint32_t hw_index; + rte_iova_t base_pa; + rte_iova_t sg_base_pa; +}; + +struct iocpt_cq { + uint16_t tail_idx; + uint16_t num_descs; + uint16_t size_mask; + bool done_color; + void *base; + rte_iova_t base_pa; +}; + +#define IOCPT_COMMON_FIELDS \ + struct iocpt_queue q; \ + struct iocpt_cq cq; \ + struct iocpt_dev *dev; \ + const struct rte_memzone *base_z; \ + void *base; \ + rte_iova_t base_pa + +struct iocpt_common_q { + IOCPT_COMMON_FIELDS; +}; + +struct iocpt_admin_q { + IOCPT_COMMON_FIELDS; + + uint16_t flags; +}; + +#define IOCPT_DEV_F_INITED BIT(0) +#define IOCPT_DEV_F_UP BIT(1) +#define IOCPT_DEV_F_FW_RESET BIT(2) + +/* Combined dev / LIF object */ +struct iocpt_dev { + const char *name; + char fw_version[IOCPT_FWVERS_BUFLEN]; + struct iocpt_identity ident; + + void *bus_dev; + struct rte_cryptodev *crypto_dev; + + union iocpt_dev_info_regs __iomem *dev_info; + union iocpt_dev_cmd_regs __iomem *dev_cmd; + + struct ionic_doorbell __iomem *db_pages; + struct ionic_intr __iomem *intr_ctrl; + + uint32_t max_qps; + uint32_t max_sessions; + uint16_t state; + uint8_t driver_id; + uint8_t socket_id; + + rte_spinlock_t adminq_lock; + rte_spinlock_t adminq_service_lock; + + struct iocpt_admin_q *adminq; + + uint64_t features; + uint32_t hw_features; + + uint32_t info_sz; + struct iocpt_lif_info *info; + rte_iova_t info_pa; + const struct rte_memzone *info_z; + + struct iocpt_qtype_info qtype_info[IOCPT_QTYPE_MAX]; + uint8_t qtype_ver[IOCPT_QTYPE_MAX]; + + struct rte_cryptodev_stats stats_base; +}; + +/** iocpt_admin_ctx - Admin command context. + * @pending_work: Flag that indicates a completion. + * @cmd: Admin command (64B) to be copied to the queue. + * @comp: Admin completion (16B) copied from the queue. + */ +struct iocpt_admin_ctx { + bool pending_work; + union iocpt_adminq_cmd cmd; + union iocpt_adminq_comp comp; +}; + +int iocpt_dev_identify(struct iocpt_dev *dev); +int iocpt_dev_init(struct iocpt_dev *dev, rte_iova_t info_pa); +int iocpt_dev_adminq_init(struct iocpt_dev *dev); +void iocpt_dev_reset(struct iocpt_dev *dev); + +int iocpt_adminq_post_wait(struct iocpt_dev *dev, struct iocpt_admin_ctx *ctx); + +struct ionic_doorbell __iomem *iocpt_db_map(struct iocpt_dev *dev, + struct iocpt_queue *q); + +typedef bool (*iocpt_cq_cb)(struct iocpt_cq *cq, uint16_t cq_desc_index, + void *cb_arg); +uint32_t iocpt_cq_service(struct iocpt_cq *cq, uint32_t work_to_do, + iocpt_cq_cb cb, void *cb_arg); + +static inline uint16_t +iocpt_q_space_avail(struct iocpt_queue *q) +{ + uint16_t avail = q->tail_idx; + + if (q->head_idx >= avail) + avail += q->num_descs - q->head_idx - 1; + else + avail -= q->head_idx + 1; + + return avail; +} + +static inline void +iocpt_q_flush(struct iocpt_queue *q) +{ + uint64_t val = IONIC_DBELL_QID(q->hw_index) | q->head_idx; + +#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA) + /* On some devices the standard 'dmb' barrier is insufficient */ + asm volatile("dsb st" : : : "memory"); + rte_write64_relaxed(rte_cpu_to_le_64(val), q->db); +#else + rte_write64(rte_cpu_to_le_64(val), q->db); +#endif +} + +static inline bool +iocpt_is_embedded(void) +{ +#if defined(RTE_LIBRTE_IONIC_PMD_EMBEDDED) + return true; +#else + return false; +#endif +} + +#endif /* _IONIC_CRYPTO_H_ */ diff --git a/drivers/crypto/ionic/ionic_crypto_cmds.c b/drivers/crypto/ionic/ionic_crypto_cmds.c new file mode 100644 index 0000000000..44e6985eb1 --- /dev/null +++ b/drivers/crypto/ionic/ionic_crypto_cmds.c @@ -0,0 +1,651 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021-2024 Advanced Micro Devices, Inc. + */ + +#include + +#include +#include + +#include "ionic_crypto.h" + +/* queuetype support level */ +static const uint8_t iocpt_qtype_vers[IOCPT_QTYPE_MAX] = { + [IOCPT_QTYPE_ADMINQ] = 0, /* 0 = Base version */ + [IOCPT_QTYPE_NOTIFYQ] = 0, /* 0 = Base version */ + [IOCPT_QTYPE_CRYPTOQ] = 0, /* 0 = Base version */ +}; + +static const char * +iocpt_error_to_str(enum iocpt_status_code code) +{ + switch (code) { + case IOCPT_RC_SUCCESS: + return "IOCPT_RC_SUCCESS"; + case IOCPT_RC_EVERSION: + return "IOCPT_RC_EVERSION"; + case IOCPT_RC_EOPCODE: + return "IOCPT_RC_EOPCODE"; + case IOCPT_RC_EIO: + return "IOCPT_RC_EIO"; + case IOCPT_RC_EPERM: + return "IOCPT_RC_EPERM"; + case IOCPT_RC_EQID: + return "IOCPT_RC_EQID"; + case IOCPT_RC_EQTYPE: + return "IOCPT_RC_EQTYPE"; + case IOCPT_RC_ENOENT: + return "IOCPT_RC_ENOENT"; + case IOCPT_RC_EINTR: + return "IOCPT_RC_EINTR"; + case IOCPT_RC_EAGAIN: + return "IOCPT_RC_EAGAIN"; + case IOCPT_RC_ENOMEM: + return "IOCPT_RC_ENOMEM"; + case IOCPT_RC_EFAULT: + return "IOCPT_RC_EFAULT"; + case IOCPT_RC_EBUSY: + return "IOCPT_RC_EBUSY"; + case IOCPT_RC_EEXIST: + return "IOCPT_RC_EEXIST"; + case IOCPT_RC_EINVAL: + return "IOCPT_RC_EINVAL"; + case IOCPT_RC_ENOSPC: + return "IOCPT_RC_ENOSPC"; + case IOCPT_RC_ERANGE: + return "IOCPT_RC_ERANGE"; + case IOCPT_RC_BAD_ADDR: + return "IOCPT_RC_BAD_ADDR"; + case IOCPT_RC_DEV_CMD: + return "IOCPT_RC_DEV_CMD"; + case IOCPT_RC_ERROR: + return "IOCPT_RC_ERROR"; + default: + return "IOCPT_RC_UNKNOWN"; + } +} + +static const char * +iocpt_opcode_to_str(enum iocpt_cmd_opcode opcode) +{ + switch (opcode) { + case IOCPT_CMD_NOP: + return "IOCPT_CMD_NOP"; + case IOCPT_CMD_IDENTIFY: + return "IOCPT_CMD_IDENTIFY"; + case IOCPT_CMD_RESET: + return "IOCPT_CMD_RESET"; + case IOCPT_CMD_LIF_IDENTIFY: + return "IOCPT_CMD_LIF_IDENTIFY"; + case IOCPT_CMD_LIF_INIT: + return "IOCPT_CMD_LIF_INIT"; + case IOCPT_CMD_LIF_RESET: + return "IOCPT_CMD_LIF_RESET"; + case IOCPT_CMD_LIF_GETATTR: + return "IOCPT_CMD_LIF_GETATTR"; + case IOCPT_CMD_LIF_SETATTR: + return "IOCPT_CMD_LIF_SETATTR"; + case IOCPT_CMD_Q_IDENTIFY: + return "IOCPT_CMD_Q_IDENTIFY"; + case IOCPT_CMD_Q_INIT: + return "IOCPT_CMD_Q_INIT"; + case IOCPT_CMD_Q_CONTROL: + return "IOCPT_CMD_Q_CONTROL"; + case IOCPT_CMD_SESS_CONTROL: + return "IOCPT_CMD_SESS_CONTROL"; + default: + return "DEVCMD_UNKNOWN"; + } +} + +/* Dev_cmd Interface */ + +static void +iocpt_dev_cmd_go(struct iocpt_dev *dev, union iocpt_dev_cmd *cmd) +{ + uint32_t cmd_size = RTE_DIM(cmd->words); + uint32_t i; + + IOCPT_PRINT(DEBUG, "Sending %s (%d) via dev_cmd", + iocpt_opcode_to_str(cmd->cmd.opcode), cmd->cmd.opcode); + + for (i = 0; i < cmd_size; i++) + iowrite32(cmd->words[i], &dev->dev_cmd->cmd.words[i]); + + iowrite32(0, &dev->dev_cmd->done); + iowrite32(1, &dev->dev_cmd->doorbell); +} + +static int +iocpt_dev_cmd_wait(struct iocpt_dev *dev, unsigned long max_wait) +{ + unsigned long step_usec = IONIC_DEVCMD_CHECK_PERIOD_US; + unsigned long max_wait_usec = max_wait * 1000000L; + unsigned long elapsed_usec = 0; + int done; + + /* Wait for dev cmd to complete.. but no more than max_wait sec */ + + do { + done = ioread32(&dev->dev_cmd->done) & IONIC_DEV_CMD_DONE; + if (done != 0) { + IOCPT_PRINT(DEBUG, "DEVCMD %d done took %lu usecs", + ioread8(&dev->dev_cmd->cmd.cmd.opcode), + elapsed_usec); + return 0; + } + + rte_delay_us_block(step_usec); + + elapsed_usec += step_usec; + } while (elapsed_usec < max_wait_usec); + + IOCPT_PRINT(ERR, "DEVCMD %d timeout after %lu usecs", + ioread8(&dev->dev_cmd->cmd.cmd.opcode), elapsed_usec); + + return -ETIMEDOUT; +} + +static void +iocpt_dev_cmd_comp(struct iocpt_dev *dev, void *mem) +{ + union iocpt_dev_cmd_comp *comp = mem; + uint32_t comp_size = RTE_DIM(comp->words); + uint32_t i; + + for (i = 0; i < comp_size; i++) + comp->words[i] = ioread32(&dev->dev_cmd->comp.words[i]); +} + + +static int +iocpt_dev_cmd_wait_check(struct iocpt_dev *dev, unsigned long max_wait) +{ + uint8_t status; + int err; + + err = iocpt_dev_cmd_wait(dev, max_wait); + if (err == 0) { + status = ioread8(&dev->dev_cmd->comp.comp.status); + if (status == IOCPT_RC_EAGAIN) + err = -EAGAIN; + else if (status != 0) + err = -EIO; + } + + IOCPT_PRINT(DEBUG, "dev_cmd returned %d", err); + return err; +} + +/* Dev_cmds */ + +static void +iocpt_dev_cmd_reset(struct iocpt_dev *dev) +{ + union iocpt_dev_cmd cmd = { + .reset.opcode = IOCPT_CMD_RESET, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_lif_identify(struct iocpt_dev *dev, uint8_t ver) +{ + union iocpt_dev_cmd cmd = { + .lif_identify.opcode = IOCPT_CMD_LIF_IDENTIFY, + .lif_identify.type = IOCPT_LIF_TYPE_DEFAULT, + .lif_identify.ver = ver, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_lif_init(struct iocpt_dev *dev, rte_iova_t info_pa) +{ + union iocpt_dev_cmd cmd = { + .lif_init.opcode = IOCPT_CMD_LIF_INIT, + .lif_init.type = IOCPT_LIF_TYPE_DEFAULT, + .lif_init.info_pa = info_pa, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_lif_reset(struct iocpt_dev *dev) +{ + union iocpt_dev_cmd cmd = { + .lif_reset.opcode = IOCPT_CMD_LIF_RESET, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_queue_identify(struct iocpt_dev *dev, + uint8_t qtype, uint8_t qver) +{ + union iocpt_dev_cmd cmd = { + .q_identify.opcode = IOCPT_CMD_Q_IDENTIFY, + .q_identify.type = qtype, + .q_identify.ver = qver, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_adminq_init(struct iocpt_dev *dev) +{ + struct iocpt_queue *q = &dev->adminq->q; + struct iocpt_cq *cq = &dev->adminq->cq; + + union iocpt_dev_cmd cmd = { + .q_init.opcode = IOCPT_CMD_Q_INIT, + .q_init.type = q->type, + .q_init.ver = dev->qtype_info[q->type].version, + .q_init.index = rte_cpu_to_le_32(q->index), + .q_init.flags = rte_cpu_to_le_16(IOCPT_QINIT_F_ENA), + .q_init.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE), + .q_init.ring_size = rte_log2_u32(q->num_descs), + .q_init.ring_base = rte_cpu_to_le_64(q->base_pa), + .q_init.cq_ring_base = rte_cpu_to_le_64(cq->base_pa), + }; + + IOCPT_PRINT(DEBUG, "adminq.q_init.ver %u", cmd.q_init.ver); + + iocpt_dev_cmd_go(dev, &cmd); +} + +/* Dev_cmd consumers */ + +static void +iocpt_queue_identify(struct iocpt_dev *dev) +{ + union iocpt_q_identity *q_ident = &dev->ident.q; + uint32_t q_words = RTE_DIM(q_ident->words); + uint32_t cmd_words = RTE_DIM(dev->dev_cmd->data); + uint32_t i, nwords, qtype; + int err; + + for (qtype = 0; qtype < RTE_DIM(iocpt_qtype_vers); qtype++) { + struct iocpt_qtype_info *qti = &dev->qtype_info[qtype]; + + /* Filter out the types this driver knows about */ + switch (qtype) { + case IOCPT_QTYPE_ADMINQ: + case IOCPT_QTYPE_NOTIFYQ: + case IOCPT_QTYPE_CRYPTOQ: + break; + default: + continue; + } + + memset(qti, 0, sizeof(*qti)); + + if (iocpt_is_embedded()) { + /* When embedded, FW will always match the driver */ + qti->version = iocpt_qtype_vers[qtype]; + continue; + } + + /* On the host, query the FW for info */ + iocpt_dev_cmd_queue_identify(dev, + qtype, iocpt_qtype_vers[qtype]); + err = iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + if (err == -EINVAL) { + IOCPT_PRINT(ERR, "qtype %d not supported", qtype); + continue; + } else if (err == -EIO) { + IOCPT_PRINT(ERR, "q_ident failed, older FW"); + return; + } else if (err != 0) { + IOCPT_PRINT(ERR, "q_ident failed, qtype %d: %d", + qtype, err); + return; + } + + nwords = RTE_MIN(q_words, cmd_words); + for (i = 0; i < nwords; i++) + q_ident->words[i] = ioread32(&dev->dev_cmd->data[i]); + + qti->version = q_ident->version; + qti->supported = q_ident->supported; + qti->features = rte_le_to_cpu_64(q_ident->features); + qti->desc_sz = rte_le_to_cpu_16(q_ident->desc_sz); + qti->comp_sz = rte_le_to_cpu_16(q_ident->comp_sz); + qti->sg_desc_sz = rte_le_to_cpu_16(q_ident->sg_desc_sz); + qti->max_sg_elems = rte_le_to_cpu_16(q_ident->max_sg_elems); + qti->sg_desc_stride = + rte_le_to_cpu_16(q_ident->sg_desc_stride); + + IOCPT_PRINT(DEBUG, " qtype[%d].version = %d", + qtype, qti->version); + IOCPT_PRINT(DEBUG, " qtype[%d].supported = %#x", + qtype, qti->supported); + IOCPT_PRINT(DEBUG, " qtype[%d].features = %#jx", + qtype, qti->features); + IOCPT_PRINT(DEBUG, " qtype[%d].desc_sz = %d", + qtype, qti->desc_sz); + IOCPT_PRINT(DEBUG, " qtype[%d].comp_sz = %d", + qtype, qti->comp_sz); + IOCPT_PRINT(DEBUG, " qtype[%d].sg_desc_sz = %d", + qtype, qti->sg_desc_sz); + IOCPT_PRINT(DEBUG, " qtype[%d].max_sg_elems = %d", + qtype, qti->max_sg_elems); + IOCPT_PRINT(DEBUG, " qtype[%d].sg_desc_stride = %d", + qtype, qti->sg_desc_stride); + } +} + +int +iocpt_dev_identify(struct iocpt_dev *dev) +{ + union iocpt_lif_identity *ident = &dev->ident.lif; + union iocpt_lif_config *cfg = &ident->config; + uint64_t features; + uint32_t cmd_size = RTE_DIM(dev->dev_cmd->data); + uint32_t dev_size = RTE_DIM(ident->words); + uint32_t i, nwords; + int err; + + memset(ident, 0, sizeof(*ident)); + + iocpt_dev_cmd_lif_identify(dev, IOCPT_IDENTITY_VERSION_1); + err = iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + if (err != 0) + return err; + + nwords = RTE_MIN(dev_size, cmd_size); + for (i = 0; i < nwords; i++) + ident->words[i] = ioread32(&dev->dev_cmd->data[i]); + + dev->max_qps = + rte_le_to_cpu_32(cfg->queue_count[IOCPT_QTYPE_CRYPTOQ]); + dev->max_sessions = + rte_le_to_cpu_32(ident->max_nb_sessions); + + features = rte_le_to_cpu_64(ident->features); + dev->features = RTE_CRYPTODEV_FF_HW_ACCELERATED; + if (features & IOCPT_HW_SYM) + dev->features |= RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO; + if (features & IOCPT_HW_ASYM) + dev->features |= RTE_CRYPTODEV_FF_ASYMMETRIC_CRYPTO; + if (features & IOCPT_HW_CHAIN) + dev->features |= RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING; + if (features & IOCPT_HW_IP) + dev->features |= RTE_CRYPTODEV_FF_IN_PLACE_SGL; + if (features & IOCPT_HW_OOP) { + dev->features |= RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT; + dev->features |= RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT; + dev->features |= RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; + dev->features |= RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT; + } + + IOCPT_PRINT(INFO, "crypto.features %#jx", + rte_le_to_cpu_64(ident->features)); + IOCPT_PRINT(INFO, "crypto.features_active %#jx", + rte_le_to_cpu_64(cfg->features)); + IOCPT_PRINT(INFO, "crypto.queue_count[IOCPT_QTYPE_ADMINQ] %#x", + rte_le_to_cpu_32(cfg->queue_count[IOCPT_QTYPE_ADMINQ])); + IOCPT_PRINT(INFO, "crypto.queue_count[IOCPT_QTYPE_NOTIFYQ] %#x", + rte_le_to_cpu_32(cfg->queue_count[IOCPT_QTYPE_NOTIFYQ])); + IOCPT_PRINT(INFO, "crypto.queue_count[IOCPT_QTYPE_CRYPTOQ] %#x", + rte_le_to_cpu_32(cfg->queue_count[IOCPT_QTYPE_CRYPTOQ])); + IOCPT_PRINT(INFO, "crypto.max_sessions %u", + rte_le_to_cpu_32(ident->max_nb_sessions)); + + iocpt_queue_identify(dev); + + return 0; +} + +int +iocpt_dev_init(struct iocpt_dev *dev, rte_iova_t info_pa) +{ + uint32_t retries = 5; + int err; + +retry_lif_init: + iocpt_dev_cmd_lif_init(dev, info_pa); + + err = iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + if (err == -EAGAIN && retries > 0) { + retries--; + rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US); + goto retry_lif_init; + } + + return err; +} + +void +iocpt_dev_reset(struct iocpt_dev *dev) +{ + iocpt_dev_cmd_lif_reset(dev); + (void)iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + + iocpt_dev_cmd_reset(dev); + (void)iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); +} + +int +iocpt_dev_adminq_init(struct iocpt_dev *dev) +{ + struct iocpt_queue *q = &dev->adminq->q; + struct iocpt_q_init_comp comp; + uint32_t retries = 5; + int err; + +retry_adminq_init: + iocpt_dev_cmd_adminq_init(dev); + + err = iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + if (err == -EAGAIN && retries > 0) { + retries--; + rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US); + goto retry_adminq_init; + } + if (err != 0) + return err; + + iocpt_dev_cmd_comp(dev, &comp); + + q->hw_type = comp.hw_type; + q->hw_index = rte_le_to_cpu_32(comp.hw_index); + q->db = iocpt_db_map(dev, q); + + IOCPT_PRINT(DEBUG, "adminq->hw_type %d", q->hw_type); + IOCPT_PRINT(DEBUG, "adminq->hw_index %d", q->hw_index); + IOCPT_PRINT(DEBUG, "adminq->db %p", q->db); + + dev->adminq->flags |= IOCPT_Q_F_INITED; + + return 0; +} + +/* Admin_cmd interface */ + +static bool +iocpt_adminq_service(struct iocpt_cq *cq, uint16_t cq_desc_index, + void *cb_arg __rte_unused) +{ + struct iocpt_admin_comp *cq_desc_base = cq->base; + struct iocpt_admin_comp *cq_desc = &cq_desc_base[cq_desc_index]; + struct iocpt_admin_q *adminq = + container_of(cq, struct iocpt_admin_q, cq); + struct iocpt_queue *q = &adminq->q; + struct iocpt_admin_ctx *ctx; + uint16_t curr_q_tail_idx; + uint16_t stop_index; + void **info; + + if (!iocpt_color_match(cq_desc->color, cq->done_color)) + return false; + + stop_index = rte_le_to_cpu_16(cq_desc->comp_index); + + do { + info = IOCPT_INFO_PTR(q, q->tail_idx); + + ctx = info[0]; + if (ctx != NULL) { + memcpy(&ctx->comp, cq_desc, sizeof(*cq_desc)); + ctx->pending_work = false; /* done */ + } + + curr_q_tail_idx = q->tail_idx; + q->tail_idx = Q_NEXT_TO_SRVC(q, 1); + } while (curr_q_tail_idx != stop_index); + + return true; +} + +/** iocpt_adminq_post - Post an admin command. + * @dev: Handle to dev. + * @cmd_ctx: Api admin command context. + * + * Return: zero or negative error status. + */ +static int +iocpt_adminq_post(struct iocpt_dev *dev, struct iocpt_admin_ctx *ctx) +{ + struct iocpt_queue *q = &dev->adminq->q; + struct iocpt_admin_cmd *q_desc_base = q->base; + struct iocpt_admin_cmd *q_desc; + void **info; + int err = 0; + + rte_spinlock_lock(&dev->adminq_lock); + + if (iocpt_q_space_avail(q) < 1) { + err = -ENOSPC; + goto err_out; + } + + q_desc = &q_desc_base[q->head_idx]; + + memcpy(q_desc, &ctx->cmd, sizeof(ctx->cmd)); + + info = IOCPT_INFO_PTR(q, q->head_idx); + info[0] = ctx; + + q->head_idx = Q_NEXT_TO_POST(q, 1); + + /* Ring doorbell */ + iocpt_q_flush(q); + +err_out: + rte_spinlock_unlock(&dev->adminq_lock); + + return err; +} + +static int +iocpt_adminq_wait_for_completion(struct iocpt_dev *dev, + struct iocpt_admin_ctx *ctx, unsigned long max_wait) +{ + struct iocpt_queue *q = &dev->adminq->q; + unsigned long step_usec = IONIC_DEVCMD_CHECK_PERIOD_US; + unsigned long step_deadline; + unsigned long max_wait_usec = max_wait * 1000000L; + unsigned long elapsed_usec = 0; + int budget = 8; + uint16_t idx; + void **info; + + step_deadline = IONIC_ADMINQ_WDOG_MS * 1000 / step_usec; + + while (ctx->pending_work && elapsed_usec < max_wait_usec) { + /* + * Locking here as adminq is served inline and could be + * called from multiple places + */ + rte_spinlock_lock(&dev->adminq_service_lock); + + iocpt_cq_service(&dev->adminq->cq, budget, + iocpt_adminq_service, NULL); + + /* + * Ring the doorbell again if work is pending after step_usec. + */ + if (ctx->pending_work && !step_deadline) { + step_deadline = IONIC_ADMINQ_WDOG_MS * + 1000 / step_usec; + + rte_spinlock_lock(&dev->adminq_lock); + idx = Q_NEXT_TO_POST(q, -1); + info = IOCPT_INFO_PTR(q, idx); + if (info[0] == ctx) + iocpt_q_flush(q); + rte_spinlock_unlock(&dev->adminq_lock); + } + + rte_spinlock_unlock(&dev->adminq_service_lock); + + rte_delay_us_block(step_usec); + elapsed_usec += step_usec; + step_deadline--; + } + + return (!ctx->pending_work); +} + +static int +iocpt_adminq_check_err(struct iocpt_admin_ctx *ctx, bool timeout) +{ + const char *name; + const char *status; + + name = iocpt_opcode_to_str(ctx->cmd.cmd.opcode); + + if (ctx->comp.comp.status == IOCPT_RC_EAGAIN) { + IOCPT_PRINT(DEBUG, "%s (%d) returned EAGAIN (%d)", + name, ctx->cmd.cmd.opcode, + ctx->comp.comp.status); + return -EAGAIN; + } + if (ctx->comp.comp.status != 0 || timeout) { + status = iocpt_error_to_str(ctx->comp.comp.status); + IOCPT_PRINT(ERR, "%s (%d) failed: %s (%d)", + name, + ctx->cmd.cmd.opcode, + timeout ? "TIMEOUT" : status, + timeout ? -1 : ctx->comp.comp.status); + return -EIO; + } + + if (ctx->cmd.cmd.opcode != IOCPT_CMD_SESS_CONTROL) { + IOCPT_PRINT(DEBUG, "%s (%d) succeeded", + name, ctx->cmd.cmd.opcode); + } + + return 0; +} + +int +iocpt_adminq_post_wait(struct iocpt_dev *dev, struct iocpt_admin_ctx *ctx) +{ + bool done; + int err; + + if (ctx->cmd.cmd.opcode != IOCPT_CMD_SESS_CONTROL) { + IOCPT_PRINT(DEBUG, "Sending %s (%d) via the admin queue", + iocpt_opcode_to_str(ctx->cmd.cmd.opcode), + ctx->cmd.cmd.opcode); + } + + err = iocpt_adminq_post(dev, ctx); + if (err != 0) { + IOCPT_PRINT(ERR, "Failure posting %d to the admin queue (%d)", + ctx->cmd.cmd.opcode, err); + return err; + } + + done = iocpt_adminq_wait_for_completion(dev, ctx, + IONIC_DEVCMD_TIMEOUT); + + return iocpt_adminq_check_err(ctx, !done /* timed out */); +} diff --git a/drivers/crypto/ionic/ionic_crypto_main.c b/drivers/crypto/ionic/ionic_crypto_main.c new file mode 100644 index 0000000000..7b26080bd1 --- /dev/null +++ b/drivers/crypto/ionic/ionic_crypto_main.c @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021-2024 Advanced Micro Devices, Inc. + */ + +#include + +#include +#include +#include + +#include "ionic_crypto.h" + +int iocpt_logtype; + +uint32_t +iocpt_cq_service(struct iocpt_cq *cq, uint32_t work_to_do, + iocpt_cq_cb cb, void *cb_arg) +{ + uint32_t work_done = 0; + + if (work_to_do == 0) + return 0; + + while (cb(cq, cq->tail_idx, cb_arg)) { + cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1); + if (cq->tail_idx == 0) + cq->done_color = !cq->done_color; + + if (++work_done == work_to_do) + break; + } + + return work_done; +} + +struct ionic_doorbell * +iocpt_db_map(struct iocpt_dev *dev, struct iocpt_queue *q) +{ + return dev->db_pages + q->hw_type; +} + +RTE_LOG_REGISTER_DEFAULT(iocpt_logtype, NOTICE); diff --git a/drivers/crypto/ionic/meson.build b/drivers/crypto/ionic/meson.build new file mode 100644 index 0000000000..6eaef41196 --- /dev/null +++ b/drivers/crypto/ionic/meson.build @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021-2024 Advanced Micro Devices, Inc. + +deps += ['common_ionic'] + +sources = files( + 'ionic_crypto_cmds.c', + 'ionic_crypto_main.c', +) +name = 'ionic_crypto' + +includes += include_directories('../../common/ionic')