From patchwork Wed Aug 30 15:58:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130882 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 71C8A41FD0; Wed, 30 Aug 2023 17:59:40 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6398740298; Wed, 30 Aug 2023 17:59:36 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 00AA340284 for ; Wed, 30 Aug 2023 17:59:34 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7ow004368 for ; Wed, 30 Aug 2023 08:59:34 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=q89M3QsvGVgvDOKmusnZQ5LDdL8/EpR2K6IDed9T4Hc=; b=ig5VEwMp5JDlywgOS379lE77n+qBD20fqtF2sxyHwr/fUYLn8Q3WURz7yBaVBfZJCsX0 ylVFDMjwcw56h5pSVcwtsqwPwPnxS3cLnlGupOg9eZjFmTKnbXKW8T6HtP82v/CPJrLt dp988PmazvSQ1NUOqjigDMyRnp30sB7JlJcDYjkoh9BuCzB0uXFTJHzXgboVQ4wiPxeH DL2v1yW9kVxK+PLu2LE5CsLmR3sMb7SayUtdCvOqclC1ZagTTfn18FoMvSnqD1API2ES xd02AE2C99ujOmhSbSmgdfua+oaPs7gbhIHuBJ3ZZamXM1GyIpJno6HQDNzd7Z3mIvNg gw== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysst5-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:33 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:32 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:32 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id E26273F707E; Wed, 30 Aug 2023 08:59:31 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 01/34] ml/cnxk: drop support for register polling Date: Wed, 30 Aug 2023 08:58:51 -0700 Message-ID: <20230830155927.3566-2-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: nfGTpF1UFxG4mNjylRpf6y0H_6rLDNhy X-Proofpoint-ORIG-GUID: nfGTpF1UFxG4mNjylRpf6y0H_6rLDNhy X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Dropped support for device argument "poll_mem" for cnxk ML driver. Support to use registers for polling is removed and DDR addresses would be used for polling. Signed-off-by: Srikanth Yalavarthi --- Depends-on: series-29375 ("Spec changes to support multi I/O models") doc/guides/mldevs/cnxk.rst | 16 ----- drivers/ml/cnxk/cn10k_ml_dev.c | 36 +---------- drivers/ml/cnxk/cn10k_ml_dev.h | 13 +--- drivers/ml/cnxk/cn10k_ml_ops.c | 111 ++++----------------------------- drivers/ml/cnxk/cn10k_ml_ops.h | 6 -- 5 files changed, 18 insertions(+), 164 deletions(-) diff --git a/doc/guides/mldevs/cnxk.rst b/doc/guides/mldevs/cnxk.rst index b79bc540d95..1834b1f905a 100644 --- a/doc/guides/mldevs/cnxk.rst +++ b/doc/guides/mldevs/cnxk.rst @@ -180,22 +180,6 @@ Runtime Config Options in the fast path enqueue burst operation. -**Polling memory location** (default ``ddr``) - - ML cnxk driver provides the option to select the memory location to be used - for polling to check the inference request completion. - Driver supports using either the DDR address space (``ddr``) - or ML registers (``register``) as polling locations. - The parameter ``poll_mem`` is used to specify the poll location. - - For example:: - - -a 0000:00:10.0,poll_mem="register" - - With the above configuration, ML cnxk driver is configured to use ML registers - for polling in fastpath requests. - - Debugging Options ----------------- diff --git a/drivers/ml/cnxk/cn10k_ml_dev.c b/drivers/ml/cnxk/cn10k_ml_dev.c index 983138a7f23..e3c2badcef5 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.c +++ b/drivers/ml/cnxk/cn10k_ml_dev.c @@ -23,7 +23,6 @@ #define CN10K_ML_DEV_CACHE_MODEL_DATA "cache_model_data" #define CN10K_ML_OCM_ALLOC_MODE "ocm_alloc_mode" #define CN10K_ML_DEV_HW_QUEUE_LOCK "hw_queue_lock" -#define CN10K_ML_FW_POLL_MEM "poll_mem" #define CN10K_ML_OCM_PAGE_SIZE "ocm_page_size" #define CN10K_ML_FW_PATH_DEFAULT "/lib/firmware/mlip-fw.bin" @@ -32,7 +31,6 @@ #define CN10K_ML_DEV_CACHE_MODEL_DATA_DEFAULT 1 #define CN10K_ML_OCM_ALLOC_MODE_DEFAULT "lowest" #define CN10K_ML_DEV_HW_QUEUE_LOCK_DEFAULT 1 -#define CN10K_ML_FW_POLL_MEM_DEFAULT "ddr" #define CN10K_ML_OCM_PAGE_SIZE_DEFAULT 16384 /* ML firmware macros */ @@ -54,7 +52,6 @@ static const char *const valid_args[] = {CN10K_ML_FW_PATH, CN10K_ML_DEV_CACHE_MODEL_DATA, CN10K_ML_OCM_ALLOC_MODE, CN10K_ML_DEV_HW_QUEUE_LOCK, - CN10K_ML_FW_POLL_MEM, CN10K_ML_OCM_PAGE_SIZE, NULL}; @@ -103,9 +100,7 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde bool hw_queue_lock_set = false; bool ocm_page_size_set = false; char *ocm_alloc_mode = NULL; - bool poll_mem_set = false; bool fw_path_set = false; - char *poll_mem = NULL; char *fw_path = NULL; int ret = 0; bool found; @@ -189,17 +184,6 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde hw_queue_lock_set = true; } - if (rte_kvargs_count(kvlist, CN10K_ML_FW_POLL_MEM) == 1) { - ret = rte_kvargs_process(kvlist, CN10K_ML_FW_POLL_MEM, &parse_string_arg, - &poll_mem); - if (ret < 0) { - plt_err("Error processing arguments, key = %s\n", CN10K_ML_FW_POLL_MEM); - ret = -EINVAL; - goto exit; - } - poll_mem_set = true; - } - if (rte_kvargs_count(kvlist, CN10K_ML_OCM_PAGE_SIZE) == 1) { ret = rte_kvargs_process(kvlist, CN10K_ML_OCM_PAGE_SIZE, &parse_integer_arg, &mldev->ocm_page_size); @@ -280,18 +264,6 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde } plt_info("ML: %s = %d", CN10K_ML_DEV_HW_QUEUE_LOCK, mldev->hw_queue_lock); - if (!poll_mem_set) { - mldev->fw.poll_mem = CN10K_ML_FW_POLL_MEM_DEFAULT; - } else { - if (!((strcmp(poll_mem, "ddr") == 0) || (strcmp(poll_mem, "register") == 0))) { - plt_err("Invalid argument, %s = %s\n", CN10K_ML_FW_POLL_MEM, poll_mem); - ret = -EINVAL; - goto exit; - } - mldev->fw.poll_mem = poll_mem; - } - plt_info("ML: %s = %s", CN10K_ML_FW_POLL_MEM, mldev->fw.poll_mem); - if (!ocm_page_size_set) { mldev->ocm_page_size = CN10K_ML_OCM_PAGE_SIZE_DEFAULT; } else { @@ -450,10 +422,7 @@ cn10k_ml_fw_flags_get(struct cn10k_ml_fw *fw) if (fw->report_dpe_warnings) flags = flags | FW_REPORT_DPE_WARNING_BITMASK; - if (strcmp(fw->poll_mem, "ddr") == 0) - flags = flags | FW_USE_DDR_POLL_ADDR_FP; - else if (strcmp(fw->poll_mem, "register") == 0) - flags = flags & ~FW_USE_DDR_POLL_ADDR_FP; + flags = flags | FW_USE_DDR_POLL_ADDR_FP; return flags; } @@ -863,5 +832,4 @@ RTE_PMD_REGISTER_PARAM_STRING(MLDEV_NAME_CN10K_PMD, CN10K_ML_FW_PATH "=<0|1>" CN10K_ML_DEV_CACHE_MODEL_DATA "=<0|1>" CN10K_ML_OCM_ALLOC_MODE "=" CN10K_ML_DEV_HW_QUEUE_LOCK - "=<0|1>" CN10K_ML_FW_POLL_MEM "=" CN10K_ML_OCM_PAGE_SIZE - "=<1024|2048|4096|8192|16384>"); + "=<0|1>" CN10K_ML_OCM_PAGE_SIZE "=<1024|2048|4096|8192|16384>"); diff --git a/drivers/ml/cnxk/cn10k_ml_dev.h b/drivers/ml/cnxk/cn10k_ml_dev.h index c73bf7d001a..4aaeecff03d 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.h +++ b/drivers/ml/cnxk/cn10k_ml_dev.h @@ -390,9 +390,6 @@ struct cn10k_ml_fw { /* Report DPE warnings */ int report_dpe_warnings; - /* Memory to be used for polling in fast-path requests */ - const char *poll_mem; - /* Data buffer */ uint8_t *data; @@ -525,13 +522,9 @@ struct cn10k_ml_dev { bool (*ml_jcmdq_enqueue)(struct roc_ml *roc_ml, struct ml_job_cmd_s *job_cmd); /* Poll handling function pointers */ - void (*set_poll_addr)(struct cn10k_ml_qp *qp, struct cn10k_ml_req *req, uint64_t idx); - void (*set_poll_ptr)(struct roc_ml *roc_ml, struct cn10k_ml_req *req); - uint64_t (*get_poll_ptr)(struct roc_ml *roc_ml, struct cn10k_ml_req *req); - - /* Memory barrier function pointers to handle synchronization */ - void (*set_enq_barrier)(void); - void (*set_deq_barrier)(void); + void (*set_poll_addr)(struct cn10k_ml_req *req); + void (*set_poll_ptr)(struct cn10k_ml_req *req); + uint64_t (*get_poll_ptr)(struct cn10k_ml_req *req); }; uint64_t cn10k_ml_fw_flags_get(struct cn10k_ml_fw *fw); diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 4abf4ae0d39..11531afd8c1 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -23,11 +23,6 @@ #define ML_FLAGS_POLL_COMPL BIT(0) #define ML_FLAGS_SSO_COMPL BIT(1) -/* Scratch register range for poll mode requests */ -#define ML_POLL_REGISTER_SYNC 1023 -#define ML_POLL_REGISTER_START 1024 -#define ML_POLL_REGISTER_END 2047 - /* Error message length */ #define ERRMSG_LEN 32 @@ -82,79 +77,23 @@ print_line(FILE *fp, int len) } static inline void -cn10k_ml_set_poll_addr_ddr(struct cn10k_ml_qp *qp, struct cn10k_ml_req *req, uint64_t idx) +cn10k_ml_set_poll_addr(struct cn10k_ml_req *req) { - PLT_SET_USED(qp); - PLT_SET_USED(idx); - req->compl_W1 = PLT_U64_CAST(&req->status); } static inline void -cn10k_ml_set_poll_addr_reg(struct cn10k_ml_qp *qp, struct cn10k_ml_req *req, uint64_t idx) -{ - req->compl_W1 = ML_SCRATCH(qp->block_start + idx % qp->block_size); -} - -static inline void -cn10k_ml_set_poll_ptr_ddr(struct roc_ml *roc_ml, struct cn10k_ml_req *req) +cn10k_ml_set_poll_ptr(struct cn10k_ml_req *req) { - PLT_SET_USED(roc_ml); - plt_write64(ML_CN10K_POLL_JOB_START, req->compl_W1); } -static inline void -cn10k_ml_set_poll_ptr_reg(struct roc_ml *roc_ml, struct cn10k_ml_req *req) -{ - roc_ml_reg_write64(roc_ml, ML_CN10K_POLL_JOB_START, req->compl_W1); -} - static inline uint64_t -cn10k_ml_get_poll_ptr_ddr(struct roc_ml *roc_ml, struct cn10k_ml_req *req) +cn10k_ml_get_poll_ptr(struct cn10k_ml_req *req) { - PLT_SET_USED(roc_ml); - return plt_read64(req->compl_W1); } -static inline uint64_t -cn10k_ml_get_poll_ptr_reg(struct roc_ml *roc_ml, struct cn10k_ml_req *req) -{ - return roc_ml_reg_read64(roc_ml, req->compl_W1); -} - -static inline void -cn10k_ml_set_sync_addr(struct cn10k_ml_dev *mldev, struct cn10k_ml_req *req) -{ - if (strcmp(mldev->fw.poll_mem, "ddr") == 0) - req->compl_W1 = PLT_U64_CAST(&req->status); - else if (strcmp(mldev->fw.poll_mem, "register") == 0) - req->compl_W1 = ML_SCRATCH(ML_POLL_REGISTER_SYNC); -} - -static inline void -cn10k_ml_enq_barrier_ddr(void) -{ -} - -static inline void -cn10k_ml_deq_barrier_ddr(void) -{ -} - -static inline void -cn10k_ml_enq_barrier_register(void) -{ - dmb_st; -} - -static inline void -cn10k_ml_deq_barrier_register(void) -{ - dsb_st; -} - static void qp_memzone_name_get(char *name, int size, int dev_id, int qp_id) { @@ -242,9 +181,6 @@ cn10k_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_des qp->stats.dequeued_count = 0; qp->stats.enqueue_err_count = 0; qp->stats.dequeue_err_count = 0; - qp->block_size = - (ML_POLL_REGISTER_END - ML_POLL_REGISTER_START + 1) / dev->data->nb_queue_pairs; - qp->block_start = ML_POLL_REGISTER_START + qp_id * qp->block_size; /* Initialize job command */ for (i = 0; i < qp->nb_desc; i++) { @@ -933,11 +869,7 @@ cn10k_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) else dev_info->max_queue_pairs = ML_CN10K_MAX_QP_PER_DEVICE_LF; - if (strcmp(mldev->fw.poll_mem, "register") == 0) - dev_info->max_desc = ML_CN10K_MAX_DESC_PER_QP / dev_info->max_queue_pairs; - else if (strcmp(mldev->fw.poll_mem, "ddr") == 0) - dev_info->max_desc = ML_CN10K_MAX_DESC_PER_QP; - + dev_info->max_desc = ML_CN10K_MAX_DESC_PER_QP; dev_info->max_io = ML_CN10K_MAX_INPUT_OUTPUT; dev_info->max_segments = ML_CN10K_MAX_SEGMENTS; dev_info->align_size = ML_CN10K_ALIGN_SIZE; @@ -1118,24 +1050,9 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c mldev->ml_jcmdq_enqueue = roc_ml_jcmdq_enqueue_lf; /* Set polling function pointers */ - if (strcmp(mldev->fw.poll_mem, "ddr") == 0) { - mldev->set_poll_addr = cn10k_ml_set_poll_addr_ddr; - mldev->set_poll_ptr = cn10k_ml_set_poll_ptr_ddr; - mldev->get_poll_ptr = cn10k_ml_get_poll_ptr_ddr; - } else if (strcmp(mldev->fw.poll_mem, "register") == 0) { - mldev->set_poll_addr = cn10k_ml_set_poll_addr_reg; - mldev->set_poll_ptr = cn10k_ml_set_poll_ptr_reg; - mldev->get_poll_ptr = cn10k_ml_get_poll_ptr_reg; - } - - /* Set barrier function pointers */ - if (strcmp(mldev->fw.poll_mem, "ddr") == 0) { - mldev->set_enq_barrier = cn10k_ml_enq_barrier_ddr; - mldev->set_deq_barrier = cn10k_ml_deq_barrier_ddr; - } else if (strcmp(mldev->fw.poll_mem, "register") == 0) { - mldev->set_enq_barrier = cn10k_ml_enq_barrier_register; - mldev->set_deq_barrier = cn10k_ml_deq_barrier_register; - } + mldev->set_poll_addr = cn10k_ml_set_poll_addr; + mldev->set_poll_ptr = cn10k_ml_set_poll_ptr; + mldev->get_poll_ptr = cn10k_ml_get_poll_ptr; dev->enqueue_burst = cn10k_ml_enqueue_burst; dev->dequeue_burst = cn10k_ml_dequeue_burst; @@ -2390,15 +2307,14 @@ cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op op = ops[count]; req = &queue->reqs[head]; - mldev->set_poll_addr(qp, req, head); + mldev->set_poll_addr(req); cn10k_ml_prep_fp_job_descriptor(dev, req, op); memset(&req->result, 0, sizeof(struct cn10k_ml_result)); req->result.error_code.s.etype = ML_ETYPE_UNKNOWN; req->result.user_ptr = op->user_ptr; - mldev->set_enq_barrier(); - mldev->set_poll_ptr(&mldev->roc, req); + mldev->set_poll_ptr(req); enqueued = mldev->ml_jcmdq_enqueue(&mldev->roc, &req->jcmd); if (unlikely(!enqueued)) goto jcmdq_full; @@ -2445,7 +2361,7 @@ cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op dequeue_req: req = &queue->reqs[tail]; - status = mldev->get_poll_ptr(&mldev->roc, req); + status = mldev->get_poll_ptr(req); if (unlikely(status != ML_CN10K_POLL_JOB_FINISH)) { if (plt_tsc_cycles() < req->timeout) goto empty_or_active; @@ -2453,7 +2369,6 @@ cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op req->result.error_code.s.etype = ML_ETYPE_DRIVER; } - mldev->set_deq_barrier(); cn10k_ml_result_update(dev, qp_id, &req->result, req->op); ops[count] = req->op; @@ -2515,14 +2430,14 @@ cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) model = dev->data->models[op->model_id]; req = model->req; - cn10k_ml_set_sync_addr(mldev, req); + cn10k_ml_set_poll_addr(req); cn10k_ml_prep_fp_job_descriptor(dev, req, op); memset(&req->result, 0, sizeof(struct cn10k_ml_result)); req->result.error_code.s.etype = ML_ETYPE_UNKNOWN; req->result.user_ptr = op->user_ptr; - mldev->set_poll_ptr(&mldev->roc, req); + mldev->set_poll_ptr(req); req->jcmd.w1.s.jobptr = PLT_U64_CAST(&req->jd); timeout = true; @@ -2542,7 +2457,7 @@ cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) timeout = true; do { - if (mldev->get_poll_ptr(&mldev->roc, req) == ML_CN10K_POLL_JOB_FINISH) { + if (mldev->get_poll_ptr(req) == ML_CN10K_POLL_JOB_FINISH) { timeout = false; break; } diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index d64a9f27e6c..005b093e45d 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -67,12 +67,6 @@ struct cn10k_ml_qp { /* Statistics per queue-pair */ struct rte_ml_dev_stats stats; - - /* Register block start for polling */ - uint32_t block_start; - - /* Register block end for polling */ - uint32_t block_size; }; /* Device ops */ From patchwork Wed Aug 30 15:58:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130883 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CB48D41FD0; Wed, 30 Aug 2023 17:59:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7C9C9402A2; Wed, 30 Aug 2023 17:59:37 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 25A0E4027D for ; Wed, 30 Aug 2023 17:59:35 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I6q012554 for ; Wed, 30 Aug 2023 08:59:34 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=iw3xrBlgCZR3DdZqr320K2F7MGR5HFZeqi5XggAT2sM=; b=d7U3x+12hAeIwl86A2MgaD9EP5OJ2L53sqGv3SSoD47GdBVqCvZl+jN6YcujZEgRiuVj 6D79QsOWJaWNGlI3idwgNx93shU9hih5sHmG0rnkR2cF5m9dHddCxTVhslHo79/Q56u2 9arXFG4m3gOgySLQI2R5DHlelh1kV5EFad4bdoIMXflOQLlpSwLX3YeVgQPazxTHJatq g7v7Slt6wstVmc32GCqf+AzS4Bh5c9HVsm9w1ZojBWWTety2twdPF2Di9TZ4qtLecA78 /WCoYUOVbQpyThBh44ajiBDAnOIh+VCsck55yQRBTiSg/jg4lNCUybUWKvzBjIgeg0Gb hQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:34 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:32 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:32 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 3D94D3F707F; Wed, 30 Aug 2023 08:59:32 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi , Prince Takkar CC: , , Subject: [PATCH v1 02/34] ml/cnxk: drop use of RTE API for firmware read Date: Wed, 30 Aug 2023 08:58:52 -0700 Message-ID: <20230830155927.3566-3-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 5vDEwaGcS1n5eCosKOslH8KvL5cd1rQc X-Proofpoint-GUID: 5vDEwaGcS1n5eCosKOslH8KvL5cd1rQc X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Dropped use of rte_firmware_read API to read ML firmware binary. When DPDK is built with libarchive aaupport, the the RTE API assumes the binary file as a compressed archive. This causes the ML firmware binary to be parsed incorrectly. Fixes: c29da752ffa8 ("ml/cnxk: support firmware load and device reset") Cc: syalavarthi@marvell.com Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.c | 64 +++++++++++++++++++++++++++++++--- 1 file changed, 60 insertions(+), 4 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_dev.c b/drivers/ml/cnxk/cn10k_ml_dev.c index e3c2badcef5..b7e6ed9a00e 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.c +++ b/drivers/ml/cnxk/cn10k_ml_dev.c @@ -2,6 +2,11 @@ * Copyright (c) 2022 Marvell. */ +#include +#include +#include +#include + #include #include #include @@ -61,6 +66,57 @@ static const int valid_ocm_page_size[] = {1024, 2048, 4096, 8192, 16384}; /* Dummy operations for ML device */ struct rte_ml_dev_ops ml_dev_dummy_ops = {0}; +static int +ml_read_file(const char *file, size_t *size, char **buffer) +{ + char *file_buffer = NULL; + struct stat file_stat; + char *file_map; + int ret; + int fd; + + fd = open(file, O_RDONLY); + if (fd == -1) { + plt_err("Failed to open file: %s\n", file); + return -errno; + } + + if (fstat(fd, &file_stat) != 0) { + plt_err("fstat failed for file: %s\n", file); + close(fd); + return -errno; + } + + file_buffer = rte_malloc("ml_firmware", file_stat.st_size, PLT_CACHE_LINE_SIZE); + if (file_buffer == NULL) { + plt_err("Failed to allocate memory: %s\n", file); + ret = -ENOMEM; + goto error; + } + + file_map = mmap(0, file_stat.st_size, PROT_READ, MAP_PRIVATE, fd, 0); + if (file_map == MAP_FAILED) { + plt_err("Failed to map file: %s\n", file); + ret = -errno; + goto error; + } + + rte_memcpy(file_buffer, file_map, file_stat.st_size); + munmap(file_map, file_stat.st_size); + close(fd); + + *size = file_stat.st_size; + *buffer = file_buffer; + + return 0; + +error: + free(file_buffer); + close(fd); + + return ret; +} + static int parse_string_arg(const char *key __rte_unused, const char *value, void *extra_args) { @@ -736,7 +792,7 @@ cn10k_ml_fw_load(struct cn10k_ml_dev *mldev) { const struct plt_memzone *mz; struct cn10k_ml_fw *fw; - void *fw_buffer = NULL; + char *fw_buffer = NULL; uint64_t mz_size = 0; uint64_t fw_size = 0; int ret = 0; @@ -746,7 +802,7 @@ cn10k_ml_fw_load(struct cn10k_ml_dev *mldev) if (roc_env_is_emulator() || roc_env_is_hw()) { /* Read firmware image to a buffer */ - ret = rte_firmware_read(fw->path, &fw_buffer, &fw_size); + ret = ml_read_file(fw->path, &fw_size, &fw_buffer); if ((ret < 0) || (fw_buffer == NULL)) { plt_err("Unable to read firmware data: %s\n", fw->path); return ret; @@ -763,7 +819,7 @@ cn10k_ml_fw_load(struct cn10k_ml_dev *mldev) mz = plt_memzone_reserve_aligned(FW_MEMZONE_NAME, mz_size, 0, ML_CN10K_ALIGN_SIZE); if (mz == NULL) { plt_err("plt_memzone_reserve failed : %s", FW_MEMZONE_NAME); - free(fw_buffer); + rte_free(fw_buffer); return -ENOMEM; } fw->req = mz->addr; @@ -780,7 +836,7 @@ cn10k_ml_fw_load(struct cn10k_ml_dev *mldev) if (roc_env_is_emulator() || roc_env_is_hw()) { fw->data = PLT_PTR_ADD(mz->addr, sizeof(struct cn10k_ml_req)); ret = cn10k_ml_fw_load_cn10ka(fw, fw_buffer, fw_size); - free(fw_buffer); + rte_free(fw_buffer); } else if (roc_env_is_asim()) { fw->data = NULL; ret = cn10k_ml_fw_load_asim(fw); From patchwork Wed Aug 30 15:58:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130884 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 00E0441FD0; Wed, 30 Aug 2023 17:59:56 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9E569402A7; Wed, 30 Aug 2023 17:59:39 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 072BE40284 for ; Wed, 30 Aug 2023 17:59:35 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7ox004368 for ; Wed, 30 Aug 2023 08:59:35 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=lrwpKV7hnA6SLo3No3V5PESqqnTOFFdoQfDoIY8M7fY=; b=hEv6lSOKTGGVp9cdqJYpzewqMvJjaGqwXAzWnXXEIQY10tD6fQaSW+t7mHs72+XV2BbQ SpSspMq+DNn3c1qhXXTVP84CLKADbPdiPTwimRquVFa0QUyT4T5sZ3GvHEXkmUTSwAAV w63MczNExx1LJy1q03dqYONMwcgGNGWqSnCqCHRv4bQj62YWHaTdK26W+hFVeJnPi0SR Wg3DD+rDMq6nJQLFZaBOxJ9kQEGutCFqDy9xezz1ccVzcEFnC0z7yXyIUVeYQffDFQ6J y7AmnJrzyTPVsapYX7QJeSt0ljAk+cZLM0alv2psZiUsx/bZLa2qRa6TvTsN9OSst1sh ew== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysst5-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:34 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:32 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:32 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 973D63F707D; Wed, 30 Aug 2023 08:59:32 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 03/34] ml/cnxk: add generic cnxk device structure Date: Wed, 30 Aug 2023 08:58:53 -0700 Message-ID: <20230830155927.3566-4-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: PiAEwHmjyNoZB-MfagU5Xrrdl_Wbg-T6 X-Proofpoint-ORIG-GUID: PiAEwHmjyNoZB-MfagU5Xrrdl_Wbg-T6 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Introduce generic cnxk device structure. This structure is a top level device structure for the driver, which would encapsulate the target / platform specific device structure. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.c | 315 ++++++++++---------- drivers/ml/cnxk/cn10k_ml_dev.h | 47 +-- drivers/ml/cnxk/cn10k_ml_model.c | 14 +- drivers/ml/cnxk/cn10k_ml_model.h | 8 +- drivers/ml/cnxk/cn10k_ml_ocm.c | 56 ++-- drivers/ml/cnxk/cn10k_ml_ops.c | 494 +++++++++++++++++-------------- drivers/ml/cnxk/cnxk_ml_dev.c | 11 + drivers/ml/cnxk/cnxk_ml_dev.h | 58 ++++ drivers/ml/cnxk/meson.build | 2 + 9 files changed, 562 insertions(+), 443 deletions(-) create mode 100644 drivers/ml/cnxk/cnxk_ml_dev.c create mode 100644 drivers/ml/cnxk/cnxk_ml_dev.h diff --git a/drivers/ml/cnxk/cn10k_ml_dev.c b/drivers/ml/cnxk/cn10k_ml_dev.c index b7e6ed9a00e..367fb7014c4 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.c +++ b/drivers/ml/cnxk/cn10k_ml_dev.c @@ -15,13 +15,15 @@ #include #include -#include - #include +#include + #include "cn10k_ml_dev.h" #include "cn10k_ml_ops.h" +#include "cnxk_ml_dev.h" + #define CN10K_ML_FW_PATH "fw_path" #define CN10K_ML_FW_ENABLE_DPE_WARNINGS "enable_dpe_warnings" #define CN10K_ML_FW_REPORT_DPE_WARNINGS "report_dpe_warnings" @@ -63,9 +65,6 @@ static const char *const valid_args[] = {CN10K_ML_FW_PATH, /* Supported OCM page sizes: 1KB, 2KB, 4KB, 8KB and 16KB */ static const int valid_ocm_page_size[] = {1024, 2048, 4096, 8192, 16384}; -/* Dummy operations for ML device */ -struct rte_ml_dev_ops ml_dev_dummy_ops = {0}; - static int ml_read_file(const char *file, size_t *size, char **buffer) { @@ -146,7 +145,7 @@ parse_integer_arg(const char *key __rte_unused, const char *value, void *extra_a } static int -cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mldev) +cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *cn10k_mldev) { bool enable_dpe_warnings_set = false; bool report_dpe_warnings_set = false; @@ -183,7 +182,7 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde if (rte_kvargs_count(kvlist, CN10K_ML_FW_ENABLE_DPE_WARNINGS) == 1) { ret = rte_kvargs_process(kvlist, CN10K_ML_FW_ENABLE_DPE_WARNINGS, - &parse_integer_arg, &mldev->fw.enable_dpe_warnings); + &parse_integer_arg, &cn10k_mldev->fw.enable_dpe_warnings); if (ret < 0) { plt_err("Error processing arguments, key = %s\n", CN10K_ML_FW_ENABLE_DPE_WARNINGS); @@ -195,7 +194,7 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde if (rte_kvargs_count(kvlist, CN10K_ML_FW_REPORT_DPE_WARNINGS) == 1) { ret = rte_kvargs_process(kvlist, CN10K_ML_FW_REPORT_DPE_WARNINGS, - &parse_integer_arg, &mldev->fw.report_dpe_warnings); + &parse_integer_arg, &cn10k_mldev->fw.report_dpe_warnings); if (ret < 0) { plt_err("Error processing arguments, key = %s\n", CN10K_ML_FW_REPORT_DPE_WARNINGS); @@ -207,7 +206,7 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde if (rte_kvargs_count(kvlist, CN10K_ML_DEV_CACHE_MODEL_DATA) == 1) { ret = rte_kvargs_process(kvlist, CN10K_ML_DEV_CACHE_MODEL_DATA, &parse_integer_arg, - &mldev->cache_model_data); + &cn10k_mldev->cache_model_data); if (ret < 0) { plt_err("Error processing arguments, key = %s\n", CN10K_ML_DEV_CACHE_MODEL_DATA); @@ -230,7 +229,7 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde if (rte_kvargs_count(kvlist, CN10K_ML_DEV_HW_QUEUE_LOCK) == 1) { ret = rte_kvargs_process(kvlist, CN10K_ML_DEV_HW_QUEUE_LOCK, &parse_integer_arg, - &mldev->hw_queue_lock); + &cn10k_mldev->hw_queue_lock); if (ret < 0) { plt_err("Error processing arguments, key = %s\n", CN10K_ML_DEV_HW_QUEUE_LOCK); @@ -242,7 +241,7 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde if (rte_kvargs_count(kvlist, CN10K_ML_OCM_PAGE_SIZE) == 1) { ret = rte_kvargs_process(kvlist, CN10K_ML_OCM_PAGE_SIZE, &parse_integer_arg, - &mldev->ocm_page_size); + &cn10k_mldev->ocm_page_size); if (ret < 0) { plt_err("Error processing arguments, key = %s\n", CN10K_ML_OCM_PAGE_SIZE); ret = -EINVAL; @@ -253,49 +252,53 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde check_args: if (!fw_path_set) - mldev->fw.path = CN10K_ML_FW_PATH_DEFAULT; + cn10k_mldev->fw.path = CN10K_ML_FW_PATH_DEFAULT; else - mldev->fw.path = fw_path; - plt_info("ML: %s = %s", CN10K_ML_FW_PATH, mldev->fw.path); + cn10k_mldev->fw.path = fw_path; + plt_info("ML: %s = %s", CN10K_ML_FW_PATH, cn10k_mldev->fw.path); if (!enable_dpe_warnings_set) { - mldev->fw.enable_dpe_warnings = CN10K_ML_FW_ENABLE_DPE_WARNINGS_DEFAULT; + cn10k_mldev->fw.enable_dpe_warnings = CN10K_ML_FW_ENABLE_DPE_WARNINGS_DEFAULT; } else { - if ((mldev->fw.enable_dpe_warnings < 0) || (mldev->fw.enable_dpe_warnings > 1)) { + if ((cn10k_mldev->fw.enable_dpe_warnings < 0) || + (cn10k_mldev->fw.enable_dpe_warnings > 1)) { plt_err("Invalid argument, %s = %d\n", CN10K_ML_FW_ENABLE_DPE_WARNINGS, - mldev->fw.enable_dpe_warnings); + cn10k_mldev->fw.enable_dpe_warnings); ret = -EINVAL; goto exit; } } - plt_info("ML: %s = %d", CN10K_ML_FW_ENABLE_DPE_WARNINGS, mldev->fw.enable_dpe_warnings); + plt_info("ML: %s = %d", CN10K_ML_FW_ENABLE_DPE_WARNINGS, + cn10k_mldev->fw.enable_dpe_warnings); if (!report_dpe_warnings_set) { - mldev->fw.report_dpe_warnings = CN10K_ML_FW_REPORT_DPE_WARNINGS_DEFAULT; + cn10k_mldev->fw.report_dpe_warnings = CN10K_ML_FW_REPORT_DPE_WARNINGS_DEFAULT; } else { - if ((mldev->fw.report_dpe_warnings < 0) || (mldev->fw.report_dpe_warnings > 1)) { + if ((cn10k_mldev->fw.report_dpe_warnings < 0) || + (cn10k_mldev->fw.report_dpe_warnings > 1)) { plt_err("Invalid argument, %s = %d\n", CN10K_ML_FW_REPORT_DPE_WARNINGS, - mldev->fw.report_dpe_warnings); + cn10k_mldev->fw.report_dpe_warnings); ret = -EINVAL; goto exit; } } - plt_info("ML: %s = %d", CN10K_ML_FW_REPORT_DPE_WARNINGS, mldev->fw.report_dpe_warnings); + plt_info("ML: %s = %d", CN10K_ML_FW_REPORT_DPE_WARNINGS, + cn10k_mldev->fw.report_dpe_warnings); if (!cache_model_data_set) { - mldev->cache_model_data = CN10K_ML_DEV_CACHE_MODEL_DATA_DEFAULT; + cn10k_mldev->cache_model_data = CN10K_ML_DEV_CACHE_MODEL_DATA_DEFAULT; } else { - if ((mldev->cache_model_data < 0) || (mldev->cache_model_data > 1)) { + if ((cn10k_mldev->cache_model_data < 0) || (cn10k_mldev->cache_model_data > 1)) { plt_err("Invalid argument, %s = %d\n", CN10K_ML_DEV_CACHE_MODEL_DATA, - mldev->cache_model_data); + cn10k_mldev->cache_model_data); ret = -EINVAL; goto exit; } } - plt_info("ML: %s = %d", CN10K_ML_DEV_CACHE_MODEL_DATA, mldev->cache_model_data); + plt_info("ML: %s = %d", CN10K_ML_DEV_CACHE_MODEL_DATA, cn10k_mldev->cache_model_data); if (!ocm_alloc_mode_set) { - mldev->ocm.alloc_mode = CN10K_ML_OCM_ALLOC_MODE_DEFAULT; + cn10k_mldev->ocm.alloc_mode = CN10K_ML_OCM_ALLOC_MODE_DEFAULT; } else { if (!((strcmp(ocm_alloc_mode, "lowest") == 0) || (strcmp(ocm_alloc_mode, "largest") == 0))) { @@ -304,47 +307,47 @@ cn10k_mldev_parse_devargs(struct rte_devargs *devargs, struct cn10k_ml_dev *mlde ret = -EINVAL; goto exit; } - mldev->ocm.alloc_mode = ocm_alloc_mode; + cn10k_mldev->ocm.alloc_mode = ocm_alloc_mode; } - plt_info("ML: %s = %s", CN10K_ML_OCM_ALLOC_MODE, mldev->ocm.alloc_mode); + plt_info("ML: %s = %s", CN10K_ML_OCM_ALLOC_MODE, cn10k_mldev->ocm.alloc_mode); if (!hw_queue_lock_set) { - mldev->hw_queue_lock = CN10K_ML_DEV_HW_QUEUE_LOCK_DEFAULT; + cn10k_mldev->hw_queue_lock = CN10K_ML_DEV_HW_QUEUE_LOCK_DEFAULT; } else { - if ((mldev->hw_queue_lock < 0) || (mldev->hw_queue_lock > 1)) { + if ((cn10k_mldev->hw_queue_lock < 0) || (cn10k_mldev->hw_queue_lock > 1)) { plt_err("Invalid argument, %s = %d\n", CN10K_ML_DEV_HW_QUEUE_LOCK, - mldev->hw_queue_lock); + cn10k_mldev->hw_queue_lock); ret = -EINVAL; goto exit; } } - plt_info("ML: %s = %d", CN10K_ML_DEV_HW_QUEUE_LOCK, mldev->hw_queue_lock); + plt_info("ML: %s = %d", CN10K_ML_DEV_HW_QUEUE_LOCK, cn10k_mldev->hw_queue_lock); if (!ocm_page_size_set) { - mldev->ocm_page_size = CN10K_ML_OCM_PAGE_SIZE_DEFAULT; + cn10k_mldev->ocm_page_size = CN10K_ML_OCM_PAGE_SIZE_DEFAULT; } else { - if (mldev->ocm_page_size < 0) { + if (cn10k_mldev->ocm_page_size < 0) { plt_err("Invalid argument, %s = %d\n", CN10K_ML_OCM_PAGE_SIZE, - mldev->ocm_page_size); + cn10k_mldev->ocm_page_size); ret = -EINVAL; goto exit; } found = false; for (i = 0; i < PLT_DIM(valid_ocm_page_size); i++) { - if (mldev->ocm_page_size == valid_ocm_page_size[i]) { + if (cn10k_mldev->ocm_page_size == valid_ocm_page_size[i]) { found = true; break; } } if (!found) { - plt_err("Unsupported ocm_page_size = %d\n", mldev->ocm_page_size); + plt_err("Unsupported ocm_page_size = %d\n", cn10k_mldev->ocm_page_size); ret = -EINVAL; goto exit; } } - plt_info("ML: %s = %d", CN10K_ML_OCM_PAGE_SIZE, mldev->ocm_page_size); + plt_info("ML: %s = %d", CN10K_ML_OCM_PAGE_SIZE, cn10k_mldev->ocm_page_size); exit: rte_kvargs_free(kvlist); @@ -356,7 +359,8 @@ static int cn10k_ml_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) { struct rte_ml_dev_pmd_init_params init_params; - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; char name[RTE_ML_STR_MAX]; struct rte_ml_dev *dev; int ret; @@ -364,7 +368,7 @@ cn10k_ml_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_de PLT_SET_USED(pci_drv); init_params = (struct rte_ml_dev_pmd_init_params){ - .socket_id = rte_socket_id(), .private_data_size = sizeof(struct cn10k_ml_dev)}; + .socket_id = rte_socket_id(), .private_data_size = sizeof(struct cnxk_ml_dev)}; ret = roc_plt_init(); if (ret < 0) { @@ -380,18 +384,20 @@ cn10k_ml_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_de } /* Get private data space allocated */ - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cnxk_mldev->mldev = dev; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; if (rte_eal_process_type() == RTE_PROC_PRIMARY) { - mldev->roc.pci_dev = pci_dev; + cn10k_mldev->roc.pci_dev = pci_dev; - ret = cn10k_mldev_parse_devargs(dev->device->devargs, mldev); + ret = cn10k_mldev_parse_devargs(dev->device->devargs, cn10k_mldev); if (ret) { plt_err("Failed to parse devargs ret = %d", ret); goto pmd_destroy; } - ret = roc_ml_dev_init(&mldev->roc); + ret = roc_ml_dev_init(&cn10k_mldev->roc); if (ret) { plt_err("Failed to initialize ML ROC, ret = %d", ret); goto pmd_destroy; @@ -407,7 +413,7 @@ cn10k_ml_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_de dev->dequeue_burst = NULL; dev->op_error_get = NULL; - mldev->state = ML_CN10K_DEV_STATE_PROBED; + cnxk_mldev->state = ML_CNXK_DEV_STATE_PROBED; return 0; @@ -424,7 +430,7 @@ cn10k_ml_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_de static int cn10k_ml_pci_remove(struct rte_pci_device *pci_dev) { - struct cn10k_ml_dev *mldev; + struct cnxk_ml_dev *cnxk_mldev; char name[RTE_ML_STR_MAX]; struct rte_ml_dev *dev; int ret; @@ -439,8 +445,8 @@ cn10k_ml_pci_remove(struct rte_pci_device *pci_dev) return -ENODEV; if (rte_eal_process_type() == RTE_PROC_PRIMARY) { - mldev = dev->data->dev_private; - ret = roc_ml_dev_fini(&mldev->roc); + cnxk_mldev = dev->data->dev_private; + ret = roc_ml_dev_fini(&cnxk_mldev->cn10k_mldev.roc); if (ret) return ret; } @@ -486,45 +492,45 @@ cn10k_ml_fw_flags_get(struct cn10k_ml_fw *fw) static int cn10k_ml_fw_load_asim(struct cn10k_ml_fw *fw) { - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; uint64_t timeout_cycle; uint64_t reg_val64; bool timeout; int ret = 0; - mldev = fw->mldev; + cn10k_mldev = fw->cn10k_mldev; /* Reset HEAD and TAIL debug pointer registers */ - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C0); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C0); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C1); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C1); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_EXCEPTION_SP_C0); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_EXCEPTION_SP_C1); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C0); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C0); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C1); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C1); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_EXCEPTION_SP_C0); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_EXCEPTION_SP_C1); /* Set ML_MLR_BASE to base IOVA of the ML region in LLC/DRAM. */ reg_val64 = rte_eal_get_baseaddr(); - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_MLR_BASE); - plt_ml_dbg("ML_MLR_BASE = 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_MLR_BASE)); - roc_ml_reg_save(&mldev->roc, ML_MLR_BASE); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_MLR_BASE); + plt_ml_dbg("ML_MLR_BASE = 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_MLR_BASE)); + roc_ml_reg_save(&cn10k_mldev->roc, ML_MLR_BASE); /* Update FW load completion structure */ fw->req->jd.hdr.jce.w1.u64 = PLT_U64_CAST(&fw->req->status); fw->req->jd.hdr.job_type = ML_CN10K_JOB_TYPE_FIRMWARE_LOAD; - fw->req->jd.hdr.result = roc_ml_addr_ap2mlip(&mldev->roc, &fw->req->result); + fw->req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &fw->req->result); fw->req->jd.fw_load.flags = cn10k_ml_fw_flags_get(fw); - plt_write64(ML_CN10K_POLL_JOB_START, &fw->req->status); + plt_write64(ML_CNXK_POLL_JOB_START, &fw->req->status); plt_wmb(); /* Enqueue FW load through scratch registers */ timeout = true; - timeout_cycle = plt_tsc_cycles() + ML_CN10K_CMD_TIMEOUT * plt_tsc_hz(); - roc_ml_scratch_enqueue(&mldev->roc, &fw->req->jd); + timeout_cycle = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); + roc_ml_scratch_enqueue(&cn10k_mldev->roc, &fw->req->jd); plt_rmb(); do { - if (roc_ml_scratch_is_done_bit_set(&mldev->roc) && - (plt_read64(&fw->req->status) == ML_CN10K_POLL_JOB_FINISH)) { + if (roc_ml_scratch_is_done_bit_set(&cn10k_mldev->roc) && + (plt_read64(&fw->req->status) == ML_CNXK_POLL_JOB_FINISH)) { timeout = false; break; } @@ -536,11 +542,11 @@ cn10k_ml_fw_load_asim(struct cn10k_ml_fw *fw) } else { /* Set ML to disable new jobs */ reg_val64 = (ROC_ML_CFG_JD_SIZE | ROC_ML_CFG_MLIP_ENA); - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); /* Clear scratch registers */ - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_WORK_PTR); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_FW_CTRL); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_WORK_PTR); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_FW_CTRL); if (timeout) { plt_err("Firmware load timeout"); @@ -554,14 +560,14 @@ cn10k_ml_fw_load_asim(struct cn10k_ml_fw *fw) } /* Reset scratch registers */ - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_FW_CTRL); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_WORK_PTR); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_FW_CTRL); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_WORK_PTR); /* Disable job execution, to be enabled in start */ - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_CFG); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); reg_val64 &= ~ROC_ML_CFG_ENA; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); - plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_CFG)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); + plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); return ret; } @@ -571,7 +577,7 @@ cn10k_ml_fw_load_cn10ka(struct cn10k_ml_fw *fw, void *buffer, uint64_t size) { union ml_a35_0_rst_vector_base_s a35_0_rst_vector_base; union ml_a35_0_rst_vector_base_s a35_1_rst_vector_base; - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; uint64_t timeout_cycle; uint64_t reg_val64; uint32_t reg_val32; @@ -580,24 +586,24 @@ cn10k_ml_fw_load_cn10ka(struct cn10k_ml_fw *fw, void *buffer, uint64_t size) int ret = 0; uint8_t i; - mldev = fw->mldev; + cn10k_mldev = fw->cn10k_mldev; /* Reset HEAD and TAIL debug pointer registers */ - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C0); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C0); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C1); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C1); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_EXCEPTION_SP_C0); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_EXCEPTION_SP_C1); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C0); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C0); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C1); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C1); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_EXCEPTION_SP_C0); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_EXCEPTION_SP_C1); /* (1) Write firmware images for ACC's two A35 cores to the ML region in LLC / DRAM. */ rte_memcpy(PLT_PTR_ADD(fw->data, FW_LINKER_OFFSET), buffer, size); /* (2) Set ML(0)_MLR_BASE = Base IOVA of the ML region in LLC/DRAM. */ reg_val64 = PLT_PTR_SUB_U64_CAST(fw->data, rte_eal_get_baseaddr()); - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_MLR_BASE); - plt_ml_dbg("ML_MLR_BASE => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_MLR_BASE)); - roc_ml_reg_save(&mldev->roc, ML_MLR_BASE); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_MLR_BASE); + plt_ml_dbg("ML_MLR_BASE => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_MLR_BASE)); + roc_ml_reg_save(&cn10k_mldev->roc, ML_MLR_BASE); /* (3) Set ML(0)_AXI_BRIDGE_CTRL(1) = 0x184003 to remove back-pressure check on DMA AXI * bridge. @@ -605,9 +611,9 @@ cn10k_ml_fw_load_cn10ka(struct cn10k_ml_fw *fw, void *buffer, uint64_t size) reg_val64 = (ROC_ML_AXI_BRIDGE_CTRL_AXI_RESP_CTRL | ROC_ML_AXI_BRIDGE_CTRL_BRIDGE_CTRL_MODE | ROC_ML_AXI_BRIDGE_CTRL_NCB_WR_BLK | ROC_ML_AXI_BRIDGE_CTRL_FORCE_WRESP_OK | ROC_ML_AXI_BRIDGE_CTRL_FORCE_RRESP_OK); - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_AXI_BRIDGE_CTRL(1)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_AXI_BRIDGE_CTRL(1)); plt_ml_dbg("ML_AXI_BRIDGE_CTRL(1) => 0x%016lx", - roc_ml_reg_read64(&mldev->roc, ML_AXI_BRIDGE_CTRL(1))); + roc_ml_reg_read64(&cn10k_mldev->roc, ML_AXI_BRIDGE_CTRL(1))); /* (4) Set ML(0)_ANB(0..2)_BACKP_DISABLE = 0x3 to remove back-pressure on the AXI to NCB * bridges. @@ -615,9 +621,9 @@ cn10k_ml_fw_load_cn10ka(struct cn10k_ml_fw *fw, void *buffer, uint64_t size) for (i = 0; i < ML_ANBX_NR; i++) { reg_val64 = (ROC_ML_ANBX_BACKP_DISABLE_EXTMSTR_B_BACKP_DISABLE | ROC_ML_ANBX_BACKP_DISABLE_EXTMSTR_R_BACKP_DISABLE); - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_ANBX_BACKP_DISABLE(i)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_ANBX_BACKP_DISABLE(i)); plt_ml_dbg("ML_ANBX_BACKP_DISABLE(%u) => 0x%016lx", i, - roc_ml_reg_read64(&mldev->roc, ML_ANBX_BACKP_DISABLE(i))); + roc_ml_reg_read64(&cn10k_mldev->roc, ML_ANBX_BACKP_DISABLE(i))); } /* (5) Set ML(0)_ANB(0..2)_NCBI_P_OVR = 0x3000 and ML(0)_ANB(0..2)_NCBI_NP_OVR = 0x3000 to @@ -626,39 +632,40 @@ cn10k_ml_fw_load_cn10ka(struct cn10k_ml_fw *fw, void *buffer, uint64_t size) for (i = 0; i < ML_ANBX_NR; i++) { reg_val64 = (ML_ANBX_NCBI_P_OVR_ANB_NCBI_P_NS_OVR | ML_ANBX_NCBI_P_OVR_ANB_NCBI_P_NS_OVR_VLD); - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_ANBX_NCBI_P_OVR(i)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_ANBX_NCBI_P_OVR(i)); plt_ml_dbg("ML_ANBX_NCBI_P_OVR(%u) => 0x%016lx", i, - roc_ml_reg_read64(&mldev->roc, ML_ANBX_NCBI_P_OVR(i))); + roc_ml_reg_read64(&cn10k_mldev->roc, ML_ANBX_NCBI_P_OVR(i))); reg_val64 |= (ML_ANBX_NCBI_NP_OVR_ANB_NCBI_NP_NS_OVR | ML_ANBX_NCBI_NP_OVR_ANB_NCBI_NP_NS_OVR_VLD); - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_ANBX_NCBI_NP_OVR(i)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_ANBX_NCBI_NP_OVR(i)); plt_ml_dbg("ML_ANBX_NCBI_NP_OVR(%u) => 0x%016lx", i, - roc_ml_reg_read64(&mldev->roc, ML_ANBX_NCBI_NP_OVR(i))); + roc_ml_reg_read64(&cn10k_mldev->roc, ML_ANBX_NCBI_NP_OVR(i))); } /* (6) Set ML(0)_CFG[MLIP_CLK_FORCE] = 1, to force turning on the MLIP clock. */ - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_CFG); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); reg_val64 |= ROC_ML_CFG_MLIP_CLK_FORCE; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); - plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_CFG)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); + plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); /* (7) Set ML(0)_JOB_MGR_CTRL[STALL_ON_IDLE] = 0, to make sure the boot request is accepted * when there is no job in the command queue. */ - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_JOB_MGR_CTRL); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_JOB_MGR_CTRL); reg_val64 &= ~ROC_ML_JOB_MGR_CTRL_STALL_ON_IDLE; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_JOB_MGR_CTRL); - plt_ml_dbg("ML_JOB_MGR_CTRL => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_JOB_MGR_CTRL)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_JOB_MGR_CTRL); + plt_ml_dbg("ML_JOB_MGR_CTRL => 0x%016lx", + roc_ml_reg_read64(&cn10k_mldev->roc, ML_JOB_MGR_CTRL)); /* (8) Set ML(0)_CFG[ENA] = 0 and ML(0)_CFG[MLIP_ENA] = 1 to bring MLIP out of reset while * keeping the job manager disabled. */ - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_CFG); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); reg_val64 |= ROC_ML_CFG_MLIP_ENA; reg_val64 &= ~ROC_ML_CFG_ENA; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); - plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_CFG)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); + plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); /* (9) Wait at least 70 coprocessor clock cycles. */ plt_delay_us(FW_WAIT_CYCLES); @@ -669,53 +676,57 @@ cn10k_ml_fw_load_cn10ka(struct cn10k_ml_fw *fw, void *buffer, uint64_t size) * AXI outbound address divided by 4. Read after write. */ offset = PLT_PTR_ADD_U64_CAST( - fw->data, FW_LINKER_OFFSET - roc_ml_reg_read64(&mldev->roc, ML_MLR_BASE)); + fw->data, FW_LINKER_OFFSET - roc_ml_reg_read64(&cn10k_mldev->roc, ML_MLR_BASE)); a35_0_rst_vector_base.s.addr = (offset + ML_AXI_START_ADDR) / 4; a35_1_rst_vector_base.s.addr = (offset + ML_AXI_START_ADDR) / 4; - roc_ml_reg_write32(&mldev->roc, a35_0_rst_vector_base.w.w0, ML_A35_0_RST_VECTOR_BASE_W(0)); - reg_val32 = roc_ml_reg_read32(&mldev->roc, ML_A35_0_RST_VECTOR_BASE_W(0)); + roc_ml_reg_write32(&cn10k_mldev->roc, a35_0_rst_vector_base.w.w0, + ML_A35_0_RST_VECTOR_BASE_W(0)); + reg_val32 = roc_ml_reg_read32(&cn10k_mldev->roc, ML_A35_0_RST_VECTOR_BASE_W(0)); plt_ml_dbg("ML_A35_0_RST_VECTOR_BASE_W(0) => 0x%08x", reg_val32); - roc_ml_reg_write32(&mldev->roc, a35_0_rst_vector_base.w.w1, ML_A35_0_RST_VECTOR_BASE_W(1)); - reg_val32 = roc_ml_reg_read32(&mldev->roc, ML_A35_0_RST_VECTOR_BASE_W(1)); + roc_ml_reg_write32(&cn10k_mldev->roc, a35_0_rst_vector_base.w.w1, + ML_A35_0_RST_VECTOR_BASE_W(1)); + reg_val32 = roc_ml_reg_read32(&cn10k_mldev->roc, ML_A35_0_RST_VECTOR_BASE_W(1)); plt_ml_dbg("ML_A35_0_RST_VECTOR_BASE_W(1) => 0x%08x", reg_val32); - roc_ml_reg_write32(&mldev->roc, a35_1_rst_vector_base.w.w0, ML_A35_1_RST_VECTOR_BASE_W(0)); - reg_val32 = roc_ml_reg_read32(&mldev->roc, ML_A35_1_RST_VECTOR_BASE_W(0)); + roc_ml_reg_write32(&cn10k_mldev->roc, a35_1_rst_vector_base.w.w0, + ML_A35_1_RST_VECTOR_BASE_W(0)); + reg_val32 = roc_ml_reg_read32(&cn10k_mldev->roc, ML_A35_1_RST_VECTOR_BASE_W(0)); plt_ml_dbg("ML_A35_1_RST_VECTOR_BASE_W(0) => 0x%08x", reg_val32); - roc_ml_reg_write32(&mldev->roc, a35_1_rst_vector_base.w.w1, ML_A35_1_RST_VECTOR_BASE_W(1)); - reg_val32 = roc_ml_reg_read32(&mldev->roc, ML_A35_1_RST_VECTOR_BASE_W(1)); + roc_ml_reg_write32(&cn10k_mldev->roc, a35_1_rst_vector_base.w.w1, + ML_A35_1_RST_VECTOR_BASE_W(1)); + reg_val32 = roc_ml_reg_read32(&cn10k_mldev->roc, ML_A35_1_RST_VECTOR_BASE_W(1)); plt_ml_dbg("ML_A35_1_RST_VECTOR_BASE_W(1) => 0x%08x", reg_val32); /* (11) Clear MLIP's ML(0)_SW_RST_CTRL[ACC_RST]. This will bring the ACC cores and other * MLIP components out of reset. The cores will execute firmware from the ML region as * written in step 1. */ - reg_val32 = roc_ml_reg_read32(&mldev->roc, ML_SW_RST_CTRL); + reg_val32 = roc_ml_reg_read32(&cn10k_mldev->roc, ML_SW_RST_CTRL); reg_val32 &= ~ROC_ML_SW_RST_CTRL_ACC_RST; - roc_ml_reg_write32(&mldev->roc, reg_val32, ML_SW_RST_CTRL); - reg_val32 = roc_ml_reg_read32(&mldev->roc, ML_SW_RST_CTRL); + roc_ml_reg_write32(&cn10k_mldev->roc, reg_val32, ML_SW_RST_CTRL); + reg_val32 = roc_ml_reg_read32(&cn10k_mldev->roc, ML_SW_RST_CTRL); plt_ml_dbg("ML_SW_RST_CTRL => 0x%08x", reg_val32); /* (12) Wait for notification from firmware that ML is ready for job execution. */ fw->req->jd.hdr.jce.w1.u64 = PLT_U64_CAST(&fw->req->status); fw->req->jd.hdr.job_type = ML_CN10K_JOB_TYPE_FIRMWARE_LOAD; - fw->req->jd.hdr.result = roc_ml_addr_ap2mlip(&mldev->roc, &fw->req->result); + fw->req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &fw->req->result); fw->req->jd.fw_load.flags = cn10k_ml_fw_flags_get(fw); - plt_write64(ML_CN10K_POLL_JOB_START, &fw->req->status); + plt_write64(ML_CNXK_POLL_JOB_START, &fw->req->status); plt_wmb(); /* Enqueue FW load through scratch registers */ timeout = true; - timeout_cycle = plt_tsc_cycles() + ML_CN10K_CMD_TIMEOUT * plt_tsc_hz(); - roc_ml_scratch_enqueue(&mldev->roc, &fw->req->jd); + timeout_cycle = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); + roc_ml_scratch_enqueue(&cn10k_mldev->roc, &fw->req->jd); plt_rmb(); do { - if (roc_ml_scratch_is_done_bit_set(&mldev->roc) && - (plt_read64(&fw->req->status) == ML_CN10K_POLL_JOB_FINISH)) { + if (roc_ml_scratch_is_done_bit_set(&cn10k_mldev->roc) && + (plt_read64(&fw->req->status) == ML_CNXK_POLL_JOB_FINISH)) { timeout = false; break; } @@ -727,11 +738,11 @@ cn10k_ml_fw_load_cn10ka(struct cn10k_ml_fw *fw, void *buffer, uint64_t size) } else { /* Set ML to disable new jobs */ reg_val64 = (ROC_ML_CFG_JD_SIZE | ROC_ML_CFG_MLIP_ENA); - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); /* Clear scratch registers */ - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_WORK_PTR); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_FW_CTRL); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_WORK_PTR); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_FW_CTRL); if (timeout) { plt_err("Firmware load timeout"); @@ -747,49 +758,51 @@ cn10k_ml_fw_load_cn10ka(struct cn10k_ml_fw *fw, void *buffer, uint64_t size) /* (13) Set ML(0)_JOB_MGR_CTRL[STALL_ON_IDLE] = 0x1; this is needed to shut down the MLIP * clock when there are no more jobs to process. */ - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_JOB_MGR_CTRL); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_JOB_MGR_CTRL); reg_val64 |= ROC_ML_JOB_MGR_CTRL_STALL_ON_IDLE; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_JOB_MGR_CTRL); - plt_ml_dbg("ML_JOB_MGR_CTRL => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_JOB_MGR_CTRL)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_JOB_MGR_CTRL); + plt_ml_dbg("ML_JOB_MGR_CTRL => 0x%016lx", + roc_ml_reg_read64(&cn10k_mldev->roc, ML_JOB_MGR_CTRL)); /* (14) Set ML(0)_CFG[MLIP_CLK_FORCE] = 0; the MLIP clock will be turned on/off based on job * activities. */ - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_CFG); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); reg_val64 &= ~ROC_ML_CFG_MLIP_CLK_FORCE; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); - plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_CFG)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); + plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); /* (15) Set ML(0)_CFG[ENA] to enable ML job execution. */ - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_CFG); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); reg_val64 |= ROC_ML_CFG_ENA; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); - plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_CFG)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); + plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); /* Reset scratch registers */ - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_FW_CTRL); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_WORK_PTR); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_FW_CTRL); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_WORK_PTR); /* Disable job execution, to be enabled in start */ - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_CFG); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); reg_val64 &= ~ROC_ML_CFG_ENA; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); - plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_CFG)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); + plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); /* Additional fixes: Set RO bit to fix O2D DMA bandwidth issue on cn10ka */ for (i = 0; i < ML_ANBX_NR; i++) { - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_ANBX_NCBI_P_OVR(i)); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_ANBX_NCBI_P_OVR(i)); reg_val64 |= (ML_ANBX_NCBI_P_OVR_ANB_NCBI_P_RO_OVR | ML_ANBX_NCBI_P_OVR_ANB_NCBI_P_RO_OVR_VLD); - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_ANBX_NCBI_P_OVR(i)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_ANBX_NCBI_P_OVR(i)); } return ret; } int -cn10k_ml_fw_load(struct cn10k_ml_dev *mldev) +cn10k_ml_fw_load(struct cnxk_ml_dev *cnxk_mldev) { + struct cn10k_ml_dev *cn10k_mldev; const struct plt_memzone *mz; struct cn10k_ml_fw *fw; char *fw_buffer = NULL; @@ -797,8 +810,9 @@ cn10k_ml_fw_load(struct cn10k_ml_dev *mldev) uint64_t fw_size = 0; int ret = 0; - fw = &mldev->fw; - fw->mldev = mldev; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + fw = &cn10k_mldev->fw; + fw->cn10k_mldev = cn10k_mldev; if (roc_env_is_emulator() || roc_env_is_hw()) { /* Read firmware image to a buffer */ @@ -829,8 +843,8 @@ cn10k_ml_fw_load(struct cn10k_ml_dev *mldev) memset(&fw->req->jd.fw_load.version[0], '\0', MLDEV_FIRMWARE_VERSION_LENGTH); /* Reset device, if in active state */ - if (roc_ml_mlip_is_enabled(&mldev->roc)) - roc_ml_mlip_reset(&mldev->roc, true); + if (roc_ml_mlip_is_enabled(&cn10k_mldev->roc)) + roc_ml_mlip_reset(&cn10k_mldev->roc, true); /* Load firmware */ if (roc_env_is_emulator() || roc_env_is_hw()) { @@ -843,22 +857,25 @@ cn10k_ml_fw_load(struct cn10k_ml_dev *mldev) } if (ret < 0) - cn10k_ml_fw_unload(mldev); + cn10k_ml_fw_unload(cnxk_mldev); return ret; } void -cn10k_ml_fw_unload(struct cn10k_ml_dev *mldev) +cn10k_ml_fw_unload(struct cnxk_ml_dev *cnxk_mldev) { + struct cn10k_ml_dev *cn10k_mldev; const struct plt_memzone *mz; uint64_t reg_val; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + /* Disable and reset device */ - reg_val = roc_ml_reg_read64(&mldev->roc, ML_CFG); + reg_val = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); reg_val &= ~ROC_ML_CFG_MLIP_ENA; - roc_ml_reg_write64(&mldev->roc, reg_val, ML_CFG); - roc_ml_mlip_reset(&mldev->roc, true); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val, ML_CFG); + roc_ml_mlip_reset(&cn10k_mldev->roc, true); mz = plt_memzone_lookup(FW_MEMZONE_NAME); if (mz != NULL) diff --git a/drivers/ml/cnxk/cn10k_ml_dev.h b/drivers/ml/cnxk/cn10k_ml_dev.h index 4aaeecff03d..f9da1548c4a 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.h +++ b/drivers/ml/cnxk/cn10k_ml_dev.h @@ -9,6 +9,9 @@ #include "cn10k_ml_ocm.h" +/* Dummy Device ops */ +extern struct rte_ml_dev_ops ml_dev_dummy_ops; + /* Marvell OCTEON CN10K ML PMD device name */ #define MLDEV_NAME_CN10K_PMD ml_cn10k @@ -36,17 +39,10 @@ /* Maximum number of segments for IO data */ #define ML_CN10K_MAX_SEGMENTS 1 -/* ML command timeout in seconds */ -#define ML_CN10K_CMD_TIMEOUT 5 - /* ML slow-path job flags */ #define ML_CN10K_SP_FLAGS_OCM_NONRELOCATABLE BIT(0) #define ML_CN10K_SP_FLAGS_EXTENDED_LOAD_JD BIT(1) -/* Poll mode job state */ -#define ML_CN10K_POLL_JOB_START 0 -#define ML_CN10K_POLL_JOB_FINISH 1 - /* Memory barrier macros */ #if defined(RTE_ARCH_ARM) #define dmb_st ({ asm volatile("dmb st" : : : "memory"); }) @@ -56,6 +52,7 @@ #define dsb_st #endif +struct cnxk_ml_dev; struct cn10k_ml_req; struct cn10k_ml_qp; @@ -68,21 +65,6 @@ enum cn10k_ml_job_type { ML_CN10K_JOB_TYPE_FIRMWARE_SELFTEST, }; -/* Device configuration state enum */ -enum cn10k_ml_dev_state { - /* Probed and not configured */ - ML_CN10K_DEV_STATE_PROBED = 0, - - /* Configured */ - ML_CN10K_DEV_STATE_CONFIGURED, - - /* Started */ - ML_CN10K_DEV_STATE_STARTED, - - /* Closed */ - ML_CN10K_DEV_STATE_CLOSED -}; - /* Error types enumeration */ enum cn10k_ml_error_etype { /* 0x0 */ ML_ETYPE_NO_ERROR = 0, /* No error */ @@ -379,7 +361,7 @@ struct cn10k_ml_jd { /* ML firmware structure */ struct cn10k_ml_fw { /* Device reference */ - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; /* Firmware file path */ const char *path; @@ -485,27 +467,12 @@ struct cn10k_ml_dev { /* Device ROC */ struct roc_ml roc; - /* Configuration state */ - enum cn10k_ml_dev_state state; - /* Firmware */ struct cn10k_ml_fw fw; /* OCM info */ struct cn10k_ml_ocm ocm; - /* Number of models loaded */ - uint16_t nb_models_loaded; - - /* Number of models unloaded */ - uint16_t nb_models_unloaded; - - /* Number of models started */ - uint16_t nb_models_started; - - /* Number of models stopped */ - uint16_t nb_models_stopped; - /* Extended stats data */ struct cn10k_ml_xstats xstats; @@ -528,7 +495,7 @@ struct cn10k_ml_dev { }; uint64_t cn10k_ml_fw_flags_get(struct cn10k_ml_fw *fw); -int cn10k_ml_fw_load(struct cn10k_ml_dev *mldev); -void cn10k_ml_fw_unload(struct cn10k_ml_dev *mldev); +int cn10k_ml_fw_load(struct cnxk_ml_dev *cnxk_mldev); +void cn10k_ml_fw_unload(struct cnxk_ml_dev *cnxk_mldev); #endif /* _CN10K_ML_DEV_H_ */ diff --git a/drivers/ml/cnxk/cn10k_ml_model.c b/drivers/ml/cnxk/cn10k_ml_model.c index e0b750cd8ef..d146535866a 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.c +++ b/drivers/ml/cnxk/cn10k_ml_model.c @@ -10,6 +10,8 @@ #include "cn10k_ml_model.h" #include "cn10k_ml_ocm.h" +#include "cnxk_ml_dev.h" + static enum rte_ml_io_type cn10k_ml_io_type_map(uint8_t type) { @@ -461,7 +463,7 @@ cn10k_ml_model_addr_update(struct cn10k_ml_model *model, uint8_t *buffer, uint8_ } int -cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *mldev, uint16_t model_id, uint8_t *buffer, +cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_id, uint8_t *buffer, uint16_t *wb_pages, uint16_t *scratch_pages) { struct cn10k_ml_model_metadata *metadata; @@ -470,7 +472,7 @@ cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *mldev, uint16_t model_id, ui uint64_t wb_size; metadata = (struct cn10k_ml_model_metadata *)buffer; - ocm = &mldev->ocm; + ocm = &cn10k_mldev->ocm; /* Assume wb_size is zero for non-relocatable models */ if (metadata->model.ocm_relocatable) @@ -494,11 +496,11 @@ cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *mldev, uint16_t model_id, ui scratch_size, *scratch_pages); /* Check if the model can be loaded on OCM */ - if ((*wb_pages + *scratch_pages) > mldev->ocm.num_pages) { + if ((*wb_pages + *scratch_pages) > cn10k_mldev->ocm.num_pages) { plt_err("Cannot create the model, OCM relocatable = %u", metadata->model.ocm_relocatable); plt_err("wb_pages (%u) + scratch_pages (%u) > %u", *wb_pages, *scratch_pages, - mldev->ocm.num_pages); + cn10k_mldev->ocm.num_pages); return -ENOMEM; } @@ -506,8 +508,8 @@ cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *mldev, uint16_t model_id, ui * prevent the library from allocating the remaining space on the tile to other models. */ if (!metadata->model.ocm_relocatable) - *scratch_pages = - PLT_MAX(PLT_U64_CAST(*scratch_pages), PLT_U64_CAST(mldev->ocm.num_pages)); + *scratch_pages = PLT_MAX(PLT_U64_CAST(*scratch_pages), + PLT_U64_CAST(cn10k_mldev->ocm.num_pages)); return 0; } diff --git a/drivers/ml/cnxk/cn10k_ml_model.h b/drivers/ml/cnxk/cn10k_ml_model.h index 4cc0744891b..3128b28db73 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.h +++ b/drivers/ml/cnxk/cn10k_ml_model.h @@ -13,6 +13,8 @@ #include "cn10k_ml_ocm.h" #include "cn10k_ml_ops.h" +struct cnxk_ml_dev; + /* Model state */ enum cn10k_ml_model_state { ML_CN10K_MODEL_STATE_LOADED, @@ -489,7 +491,7 @@ struct cn10k_ml_model_stats { /* Model Object */ struct cn10k_ml_model { /* Device reference */ - struct cn10k_ml_dev *mldev; + struct cnxk_ml_dev *mldev; /* Name */ char name[RTE_ML_STR_MAX]; @@ -537,8 +539,8 @@ int cn10k_ml_model_metadata_check(uint8_t *buffer, uint64_t size); void cn10k_ml_model_metadata_update(struct cn10k_ml_model_metadata *metadata); void cn10k_ml_model_addr_update(struct cn10k_ml_model *model, uint8_t *buffer, uint8_t *base_dma_addr); -int cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *mldev, uint16_t model_id, uint8_t *buffer, - uint16_t *wb_pages, uint16_t *scratch_pages); +int cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_id, + uint8_t *buffer, uint16_t *wb_pages, uint16_t *scratch_pages); void cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cn10k_ml_model *model); #endif /* _CN10K_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/cn10k_ml_ocm.c b/drivers/ml/cnxk/cn10k_ml_ocm.c index 93505c9c09b..d0f716bccea 100644 --- a/drivers/ml/cnxk/cn10k_ml_ocm.c +++ b/drivers/ml/cnxk/cn10k_ml_ocm.c @@ -4,11 +4,13 @@ #include +#include + #include "cn10k_ml_dev.h" #include "cn10k_ml_model.h" #include "cn10k_ml_ocm.h" -#include "roc_api.h" +#include "cnxk_ml_dev.h" /* OCM macros */ #define BYTE_LEN 8 @@ -217,7 +219,8 @@ int cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t wb_pages, uint16_t scratch_pages, uint64_t *tilemask) { - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_ocm *ocm; uint16_t used_scratch_pages_max; @@ -236,8 +239,9 @@ cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t w int max_slot_sz; int page_id; - mldev = dev->data->dev_private; - ocm = &mldev->ocm; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + ocm = &cn10k_mldev->ocm; if (num_tiles > ML_CN10K_OCM_NUMTILES) { plt_err("Invalid num_tiles = %u (> %u)", num_tiles, ML_CN10K_OCM_NUMTILES); @@ -254,8 +258,8 @@ cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t w tile_start = 0; search_end_tile = ocm->num_tiles - num_tiles; - /* allocate for local ocm mask */ - local_ocm_mask = rte_zmalloc("local_ocm_mask", mldev->ocm.mask_words, RTE_CACHE_LINE_SIZE); + /* Allocate for local ocm mask */ + local_ocm_mask = rte_zmalloc("local_ocm_mask", ocm->mask_words, RTE_CACHE_LINE_SIZE); if (local_ocm_mask == NULL) { plt_err("Unable to allocate memory for local_ocm_mask"); return -1; @@ -271,7 +275,7 @@ cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t w PLT_MAX(ocm->tile_ocm_info[tile_id].last_wb_page, used_last_wb_page_max); } - memset(local_ocm_mask, 0, mldev->ocm.mask_words); + memset(local_ocm_mask, 0, ocm->mask_words); for (tile_id = tile_start; tile_id < tile_start + num_tiles; tile_id++) { for (word_id = 0; word_id < ocm->mask_words; word_id++) local_ocm_mask[word_id] |= ocm->tile_ocm_info[tile_id].ocm_mask[word_id]; @@ -333,8 +337,9 @@ void cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint64_t tilemask, int wb_page_start, uint16_t wb_pages, uint16_t scratch_pages) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_ocm *ocm; int scratch_page_start; @@ -345,8 +350,9 @@ cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint64_t t int tile_id; int page_id; - mldev = dev->data->dev_private; - ocm = &mldev->ocm; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + ocm = &cn10k_mldev->ocm; model = dev->data->models[model_id]; /* Get first set bit, tile_start */ @@ -391,8 +397,9 @@ void cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id) { struct cn10k_ml_model *local_model; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_ocm *ocm; int scratch_resize_pages; @@ -404,8 +411,9 @@ cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id) int page_id; uint16_t i; - mldev = dev->data->dev_private; - ocm = &mldev->ocm; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + ocm = &cn10k_mldev->ocm; model = dev->data->models[model_id]; /* Update OCM info for WB memory */ @@ -453,35 +461,37 @@ cn10k_ml_ocm_pagemask_to_str(struct cn10k_ml_ocm_tile_info *tile_info, uint16_t char *p = str; int word; - /* add prefix 0x */ + /* Add prefix 0x */ *p++ = '0'; *p++ = 'x'; - /* build one word at a time */ + /* Build hex string */ for (word = nwords - 1; word >= 0; word--) { sprintf(p, "%02X", tile_info->ocm_mask[word]); p += 2; } - /* terminate */ + /* Terminate */ *p++ = 0; } void cn10k_ml_ocm_print(struct rte_ml_dev *dev, FILE *fp) { - char *str; - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_ocm *ocm; uint8_t tile_id; uint8_t word_id; int wb_pages; + char *str; - mldev = dev->data->dev_private; - ocm = &mldev->ocm; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + ocm = &cn10k_mldev->ocm; - /* nibbles + prefix '0x' */ - str = rte_zmalloc("ocm_mask_str", mldev->ocm.num_pages / 4 + 2, RTE_CACHE_LINE_SIZE); + /* Nibbles + prefix '0x' */ + str = rte_zmalloc("ocm_mask_str", ocm->num_pages / 4 + 2, RTE_CACHE_LINE_SIZE); if (str == NULL) { plt_err("Unable to allocate memory for ocm_mask_str"); return; @@ -492,7 +502,7 @@ cn10k_ml_ocm_print(struct rte_ml_dev *dev, FILE *fp) cn10k_ml_ocm_pagemask_to_str(&ocm->tile_ocm_info[tile_id], ocm->mask_words, str); wb_pages = 0 - ocm->tile_ocm_info[tile_id].scratch_pages; - for (word_id = 0; word_id < mldev->ocm.mask_words; word_id++) + for (word_id = 0; word_id < ocm->mask_words; word_id++) wb_pages += __builtin_popcount(ocm->tile_ocm_info[tile_id].ocm_mask[word_id]); diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 11531afd8c1..3385bf50c0d 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -11,6 +11,8 @@ #include "cn10k_ml_model.h" #include "cn10k_ml_ops.h" +#include "cnxk_ml_dev.h" + /* ML model macros */ #define CN10K_ML_MODEL_MEMZONE_NAME "ml_cn10k_model_mz" @@ -85,7 +87,7 @@ cn10k_ml_set_poll_addr(struct cn10k_ml_req *req) static inline void cn10k_ml_set_poll_ptr(struct cn10k_ml_req *req) { - plt_write64(ML_CN10K_POLL_JOB_START, req->compl_W1); + plt_write64(ML_CNXK_POLL_JOB_START, req->compl_W1); } static inline uint64_t @@ -175,7 +177,7 @@ cn10k_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_des qp->queue.reqs = (struct cn10k_ml_req *)va; qp->queue.head = 0; qp->queue.tail = 0; - qp->queue.wait_cycles = ML_CN10K_CMD_TIMEOUT * plt_tsc_hz(); + qp->queue.wait_cycles = ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); qp->nb_desc = nb_desc; qp->stats.enqueued_count = 0; qp->stats.dequeued_count = 0; @@ -199,16 +201,17 @@ cn10k_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_des static void cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) { - + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_ocm *ocm; char str[STR_LEN]; uint8_t i; uint8_t j; - mldev = dev->data->dev_private; - ocm = &mldev->ocm; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + ocm = &cn10k_mldev->ocm; model = dev->data->models[model_id]; /* Print debug info */ @@ -249,7 +252,7 @@ cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) fprintf(fp, "%*s : 0x%0*" PRIx64 "\n", FIELD_LEN, "tilemask", ML_CN10K_OCM_NUMTILES / 4, model->model_mem_map.tilemask); fprintf(fp, "%*s : 0x%" PRIx64 "\n", FIELD_LEN, "ocm_wb_start", - model->model_mem_map.wb_page_start * mldev->ocm.page_size); + model->model_mem_map.wb_page_start * cn10k_mldev->ocm.page_size); } fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_inputs", model->metadata.model.num_input); @@ -325,7 +328,7 @@ cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) } static void -cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *mldev, struct cn10k_ml_model *model, +cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cn10k_ml_model *model, struct cn10k_ml_req *req, enum cn10k_ml_job_type job_type) { struct cn10k_ml_model_metadata *metadata; @@ -340,7 +343,7 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *mldev, struct cn10k_ml_mode req->jd.hdr.model_id = model->model_id; req->jd.hdr.job_type = job_type; req->jd.hdr.fp_flags = 0x0; - req->jd.hdr.result = roc_ml_addr_ap2mlip(&mldev->roc, &req->result); + req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->result); if (job_type == ML_CN10K_JOB_TYPE_MODEL_START) { if (!model->metadata.model.ocm_relocatable) @@ -350,9 +353,9 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *mldev, struct cn10k_ml_mode req->jd.hdr.sp_flags |= ML_CN10K_SP_FLAGS_EXTENDED_LOAD_JD; req->jd.model_start.extended_args = - PLT_U64_CAST(roc_ml_addr_ap2mlip(&mldev->roc, &req->extended_args)); + PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->extended_args)); req->jd.model_start.model_dst_ddr_addr = - PLT_U64_CAST(roc_ml_addr_ap2mlip(&mldev->roc, addr->init_run_addr)); + PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, addr->init_run_addr)); req->jd.model_start.model_init_offset = 0x0; req->jd.model_start.model_main_offset = metadata->init_model.file_size; req->jd.model_start.model_finish_offset = @@ -372,7 +375,7 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *mldev, struct cn10k_ml_mode req->jd.model_start.ocm_wb_range_start = metadata->model.ocm_wb_range_start; req->jd.model_start.ocm_wb_range_end = metadata->model.ocm_wb_range_end; req->jd.model_start.ddr_wb_base_address = PLT_U64_CAST(roc_ml_addr_ap2mlip( - &mldev->roc, + &cn10k_mldev->roc, PLT_PTR_ADD(addr->finish_load_addr, metadata->finish_model.file_size))); req->jd.model_start.ddr_wb_range_start = metadata->model.ddr_wb_range_start; req->jd.model_start.ddr_wb_range_end = metadata->model.ddr_wb_range_end; @@ -383,7 +386,7 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *mldev, struct cn10k_ml_mode req->jd.model_start.output.s.ddr_range_end = metadata->model.ddr_output_range_end; req->extended_args.start.ddr_scratch_base_address = PLT_U64_CAST( - roc_ml_addr_ap2mlip(&mldev->roc, model->addr.scratch_base_addr)); + roc_ml_addr_ap2mlip(&cn10k_mldev->roc, model->addr.scratch_base_addr)); req->extended_args.start.ddr_scratch_range_start = metadata->model.ddr_scratch_range_start; req->extended_args.start.ddr_scratch_range_end = @@ -392,24 +395,20 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *mldev, struct cn10k_ml_mode } static __rte_always_inline void -cn10k_ml_prep_fp_job_descriptor(struct rte_ml_dev *dev, struct cn10k_ml_req *req, +cn10k_ml_prep_fp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cn10k_ml_req *req, struct rte_ml_op *op) { - struct cn10k_ml_dev *mldev; - - mldev = dev->data->dev_private; - req->jd.hdr.jce.w0.u64 = 0; req->jd.hdr.jce.w1.u64 = req->compl_W1; req->jd.hdr.model_id = op->model_id; req->jd.hdr.job_type = ML_CN10K_JOB_TYPE_MODEL_RUN; req->jd.hdr.fp_flags = ML_FLAGS_POLL_COMPL; req->jd.hdr.sp_flags = 0x0; - req->jd.hdr.result = roc_ml_addr_ap2mlip(&mldev->roc, &req->result); + req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->result); req->jd.model_run.input_ddr_addr = - PLT_U64_CAST(roc_ml_addr_ap2mlip(&mldev->roc, op->input[0]->addr)); + PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, op->input[0]->addr)); req->jd.model_run.output_ddr_addr = - PLT_U64_CAST(roc_ml_addr_ap2mlip(&mldev->roc, op->output[0]->addr)); + PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, op->output[0]->addr)); req->jd.model_run.num_batches = op->nb_batches; } @@ -436,66 +435,69 @@ static const struct xstat_info model_stats[] = { static int cn10k_ml_xstats_init(struct rte_ml_dev *dev) { - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; uint16_t nb_stats; uint16_t stat_id; uint16_t model; uint16_t i; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; /* Allocate memory for xstats entries. Don't allocate during reconfigure */ nb_stats = RTE_DIM(device_stats) + ML_CN10K_MAX_MODELS * RTE_DIM(model_stats); - if (mldev->xstats.entries == NULL) - mldev->xstats.entries = rte_zmalloc("cn10k_ml_xstats", - sizeof(struct cn10k_ml_xstats_entry) * nb_stats, - PLT_CACHE_LINE_SIZE); + if (cn10k_mldev->xstats.entries == NULL) + cn10k_mldev->xstats.entries = rte_zmalloc( + "cn10k_ml_xstats", sizeof(struct cn10k_ml_xstats_entry) * nb_stats, + PLT_CACHE_LINE_SIZE); - if (mldev->xstats.entries == NULL) + if (cn10k_mldev->xstats.entries == NULL) return -ENOMEM; /* Initialize device xstats */ stat_id = 0; for (i = 0; i < RTE_DIM(device_stats); i++) { - mldev->xstats.entries[stat_id].map.id = stat_id; - snprintf(mldev->xstats.entries[stat_id].map.name, - sizeof(mldev->xstats.entries[stat_id].map.name), "%s", + cn10k_mldev->xstats.entries[stat_id].map.id = stat_id; + snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, + sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), "%s", device_stats[i].name); - mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_DEVICE; - mldev->xstats.entries[stat_id].type = device_stats[i].type; - mldev->xstats.entries[stat_id].fn_id = CN10K_ML_XSTATS_FN_DEVICE; - mldev->xstats.entries[stat_id].obj_idx = 0; - mldev->xstats.entries[stat_id].reset_allowed = device_stats[i].reset_allowed; + cn10k_mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_DEVICE; + cn10k_mldev->xstats.entries[stat_id].type = device_stats[i].type; + cn10k_mldev->xstats.entries[stat_id].fn_id = CN10K_ML_XSTATS_FN_DEVICE; + cn10k_mldev->xstats.entries[stat_id].obj_idx = 0; + cn10k_mldev->xstats.entries[stat_id].reset_allowed = device_stats[i].reset_allowed; stat_id++; } - mldev->xstats.count_mode_device = stat_id; + cn10k_mldev->xstats.count_mode_device = stat_id; /* Initialize model xstats */ for (model = 0; model < ML_CN10K_MAX_MODELS; model++) { - mldev->xstats.offset_for_model[model] = stat_id; + cn10k_mldev->xstats.offset_for_model[model] = stat_id; for (i = 0; i < RTE_DIM(model_stats); i++) { - mldev->xstats.entries[stat_id].map.id = stat_id; - mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_MODEL; - mldev->xstats.entries[stat_id].type = model_stats[i].type; - mldev->xstats.entries[stat_id].fn_id = CN10K_ML_XSTATS_FN_MODEL; - mldev->xstats.entries[stat_id].obj_idx = model; - mldev->xstats.entries[stat_id].reset_allowed = model_stats[i].reset_allowed; + cn10k_mldev->xstats.entries[stat_id].map.id = stat_id; + cn10k_mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_MODEL; + cn10k_mldev->xstats.entries[stat_id].type = model_stats[i].type; + cn10k_mldev->xstats.entries[stat_id].fn_id = CN10K_ML_XSTATS_FN_MODEL; + cn10k_mldev->xstats.entries[stat_id].obj_idx = model; + cn10k_mldev->xstats.entries[stat_id].reset_allowed = + model_stats[i].reset_allowed; /* Name of xstat is updated during model load */ - snprintf(mldev->xstats.entries[stat_id].map.name, - sizeof(mldev->xstats.entries[stat_id].map.name), "Model-%u-%s", - model, model_stats[i].name); + snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, + sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), + "Model-%u-%s", model, model_stats[i].name); stat_id++; } - mldev->xstats.count_per_model[model] = RTE_DIM(model_stats); + cn10k_mldev->xstats.count_per_model[model] = RTE_DIM(model_stats); } - mldev->xstats.count_mode_model = stat_id - mldev->xstats.count_mode_device; - mldev->xstats.count = stat_id; + cn10k_mldev->xstats.count_mode_model = stat_id - cn10k_mldev->xstats.count_mode_device; + cn10k_mldev->xstats.count = stat_id; return 0; } @@ -503,28 +505,32 @@ cn10k_ml_xstats_init(struct rte_ml_dev *dev) static void cn10k_ml_xstats_uninit(struct rte_ml_dev *dev) { - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; - rte_free(mldev->xstats.entries); - mldev->xstats.entries = NULL; + rte_free(cn10k_mldev->xstats.entries); + cn10k_mldev->xstats.entries = NULL; - mldev->xstats.count = 0; + cn10k_mldev->xstats.count = 0; } static void cn10k_ml_xstats_model_name_update(struct rte_ml_dev *dev, uint16_t model_id) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; uint16_t rclk_freq; uint16_t sclk_freq; uint16_t stat_id; char suffix[8]; uint16_t i; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; model = dev->data->models[model_id]; stat_id = RTE_DIM(device_stats) + model_id * RTE_DIM(model_stats); @@ -536,8 +542,8 @@ cn10k_ml_xstats_model_name_update(struct rte_ml_dev *dev, uint16_t model_id) /* Update xstat name based on model name and sclk availability */ for (i = 0; i < RTE_DIM(model_stats); i++) { - snprintf(mldev->xstats.entries[stat_id].map.name, - sizeof(mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", + snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, + sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", model->metadata.model.name, model_stats[i].name, suffix); stat_id++; } @@ -547,19 +553,19 @@ static uint64_t cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, enum cn10k_ml_xstats_type type) { - struct cn10k_ml_dev *mldev; + struct cnxk_ml_dev *cnxk_mldev; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; switch (type) { case nb_models_loaded: - return mldev->nb_models_loaded; + return cnxk_mldev->nb_models_loaded; case nb_models_unloaded: - return mldev->nb_models_unloaded; + return cnxk_mldev->nb_models_unloaded; case nb_models_started: - return mldev->nb_models_started; + return cnxk_mldev->nb_models_started; case nb_models_stopped: - return mldev->nb_models_stopped; + return cnxk_mldev->nb_models_stopped; default: return -1; } @@ -651,15 +657,17 @@ static int cn10k_ml_device_xstats_reset(struct rte_ml_dev *dev, const uint16_t stat_ids[], uint16_t nb_ids) { struct cn10k_ml_xstats_entry *xs; - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; uint16_t nb_stats; uint16_t stat_id; uint32_t i; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; if (stat_ids == NULL) - nb_stats = mldev->xstats.count_mode_device; + nb_stats = cn10k_mldev->xstats.count_mode_device; else nb_stats = nb_ids; @@ -669,10 +677,10 @@ cn10k_ml_device_xstats_reset(struct rte_ml_dev *dev, const uint16_t stat_ids[], else stat_id = stat_ids[i]; - if (stat_id >= mldev->xstats.count_mode_device) + if (stat_id >= cn10k_mldev->xstats.count_mode_device) return -EINVAL; - xs = &mldev->xstats.entries[stat_id]; + xs = &cn10k_mldev->xstats.entries[stat_id]; if (!xs->reset_allowed) continue; @@ -740,15 +748,17 @@ cn10k_ml_model_xstats_reset(struct rte_ml_dev *dev, int32_t model_id, const uint uint16_t nb_ids) { struct cn10k_ml_xstats_entry *xs; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; int32_t lcl_model_id = 0; uint16_t start_id; uint16_t end_id; int32_t i; int32_t j; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; for (i = 0; i < ML_CN10K_MAX_MODELS; i++) { if (model_id == -1) { model = dev->data->models[i]; @@ -765,12 +775,13 @@ cn10k_ml_model_xstats_reset(struct rte_ml_dev *dev, int32_t model_id, const uint } } - start_id = mldev->xstats.offset_for_model[i]; - end_id = mldev->xstats.offset_for_model[i] + mldev->xstats.count_per_model[i] - 1; + start_id = cn10k_mldev->xstats.offset_for_model[i]; + end_id = cn10k_mldev->xstats.offset_for_model[i] + + cn10k_mldev->xstats.count_per_model[i] - 1; if (stat_ids == NULL) { for (j = start_id; j <= end_id; j++) { - xs = &mldev->xstats.entries[j]; + xs = &cn10k_mldev->xstats.entries[j]; cn10k_ml_reset_model_stat(dev, i, xs->type); } } else { @@ -780,7 +791,7 @@ cn10k_ml_model_xstats_reset(struct rte_ml_dev *dev, int32_t model_id, const uint stat_ids[j], lcl_model_id); return -EINVAL; } - xs = &mldev->xstats.entries[stat_ids[j]]; + xs = &cn10k_mldev->xstats.entries[stat_ids[j]]; cn10k_ml_reset_model_stat(dev, i, xs->type); } } @@ -854,17 +865,19 @@ cn10k_ml_cache_model_data(struct rte_ml_dev *dev, uint16_t model_id) static int cn10k_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) { - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; if (dev_info == NULL) return -EINVAL; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; memset(dev_info, 0, sizeof(struct rte_ml_dev_info)); dev_info->driver_name = dev->device->driver->name; dev_info->max_models = ML_CN10K_MAX_MODELS; - if (mldev->hw_queue_lock) + if (cn10k_mldev->hw_queue_lock) dev_info->max_queue_pairs = ML_CN10K_MAX_QP_PER_DEVICE_SL; else dev_info->max_queue_pairs = ML_CN10K_MAX_QP_PER_DEVICE_LF; @@ -881,8 +894,9 @@ static int cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *conf) { struct rte_ml_dev_info dev_info; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_ocm *ocm; struct cn10k_ml_qp *qp; uint16_t model_id; @@ -895,7 +909,8 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c return -EINVAL; /* Get CN10K device handle */ - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; cn10k_ml_dev_info_get(dev, &dev_info); if (conf->nb_models > dev_info.max_models) { @@ -908,21 +923,21 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c return -EINVAL; } - if (mldev->state == ML_CN10K_DEV_STATE_PROBED) { + if (cnxk_mldev->state == ML_CNXK_DEV_STATE_PROBED) { plt_ml_dbg("Configuring ML device, nb_queue_pairs = %u, nb_models = %u", conf->nb_queue_pairs, conf->nb_models); /* Load firmware */ - ret = cn10k_ml_fw_load(mldev); + ret = cn10k_ml_fw_load(cnxk_mldev); if (ret != 0) return ret; - } else if (mldev->state == ML_CN10K_DEV_STATE_CONFIGURED) { + } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_CONFIGURED) { plt_ml_dbg("Re-configuring ML device, nb_queue_pairs = %u, nb_models = %u", conf->nb_queue_pairs, conf->nb_models); - } else if (mldev->state == ML_CN10K_DEV_STATE_STARTED) { + } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_STARTED) { plt_err("Device can't be reconfigured in started state\n"); return -ENOTSUP; - } else if (mldev->state == ML_CN10K_DEV_STATE_CLOSED) { + } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_CLOSED) { plt_err("Device can't be reconfigured after close\n"); return -ENOTSUP; } @@ -1013,10 +1028,10 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c } dev->data->nb_models = conf->nb_models; - ocm = &mldev->ocm; + ocm = &cn10k_mldev->ocm; ocm->num_tiles = ML_CN10K_OCM_NUMTILES; ocm->size_per_tile = ML_CN10K_OCM_TILESIZE; - ocm->page_size = mldev->ocm_page_size; + ocm->page_size = cn10k_mldev->ocm_page_size; ocm->num_pages = ocm->size_per_tile / ocm->page_size; ocm->mask_words = ocm->num_pages / (8 * sizeof(uint8_t)); @@ -1044,25 +1059,25 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c } /* Set JCMDQ enqueue function */ - if (mldev->hw_queue_lock == 1) - mldev->ml_jcmdq_enqueue = roc_ml_jcmdq_enqueue_sl; + if (cn10k_mldev->hw_queue_lock == 1) + cn10k_mldev->ml_jcmdq_enqueue = roc_ml_jcmdq_enqueue_sl; else - mldev->ml_jcmdq_enqueue = roc_ml_jcmdq_enqueue_lf; + cn10k_mldev->ml_jcmdq_enqueue = roc_ml_jcmdq_enqueue_lf; /* Set polling function pointers */ - mldev->set_poll_addr = cn10k_ml_set_poll_addr; - mldev->set_poll_ptr = cn10k_ml_set_poll_ptr; - mldev->get_poll_ptr = cn10k_ml_get_poll_ptr; + cn10k_mldev->set_poll_addr = cn10k_ml_set_poll_addr; + cn10k_mldev->set_poll_ptr = cn10k_ml_set_poll_ptr; + cn10k_mldev->get_poll_ptr = cn10k_ml_get_poll_ptr; dev->enqueue_burst = cn10k_ml_enqueue_burst; dev->dequeue_burst = cn10k_ml_dequeue_burst; dev->op_error_get = cn10k_ml_op_error_get; - mldev->nb_models_loaded = 0; - mldev->nb_models_started = 0; - mldev->nb_models_stopped = 0; - mldev->nb_models_unloaded = 0; - mldev->state = ML_CN10K_DEV_STATE_CONFIGURED; + cnxk_mldev->nb_models_loaded = 0; + cnxk_mldev->nb_models_started = 0; + cnxk_mldev->nb_models_stopped = 0; + cnxk_mldev->nb_models_unloaded = 0; + cnxk_mldev->state = ML_CNXK_DEV_STATE_CONFIGURED; return 0; @@ -1077,8 +1092,9 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c static int cn10k_ml_dev_close(struct rte_ml_dev *dev) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_qp *qp; uint16_t model_id; uint16_t qp_id; @@ -1086,10 +1102,11 @@ cn10k_ml_dev_close(struct rte_ml_dev *dev) if (dev == NULL) return -EINVAL; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; /* Release ocm_mask memory */ - rte_free(mldev->ocm.ocm_mask); + rte_free(cn10k_mldev->ocm.ocm_mask); /* Stop and unload all models */ for (model_id = 0; model_id < dev->data->nb_models; model_id++) { @@ -1125,21 +1142,21 @@ cn10k_ml_dev_close(struct rte_ml_dev *dev) cn10k_ml_xstats_uninit(dev); /* Unload firmware */ - cn10k_ml_fw_unload(mldev); + cn10k_ml_fw_unload(cnxk_mldev); /* Clear scratch registers */ - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_WORK_PTR); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_FW_CTRL); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C0); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C0); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C1); - roc_ml_reg_write64(&mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C1); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_WORK_PTR); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_FW_CTRL); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C0); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C0); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_HEAD_C1); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_SCRATCH_DBG_BUFFER_TAIL_C1); /* Reset ML_MLR_BASE */ - roc_ml_reg_write64(&mldev->roc, 0, ML_MLR_BASE); - plt_ml_dbg("ML_MLR_BASE = 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_MLR_BASE)); + roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_MLR_BASE); + plt_ml_dbg("ML_MLR_BASE = 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_MLR_BASE)); - mldev->state = ML_CN10K_DEV_STATE_CLOSED; + cnxk_mldev->state = ML_CNXK_DEV_STATE_CLOSED; /* Remove PCI device */ return rte_dev_remove(dev->device); @@ -1148,17 +1165,19 @@ cn10k_ml_dev_close(struct rte_ml_dev *dev) static int cn10k_ml_dev_start(struct rte_ml_dev *dev) { - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; uint64_t reg_val64; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_CFG); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); reg_val64 |= ROC_ML_CFG_ENA; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); - plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_CFG)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); + plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); - mldev->state = ML_CN10K_DEV_STATE_STARTED; + cnxk_mldev->state = ML_CNXK_DEV_STATE_STARTED; return 0; } @@ -1166,17 +1185,19 @@ cn10k_ml_dev_start(struct rte_ml_dev *dev) static int cn10k_ml_dev_stop(struct rte_ml_dev *dev) { - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; uint64_t reg_val64; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; - reg_val64 = roc_ml_reg_read64(&mldev->roc, ML_CFG); + reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); reg_val64 &= ~ROC_ML_CFG_ENA; - roc_ml_reg_write64(&mldev->roc, reg_val64, ML_CFG); - plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&mldev->roc, ML_CFG)); + roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); + plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); - mldev->state = ML_CN10K_DEV_STATE_CONFIGURED; + cnxk_mldev->state = ML_CNXK_DEV_STATE_CONFIGURED; return 0; } @@ -1259,22 +1280,24 @@ cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mod int32_t model_id, struct rte_ml_dev_xstats_map *xstats_map, uint32_t size) { - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; uint32_t xstats_mode_count; uint32_t idx = 0; uint32_t i; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; xstats_mode_count = 0; switch (mode) { case RTE_ML_DEV_XSTATS_DEVICE: - xstats_mode_count = mldev->xstats.count_mode_device; + xstats_mode_count = cn10k_mldev->xstats.count_mode_device; break; case RTE_ML_DEV_XSTATS_MODEL: if (model_id >= ML_CN10K_MAX_MODELS) break; - xstats_mode_count = mldev->xstats.count_per_model[model_id]; + xstats_mode_count = cn10k_mldev->xstats.count_per_model[model_id]; break; default: return -EINVAL; @@ -1283,16 +1306,17 @@ cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mod if (xstats_mode_count > size || xstats_map == NULL) return xstats_mode_count; - for (i = 0; i < mldev->xstats.count && idx < size; i++) { - if (mldev->xstats.entries[i].mode != mode) + for (i = 0; i < cn10k_mldev->xstats.count && idx < size; i++) { + if (cn10k_mldev->xstats.entries[i].mode != mode) continue; if (mode != RTE_ML_DEV_XSTATS_DEVICE && - model_id != mldev->xstats.entries[i].obj_idx) + model_id != cn10k_mldev->xstats.entries[i].obj_idx) continue; - strncpy(xstats_map[idx].name, mldev->xstats.entries[i].map.name, RTE_ML_STR_MAX); - xstats_map[idx].id = mldev->xstats.entries[i].map.id; + strncpy(xstats_map[idx].name, cn10k_mldev->xstats.entries[i].map.name, + RTE_ML_STR_MAX); + xstats_map[idx].id = cn10k_mldev->xstats.entries[i].map.id; idx++; } @@ -1304,13 +1328,15 @@ cn10k_ml_dev_xstats_by_name_get(struct rte_ml_dev *dev, const char *name, uint16 uint64_t *value) { struct cn10k_ml_xstats_entry *xs; - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; cn10k_ml_xstats_fn fn; uint32_t i; - mldev = dev->data->dev_private; - for (i = 0; i < mldev->xstats.count; i++) { - xs = &mldev->xstats.entries[i]; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + for (i = 0; i < cn10k_mldev->xstats.count; i++) { + xs = &cn10k_mldev->xstats.entries[i]; if (strncmp(xs->map.name, name, RTE_ML_STR_MAX) == 0) { if (stat_id != NULL) *stat_id = xs->map.id; @@ -1344,24 +1370,26 @@ cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode const uint16_t stat_ids[], uint64_t values[], uint16_t nb_ids) { struct cn10k_ml_xstats_entry *xs; - struct cn10k_ml_dev *mldev; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; uint32_t xstats_mode_count; cn10k_ml_xstats_fn fn; uint64_t val; uint32_t idx; uint32_t i; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; xstats_mode_count = 0; switch (mode) { case RTE_ML_DEV_XSTATS_DEVICE: - xstats_mode_count = mldev->xstats.count_mode_device; + xstats_mode_count = cn10k_mldev->xstats.count_mode_device; break; case RTE_ML_DEV_XSTATS_MODEL: if (model_id >= ML_CN10K_MAX_MODELS) return -EINVAL; - xstats_mode_count = mldev->xstats.count_per_model[model_id]; + xstats_mode_count = cn10k_mldev->xstats.count_per_model[model_id]; break; default: return -EINVAL; @@ -1369,8 +1397,8 @@ cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode idx = 0; for (i = 0; i < nb_ids && idx < xstats_mode_count; i++) { - xs = &mldev->xstats.entries[stat_ids[i]]; - if (stat_ids[i] > mldev->xstats.count || xs->mode != mode) + xs = &cn10k_mldev->xstats.entries[stat_ids[i]]; + if (stat_ids[i] > cn10k_mldev->xstats.count || xs->mode != mode) continue; if (mode == RTE_ML_DEV_XSTATS_MODEL && model_id != xs->obj_idx) { @@ -1418,8 +1446,9 @@ cn10k_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mo static int cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_fw *fw; uint32_t head_loc; @@ -1432,8 +1461,9 @@ cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) if (roc_env_is_asim()) return 0; - mldev = dev->data->dev_private; - fw = &mldev->fw; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + fw = &cn10k_mldev->fw; /* Dump model info */ for (model_id = 0; model_id < dev->data->nb_models; model_id++) { @@ -1451,15 +1481,19 @@ cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) for (core_id = 0; core_id <= 1; core_id++) { bufsize = fw->req->jd.fw_load.debug.debug_buffer_size; if (core_id == 0) { - head_loc = roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_DBG_BUFFER_HEAD_C0); - tail_loc = roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_DBG_BUFFER_TAIL_C0); + head_loc = + roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_DBG_BUFFER_HEAD_C0); + tail_loc = + roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_DBG_BUFFER_TAIL_C0); head_ptr = PLT_PTR_CAST(fw->req->jd.fw_load.debug.core0_debug_ptr); - head_ptr = roc_ml_addr_mlip2ap(&mldev->roc, head_ptr); + head_ptr = roc_ml_addr_mlip2ap(&cn10k_mldev->roc, head_ptr); } else { - head_loc = roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_DBG_BUFFER_HEAD_C1); - tail_loc = roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_DBG_BUFFER_TAIL_C1); + head_loc = + roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_DBG_BUFFER_HEAD_C1); + tail_loc = + roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_DBG_BUFFER_TAIL_C1); head_ptr = PLT_PTR_CAST(fw->req->jd.fw_load.debug.core1_debug_ptr); - head_ptr = roc_ml_addr_mlip2ap(&mldev->roc, head_ptr); + head_ptr = roc_ml_addr_mlip2ap(&cn10k_mldev->roc, head_ptr); } if (head_loc < tail_loc) { fprintf(fp, "%.*s\n", tail_loc - head_loc, &head_ptr[head_loc]); @@ -1473,18 +1507,18 @@ cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) for (core_id = 0; core_id <= 1; core_id++) { bufsize = fw->req->jd.fw_load.debug.exception_state_size; if ((core_id == 0) && - (roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0) != 0)) { + (roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0) != 0)) { head_ptr = PLT_PTR_CAST(fw->req->jd.fw_load.debug.core0_exception_buffer); fprintf(fp, "ML_SCRATCH_EXCEPTION_SP_C0 = 0x%016lx", - roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0)); - head_ptr = roc_ml_addr_mlip2ap(&mldev->roc, head_ptr); + roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0)); + head_ptr = roc_ml_addr_mlip2ap(&cn10k_mldev->roc, head_ptr); fprintf(fp, "%.*s", bufsize, head_ptr); - } else if ((core_id == 1) && - (roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_EXCEPTION_SP_C1) != 0)) { + } else if ((core_id == 1) && (roc_ml_reg_read64(&cn10k_mldev->roc, + ML_SCRATCH_EXCEPTION_SP_C1) != 0)) { head_ptr = PLT_PTR_CAST(fw->req->jd.fw_load.debug.core1_exception_buffer); fprintf(fp, "ML_SCRATCH_EXCEPTION_SP_C1 = 0x%016lx", - roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_EXCEPTION_SP_C1)); - head_ptr = roc_ml_addr_mlip2ap(&mldev->roc, head_ptr); + roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C1)); + head_ptr = roc_ml_addr_mlip2ap(&cn10k_mldev->roc, head_ptr); fprintf(fp, "%.*s", bufsize, head_ptr); } } @@ -1495,14 +1529,16 @@ cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) static int cn10k_ml_dev_selftest(struct rte_ml_dev *dev) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; const struct plt_memzone *mz; - struct cn10k_ml_dev *mldev; struct cn10k_ml_req *req; uint64_t timeout_cycle; bool timeout; int ret; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; mz = plt_memzone_reserve_aligned("dev_selftest", sizeof(struct cn10k_ml_req), 0, ML_CN10K_ALIGN_SIZE); if (mz == NULL) { @@ -1515,20 +1551,20 @@ cn10k_ml_dev_selftest(struct rte_ml_dev *dev) memset(&req->jd, 0, sizeof(struct cn10k_ml_jd)); req->jd.hdr.jce.w1.u64 = PLT_U64_CAST(&req->status); req->jd.hdr.job_type = ML_CN10K_JOB_TYPE_FIRMWARE_SELFTEST; - req->jd.hdr.result = roc_ml_addr_ap2mlip(&mldev->roc, &req->result); - req->jd.fw_load.flags = cn10k_ml_fw_flags_get(&mldev->fw); - plt_write64(ML_CN10K_POLL_JOB_START, &req->status); + req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->result); + req->jd.fw_load.flags = cn10k_ml_fw_flags_get(&cn10k_mldev->fw); + plt_write64(ML_CNXK_POLL_JOB_START, &req->status); plt_wmb(); /* Enqueue firmware selftest request through scratch registers */ timeout = true; - timeout_cycle = plt_tsc_cycles() + ML_CN10K_CMD_TIMEOUT * plt_tsc_hz(); - roc_ml_scratch_enqueue(&mldev->roc, &req->jd); + timeout_cycle = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); + roc_ml_scratch_enqueue(&cn10k_mldev->roc, &req->jd); plt_rmb(); do { - if (roc_ml_scratch_is_done_bit_set(&mldev->roc) && - (plt_read64(&req->status) == ML_CN10K_POLL_JOB_FINISH)) { + if (roc_ml_scratch_is_done_bit_set(&cn10k_mldev->roc) && + (plt_read64(&req->status) == ML_CNXK_POLL_JOB_FINISH)) { timeout = false; break; } @@ -1552,8 +1588,8 @@ int cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, uint16_t *model_id) { struct cn10k_ml_model_metadata *metadata; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; char str[RTE_MEMZONE_NAMESIZE]; const struct plt_memzone *mz; @@ -1574,7 +1610,7 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, if (ret != 0) return ret; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; /* Find model ID */ found = false; @@ -1591,7 +1627,8 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, } /* Get WB and scratch pages, check if model can be loaded. */ - ret = cn10k_ml_model_ocm_pages_count(mldev, idx, params->addr, &wb_pages, &scratch_pages); + ret = cn10k_ml_model_ocm_pages_count(&cnxk_mldev->cn10k_mldev, idx, params->addr, &wb_pages, + &scratch_pages); if (ret < 0) return ret; @@ -1623,7 +1660,7 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, } model = mz->addr; - model->mldev = mldev; + model->mldev = cnxk_mldev; model->model_id = idx; rte_memcpy(&model->metadata, params->addr, sizeof(struct cn10k_ml_model_metadata)); @@ -1680,7 +1717,7 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, plt_spinlock_init(&model->lock); model->state = ML_CN10K_MODEL_STATE_LOADED; dev->data->models[idx] = model; - mldev->nb_models_loaded++; + cnxk_mldev->nb_models_loaded++; /* Update xstats names */ cn10k_ml_xstats_model_name_update(dev, idx); @@ -1695,9 +1732,9 @@ cn10k_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) { char str[RTE_MEMZONE_NAMESIZE]; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; + struct cnxk_ml_dev *cnxk_mldev; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; model = dev->data->models[model_id]; if (model == NULL) { @@ -1711,7 +1748,7 @@ cn10k_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) } dev->data->models[model_id] = NULL; - mldev->nb_models_unloaded++; + cnxk_mldev->nb_models_unloaded++; snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", CN10K_ML_MODEL_MEMZONE_NAME, model_id); return plt_memzone_free(plt_memzone_lookup(str)); @@ -1720,8 +1757,9 @@ cn10k_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) int cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_ocm *ocm; struct cn10k_ml_req *req; @@ -1735,8 +1773,9 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) bool locked; int ret = 0; - mldev = dev->data->dev_private; - ocm = &mldev->ocm; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + ocm = &cn10k_mldev->ocm; model = dev->data->models[model_id]; if (model == NULL) { @@ -1746,11 +1785,11 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) /* Prepare JD */ req = model->req; - cn10k_ml_prep_sp_job_descriptor(mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_START); + cn10k_ml_prep_sp_job_descriptor(cn10k_mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_START); req->result.error_code.u64 = 0x0; req->result.user_ptr = NULL; - plt_write64(ML_CN10K_POLL_JOB_START, &req->status); + plt_write64(ML_CNXK_POLL_JOB_START, &req->status); plt_wmb(); num_tiles = model->metadata.model.tile_end - model->metadata.model.tile_start + 1; @@ -1815,26 +1854,26 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) job_dequeued = false; do { if (!job_enqueued) { - req->timeout = plt_tsc_cycles() + ML_CN10K_CMD_TIMEOUT * plt_tsc_hz(); - job_enqueued = roc_ml_scratch_enqueue(&mldev->roc, &req->jd); + req->timeout = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); + job_enqueued = roc_ml_scratch_enqueue(&cn10k_mldev->roc, &req->jd); } if (job_enqueued && !job_dequeued) - job_dequeued = roc_ml_scratch_dequeue(&mldev->roc, &req->jd); + job_dequeued = roc_ml_scratch_dequeue(&cn10k_mldev->roc, &req->jd); if (job_dequeued) break; } while (plt_tsc_cycles() < req->timeout); if (job_dequeued) { - if (plt_read64(&req->status) == ML_CN10K_POLL_JOB_FINISH) { + if (plt_read64(&req->status) == ML_CNXK_POLL_JOB_FINISH) { if (req->result.error_code.u64 == 0) ret = 0; else ret = -1; } } else { /* Reset scratch registers */ - roc_ml_scratch_queue_reset(&mldev->roc); + roc_ml_scratch_queue_reset(&cn10k_mldev->roc); ret = -ETIME; } @@ -1843,7 +1882,7 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) if (plt_spinlock_trylock(&model->lock) != 0) { if (ret == 0) { model->state = ML_CN10K_MODEL_STATE_STARTED; - mldev->nb_models_started++; + cnxk_mldev->nb_models_started++; } else { model->state = ML_CN10K_MODEL_STATE_UNKNOWN; } @@ -1867,7 +1906,7 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) if (ret < 0) { /* Call unload to update model and FW state, ignore error */ rte_ml_model_stop(dev->data->dev_id, model_id); } else { - if (mldev->cache_model_data && roc_model_is_cn10ka()) + if (cn10k_mldev->cache_model_data && roc_model_is_cn10ka()) ret = cn10k_ml_cache_model_data(dev, model_id); } @@ -1877,8 +1916,9 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) int cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_ocm *ocm; struct cn10k_ml_req *req; @@ -1887,8 +1927,9 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) bool locked; int ret = 0; - mldev = dev->data->dev_private; - ocm = &mldev->ocm; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + ocm = &cn10k_mldev->ocm; model = dev->data->models[model_id]; if (model == NULL) { @@ -1898,11 +1939,11 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) /* Prepare JD */ req = model->req; - cn10k_ml_prep_sp_job_descriptor(mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_STOP); + cn10k_ml_prep_sp_job_descriptor(cn10k_mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_STOP); req->result.error_code.u64 = 0x0; req->result.user_ptr = NULL; - plt_write64(ML_CN10K_POLL_JOB_START, &req->status); + plt_write64(ML_CNXK_POLL_JOB_START, &req->status); plt_wmb(); locked = false; @@ -1941,33 +1982,33 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) job_dequeued = false; do { if (!job_enqueued) { - req->timeout = plt_tsc_cycles() + ML_CN10K_CMD_TIMEOUT * plt_tsc_hz(); - job_enqueued = roc_ml_scratch_enqueue(&mldev->roc, &req->jd); + req->timeout = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); + job_enqueued = roc_ml_scratch_enqueue(&cn10k_mldev->roc, &req->jd); } if (job_enqueued && !job_dequeued) - job_dequeued = roc_ml_scratch_dequeue(&mldev->roc, &req->jd); + job_dequeued = roc_ml_scratch_dequeue(&cn10k_mldev->roc, &req->jd); if (job_dequeued) break; } while (plt_tsc_cycles() < req->timeout); if (job_dequeued) { - if (plt_read64(&req->status) == ML_CN10K_POLL_JOB_FINISH) { + if (plt_read64(&req->status) == ML_CNXK_POLL_JOB_FINISH) { if (req->result.error_code.u64 == 0x0) ret = 0; else ret = -1; } } else { - roc_ml_scratch_queue_reset(&mldev->roc); + roc_ml_scratch_queue_reset(&cn10k_mldev->roc); ret = -ETIME; } locked = false; while (!locked) { if (plt_spinlock_trylock(&model->lock) != 0) { - mldev->nb_models_stopped++; + cnxk_mldev->nb_models_stopped++; model->state = ML_CN10K_MODEL_STATE_LOADED; plt_spinlock_unlock(&model->lock); locked = true; @@ -2211,8 +2252,9 @@ cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cn10k_ml_result struct rte_ml_op *op) { struct cn10k_ml_model_stats *stats; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_qp *qp; uint64_t hw_latency; uint64_t fw_latency; @@ -2258,14 +2300,16 @@ cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cn10k_ml_result /* Handle driver error */ if (result->error_code.s.etype == ML_ETYPE_DRIVER) { - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; /* Check for exception */ - if ((roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0) != 0) || - (roc_ml_reg_read64(&mldev->roc, ML_SCRATCH_EXCEPTION_SP_C1) != 0)) + if ((roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0) != + 0) || + (roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C1) != 0)) result->error_code.s.stype = ML_DRIVER_ERR_EXCEPTION; - else if ((roc_ml_reg_read64(&mldev->roc, ML_CORE_INT_LO) != 0) || - (roc_ml_reg_read64(&mldev->roc, ML_CORE_INT_HI) != 0)) + else if ((roc_ml_reg_read64(&cn10k_mldev->roc, ML_CORE_INT_LO) != 0) || + (roc_ml_reg_read64(&cn10k_mldev->roc, ML_CORE_INT_HI) != 0)) result->error_code.s.stype = ML_DRIVER_ERR_FW_ERROR; else result->error_code.s.stype = ML_DRIVER_ERR_UNKNOWN; @@ -2282,8 +2326,9 @@ __rte_hot uint16_t cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, uint16_t nb_ops) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_queue *queue; - struct cn10k_ml_dev *mldev; struct cn10k_ml_req *req; struct cn10k_ml_qp *qp; struct rte_ml_op *op; @@ -2292,7 +2337,8 @@ cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op uint64_t head; bool enqueued; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; qp = dev->data->queue_pairs[qp_id]; queue = &qp->queue; @@ -2307,15 +2353,15 @@ cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op op = ops[count]; req = &queue->reqs[head]; - mldev->set_poll_addr(req); - cn10k_ml_prep_fp_job_descriptor(dev, req, op); + cn10k_mldev->set_poll_addr(req); + cn10k_ml_prep_fp_job_descriptor(cn10k_mldev, req, op); memset(&req->result, 0, sizeof(struct cn10k_ml_result)); req->result.error_code.s.etype = ML_ETYPE_UNKNOWN; req->result.user_ptr = op->user_ptr; - mldev->set_poll_ptr(req); - enqueued = mldev->ml_jcmdq_enqueue(&mldev->roc, &req->jcmd); + cn10k_mldev->set_poll_ptr(req); + enqueued = cn10k_mldev->ml_jcmdq_enqueue(&cn10k_mldev->roc, &req->jcmd); if (unlikely(!enqueued)) goto jcmdq_full; @@ -2339,8 +2385,9 @@ __rte_hot uint16_t cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, uint16_t nb_ops) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_queue *queue; - struct cn10k_ml_dev *mldev; struct cn10k_ml_req *req; struct cn10k_ml_qp *qp; @@ -2348,7 +2395,8 @@ cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op uint16_t count; uint64_t tail; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; qp = dev->data->queue_pairs[qp_id]; queue = &qp->queue; @@ -2361,8 +2409,8 @@ cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op dequeue_req: req = &queue->reqs[tail]; - status = mldev->get_poll_ptr(req); - if (unlikely(status != ML_CN10K_POLL_JOB_FINISH)) { + status = cn10k_mldev->get_poll_ptr(req); + if (unlikely(status != ML_CNXK_POLL_JOB_FINISH)) { if (plt_tsc_cycles() < req->timeout) goto empty_or_active; else /* Timeout, set indication of driver error */ @@ -2420,30 +2468,32 @@ cn10k_ml_op_error_get(struct rte_ml_dev *dev, struct rte_ml_op *op, struct rte_m __rte_hot int cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) { + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_model *model; - struct cn10k_ml_dev *mldev; struct cn10k_ml_req *req; bool timeout; int ret = 0; - mldev = dev->data->dev_private; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; model = dev->data->models[op->model_id]; req = model->req; cn10k_ml_set_poll_addr(req); - cn10k_ml_prep_fp_job_descriptor(dev, req, op); + cn10k_ml_prep_fp_job_descriptor(cn10k_mldev, req, op); memset(&req->result, 0, sizeof(struct cn10k_ml_result)); req->result.error_code.s.etype = ML_ETYPE_UNKNOWN; req->result.user_ptr = op->user_ptr; - mldev->set_poll_ptr(req); + cn10k_mldev->set_poll_ptr(req); req->jcmd.w1.s.jobptr = PLT_U64_CAST(&req->jd); timeout = true; - req->timeout = plt_tsc_cycles() + ML_CN10K_CMD_TIMEOUT * plt_tsc_hz(); + req->timeout = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); do { - if (mldev->ml_jcmdq_enqueue(&mldev->roc, &req->jcmd)) { + if (cn10k_mldev->ml_jcmdq_enqueue(&cn10k_mldev->roc, &req->jcmd)) { req->op = op; timeout = false; break; @@ -2457,7 +2507,7 @@ cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) timeout = true; do { - if (mldev->get_poll_ptr(req) == ML_CN10K_POLL_JOB_FINISH) { + if (cn10k_mldev->get_poll_ptr(req) == ML_CNXK_POLL_JOB_FINISH) { timeout = false; break; } diff --git a/drivers/ml/cnxk/cnxk_ml_dev.c b/drivers/ml/cnxk/cnxk_ml_dev.c new file mode 100644 index 00000000000..2a5c17c973b --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_dev.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#include +#include + +#include "cnxk_ml_dev.h" + +/* Dummy operations for ML device */ +struct rte_ml_dev_ops ml_dev_dummy_ops = {0}; diff --git a/drivers/ml/cnxk/cnxk_ml_dev.h b/drivers/ml/cnxk/cnxk_ml_dev.h new file mode 100644 index 00000000000..51315de6227 --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_dev.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#ifndef _CNXK_ML_DEV_H_ +#define _CNXK_ML_DEV_H_ + +#include + +#include "cn10k_ml_dev.h" + +/* ML command timeout in seconds */ +#define ML_CNXK_CMD_TIMEOUT 5 + +/* Poll mode job state */ +#define ML_CNXK_POLL_JOB_START 0 +#define ML_CNXK_POLL_JOB_FINISH 1 + +/* Device configuration state enum */ +enum cnxk_ml_dev_state { + /* Probed and not configured */ + ML_CNXK_DEV_STATE_PROBED = 0, + + /* Configured */ + ML_CNXK_DEV_STATE_CONFIGURED, + + /* Started */ + ML_CNXK_DEV_STATE_STARTED, + + /* Closed */ + ML_CNXK_DEV_STATE_CLOSED +}; + +/* Device private data */ +struct cnxk_ml_dev { + /* RTE device */ + struct rte_ml_dev *mldev; + + /* Configuration state */ + enum cnxk_ml_dev_state state; + + /* Number of models loaded */ + uint16_t nb_models_loaded; + + /* Number of models unloaded */ + uint16_t nb_models_unloaded; + + /* Number of models started */ + uint16_t nb_models_started; + + /* Number of models stopped */ + uint16_t nb_models_stopped; + + /* CN10K device structure */ + struct cn10k_ml_dev cn10k_mldev; +}; + +#endif /* _CNXK_ML_DEV_H_ */ diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 94fa4283b13..03a2d4ecf2f 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -12,6 +12,7 @@ driver_sdk_headers = files( 'cn10k_ml_ops.h', 'cn10k_ml_model.h', 'cn10k_ml_ocm.h', + 'cnxk_ml_dev.h', ) sources = files( @@ -19,6 +20,7 @@ sources = files( 'cn10k_ml_ops.c', 'cn10k_ml_model.c', 'cn10k_ml_ocm.c', + 'cnxk_ml_dev.c', ) deps += ['mldev', 'common_cnxk', 'kvargs', 'hash'] From patchwork Wed Aug 30 15:58:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130885 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A11AB41FD2; Wed, 30 Aug 2023 18:00:08 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B34B6402AB; Wed, 30 Aug 2023 17:59:40 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 25D4F4028D for ; Wed, 30 Aug 2023 17:59:36 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I6r012554 for ; Wed, 30 Aug 2023 08:59:35 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=u8eX7+IVFcZeQ88coYsxshvRZlsljKZYeULt5XWPivk=; b=aWBeyHuoTp4XQF6ExaL+1EL+ORTx4xmw8kL3o0IJhJYIQglLrfF7MZERt5tK1jUSa4gH 8RhnPprUNmSxFcwB42IACXGPG/9px2YOBqkIV5gSton6x1MZv8NUuDzHBjAuo4oDs5WD nZsWbjsAX3R2ObQS3E061PxlMu9fJnRXsEtkYor9JU8mmW3YTHh3mNkONpAI6zJj+xJc rCfDW7pojSr3C112ihwcFaDwyJ1dbz11YTr8uL3v5SVPsBaOw36sx6KYZR2gs54+F/sf RVEtdrsxcfcN9e60u4gVJJSiiOPno3vXNIafyvlOrJHHqHjwG8eQe0YAixm3EYu1FNsI bw== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:35 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:33 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:33 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 01D473F7080; Wed, 30 Aug 2023 08:59:32 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 04/34] ml/cnxk: add generic model and layer structures Date: Wed, 30 Aug 2023 08:58:54 -0700 Message-ID: <20230830155927.3566-5-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: LodvVmdJXgNH8bU1eY1KPpLHG4x4KKY_ X-Proofpoint-GUID: LodvVmdJXgNH8bU1eY1KPpLHG4x4KKY_ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Introduce generic cnxk model and layer structure. These structures would enable supporting models with multiple layers. A model is a collection of multiple independent layers with flow dependencies between the layers. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.h | 9 +- drivers/ml/cnxk/cn10k_ml_model.c | 244 ++++++++-------- drivers/ml/cnxk/cn10k_ml_model.h | 122 ++------ drivers/ml/cnxk/cn10k_ml_ocm.c | 49 +++- drivers/ml/cnxk/cn10k_ml_ocm.h | 9 +- drivers/ml/cnxk/cn10k_ml_ops.c | 487 +++++++++++++++++-------------- drivers/ml/cnxk/cnxk_ml_io.h | 79 +++++ drivers/ml/cnxk/cnxk_ml_model.c | 7 + drivers/ml/cnxk/cnxk_ml_model.h | 111 +++++++ drivers/ml/cnxk/meson.build | 3 + 10 files changed, 653 insertions(+), 467 deletions(-) create mode 100644 drivers/ml/cnxk/cnxk_ml_io.h create mode 100644 drivers/ml/cnxk/cnxk_ml_model.c create mode 100644 drivers/ml/cnxk/cnxk_ml_model.h diff --git a/drivers/ml/cnxk/cn10k_ml_dev.h b/drivers/ml/cnxk/cn10k_ml_dev.h index f9da1548c4a..99ff0a344a2 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.h +++ b/drivers/ml/cnxk/cn10k_ml_dev.h @@ -9,6 +9,8 @@ #include "cn10k_ml_ocm.h" +#include "cnxk_ml_io.h" + /* Dummy Device ops */ extern struct rte_ml_dev_ops ml_dev_dummy_ops; @@ -21,9 +23,6 @@ extern struct rte_ml_dev_ops ml_dev_dummy_ops; /* Device alignment size */ #define ML_CN10K_ALIGN_SIZE 128 -/* Maximum number of models per device */ -#define ML_CN10K_MAX_MODELS 16 - /* Maximum number of queue-pairs per device, spinlock version */ #define ML_CN10K_MAX_QP_PER_DEVICE_SL 16 @@ -455,8 +454,8 @@ struct cn10k_ml_xstats { struct cn10k_ml_xstats_entry *entries; /* Store num stats and offset of the stats for each model */ - uint16_t count_per_model[ML_CN10K_MAX_MODELS]; - uint16_t offset_for_model[ML_CN10K_MAX_MODELS]; + uint16_t count_per_model[ML_CNXK_MAX_MODELS]; + uint16_t offset_for_model[ML_CNXK_MAX_MODELS]; uint16_t count_mode_device; uint16_t count_mode_model; uint16_t count; diff --git a/drivers/ml/cnxk/cn10k_ml_model.c b/drivers/ml/cnxk/cn10k_ml_model.c index d146535866a..0ea6520bf78 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.c +++ b/drivers/ml/cnxk/cn10k_ml_model.c @@ -11,6 +11,7 @@ #include "cn10k_ml_ocm.h" #include "cnxk_ml_dev.h" +#include "cnxk_ml_model.h" static enum rte_ml_io_type cn10k_ml_io_type_map(uint8_t type) @@ -312,19 +313,17 @@ cn10k_ml_model_metadata_update(struct cn10k_ml_model_metadata *metadata) } void -cn10k_ml_model_addr_update(struct cn10k_ml_model *model, uint8_t *buffer, uint8_t *base_dma_addr) +cn10k_ml_layer_addr_update(struct cnxk_ml_layer *layer, uint8_t *buffer, uint8_t *base_dma_addr) { struct cn10k_ml_model_metadata *metadata; - struct cn10k_ml_model_addr *addr; + struct cn10k_ml_layer_addr *addr; size_t model_data_size; uint8_t *dma_addr_load; uint8_t *dma_addr_run; - uint8_t i; - uint8_t j; int fpos; - metadata = &model->metadata; - addr = &model->addr; + metadata = &layer->glow.metadata; + addr = &layer->glow.addr; model_data_size = metadata->init_model.file_size + metadata->main_model.file_size + metadata->finish_model.file_size + metadata->weights_bias.file_size; @@ -362,102 +361,136 @@ cn10k_ml_model_addr_update(struct cn10k_ml_model *model, uint8_t *buffer, uint8_ addr->wb_base_addr = PLT_PTR_SUB(dma_addr_load, metadata->weights_bias.mem_offset); addr->wb_load_addr = PLT_PTR_ADD(addr->wb_base_addr, metadata->weights_bias.mem_offset); rte_memcpy(addr->wb_load_addr, PLT_PTR_ADD(buffer, fpos), metadata->weights_bias.file_size); +} + +void +cn10k_ml_layer_info_update(struct cnxk_ml_layer *layer) +{ + struct cn10k_ml_model_metadata *metadata; + uint8_t i; + uint8_t j; + + metadata = &layer->glow.metadata; /* Inputs */ - addr->total_input_sz_d = 0; - addr->total_input_sz_q = 0; + layer->info.nb_inputs = metadata->model.num_input; + layer->info.total_input_sz_d = 0; + layer->info.total_input_sz_q = 0; for (i = 0; i < metadata->model.num_input; i++) { if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - addr->input[i].nb_dims = 4; - addr->input[i].shape[0] = metadata->input1[i].shape.w; - addr->input[i].shape[1] = metadata->input1[i].shape.x; - addr->input[i].shape[2] = metadata->input1[i].shape.y; - addr->input[i].shape[3] = metadata->input1[i].shape.z; - - addr->input[i].nb_elements = + strncpy(layer->info.input[i].name, (char *)metadata->input1[i].input_name, + MRVL_ML_INPUT_NAME_LEN); + layer->info.input[i].dtype = metadata->input1[i].input_type; + layer->info.input[i].qtype = metadata->input1[i].model_input_type; + layer->info.input[i].nb_dims = 4; + layer->info.input[i].shape[0] = metadata->input1[i].shape.w; + layer->info.input[i].shape[1] = metadata->input1[i].shape.x; + layer->info.input[i].shape[2] = metadata->input1[i].shape.y; + layer->info.input[i].shape[3] = metadata->input1[i].shape.z; + layer->info.input[i].nb_elements = metadata->input1[i].shape.w * metadata->input1[i].shape.x * metadata->input1[i].shape.y * metadata->input1[i].shape.z; - addr->input[i].sz_d = - addr->input[i].nb_elements * + layer->info.input[i].sz_d = + layer->info.input[i].nb_elements * rte_ml_io_type_size_get(metadata->input1[i].input_type); - addr->input[i].sz_q = - addr->input[i].nb_elements * + layer->info.input[i].sz_q = + layer->info.input[i].nb_elements * rte_ml_io_type_size_get(metadata->input1[i].model_input_type); - addr->total_input_sz_d += addr->input[i].sz_d; - addr->total_input_sz_q += addr->input[i].sz_q; + layer->info.input[i].scale = metadata->input1[i].qscale; + + layer->info.total_input_sz_d += layer->info.input[i].sz_d; + layer->info.total_input_sz_q += layer->info.input[i].sz_q; plt_ml_dbg( - "model_id = %u, input[%u] - w:%u x:%u y:%u z:%u, sz_d = %u sz_q = %u", - model->model_id, i, metadata->input1[i].shape.w, + "index = %u, input1[%u] - w:%u x:%u y:%u z:%u, sz_d = %u sz_q = %u", + layer->index, i, metadata->input1[i].shape.w, metadata->input1[i].shape.x, metadata->input1[i].shape.y, - metadata->input1[i].shape.z, addr->input[i].sz_d, - addr->input[i].sz_q); + metadata->input1[i].shape.z, layer->info.input[i].sz_d, + layer->info.input[i].sz_q); } else { j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - addr->input[i].nb_dims = 4; - addr->input[i].shape[0] = metadata->input2[j].shape.w; - addr->input[i].shape[1] = metadata->input2[j].shape.x; - addr->input[i].shape[2] = metadata->input2[j].shape.y; - addr->input[i].shape[3] = metadata->input2[j].shape.z; - - addr->input[i].nb_elements = + strncpy(layer->info.input[i].name, (char *)metadata->input2[j].input_name, + MRVL_ML_INPUT_NAME_LEN); + layer->info.input[i].dtype = metadata->input2[j].input_type; + layer->info.input[i].qtype = metadata->input2[j].model_input_type; + layer->info.input[i].nb_dims = 4; + layer->info.input[i].shape[0] = metadata->input2[j].shape.w; + layer->info.input[i].shape[1] = metadata->input2[j].shape.x; + layer->info.input[i].shape[2] = metadata->input2[j].shape.y; + layer->info.input[i].shape[3] = metadata->input2[j].shape.z; + layer->info.input[i].nb_elements = metadata->input2[j].shape.w * metadata->input2[j].shape.x * metadata->input2[j].shape.y * metadata->input2[j].shape.z; - addr->input[i].sz_d = - addr->input[i].nb_elements * + layer->info.input[i].sz_d = + layer->info.input[i].nb_elements * rte_ml_io_type_size_get(metadata->input2[j].input_type); - addr->input[i].sz_q = - addr->input[i].nb_elements * + layer->info.input[i].sz_q = + layer->info.input[i].nb_elements * rte_ml_io_type_size_get(metadata->input2[j].model_input_type); - addr->total_input_sz_d += addr->input[i].sz_d; - addr->total_input_sz_q += addr->input[i].sz_q; + layer->info.input[i].scale = metadata->input2[j].qscale; + + layer->info.total_input_sz_d += layer->info.input[i].sz_d; + layer->info.total_input_sz_q += layer->info.input[i].sz_q; plt_ml_dbg( - "model_id = %u, input2[%u] - w:%u x:%u y:%u z:%u, sz_d = %u sz_q = %u", - model->model_id, j, metadata->input2[j].shape.w, + "index = %u, input2[%u] - w:%u x:%u y:%u z:%u, sz_d = %u sz_q = %u", + layer->index, j, metadata->input2[j].shape.w, metadata->input2[j].shape.x, metadata->input2[j].shape.y, - metadata->input2[j].shape.z, addr->input[i].sz_d, - addr->input[i].sz_q); + metadata->input2[j].shape.z, layer->info.input[i].sz_d, + layer->info.input[i].sz_q); } } /* Outputs */ - addr->total_output_sz_q = 0; - addr->total_output_sz_d = 0; + layer->info.nb_outputs = metadata->model.num_output; + layer->info.total_output_sz_q = 0; + layer->info.total_output_sz_d = 0; for (i = 0; i < metadata->model.num_output; i++) { if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - addr->output[i].nb_dims = 1; - addr->output[i].shape[0] = metadata->output1[i].size; - addr->output[i].nb_elements = metadata->output1[i].size; - addr->output[i].sz_d = - addr->output[i].nb_elements * + strncpy(layer->info.output[i].name, + (char *)metadata->output1[i].output_name, MRVL_ML_OUTPUT_NAME_LEN); + layer->info.output[i].dtype = metadata->output1[i].output_type; + layer->info.output[i].qtype = metadata->output1[i].model_output_type; + layer->info.output[i].nb_dims = 1; + layer->info.output[i].shape[0] = metadata->output1[i].size; + layer->info.output[i].nb_elements = metadata->output1[i].size; + layer->info.output[i].sz_d = + layer->info.output[i].nb_elements * rte_ml_io_type_size_get(metadata->output1[i].output_type); - addr->output[i].sz_q = - addr->output[i].nb_elements * + layer->info.output[i].sz_q = + layer->info.output[i].nb_elements * rte_ml_io_type_size_get(metadata->output1[i].model_output_type); - addr->total_output_sz_q += addr->output[i].sz_q; - addr->total_output_sz_d += addr->output[i].sz_d; + layer->info.output[i].scale = metadata->output1[i].dscale; - plt_ml_dbg("model_id = %u, output[%u] - sz_d = %u, sz_q = %u", - model->model_id, i, addr->output[i].sz_d, addr->output[i].sz_q); + layer->info.total_output_sz_q += layer->info.output[i].sz_q; + layer->info.total_output_sz_d += layer->info.output[i].sz_d; + + plt_ml_dbg("index = %u, output1[%u] - sz_d = %u, sz_q = %u", layer->index, + i, layer->info.output[i].sz_d, layer->info.output[i].sz_q); } else { j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - addr->output[i].nb_dims = 1; - addr->output[i].shape[0] = metadata->output2[j].size; - addr->output[i].nb_elements = metadata->output2[j].size; - addr->output[i].sz_d = - addr->output[i].nb_elements * + strncpy(layer->info.output[i].name, + (char *)metadata->output2[j].output_name, MRVL_ML_OUTPUT_NAME_LEN); + layer->info.output[i].dtype = metadata->output2[j].output_type; + layer->info.output[i].qtype = metadata->output2[j].model_output_type; + layer->info.output[i].nb_dims = 1; + layer->info.output[i].shape[0] = metadata->output2[j].size; + layer->info.output[i].nb_elements = metadata->output2[j].size; + layer->info.output[i].sz_d = + layer->info.output[i].nb_elements * rte_ml_io_type_size_get(metadata->output2[j].output_type); - addr->output[i].sz_q = - addr->output[i].nb_elements * + layer->info.output[i].sz_q = + layer->info.output[i].nb_elements * rte_ml_io_type_size_get(metadata->output2[j].model_output_type); - addr->total_output_sz_q += addr->output[i].sz_q; - addr->total_output_sz_d += addr->output[i].sz_d; + layer->info.output[i].scale = metadata->output2[j].dscale; + + layer->info.total_output_sz_q += layer->info.output[i].sz_q; + layer->info.total_output_sz_d += layer->info.output[i].sz_d; - plt_ml_dbg("model_id = %u, output2[%u] - sz_d = %u, sz_q = %u", - model->model_id, j, addr->output[i].sz_d, addr->output[i].sz_q); + plt_ml_dbg("index = %u, output2[%u] - sz_d = %u, sz_q = %u", layer->index, + j, layer->info.output[i].sz_d, layer->info.output[i].sz_q); } } } @@ -515,23 +548,23 @@ cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_ } void -cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cn10k_ml_model *model) +cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cnxk_ml_model *model) { struct cn10k_ml_model_metadata *metadata; - struct cn10k_ml_model_addr *addr; + struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct rte_ml_model_info *info; struct rte_ml_io_info *output; struct rte_ml_io_info *input; - struct cn10k_ml_dev *mldev; + struct cnxk_ml_layer *layer; uint8_t i; - uint8_t j; - mldev = dev->data->dev_private; - metadata = &model->metadata; + cnxk_mldev = dev->data->dev_private; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + metadata = &model->glow.metadata; info = PLT_PTR_CAST(model->info); input = PLT_PTR_ADD(info, sizeof(struct rte_ml_model_info)); output = PLT_PTR_ADD(input, metadata->model.num_input * sizeof(struct rte_ml_io_info)); - addr = &model->addr; /* Set model info */ memset(info, 0, sizeof(struct rte_ml_model_info)); @@ -543,7 +576,8 @@ cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cn10k_ml_model *model) info->device_id = dev->data->dev_id; info->io_layout = RTE_ML_IO_LAYOUT_PACKED; info->min_batches = model->batch_size; - info->max_batches = mldev->fw.req->jd.fw_load.cap.s.max_num_batches / model->batch_size; + info->max_batches = + cn10k_mldev->fw.req->jd.fw_load.cap.s.max_num_batches / model->batch_size; info->nb_inputs = metadata->model.num_input; info->input_info = input; info->nb_outputs = metadata->model.num_output; @@ -551,56 +585,26 @@ cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cn10k_ml_model *model) info->wb_size = metadata->weights_bias.file_size; /* Set input info */ + layer = &model->layer[0]; for (i = 0; i < info->nb_inputs; i++) { - if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - rte_memcpy(input[i].name, metadata->input1[i].input_name, - MRVL_ML_INPUT_NAME_LEN); - input[i].nb_dims = addr->input[i].nb_dims; - input[i].shape = addr->input[i].shape; - input[i].type = metadata->input1[i].model_input_type; - input[i].nb_elements = addr->input[i].nb_elements; - input[i].size = - addr->input[i].nb_elements * - rte_ml_io_type_size_get(metadata->input1[i].model_input_type); - } else { - j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - - rte_memcpy(input[i].name, metadata->input2[j].input_name, - MRVL_ML_INPUT_NAME_LEN); - input[i].nb_dims = addr->input[i].nb_dims; - input[i].shape = addr->input[i].shape; - input[i].type = metadata->input2[j].model_input_type; - input[i].nb_elements = addr->input[i].nb_elements; - input[i].size = - addr->input[i].nb_elements * - rte_ml_io_type_size_get(metadata->input2[j].model_input_type); - } + rte_memcpy(input[i].name, layer->info.input[i].name, MRVL_ML_INPUT_NAME_LEN); + input[i].nb_dims = layer->info.input[i].nb_dims; + input[i].shape = &layer->info.input[i].shape[0]; + input[i].type = layer->info.input[i].qtype; + input[i].nb_elements = layer->info.input[i].nb_elements; + input[i].size = layer->info.input[i].nb_elements * + rte_ml_io_type_size_get(layer->info.input[i].qtype); } /* Set output info */ + layer = &model->layer[0]; for (i = 0; i < info->nb_outputs; i++) { - if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - rte_memcpy(output[i].name, metadata->output1[i].output_name, - MRVL_ML_OUTPUT_NAME_LEN); - output[i].nb_dims = addr->output[i].nb_dims; - output[i].shape = addr->output[i].shape; - output[i].type = metadata->output1[i].model_output_type; - output[i].nb_elements = addr->output[i].nb_elements; - output[i].size = - addr->output[i].nb_elements * - rte_ml_io_type_size_get(metadata->output1[i].model_output_type); - } else { - j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - - rte_memcpy(output[i].name, metadata->output2[j].output_name, - MRVL_ML_OUTPUT_NAME_LEN); - output[i].nb_dims = addr->output[i].nb_dims; - output[i].shape = addr->output[i].shape; - output[i].type = metadata->output2[j].model_output_type; - output[i].nb_elements = addr->output[i].nb_elements; - output[i].size = - addr->output[i].nb_elements * - rte_ml_io_type_size_get(metadata->output2[j].model_output_type); - } + rte_memcpy(output[i].name, layer->info.output[i].name, MRVL_ML_INPUT_NAME_LEN); + output[i].nb_dims = layer->info.output[i].nb_dims; + output[i].shape = &layer->info.output[i].shape[0]; + output[i].type = layer->info.output[i].qtype; + output[i].nb_elements = layer->info.output[i].nb_elements; + output[i].size = layer->info.output[i].nb_elements * + rte_ml_io_type_size_get(layer->info.output[i].qtype); } } diff --git a/drivers/ml/cnxk/cn10k_ml_model.h b/drivers/ml/cnxk/cn10k_ml_model.h index 3128b28db73..206a369ca75 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.h +++ b/drivers/ml/cnxk/cn10k_ml_model.h @@ -13,15 +13,8 @@ #include "cn10k_ml_ocm.h" #include "cn10k_ml_ops.h" -struct cnxk_ml_dev; - -/* Model state */ -enum cn10k_ml_model_state { - ML_CN10K_MODEL_STATE_LOADED, - ML_CN10K_MODEL_STATE_JOB_ACTIVE, - ML_CN10K_MODEL_STATE_STARTED, - ML_CN10K_MODEL_STATE_UNKNOWN, -}; +struct cnxk_ml_model; +struct cnxk_ml_layer; /* Model Metadata : v 2.3.0.1 */ #define MRVL_ML_MODEL_MAGIC_STRING "MRVL" @@ -369,7 +362,7 @@ struct cn10k_ml_model_metadata { }; /* Model address structure */ -struct cn10k_ml_model_addr { +struct cn10k_ml_layer_addr { /* Base DMA address for load */ void *base_dma_addr_load; @@ -408,58 +401,10 @@ struct cn10k_ml_model_addr { /* End tile */ uint8_t tile_end; - - /* Input address and size */ - struct { - /* Number of dimensions in shape */ - uint32_t nb_dims; - - /* Shape of input */ - uint32_t shape[4]; - - /* Number of elements */ - uint32_t nb_elements; - - /* Dequantized input size */ - uint32_t sz_d; - - /* Quantized input size */ - uint32_t sz_q; - } input[MRVL_ML_NUM_INPUT_OUTPUT]; - - /* Output address and size */ - struct { - /* Number of dimensions in shape */ - uint32_t nb_dims; - - /* Shape of input */ - uint32_t shape[4]; - - /* Number of elements */ - uint32_t nb_elements; - - /* Dequantize output size */ - uint32_t sz_d; - - /* Quantized output size */ - uint32_t sz_q; - } output[MRVL_ML_NUM_INPUT_OUTPUT]; - - /* Total size of quantized input */ - uint32_t total_input_sz_q; - - /* Total size of dequantized input */ - uint32_t total_input_sz_d; - - /* Total size of quantized output */ - uint32_t total_output_sz_q; - - /* Total size of dequantized output */ - uint32_t total_output_sz_d; }; /* Model fast-path stats */ -struct cn10k_ml_model_stats { +struct cn10k_ml_layer_stats { /* Total hardware latency, sum of all inferences */ uint64_t hw_latency_tot; @@ -488,59 +433,38 @@ struct cn10k_ml_model_stats { uint64_t fw_reset_count; }; -/* Model Object */ -struct cn10k_ml_model { - /* Device reference */ - struct cnxk_ml_dev *mldev; - - /* Name */ - char name[RTE_ML_STR_MAX]; - - /* ID */ - uint16_t model_id; - - /* Batch size */ - uint32_t batch_size; - - /* Metadata */ +struct cn10k_ml_layer_data { + /* Model / Layer: metadata */ struct cn10k_ml_model_metadata metadata; - /* Address structure */ - struct cn10k_ml_model_addr addr; + /* Layer: address structure */ + struct cn10k_ml_layer_addr addr; - /* Tile and memory information object */ - struct cn10k_ml_ocm_model_map model_mem_map; + /* Layer: Tile and memory information object */ + struct cn10k_ml_ocm_layer_map ocm_map; - /* Internal model information structure - * Size of the buffer = sizeof(struct rte_ml_model_info) - * + num_inputs * sizeof(struct rte_ml_io_info) - * + num_outputs * sizeof(struct rte_ml_io_info). - * Structures would be arranged in the same order in the buffer. - */ - uint8_t *info; - - /* Spinlock, used to update model state */ - plt_spinlock_t lock; - - /* State */ - enum cn10k_ml_model_state state; - - /* Slow-path operations request pointer */ + /* Layer: Slow-path operations request pointer */ struct cn10k_ml_req *req; - /* Stats for burst ops */ - struct cn10k_ml_model_stats *burst_stats; + /* Layer: Stats for burst ops */ + struct cn10k_ml_layer_stats *burst_stats; - /* Stats for sync ops */ - struct cn10k_ml_model_stats *sync_stats; + /* Layer: Stats for sync ops */ + struct cn10k_ml_layer_stats *sync_stats; +}; + +struct cn10k_ml_model_data { + /* Model / Layer: metadata */ + struct cn10k_ml_model_metadata metadata; }; int cn10k_ml_model_metadata_check(uint8_t *buffer, uint64_t size); void cn10k_ml_model_metadata_update(struct cn10k_ml_model_metadata *metadata); -void cn10k_ml_model_addr_update(struct cn10k_ml_model *model, uint8_t *buffer, +void cn10k_ml_layer_addr_update(struct cnxk_ml_layer *layer, uint8_t *buffer, uint8_t *base_dma_addr); +void cn10k_ml_layer_info_update(struct cnxk_ml_layer *layer); int cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_id, uint8_t *buffer, uint16_t *wb_pages, uint16_t *scratch_pages); -void cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cn10k_ml_model *model); +void cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cnxk_ml_model *model); #endif /* _CN10K_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/cn10k_ml_ocm.c b/drivers/ml/cnxk/cn10k_ml_ocm.c index d0f716bccea..639f329f8aa 100644 --- a/drivers/ml/cnxk/cn10k_ml_ocm.c +++ b/drivers/ml/cnxk/cn10k_ml_ocm.c @@ -11,6 +11,7 @@ #include "cn10k_ml_ocm.h" #include "cnxk_ml_dev.h" +#include "cnxk_ml_model.h" /* OCM macros */ #define BYTE_LEN 8 @@ -334,12 +335,14 @@ cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t w } void -cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint64_t tilemask, - int wb_page_start, uint16_t wb_pages, uint16_t scratch_pages) +cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t layer_id, + uint64_t tilemask, int wb_page_start, uint16_t wb_pages, + uint16_t scratch_pages) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; struct cn10k_ml_ocm *ocm; int scratch_page_start; @@ -354,6 +357,7 @@ cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint64_t t cn10k_mldev = &cnxk_mldev->cn10k_mldev; ocm = &cn10k_mldev->ocm; model = dev->data->models[model_id]; + layer = &model->layer[layer_id]; /* Get first set bit, tile_start */ tile_start = 0; @@ -383,8 +387,8 @@ cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint64_t t PLT_MAX(ocm->tile_ocm_info[tile_id].last_wb_page, wb_page_end); } - model->addr.tile_start = tile_start; - model->addr.tile_end = tile_end; + layer->glow.addr.tile_start = tile_start; + layer->glow.addr.tile_end = tile_end; plt_ml_dbg("model_id = %u, tilemask = 0x%016lx", model_id, tilemask); plt_ml_dbg("model_id = %u, wb_page_start = %d, wb_page_end = %d", model_id, wb_page_start, @@ -394,12 +398,14 @@ cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint64_t t } void -cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id) +cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t layer_id) { - struct cn10k_ml_model *local_model; + struct cnxk_ml_model *local_model; + struct cnxk_ml_layer *local_layer; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; struct cn10k_ml_ocm *ocm; int scratch_resize_pages; @@ -410,16 +416,19 @@ cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id) int tile_id; int page_id; uint16_t i; + uint16_t j; cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; ocm = &cn10k_mldev->ocm; model = dev->data->models[model_id]; + layer = &model->layer[layer_id]; /* Update OCM info for WB memory */ - wb_page_start = model->model_mem_map.wb_page_start; - wb_page_end = wb_page_start + model->model_mem_map.wb_pages - 1; - for (tile_id = model->addr.tile_start; tile_id <= model->addr.tile_end; tile_id++) { + wb_page_start = layer->glow.ocm_map.wb_page_start; + wb_page_end = wb_page_start + layer->glow.ocm_map.wb_pages - 1; + for (tile_id = layer->glow.addr.tile_start; tile_id <= layer->glow.addr.tile_end; + tile_id++) { for (page_id = wb_page_start; page_id <= wb_page_end; page_id++) { CLEAR_BIT(ocm->tile_ocm_info[tile_id].ocm_mask[page_id / OCM_MAP_WORD_SIZE], page_id % OCM_MAP_WORD_SIZE); @@ -433,11 +442,19 @@ cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id) scratch_resize_pages = 0; for (i = 0; i < dev->data->nb_models; i++) { local_model = dev->data->models[i]; - if ((i != model_id) && (local_model != NULL)) { - if (IS_BIT_SET(local_model->model_mem_map.tilemask, tile_id)) - scratch_resize_pages = PLT_MAX( - (int)local_model->model_mem_map.scratch_pages, - scratch_resize_pages); + if (local_model == NULL) + continue; + + for (j = 0; j < local_model->nb_layers; j++) { + local_layer = &local_model->layer[j]; + if (local_layer != layer && + local_layer->glow.ocm_map.ocm_reserved) { + if (IS_BIT_SET(local_layer->glow.ocm_map.tilemask, tile_id)) + scratch_resize_pages = + PLT_MAX((int)local_layer->glow.ocm_map + .scratch_pages, + scratch_resize_pages); + } } } diff --git a/drivers/ml/cnxk/cn10k_ml_ocm.h b/drivers/ml/cnxk/cn10k_ml_ocm.h index 3404e7fd651..720f8caf766 100644 --- a/drivers/ml/cnxk/cn10k_ml_ocm.h +++ b/drivers/ml/cnxk/cn10k_ml_ocm.h @@ -27,7 +27,7 @@ struct cn10k_ml_ocm_tile_info { }; /* Model OCM map structure */ -struct cn10k_ml_ocm_model_map { +struct cn10k_ml_ocm_layer_map { /* Status of OCM reservation */ bool ocm_reserved; @@ -77,9 +77,10 @@ struct cn10k_ml_ocm { int cn10k_ml_ocm_tilecount(uint64_t tilemask, int *start, int *end); int cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t wb_pages, uint16_t scratch_pages, uint64_t *tilemask); -void cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint64_t tilemask, - int wb_page_start, uint16_t wb_pages, uint16_t scratch_pages); -void cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id); +void cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t layer_id, + uint64_t tilemask, int wb_page_start, uint16_t wb_pages, + uint16_t scratch_pages); +void cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t layer_id); void cn10k_ml_ocm_print(struct rte_ml_dev *dev, FILE *fp); #endif /* _CN10K_ML_OCM_H_ */ diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 3385bf50c0d..a52509630fe 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -12,6 +12,7 @@ #include "cn10k_ml_ops.h" #include "cnxk_ml_dev.h" +#include "cnxk_ml_model.h" /* ML model macros */ #define CN10K_ML_MODEL_MEMZONE_NAME "ml_cn10k_model_mz" @@ -203,7 +204,7 @@ cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct cn10k_ml_ocm *ocm; char str[STR_LEN]; uint8_t i; @@ -216,77 +217,80 @@ cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) /* Print debug info */ print_line(fp, LINE_LEN); - fprintf(fp, " Model Information (%s)\n", model->metadata.model.name); + fprintf(fp, " Model Information (%s)\n", model->glow.metadata.model.name); print_line(fp, LINE_LEN); - fprintf(fp, "%*s : %s\n", FIELD_LEN, "name", model->metadata.model.name); - fprintf(fp, "%*s : %u.%u.%u.%u\n", FIELD_LEN, "version", model->metadata.model.version[0], - model->metadata.model.version[1], model->metadata.model.version[2], - model->metadata.model.version[3]); + fprintf(fp, "%*s : %s\n", FIELD_LEN, "name", model->glow.metadata.model.name); + fprintf(fp, "%*s : %u.%u.%u.%u\n", FIELD_LEN, "version", + model->glow.metadata.model.version[0], model->glow.metadata.model.version[1], + model->glow.metadata.model.version[2], model->glow.metadata.model.version[3]); if (strlen(model->name) != 0) fprintf(fp, "%*s : %s\n", FIELD_LEN, "debug_name", model->name); fprintf(fp, "%*s : 0x%016lx\n", FIELD_LEN, "model", PLT_U64_CAST(model)); fprintf(fp, "%*s : %u\n", FIELD_LEN, "model_id", model->model_id); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "batch_size", model->metadata.model.batch_size); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_layers", model->metadata.model.num_layers); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "batch_size", model->glow.metadata.model.batch_size); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_layers", model->glow.metadata.model.num_layers); /* Print model state */ - if (model->state == ML_CN10K_MODEL_STATE_LOADED) + if (model->state == ML_CNXK_MODEL_STATE_LOADED) fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "loaded"); - if (model->state == ML_CN10K_MODEL_STATE_JOB_ACTIVE) + if (model->state == ML_CNXK_MODEL_STATE_JOB_ACTIVE) fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "job_active"); - if (model->state == ML_CN10K_MODEL_STATE_STARTED) + if (model->state == ML_CNXK_MODEL_STATE_STARTED) fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "started"); /* Print OCM status */ fprintf(fp, "%*s : %" PRIu64 " bytes\n", FIELD_LEN, "wb_size", - model->metadata.model.ocm_wb_range_end - model->metadata.model.ocm_wb_range_start + - 1); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "wb_pages", model->model_mem_map.wb_pages); + model->glow.metadata.model.ocm_wb_range_end - + model->glow.metadata.model.ocm_wb_range_start + 1); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "wb_pages", model->layer[0].glow.ocm_map.wb_pages); fprintf(fp, "%*s : %" PRIu64 " bytes\n", FIELD_LEN, "scratch_size", - ocm->size_per_tile - model->metadata.model.ocm_tmp_range_floor); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "scratch_pages", model->model_mem_map.scratch_pages); + ocm->size_per_tile - model->glow.metadata.model.ocm_tmp_range_floor); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "scratch_pages", + model->layer[0].glow.ocm_map.scratch_pages); fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_tiles", - model->metadata.model.tile_end - model->metadata.model.tile_start + 1); + model->glow.metadata.model.tile_end - model->glow.metadata.model.tile_start + 1); - if (model->state == ML_CN10K_MODEL_STATE_STARTED) { + if (model->state == ML_CNXK_MODEL_STATE_STARTED) { fprintf(fp, "%*s : 0x%0*" PRIx64 "\n", FIELD_LEN, "tilemask", - ML_CN10K_OCM_NUMTILES / 4, model->model_mem_map.tilemask); + ML_CN10K_OCM_NUMTILES / 4, model->layer[0].glow.ocm_map.tilemask); fprintf(fp, "%*s : 0x%" PRIx64 "\n", FIELD_LEN, "ocm_wb_start", - model->model_mem_map.wb_page_start * cn10k_mldev->ocm.page_size); + model->layer[0].glow.ocm_map.wb_page_start * cn10k_mldev->ocm.page_size); } - fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_inputs", model->metadata.model.num_input); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_outputs", model->metadata.model.num_output); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_inputs", model->glow.metadata.model.num_input); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_outputs", model->glow.metadata.model.num_output); fprintf(fp, "\n"); print_line(fp, LINE_LEN); fprintf(fp, "%8s %16s %12s %18s %12s\n", "input", "input_name", "input_type", "model_input_type", "quantize"); print_line(fp, LINE_LEN); - for (i = 0; i < model->metadata.model.num_input; i++) { + for (i = 0; i < model->glow.metadata.model.num_input; i++) { if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { fprintf(fp, "%8u ", i); - fprintf(fp, "%*s ", 16, model->metadata.input1[i].input_name); - rte_ml_io_type_to_str(model->metadata.input1[i].input_type, str, STR_LEN); + fprintf(fp, "%*s ", 16, model->glow.metadata.input1[i].input_name); + rte_ml_io_type_to_str(model->glow.metadata.input1[i].input_type, str, + STR_LEN); fprintf(fp, "%*s ", 12, str); - rte_ml_io_type_to_str(model->metadata.input1[i].model_input_type, str, + rte_ml_io_type_to_str(model->glow.metadata.input1[i].model_input_type, str, STR_LEN); fprintf(fp, "%*s ", 18, str); fprintf(fp, "%*s", 12, - (model->metadata.input1[i].quantize == 1 ? "Yes" : "No")); + (model->glow.metadata.input1[i].quantize == 1 ? "Yes" : "No")); fprintf(fp, "\n"); } else { j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; fprintf(fp, "%8u ", i); - fprintf(fp, "%*s ", 16, model->metadata.input2[j].input_name); - rte_ml_io_type_to_str(model->metadata.input2[j].input_type, str, STR_LEN); + fprintf(fp, "%*s ", 16, model->glow.metadata.input2[j].input_name); + rte_ml_io_type_to_str(model->glow.metadata.input2[j].input_type, str, + STR_LEN); fprintf(fp, "%*s ", 12, str); - rte_ml_io_type_to_str(model->metadata.input2[j].model_input_type, str, + rte_ml_io_type_to_str(model->glow.metadata.input2[j].model_input_type, str, STR_LEN); fprintf(fp, "%*s ", 18, str); fprintf(fp, "%*s", 12, - (model->metadata.input2[j].quantize == 1 ? "Yes" : "No")); + (model->glow.metadata.input2[j].quantize == 1 ? "Yes" : "No")); fprintf(fp, "\n"); } } @@ -296,29 +300,31 @@ cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) fprintf(fp, "%8s %16s %12s %18s %12s\n", "output", "output_name", "output_type", "model_output_type", "dequantize"); print_line(fp, LINE_LEN); - for (i = 0; i < model->metadata.model.num_output; i++) { + for (i = 0; i < model->glow.metadata.model.num_output; i++) { if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { fprintf(fp, "%8u ", i); - fprintf(fp, "%*s ", 16, model->metadata.output1[i].output_name); - rte_ml_io_type_to_str(model->metadata.output1[i].output_type, str, STR_LEN); - fprintf(fp, "%*s ", 12, str); - rte_ml_io_type_to_str(model->metadata.output1[i].model_output_type, str, + fprintf(fp, "%*s ", 16, model->glow.metadata.output1[i].output_name); + rte_ml_io_type_to_str(model->glow.metadata.output1[i].output_type, str, STR_LEN); + fprintf(fp, "%*s ", 12, str); + rte_ml_io_type_to_str(model->glow.metadata.output1[i].model_output_type, + str, STR_LEN); fprintf(fp, "%*s ", 18, str); fprintf(fp, "%*s", 12, - (model->metadata.output1[i].dequantize == 1 ? "Yes" : "No")); + (model->glow.metadata.output1[i].dequantize == 1 ? "Yes" : "No")); fprintf(fp, "\n"); } else { j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; fprintf(fp, "%8u ", i); - fprintf(fp, "%*s ", 16, model->metadata.output2[j].output_name); - rte_ml_io_type_to_str(model->metadata.output2[j].output_type, str, STR_LEN); - fprintf(fp, "%*s ", 12, str); - rte_ml_io_type_to_str(model->metadata.output2[j].model_output_type, str, + fprintf(fp, "%*s ", 16, model->glow.metadata.output2[j].output_name); + rte_ml_io_type_to_str(model->glow.metadata.output2[j].output_type, str, STR_LEN); + fprintf(fp, "%*s ", 12, str); + rte_ml_io_type_to_str(model->glow.metadata.output2[j].model_output_type, + str, STR_LEN); fprintf(fp, "%*s ", 18, str); fprintf(fp, "%*s", 12, - (model->metadata.output2[j].dequantize == 1 ? "Yes" : "No")); + (model->glow.metadata.output2[j].dequantize == 1 ? "Yes" : "No")); fprintf(fp, "\n"); } } @@ -328,14 +334,14 @@ cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) } static void -cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cn10k_ml_model *model, +cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml_model *model, struct cn10k_ml_req *req, enum cn10k_ml_job_type job_type) { struct cn10k_ml_model_metadata *metadata; - struct cn10k_ml_model_addr *addr; + struct cn10k_ml_layer_addr *addr; - metadata = &model->metadata; - addr = &model->addr; + metadata = &model->glow.metadata; + addr = &model->layer[0].glow.addr; memset(&req->jd, 0, sizeof(struct cn10k_ml_jd)); req->jd.hdr.jce.w0.u64 = 0; @@ -346,7 +352,7 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cn10k_m req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->result); if (job_type == ML_CN10K_JOB_TYPE_MODEL_START) { - if (!model->metadata.model.ocm_relocatable) + if (!model->glow.metadata.model.ocm_relocatable) req->jd.hdr.sp_flags = ML_CN10K_SP_FLAGS_OCM_NONRELOCATABLE; else req->jd.hdr.sp_flags = 0x0; @@ -386,7 +392,7 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cn10k_m req->jd.model_start.output.s.ddr_range_end = metadata->model.ddr_output_range_end; req->extended_args.start.ddr_scratch_base_address = PLT_U64_CAST( - roc_ml_addr_ap2mlip(&cn10k_mldev->roc, model->addr.scratch_base_addr)); + roc_ml_addr_ap2mlip(&cn10k_mldev->roc, addr->scratch_base_addr)); req->extended_args.start.ddr_scratch_range_start = metadata->model.ddr_scratch_range_start; req->extended_args.start.ddr_scratch_range_end = @@ -446,7 +452,7 @@ cn10k_ml_xstats_init(struct rte_ml_dev *dev) cn10k_mldev = &cnxk_mldev->cn10k_mldev; /* Allocate memory for xstats entries. Don't allocate during reconfigure */ - nb_stats = RTE_DIM(device_stats) + ML_CN10K_MAX_MODELS * RTE_DIM(model_stats); + nb_stats = RTE_DIM(device_stats) + ML_CNXK_MAX_MODELS * RTE_DIM(model_stats); if (cn10k_mldev->xstats.entries == NULL) cn10k_mldev->xstats.entries = rte_zmalloc( "cn10k_ml_xstats", sizeof(struct cn10k_ml_xstats_entry) * nb_stats, @@ -473,7 +479,7 @@ cn10k_ml_xstats_init(struct rte_ml_dev *dev) cn10k_mldev->xstats.count_mode_device = stat_id; /* Initialize model xstats */ - for (model = 0; model < ML_CN10K_MAX_MODELS; model++) { + for (model = 0; model < ML_CNXK_MAX_MODELS; model++) { cn10k_mldev->xstats.offset_for_model[model] = stat_id; for (i = 0; i < RTE_DIM(model_stats); i++) { @@ -522,7 +528,7 @@ cn10k_ml_xstats_model_name_update(struct rte_ml_dev *dev, uint16_t model_id) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; uint16_t rclk_freq; uint16_t sclk_freq; uint16_t stat_id; @@ -544,7 +550,7 @@ cn10k_ml_xstats_model_name_update(struct rte_ml_dev *dev, uint16_t model_id) for (i = 0; i < RTE_DIM(model_stats); i++) { snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", - model->metadata.model.name, model_stats[i].name, suffix); + model->layer[0].glow.metadata.model.name, model_stats[i].name, suffix); stat_id++; } } @@ -577,9 +583,9 @@ cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, do { \ value = 0; \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - value += model->burst_stats[qp_id].str##_latency_tot; \ - count += model->burst_stats[qp_id].dequeued_count - \ - model->burst_stats[qp_id].str##_reset_count; \ + value += model->layer[0].glow.burst_stats[qp_id].str##_latency_tot; \ + count += model->layer[0].glow.burst_stats[qp_id].dequeued_count - \ + model->layer[0].glow.burst_stats[qp_id].str##_reset_count; \ } \ if (count != 0) \ value = value / count; \ @@ -589,9 +595,10 @@ cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, do { \ value = UINT64_MAX; \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - value = PLT_MIN(value, model->burst_stats[qp_id].str##_latency_min); \ - count += model->burst_stats[qp_id].dequeued_count - \ - model->burst_stats[qp_id].str##_reset_count; \ + value = PLT_MIN( \ + value, model->layer[0].glow.burst_stats[qp_id].str##_latency_min); \ + count += model->layer[0].glow.burst_stats[qp_id].dequeued_count - \ + model->layer[0].glow.burst_stats[qp_id].str##_reset_count; \ } \ if (count == 0) \ value = 0; \ @@ -601,9 +608,10 @@ cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, do { \ value = 0; \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - value = PLT_MAX(value, model->burst_stats[qp_id].str##_latency_max); \ - count += model->burst_stats[qp_id].dequeued_count - \ - model->burst_stats[qp_id].str##_reset_count; \ + value = PLT_MAX( \ + value, model->layer[0].glow.burst_stats[qp_id].str##_latency_max); \ + count += model->layer[0].glow.burst_stats[qp_id].dequeued_count - \ + model->layer[0].glow.burst_stats[qp_id].str##_reset_count; \ } \ if (count == 0) \ value = 0; \ @@ -612,7 +620,7 @@ cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, static uint64_t cn10k_ml_model_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx, enum cn10k_ml_xstats_type type) { - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; uint16_t rclk_freq; /* MHz */ uint16_t sclk_freq; /* MHz */ uint64_t count = 0; @@ -693,28 +701,28 @@ cn10k_ml_device_xstats_reset(struct rte_ml_dev *dev, const uint16_t stat_ids[], #define ML_AVG_RESET_FOREACH_QP(dev, model, qp_id, str) \ do { \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - model->burst_stats[qp_id].str##_latency_tot = 0; \ - model->burst_stats[qp_id].str##_reset_count = \ - model->burst_stats[qp_id].dequeued_count; \ + model->layer[0].glow.burst_stats[qp_id].str##_latency_tot = 0; \ + model->layer[0].glow.burst_stats[qp_id].str##_reset_count = \ + model->layer[0].glow.burst_stats[qp_id].dequeued_count; \ } \ } while (0) #define ML_MIN_RESET_FOREACH_QP(dev, model, qp_id, str) \ do { \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) \ - model->burst_stats[qp_id].str##_latency_min = UINT64_MAX; \ + model->layer[0].glow.burst_stats[qp_id].str##_latency_min = UINT64_MAX; \ } while (0) #define ML_MAX_RESET_FOREACH_QP(dev, model, qp_id, str) \ do { \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) \ - model->burst_stats[qp_id].str##_latency_max = 0; \ + model->layer[0].glow.burst_stats[qp_id].str##_latency_max = 0; \ } while (0) static void cn10k_ml_reset_model_stat(struct rte_ml_dev *dev, uint16_t model_id, enum cn10k_ml_xstats_type type) { - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; uint32_t qp_id; model = dev->data->models[model_id]; @@ -750,7 +758,7 @@ cn10k_ml_model_xstats_reset(struct rte_ml_dev *dev, int32_t model_id, const uint struct cn10k_ml_xstats_entry *xs; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; int32_t lcl_model_id = 0; uint16_t start_id; uint16_t end_id; @@ -759,7 +767,7 @@ cn10k_ml_model_xstats_reset(struct rte_ml_dev *dev, int32_t model_id, const uint cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; - for (i = 0; i < ML_CN10K_MAX_MODELS; i++) { + for (i = 0; i < ML_CNXK_MAX_MODELS; i++) { if (model_id == -1) { model = dev->data->models[i]; if (model == NULL) /* Skip inactive models */ @@ -804,7 +812,7 @@ static int cn10k_ml_cache_model_data(struct rte_ml_dev *dev, uint16_t model_id) { struct rte_ml_model_info *info; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct rte_ml_buff_seg seg[2]; struct rte_ml_buff_seg *inp; struct rte_ml_buff_seg *out; @@ -855,7 +863,7 @@ cn10k_ml_cache_model_data(struct rte_ml_dev *dev, uint16_t model_id) op.input = &inp; op.output = &out; - memset(model->req, 0, sizeof(struct cn10k_ml_req)); + memset(model->layer[0].glow.req, 0, sizeof(struct cn10k_ml_req)); ret = cn10k_ml_inference_sync(dev, &op); plt_memzone_free(mz); @@ -876,7 +884,7 @@ cn10k_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) memset(dev_info, 0, sizeof(struct rte_ml_dev_info)); dev_info->driver_name = dev->device->driver->name; - dev_info->max_models = ML_CN10K_MAX_MODELS; + dev_info->max_models = ML_CNXK_MAX_MODELS; if (cn10k_mldev->hw_queue_lock) dev_info->max_queue_pairs = ML_CN10K_MAX_QP_PER_DEVICE_SL; else @@ -896,7 +904,7 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c struct rte_ml_dev_info dev_info; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct cn10k_ml_ocm *ocm; struct cn10k_ml_qp *qp; uint16_t model_id; @@ -1002,11 +1010,11 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c for (model_id = 0; model_id < dev->data->nb_models; model_id++) { model = dev->data->models[model_id]; if (model != NULL) { - if (model->state == ML_CN10K_MODEL_STATE_STARTED) { + if (model->state == ML_CNXK_MODEL_STATE_STARTED) { if (cn10k_ml_model_stop(dev, model_id) != 0) plt_err("Could not stop model %u", model_id); } - if (model->state == ML_CN10K_MODEL_STATE_LOADED) { + if (model->state == ML_CNXK_MODEL_STATE_LOADED) { if (cn10k_ml_model_unload(dev, model_id) != 0) plt_err("Could not unload model %u", model_id); } @@ -1094,7 +1102,7 @@ cn10k_ml_dev_close(struct rte_ml_dev *dev) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct cn10k_ml_qp *qp; uint16_t model_id; uint16_t qp_id; @@ -1112,11 +1120,11 @@ cn10k_ml_dev_close(struct rte_ml_dev *dev) for (model_id = 0; model_id < dev->data->nb_models; model_id++) { model = dev->data->models[model_id]; if (model != NULL) { - if (model->state == ML_CN10K_MODEL_STATE_STARTED) { + if (model->state == ML_CNXK_MODEL_STATE_STARTED) { if (cn10k_ml_model_stop(dev, model_id) != 0) plt_err("Could not stop model %u", model_id); } - if (model->state == ML_CN10K_MODEL_STATE_LOADED) { + if (model->state == ML_CNXK_MODEL_STATE_LOADED) { if (cn10k_ml_model_unload(dev, model_id) != 0) plt_err("Could not unload model %u", model_id); } @@ -1295,7 +1303,7 @@ cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mod xstats_mode_count = cn10k_mldev->xstats.count_mode_device; break; case RTE_ML_DEV_XSTATS_MODEL: - if (model_id >= ML_CN10K_MAX_MODELS) + if (model_id >= ML_CNXK_MAX_MODELS) break; xstats_mode_count = cn10k_mldev->xstats.count_per_model[model_id]; break; @@ -1387,7 +1395,7 @@ cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode xstats_mode_count = cn10k_mldev->xstats.count_mode_device; break; case RTE_ML_DEV_XSTATS_MODEL: - if (model_id >= ML_CN10K_MAX_MODELS) + if (model_id >= ML_CNXK_MAX_MODELS) return -EINVAL; xstats_mode_count = cn10k_mldev->xstats.count_per_model[model_id]; break; @@ -1448,7 +1456,7 @@ cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct cn10k_ml_fw *fw; uint32_t head_loc; @@ -1589,7 +1597,7 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, { struct cn10k_ml_model_metadata *metadata; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; char str[RTE_MEMZONE_NAMESIZE]; const struct plt_memzone *mz; @@ -1644,9 +1652,9 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, metadata->model.num_input * sizeof(struct rte_ml_io_info) + metadata->model.num_output * sizeof(struct rte_ml_io_info); model_info_size = PLT_ALIGN_CEIL(model_info_size, ML_CN10K_ALIGN_SIZE); - model_stats_size = (dev->data->nb_queue_pairs + 1) * sizeof(struct cn10k_ml_model_stats); + model_stats_size = (dev->data->nb_queue_pairs + 1) * sizeof(struct cn10k_ml_layer_stats); - mz_size = PLT_ALIGN_CEIL(sizeof(struct cn10k_ml_model), ML_CN10K_ALIGN_SIZE) + + mz_size = PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), ML_CN10K_ALIGN_SIZE) + 2 * model_data_size + model_scratch_size + model_info_size + PLT_ALIGN_CEIL(sizeof(struct cn10k_ml_req), ML_CN10K_ALIGN_SIZE) + model_stats_size; @@ -1660,62 +1668,85 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, } model = mz->addr; - model->mldev = cnxk_mldev; + model->cnxk_mldev = cnxk_mldev; model->model_id = idx; + dev->data->models[idx] = model; - rte_memcpy(&model->metadata, params->addr, sizeof(struct cn10k_ml_model_metadata)); - cn10k_ml_model_metadata_update(&model->metadata); + rte_memcpy(&model->glow.metadata, params->addr, sizeof(struct cn10k_ml_model_metadata)); + cn10k_ml_model_metadata_update(&model->glow.metadata); + + /* Set model name */ + rte_memcpy(model->name, (char *)model->glow.metadata.model.name, 64); /* Enable support for batch_size of 256 */ - if (model->metadata.model.batch_size == 0) + if (model->glow.metadata.model.batch_size == 0) model->batch_size = 256; else - model->batch_size = model->metadata.model.batch_size; + model->batch_size = model->glow.metadata.model.batch_size; + + /* Since the number of layers that the driver would be handling for glow models is + * always 1. consider the entire model as a model with single layer. This would + * ignore the num_layers from metadata. + */ + model->nb_layers = 1; + + /* Copy metadata to internal buffer */ + rte_memcpy(&model->layer[0].glow.metadata, params->addr, + sizeof(struct cn10k_ml_model_metadata)); + cn10k_ml_model_metadata_update(&model->layer[0].glow.metadata); + model->layer[0].model = model; /* Set DMA base address */ base_dma_addr = PLT_PTR_ADD( - mz->addr, PLT_ALIGN_CEIL(sizeof(struct cn10k_ml_model), ML_CN10K_ALIGN_SIZE)); - cn10k_ml_model_addr_update(model, params->addr, base_dma_addr); - model->addr.scratch_base_addr = PLT_PTR_ADD(base_dma_addr, 2 * model_data_size); + mz->addr, PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), ML_CN10K_ALIGN_SIZE)); + cn10k_ml_layer_addr_update(&model->layer[0], params->addr, base_dma_addr); + model->layer[0].glow.addr.scratch_base_addr = + PLT_PTR_ADD(base_dma_addr, 2 * model_data_size); /* Copy data from load to run. run address to be used by MLIP */ - rte_memcpy(model->addr.base_dma_addr_run, model->addr.base_dma_addr_load, model_data_size); + rte_memcpy(model->layer[0].glow.addr.base_dma_addr_run, + model->layer[0].glow.addr.base_dma_addr_load, model_data_size); + + /* Update internal I/O data structure */ + cn10k_ml_layer_info_update(&model->layer[0]); /* Initialize model_mem_map */ - memset(&model->model_mem_map, 0, sizeof(struct cn10k_ml_ocm_model_map)); - model->model_mem_map.ocm_reserved = false; - model->model_mem_map.tilemask = 0; - model->model_mem_map.wb_page_start = -1; - model->model_mem_map.wb_pages = wb_pages; - model->model_mem_map.scratch_pages = scratch_pages; + memset(&model->layer[0].glow.ocm_map, 0, sizeof(struct cn10k_ml_ocm_layer_map)); + model->layer[0].glow.ocm_map.ocm_reserved = false; + model->layer[0].glow.ocm_map.tilemask = 0; + model->layer[0].glow.ocm_map.wb_page_start = -1; + model->layer[0].glow.ocm_map.wb_pages = wb_pages; + model->layer[0].glow.ocm_map.scratch_pages = scratch_pages; /* Set model info */ - model->info = PLT_PTR_ADD(model->addr.scratch_base_addr, model_scratch_size); + model->info = PLT_PTR_ADD(model->layer[0].glow.addr.scratch_base_addr, model_scratch_size); cn10k_ml_model_info_set(dev, model); /* Set slow-path request address and state */ - model->req = PLT_PTR_ADD(model->info, model_info_size); + model->layer[0].glow.req = PLT_PTR_ADD(model->info, model_info_size); /* Reset burst and sync stats */ - model->burst_stats = PLT_PTR_ADD( - model->req, PLT_ALIGN_CEIL(sizeof(struct cn10k_ml_req), ML_CN10K_ALIGN_SIZE)); + model->layer[0].glow.burst_stats = + PLT_PTR_ADD(model->layer[0].glow.req, + PLT_ALIGN_CEIL(sizeof(struct cn10k_ml_req), ML_CN10K_ALIGN_SIZE)); for (qp_id = 0; qp_id < dev->data->nb_queue_pairs + 1; qp_id++) { - model->burst_stats[qp_id].hw_latency_tot = 0; - model->burst_stats[qp_id].hw_latency_min = UINT64_MAX; - model->burst_stats[qp_id].hw_latency_max = 0; - model->burst_stats[qp_id].fw_latency_tot = 0; - model->burst_stats[qp_id].fw_latency_min = UINT64_MAX; - model->burst_stats[qp_id].fw_latency_max = 0; - model->burst_stats[qp_id].hw_reset_count = 0; - model->burst_stats[qp_id].fw_reset_count = 0; - model->burst_stats[qp_id].dequeued_count = 0; - } - model->sync_stats = - PLT_PTR_ADD(model->burst_stats, - dev->data->nb_queue_pairs * sizeof(struct cn10k_ml_model_stats)); + model->layer[0].glow.burst_stats[qp_id].hw_latency_tot = 0; + model->layer[0].glow.burst_stats[qp_id].hw_latency_min = UINT64_MAX; + model->layer[0].glow.burst_stats[qp_id].hw_latency_max = 0; + model->layer[0].glow.burst_stats[qp_id].fw_latency_tot = 0; + model->layer[0].glow.burst_stats[qp_id].fw_latency_min = UINT64_MAX; + model->layer[0].glow.burst_stats[qp_id].fw_latency_max = 0; + model->layer[0].glow.burst_stats[qp_id].hw_reset_count = 0; + model->layer[0].glow.burst_stats[qp_id].fw_reset_count = 0; + model->layer[0].glow.burst_stats[qp_id].dequeued_count = 0; + } + + model->layer[0].glow.sync_stats = + PLT_PTR_ADD(model->layer[0].glow.burst_stats, + dev->data->nb_queue_pairs * sizeof(struct cn10k_ml_layer_stats)); plt_spinlock_init(&model->lock); - model->state = ML_CN10K_MODEL_STATE_LOADED; + model->state = ML_CNXK_MODEL_STATE_LOADED; dev->data->models[idx] = model; cnxk_mldev->nb_models_loaded++; @@ -1731,7 +1762,7 @@ int cn10k_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) { char str[RTE_MEMZONE_NAMESIZE]; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct cnxk_ml_dev *cnxk_mldev; cnxk_mldev = dev->data->dev_private; @@ -1742,7 +1773,7 @@ cn10k_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) return -EINVAL; } - if (model->state != ML_CN10K_MODEL_STATE_LOADED) { + if (model->state != ML_CNXK_MODEL_STATE_LOADED) { plt_err("Cannot unload. Model in use."); return -EBUSY; } @@ -1759,7 +1790,7 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct cn10k_ml_ocm *ocm; struct cn10k_ml_req *req; @@ -1784,7 +1815,7 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) } /* Prepare JD */ - req = model->req; + req = model->layer[0].glow.req; cn10k_ml_prep_sp_job_descriptor(cn10k_mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_START); req->result.error_code.u64 = 0x0; req->result.user_ptr = NULL; @@ -1792,63 +1823,66 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) plt_write64(ML_CNXK_POLL_JOB_START, &req->status); plt_wmb(); - num_tiles = model->metadata.model.tile_end - model->metadata.model.tile_start + 1; + num_tiles = model->layer[0].glow.metadata.model.tile_end - + model->layer[0].glow.metadata.model.tile_start + 1; locked = false; while (!locked) { if (plt_spinlock_trylock(&model->lock) != 0) { - if (model->state == ML_CN10K_MODEL_STATE_STARTED) { + if (model->state == ML_CNXK_MODEL_STATE_STARTED) { plt_ml_dbg("Model already started, model = 0x%016lx", PLT_U64_CAST(model)); plt_spinlock_unlock(&model->lock); return 1; } - if (model->state == ML_CN10K_MODEL_STATE_JOB_ACTIVE) { + if (model->state == ML_CNXK_MODEL_STATE_JOB_ACTIVE) { plt_err("A slow-path job is active for the model = 0x%016lx", PLT_U64_CAST(model)); plt_spinlock_unlock(&model->lock); return -EBUSY; } - model->state = ML_CN10K_MODEL_STATE_JOB_ACTIVE; + model->state = ML_CNXK_MODEL_STATE_JOB_ACTIVE; plt_spinlock_unlock(&model->lock); locked = true; } } - while (!model->model_mem_map.ocm_reserved) { + while (!model->layer[0].glow.ocm_map.ocm_reserved) { if (plt_spinlock_trylock(&ocm->lock) != 0) { wb_page_start = cn10k_ml_ocm_tilemask_find( - dev, num_tiles, model->model_mem_map.wb_pages, - model->model_mem_map.scratch_pages, &tilemask); + dev, num_tiles, model->layer[0].glow.ocm_map.wb_pages, + model->layer[0].glow.ocm_map.scratch_pages, &tilemask); if (wb_page_start == -1) { plt_err("Free pages not available on OCM tiles"); plt_err("Failed to start model = 0x%016lx, name = %s", - PLT_U64_CAST(model), model->metadata.model.name); + PLT_U64_CAST(model), + model->layer[0].glow.metadata.model.name); plt_spinlock_unlock(&ocm->lock); return -ENOMEM; } - model->model_mem_map.tilemask = tilemask; - model->model_mem_map.wb_page_start = wb_page_start; + model->layer[0].glow.ocm_map.tilemask = tilemask; + model->layer[0].glow.ocm_map.wb_page_start = wb_page_start; - cn10k_ml_ocm_reserve_pages( - dev, model->model_id, model->model_mem_map.tilemask, - model->model_mem_map.wb_page_start, model->model_mem_map.wb_pages, - model->model_mem_map.scratch_pages); - model->model_mem_map.ocm_reserved = true; + cn10k_ml_ocm_reserve_pages(dev, model->model_id, 0, + model->layer[0].glow.ocm_map.tilemask, + model->layer[0].glow.ocm_map.wb_page_start, + model->layer[0].glow.ocm_map.wb_pages, + model->layer[0].glow.ocm_map.scratch_pages); + model->layer[0].glow.ocm_map.ocm_reserved = true; plt_spinlock_unlock(&ocm->lock); } } /* Update JD */ - cn10k_ml_ocm_tilecount(model->model_mem_map.tilemask, &tile_start, &tile_end); + cn10k_ml_ocm_tilecount(model->layer[0].glow.ocm_map.tilemask, &tile_start, &tile_end); req->jd.model_start.tilemask = GENMASK_ULL(tile_end, tile_start); req->jd.model_start.ocm_wb_base_address = - model->model_mem_map.wb_page_start * ocm->page_size; + model->layer[0].glow.ocm_map.wb_page_start * ocm->page_size; job_enqueued = false; job_dequeued = false; @@ -1881,10 +1915,10 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) while (!locked) { if (plt_spinlock_trylock(&model->lock) != 0) { if (ret == 0) { - model->state = ML_CN10K_MODEL_STATE_STARTED; + model->state = ML_CNXK_MODEL_STATE_STARTED; cnxk_mldev->nb_models_started++; } else { - model->state = ML_CN10K_MODEL_STATE_UNKNOWN; + model->state = ML_CNXK_MODEL_STATE_UNKNOWN; } plt_spinlock_unlock(&model->lock); @@ -1892,12 +1926,12 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) } } - if (model->state == ML_CN10K_MODEL_STATE_UNKNOWN) { - while (model->model_mem_map.ocm_reserved) { + if (model->state == ML_CNXK_MODEL_STATE_UNKNOWN) { + while (model->layer[0].glow.ocm_map.ocm_reserved) { if (plt_spinlock_trylock(&ocm->lock) != 0) { - cn10k_ml_ocm_free_pages(dev, model->model_id); - model->model_mem_map.ocm_reserved = false; - model->model_mem_map.tilemask = 0x0; + cn10k_ml_ocm_free_pages(dev, model->model_id, 0); + model->layer[0].glow.ocm_map.ocm_reserved = false; + model->layer[0].glow.ocm_map.tilemask = 0x0; plt_spinlock_unlock(&ocm->lock); } } @@ -1918,7 +1952,7 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct cn10k_ml_ocm *ocm; struct cn10k_ml_req *req; @@ -1938,7 +1972,7 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) } /* Prepare JD */ - req = model->req; + req = model->layer[0].glow.req; cn10k_ml_prep_sp_job_descriptor(cn10k_mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_STOP); req->result.error_code.u64 = 0x0; req->result.user_ptr = NULL; @@ -1949,31 +1983,31 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) locked = false; while (!locked) { if (plt_spinlock_trylock(&model->lock) != 0) { - if (model->state == ML_CN10K_MODEL_STATE_LOADED) { + if (model->state == ML_CNXK_MODEL_STATE_LOADED) { plt_ml_dbg("Model not started, model = 0x%016lx", PLT_U64_CAST(model)); plt_spinlock_unlock(&model->lock); return 1; } - if (model->state == ML_CN10K_MODEL_STATE_JOB_ACTIVE) { + if (model->state == ML_CNXK_MODEL_STATE_JOB_ACTIVE) { plt_err("A slow-path job is active for the model = 0x%016lx", PLT_U64_CAST(model)); plt_spinlock_unlock(&model->lock); return -EBUSY; } - model->state = ML_CN10K_MODEL_STATE_JOB_ACTIVE; + model->state = ML_CNXK_MODEL_STATE_JOB_ACTIVE; plt_spinlock_unlock(&model->lock); locked = true; } } - while (model->model_mem_map.ocm_reserved) { + while (model->layer[0].glow.ocm_map.ocm_reserved) { if (plt_spinlock_trylock(&ocm->lock) != 0) { - cn10k_ml_ocm_free_pages(dev, model->model_id); - model->model_mem_map.ocm_reserved = false; - model->model_mem_map.tilemask = 0x0; + cn10k_ml_ocm_free_pages(dev, model->model_id, 0); + model->layer[0].glow.ocm_map.ocm_reserved = false; + model->layer[0].glow.ocm_map.tilemask = 0x0; plt_spinlock_unlock(&ocm->lock); } } @@ -2009,7 +2043,7 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) while (!locked) { if (plt_spinlock_trylock(&model->lock) != 0) { cnxk_mldev->nb_models_stopped++; - model->state = ML_CN10K_MODEL_STATE_LOADED; + model->state = ML_CNXK_MODEL_STATE_LOADED; plt_spinlock_unlock(&model->lock); locked = true; } @@ -2022,7 +2056,7 @@ static int cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_model_info *model_info) { - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; model = dev->data->models[model_id]; @@ -2041,7 +2075,7 @@ cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, static int cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buffer) { - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; size_t size; model = dev->data->models[model_id]; @@ -2051,19 +2085,23 @@ cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *bu return -EINVAL; } - if (model->state == ML_CN10K_MODEL_STATE_UNKNOWN) + if (model->state == ML_CNXK_MODEL_STATE_UNKNOWN) return -1; - else if (model->state != ML_CN10K_MODEL_STATE_LOADED) + else if (model->state != ML_CNXK_MODEL_STATE_LOADED) return -EBUSY; - size = model->metadata.init_model.file_size + model->metadata.main_model.file_size + - model->metadata.finish_model.file_size + model->metadata.weights_bias.file_size; + size = model->layer[0].glow.metadata.init_model.file_size + + model->layer[0].glow.metadata.main_model.file_size + + model->layer[0].glow.metadata.finish_model.file_size + + model->layer[0].glow.metadata.weights_bias.file_size; /* Update model weights & bias */ - rte_memcpy(model->addr.wb_load_addr, buffer, model->metadata.weights_bias.file_size); + rte_memcpy(model->layer[0].glow.addr.wb_load_addr, buffer, + model->layer[0].glow.metadata.weights_bias.file_size); /* Copy data from load to run. run address to be used by MLIP */ - rte_memcpy(model->addr.base_dma_addr_run, model->addr.base_dma_addr_load, size); + rte_memcpy(model->layer[0].glow.addr.base_dma_addr_run, + model->layer[0].glow.addr.base_dma_addr_load, size); return 0; } @@ -2072,7 +2110,7 @@ static int cn10k_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buff_seg **dbuffer, struct rte_ml_buff_seg **qbuffer) { - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; uint8_t model_input_type; uint8_t *lcl_dbuffer; uint8_t *lcl_qbuffer; @@ -2092,57 +2130,58 @@ cn10k_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_bu lcl_dbuffer = dbuffer[0]->addr; lcl_qbuffer = qbuffer[0]->addr; - for (i = 0; i < model->metadata.model.num_input; i++) { + for (i = 0; i < model->layer[0].glow.metadata.model.num_input; i++) { if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - input_type = model->metadata.input1[i].input_type; - model_input_type = model->metadata.input1[i].model_input_type; - qscale = model->metadata.input1[i].qscale; + input_type = model->layer[0].glow.metadata.input1[i].input_type; + model_input_type = model->layer[0].glow.metadata.input1[i].model_input_type; + qscale = model->layer[0].glow.metadata.input1[i].qscale; } else { j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - input_type = model->metadata.input2[j].input_type; - model_input_type = model->metadata.input2[j].model_input_type; - qscale = model->metadata.input2[j].qscale; + input_type = model->layer[0].glow.metadata.input2[j].input_type; + model_input_type = model->layer[0].glow.metadata.input2[j].model_input_type; + qscale = model->layer[0].glow.metadata.input2[j].qscale; } if (input_type == model_input_type) { - rte_memcpy(lcl_qbuffer, lcl_dbuffer, model->addr.input[i].sz_d); + rte_memcpy(lcl_qbuffer, lcl_dbuffer, model->layer[0].info.input[i].sz_d); } else { - switch (model->metadata.input1[i].model_input_type) { + switch (model->layer[0].glow.metadata.input1[i].model_input_type) { case RTE_ML_IO_TYPE_INT8: - ret = rte_ml_io_float32_to_int8(qscale, - model->addr.input[i].nb_elements, - lcl_dbuffer, lcl_qbuffer); + ret = rte_ml_io_float32_to_int8( + qscale, model->layer[0].info.input[i].nb_elements, + lcl_dbuffer, lcl_qbuffer); break; case RTE_ML_IO_TYPE_UINT8: - ret = rte_ml_io_float32_to_uint8(qscale, - model->addr.input[i].nb_elements, - lcl_dbuffer, lcl_qbuffer); + ret = rte_ml_io_float32_to_uint8( + qscale, model->layer[0].info.input[i].nb_elements, + lcl_dbuffer, lcl_qbuffer); break; case RTE_ML_IO_TYPE_INT16: - ret = rte_ml_io_float32_to_int16(qscale, - model->addr.input[i].nb_elements, - lcl_dbuffer, lcl_qbuffer); + ret = rte_ml_io_float32_to_int16( + qscale, model->layer[0].info.input[i].nb_elements, + lcl_dbuffer, lcl_qbuffer); break; case RTE_ML_IO_TYPE_UINT16: - ret = rte_ml_io_float32_to_uint16(qscale, - model->addr.input[i].nb_elements, - lcl_dbuffer, lcl_qbuffer); + ret = rte_ml_io_float32_to_uint16( + qscale, model->layer[0].info.input[i].nb_elements, + lcl_dbuffer, lcl_qbuffer); break; case RTE_ML_IO_TYPE_FP16: - ret = rte_ml_io_float32_to_float16(model->addr.input[i].nb_elements, - lcl_dbuffer, lcl_qbuffer); + ret = rte_ml_io_float32_to_float16( + model->layer[0].info.input[i].nb_elements, lcl_dbuffer, + lcl_qbuffer); break; default: plt_err("Unsupported model_input_type[%u] : %u", i, - model->metadata.input1[i].model_input_type); + model->layer[0].glow.metadata.input1[i].model_input_type); ret = -ENOTSUP; } if (ret < 0) return ret; } - lcl_dbuffer += model->addr.input[i].sz_d; - lcl_qbuffer += model->addr.input[i].sz_q; + lcl_dbuffer += model->layer[0].info.input[i].sz_d; + lcl_qbuffer += model->layer[0].info.input[i].sz_q; } return 0; @@ -2152,7 +2191,7 @@ static int cn10k_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buff_seg **qbuffer, struct rte_ml_buff_seg **dbuffer) { - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; uint8_t model_output_type; uint8_t *lcl_qbuffer; uint8_t *lcl_dbuffer; @@ -2172,58 +2211,60 @@ cn10k_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_ lcl_dbuffer = dbuffer[0]->addr; lcl_qbuffer = qbuffer[0]->addr; - for (i = 0; i < model->metadata.model.num_output; i++) { + for (i = 0; i < model->layer[0].glow.metadata.model.num_output; i++) { if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - output_type = model->metadata.output1[i].output_type; - model_output_type = model->metadata.output1[i].model_output_type; - dscale = model->metadata.output1[i].dscale; + output_type = model->layer[0].glow.metadata.output1[i].output_type; + model_output_type = + model->layer[0].glow.metadata.output1[i].model_output_type; + dscale = model->layer[0].glow.metadata.output1[i].dscale; } else { j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - output_type = model->metadata.output2[j].output_type; - model_output_type = model->metadata.output2[j].model_output_type; - dscale = model->metadata.output2[j].dscale; + output_type = model->layer[0].glow.metadata.output2[j].output_type; + model_output_type = + model->layer[0].glow.metadata.output2[j].model_output_type; + dscale = model->layer[0].glow.metadata.output2[j].dscale; } if (output_type == model_output_type) { - rte_memcpy(lcl_dbuffer, lcl_qbuffer, model->addr.output[i].sz_q); + rte_memcpy(lcl_dbuffer, lcl_qbuffer, model->layer[0].info.output[i].sz_q); } else { - switch (model->metadata.output1[i].model_output_type) { + switch (model->layer[0].glow.metadata.output1[i].model_output_type) { case RTE_ML_IO_TYPE_INT8: - ret = rte_ml_io_int8_to_float32(dscale, - model->addr.output[i].nb_elements, - lcl_qbuffer, lcl_dbuffer); + ret = rte_ml_io_int8_to_float32( + dscale, model->layer[0].info.output[i].nb_elements, + lcl_qbuffer, lcl_dbuffer); break; case RTE_ML_IO_TYPE_UINT8: - ret = rte_ml_io_uint8_to_float32(dscale, - model->addr.output[i].nb_elements, - lcl_qbuffer, lcl_dbuffer); + ret = rte_ml_io_uint8_to_float32( + dscale, model->layer[0].info.output[i].nb_elements, + lcl_qbuffer, lcl_dbuffer); break; case RTE_ML_IO_TYPE_INT16: - ret = rte_ml_io_int16_to_float32(dscale, - model->addr.output[i].nb_elements, - lcl_qbuffer, lcl_dbuffer); + ret = rte_ml_io_int16_to_float32( + dscale, model->layer[0].info.output[i].nb_elements, + lcl_qbuffer, lcl_dbuffer); break; case RTE_ML_IO_TYPE_UINT16: - ret = rte_ml_io_uint16_to_float32(dscale, - model->addr.output[i].nb_elements, - lcl_qbuffer, lcl_dbuffer); + ret = rte_ml_io_uint16_to_float32( + dscale, model->layer[0].info.output[i].nb_elements, + lcl_qbuffer, lcl_dbuffer); break; case RTE_ML_IO_TYPE_FP16: ret = rte_ml_io_float16_to_float32( - model->addr.output[i].nb_elements, lcl_qbuffer, + model->layer[0].info.output[i].nb_elements, lcl_qbuffer, lcl_dbuffer); break; default: plt_err("Unsupported model_output_type[%u] : %u", i, - model->metadata.output1[i].model_output_type); + model->layer[0].glow.metadata.output1[i].model_output_type); ret = -ENOTSUP; } if (ret < 0) return ret; } - lcl_qbuffer += model->addr.output[i].sz_q; - lcl_dbuffer += model->addr.output[i].sz_d; + lcl_qbuffer += model->layer[0].info.output[i].sz_q; + lcl_dbuffer += model->layer[0].info.output[i].sz_d; } return 0; @@ -2251,10 +2292,10 @@ static __rte_always_inline void cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cn10k_ml_result *result, struct rte_ml_op *op) { - struct cn10k_ml_model_stats *stats; + struct cn10k_ml_layer_stats *stats; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct cn10k_ml_qp *qp; uint64_t hw_latency; uint64_t fw_latency; @@ -2264,9 +2305,9 @@ cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cn10k_ml_result if (likely(qp_id >= 0)) { qp = dev->data->queue_pairs[qp_id]; qp->stats.dequeued_count++; - stats = &model->burst_stats[qp_id]; + stats = &model->layer[0].glow.burst_stats[qp_id]; } else { - stats = model->sync_stats; + stats = model->layer[0].glow.sync_stats; } if (unlikely(stats->dequeued_count == stats->hw_reset_count)) { @@ -2470,7 +2511,7 @@ cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_model *model; + struct cnxk_ml_model *model; struct cn10k_ml_req *req; bool timeout; int ret = 0; @@ -2478,7 +2519,7 @@ cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; model = dev->data->models[op->model_id]; - req = model->req; + req = model->layer[0].glow.req; cn10k_ml_set_poll_addr(req); cn10k_ml_prep_fp_job_descriptor(cn10k_mldev, req, op); diff --git a/drivers/ml/cnxk/cnxk_ml_io.h b/drivers/ml/cnxk/cnxk_ml_io.h new file mode 100644 index 00000000000..29ec7ec5112 --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_io.h @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#ifndef _CNXK_ML_IO_H_ +#define _CNXK_ML_IO_H_ + +#include + +/* Maximum number of models per device */ +#define ML_CNXK_MAX_MODELS 16 + +/* Maximum number of layers per model */ +#define ML_CNXK_MODEL_MAX_LAYERS 32 + +/* Maximum number of inputs or outputs per layer or model */ +#define ML_CNXK_MODEL_MAX_INPUT_OUTPUT 32 + +/* Maximum number of dimensions per I/O shape */ +#define ML_CNXK_MODEL_MAX_DIMS 8 + +/* Input / Output structure */ +struct cnxk_ml_io { + /* name */ + char name[RTE_ML_STR_MAX]; + + /* dequantized data type */ + enum rte_ml_io_type dtype; + + /* quantized data type */ + enum rte_ml_io_type qtype; + + /* Number of dimensions in shape */ + uint32_t nb_dims; + + /* Shape of input */ + uint32_t shape[ML_CNXK_MODEL_MAX_DIMS]; + + /* Number of elements */ + uint32_t nb_elements; + + /* Dequantized input size */ + uint32_t sz_d; + + /* Quantized input size */ + uint32_t sz_q; + + /* Scale */ + float scale; +}; + +/* Model / Layer IO structure */ +struct cnxk_ml_io_info { + /* Number of inputs */ + uint16_t nb_inputs; + + /* Model / Layer inputs */ + struct cnxk_ml_io input[ML_CNXK_MODEL_MAX_INPUT_OUTPUT]; + + /* Total size of quantized input */ + uint32_t total_input_sz_q; + + /* Total size of dequantized input */ + uint32_t total_input_sz_d; + + /* Number of outputs */ + uint16_t nb_outputs; + + /* Model / Layer outputs */ + struct cnxk_ml_io output[ML_CNXK_MODEL_MAX_INPUT_OUTPUT]; + + /* Total size of quantized output */ + uint32_t total_output_sz_q; + + /* Total size of dequantized output */ + uint32_t total_output_sz_d; +}; + +#endif /* _CNXK_ML_IO_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_model.c b/drivers/ml/cnxk/cnxk_ml_model.c new file mode 100644 index 00000000000..3d735ced3e1 --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_model.c @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#include + +#include "cnxk_ml_model.h" diff --git a/drivers/ml/cnxk/cnxk_ml_model.h b/drivers/ml/cnxk/cnxk_ml_model.h new file mode 100644 index 00000000000..a2994dbb71e --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_model.h @@ -0,0 +1,111 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#ifndef _CNXK_ML_MODEL_H_ +#define _CNXK_ML_MODEL_H_ + +#include + +#include + +#include "cn10k_ml_model.h" + +#include "cnxk_ml_io.h" + +struct cnxk_ml_dev; +struct cnxk_ml_model; + +/* Model state */ +enum cnxk_ml_model_state { + /* Unknown state */ + ML_CNXK_MODEL_STATE_UNKNOWN, + + /* Model loaded */ + ML_CNXK_MODEL_STATE_LOADED, + + /* A slow-path job is active, start or stop */ + ML_CNXK_MODEL_STATE_JOB_ACTIVE, + + /* Model started */ + ML_CNXK_MODEL_STATE_STARTED, +}; + +/* Layer state */ +enum cnxk_ml_layer_state { + /* Unknown state */ + ML_CNXK_LAYER_STATE_UNKNOWN, + + /* Layer loaded */ + ML_CNXK_LAYER_STATE_LOADED, + + /* A slow-path job is active, start or stop */ + ML_CNXK_LAYER_STATE_JOB_ACTIVE, + + /* Layer started */ + ML_CNXK_LAYER_STATE_STARTED, +}; + +/* Layer object */ +struct cnxk_ml_layer { + /* Name*/ + char name[RTE_ML_STR_MAX]; + + /* Model handle */ + struct cnxk_ml_model *model; + + /* Index mapped with firmware's model_id */ + uint16_t index; + + /* Input / Output */ + struct cnxk_ml_io_info info; + + /* Batch size */ + uint32_t batch_size; + + /* State */ + enum cnxk_ml_layer_state state; + + /* Glow layer specific data */ + struct cn10k_ml_layer_data glow; +}; + +/* Model Object */ +struct cnxk_ml_model { + /* Device reference */ + struct cnxk_ml_dev *cnxk_mldev; + + /* ID */ + uint16_t model_id; + + /* Name */ + char name[RTE_ML_STR_MAX]; + + /* Model specific data - glow */ + struct cn10k_ml_model_data glow; + + /* Batch size */ + uint32_t batch_size; + + /* Number of layers */ + uint16_t nb_layers; + + /* Layer info */ + struct cnxk_ml_layer layer[ML_CNXK_MODEL_MAX_LAYERS]; + + /* State */ + enum cnxk_ml_model_state state; + + /* Internal model information structure + * Size of the buffer = sizeof(struct rte_ml_model_info) + * + num_inputs * sizeof(struct rte_ml_io_info) + * + num_outputs * sizeof(struct rte_ml_io_info). + * Structures would be arranged in the same order in the buffer. + */ + uint8_t *info; + + /* Spinlock, used to update model state */ + plt_spinlock_t lock; +}; + +#endif /* _CNXK_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 03a2d4ecf2f..72e03b15b5b 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -13,6 +13,8 @@ driver_sdk_headers = files( 'cn10k_ml_model.h', 'cn10k_ml_ocm.h', 'cnxk_ml_dev.h', + 'cnxk_ml_io.h', + 'cnxk_ml_model.h', ) sources = files( @@ -21,6 +23,7 @@ sources = files( 'cn10k_ml_model.c', 'cn10k_ml_ocm.c', 'cnxk_ml_dev.c', + 'cnxk_ml_model.c', ) deps += ['mldev', 'common_cnxk', 'kvargs', 'hash'] From patchwork Wed Aug 30 15:58:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130887 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4A80D41FD1; Wed, 30 Aug 2023 18:00:31 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E5913402B7; Wed, 30 Aug 2023 17:59:42 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 0A7704028D for ; Wed, 30 Aug 2023 17:59:36 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7p0004368 for ; Wed, 30 Aug 2023 08:59:36 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=1pnAgzZ1QMPmZ7OC5oaJQ0B0uuzYw5hBDygGLKKg+aA=; b=dkXgXArkKY9hLE+uvE6cKnu+R77W3F00FgCDU4uAvXONbviLxsUcuOgMcI3Y9MsML/1e TQYiP9xUUmXcgvgKlZqTfeFuMOMzII/KPZZnFwNtnifzLOyOupeJWpm9kxPx/cGNYC1y 95Bti1VMfwVX2XWusogvG0+avvf2sDq4gwHRHTGU449+o8xjpU++zRvxzy7YCQVCC7Uj ZtBIbicN1IZL09FSDFEVeZ1P5siiuJgH8gNse8TKA5r5C6tPIys3Ur3h3K1VEeBAHHFI JU+aMPJ0Bf8J6e1OC8iMsZ78nvwv5k3rWI8oFWlD/nF6iJ4s3r4s3TssxMxjdj3XpWA4 IQ== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysst5-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:35 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:33 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:33 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 5A1C63F707E; Wed, 30 Aug 2023 08:59:33 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 05/34] ml/cnxk: add generic cnxk request structure Date: Wed, 30 Aug 2023 08:58:55 -0700 Message-ID: <20230830155927.3566-6-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: 6_nKpJ9F_-tGbkxU9bhbB_xbA5dqFo8d X-Proofpoint-ORIG-GUID: 6_nKpJ9F_-tGbkxU9bhbB_xbA5dqFo8d X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added generic cnxk request structure. Moved common fields from cn10k structures to cnxk structure. Moved job related structures and enumerations to ops headers. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.c | 70 ++++--- drivers/ml/cnxk/cn10k_ml_dev.h | 269 +------------------------ drivers/ml/cnxk/cn10k_ml_model.c | 6 +- drivers/ml/cnxk/cn10k_ml_model.h | 4 +- drivers/ml/cnxk/cn10k_ml_ops.c | 329 +++++++++++++++++-------------- drivers/ml/cnxk/cn10k_ml_ops.h | 296 +++++++++++++++++++++++---- drivers/ml/cnxk/cnxk_ml_ops.c | 7 + drivers/ml/cnxk/cnxk_ml_ops.h | 63 ++++++ drivers/ml/cnxk/meson.build | 2 + 9 files changed, 558 insertions(+), 488 deletions(-) create mode 100644 drivers/ml/cnxk/cnxk_ml_ops.c create mode 100644 drivers/ml/cnxk/cnxk_ml_ops.h diff --git a/drivers/ml/cnxk/cn10k_ml_dev.c b/drivers/ml/cnxk/cn10k_ml_dev.c index 367fb7014c4..f6e05cfc472 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.c +++ b/drivers/ml/cnxk/cn10k_ml_dev.c @@ -23,6 +23,7 @@ #include "cn10k_ml_ops.h" #include "cnxk_ml_dev.h" +#include "cnxk_ml_ops.h" #define CN10K_ML_FW_PATH "fw_path" #define CN10K_ML_FW_ENABLE_DPE_WARNINGS "enable_dpe_warnings" @@ -457,20 +458,23 @@ cn10k_ml_pci_remove(struct rte_pci_device *pci_dev) static void cn10k_ml_fw_print_info(struct cn10k_ml_fw *fw) { - plt_info("ML Firmware Version = %s", fw->req->jd.fw_load.version); - - plt_ml_dbg("Firmware capabilities = 0x%016lx", fw->req->jd.fw_load.cap.u64); - plt_ml_dbg("Version = %s", fw->req->jd.fw_load.version); - plt_ml_dbg("core0_debug_ptr = 0x%016lx", fw->req->jd.fw_load.debug.core0_debug_ptr); - plt_ml_dbg("core1_debug_ptr = 0x%016lx", fw->req->jd.fw_load.debug.core1_debug_ptr); - plt_ml_dbg("debug_buffer_size = %u bytes", fw->req->jd.fw_load.debug.debug_buffer_size); + plt_info("ML Firmware Version = %s", fw->req->cn10k_req.jd.fw_load.version); + + plt_ml_dbg("Firmware capabilities = 0x%016lx", fw->req->cn10k_req.jd.fw_load.cap.u64); + plt_ml_dbg("Version = %s", fw->req->cn10k_req.jd.fw_load.version); + plt_ml_dbg("core0_debug_ptr = 0x%016lx", + fw->req->cn10k_req.jd.fw_load.debug.core0_debug_ptr); + plt_ml_dbg("core1_debug_ptr = 0x%016lx", + fw->req->cn10k_req.jd.fw_load.debug.core1_debug_ptr); + plt_ml_dbg("debug_buffer_size = %u bytes", + fw->req->cn10k_req.jd.fw_load.debug.debug_buffer_size); plt_ml_dbg("core0_exception_buffer = 0x%016lx", - fw->req->jd.fw_load.debug.core0_exception_buffer); + fw->req->cn10k_req.jd.fw_load.debug.core0_exception_buffer); plt_ml_dbg("core1_exception_buffer = 0x%016lx", - fw->req->jd.fw_load.debug.core1_exception_buffer); + fw->req->cn10k_req.jd.fw_load.debug.core1_exception_buffer); plt_ml_dbg("exception_state_size = %u bytes", - fw->req->jd.fw_load.debug.exception_state_size); - plt_ml_dbg("flags = 0x%016lx", fw->req->jd.fw_load.flags); + fw->req->cn10k_req.jd.fw_load.debug.exception_state_size); + plt_ml_dbg("flags = 0x%016lx", fw->req->cn10k_req.jd.fw_load.flags); } uint64_t @@ -515,29 +519,30 @@ cn10k_ml_fw_load_asim(struct cn10k_ml_fw *fw) roc_ml_reg_save(&cn10k_mldev->roc, ML_MLR_BASE); /* Update FW load completion structure */ - fw->req->jd.hdr.jce.w1.u64 = PLT_U64_CAST(&fw->req->status); - fw->req->jd.hdr.job_type = ML_CN10K_JOB_TYPE_FIRMWARE_LOAD; - fw->req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &fw->req->result); - fw->req->jd.fw_load.flags = cn10k_ml_fw_flags_get(fw); - plt_write64(ML_CNXK_POLL_JOB_START, &fw->req->status); + fw->req->cn10k_req.jd.hdr.jce.w1.u64 = PLT_U64_CAST(&fw->req->cn10k_req.status); + fw->req->cn10k_req.jd.hdr.job_type = ML_CN10K_JOB_TYPE_FIRMWARE_LOAD; + fw->req->cn10k_req.jd.hdr.result = + roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &fw->req->cn10k_req.result); + fw->req->cn10k_req.jd.fw_load.flags = cn10k_ml_fw_flags_get(fw); + plt_write64(ML_CNXK_POLL_JOB_START, &fw->req->cn10k_req.status); plt_wmb(); /* Enqueue FW load through scratch registers */ timeout = true; timeout_cycle = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); - roc_ml_scratch_enqueue(&cn10k_mldev->roc, &fw->req->jd); + roc_ml_scratch_enqueue(&cn10k_mldev->roc, &fw->req->cn10k_req.jd); plt_rmb(); do { if (roc_ml_scratch_is_done_bit_set(&cn10k_mldev->roc) && - (plt_read64(&fw->req->status) == ML_CNXK_POLL_JOB_FINISH)) { + (plt_read64(&fw->req->cn10k_req.status) == ML_CNXK_POLL_JOB_FINISH)) { timeout = false; break; } } while (plt_tsc_cycles() < timeout_cycle); /* Check firmware load status, clean-up and exit on failure. */ - if ((!timeout) && (fw->req->result.error_code.u64 == 0)) { + if ((!timeout) && (fw->req->cn10k_req.result.error_code == 0)) { cn10k_ml_fw_print_info(fw); } else { /* Set ML to disable new jobs */ @@ -711,29 +716,30 @@ cn10k_ml_fw_load_cn10ka(struct cn10k_ml_fw *fw, void *buffer, uint64_t size) plt_ml_dbg("ML_SW_RST_CTRL => 0x%08x", reg_val32); /* (12) Wait for notification from firmware that ML is ready for job execution. */ - fw->req->jd.hdr.jce.w1.u64 = PLT_U64_CAST(&fw->req->status); - fw->req->jd.hdr.job_type = ML_CN10K_JOB_TYPE_FIRMWARE_LOAD; - fw->req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &fw->req->result); - fw->req->jd.fw_load.flags = cn10k_ml_fw_flags_get(fw); - plt_write64(ML_CNXK_POLL_JOB_START, &fw->req->status); + fw->req->cn10k_req.jd.hdr.jce.w1.u64 = PLT_U64_CAST(&fw->req->cn10k_req.status); + fw->req->cn10k_req.jd.hdr.job_type = ML_CN10K_JOB_TYPE_FIRMWARE_LOAD; + fw->req->cn10k_req.jd.hdr.result = + roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &fw->req->cn10k_req.result); + fw->req->cn10k_req.jd.fw_load.flags = cn10k_ml_fw_flags_get(fw); + plt_write64(ML_CNXK_POLL_JOB_START, &fw->req->cn10k_req.status); plt_wmb(); /* Enqueue FW load through scratch registers */ timeout = true; timeout_cycle = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); - roc_ml_scratch_enqueue(&cn10k_mldev->roc, &fw->req->jd); + roc_ml_scratch_enqueue(&cn10k_mldev->roc, &fw->req->cn10k_req.jd); plt_rmb(); do { if (roc_ml_scratch_is_done_bit_set(&cn10k_mldev->roc) && - (plt_read64(&fw->req->status) == ML_CNXK_POLL_JOB_FINISH)) { + (plt_read64(&fw->req->cn10k_req.status) == ML_CNXK_POLL_JOB_FINISH)) { timeout = false; break; } } while (plt_tsc_cycles() < timeout_cycle); /* Check firmware load status, clean-up and exit on failure. */ - if ((!timeout) && (fw->req->result.error_code.u64 == 0)) { + if ((!timeout) && (fw->req->cn10k_req.result.error_code == 0)) { cn10k_ml_fw_print_info(fw); } else { /* Set ML to disable new jobs */ @@ -823,11 +829,11 @@ cn10k_ml_fw_load(struct cnxk_ml_dev *cnxk_mldev) } /* Reserve memzone for firmware load completion and data */ - mz_size = sizeof(struct cn10k_ml_req) + fw_size + FW_STACK_BUFFER_SIZE + + mz_size = sizeof(struct cnxk_ml_req) + fw_size + FW_STACK_BUFFER_SIZE + FW_DEBUG_BUFFER_SIZE + FW_EXCEPTION_BUFFER_SIZE; } else if (roc_env_is_asim()) { /* Reserve memzone for firmware load completion */ - mz_size = sizeof(struct cn10k_ml_req); + mz_size = sizeof(struct cnxk_ml_req); } mz = plt_memzone_reserve_aligned(FW_MEMZONE_NAME, mz_size, 0, ML_CN10K_ALIGN_SIZE); @@ -839,8 +845,8 @@ cn10k_ml_fw_load(struct cnxk_ml_dev *cnxk_mldev) fw->req = mz->addr; /* Reset firmware load completion structure */ - memset(&fw->req->jd, 0, sizeof(struct cn10k_ml_jd)); - memset(&fw->req->jd.fw_load.version[0], '\0', MLDEV_FIRMWARE_VERSION_LENGTH); + memset(&fw->req->cn10k_req.jd, 0, sizeof(struct cn10k_ml_jd)); + memset(&fw->req->cn10k_req.jd.fw_load.version[0], '\0', MLDEV_FIRMWARE_VERSION_LENGTH); /* Reset device, if in active state */ if (roc_ml_mlip_is_enabled(&cn10k_mldev->roc)) @@ -848,7 +854,7 @@ cn10k_ml_fw_load(struct cnxk_ml_dev *cnxk_mldev) /* Load firmware */ if (roc_env_is_emulator() || roc_env_is_hw()) { - fw->data = PLT_PTR_ADD(mz->addr, sizeof(struct cn10k_ml_req)); + fw->data = PLT_PTR_ADD(mz->addr, sizeof(struct cnxk_ml_req)); ret = cn10k_ml_fw_load_cn10ka(fw, fw_buffer, fw_size); rte_free(fw_buffer); } else if (roc_env_is_asim()) { diff --git a/drivers/ml/cnxk/cn10k_ml_dev.h b/drivers/ml/cnxk/cn10k_ml_dev.h index 99ff0a344a2..1852d4f6c9a 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.h +++ b/drivers/ml/cnxk/cn10k_ml_dev.h @@ -17,9 +17,6 @@ extern struct rte_ml_dev_ops ml_dev_dummy_ops; /* Marvell OCTEON CN10K ML PMD device name */ #define MLDEV_NAME_CN10K_PMD ml_cn10k -/* Firmware version string length */ -#define MLDEV_FIRMWARE_VERSION_LENGTH 32 - /* Device alignment size */ #define ML_CN10K_ALIGN_SIZE 128 @@ -52,17 +49,8 @@ extern struct rte_ml_dev_ops ml_dev_dummy_ops; #endif struct cnxk_ml_dev; -struct cn10k_ml_req; -struct cn10k_ml_qp; - -/* Job types */ -enum cn10k_ml_job_type { - ML_CN10K_JOB_TYPE_MODEL_RUN = 0, - ML_CN10K_JOB_TYPE_MODEL_STOP, - ML_CN10K_JOB_TYPE_MODEL_START, - ML_CN10K_JOB_TYPE_FIRMWARE_LOAD, - ML_CN10K_JOB_TYPE_FIRMWARE_SELFTEST, -}; +struct cnxk_ml_req; +struct cnxk_ml_qp; /* Error types enumeration */ enum cn10k_ml_error_etype { @@ -112,251 +100,6 @@ union cn10k_ml_error_code { uint64_t u64; }; -/* Firmware stats */ -struct cn10k_ml_fw_stats { - /* Firmware start cycle */ - uint64_t fw_start; - - /* Firmware end cycle */ - uint64_t fw_end; - - /* Hardware start cycle */ - uint64_t hw_start; - - /* Hardware end cycle */ - uint64_t hw_end; -}; - -/* Result structure */ -struct cn10k_ml_result { - /* Job error code */ - union cn10k_ml_error_code error_code; - - /* Firmware stats */ - struct cn10k_ml_fw_stats stats; - - /* User context pointer */ - void *user_ptr; -}; - -/* Firmware capability structure */ -union cn10k_ml_fw_cap { - uint64_t u64; - - struct { - /* CMPC completion support */ - uint64_t cmpc_completions : 1; - - /* Poll mode completion support */ - uint64_t poll_completions : 1; - - /* SSO completion support */ - uint64_t sso_completions : 1; - - /* Support for model side loading */ - uint64_t side_load_model : 1; - - /* Batch execution */ - uint64_t batch_run : 1; - - /* Max number of models to be loaded in parallel */ - uint64_t max_models : 8; - - /* Firmware statistics */ - uint64_t fw_stats : 1; - - /* Hardware statistics */ - uint64_t hw_stats : 1; - - /* Max number of batches */ - uint64_t max_num_batches : 16; - - uint64_t rsvd : 33; - } s; -}; - -/* Firmware debug info structure */ -struct cn10k_ml_fw_debug { - /* ACC core 0 debug buffer */ - uint64_t core0_debug_ptr; - - /* ACC core 1 debug buffer */ - uint64_t core1_debug_ptr; - - /* ACC core 0 exception state buffer */ - uint64_t core0_exception_buffer; - - /* ACC core 1 exception state buffer */ - uint64_t core1_exception_buffer; - - /* Debug buffer size per core */ - uint32_t debug_buffer_size; - - /* Exception state dump size */ - uint32_t exception_state_size; -}; - -/* Job descriptor header (32 bytes) */ -struct cn10k_ml_jd_header { - /* Job completion structure */ - struct ml_jce_s jce; - - /* Model ID */ - uint64_t model_id : 8; - - /* Job type */ - uint64_t job_type : 8; - - /* Flags for fast-path jobs */ - uint64_t fp_flags : 16; - - /* Flags for slow-path jobs */ - uint64_t sp_flags : 16; - uint64_t rsvd : 16; - - /* Job result pointer */ - uint64_t *result; -}; - -/* Extra arguments for job descriptor */ -union cn10k_ml_jd_extended_args { - struct cn10k_ml_jd_extended_args_section_start { - /** DDR Scratch base address */ - uint64_t ddr_scratch_base_address; - - /** DDR Scratch range start */ - uint64_t ddr_scratch_range_start; - - /** DDR Scratch range end */ - uint64_t ddr_scratch_range_end; - - uint8_t rsvd[104]; - } start; -}; - -/* Job descriptor structure */ -struct cn10k_ml_jd { - /* Job descriptor header (32 bytes) */ - struct cn10k_ml_jd_header hdr; - - union { - struct cn10k_ml_jd_section_fw_load { - /* Firmware capability structure (8 bytes) */ - union cn10k_ml_fw_cap cap; - - /* Firmware version (32 bytes) */ - uint8_t version[MLDEV_FIRMWARE_VERSION_LENGTH]; - - /* Debug capability structure (40 bytes) */ - struct cn10k_ml_fw_debug debug; - - /* Flags to control error handling */ - uint64_t flags; - - uint8_t rsvd[8]; - } fw_load; - - struct cn10k_ml_jd_section_model_start { - /* Extended arguments */ - uint64_t extended_args; - - /* Destination model start address in DDR relative to ML_MLR_BASE */ - uint64_t model_dst_ddr_addr; - - /* Offset to model init section in the model */ - uint64_t model_init_offset : 32; - - /* Size of init section in the model */ - uint64_t model_init_size : 32; - - /* Offset to model main section in the model */ - uint64_t model_main_offset : 32; - - /* Size of main section in the model */ - uint64_t model_main_size : 32; - - /* Offset to model finish section in the model */ - uint64_t model_finish_offset : 32; - - /* Size of finish section in the model */ - uint64_t model_finish_size : 32; - - /* Offset to WB in model bin */ - uint64_t model_wb_offset : 32; - - /* Number of model layers */ - uint64_t num_layers : 8; - - /* Number of gather entries, 0 means linear input mode (= no gather) */ - uint64_t num_gather_entries : 8; - - /* Number of scatter entries 0 means linear input mode (= no scatter) */ - uint64_t num_scatter_entries : 8; - - /* Tile mask to load model */ - uint64_t tilemask : 8; - - /* Batch size of model */ - uint64_t batch_size : 32; - - /* OCM WB base address */ - uint64_t ocm_wb_base_address : 32; - - /* OCM WB range start */ - uint64_t ocm_wb_range_start : 32; - - /* OCM WB range End */ - uint64_t ocm_wb_range_end : 32; - - /* DDR WB address */ - uint64_t ddr_wb_base_address; - - /* DDR WB range start */ - uint64_t ddr_wb_range_start : 32; - - /* DDR WB range end */ - uint64_t ddr_wb_range_end : 32; - - union { - /* Points to gather list if num_gather_entries > 0 */ - void *gather_list; - struct { - /* Linear input mode */ - uint64_t ddr_range_start : 32; - uint64_t ddr_range_end : 32; - } s; - } input; - - union { - /* Points to scatter list if num_scatter_entries > 0 */ - void *scatter_list; - struct { - /* Linear output mode */ - uint64_t ddr_range_start : 32; - uint64_t ddr_range_end : 32; - } s; - } output; - } model_start; - - struct cn10k_ml_jd_section_model_stop { - uint8_t rsvd[96]; - } model_stop; - - struct cn10k_ml_jd_section_model_run { - /* Address of the input for the run relative to ML_MLR_BASE */ - uint64_t input_ddr_addr; - - /* Address of the output for the run relative to ML_MLR_BASE */ - uint64_t output_ddr_addr; - - /* Number of batches to run in variable batch processing */ - uint16_t num_batches; - - uint8_t rsvd[78]; - } model_run; - }; -}; - /* ML firmware structure */ struct cn10k_ml_fw { /* Device reference */ @@ -375,7 +118,7 @@ struct cn10k_ml_fw { uint8_t *data; /* Firmware load / handshake request structure */ - struct cn10k_ml_req *req; + struct cnxk_ml_req *req; }; /* Extended stats types enum */ @@ -488,9 +231,9 @@ struct cn10k_ml_dev { bool (*ml_jcmdq_enqueue)(struct roc_ml *roc_ml, struct ml_job_cmd_s *job_cmd); /* Poll handling function pointers */ - void (*set_poll_addr)(struct cn10k_ml_req *req); - void (*set_poll_ptr)(struct cn10k_ml_req *req); - uint64_t (*get_poll_ptr)(struct cn10k_ml_req *req); + void (*set_poll_addr)(struct cnxk_ml_req *req); + void (*set_poll_ptr)(struct cnxk_ml_req *req); + uint64_t (*get_poll_ptr)(struct cnxk_ml_req *req); }; uint64_t cn10k_ml_fw_flags_get(struct cn10k_ml_fw *fw); diff --git a/drivers/ml/cnxk/cn10k_ml_model.c b/drivers/ml/cnxk/cn10k_ml_model.c index 0ea6520bf78..2a0ae44cfd5 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.c +++ b/drivers/ml/cnxk/cn10k_ml_model.c @@ -12,6 +12,7 @@ #include "cnxk_ml_dev.h" #include "cnxk_ml_model.h" +#include "cnxk_ml_ops.h" static enum rte_ml_io_type cn10k_ml_io_type_map(uint8_t type) @@ -551,7 +552,6 @@ void cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cnxk_ml_model *model) { struct cn10k_ml_model_metadata *metadata; - struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; struct rte_ml_model_info *info; struct rte_ml_io_info *output; @@ -560,7 +560,6 @@ cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cnxk_ml_model *model) uint8_t i; cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; metadata = &model->glow.metadata; info = PLT_PTR_CAST(model->info); input = PLT_PTR_ADD(info, sizeof(struct rte_ml_model_info)); @@ -577,7 +576,8 @@ cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cnxk_ml_model *model) info->io_layout = RTE_ML_IO_LAYOUT_PACKED; info->min_batches = model->batch_size; info->max_batches = - cn10k_mldev->fw.req->jd.fw_load.cap.s.max_num_batches / model->batch_size; + cnxk_mldev->cn10k_mldev.fw.req->cn10k_req.jd.fw_load.cap.s.max_num_batches / + model->batch_size; info->nb_inputs = metadata->model.num_input; info->input_info = input; info->nb_outputs = metadata->model.num_output; diff --git a/drivers/ml/cnxk/cn10k_ml_model.h b/drivers/ml/cnxk/cn10k_ml_model.h index 206a369ca75..74ada1531a8 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.h +++ b/drivers/ml/cnxk/cn10k_ml_model.h @@ -11,10 +11,10 @@ #include "cn10k_ml_dev.h" #include "cn10k_ml_ocm.h" -#include "cn10k_ml_ops.h" struct cnxk_ml_model; struct cnxk_ml_layer; +struct cnxk_ml_req; /* Model Metadata : v 2.3.0.1 */ #define MRVL_ML_MODEL_MAGIC_STRING "MRVL" @@ -444,7 +444,7 @@ struct cn10k_ml_layer_data { struct cn10k_ml_ocm_layer_map ocm_map; /* Layer: Slow-path operations request pointer */ - struct cn10k_ml_req *req; + struct cnxk_ml_req *req; /* Layer: Stats for burst ops */ struct cn10k_ml_layer_stats *burst_stats; diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index a52509630fe..2b1fa08154d 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -13,6 +13,7 @@ #include "cnxk_ml_dev.h" #include "cnxk_ml_model.h" +#include "cnxk_ml_ops.h" /* ML model macros */ #define CN10K_ML_MODEL_MEMZONE_NAME "ml_cn10k_model_mz" @@ -80,31 +81,31 @@ print_line(FILE *fp, int len) } static inline void -cn10k_ml_set_poll_addr(struct cn10k_ml_req *req) +cn10k_ml_set_poll_addr(struct cnxk_ml_req *req) { - req->compl_W1 = PLT_U64_CAST(&req->status); + req->status = &req->cn10k_req.status; } static inline void -cn10k_ml_set_poll_ptr(struct cn10k_ml_req *req) +cn10k_ml_set_poll_ptr(struct cnxk_ml_req *req) { - plt_write64(ML_CNXK_POLL_JOB_START, req->compl_W1); + plt_write64(ML_CNXK_POLL_JOB_START, req->status); } static inline uint64_t -cn10k_ml_get_poll_ptr(struct cn10k_ml_req *req) +cn10k_ml_get_poll_ptr(struct cnxk_ml_req *req) { - return plt_read64(req->compl_W1); + return plt_read64(req->status); } static void qp_memzone_name_get(char *name, int size, int dev_id, int qp_id) { - snprintf(name, size, "cn10k_ml_qp_mem_%u:%u", dev_id, qp_id); + snprintf(name, size, "cnxk_ml_qp_mem_%u:%u", dev_id, qp_id); } static int -cn10k_ml_qp_destroy(const struct rte_ml_dev *dev, struct cn10k_ml_qp *qp) +cnxk_ml_qp_destroy(const struct rte_ml_dev *dev, struct cnxk_ml_qp *qp) { const struct rte_memzone *qp_mem; char name[RTE_MEMZONE_NAMESIZE]; @@ -124,14 +125,14 @@ cn10k_ml_qp_destroy(const struct rte_ml_dev *dev, struct cn10k_ml_qp *qp) static int cn10k_ml_dev_queue_pair_release(struct rte_ml_dev *dev, uint16_t queue_pair_id) { - struct cn10k_ml_qp *qp; + struct cnxk_ml_qp *qp; int ret; qp = dev->data->queue_pairs[queue_pair_id]; if (qp == NULL) return -EINVAL; - ret = cn10k_ml_qp_destroy(dev, qp); + ret = cnxk_ml_qp_destroy(dev, qp); if (ret) { plt_err("Could not destroy queue pair %u", queue_pair_id); return ret; @@ -142,18 +143,18 @@ cn10k_ml_dev_queue_pair_release(struct rte_ml_dev *dev, uint16_t queue_pair_id) return 0; } -static struct cn10k_ml_qp * -cn10k_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_desc, int socket_id) +static struct cnxk_ml_qp * +cnxk_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_desc, int socket_id) { const struct rte_memzone *qp_mem; char name[RTE_MEMZONE_NAMESIZE]; - struct cn10k_ml_qp *qp; + struct cnxk_ml_qp *qp; uint32_t len; uint8_t *va; uint64_t i; /* Allocate queue pair */ - qp = rte_zmalloc_socket("cn10k_ml_pmd_queue_pair", sizeof(struct cn10k_ml_qp), ROC_ALIGN, + qp = rte_zmalloc_socket("cn10k_ml_pmd_queue_pair", sizeof(struct cnxk_ml_qp), ROC_ALIGN, socket_id); if (qp == NULL) { plt_err("Could not allocate queue pair"); @@ -161,7 +162,7 @@ cn10k_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_des } /* For request queue */ - len = nb_desc * sizeof(struct cn10k_ml_req); + len = nb_desc * sizeof(struct cnxk_ml_req); qp_memzone_name_get(name, RTE_MEMZONE_NAMESIZE, dev->data->dev_id, qp_id); qp_mem = rte_memzone_reserve_aligned( name, len, socket_id, RTE_MEMZONE_SIZE_HINT_ONLY | RTE_MEMZONE_256MB, ROC_ALIGN); @@ -175,7 +176,7 @@ cn10k_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_des /* Initialize Request queue */ qp->id = qp_id; - qp->queue.reqs = (struct cn10k_ml_req *)va; + qp->queue.reqs = (struct cnxk_ml_req *)va; qp->queue.head = 0; qp->queue.tail = 0; qp->queue.wait_cycles = ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); @@ -187,8 +188,9 @@ cn10k_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_des /* Initialize job command */ for (i = 0; i < qp->nb_desc; i++) { - memset(&qp->queue.reqs[i].jd, 0, sizeof(struct cn10k_ml_jd)); - qp->queue.reqs[i].jcmd.w1.s.jobptr = PLT_U64_CAST(&qp->queue.reqs[i].jd); + memset(&qp->queue.reqs[i].cn10k_req.jd, 0, sizeof(struct cn10k_ml_jd)); + qp->queue.reqs[i].cn10k_req.jcmd.w1.s.jobptr = + PLT_U64_CAST(&qp->queue.reqs[i].cn10k_req.jd); } return qp; @@ -335,7 +337,7 @@ cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) static void cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml_model *model, - struct cn10k_ml_req *req, enum cn10k_ml_job_type job_type) + struct cnxk_ml_req *req, enum cn10k_ml_job_type job_type) { struct cn10k_ml_model_metadata *metadata; struct cn10k_ml_layer_addr *addr; @@ -343,79 +345,88 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml metadata = &model->glow.metadata; addr = &model->layer[0].glow.addr; - memset(&req->jd, 0, sizeof(struct cn10k_ml_jd)); - req->jd.hdr.jce.w0.u64 = 0; - req->jd.hdr.jce.w1.u64 = PLT_U64_CAST(&req->status); - req->jd.hdr.model_id = model->model_id; - req->jd.hdr.job_type = job_type; - req->jd.hdr.fp_flags = 0x0; - req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->result); + memset(&req->cn10k_req.jd, 0, sizeof(struct cn10k_ml_jd)); + req->cn10k_req.jd.hdr.jce.w0.u64 = 0; + req->cn10k_req.jd.hdr.jce.w1.u64 = PLT_U64_CAST(&req->cn10k_req.status); + req->cn10k_req.jd.hdr.model_id = model->model_id; + req->cn10k_req.jd.hdr.job_type = job_type; + req->cn10k_req.jd.hdr.fp_flags = 0x0; + req->cn10k_req.jd.hdr.result = + roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->cn10k_req.result); if (job_type == ML_CN10K_JOB_TYPE_MODEL_START) { if (!model->glow.metadata.model.ocm_relocatable) - req->jd.hdr.sp_flags = ML_CN10K_SP_FLAGS_OCM_NONRELOCATABLE; + req->cn10k_req.jd.hdr.sp_flags = ML_CN10K_SP_FLAGS_OCM_NONRELOCATABLE; else - req->jd.hdr.sp_flags = 0x0; + req->cn10k_req.jd.hdr.sp_flags = 0x0; - req->jd.hdr.sp_flags |= ML_CN10K_SP_FLAGS_EXTENDED_LOAD_JD; - req->jd.model_start.extended_args = - PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->extended_args)); - req->jd.model_start.model_dst_ddr_addr = + req->cn10k_req.jd.hdr.sp_flags |= ML_CN10K_SP_FLAGS_EXTENDED_LOAD_JD; + req->cn10k_req.jd.model_start.extended_args = PLT_U64_CAST( + roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->cn10k_req.extended_args)); + req->cn10k_req.jd.model_start.model_dst_ddr_addr = PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, addr->init_run_addr)); - req->jd.model_start.model_init_offset = 0x0; - req->jd.model_start.model_main_offset = metadata->init_model.file_size; - req->jd.model_start.model_finish_offset = + req->cn10k_req.jd.model_start.model_init_offset = 0x0; + req->cn10k_req.jd.model_start.model_main_offset = metadata->init_model.file_size; + req->cn10k_req.jd.model_start.model_finish_offset = metadata->init_model.file_size + metadata->main_model.file_size; - req->jd.model_start.model_init_size = metadata->init_model.file_size; - req->jd.model_start.model_main_size = metadata->main_model.file_size; - req->jd.model_start.model_finish_size = metadata->finish_model.file_size; - req->jd.model_start.model_wb_offset = metadata->init_model.file_size + - metadata->main_model.file_size + - metadata->finish_model.file_size; - req->jd.model_start.num_layers = metadata->model.num_layers; - req->jd.model_start.num_gather_entries = 0; - req->jd.model_start.num_scatter_entries = 0; - req->jd.model_start.tilemask = 0; /* Updated after reserving pages */ - req->jd.model_start.batch_size = model->batch_size; - req->jd.model_start.ocm_wb_base_address = 0; /* Updated after reserving pages */ - req->jd.model_start.ocm_wb_range_start = metadata->model.ocm_wb_range_start; - req->jd.model_start.ocm_wb_range_end = metadata->model.ocm_wb_range_end; - req->jd.model_start.ddr_wb_base_address = PLT_U64_CAST(roc_ml_addr_ap2mlip( - &cn10k_mldev->roc, - PLT_PTR_ADD(addr->finish_load_addr, metadata->finish_model.file_size))); - req->jd.model_start.ddr_wb_range_start = metadata->model.ddr_wb_range_start; - req->jd.model_start.ddr_wb_range_end = metadata->model.ddr_wb_range_end; - req->jd.model_start.input.s.ddr_range_start = metadata->model.ddr_input_range_start; - req->jd.model_start.input.s.ddr_range_end = metadata->model.ddr_input_range_end; - req->jd.model_start.output.s.ddr_range_start = + req->cn10k_req.jd.model_start.model_init_size = metadata->init_model.file_size; + req->cn10k_req.jd.model_start.model_main_size = metadata->main_model.file_size; + req->cn10k_req.jd.model_start.model_finish_size = metadata->finish_model.file_size; + req->cn10k_req.jd.model_start.model_wb_offset = metadata->init_model.file_size + + metadata->main_model.file_size + + metadata->finish_model.file_size; + req->cn10k_req.jd.model_start.num_layers = metadata->model.num_layers; + req->cn10k_req.jd.model_start.num_gather_entries = 0; + req->cn10k_req.jd.model_start.num_scatter_entries = 0; + req->cn10k_req.jd.model_start.tilemask = 0; /* Updated after reserving pages */ + req->cn10k_req.jd.model_start.batch_size = model->batch_size; + req->cn10k_req.jd.model_start.ocm_wb_base_address = + 0; /* Updated after reserving pages */ + req->cn10k_req.jd.model_start.ocm_wb_range_start = + metadata->model.ocm_wb_range_start; + req->cn10k_req.jd.model_start.ocm_wb_range_end = metadata->model.ocm_wb_range_end; + req->cn10k_req.jd.model_start.ddr_wb_base_address = + PLT_U64_CAST(roc_ml_addr_ap2mlip( + &cn10k_mldev->roc, PLT_PTR_ADD(addr->finish_load_addr, + metadata->finish_model.file_size))); + req->cn10k_req.jd.model_start.ddr_wb_range_start = + metadata->model.ddr_wb_range_start; + req->cn10k_req.jd.model_start.ddr_wb_range_end = metadata->model.ddr_wb_range_end; + req->cn10k_req.jd.model_start.input.s.ddr_range_start = + metadata->model.ddr_input_range_start; + req->cn10k_req.jd.model_start.input.s.ddr_range_end = + metadata->model.ddr_input_range_end; + req->cn10k_req.jd.model_start.output.s.ddr_range_start = metadata->model.ddr_output_range_start; - req->jd.model_start.output.s.ddr_range_end = metadata->model.ddr_output_range_end; + req->cn10k_req.jd.model_start.output.s.ddr_range_end = + metadata->model.ddr_output_range_end; - req->extended_args.start.ddr_scratch_base_address = PLT_U64_CAST( + req->cn10k_req.extended_args.start.ddr_scratch_base_address = PLT_U64_CAST( roc_ml_addr_ap2mlip(&cn10k_mldev->roc, addr->scratch_base_addr)); - req->extended_args.start.ddr_scratch_range_start = + req->cn10k_req.extended_args.start.ddr_scratch_range_start = metadata->model.ddr_scratch_range_start; - req->extended_args.start.ddr_scratch_range_end = + req->cn10k_req.extended_args.start.ddr_scratch_range_end = metadata->model.ddr_scratch_range_end; } } static __rte_always_inline void -cn10k_ml_prep_fp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cn10k_ml_req *req, +cn10k_ml_prep_fp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml_req *req, struct rte_ml_op *op) { - req->jd.hdr.jce.w0.u64 = 0; - req->jd.hdr.jce.w1.u64 = req->compl_W1; - req->jd.hdr.model_id = op->model_id; - req->jd.hdr.job_type = ML_CN10K_JOB_TYPE_MODEL_RUN; - req->jd.hdr.fp_flags = ML_FLAGS_POLL_COMPL; - req->jd.hdr.sp_flags = 0x0; - req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->result); - req->jd.model_run.input_ddr_addr = + req->cn10k_req.jd.hdr.jce.w0.u64 = 0; + req->cn10k_req.jd.hdr.jce.w1.u64 = PLT_U64_CAST(req->status); + req->cn10k_req.jd.hdr.model_id = op->model_id; + req->cn10k_req.jd.hdr.job_type = ML_CN10K_JOB_TYPE_MODEL_RUN; + req->cn10k_req.jd.hdr.fp_flags = ML_FLAGS_POLL_COMPL; + req->cn10k_req.jd.hdr.sp_flags = 0x0; + req->cn10k_req.jd.hdr.result = + roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->cn10k_req.result); + req->cn10k_req.jd.model_run.input_ddr_addr = PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, op->input[0]->addr)); - req->jd.model_run.output_ddr_addr = + req->cn10k_req.jd.model_run.output_ddr_addr = PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, op->output[0]->addr)); - req->jd.model_run.num_batches = op->nb_batches; + req->cn10k_req.jd.model_run.num_batches = op->nb_batches; } struct xstat_info { @@ -863,7 +874,7 @@ cn10k_ml_cache_model_data(struct rte_ml_dev *dev, uint16_t model_id) op.input = &inp; op.output = &out; - memset(model->layer[0].glow.req, 0, sizeof(struct cn10k_ml_req)); + memset(model->layer[0].glow.req, 0, sizeof(struct cnxk_ml_req)); ret = cn10k_ml_inference_sync(dev, &op); plt_memzone_free(mz); @@ -906,7 +917,7 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; struct cn10k_ml_ocm *ocm; - struct cn10k_ml_qp *qp; + struct cnxk_ml_qp *qp; uint16_t model_id; uint32_t mz_size; uint16_t tile_id; @@ -1103,7 +1114,7 @@ cn10k_ml_dev_close(struct rte_ml_dev *dev) struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; - struct cn10k_ml_qp *qp; + struct cnxk_ml_qp *qp; uint16_t model_id; uint16_t qp_id; @@ -1138,7 +1149,7 @@ cn10k_ml_dev_close(struct rte_ml_dev *dev) for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { qp = dev->data->queue_pairs[qp_id]; if (qp != NULL) { - if (cn10k_ml_qp_destroy(dev, qp) != 0) + if (cnxk_ml_qp_destroy(dev, qp) != 0) plt_err("Could not destroy queue pair %u", qp_id); dev->data->queue_pairs[qp_id] = NULL; } @@ -1215,7 +1226,7 @@ cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, const struct rte_ml_dev_qp_conf *qp_conf, int socket_id) { struct rte_ml_dev_info dev_info; - struct cn10k_ml_qp *qp; + struct cnxk_ml_qp *qp; uint32_t nb_desc; if (queue_pair_id >= dev->data->nb_queue_pairs) { @@ -1241,7 +1252,7 @@ cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, */ nb_desc = (qp_conf->nb_desc == dev_info.max_desc) ? dev_info.max_desc : qp_conf->nb_desc + 1; - qp = cn10k_ml_qp_create(dev, queue_pair_id, nb_desc, socket_id); + qp = cnxk_ml_qp_create(dev, queue_pair_id, nb_desc, socket_id); if (qp == NULL) { plt_err("Could not create queue pair %u", queue_pair_id); return -ENOMEM; @@ -1254,7 +1265,7 @@ cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, static int cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats) { - struct cn10k_ml_qp *qp; + struct cnxk_ml_qp *qp; int qp_id; for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { @@ -1271,7 +1282,7 @@ cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats) static void cn10k_ml_dev_stats_reset(struct rte_ml_dev *dev) { - struct cn10k_ml_qp *qp; + struct cnxk_ml_qp *qp; int qp_id; for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { @@ -1487,20 +1498,22 @@ cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) /* Dump debug buffer */ for (core_id = 0; core_id <= 1; core_id++) { - bufsize = fw->req->jd.fw_load.debug.debug_buffer_size; + bufsize = fw->req->cn10k_req.jd.fw_load.debug.debug_buffer_size; if (core_id == 0) { head_loc = roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_DBG_BUFFER_HEAD_C0); tail_loc = roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_DBG_BUFFER_TAIL_C0); - head_ptr = PLT_PTR_CAST(fw->req->jd.fw_load.debug.core0_debug_ptr); + head_ptr = + PLT_PTR_CAST(fw->req->cn10k_req.jd.fw_load.debug.core0_debug_ptr); head_ptr = roc_ml_addr_mlip2ap(&cn10k_mldev->roc, head_ptr); } else { head_loc = roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_DBG_BUFFER_HEAD_C1); tail_loc = roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_DBG_BUFFER_TAIL_C1); - head_ptr = PLT_PTR_CAST(fw->req->jd.fw_load.debug.core1_debug_ptr); + head_ptr = + PLT_PTR_CAST(fw->req->cn10k_req.jd.fw_load.debug.core1_debug_ptr); head_ptr = roc_ml_addr_mlip2ap(&cn10k_mldev->roc, head_ptr); } if (head_loc < tail_loc) { @@ -1513,17 +1526,19 @@ cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) /* Dump exception info */ for (core_id = 0; core_id <= 1; core_id++) { - bufsize = fw->req->jd.fw_load.debug.exception_state_size; + bufsize = fw->req->cn10k_req.jd.fw_load.debug.exception_state_size; if ((core_id == 0) && (roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0) != 0)) { - head_ptr = PLT_PTR_CAST(fw->req->jd.fw_load.debug.core0_exception_buffer); + head_ptr = PLT_PTR_CAST( + fw->req->cn10k_req.jd.fw_load.debug.core0_exception_buffer); fprintf(fp, "ML_SCRATCH_EXCEPTION_SP_C0 = 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0)); head_ptr = roc_ml_addr_mlip2ap(&cn10k_mldev->roc, head_ptr); fprintf(fp, "%.*s", bufsize, head_ptr); } else if ((core_id == 1) && (roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C1) != 0)) { - head_ptr = PLT_PTR_CAST(fw->req->jd.fw_load.debug.core1_exception_buffer); + head_ptr = PLT_PTR_CAST( + fw->req->cn10k_req.jd.fw_load.debug.core1_exception_buffer); fprintf(fp, "ML_SCRATCH_EXCEPTION_SP_C1 = 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C1)); head_ptr = roc_ml_addr_mlip2ap(&cn10k_mldev->roc, head_ptr); @@ -1540,14 +1555,14 @@ cn10k_ml_dev_selftest(struct rte_ml_dev *dev) struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; const struct plt_memzone *mz; - struct cn10k_ml_req *req; + struct cnxk_ml_req *req; uint64_t timeout_cycle; bool timeout; int ret; cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; - mz = plt_memzone_reserve_aligned("dev_selftest", sizeof(struct cn10k_ml_req), 0, + mz = plt_memzone_reserve_aligned("dev_selftest", sizeof(struct cnxk_ml_req), 0, ML_CN10K_ALIGN_SIZE); if (mz == NULL) { plt_err("Could not allocate reserved memzone"); @@ -1556,23 +1571,24 @@ cn10k_ml_dev_selftest(struct rte_ml_dev *dev) req = mz->addr; /* Prepare load completion structure */ - memset(&req->jd, 0, sizeof(struct cn10k_ml_jd)); - req->jd.hdr.jce.w1.u64 = PLT_U64_CAST(&req->status); - req->jd.hdr.job_type = ML_CN10K_JOB_TYPE_FIRMWARE_SELFTEST; - req->jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->result); - req->jd.fw_load.flags = cn10k_ml_fw_flags_get(&cn10k_mldev->fw); - plt_write64(ML_CNXK_POLL_JOB_START, &req->status); + memset(&req->cn10k_req.jd, 0, sizeof(struct cn10k_ml_jd)); + req->cn10k_req.jd.hdr.jce.w1.u64 = PLT_U64_CAST(&req->cn10k_req.status); + req->cn10k_req.jd.hdr.job_type = ML_CN10K_JOB_TYPE_FIRMWARE_SELFTEST; + req->cn10k_req.jd.hdr.result = + roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->cn10k_req.result); + req->cn10k_req.jd.fw_load.flags = cn10k_ml_fw_flags_get(&cn10k_mldev->fw); + plt_write64(ML_CNXK_POLL_JOB_START, &req->cn10k_req.status); plt_wmb(); /* Enqueue firmware selftest request through scratch registers */ timeout = true; timeout_cycle = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); - roc_ml_scratch_enqueue(&cn10k_mldev->roc, &req->jd); + roc_ml_scratch_enqueue(&cn10k_mldev->roc, &req->cn10k_req.jd); plt_rmb(); do { if (roc_ml_scratch_is_done_bit_set(&cn10k_mldev->roc) && - (plt_read64(&req->status) == ML_CNXK_POLL_JOB_FINISH)) { + (plt_read64(&req->cn10k_req.status) == ML_CNXK_POLL_JOB_FINISH)) { timeout = false; break; } @@ -1583,7 +1599,7 @@ cn10k_ml_dev_selftest(struct rte_ml_dev *dev) if (timeout) { ret = -ETIME; } else { - if (req->result.error_code.u64 != 0) + if (req->cn10k_req.result.error_code != 0) ret = -1; } @@ -1656,7 +1672,7 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, mz_size = PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), ML_CN10K_ALIGN_SIZE) + 2 * model_data_size + model_scratch_size + model_info_size + - PLT_ALIGN_CEIL(sizeof(struct cn10k_ml_req), ML_CN10K_ALIGN_SIZE) + + PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_req), ML_CN10K_ALIGN_SIZE) + model_stats_size; /* Allocate memzone for model object and model data */ @@ -1728,7 +1744,7 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, /* Reset burst and sync stats */ model->layer[0].glow.burst_stats = PLT_PTR_ADD(model->layer[0].glow.req, - PLT_ALIGN_CEIL(sizeof(struct cn10k_ml_req), ML_CN10K_ALIGN_SIZE)); + PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_req), ML_CN10K_ALIGN_SIZE)); for (qp_id = 0; qp_id < dev->data->nb_queue_pairs + 1; qp_id++) { model->layer[0].glow.burst_stats[qp_id].hw_latency_tot = 0; model->layer[0].glow.burst_stats[qp_id].hw_latency_min = UINT64_MAX; @@ -1792,7 +1808,7 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; struct cn10k_ml_ocm *ocm; - struct cn10k_ml_req *req; + struct cnxk_ml_req *req; bool job_enqueued; bool job_dequeued; @@ -1817,10 +1833,10 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) /* Prepare JD */ req = model->layer[0].glow.req; cn10k_ml_prep_sp_job_descriptor(cn10k_mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_START); - req->result.error_code.u64 = 0x0; - req->result.user_ptr = NULL; + req->cn10k_req.result.error_code = 0x0; + req->cn10k_req.result.user_ptr = NULL; - plt_write64(ML_CNXK_POLL_JOB_START, &req->status); + plt_write64(ML_CNXK_POLL_JOB_START, &req->cn10k_req.status); plt_wmb(); num_tiles = model->layer[0].glow.metadata.model.tile_end - @@ -1880,8 +1896,8 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) /* Update JD */ cn10k_ml_ocm_tilecount(model->layer[0].glow.ocm_map.tilemask, &tile_start, &tile_end); - req->jd.model_start.tilemask = GENMASK_ULL(tile_end, tile_start); - req->jd.model_start.ocm_wb_base_address = + req->cn10k_req.jd.model_start.tilemask = GENMASK_ULL(tile_end, tile_start); + req->cn10k_req.jd.model_start.ocm_wb_base_address = model->layer[0].glow.ocm_map.wb_page_start * ocm->page_size; job_enqueued = false; @@ -1889,19 +1905,21 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) do { if (!job_enqueued) { req->timeout = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); - job_enqueued = roc_ml_scratch_enqueue(&cn10k_mldev->roc, &req->jd); + job_enqueued = + roc_ml_scratch_enqueue(&cn10k_mldev->roc, &req->cn10k_req.jd); } if (job_enqueued && !job_dequeued) - job_dequeued = roc_ml_scratch_dequeue(&cn10k_mldev->roc, &req->jd); + job_dequeued = + roc_ml_scratch_dequeue(&cn10k_mldev->roc, &req->cn10k_req.jd); if (job_dequeued) break; } while (plt_tsc_cycles() < req->timeout); if (job_dequeued) { - if (plt_read64(&req->status) == ML_CNXK_POLL_JOB_FINISH) { - if (req->result.error_code.u64 == 0) + if (plt_read64(&req->cn10k_req.status) == ML_CNXK_POLL_JOB_FINISH) { + if (req->cn10k_req.result.error_code == 0) ret = 0; else ret = -1; @@ -1954,7 +1972,7 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; struct cn10k_ml_ocm *ocm; - struct cn10k_ml_req *req; + struct cnxk_ml_req *req; bool job_enqueued; bool job_dequeued; @@ -1974,10 +1992,10 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) /* Prepare JD */ req = model->layer[0].glow.req; cn10k_ml_prep_sp_job_descriptor(cn10k_mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_STOP); - req->result.error_code.u64 = 0x0; - req->result.user_ptr = NULL; + req->cn10k_req.result.error_code = 0x0; + req->cn10k_req.result.user_ptr = NULL; - plt_write64(ML_CNXK_POLL_JOB_START, &req->status); + plt_write64(ML_CNXK_POLL_JOB_START, &req->cn10k_req.status); plt_wmb(); locked = false; @@ -2017,19 +2035,21 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) do { if (!job_enqueued) { req->timeout = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); - job_enqueued = roc_ml_scratch_enqueue(&cn10k_mldev->roc, &req->jd); + job_enqueued = + roc_ml_scratch_enqueue(&cn10k_mldev->roc, &req->cn10k_req.jd); } if (job_enqueued && !job_dequeued) - job_dequeued = roc_ml_scratch_dequeue(&cn10k_mldev->roc, &req->jd); + job_dequeued = + roc_ml_scratch_dequeue(&cn10k_mldev->roc, &req->cn10k_req.jd); if (job_dequeued) break; } while (plt_tsc_cycles() < req->timeout); if (job_dequeued) { - if (plt_read64(&req->status) == ML_CNXK_POLL_JOB_FINISH) { - if (req->result.error_code.u64 == 0x0) + if (plt_read64(&req->cn10k_req.status) == ML_CNXK_POLL_JOB_FINISH) { + if (req->cn10k_req.result.error_code == 0x0) ret = 0; else ret = -1; @@ -2289,18 +2309,23 @@ queue_free_count(uint64_t head, uint64_t tail, uint64_t nb_desc) } static __rte_always_inline void -cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cn10k_ml_result *result, - struct rte_ml_op *op) +cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cnxk_ml_req *req) { + union cn10k_ml_error_code *error_code; struct cn10k_ml_layer_stats *stats; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; + struct cn10k_ml_result *result; struct cnxk_ml_model *model; - struct cn10k_ml_qp *qp; + struct cnxk_ml_qp *qp; + struct rte_ml_op *op; uint64_t hw_latency; uint64_t fw_latency; - if (likely(result->error_code.u64 == 0)) { + result = &req->cn10k_req.result; + op = req->op; + + if (likely(result->error_code == 0)) { model = dev->data->models[op->model_id]; if (likely(qp_id >= 0)) { qp = dev->data->queue_pairs[qp_id]; @@ -2331,7 +2356,7 @@ cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cn10k_ml_result stats->fw_latency_max = PLT_MAX(stats->fw_latency_max, fw_latency); stats->dequeued_count++; - op->impl_opaque = result->error_code.u64; + op->impl_opaque = result->error_code; op->status = RTE_ML_OP_STATUS_SUCCESS; } else { if (likely(qp_id >= 0)) { @@ -2340,7 +2365,8 @@ cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cn10k_ml_result } /* Handle driver error */ - if (result->error_code.s.etype == ML_ETYPE_DRIVER) { + error_code = (union cn10k_ml_error_code *)&result->error_code; + if (error_code->s.etype == ML_ETYPE_DRIVER) { cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; @@ -2348,15 +2374,15 @@ cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cn10k_ml_result if ((roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0) != 0) || (roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C1) != 0)) - result->error_code.s.stype = ML_DRIVER_ERR_EXCEPTION; + error_code->s.stype = ML_DRIVER_ERR_EXCEPTION; else if ((roc_ml_reg_read64(&cn10k_mldev->roc, ML_CORE_INT_LO) != 0) || (roc_ml_reg_read64(&cn10k_mldev->roc, ML_CORE_INT_HI) != 0)) - result->error_code.s.stype = ML_DRIVER_ERR_FW_ERROR; + error_code->s.stype = ML_DRIVER_ERR_FW_ERROR; else - result->error_code.s.stype = ML_DRIVER_ERR_UNKNOWN; + error_code->s.stype = ML_DRIVER_ERR_UNKNOWN; } - op->impl_opaque = result->error_code.u64; + op->impl_opaque = result->error_code; op->status = RTE_ML_OP_STATUS_ERROR; } @@ -2367,11 +2393,12 @@ __rte_hot uint16_t cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, uint16_t nb_ops) { + union cn10k_ml_error_code *error_code; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_queue *queue; - struct cn10k_ml_req *req; - struct cn10k_ml_qp *qp; + struct cnxk_ml_queue *queue; + struct cnxk_ml_req *req; + struct cnxk_ml_qp *qp; struct rte_ml_op *op; uint16_t count; @@ -2397,12 +2424,13 @@ cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op cn10k_mldev->set_poll_addr(req); cn10k_ml_prep_fp_job_descriptor(cn10k_mldev, req, op); - memset(&req->result, 0, sizeof(struct cn10k_ml_result)); - req->result.error_code.s.etype = ML_ETYPE_UNKNOWN; - req->result.user_ptr = op->user_ptr; + memset(&req->cn10k_req.result, 0, sizeof(struct cn10k_ml_result)); + error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; + error_code->s.etype = ML_ETYPE_UNKNOWN; + req->cn10k_req.result.user_ptr = op->user_ptr; cn10k_mldev->set_poll_ptr(req); - enqueued = cn10k_mldev->ml_jcmdq_enqueue(&cn10k_mldev->roc, &req->jcmd); + enqueued = cn10k_mldev->ml_jcmdq_enqueue(&cn10k_mldev->roc, &req->cn10k_req.jcmd); if (unlikely(!enqueued)) goto jcmdq_full; @@ -2426,11 +2454,12 @@ __rte_hot uint16_t cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, uint16_t nb_ops) { + union cn10k_ml_error_code *error_code; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - struct cn10k_ml_queue *queue; - struct cn10k_ml_req *req; - struct cn10k_ml_qp *qp; + struct cnxk_ml_queue *queue; + struct cnxk_ml_req *req; + struct cnxk_ml_qp *qp; uint64_t status; uint16_t count; @@ -2452,13 +2481,15 @@ cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op req = &queue->reqs[tail]; status = cn10k_mldev->get_poll_ptr(req); if (unlikely(status != ML_CNXK_POLL_JOB_FINISH)) { - if (plt_tsc_cycles() < req->timeout) + if (plt_tsc_cycles() < req->timeout) { goto empty_or_active; - else /* Timeout, set indication of driver error */ - req->result.error_code.s.etype = ML_ETYPE_DRIVER; + } else { /* Timeout, set indication of driver error */ + error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; + error_code->s.etype = ML_ETYPE_DRIVER; + } } - cn10k_ml_result_update(dev, qp_id, &req->result, req->op); + cn10k_ml_result_update(dev, qp_id, req); ops[count] = req->op; queue_index_advance(&tail, qp->nb_desc); @@ -2509,10 +2540,11 @@ cn10k_ml_op_error_get(struct rte_ml_dev *dev, struct rte_ml_op *op, struct rte_m __rte_hot int cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) { + union cn10k_ml_error_code *error_code; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; - struct cn10k_ml_req *req; + struct cnxk_ml_req *req; bool timeout; int ret = 0; @@ -2524,17 +2556,18 @@ cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) cn10k_ml_set_poll_addr(req); cn10k_ml_prep_fp_job_descriptor(cn10k_mldev, req, op); - memset(&req->result, 0, sizeof(struct cn10k_ml_result)); - req->result.error_code.s.etype = ML_ETYPE_UNKNOWN; - req->result.user_ptr = op->user_ptr; + memset(&req->cn10k_req.result, 0, sizeof(struct cn10k_ml_result)); + error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; + error_code->s.etype = ML_ETYPE_UNKNOWN; + req->cn10k_req.result.user_ptr = op->user_ptr; cn10k_mldev->set_poll_ptr(req); - req->jcmd.w1.s.jobptr = PLT_U64_CAST(&req->jd); + req->cn10k_req.jcmd.w1.s.jobptr = PLT_U64_CAST(&req->cn10k_req.jd); timeout = true; req->timeout = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); do { - if (cn10k_mldev->ml_jcmdq_enqueue(&cn10k_mldev->roc, &req->jcmd)) { + if (cn10k_mldev->ml_jcmdq_enqueue(&cn10k_mldev->roc, &req->cn10k_req.jcmd)) { req->op = op; timeout = false; break; @@ -2557,7 +2590,7 @@ cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) if (timeout) ret = -ETIME; else - cn10k_ml_result_update(dev, -1, &req->result, req->op); + cn10k_ml_result_update(dev, -1, req); error_enqueue: return ret; diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 005b093e45d..fd5992e1925 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -10,63 +10,279 @@ #include -#include "cn10k_ml_dev.h" +/* Firmware version string length */ +#define MLDEV_FIRMWARE_VERSION_LENGTH 32 -/* Request structure */ -struct cn10k_ml_req { - /* Job descriptor */ - struct cn10k_ml_jd jd; +/* Job types */ +enum cn10k_ml_job_type { + ML_CN10K_JOB_TYPE_MODEL_RUN = 0, + ML_CN10K_JOB_TYPE_MODEL_STOP, + ML_CN10K_JOB_TYPE_MODEL_START, + ML_CN10K_JOB_TYPE_FIRMWARE_LOAD, + ML_CN10K_JOB_TYPE_FIRMWARE_SELFTEST, +}; - /* Job descriptor extra arguments */ - union cn10k_ml_jd_extended_args extended_args; +/* Firmware stats */ +struct cn10k_ml_stats { + /* Firmware start cycle */ + uint64_t fw_start; - /* Job result */ - struct cn10k_ml_result result; + /* Firmware end cycle */ + uint64_t fw_end; - /* Status field for poll mode requests */ - volatile uint64_t status; + /* Hardware start cycle */ + uint64_t hw_start; - /* Job command */ - struct ml_job_cmd_s jcmd; + /* Hardware end cycle */ + uint64_t hw_end; +}; + +/* Result structure */ +struct cn10k_ml_result { + /* Job error code */ + uint64_t error_code; + + /* Stats */ + struct cn10k_ml_stats stats; + + /* User context pointer */ + void *user_ptr; +}; + +/* Firmware capability structure */ +union cn10k_ml_fw_cap { + uint64_t u64; + + struct { + /* CMPC completion support */ + uint64_t cmpc_completions : 1; + + /* Poll mode completion support */ + uint64_t poll_completions : 1; + + /* SSO completion support */ + uint64_t sso_completions : 1; + + /* Support for model side loading */ + uint64_t side_load_model : 1; - /* Job completion W1 */ - uint64_t compl_W1; + /* Batch execution */ + uint64_t batch_run : 1; - /* Timeout cycle */ - uint64_t timeout; + /* Max number of models to be loaded in parallel */ + uint64_t max_models : 8; - /* Op */ - struct rte_ml_op *op; -} __rte_aligned(ROC_ALIGN); + /* Firmware statistics */ + uint64_t fw_stats : 1; -/* Request queue */ -struct cn10k_ml_queue { - /* Array of requests */ - struct cn10k_ml_req *reqs; + /* Hardware statistics */ + uint64_t hw_stats : 1; - /* Head of the queue, used for enqueue */ - uint64_t head; + /* Max number of batches */ + uint64_t max_num_batches : 16; - /* Tail of the queue, used for dequeue */ - uint64_t tail; + uint64_t rsvd : 33; + } s; +}; + +/* Firmware debug info structure */ +struct cn10k_ml_fw_debug { + /* ACC core 0 debug buffer */ + uint64_t core0_debug_ptr; + + /* ACC core 1 debug buffer */ + uint64_t core1_debug_ptr; + + /* ACC core 0 exception state buffer */ + uint64_t core0_exception_buffer; + + /* ACC core 1 exception state buffer */ + uint64_t core1_exception_buffer; + + /* Debug buffer size per core */ + uint32_t debug_buffer_size; - /* Wait cycles before timeout */ - uint64_t wait_cycles; + /* Exception state dump size */ + uint32_t exception_state_size; }; -/* Queue-pair structure */ -struct cn10k_ml_qp { - /* ID */ - uint32_t id; +/* Job descriptor header (32 bytes) */ +struct cn10k_ml_jd_header { + /* Job completion structure */ + struct ml_jce_s jce; + + /* Model ID */ + uint64_t model_id : 8; + + /* Job type */ + uint64_t job_type : 8; + + /* Flags for fast-path jobs */ + uint64_t fp_flags : 16; + + /* Flags for slow-path jobs */ + uint64_t sp_flags : 16; + uint64_t rsvd : 16; + + /* Job result pointer */ + uint64_t *result; +}; + +/* Extra arguments for job descriptor */ +union cn10k_ml_jd_extended_args { + struct cn10k_ml_jd_extended_args_section_start { + /* DDR Scratch base address */ + uint64_t ddr_scratch_base_address; + + /* DDR Scratch range start */ + uint64_t ddr_scratch_range_start; + + /* DDR Scratch range end */ + uint64_t ddr_scratch_range_end; + + uint8_t rsvd[104]; + } start; +}; + +/* Job descriptor structure */ +struct cn10k_ml_jd { + /* Job descriptor header (32 bytes) */ + struct cn10k_ml_jd_header hdr; + + union { + struct cn10k_ml_jd_section_fw_load { + /* Firmware capability structure (8 bytes) */ + union cn10k_ml_fw_cap cap; + + /* Firmware version (32 bytes) */ + uint8_t version[MLDEV_FIRMWARE_VERSION_LENGTH]; + + /* Debug capability structure (40 bytes) */ + struct cn10k_ml_fw_debug debug; - /* Number of descriptors */ - uint64_t nb_desc; + /* Flags to control error handling */ + uint64_t flags; - /* Request queue */ - struct cn10k_ml_queue queue; + uint8_t rsvd[8]; + } fw_load; - /* Statistics per queue-pair */ - struct rte_ml_dev_stats stats; + struct cn10k_ml_jd_section_model_start { + /* Extended arguments */ + uint64_t extended_args; + + /* Destination model start address in DDR relative to ML_MLR_BASE */ + uint64_t model_dst_ddr_addr; + + /* Offset to model init section in the model */ + uint64_t model_init_offset : 32; + + /* Size of init section in the model */ + uint64_t model_init_size : 32; + + /* Offset to model main section in the model */ + uint64_t model_main_offset : 32; + + /* Size of main section in the model */ + uint64_t model_main_size : 32; + + /* Offset to model finish section in the model */ + uint64_t model_finish_offset : 32; + + /* Size of finish section in the model */ + uint64_t model_finish_size : 32; + + /* Offset to WB in model bin */ + uint64_t model_wb_offset : 32; + + /* Number of model layers */ + uint64_t num_layers : 8; + + /* Number of gather entries, 0 means linear input mode (= no gather) */ + uint64_t num_gather_entries : 8; + + /* Number of scatter entries 0 means linear input mode (= no scatter) */ + uint64_t num_scatter_entries : 8; + + /* Tile mask to load model */ + uint64_t tilemask : 8; + + /* Batch size of model */ + uint64_t batch_size : 32; + + /* OCM WB base address */ + uint64_t ocm_wb_base_address : 32; + + /* OCM WB range start */ + uint64_t ocm_wb_range_start : 32; + + /* OCM WB range End */ + uint64_t ocm_wb_range_end : 32; + + /* DDR WB address */ + uint64_t ddr_wb_base_address; + + /* DDR WB range start */ + uint64_t ddr_wb_range_start : 32; + + /* DDR WB range end */ + uint64_t ddr_wb_range_end : 32; + + union { + /* Points to gather list if num_gather_entries > 0 */ + void *gather_list; + struct { + /* Linear input mode */ + uint64_t ddr_range_start : 32; + uint64_t ddr_range_end : 32; + } s; + } input; + + union { + /* Points to scatter list if num_scatter_entries > 0 */ + void *scatter_list; + struct { + /* Linear output mode */ + uint64_t ddr_range_start : 32; + uint64_t ddr_range_end : 32; + } s; + } output; + } model_start; + + struct cn10k_ml_jd_section_model_stop { + uint8_t rsvd[96]; + } model_stop; + + struct cn10k_ml_jd_section_model_run { + /* Address of the input for the run relative to ML_MLR_BASE */ + uint64_t input_ddr_addr; + + /* Address of the output for the run relative to ML_MLR_BASE */ + uint64_t output_ddr_addr; + + /* Number of batches to run in variable batch processing */ + uint16_t num_batches; + + uint8_t rsvd[78]; + } model_run; + }; +} __plt_aligned(ROC_ALIGN); + +/* CN10K specific request */ +struct cn10k_ml_req { + /* Job descriptor */ + struct cn10k_ml_jd jd; + + /* Job descriptor extra arguments */ + union cn10k_ml_jd_extended_args extended_args; + + /* Status field for poll mode requests */ + volatile uint64_t status; + + /* Job command */ + struct ml_job_cmd_s jcmd; + + /* Result */ + struct cn10k_ml_result result; }; /* Device ops */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c new file mode 100644 index 00000000000..f1872dcf7c6 --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#include + +#include "cnxk_ml_ops.h" diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h new file mode 100644 index 00000000000..b953fb0f5fc --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_ops.h @@ -0,0 +1,63 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#ifndef _CNXK_ML_OPS_H_ +#define _CNXK_ML_OPS_H_ + +#include +#include + +#include + +#include "cn10k_ml_ops.h" + +/* Request structure */ +struct cnxk_ml_req { + /* Device specific request */ + union { + /* CN10K */ + struct cn10k_ml_req cn10k_req; + }; + + /* Address of status field */ + volatile uint64_t *status; + + /* Timeout cycle */ + uint64_t timeout; + + /* Op */ + struct rte_ml_op *op; +} __rte_aligned(ROC_ALIGN); + +/* Request queue */ +struct cnxk_ml_queue { + /* Array of requests */ + struct cnxk_ml_req *reqs; + + /* Head of the queue, used for enqueue */ + uint64_t head; + + /* Tail of the queue, used for dequeue */ + uint64_t tail; + + /* Wait cycles before timeout */ + uint64_t wait_cycles; +}; + +/* Queue-pair structure */ +struct cnxk_ml_qp { + /* ID */ + uint32_t id; + + /* Number of descriptors */ + uint64_t nb_desc; + + /* Request queue */ + struct cnxk_ml_queue queue; + + /* Statistics per queue-pair */ + struct rte_ml_dev_stats stats; +}; + +#endif /* _CNXK_ML_OPS_H_ */ diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 72e03b15b5b..73db458fcd9 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -15,6 +15,7 @@ driver_sdk_headers = files( 'cnxk_ml_dev.h', 'cnxk_ml_io.h', 'cnxk_ml_model.h', + 'cnxk_ml_ops.h', ) sources = files( @@ -24,6 +25,7 @@ sources = files( 'cn10k_ml_ocm.c', 'cnxk_ml_dev.c', 'cnxk_ml_model.c', + 'cnxk_ml_ops.c', ) deps += ['mldev', 'common_cnxk', 'kvargs', 'hash'] From patchwork Wed Aug 30 15:58:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130888 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E0CAD41FD1; Wed, 30 Aug 2023 18:00:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 18A44402BA; Wed, 30 Aug 2023 17:59:44 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 6DD1B402A0 for ; Wed, 30 Aug 2023 17:59:37 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7p1004368 for ; Wed, 30 Aug 2023 08:59:36 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=Cj55i7ffHAyRvLjfYKTwKwsrqJIBewZALDV6dkwoTJA=; b=dGX09vr6DqUelhkc+/LPBz8zvmOnUJTuvWPo5TJtwmlGjJfUNimT5zu1cbRWp2QcCb0U 2Ro7lufDl8l6fWVek/G43CnoJMObhP5FKrFUBpkFXfRXuyrDO6MdunLGjl7unfAUYXjb Pnp2rU1Y8cQI80INnJ0b3vhP4i58teBFZvImwvCM1nJRsi7hPgr0r+HW6LnDvsCBAGHQ 0G+f757hD/P/f/2JvPaZS5vv67Sfjv5aUU3QwSkamQq9EityGGyEHPkaPt+uP8fN3VaE oZxN8Pi+AtzTIZRwGtWSuPgv0VjvmOfsw8hC0ewwebZe8QExv8c08pGGBNzHu1FXXD6L yA== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysst5-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:36 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:33 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:33 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id ACDC03F707F; Wed, 30 Aug 2023 08:59:33 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 06/34] ml/cnxk: add generic cnxk xstats structures Date: Wed, 30 Aug 2023 08:58:56 -0700 Message-ID: <20230830155927.3566-7-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: OSkLwaMvcCRnoKg3koIyd2EJwP3dp_k1 X-Proofpoint-ORIG-GUID: OSkLwaMvcCRnoKg3koIyd2EJwP3dp_k1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Introduced generic xstats structures and renamed cn10k xstats enumerations with cnxk prefix. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.h | 86 +--------------- drivers/ml/cnxk/cn10k_ml_model.h | 6 +- drivers/ml/cnxk/cn10k_ml_ops.c | 169 ++++++++++++++----------------- drivers/ml/cnxk/cnxk_ml_xstats.h | 128 +++++++++++++++++++++++ drivers/ml/cnxk/meson.build | 1 + 5 files changed, 210 insertions(+), 180 deletions(-) create mode 100644 drivers/ml/cnxk/cnxk_ml_xstats.h diff --git a/drivers/ml/cnxk/cn10k_ml_dev.h b/drivers/ml/cnxk/cn10k_ml_dev.h index 1852d4f6c9a..be989e0a207 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.h +++ b/drivers/ml/cnxk/cn10k_ml_dev.h @@ -10,6 +10,7 @@ #include "cn10k_ml_ocm.h" #include "cnxk_ml_io.h" +#include "cnxk_ml_xstats.h" /* Dummy Device ops */ extern struct rte_ml_dev_ops ml_dev_dummy_ops; @@ -121,89 +122,6 @@ struct cn10k_ml_fw { struct cnxk_ml_req *req; }; -/* Extended stats types enum */ -enum cn10k_ml_xstats_type { - /* Number of models loaded */ - nb_models_loaded, - - /* Number of models unloaded */ - nb_models_unloaded, - - /* Number of models started */ - nb_models_started, - - /* Number of models stopped */ - nb_models_stopped, - - /* Average inference hardware latency */ - avg_hw_latency, - - /* Minimum hardware latency */ - min_hw_latency, - - /* Maximum hardware latency */ - max_hw_latency, - - /* Average firmware latency */ - avg_fw_latency, - - /* Minimum firmware latency */ - min_fw_latency, - - /* Maximum firmware latency */ - max_fw_latency, -}; - -/* Extended stats function type enum. */ -enum cn10k_ml_xstats_fn_type { - /* Device function */ - CN10K_ML_XSTATS_FN_DEVICE, - - /* Model function */ - CN10K_ML_XSTATS_FN_MODEL, -}; - -/* Function pointer to get xstats for a type */ -typedef uint64_t (*cn10k_ml_xstats_fn)(struct rte_ml_dev *dev, uint16_t obj_idx, - enum cn10k_ml_xstats_type stat); - -/* Extended stats entry structure */ -struct cn10k_ml_xstats_entry { - /* Name-ID map */ - struct rte_ml_dev_xstats_map map; - - /* xstats mode, device or model */ - enum rte_ml_dev_xstats_mode mode; - - /* Type of xstats */ - enum cn10k_ml_xstats_type type; - - /* xstats function */ - enum cn10k_ml_xstats_fn_type fn_id; - - /* Object ID, model ID for model stat type */ - uint16_t obj_idx; - - /* Allowed to reset the stat */ - uint8_t reset_allowed; - - /* An offset to be taken away to emulate resets */ - uint64_t reset_value; -}; - -/* Extended stats data */ -struct cn10k_ml_xstats { - /* Pointer to xstats entries */ - struct cn10k_ml_xstats_entry *entries; - - /* Store num stats and offset of the stats for each model */ - uint16_t count_per_model[ML_CNXK_MAX_MODELS]; - uint16_t offset_for_model[ML_CNXK_MAX_MODELS]; - uint16_t count_mode_device; - uint16_t count_mode_model; - uint16_t count; -}; - /* Device private data */ struct cn10k_ml_dev { /* Device ROC */ @@ -216,7 +134,7 @@ struct cn10k_ml_dev { struct cn10k_ml_ocm ocm; /* Extended stats data */ - struct cn10k_ml_xstats xstats; + struct cnxk_ml_xstats xstats; /* Enable / disable model data caching */ int cache_model_data; diff --git a/drivers/ml/cnxk/cn10k_ml_model.h b/drivers/ml/cnxk/cn10k_ml_model.h index 74ada1531a8..5c32f48c68f 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.h +++ b/drivers/ml/cnxk/cn10k_ml_model.h @@ -404,7 +404,7 @@ struct cn10k_ml_layer_addr { }; /* Model fast-path stats */ -struct cn10k_ml_layer_stats { +struct cn10k_ml_layer_xstats { /* Total hardware latency, sum of all inferences */ uint64_t hw_latency_tot; @@ -447,10 +447,10 @@ struct cn10k_ml_layer_data { struct cnxk_ml_req *req; /* Layer: Stats for burst ops */ - struct cn10k_ml_layer_stats *burst_stats; + struct cn10k_ml_layer_xstats *burst_xstats; /* Layer: Stats for sync ops */ - struct cn10k_ml_layer_stats *sync_stats; + struct cn10k_ml_layer_xstats *sync_xstats; }; struct cn10k_ml_model_data { diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 2b1fa08154d..03a7447dc87 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -14,6 +14,7 @@ #include "cnxk_ml_dev.h" #include "cnxk_ml_model.h" #include "cnxk_ml_ops.h" +#include "cnxk_ml_xstats.h" /* ML model macros */ #define CN10K_ML_MODEL_MEMZONE_NAME "ml_cn10k_model_mz" @@ -429,26 +430,6 @@ cn10k_ml_prep_fp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml req->cn10k_req.jd.model_run.num_batches = op->nb_batches; } -struct xstat_info { - char name[32]; - enum cn10k_ml_xstats_type type; - uint8_t reset_allowed; -}; - -/* Note: Device stats are not allowed to be reset. */ -static const struct xstat_info device_stats[] = { - {"nb_models_loaded", nb_models_loaded, 0}, - {"nb_models_unloaded", nb_models_unloaded, 0}, - {"nb_models_started", nb_models_started, 0}, - {"nb_models_stopped", nb_models_stopped, 0}, -}; - -static const struct xstat_info model_stats[] = { - {"Avg-HW-Latency", avg_hw_latency, 1}, {"Min-HW-Latency", min_hw_latency, 1}, - {"Max-HW-Latency", max_hw_latency, 1}, {"Avg-FW-Latency", avg_fw_latency, 1}, - {"Min-FW-Latency", min_fw_latency, 1}, {"Max-FW-Latency", max_fw_latency, 1}, -}; - static int cn10k_ml_xstats_init(struct rte_ml_dev *dev) { @@ -463,10 +444,10 @@ cn10k_ml_xstats_init(struct rte_ml_dev *dev) cn10k_mldev = &cnxk_mldev->cn10k_mldev; /* Allocate memory for xstats entries. Don't allocate during reconfigure */ - nb_stats = RTE_DIM(device_stats) + ML_CNXK_MAX_MODELS * RTE_DIM(model_stats); + nb_stats = RTE_DIM(device_xstats) + ML_CNXK_MAX_MODELS * RTE_DIM(layer_xstats); if (cn10k_mldev->xstats.entries == NULL) cn10k_mldev->xstats.entries = rte_zmalloc( - "cn10k_ml_xstats", sizeof(struct cn10k_ml_xstats_entry) * nb_stats, + "cn10k_ml_xstats", sizeof(struct cnxk_ml_xstats_entry) * nb_stats, PLT_CACHE_LINE_SIZE); if (cn10k_mldev->xstats.entries == NULL) @@ -474,17 +455,17 @@ cn10k_ml_xstats_init(struct rte_ml_dev *dev) /* Initialize device xstats */ stat_id = 0; - for (i = 0; i < RTE_DIM(device_stats); i++) { + for (i = 0; i < RTE_DIM(device_xstats); i++) { cn10k_mldev->xstats.entries[stat_id].map.id = stat_id; snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), "%s", - device_stats[i].name); + device_xstats[i].name); cn10k_mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_DEVICE; - cn10k_mldev->xstats.entries[stat_id].type = device_stats[i].type; - cn10k_mldev->xstats.entries[stat_id].fn_id = CN10K_ML_XSTATS_FN_DEVICE; + cn10k_mldev->xstats.entries[stat_id].type = device_xstats[i].type; + cn10k_mldev->xstats.entries[stat_id].fn_id = CNXK_ML_XSTATS_FN_DEVICE; cn10k_mldev->xstats.entries[stat_id].obj_idx = 0; - cn10k_mldev->xstats.entries[stat_id].reset_allowed = device_stats[i].reset_allowed; + cn10k_mldev->xstats.entries[stat_id].reset_allowed = device_xstats[i].reset_allowed; stat_id++; } cn10k_mldev->xstats.count_mode_device = stat_id; @@ -493,24 +474,24 @@ cn10k_ml_xstats_init(struct rte_ml_dev *dev) for (model = 0; model < ML_CNXK_MAX_MODELS; model++) { cn10k_mldev->xstats.offset_for_model[model] = stat_id; - for (i = 0; i < RTE_DIM(model_stats); i++) { + for (i = 0; i < RTE_DIM(layer_xstats); i++) { cn10k_mldev->xstats.entries[stat_id].map.id = stat_id; cn10k_mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_MODEL; - cn10k_mldev->xstats.entries[stat_id].type = model_stats[i].type; - cn10k_mldev->xstats.entries[stat_id].fn_id = CN10K_ML_XSTATS_FN_MODEL; + cn10k_mldev->xstats.entries[stat_id].type = layer_xstats[i].type; + cn10k_mldev->xstats.entries[stat_id].fn_id = CNXK_ML_XSTATS_FN_MODEL; cn10k_mldev->xstats.entries[stat_id].obj_idx = model; cn10k_mldev->xstats.entries[stat_id].reset_allowed = - model_stats[i].reset_allowed; + layer_xstats[i].reset_allowed; /* Name of xstat is updated during model load */ snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), - "Model-%u-%s", model, model_stats[i].name); + "Model-%u-%s", model, layer_xstats[i].name); stat_id++; } - cn10k_mldev->xstats.count_per_model[model] = RTE_DIM(model_stats); + cn10k_mldev->xstats.count_per_model[model] = RTE_DIM(layer_xstats); } cn10k_mldev->xstats.count_mode_model = stat_id - cn10k_mldev->xstats.count_mode_device; @@ -549,7 +530,7 @@ cn10k_ml_xstats_model_name_update(struct rte_ml_dev *dev, uint16_t model_id) cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; model = dev->data->models[model_id]; - stat_id = RTE_DIM(device_stats) + model_id * RTE_DIM(model_stats); + stat_id = RTE_DIM(device_xstats) + model_id * RTE_DIM(layer_xstats); roc_clk_freq_get(&rclk_freq, &sclk_freq); if (sclk_freq == 0) @@ -558,17 +539,17 @@ cn10k_ml_xstats_model_name_update(struct rte_ml_dev *dev, uint16_t model_id) strcpy(suffix, "ns"); /* Update xstat name based on model name and sclk availability */ - for (i = 0; i < RTE_DIM(model_stats); i++) { + for (i = 0; i < RTE_DIM(layer_xstats); i++) { snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", - model->layer[0].glow.metadata.model.name, model_stats[i].name, suffix); + model->layer[0].glow.metadata.model.name, layer_xstats[i].name, suffix); stat_id++; } } static uint64_t cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, - enum cn10k_ml_xstats_type type) + enum cnxk_ml_xstats_type type) { struct cnxk_ml_dev *cnxk_mldev; @@ -594,9 +575,9 @@ cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, do { \ value = 0; \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - value += model->layer[0].glow.burst_stats[qp_id].str##_latency_tot; \ - count += model->layer[0].glow.burst_stats[qp_id].dequeued_count - \ - model->layer[0].glow.burst_stats[qp_id].str##_reset_count; \ + value += model->layer[0].glow.burst_xstats[qp_id].str##_latency_tot; \ + count += model->layer[0].glow.burst_xstats[qp_id].dequeued_count - \ + model->layer[0].glow.burst_xstats[qp_id].str##_reset_count; \ } \ if (count != 0) \ value = value / count; \ @@ -607,9 +588,10 @@ cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, value = UINT64_MAX; \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ value = PLT_MIN( \ - value, model->layer[0].glow.burst_stats[qp_id].str##_latency_min); \ - count += model->layer[0].glow.burst_stats[qp_id].dequeued_count - \ - model->layer[0].glow.burst_stats[qp_id].str##_reset_count; \ + value, \ + model->layer[0].glow.burst_xstats[qp_id].str##_latency_min); \ + count += model->layer[0].glow.burst_xstats[qp_id].dequeued_count - \ + model->layer[0].glow.burst_xstats[qp_id].str##_reset_count; \ } \ if (count == 0) \ value = 0; \ @@ -620,16 +602,17 @@ cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, value = 0; \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ value = PLT_MAX( \ - value, model->layer[0].glow.burst_stats[qp_id].str##_latency_max); \ - count += model->layer[0].glow.burst_stats[qp_id].dequeued_count - \ - model->layer[0].glow.burst_stats[qp_id].str##_reset_count; \ + value, \ + model->layer[0].glow.burst_xstats[qp_id].str##_latency_max); \ + count += model->layer[0].glow.burst_xstats[qp_id].dequeued_count - \ + model->layer[0].glow.burst_xstats[qp_id].str##_reset_count; \ } \ if (count == 0) \ value = 0; \ } while (0) static uint64_t -cn10k_ml_model_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx, enum cn10k_ml_xstats_type type) +cn10k_ml_model_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx, enum cnxk_ml_xstats_type type) { struct cnxk_ml_model *model; uint16_t rclk_freq; /* MHz */ @@ -675,8 +658,8 @@ cn10k_ml_model_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx, enum cn10k_ml static int cn10k_ml_device_xstats_reset(struct rte_ml_dev *dev, const uint16_t stat_ids[], uint16_t nb_ids) { - struct cn10k_ml_xstats_entry *xs; struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_xstats_entry *xs; struct cnxk_ml_dev *cnxk_mldev; uint16_t nb_stats; uint16_t stat_id; @@ -712,26 +695,26 @@ cn10k_ml_device_xstats_reset(struct rte_ml_dev *dev, const uint16_t stat_ids[], #define ML_AVG_RESET_FOREACH_QP(dev, model, qp_id, str) \ do { \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - model->layer[0].glow.burst_stats[qp_id].str##_latency_tot = 0; \ - model->layer[0].glow.burst_stats[qp_id].str##_reset_count = \ - model->layer[0].glow.burst_stats[qp_id].dequeued_count; \ + model->layer[0].glow.burst_xstats[qp_id].str##_latency_tot = 0; \ + model->layer[0].glow.burst_xstats[qp_id].str##_reset_count = \ + model->layer[0].glow.burst_xstats[qp_id].dequeued_count; \ } \ } while (0) #define ML_MIN_RESET_FOREACH_QP(dev, model, qp_id, str) \ do { \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) \ - model->layer[0].glow.burst_stats[qp_id].str##_latency_min = UINT64_MAX; \ + model->layer[0].glow.burst_xstats[qp_id].str##_latency_min = UINT64_MAX; \ } while (0) #define ML_MAX_RESET_FOREACH_QP(dev, model, qp_id, str) \ do { \ for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) \ - model->layer[0].glow.burst_stats[qp_id].str##_latency_max = 0; \ + model->layer[0].glow.burst_xstats[qp_id].str##_latency_max = 0; \ } while (0) static void -cn10k_ml_reset_model_stat(struct rte_ml_dev *dev, uint16_t model_id, enum cn10k_ml_xstats_type type) +cn10k_ml_reset_model_stat(struct rte_ml_dev *dev, uint16_t model_id, enum cnxk_ml_xstats_type type) { struct cnxk_ml_model *model; uint32_t qp_id; @@ -766,8 +749,8 @@ static int cn10k_ml_model_xstats_reset(struct rte_ml_dev *dev, int32_t model_id, const uint16_t stat_ids[], uint16_t nb_ids) { - struct cn10k_ml_xstats_entry *xs; struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_xstats_entry *xs; struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; int32_t lcl_model_id = 0; @@ -1346,10 +1329,10 @@ static int cn10k_ml_dev_xstats_by_name_get(struct rte_ml_dev *dev, const char *name, uint16_t *stat_id, uint64_t *value) { - struct cn10k_ml_xstats_entry *xs; + struct cnxk_ml_xstats_entry *xs; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; - cn10k_ml_xstats_fn fn; + cnxk_ml_xstats_fn fn; uint32_t i; cnxk_mldev = dev->data->dev_private; @@ -1361,10 +1344,10 @@ cn10k_ml_dev_xstats_by_name_get(struct rte_ml_dev *dev, const char *name, uint16 *stat_id = xs->map.id; switch (xs->fn_id) { - case CN10K_ML_XSTATS_FN_DEVICE: + case CNXK_ML_XSTATS_FN_DEVICE: fn = cn10k_ml_dev_xstat_get; break; - case CN10K_ML_XSTATS_FN_MODEL: + case CNXK_ML_XSTATS_FN_MODEL: fn = cn10k_ml_model_xstat_get; break; default: @@ -1388,11 +1371,11 @@ static int cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, int32_t model_id, const uint16_t stat_ids[], uint64_t values[], uint16_t nb_ids) { - struct cn10k_ml_xstats_entry *xs; struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_xstats_entry *xs; struct cnxk_ml_dev *cnxk_mldev; uint32_t xstats_mode_count; - cn10k_ml_xstats_fn fn; + cnxk_ml_xstats_fn fn; uint64_t val; uint32_t idx; uint32_t i; @@ -1427,10 +1410,10 @@ cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode } switch (xs->fn_id) { - case CN10K_ML_XSTATS_FN_DEVICE: + case CNXK_ML_XSTATS_FN_DEVICE: fn = cn10k_ml_dev_xstat_get; break; - case CN10K_ML_XSTATS_FN_MODEL: + case CNXK_ML_XSTATS_FN_MODEL: fn = cn10k_ml_model_xstat_get; break; default: @@ -1668,7 +1651,7 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, metadata->model.num_input * sizeof(struct rte_ml_io_info) + metadata->model.num_output * sizeof(struct rte_ml_io_info); model_info_size = PLT_ALIGN_CEIL(model_info_size, ML_CN10K_ALIGN_SIZE); - model_stats_size = (dev->data->nb_queue_pairs + 1) * sizeof(struct cn10k_ml_layer_stats); + model_stats_size = (dev->data->nb_queue_pairs + 1) * sizeof(struct cn10k_ml_layer_xstats); mz_size = PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), ML_CN10K_ALIGN_SIZE) + 2 * model_data_size + model_scratch_size + model_info_size + @@ -1742,24 +1725,24 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, model->layer[0].glow.req = PLT_PTR_ADD(model->info, model_info_size); /* Reset burst and sync stats */ - model->layer[0].glow.burst_stats = + model->layer[0].glow.burst_xstats = PLT_PTR_ADD(model->layer[0].glow.req, PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_req), ML_CN10K_ALIGN_SIZE)); for (qp_id = 0; qp_id < dev->data->nb_queue_pairs + 1; qp_id++) { - model->layer[0].glow.burst_stats[qp_id].hw_latency_tot = 0; - model->layer[0].glow.burst_stats[qp_id].hw_latency_min = UINT64_MAX; - model->layer[0].glow.burst_stats[qp_id].hw_latency_max = 0; - model->layer[0].glow.burst_stats[qp_id].fw_latency_tot = 0; - model->layer[0].glow.burst_stats[qp_id].fw_latency_min = UINT64_MAX; - model->layer[0].glow.burst_stats[qp_id].fw_latency_max = 0; - model->layer[0].glow.burst_stats[qp_id].hw_reset_count = 0; - model->layer[0].glow.burst_stats[qp_id].fw_reset_count = 0; - model->layer[0].glow.burst_stats[qp_id].dequeued_count = 0; + model->layer[0].glow.burst_xstats[qp_id].hw_latency_tot = 0; + model->layer[0].glow.burst_xstats[qp_id].hw_latency_min = UINT64_MAX; + model->layer[0].glow.burst_xstats[qp_id].hw_latency_max = 0; + model->layer[0].glow.burst_xstats[qp_id].fw_latency_tot = 0; + model->layer[0].glow.burst_xstats[qp_id].fw_latency_min = UINT64_MAX; + model->layer[0].glow.burst_xstats[qp_id].fw_latency_max = 0; + model->layer[0].glow.burst_xstats[qp_id].hw_reset_count = 0; + model->layer[0].glow.burst_xstats[qp_id].fw_reset_count = 0; + model->layer[0].glow.burst_xstats[qp_id].dequeued_count = 0; } - model->layer[0].glow.sync_stats = - PLT_PTR_ADD(model->layer[0].glow.burst_stats, - dev->data->nb_queue_pairs * sizeof(struct cn10k_ml_layer_stats)); + model->layer[0].glow.sync_xstats = + PLT_PTR_ADD(model->layer[0].glow.burst_xstats, + dev->data->nb_queue_pairs * sizeof(struct cn10k_ml_layer_xstats)); plt_spinlock_init(&model->lock); model->state = ML_CNXK_MODEL_STATE_LOADED; @@ -2312,7 +2295,7 @@ static __rte_always_inline void cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cnxk_ml_req *req) { union cn10k_ml_error_code *error_code; - struct cn10k_ml_layer_stats *stats; + struct cn10k_ml_layer_xstats *xstats; struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_result *result; @@ -2330,31 +2313,31 @@ cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cnxk_ml_req *re if (likely(qp_id >= 0)) { qp = dev->data->queue_pairs[qp_id]; qp->stats.dequeued_count++; - stats = &model->layer[0].glow.burst_stats[qp_id]; + xstats = &model->layer[0].glow.burst_xstats[qp_id]; } else { - stats = model->layer[0].glow.sync_stats; + xstats = model->layer[0].glow.sync_xstats; } - if (unlikely(stats->dequeued_count == stats->hw_reset_count)) { - stats->hw_latency_min = UINT64_MAX; - stats->hw_latency_max = 0; + if (unlikely(xstats->dequeued_count == xstats->hw_reset_count)) { + xstats->hw_latency_min = UINT64_MAX; + xstats->hw_latency_max = 0; } - if (unlikely(stats->dequeued_count == stats->fw_reset_count)) { - stats->fw_latency_min = UINT64_MAX; - stats->fw_latency_max = 0; + if (unlikely(xstats->dequeued_count == xstats->fw_reset_count)) { + xstats->fw_latency_min = UINT64_MAX; + xstats->fw_latency_max = 0; } hw_latency = result->stats.hw_end - result->stats.hw_start; fw_latency = result->stats.fw_end - result->stats.fw_start - hw_latency; - stats->hw_latency_tot += hw_latency; - stats->hw_latency_min = PLT_MIN(stats->hw_latency_min, hw_latency); - stats->hw_latency_max = PLT_MAX(stats->hw_latency_max, hw_latency); - stats->fw_latency_tot += fw_latency; - stats->fw_latency_min = PLT_MIN(stats->fw_latency_min, fw_latency); - stats->fw_latency_max = PLT_MAX(stats->fw_latency_max, fw_latency); - stats->dequeued_count++; + xstats->hw_latency_tot += hw_latency; + xstats->hw_latency_min = PLT_MIN(xstats->hw_latency_min, hw_latency); + xstats->hw_latency_max = PLT_MAX(xstats->hw_latency_max, hw_latency); + xstats->fw_latency_tot += fw_latency; + xstats->fw_latency_min = PLT_MIN(xstats->fw_latency_min, fw_latency); + xstats->fw_latency_max = PLT_MAX(xstats->fw_latency_max, fw_latency); + xstats->dequeued_count++; op->impl_opaque = result->error_code; op->status = RTE_ML_OP_STATUS_SUCCESS; diff --git a/drivers/ml/cnxk/cnxk_ml_xstats.h b/drivers/ml/cnxk/cnxk_ml_xstats.h new file mode 100644 index 00000000000..0d405679caa --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_xstats.h @@ -0,0 +1,128 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#ifndef _CNXK_ML_XSTATS_H_ +#define _CNXK_ML_XSTATS_H_ + +#include "cnxk_ml_io.h" + +/* Extended stats types enum */ +enum cnxk_ml_xstats_type { + /* Number of models loaded */ + nb_models_loaded, + + /* Number of models unloaded */ + nb_models_unloaded, + + /* Number of models started */ + nb_models_started, + + /* Number of models stopped */ + nb_models_stopped, + + /* Average inference hardware latency */ + avg_hw_latency, + + /* Minimum hardware latency */ + min_hw_latency, + + /* Maximum hardware latency */ + max_hw_latency, + + /* Average firmware latency */ + avg_fw_latency, + + /* Minimum firmware latency */ + min_fw_latency, + + /* Maximum firmware latency */ + max_fw_latency, + + /* Average runtime latency */ + avg_rt_latency, + + /* Minimum runtime latency */ + min_rt_latency, + + /* Maximum runtime latency */ + max_rt_latency, +}; + +/* Extended stats function type enum. */ +enum cnxk_ml_xstats_fn_type { + /* Device function */ + CNXK_ML_XSTATS_FN_DEVICE, + + /* Model function */ + CNXK_ML_XSTATS_FN_MODEL, +}; + +/* Function pointer to get xstats for a type */ +typedef uint64_t (*cnxk_ml_xstats_fn)(struct rte_ml_dev *cnxk_mldev, uint16_t obj_idx, + enum cnxk_ml_xstats_type stat); + +/* Extended stats entry structure */ +struct cnxk_ml_xstats_entry { + /* Name-ID map */ + struct rte_ml_dev_xstats_map map; + + /* xstats mode, device or model */ + enum rte_ml_dev_xstats_mode mode; + + /* Type of xstats */ + enum cnxk_ml_xstats_type type; + + /* xstats function */ + enum cnxk_ml_xstats_fn_type fn_id; + + /* Object ID, model ID for model stat type */ + uint16_t obj_idx; + + /* Layer ID, valid for model stat type */ + int32_t layer_id; + + /* Allowed to reset the stat */ + uint8_t reset_allowed; + + /* An offset to be taken away to emulate resets */ + uint64_t reset_value; +}; + +/* Extended stats data */ +struct cnxk_ml_xstats { + /* Pointer to xstats entries */ + struct cnxk_ml_xstats_entry *entries; + + /* Store num stats and offset of the stats for each model */ + uint16_t count_per_model[ML_CNXK_MAX_MODELS]; + uint16_t offset_for_model[ML_CNXK_MAX_MODELS]; + uint16_t count_per_layer[ML_CNXK_MAX_MODELS][ML_CNXK_MODEL_MAX_LAYERS]; + uint16_t offset_for_layer[ML_CNXK_MAX_MODELS][ML_CNXK_MODEL_MAX_LAYERS]; + uint16_t count_mode_device; + uint16_t count_mode_model; + uint16_t count; +}; + +struct cnxk_ml_xstat_info { + char name[32]; + enum cnxk_ml_xstats_type type; + uint8_t reset_allowed; +}; + +/* Device xstats. Note: Device stats are not allowed to be reset. */ +static const struct cnxk_ml_xstat_info device_xstats[] = { + {"nb_models_loaded", nb_models_loaded, 0}, + {"nb_models_unloaded", nb_models_unloaded, 0}, + {"nb_models_started", nb_models_started, 0}, + {"nb_models_stopped", nb_models_stopped, 0}, +}; + +/* Layer xstats */ +static const struct cnxk_ml_xstat_info layer_xstats[] = { + {"Avg-HW-Latency", avg_hw_latency, 1}, {"Min-HW-Latency", min_hw_latency, 1}, + {"Max-HW-Latency", max_hw_latency, 1}, {"Avg-FW-Latency", avg_fw_latency, 1}, + {"Min-FW-Latency", min_fw_latency, 1}, {"Max-FW-Latency", max_fw_latency, 1}, +}; + +#endif /* _CNXK_ML_XSTATS_H_ */ diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 73db458fcd9..6385ac45481 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -16,6 +16,7 @@ driver_sdk_headers = files( 'cnxk_ml_io.h', 'cnxk_ml_model.h', 'cnxk_ml_ops.h', + 'cnxk_ml_xstats.h', ) sources = files( From patchwork Wed Aug 30 15:58:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130890 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A25A441FD1; Wed, 30 Aug 2023 18:01:06 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 52DF4402C2; Wed, 30 Aug 2023 17:59:46 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id DA144402A6 for ; Wed, 30 Aug 2023 17:59:37 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7p2004368 for ; Wed, 30 Aug 2023 08:59:37 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=EfFfz+8C1RfBgkCXKzmwBLshDxSB9EkR1MW4FrEPmRs=; b=CenRySdK7jn15ja0xA7PdXIsATcTEShmm4iF9HMs7KRfpWiusRQx91H72gLdhwzuuHhj PRrTlrzzF4S4wUsdT4v/+F5B/gP8xd0G9qL9/iYQZJEX5g/Kzdkz+9t8aGem1lhPSq/d QKXbeE3gEDMkRbeSp1zyL7asa0slI/YhNKxfEC2JUMSZLYKyjpJj+MViDKxw0uqHFEXw lxGFHUNK+T3kY/SiNctua3jfiavqasuB3VHiWQrT1qMxodgJcoPOQ8RyH7zG2A2/PU6d RhzQSrdCfp2TcJFB2/5NUUyGtWf1pFJ6VK5ahynGM0BjUsm7hsAd7QN51lJk/pM1RO+n Jg== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysst5-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:36 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:34 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:34 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 0C8A33F707D; Wed, 30 Aug 2023 08:59:34 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 07/34] ml/cnxk: rename cnxk ops function pointers struct Date: Wed, 30 Aug 2023 08:58:57 -0700 Message-ID: <20230830155927.3566-8-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: xVp-CwbVP8V-WoodFDti2HKQAg4DQbvj X-Proofpoint-ORIG-GUID: xVp-CwbVP8V-WoodFDti2HKQAg4DQbvj X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Renamed cn10k ML ops structure with cnxk prefix. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.c | 2 +- drivers/ml/cnxk/cn10k_ml_ops.c | 73 +++++++++------------------------- drivers/ml/cnxk/cn10k_ml_ops.h | 34 +++++++++++++++- drivers/ml/cnxk/cnxk_ml_ops.c | 38 ++++++++++++++++++ drivers/ml/cnxk/cnxk_ml_ops.h | 2 + 5 files changed, 93 insertions(+), 56 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_dev.c b/drivers/ml/cnxk/cn10k_ml_dev.c index f6e05cfc472..20c114b8bf7 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.c +++ b/drivers/ml/cnxk/cn10k_ml_dev.c @@ -404,7 +404,7 @@ cn10k_ml_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_de goto pmd_destroy; } - dev->dev_ops = &cn10k_ml_ops; + dev->dev_ops = &cnxk_ml_ops; } else { plt_err("CN10K ML Ops are not supported on secondary process"); dev->dev_ops = &ml_dev_dummy_ops; diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 03a7447dc87..e6383283d31 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -123,7 +123,7 @@ cnxk_ml_qp_destroy(const struct rte_ml_dev *dev, struct cnxk_ml_qp *qp) return 0; } -static int +int cn10k_ml_dev_queue_pair_release(struct rte_ml_dev *dev, uint16_t queue_pair_id) { struct cnxk_ml_qp *qp; @@ -864,7 +864,7 @@ cn10k_ml_cache_model_data(struct rte_ml_dev *dev, uint16_t model_id) return ret; } -static int +int cn10k_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) { struct cn10k_ml_dev *cn10k_mldev; @@ -892,7 +892,7 @@ cn10k_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) return 0; } -static int +int cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *conf) { struct rte_ml_dev_info dev_info; @@ -1091,7 +1091,7 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c return ret; } -static int +int cn10k_ml_dev_close(struct rte_ml_dev *dev) { struct cn10k_ml_dev *cn10k_mldev; @@ -1164,7 +1164,7 @@ cn10k_ml_dev_close(struct rte_ml_dev *dev) return rte_dev_remove(dev->device); } -static int +int cn10k_ml_dev_start(struct rte_ml_dev *dev) { struct cn10k_ml_dev *cn10k_mldev; @@ -1184,7 +1184,7 @@ cn10k_ml_dev_start(struct rte_ml_dev *dev) return 0; } -static int +int cn10k_ml_dev_stop(struct rte_ml_dev *dev) { struct cn10k_ml_dev *cn10k_mldev; @@ -1204,7 +1204,7 @@ cn10k_ml_dev_stop(struct rte_ml_dev *dev) return 0; } -static int +int cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, const struct rte_ml_dev_qp_conf *qp_conf, int socket_id) { @@ -1245,7 +1245,7 @@ cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, return 0; } -static int +int cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats) { struct cnxk_ml_qp *qp; @@ -1262,7 +1262,7 @@ cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats) return 0; } -static void +void cn10k_ml_dev_stats_reset(struct rte_ml_dev *dev) { struct cnxk_ml_qp *qp; @@ -1277,7 +1277,7 @@ cn10k_ml_dev_stats_reset(struct rte_ml_dev *dev) } } -static int +int cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, int32_t model_id, struct rte_ml_dev_xstats_map *xstats_map, uint32_t size) @@ -1325,7 +1325,7 @@ cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mod return idx; } -static int +int cn10k_ml_dev_xstats_by_name_get(struct rte_ml_dev *dev, const char *name, uint16_t *stat_id, uint64_t *value) { @@ -1367,7 +1367,7 @@ cn10k_ml_dev_xstats_by_name_get(struct rte_ml_dev *dev, const char *name, uint16 return -EINVAL; } -static int +int cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, int32_t model_id, const uint16_t stat_ids[], uint64_t values[], uint16_t nb_ids) { @@ -1431,7 +1431,7 @@ cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode return idx; } -static int +int cn10k_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, int32_t model_id, const uint16_t stat_ids[], uint16_t nb_ids) { @@ -1445,7 +1445,7 @@ cn10k_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mo return 0; } -static int +int cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) { struct cn10k_ml_dev *cn10k_mldev; @@ -1532,7 +1532,7 @@ cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) return 0; } -static int +int cn10k_ml_dev_selftest(struct rte_ml_dev *dev) { struct cn10k_ml_dev *cn10k_mldev; @@ -2055,7 +2055,7 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) return ret; } -static int +int cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_model_info *model_info) { @@ -2075,7 +2075,7 @@ cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, return 0; } -static int +int cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buffer) { struct cnxk_ml_model *model; @@ -2109,7 +2109,7 @@ cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *bu return 0; } -static int +int cn10k_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buff_seg **dbuffer, struct rte_ml_buff_seg **qbuffer) { @@ -2190,7 +2190,7 @@ cn10k_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_bu return 0; } -static int +int cn10k_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buff_seg **qbuffer, struct rte_ml_buff_seg **dbuffer) { @@ -2578,38 +2578,3 @@ cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) error_enqueue: return ret; } - -struct rte_ml_dev_ops cn10k_ml_ops = { - /* Device control ops */ - .dev_info_get = cn10k_ml_dev_info_get, - .dev_configure = cn10k_ml_dev_configure, - .dev_close = cn10k_ml_dev_close, - .dev_start = cn10k_ml_dev_start, - .dev_stop = cn10k_ml_dev_stop, - .dev_dump = cn10k_ml_dev_dump, - .dev_selftest = cn10k_ml_dev_selftest, - - /* Queue-pair handling ops */ - .dev_queue_pair_setup = cn10k_ml_dev_queue_pair_setup, - .dev_queue_pair_release = cn10k_ml_dev_queue_pair_release, - - /* Stats ops */ - .dev_stats_get = cn10k_ml_dev_stats_get, - .dev_stats_reset = cn10k_ml_dev_stats_reset, - .dev_xstats_names_get = cn10k_ml_dev_xstats_names_get, - .dev_xstats_by_name_get = cn10k_ml_dev_xstats_by_name_get, - .dev_xstats_get = cn10k_ml_dev_xstats_get, - .dev_xstats_reset = cn10k_ml_dev_xstats_reset, - - /* Model ops */ - .model_load = cn10k_ml_model_load, - .model_unload = cn10k_ml_model_unload, - .model_start = cn10k_ml_model_start, - .model_stop = cn10k_ml_model_stop, - .model_info_get = cn10k_ml_model_info_get, - .model_params_update = cn10k_ml_model_params_update, - - /* I/O ops */ - .io_quantize = cn10k_ml_io_quantize, - .io_dequantize = cn10k_ml_io_dequantize, -}; diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index fd5992e1925..16480b9ad89 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -286,7 +286,29 @@ struct cn10k_ml_req { }; /* Device ops */ -extern struct rte_ml_dev_ops cn10k_ml_ops; +int cn10k_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info); +int cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *conf); +int cn10k_ml_dev_close(struct rte_ml_dev *dev); +int cn10k_ml_dev_start(struct rte_ml_dev *dev); +int cn10k_ml_dev_stop(struct rte_ml_dev *dev); +int cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp); +int cn10k_ml_dev_selftest(struct rte_ml_dev *dev); +int cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, + const struct rte_ml_dev_qp_conf *qp_conf, int socket_id); +int cn10k_ml_dev_queue_pair_release(struct rte_ml_dev *dev, uint16_t queue_pair_id); + +int cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats); +void cn10k_ml_dev_stats_reset(struct rte_ml_dev *dev); +int cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, + int32_t model_id, struct rte_ml_dev_xstats_map *xstats_map, + uint32_t size); +int cn10k_ml_dev_xstats_by_name_get(struct rte_ml_dev *dev, const char *name, uint16_t *stat_id, + uint64_t *value); +int cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, + int32_t model_id, const uint16_t stat_ids[], uint64_t values[], + uint16_t nb_ids); +int cn10k_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, + int32_t model_id, const uint16_t stat_ids[], uint16_t nb_ids); /* Slow-path ops */ int cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, @@ -294,6 +316,16 @@ int cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *para int cn10k_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id); int cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id); int cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id); +int cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, + struct rte_ml_model_info *model_info); +int cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buffer); + +/* I/O ops */ +int cn10k_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, + struct rte_ml_buff_seg **dbuffer, struct rte_ml_buff_seg **qbuffer); + +int cn10k_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, + struct rte_ml_buff_seg **qbuffer, struct rte_ml_buff_seg **dbuffer); /* Fast-path ops */ __rte_hot uint16_t cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index f1872dcf7c6..89e0d9d32c3 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -3,5 +3,43 @@ */ #include +#include + +#include "cn10k_ml_ops.h" #include "cnxk_ml_ops.h" + +struct rte_ml_dev_ops cnxk_ml_ops = { + /* Device control ops */ + .dev_info_get = cn10k_ml_dev_info_get, + .dev_configure = cn10k_ml_dev_configure, + .dev_close = cn10k_ml_dev_close, + .dev_start = cn10k_ml_dev_start, + .dev_stop = cn10k_ml_dev_stop, + .dev_dump = cn10k_ml_dev_dump, + .dev_selftest = cn10k_ml_dev_selftest, + + /* Queue-pair handling ops */ + .dev_queue_pair_setup = cn10k_ml_dev_queue_pair_setup, + .dev_queue_pair_release = cn10k_ml_dev_queue_pair_release, + + /* Stats ops */ + .dev_stats_get = cn10k_ml_dev_stats_get, + .dev_stats_reset = cn10k_ml_dev_stats_reset, + .dev_xstats_names_get = cn10k_ml_dev_xstats_names_get, + .dev_xstats_by_name_get = cn10k_ml_dev_xstats_by_name_get, + .dev_xstats_get = cn10k_ml_dev_xstats_get, + .dev_xstats_reset = cn10k_ml_dev_xstats_reset, + + /* Model ops */ + .model_load = cn10k_ml_model_load, + .model_unload = cn10k_ml_model_unload, + .model_start = cn10k_ml_model_start, + .model_stop = cn10k_ml_model_stop, + .model_info_get = cn10k_ml_model_info_get, + .model_params_update = cn10k_ml_model_params_update, + + /* I/O ops */ + .io_quantize = cn10k_ml_io_quantize, + .io_dequantize = cn10k_ml_io_dequantize, +}; diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h index b953fb0f5fc..a925c075809 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.h +++ b/drivers/ml/cnxk/cnxk_ml_ops.h @@ -60,4 +60,6 @@ struct cnxk_ml_qp { struct rte_ml_dev_stats stats; }; +extern struct rte_ml_dev_ops cnxk_ml_ops; + #endif /* _CNXK_ML_OPS_H_ */ From patchwork Wed Aug 30 15:58:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130886 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BA02541FD1; Wed, 30 Aug 2023 18:00:19 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CD259402B3; Wed, 30 Aug 2023 17:59:41 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id CC0154029C for ; Wed, 30 Aug 2023 17:59:36 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I6s012554 for ; Wed, 30 Aug 2023 08:59:36 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=hatF2CHhzl/qeOROzVk54Wuu3GZEikzN79jiAwd7sUk=; b=CTZAf1b4Bs292CjwITLjhsEMXadDazOFFInJ8p8INmi2cbXpMjINRADLlw0SsA9cPioO kBJNG2/MTveuxB07/88ZyKCBgyun5mSjZLvtAtwhDbPCw/1nH98IFeQ2idNZnUfcr1j2 /dqfzyepAuRimipbgvtYu6B2naZTd2uQz7LDlcPuTfw/g7kwCckJFSZW4fYLUydx+44c QYGoTI/cGxYF9sOWOqH8Y7JMSU5qL7e2rKnOx2rI5yy758CGrKx1y0BsrjPz4oK77dRP c4tId0OWrfASD2otVLvvWx+tKvvHcA9mWeD0SFwyJzXXM1ERopkAyC9w8glBQPklSXR+ dw== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:35 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:34 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:34 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 637F83F7080; Wed, 30 Aug 2023 08:59:34 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 08/34] ml/cnxk: update device handling functions Date: Wed, 30 Aug 2023 08:58:58 -0700 Message-ID: <20230830155927.3566-9-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: MMKDRMNZOSS4nnNjNBwT4gt1O9yAeIQg X-Proofpoint-GUID: MMKDRMNZOSS4nnNjNBwT4gt1O9yAeIQg X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implement CNXK wrapper functions for dev_info_get, dev_configure, dev_close, dev_start and dev_stop. The wrapper functions allocate / release common resources for the ML driver and invoke device specific functions. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ops.c | 230 ++------------------------ drivers/ml/cnxk/cn10k_ml_ops.h | 16 +- drivers/ml/cnxk/cnxk_ml_dev.h | 3 + drivers/ml/cnxk/cnxk_ml_ops.c | 286 ++++++++++++++++++++++++++++++++- drivers/ml/cnxk/cnxk_ml_ops.h | 3 + 5 files changed, 314 insertions(+), 224 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index e6383283d31..0f32f3b2bbe 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -105,7 +105,7 @@ qp_memzone_name_get(char *name, int size, int dev_id, int qp_id) snprintf(name, size, "cnxk_ml_qp_mem_%u:%u", dev_id, qp_id); } -static int +int cnxk_ml_qp_destroy(const struct rte_ml_dev *dev, struct cnxk_ml_qp *qp) { const struct rte_memzone *qp_mem; @@ -865,20 +865,12 @@ cn10k_ml_cache_model_data(struct rte_ml_dev *dev, uint16_t model_id) } int -cn10k_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) +cn10k_ml_dev_info_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_dev_info *dev_info) { struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - if (dev_info == NULL) - return -EINVAL; - - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; - memset(dev_info, 0, sizeof(struct rte_ml_dev_info)); - dev_info->driver_name = dev->device->driver->name; - dev_info->max_models = ML_CNXK_MAX_MODELS; if (cn10k_mldev->hw_queue_lock) dev_info->max_queue_pairs = ML_CN10K_MAX_QP_PER_DEVICE_SL; else @@ -893,143 +885,17 @@ cn10k_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) } int -cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *conf) +cn10k_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf) { - struct rte_ml_dev_info dev_info; struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - struct cnxk_ml_model *model; struct cn10k_ml_ocm *ocm; - struct cnxk_ml_qp *qp; - uint16_t model_id; - uint32_t mz_size; uint16_t tile_id; - uint16_t qp_id; int ret; - if (dev == NULL || conf == NULL) - return -EINVAL; + RTE_SET_USED(conf); - /* Get CN10K device handle */ - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; - cn10k_ml_dev_info_get(dev, &dev_info); - if (conf->nb_models > dev_info.max_models) { - plt_err("Invalid device config, nb_models > %u\n", dev_info.max_models); - return -EINVAL; - } - - if (conf->nb_queue_pairs > dev_info.max_queue_pairs) { - plt_err("Invalid device config, nb_queue_pairs > %u\n", dev_info.max_queue_pairs); - return -EINVAL; - } - - if (cnxk_mldev->state == ML_CNXK_DEV_STATE_PROBED) { - plt_ml_dbg("Configuring ML device, nb_queue_pairs = %u, nb_models = %u", - conf->nb_queue_pairs, conf->nb_models); - - /* Load firmware */ - ret = cn10k_ml_fw_load(cnxk_mldev); - if (ret != 0) - return ret; - } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_CONFIGURED) { - plt_ml_dbg("Re-configuring ML device, nb_queue_pairs = %u, nb_models = %u", - conf->nb_queue_pairs, conf->nb_models); - } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_STARTED) { - plt_err("Device can't be reconfigured in started state\n"); - return -ENOTSUP; - } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_CLOSED) { - plt_err("Device can't be reconfigured after close\n"); - return -ENOTSUP; - } - - /* Configure queue-pairs */ - if (dev->data->queue_pairs == NULL) { - mz_size = sizeof(dev->data->queue_pairs[0]) * conf->nb_queue_pairs; - dev->data->queue_pairs = - rte_zmalloc("cn10k_mldev_queue_pairs", mz_size, RTE_CACHE_LINE_SIZE); - if (dev->data->queue_pairs == NULL) { - dev->data->nb_queue_pairs = 0; - plt_err("Failed to get memory for queue_pairs, nb_queue_pairs %u", - conf->nb_queue_pairs); - return -ENOMEM; - } - } else { /* Re-configure */ - void **queue_pairs; - - /* Release all queue pairs as ML spec doesn't support queue_pair_destroy. */ - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - qp = dev->data->queue_pairs[qp_id]; - if (qp != NULL) { - ret = cn10k_ml_dev_queue_pair_release(dev, qp_id); - if (ret < 0) - return ret; - } - } - - queue_pairs = dev->data->queue_pairs; - queue_pairs = - rte_realloc(queue_pairs, sizeof(queue_pairs[0]) * conf->nb_queue_pairs, - RTE_CACHE_LINE_SIZE); - if (queue_pairs == NULL) { - dev->data->nb_queue_pairs = 0; - plt_err("Failed to realloc queue_pairs, nb_queue_pairs = %u", - conf->nb_queue_pairs); - ret = -ENOMEM; - goto error; - } - - memset(queue_pairs, 0, sizeof(queue_pairs[0]) * conf->nb_queue_pairs); - dev->data->queue_pairs = queue_pairs; - } - dev->data->nb_queue_pairs = conf->nb_queue_pairs; - - /* Allocate ML models */ - if (dev->data->models == NULL) { - mz_size = sizeof(dev->data->models[0]) * conf->nb_models; - dev->data->models = rte_zmalloc("cn10k_mldev_models", mz_size, RTE_CACHE_LINE_SIZE); - if (dev->data->models == NULL) { - dev->data->nb_models = 0; - plt_err("Failed to get memory for ml_models, nb_models %u", - conf->nb_models); - ret = -ENOMEM; - goto error; - } - } else { - /* Re-configure */ - void **models; - - /* Stop and unload all models */ - for (model_id = 0; model_id < dev->data->nb_models; model_id++) { - model = dev->data->models[model_id]; - if (model != NULL) { - if (model->state == ML_CNXK_MODEL_STATE_STARTED) { - if (cn10k_ml_model_stop(dev, model_id) != 0) - plt_err("Could not stop model %u", model_id); - } - if (model->state == ML_CNXK_MODEL_STATE_LOADED) { - if (cn10k_ml_model_unload(dev, model_id) != 0) - plt_err("Could not unload model %u", model_id); - } - dev->data->models[model_id] = NULL; - } - } - - models = dev->data->models; - models = rte_realloc(models, sizeof(models[0]) * conf->nb_models, - RTE_CACHE_LINE_SIZE); - if (models == NULL) { - dev->data->nb_models = 0; - plt_err("Failed to realloc ml_models, nb_models = %u", conf->nb_models); - ret = -ENOMEM; - goto error; - } - memset(models, 0, sizeof(models[0]) * conf->nb_models); - dev->data->models = models; - } - dev->data->nb_models = conf->nb_models; - ocm = &cn10k_mldev->ocm; ocm->num_tiles = ML_CN10K_OCM_NUMTILES; ocm->size_per_tile = ML_CN10K_OCM_TILESIZE; @@ -1042,8 +908,7 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c rte_zmalloc("ocm_mask", ocm->mask_words * ocm->num_tiles, RTE_CACHE_LINE_SIZE); if (ocm->ocm_mask == NULL) { plt_err("Unable to allocate memory for OCM mask"); - ret = -ENOMEM; - goto error; + return -ENOMEM; } for (tile_id = 0; tile_id < ocm->num_tiles; tile_id++) { @@ -1054,10 +919,10 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c rte_spinlock_init(&ocm->lock); /* Initialize xstats */ - ret = cn10k_ml_xstats_init(dev); + ret = cn10k_ml_xstats_init(cnxk_mldev->mldev); if (ret != 0) { plt_err("Failed to initialize xstats"); - goto error; + return ret; } /* Set JCMDQ enqueue function */ @@ -1071,77 +936,25 @@ cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *c cn10k_mldev->set_poll_ptr = cn10k_ml_set_poll_ptr; cn10k_mldev->get_poll_ptr = cn10k_ml_get_poll_ptr; - dev->enqueue_burst = cn10k_ml_enqueue_burst; - dev->dequeue_burst = cn10k_ml_dequeue_burst; - dev->op_error_get = cn10k_ml_op_error_get; - - cnxk_mldev->nb_models_loaded = 0; - cnxk_mldev->nb_models_started = 0; - cnxk_mldev->nb_models_stopped = 0; - cnxk_mldev->nb_models_unloaded = 0; - cnxk_mldev->state = ML_CNXK_DEV_STATE_CONFIGURED; + cnxk_mldev->mldev->enqueue_burst = cn10k_ml_enqueue_burst; + cnxk_mldev->mldev->dequeue_burst = cn10k_ml_dequeue_burst; + cnxk_mldev->mldev->op_error_get = cn10k_ml_op_error_get; return 0; - -error: - rte_free(dev->data->queue_pairs); - - rte_free(dev->data->models); - - return ret; } int -cn10k_ml_dev_close(struct rte_ml_dev *dev) +cn10k_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev) { struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - struct cnxk_ml_model *model; - struct cnxk_ml_qp *qp; - uint16_t model_id; - uint16_t qp_id; - if (dev == NULL) - return -EINVAL; - - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; /* Release ocm_mask memory */ rte_free(cn10k_mldev->ocm.ocm_mask); - /* Stop and unload all models */ - for (model_id = 0; model_id < dev->data->nb_models; model_id++) { - model = dev->data->models[model_id]; - if (model != NULL) { - if (model->state == ML_CNXK_MODEL_STATE_STARTED) { - if (cn10k_ml_model_stop(dev, model_id) != 0) - plt_err("Could not stop model %u", model_id); - } - if (model->state == ML_CNXK_MODEL_STATE_LOADED) { - if (cn10k_ml_model_unload(dev, model_id) != 0) - plt_err("Could not unload model %u", model_id); - } - dev->data->models[model_id] = NULL; - } - } - - rte_free(dev->data->models); - - /* Destroy all queue pairs */ - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - qp = dev->data->queue_pairs[qp_id]; - if (qp != NULL) { - if (cnxk_ml_qp_destroy(dev, qp) != 0) - plt_err("Could not destroy queue pair %u", qp_id); - dev->data->queue_pairs[qp_id] = NULL; - } - } - - rte_free(dev->data->queue_pairs); - /* Un-initialize xstats */ - cn10k_ml_xstats_uninit(dev); + cn10k_ml_xstats_uninit(cnxk_mldev->mldev); /* Unload firmware */ cn10k_ml_fw_unload(cnxk_mldev); @@ -1158,20 +971,15 @@ cn10k_ml_dev_close(struct rte_ml_dev *dev) roc_ml_reg_write64(&cn10k_mldev->roc, 0, ML_MLR_BASE); plt_ml_dbg("ML_MLR_BASE = 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_MLR_BASE)); - cnxk_mldev->state = ML_CNXK_DEV_STATE_CLOSED; - - /* Remove PCI device */ - return rte_dev_remove(dev->device); + return 0; } int -cn10k_ml_dev_start(struct rte_ml_dev *dev) +cn10k_ml_dev_start(struct cnxk_ml_dev *cnxk_mldev) { struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; uint64_t reg_val64; - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); @@ -1179,19 +987,15 @@ cn10k_ml_dev_start(struct rte_ml_dev *dev) roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); - cnxk_mldev->state = ML_CNXK_DEV_STATE_STARTED; - return 0; } int -cn10k_ml_dev_stop(struct rte_ml_dev *dev) +cn10k_ml_dev_stop(struct cnxk_ml_dev *cnxk_mldev) { struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; uint64_t reg_val64; - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; reg_val64 = roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG); @@ -1199,8 +1003,6 @@ cn10k_ml_dev_stop(struct rte_ml_dev *dev) roc_ml_reg_write64(&cn10k_mldev->roc, reg_val64, ML_CFG); plt_ml_dbg("ML_CFG => 0x%016lx", roc_ml_reg_read64(&cn10k_mldev->roc, ML_CFG)); - cnxk_mldev->state = ML_CNXK_DEV_STATE_CONFIGURED; - return 0; } @@ -1221,7 +1023,7 @@ cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, if (dev->data->queue_pairs[queue_pair_id] != NULL) cn10k_ml_dev_queue_pair_release(dev, queue_pair_id); - cn10k_ml_dev_info_get(dev, &dev_info); + cnxk_ml_dev_info_get(dev, &dev_info); if ((qp_conf->nb_desc > dev_info.max_desc) || (qp_conf->nb_desc == 0)) { plt_err("Could not setup queue pair for %u descriptors", qp_conf->nb_desc); return -EINVAL; diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 16480b9ad89..d50b5bede71 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -10,6 +10,9 @@ #include +struct cnxk_ml_dev; +struct cnxk_ml_qp; + /* Firmware version string length */ #define MLDEV_FIRMWARE_VERSION_LENGTH 32 @@ -286,11 +289,11 @@ struct cn10k_ml_req { }; /* Device ops */ -int cn10k_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info); -int cn10k_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *conf); -int cn10k_ml_dev_close(struct rte_ml_dev *dev); -int cn10k_ml_dev_start(struct rte_ml_dev *dev); -int cn10k_ml_dev_stop(struct rte_ml_dev *dev); +int cn10k_ml_dev_info_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_dev_info *dev_info); +int cn10k_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf); +int cn10k_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev); +int cn10k_ml_dev_start(struct cnxk_ml_dev *cnxk_mldev); +int cn10k_ml_dev_stop(struct cnxk_ml_dev *cnxk_mldev); int cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp); int cn10k_ml_dev_selftest(struct rte_ml_dev *dev); int cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, @@ -336,4 +339,7 @@ __rte_hot int cn10k_ml_op_error_get(struct rte_ml_dev *dev, struct rte_ml_op *op struct rte_ml_op_error *error); __rte_hot int cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op); +/* Temporarily set below functions as non-static */ +int cnxk_ml_qp_destroy(const struct rte_ml_dev *dev, struct cnxk_ml_qp *qp); + #endif /* _CN10K_ML_OPS_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_dev.h b/drivers/ml/cnxk/cnxk_ml_dev.h index 51315de6227..02605fa28fc 100644 --- a/drivers/ml/cnxk/cnxk_ml_dev.h +++ b/drivers/ml/cnxk/cnxk_ml_dev.h @@ -53,6 +53,9 @@ struct cnxk_ml_dev { /* CN10K device structure */ struct cn10k_ml_dev cn10k_mldev; + + /* Maximum number of layers */ + uint64_t max_nb_layers; }; #endif /* _CNXK_ML_DEV_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 89e0d9d32c3..83d5cbae58b 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -7,15 +7,291 @@ #include "cn10k_ml_ops.h" +#include "cnxk_ml_dev.h" +#include "cnxk_ml_io.h" +#include "cnxk_ml_model.h" #include "cnxk_ml_ops.h" +int +cnxk_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) +{ + struct cnxk_ml_dev *cnxk_mldev; + + if (dev == NULL || dev_info == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + memset(dev_info, 0, sizeof(struct rte_ml_dev_info)); + dev_info->driver_name = dev->device->driver->name; + dev_info->max_models = ML_CNXK_MAX_MODELS; + + return cn10k_ml_dev_info_get(cnxk_mldev, dev_info); +} + +static int +cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *conf) +{ + struct rte_ml_dev_info dev_info; + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + struct cnxk_ml_qp *qp; + uint16_t model_id; + uint32_t mz_size; + uint16_t qp_id; + int ret; + + if (dev == NULL) + return -EINVAL; + + /* Get CNXK device handle */ + cnxk_mldev = dev->data->dev_private; + + cnxk_ml_dev_info_get(dev, &dev_info); + if (conf->nb_models > dev_info.max_models) { + plt_err("Invalid device config, nb_models > %u\n", dev_info.max_models); + return -EINVAL; + } + + if (conf->nb_queue_pairs > dev_info.max_queue_pairs) { + plt_err("Invalid device config, nb_queue_pairs > %u\n", dev_info.max_queue_pairs); + return -EINVAL; + } + + if (cnxk_mldev->state == ML_CNXK_DEV_STATE_PROBED) { + plt_ml_dbg("Configuring ML device, nb_queue_pairs = %u, nb_models = %u", + conf->nb_queue_pairs, conf->nb_models); + + /* Load firmware */ + ret = cn10k_ml_fw_load(cnxk_mldev); + if (ret != 0) + return ret; + } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_CONFIGURED) { + plt_ml_dbg("Re-configuring ML device, nb_queue_pairs = %u, nb_models = %u", + conf->nb_queue_pairs, conf->nb_models); + } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_STARTED) { + plt_err("Device can't be reconfigured in started state\n"); + return -ENOTSUP; + } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_CLOSED) { + plt_err("Device can't be reconfigured after close\n"); + return -ENOTSUP; + } + + /* Configure queue-pairs */ + if (dev->data->queue_pairs == NULL) { + mz_size = sizeof(dev->data->queue_pairs[0]) * conf->nb_queue_pairs; + dev->data->queue_pairs = + rte_zmalloc("cnxk_mldev_queue_pairs", mz_size, RTE_CACHE_LINE_SIZE); + if (dev->data->queue_pairs == NULL) { + dev->data->nb_queue_pairs = 0; + plt_err("Failed to get memory for queue_pairs, nb_queue_pairs %u", + conf->nb_queue_pairs); + return -ENOMEM; + } + } else { /* Re-configure */ + void **queue_pairs; + + /* Release all queue pairs as ML spec doesn't support queue_pair_destroy. */ + for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { + qp = dev->data->queue_pairs[qp_id]; + if (qp != NULL) { + ret = cn10k_ml_dev_queue_pair_release(dev, qp_id); + if (ret < 0) + return ret; + } + } + + queue_pairs = dev->data->queue_pairs; + queue_pairs = + rte_realloc(queue_pairs, sizeof(queue_pairs[0]) * conf->nb_queue_pairs, + RTE_CACHE_LINE_SIZE); + if (queue_pairs == NULL) { + dev->data->nb_queue_pairs = 0; + plt_err("Failed to realloc queue_pairs, nb_queue_pairs = %u", + conf->nb_queue_pairs); + ret = -ENOMEM; + goto error; + } + + memset(queue_pairs, 0, sizeof(queue_pairs[0]) * conf->nb_queue_pairs); + dev->data->queue_pairs = queue_pairs; + } + dev->data->nb_queue_pairs = conf->nb_queue_pairs; + + /* Allocate ML models */ + if (dev->data->models == NULL) { + mz_size = sizeof(dev->data->models[0]) * conf->nb_models; + dev->data->models = rte_zmalloc("cnxk_mldev_models", mz_size, RTE_CACHE_LINE_SIZE); + if (dev->data->models == NULL) { + dev->data->nb_models = 0; + plt_err("Failed to get memory for ml_models, nb_models %u", + conf->nb_models); + ret = -ENOMEM; + goto error; + } + } else { + /* Re-configure */ + void **models; + + /* Stop and unload all models */ + for (model_id = 0; model_id < dev->data->nb_models; model_id++) { + model = dev->data->models[model_id]; + if (model != NULL) { + if (model->state == ML_CNXK_MODEL_STATE_STARTED) { + if (cn10k_ml_model_stop(dev, model_id) != 0) + plt_err("Could not stop model %u", model_id); + } + if (model->state == ML_CNXK_MODEL_STATE_LOADED) { + if (cn10k_ml_model_unload(dev, model_id) != 0) + plt_err("Could not unload model %u", model_id); + } + dev->data->models[model_id] = NULL; + } + } + + models = dev->data->models; + models = rte_realloc(models, sizeof(models[0]) * conf->nb_models, + RTE_CACHE_LINE_SIZE); + if (models == NULL) { + dev->data->nb_models = 0; + plt_err("Failed to realloc ml_models, nb_models = %u", conf->nb_models); + ret = -ENOMEM; + goto error; + } + memset(models, 0, sizeof(models[0]) * conf->nb_models); + dev->data->models = models; + } + dev->data->nb_models = conf->nb_models; + + ret = cn10k_ml_dev_configure(cnxk_mldev, conf); + if (ret != 0) { + plt_err("Failed to configure CN10K ML Device"); + goto error; + } + + /* Set device capabilities */ + cnxk_mldev->max_nb_layers = + cnxk_mldev->cn10k_mldev.fw.req->cn10k_req.jd.fw_load.cap.s.max_models; + + cnxk_mldev->nb_models_loaded = 0; + cnxk_mldev->nb_models_started = 0; + cnxk_mldev->nb_models_stopped = 0; + cnxk_mldev->nb_models_unloaded = 0; + cnxk_mldev->state = ML_CNXK_DEV_STATE_CONFIGURED; + + return 0; + +error: + rte_free(dev->data->queue_pairs); + rte_free(dev->data->models); + + return ret; +} + +static int +cnxk_ml_dev_close(struct rte_ml_dev *dev) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + struct cnxk_ml_qp *qp; + uint16_t model_id; + uint16_t qp_id; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + if (cn10k_ml_dev_close(cnxk_mldev) != 0) + plt_err("Failed to close CN10K ML Device"); + + /* Stop and unload all models */ + for (model_id = 0; model_id < dev->data->nb_models; model_id++) { + model = dev->data->models[model_id]; + if (model != NULL) { + if (model->state == ML_CNXK_MODEL_STATE_STARTED) { + if (cn10k_ml_model_stop(dev, model_id) != 0) + plt_err("Could not stop model %u", model_id); + } + if (model->state == ML_CNXK_MODEL_STATE_LOADED) { + if (cn10k_ml_model_unload(dev, model_id) != 0) + plt_err("Could not unload model %u", model_id); + } + dev->data->models[model_id] = NULL; + } + } + + rte_free(dev->data->models); + + /* Destroy all queue pairs */ + for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { + qp = dev->data->queue_pairs[qp_id]; + if (qp != NULL) { + if (cnxk_ml_qp_destroy(dev, qp) != 0) + plt_err("Could not destroy queue pair %u", qp_id); + dev->data->queue_pairs[qp_id] = NULL; + } + } + + rte_free(dev->data->queue_pairs); + + cnxk_mldev->state = ML_CNXK_DEV_STATE_CLOSED; + + /* Remove PCI device */ + return rte_dev_remove(dev->device); +} + +static int +cnxk_ml_dev_start(struct rte_ml_dev *dev) +{ + struct cnxk_ml_dev *cnxk_mldev; + int ret; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + ret = cn10k_ml_dev_start(cnxk_mldev); + if (ret != 0) { + plt_err("Failed to start CN10K ML Device"); + return ret; + } + + cnxk_mldev->state = ML_CNXK_DEV_STATE_STARTED; + + return 0; +} + +static int +cnxk_ml_dev_stop(struct rte_ml_dev *dev) +{ + struct cnxk_ml_dev *cnxk_mldev; + int ret; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + ret = cn10k_ml_dev_stop(cnxk_mldev); + if (ret != 0) { + plt_err("Failed to stop CN10K ML Device"); + return ret; + } + + cnxk_mldev->state = ML_CNXK_DEV_STATE_CONFIGURED; + + return 0; +} + struct rte_ml_dev_ops cnxk_ml_ops = { /* Device control ops */ - .dev_info_get = cn10k_ml_dev_info_get, - .dev_configure = cn10k_ml_dev_configure, - .dev_close = cn10k_ml_dev_close, - .dev_start = cn10k_ml_dev_start, - .dev_stop = cn10k_ml_dev_stop, + .dev_info_get = cnxk_ml_dev_info_get, + .dev_configure = cnxk_ml_dev_configure, + .dev_close = cnxk_ml_dev_close, + .dev_start = cnxk_ml_dev_start, + .dev_stop = cnxk_ml_dev_stop, .dev_dump = cn10k_ml_dev_dump, .dev_selftest = cn10k_ml_dev_selftest, diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h index a925c075809..2996928d7d0 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.h +++ b/drivers/ml/cnxk/cnxk_ml_ops.h @@ -62,4 +62,7 @@ struct cnxk_ml_qp { extern struct rte_ml_dev_ops cnxk_ml_ops; +/* Temporarily set cnxk driver functions as non-static */ +int cnxk_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info); + #endif /* _CNXK_ML_OPS_H_ */ From patchwork Wed Aug 30 15:58:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130892 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0583941FD1; Wed, 30 Aug 2023 18:01:26 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1D09D402CC; Wed, 30 Aug 2023 17:59:48 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 5A78D402A3 for ; Wed, 30 Aug 2023 17:59:38 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7p3004368 for ; Wed, 30 Aug 2023 08:59:37 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=EkmwyxVT2u3zDhj2fK5wpuHGXG2WSxk9aNYgZ9Sx6dY=; b=JkhPk6RwQ529PwpaqKaSU4sbIShiy/7zHJb+TCBeoSFm4brnS/ECOkeikdWlJVKELAVj gGxTcHHcOsa5MM0ihP6TqvNjSRaw9/Botj4kQ2YQJm2T0CL6dQlDCXZ8WVslCRRW99pN Tho+nZPxrQ+ewNGHEBEQClnghORWB4SDwF29W68tvfwmByh828JveWNw869JFJqtE9Jo Q9HLY9GR7132mHqG3YjK7g0MHagYueDyUG9w2X2qNeZ7gNBqNaJJFz9cEsd5ezXIJjDd fhG0A6xcCmRzga04eTyZXWrcQN5CVE+3EAlsY+AwCB5RnlIjrIHjUC+ApLXP5XRqEjgz 3w== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysst5-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:37 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:35 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:34 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id B75C83F707E; Wed, 30 Aug 2023 08:59:34 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 09/34] ml/cnxk: update queue-pair handling functions Date: Wed, 30 Aug 2023 08:58:59 -0700 Message-ID: <20230830155927.3566-10-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: OlmwJyx7Rx6WI7mSw1G_XtnZ1JJdl9IQ X-Proofpoint-ORIG-GUID: OlmwJyx7Rx6WI7mSw1G_XtnZ1JJdl9IQ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added cnxk wrapper function to handle ML device queue-pairs. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ops.c | 135 +---------------------------- drivers/ml/cnxk/cn10k_ml_ops.h | 7 +- drivers/ml/cnxk/cnxk_ml_ops.c | 153 ++++++++++++++++++++++++++++++++- drivers/ml/cnxk/cnxk_ml_ops.h | 3 - 4 files changed, 154 insertions(+), 144 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 0f32f3b2bbe..330cb050cbd 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -99,93 +99,12 @@ cn10k_ml_get_poll_ptr(struct cnxk_ml_req *req) return plt_read64(req->status); } -static void -qp_memzone_name_get(char *name, int size, int dev_id, int qp_id) -{ - snprintf(name, size, "cnxk_ml_qp_mem_%u:%u", dev_id, qp_id); -} - -int -cnxk_ml_qp_destroy(const struct rte_ml_dev *dev, struct cnxk_ml_qp *qp) -{ - const struct rte_memzone *qp_mem; - char name[RTE_MEMZONE_NAMESIZE]; - int ret; - - qp_memzone_name_get(name, RTE_MEMZONE_NAMESIZE, dev->data->dev_id, qp->id); - qp_mem = rte_memzone_lookup(name); - ret = rte_memzone_free(qp_mem); - if (ret) - return ret; - - rte_free(qp); - - return 0; -} - -int -cn10k_ml_dev_queue_pair_release(struct rte_ml_dev *dev, uint16_t queue_pair_id) -{ - struct cnxk_ml_qp *qp; - int ret; - - qp = dev->data->queue_pairs[queue_pair_id]; - if (qp == NULL) - return -EINVAL; - - ret = cnxk_ml_qp_destroy(dev, qp); - if (ret) { - plt_err("Could not destroy queue pair %u", queue_pair_id); - return ret; - } - - dev->data->queue_pairs[queue_pair_id] = NULL; - - return 0; -} - -static struct cnxk_ml_qp * -cnxk_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_desc, int socket_id) +void +cn10k_ml_qp_initialize(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_qp *qp) { - const struct rte_memzone *qp_mem; - char name[RTE_MEMZONE_NAMESIZE]; - struct cnxk_ml_qp *qp; - uint32_t len; - uint8_t *va; uint64_t i; - /* Allocate queue pair */ - qp = rte_zmalloc_socket("cn10k_ml_pmd_queue_pair", sizeof(struct cnxk_ml_qp), ROC_ALIGN, - socket_id); - if (qp == NULL) { - plt_err("Could not allocate queue pair"); - return NULL; - } - - /* For request queue */ - len = nb_desc * sizeof(struct cnxk_ml_req); - qp_memzone_name_get(name, RTE_MEMZONE_NAMESIZE, dev->data->dev_id, qp_id); - qp_mem = rte_memzone_reserve_aligned( - name, len, socket_id, RTE_MEMZONE_SIZE_HINT_ONLY | RTE_MEMZONE_256MB, ROC_ALIGN); - if (qp_mem == NULL) { - plt_err("Could not reserve memzone: %s", name); - goto qp_free; - } - - va = qp_mem->addr; - memset(va, 0, len); - - /* Initialize Request queue */ - qp->id = qp_id; - qp->queue.reqs = (struct cnxk_ml_req *)va; - qp->queue.head = 0; - qp->queue.tail = 0; - qp->queue.wait_cycles = ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); - qp->nb_desc = nb_desc; - qp->stats.enqueued_count = 0; - qp->stats.dequeued_count = 0; - qp->stats.enqueue_err_count = 0; - qp->stats.dequeue_err_count = 0; + RTE_SET_USED(cnxk_mldev); /* Initialize job command */ for (i = 0; i < qp->nb_desc; i++) { @@ -193,13 +112,6 @@ cnxk_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_desc qp->queue.reqs[i].cn10k_req.jcmd.w1.s.jobptr = PLT_U64_CAST(&qp->queue.reqs[i].cn10k_req.jd); } - - return qp; - -qp_free: - rte_free(qp); - - return NULL; } static void @@ -1006,47 +918,6 @@ cn10k_ml_dev_stop(struct cnxk_ml_dev *cnxk_mldev) return 0; } -int -cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, - const struct rte_ml_dev_qp_conf *qp_conf, int socket_id) -{ - struct rte_ml_dev_info dev_info; - struct cnxk_ml_qp *qp; - uint32_t nb_desc; - - if (queue_pair_id >= dev->data->nb_queue_pairs) { - plt_err("Queue-pair id = %u (>= max queue pairs supported, %u)\n", queue_pair_id, - dev->data->nb_queue_pairs); - return -EINVAL; - } - - if (dev->data->queue_pairs[queue_pair_id] != NULL) - cn10k_ml_dev_queue_pair_release(dev, queue_pair_id); - - cnxk_ml_dev_info_get(dev, &dev_info); - if ((qp_conf->nb_desc > dev_info.max_desc) || (qp_conf->nb_desc == 0)) { - plt_err("Could not setup queue pair for %u descriptors", qp_conf->nb_desc); - return -EINVAL; - } - plt_ml_dbg("Creating queue-pair, queue_pair_id = %u, nb_desc = %u", queue_pair_id, - qp_conf->nb_desc); - - /* As the number of usable descriptors is 1 less than the queue size being created, we - * increment the size of queue by 1 than the requested size, except when the requested size - * is equal to the maximum possible size. - */ - nb_desc = - (qp_conf->nb_desc == dev_info.max_desc) ? dev_info.max_desc : qp_conf->nb_desc + 1; - qp = cnxk_ml_qp_create(dev, queue_pair_id, nb_desc, socket_id); - if (qp == NULL) { - plt_err("Could not create queue pair %u", queue_pair_id); - return -ENOMEM; - } - dev->data->queue_pairs[queue_pair_id] = qp; - - return 0; -} - int cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats) { diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index d50b5bede71..2d0a49d5cdf 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -296,9 +296,6 @@ int cn10k_ml_dev_start(struct cnxk_ml_dev *cnxk_mldev); int cn10k_ml_dev_stop(struct cnxk_ml_dev *cnxk_mldev); int cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp); int cn10k_ml_dev_selftest(struct rte_ml_dev *dev); -int cn10k_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, - const struct rte_ml_dev_qp_conf *qp_conf, int socket_id); -int cn10k_ml_dev_queue_pair_release(struct rte_ml_dev *dev, uint16_t queue_pair_id); int cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats); void cn10k_ml_dev_stats_reset(struct rte_ml_dev *dev); @@ -339,7 +336,7 @@ __rte_hot int cn10k_ml_op_error_get(struct rte_ml_dev *dev, struct rte_ml_op *op struct rte_ml_op_error *error); __rte_hot int cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op); -/* Temporarily set below functions as non-static */ -int cnxk_ml_qp_destroy(const struct rte_ml_dev *dev, struct cnxk_ml_qp *qp); +/* Misc ops */ +void cn10k_ml_qp_initialize(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_qp *qp); #endif /* _CN10K_ML_OPS_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 83d5cbae58b..1767a8a3dbc 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -12,7 +12,107 @@ #include "cnxk_ml_model.h" #include "cnxk_ml_ops.h" -int +static void +qp_memzone_name_get(char *name, int size, int dev_id, int qp_id) +{ + snprintf(name, size, "cnxk_ml_qp_mem_%u:%u", dev_id, qp_id); +} + +static int +cnxk_ml_qp_destroy(const struct rte_ml_dev *dev, struct cnxk_ml_qp *qp) +{ + const struct rte_memzone *qp_mem; + char name[RTE_MEMZONE_NAMESIZE]; + int ret; + + qp_memzone_name_get(name, RTE_MEMZONE_NAMESIZE, dev->data->dev_id, qp->id); + qp_mem = rte_memzone_lookup(name); + ret = rte_memzone_free(qp_mem); + if (ret) + return ret; + + rte_free(qp); + + return 0; +} + +static int +cnxk_ml_dev_queue_pair_release(struct rte_ml_dev *dev, uint16_t queue_pair_id) +{ + struct cnxk_ml_qp *qp; + int ret; + + qp = dev->data->queue_pairs[queue_pair_id]; + if (qp == NULL) + return -EINVAL; + + ret = cnxk_ml_qp_destroy(dev, qp); + if (ret) { + plt_err("Could not destroy queue pair %u", queue_pair_id); + return ret; + } + + dev->data->queue_pairs[queue_pair_id] = NULL; + + return 0; +} + +static struct cnxk_ml_qp * +cnxk_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_desc, int socket_id) +{ + const struct rte_memzone *qp_mem; + char name[RTE_MEMZONE_NAMESIZE]; + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_qp *qp; + uint32_t len; + uint8_t *va; + + cnxk_mldev = dev->data->dev_private; + + /* Allocate queue pair */ + qp = rte_zmalloc_socket("cnxk_ml_pmd_queue_pair", sizeof(struct cnxk_ml_qp), ROC_ALIGN, + socket_id); + if (qp == NULL) { + plt_err("Could not allocate queue pair"); + return NULL; + } + + /* For request queue */ + len = nb_desc * sizeof(struct cnxk_ml_req); + qp_memzone_name_get(name, RTE_MEMZONE_NAMESIZE, dev->data->dev_id, qp_id); + qp_mem = rte_memzone_reserve_aligned( + name, len, socket_id, RTE_MEMZONE_SIZE_HINT_ONLY | RTE_MEMZONE_256MB, ROC_ALIGN); + if (qp_mem == NULL) { + plt_err("Could not reserve memzone: %s", name); + goto qp_free; + } + + va = qp_mem->addr; + memset(va, 0, len); + + /* Initialize Request queue */ + qp->id = qp_id; + qp->queue.reqs = (struct cnxk_ml_req *)va; + qp->queue.head = 0; + qp->queue.tail = 0; + qp->queue.wait_cycles = ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); + qp->nb_desc = nb_desc; + qp->stats.enqueued_count = 0; + qp->stats.dequeued_count = 0; + qp->stats.enqueue_err_count = 0; + qp->stats.dequeue_err_count = 0; + + cn10k_ml_qp_initialize(cnxk_mldev, qp); + + return qp; + +qp_free: + rte_free(qp); + + return NULL; +} + +static int cnxk_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) { struct cnxk_ml_dev *cnxk_mldev; @@ -95,7 +195,7 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { qp = dev->data->queue_pairs[qp_id]; if (qp != NULL) { - ret = cn10k_ml_dev_queue_pair_release(dev, qp_id); + ret = cnxk_ml_dev_queue_pair_release(dev, qp_id); if (ret < 0) return ret; } @@ -285,6 +385,51 @@ cnxk_ml_dev_stop(struct rte_ml_dev *dev) return 0; } +static int +cnxk_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, + const struct rte_ml_dev_qp_conf *qp_conf, int socket_id) +{ + struct rte_ml_dev_info dev_info; + struct cnxk_ml_qp *qp; + uint32_t nb_desc; + + if (queue_pair_id >= dev->data->nb_queue_pairs) { + plt_err("Queue-pair id = %u (>= max queue pairs supported, %u)\n", queue_pair_id, + dev->data->nb_queue_pairs); + return -EINVAL; + } + + if (dev->data->queue_pairs[queue_pair_id] != NULL) + cnxk_ml_dev_queue_pair_release(dev, queue_pair_id); + + cnxk_ml_dev_info_get(dev, &dev_info); + if (qp_conf->nb_desc == 0) { + plt_err("Could not setup queue pair for %u descriptors", qp_conf->nb_desc); + return -EINVAL; + } else if (qp_conf->nb_desc > dev_info.max_desc) { + plt_err("Could not setup queue pair for %u descriptors (> %u)", qp_conf->nb_desc, + dev_info.max_desc); + return -EINVAL; + } + plt_ml_dbg("Creating queue-pair, queue_pair_id = %u, nb_desc = %u", queue_pair_id, + qp_conf->nb_desc); + + /* As the number of usable descriptors is 1 less than the queue size being created, we + * increment the size of queue by 1 than the requested size, except when the requested size + * is equal to the maximum possible size. + */ + nb_desc = + (qp_conf->nb_desc == dev_info.max_desc) ? dev_info.max_desc : qp_conf->nb_desc + 1; + qp = cnxk_ml_qp_create(dev, queue_pair_id, nb_desc, socket_id); + if (qp == NULL) { + plt_err("Could not create queue pair %u", queue_pair_id); + return -ENOMEM; + } + dev->data->queue_pairs[queue_pair_id] = qp; + + return 0; +} + struct rte_ml_dev_ops cnxk_ml_ops = { /* Device control ops */ .dev_info_get = cnxk_ml_dev_info_get, @@ -296,8 +441,8 @@ struct rte_ml_dev_ops cnxk_ml_ops = { .dev_selftest = cn10k_ml_dev_selftest, /* Queue-pair handling ops */ - .dev_queue_pair_setup = cn10k_ml_dev_queue_pair_setup, - .dev_queue_pair_release = cn10k_ml_dev_queue_pair_release, + .dev_queue_pair_setup = cnxk_ml_dev_queue_pair_setup, + .dev_queue_pair_release = cnxk_ml_dev_queue_pair_release, /* Stats ops */ .dev_stats_get = cn10k_ml_dev_stats_get, diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h index 2996928d7d0..a925c075809 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.h +++ b/drivers/ml/cnxk/cnxk_ml_ops.h @@ -62,7 +62,4 @@ struct cnxk_ml_qp { extern struct rte_ml_dev_ops cnxk_ml_ops; -/* Temporarily set cnxk driver functions as non-static */ -int cnxk_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info); - #endif /* _CNXK_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130889 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6237D41FD1; Wed, 30 Aug 2023 18:00:55 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5622E402BE; Wed, 30 Aug 2023 17:59:45 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 87666402A3 for ; Wed, 30 Aug 2023 17:59:37 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I6t012554 for ; Wed, 30 Aug 2023 08:59:37 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=VC1S0x5xEBB9OeDhWXLmXkt9A7Utzzd34DnLhSKUtGU=; b=IfmCvKox/0mh1Eww4ILCKSipEw8NSCbl0mdCgP86l5kvXHp6aj22LQTD+BWKVX7Y8hCV AUoSwg9iZAllZIEvWoio5gmJBKyvxWkMosEwJgVgzYzZkNkpR+Pt1N3HQcqCEM8cUl8s Hl+gW1eM2HmrAWHZMjdioZMCCQc+E3hB7bivBFp42DXgVwjS/zJ/ZN0odhzLZ2DSAq6h dBOWt6PPeikJHTgp36BqVAyfMA24/6Rmg8TA0QYKUoNvdOAeZqdM27yTZoGkqbUa1fXz US5CSBOYVYrXfM0ElnDs5g8RD9GusPA5tX+afHdG/7UlO6ZcErtQJUfSaysWhQ879iAG +w== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:36 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:35 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:35 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 209883F707F; Wed, 30 Aug 2023 08:59:35 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 10/34] ml/cnxk: update model load and unload functions Date: Wed, 30 Aug 2023 08:59:00 -0700 Message-ID: <20230830155927.3566-11-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: r3R07PjjVn1QmHD1Aa1hBsE7gLk7rIjA X-Proofpoint-GUID: r3R07PjjVn1QmHD1Aa1hBsE7gLk7rIjA X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implemented cnxk wrapper functions to load and unload ML models. Wrapper functions would invoke the cn10k model load and unload functions. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_model.c | 239 ++++++++++++------------- drivers/ml/cnxk/cn10k_ml_model.h | 25 +-- drivers/ml/cnxk/cn10k_ml_ops.c | 296 ++++++++++++++++++------------- drivers/ml/cnxk/cn10k_ml_ops.h | 12 +- drivers/ml/cnxk/cnxk_ml_dev.h | 15 ++ drivers/ml/cnxk/cnxk_ml_ops.c | 144 ++++++++++++++- drivers/ml/cnxk/cnxk_ml_ops.h | 2 + 7 files changed, 455 insertions(+), 278 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_model.c b/drivers/ml/cnxk/cn10k_ml_model.c index 2a0ae44cfd5..9a336cd18f9 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.c +++ b/drivers/ml/cnxk/cn10k_ml_model.c @@ -6,7 +6,6 @@ #include -#include "cn10k_ml_dev.h" #include "cn10k_ml_model.h" #include "cn10k_ml_ocm.h" @@ -318,42 +317,31 @@ cn10k_ml_layer_addr_update(struct cnxk_ml_layer *layer, uint8_t *buffer, uint8_t { struct cn10k_ml_model_metadata *metadata; struct cn10k_ml_layer_addr *addr; - size_t model_data_size; uint8_t *dma_addr_load; - uint8_t *dma_addr_run; int fpos; metadata = &layer->glow.metadata; addr = &layer->glow.addr; - model_data_size = metadata->init_model.file_size + metadata->main_model.file_size + - metadata->finish_model.file_size + metadata->weights_bias.file_size; /* Base address */ addr->base_dma_addr_load = base_dma_addr; - addr->base_dma_addr_run = PLT_PTR_ADD(addr->base_dma_addr_load, model_data_size); /* Init section */ dma_addr_load = addr->base_dma_addr_load; - dma_addr_run = addr->base_dma_addr_run; fpos = sizeof(struct cn10k_ml_model_metadata); addr->init_load_addr = dma_addr_load; - addr->init_run_addr = dma_addr_run; rte_memcpy(dma_addr_load, PLT_PTR_ADD(buffer, fpos), metadata->init_model.file_size); /* Main section */ dma_addr_load += metadata->init_model.file_size; - dma_addr_run += metadata->init_model.file_size; fpos += metadata->init_model.file_size; addr->main_load_addr = dma_addr_load; - addr->main_run_addr = dma_addr_run; rte_memcpy(dma_addr_load, PLT_PTR_ADD(buffer, fpos), metadata->main_model.file_size); /* Finish section */ dma_addr_load += metadata->main_model.file_size; - dma_addr_run += metadata->main_model.file_size; fpos += metadata->main_model.file_size; addr->finish_load_addr = dma_addr_load; - addr->finish_run_addr = dma_addr_run; rte_memcpy(dma_addr_load, PLT_PTR_ADD(buffer, fpos), metadata->finish_model.file_size); /* Weights and Bias section */ @@ -365,140 +353,140 @@ cn10k_ml_layer_addr_update(struct cnxk_ml_layer *layer, uint8_t *buffer, uint8_t } void -cn10k_ml_layer_info_update(struct cnxk_ml_layer *layer) +cn10k_ml_layer_io_info_update(struct cnxk_ml_io_info *io_info, + struct cn10k_ml_model_metadata *metadata) { - struct cn10k_ml_model_metadata *metadata; uint8_t i; uint8_t j; - metadata = &layer->glow.metadata; - /* Inputs */ - layer->info.nb_inputs = metadata->model.num_input; - layer->info.total_input_sz_d = 0; - layer->info.total_input_sz_q = 0; + io_info->nb_inputs = metadata->model.num_input; + io_info->total_input_sz_d = 0; + io_info->total_input_sz_q = 0; for (i = 0; i < metadata->model.num_input; i++) { if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - strncpy(layer->info.input[i].name, (char *)metadata->input1[i].input_name, + strncpy(io_info->input[i].name, (char *)metadata->input1[i].input_name, MRVL_ML_INPUT_NAME_LEN); - layer->info.input[i].dtype = metadata->input1[i].input_type; - layer->info.input[i].qtype = metadata->input1[i].model_input_type; - layer->info.input[i].nb_dims = 4; - layer->info.input[i].shape[0] = metadata->input1[i].shape.w; - layer->info.input[i].shape[1] = metadata->input1[i].shape.x; - layer->info.input[i].shape[2] = metadata->input1[i].shape.y; - layer->info.input[i].shape[3] = metadata->input1[i].shape.z; - layer->info.input[i].nb_elements = + io_info->input[i].dtype = metadata->input1[i].input_type; + io_info->input[i].qtype = metadata->input1[i].model_input_type; + io_info->input[i].nb_dims = 4; + io_info->input[i].shape[0] = metadata->input1[i].shape.w; + io_info->input[i].shape[1] = metadata->input1[i].shape.x; + io_info->input[i].shape[2] = metadata->input1[i].shape.y; + io_info->input[i].shape[3] = metadata->input1[i].shape.z; + io_info->input[i].nb_elements = metadata->input1[i].shape.w * metadata->input1[i].shape.x * metadata->input1[i].shape.y * metadata->input1[i].shape.z; - layer->info.input[i].sz_d = - layer->info.input[i].nb_elements * + io_info->input[i].sz_d = + io_info->input[i].nb_elements * rte_ml_io_type_size_get(metadata->input1[i].input_type); - layer->info.input[i].sz_q = - layer->info.input[i].nb_elements * + io_info->input[i].sz_q = + io_info->input[i].nb_elements * rte_ml_io_type_size_get(metadata->input1[i].model_input_type); - layer->info.input[i].scale = metadata->input1[i].qscale; + io_info->input[i].scale = metadata->input1[i].qscale; - layer->info.total_input_sz_d += layer->info.input[i].sz_d; - layer->info.total_input_sz_q += layer->info.input[i].sz_q; + io_info->total_input_sz_d += io_info->input[i].sz_d; + io_info->total_input_sz_q += io_info->input[i].sz_q; plt_ml_dbg( - "index = %u, input1[%u] - w:%u x:%u y:%u z:%u, sz_d = %u sz_q = %u", - layer->index, i, metadata->input1[i].shape.w, + "layer_name = %s, input1[%u] - w:%u x:%u y:%u z:%u, sz_d = %u sz_q = %u", + metadata->model.name, i, metadata->input1[i].shape.w, metadata->input1[i].shape.x, metadata->input1[i].shape.y, - metadata->input1[i].shape.z, layer->info.input[i].sz_d, - layer->info.input[i].sz_q); + metadata->input1[i].shape.z, io_info->input[i].sz_d, + io_info->input[i].sz_q); } else { j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - strncpy(layer->info.input[i].name, (char *)metadata->input2[j].input_name, + strncpy(io_info->input[i].name, (char *)metadata->input2[j].input_name, MRVL_ML_INPUT_NAME_LEN); - layer->info.input[i].dtype = metadata->input2[j].input_type; - layer->info.input[i].qtype = metadata->input2[j].model_input_type; - layer->info.input[i].nb_dims = 4; - layer->info.input[i].shape[0] = metadata->input2[j].shape.w; - layer->info.input[i].shape[1] = metadata->input2[j].shape.x; - layer->info.input[i].shape[2] = metadata->input2[j].shape.y; - layer->info.input[i].shape[3] = metadata->input2[j].shape.z; - layer->info.input[i].nb_elements = + io_info->input[i].dtype = metadata->input2[j].input_type; + io_info->input[i].qtype = metadata->input2[j].model_input_type; + io_info->input[i].nb_dims = 4; + io_info->input[i].shape[0] = metadata->input2[j].shape.w; + io_info->input[i].shape[1] = metadata->input2[j].shape.x; + io_info->input[i].shape[2] = metadata->input2[j].shape.y; + io_info->input[i].shape[3] = metadata->input2[j].shape.z; + io_info->input[i].nb_elements = metadata->input2[j].shape.w * metadata->input2[j].shape.x * metadata->input2[j].shape.y * metadata->input2[j].shape.z; - layer->info.input[i].sz_d = - layer->info.input[i].nb_elements * + io_info->input[i].sz_d = + io_info->input[i].nb_elements * rte_ml_io_type_size_get(metadata->input2[j].input_type); - layer->info.input[i].sz_q = - layer->info.input[i].nb_elements * + io_info->input[i].sz_q = + io_info->input[i].nb_elements * rte_ml_io_type_size_get(metadata->input2[j].model_input_type); - layer->info.input[i].scale = metadata->input2[j].qscale; + io_info->input[i].scale = metadata->input2[j].qscale; - layer->info.total_input_sz_d += layer->info.input[i].sz_d; - layer->info.total_input_sz_q += layer->info.input[i].sz_q; + io_info->total_input_sz_d += io_info->input[i].sz_d; + io_info->total_input_sz_q += io_info->input[i].sz_q; plt_ml_dbg( - "index = %u, input2[%u] - w:%u x:%u y:%u z:%u, sz_d = %u sz_q = %u", - layer->index, j, metadata->input2[j].shape.w, + "layer_name = %s, input2[%u] - w:%u x:%u y:%u z:%u, sz_d = %u sz_q = %u", + metadata->model.name, j, metadata->input2[j].shape.w, metadata->input2[j].shape.x, metadata->input2[j].shape.y, - metadata->input2[j].shape.z, layer->info.input[i].sz_d, - layer->info.input[i].sz_q); + metadata->input2[j].shape.z, io_info->input[i].sz_d, + io_info->input[i].sz_q); } } /* Outputs */ - layer->info.nb_outputs = metadata->model.num_output; - layer->info.total_output_sz_q = 0; - layer->info.total_output_sz_d = 0; + io_info->nb_outputs = metadata->model.num_output; + io_info->total_output_sz_q = 0; + io_info->total_output_sz_d = 0; for (i = 0; i < metadata->model.num_output; i++) { if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - strncpy(layer->info.output[i].name, - (char *)metadata->output1[i].output_name, MRVL_ML_OUTPUT_NAME_LEN); - layer->info.output[i].dtype = metadata->output1[i].output_type; - layer->info.output[i].qtype = metadata->output1[i].model_output_type; - layer->info.output[i].nb_dims = 1; - layer->info.output[i].shape[0] = metadata->output1[i].size; - layer->info.output[i].nb_elements = metadata->output1[i].size; - layer->info.output[i].sz_d = - layer->info.output[i].nb_elements * + strncpy(io_info->output[i].name, (char *)metadata->output1[i].output_name, + MRVL_ML_OUTPUT_NAME_LEN); + io_info->output[i].dtype = metadata->output1[i].output_type; + io_info->output[i].qtype = metadata->output1[i].model_output_type; + io_info->output[i].nb_dims = 1; + io_info->output[i].shape[0] = metadata->output1[i].size; + io_info->output[i].nb_elements = metadata->output1[i].size; + io_info->output[i].sz_d = + io_info->output[i].nb_elements * rte_ml_io_type_size_get(metadata->output1[i].output_type); - layer->info.output[i].sz_q = - layer->info.output[i].nb_elements * + io_info->output[i].sz_q = + io_info->output[i].nb_elements * rte_ml_io_type_size_get(metadata->output1[i].model_output_type); - layer->info.output[i].scale = metadata->output1[i].dscale; + io_info->output[i].scale = metadata->output1[i].dscale; - layer->info.total_output_sz_q += layer->info.output[i].sz_q; - layer->info.total_output_sz_d += layer->info.output[i].sz_d; + io_info->total_output_sz_q += io_info->output[i].sz_q; + io_info->total_output_sz_d += io_info->output[i].sz_d; - plt_ml_dbg("index = %u, output1[%u] - sz_d = %u, sz_q = %u", layer->index, - i, layer->info.output[i].sz_d, layer->info.output[i].sz_q); + plt_ml_dbg("layer_name = %s, output1[%u] - sz_d = %u, sz_q = %u", + metadata->model.name, i, io_info->output[i].sz_d, + io_info->output[i].sz_q); } else { j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - strncpy(layer->info.output[i].name, - (char *)metadata->output2[j].output_name, MRVL_ML_OUTPUT_NAME_LEN); - layer->info.output[i].dtype = metadata->output2[j].output_type; - layer->info.output[i].qtype = metadata->output2[j].model_output_type; - layer->info.output[i].nb_dims = 1; - layer->info.output[i].shape[0] = metadata->output2[j].size; - layer->info.output[i].nb_elements = metadata->output2[j].size; - layer->info.output[i].sz_d = - layer->info.output[i].nb_elements * + strncpy(io_info->output[i].name, (char *)metadata->output2[j].output_name, + MRVL_ML_OUTPUT_NAME_LEN); + io_info->output[i].dtype = metadata->output2[j].output_type; + io_info->output[i].qtype = metadata->output2[j].model_output_type; + io_info->output[i].nb_dims = 1; + io_info->output[i].shape[0] = metadata->output2[j].size; + io_info->output[i].nb_elements = metadata->output2[j].size; + io_info->output[i].sz_d = + io_info->output[i].nb_elements * rte_ml_io_type_size_get(metadata->output2[j].output_type); - layer->info.output[i].sz_q = - layer->info.output[i].nb_elements * + io_info->output[i].sz_q = + io_info->output[i].nb_elements * rte_ml_io_type_size_get(metadata->output2[j].model_output_type); - layer->info.output[i].scale = metadata->output2[j].dscale; + io_info->output[i].scale = metadata->output2[j].dscale; - layer->info.total_output_sz_q += layer->info.output[i].sz_q; - layer->info.total_output_sz_d += layer->info.output[i].sz_d; + io_info->total_output_sz_q += io_info->output[i].sz_q; + io_info->total_output_sz_d += io_info->output[i].sz_d; - plt_ml_dbg("index = %u, output2[%u] - sz_d = %u, sz_q = %u", layer->index, - j, layer->info.output[i].sz_d, layer->info.output[i].sz_q); + plt_ml_dbg("layer_name = %s, output2[%u] - sz_d = %u, sz_q = %u", + metadata->model.name, j, io_info->output[i].sz_d, + io_info->output[i].sz_q); } } } int -cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_id, uint8_t *buffer, - uint16_t *wb_pages, uint16_t *scratch_pages) +cn10k_ml_model_ocm_pages_count(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, + uint8_t *buffer, uint16_t *wb_pages, uint16_t *scratch_pages) { struct cn10k_ml_model_metadata *metadata; struct cn10k_ml_ocm *ocm; @@ -506,7 +494,7 @@ cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_ uint64_t wb_size; metadata = (struct cn10k_ml_model_metadata *)buffer; - ocm = &cn10k_mldev->ocm; + ocm = &cnxk_mldev->cn10k_mldev.ocm; /* Assume wb_size is zero for non-relocatable models */ if (metadata->model.ocm_relocatable) @@ -518,7 +506,7 @@ cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_ *wb_pages = wb_size / ocm->page_size + 1; else *wb_pages = wb_size / ocm->page_size; - plt_ml_dbg("model_id = %u, wb_size = %" PRIu64 ", wb_pages = %u", model_id, wb_size, + plt_ml_dbg("index = %u, wb_size = %" PRIu64 ", wb_pages = %u", layer->index, wb_size, *wb_pages); scratch_size = ocm->size_per_tile - metadata->model.ocm_tmp_range_floor; @@ -526,15 +514,15 @@ cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_ *scratch_pages = scratch_size / ocm->page_size + 1; else *scratch_pages = scratch_size / ocm->page_size; - plt_ml_dbg("model_id = %u, scratch_size = %" PRIu64 ", scratch_pages = %u", model_id, + plt_ml_dbg("index = %u, scratch_size = %" PRIu64 ", scratch_pages = %u", layer->index, scratch_size, *scratch_pages); /* Check if the model can be loaded on OCM */ - if ((*wb_pages + *scratch_pages) > cn10k_mldev->ocm.num_pages) { + if ((*wb_pages + *scratch_pages) > ocm->num_pages) { plt_err("Cannot create the model, OCM relocatable = %u", metadata->model.ocm_relocatable); plt_err("wb_pages (%u) + scratch_pages (%u) > %u", *wb_pages, *scratch_pages, - cn10k_mldev->ocm.num_pages); + ocm->num_pages); return -ENOMEM; } @@ -542,28 +530,25 @@ cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_ * prevent the library from allocating the remaining space on the tile to other models. */ if (!metadata->model.ocm_relocatable) - *scratch_pages = PLT_MAX(PLT_U64_CAST(*scratch_pages), - PLT_U64_CAST(cn10k_mldev->ocm.num_pages)); + *scratch_pages = + PLT_MAX(PLT_U64_CAST(*scratch_pages), PLT_U64_CAST(ocm->num_pages)); return 0; } void -cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cnxk_ml_model *model) +cn10k_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, + struct cnxk_ml_io_info *io_info, struct cn10k_ml_model_metadata *metadata) { - struct cn10k_ml_model_metadata *metadata; - struct cnxk_ml_dev *cnxk_mldev; struct rte_ml_model_info *info; struct rte_ml_io_info *output; struct rte_ml_io_info *input; - struct cnxk_ml_layer *layer; uint8_t i; - cnxk_mldev = dev->data->dev_private; metadata = &model->glow.metadata; info = PLT_PTR_CAST(model->info); input = PLT_PTR_ADD(info, sizeof(struct rte_ml_model_info)); - output = PLT_PTR_ADD(input, metadata->model.num_input * sizeof(struct rte_ml_io_info)); + output = PLT_PTR_ADD(input, ML_CNXK_MODEL_MAX_INPUT_OUTPUT * sizeof(struct rte_ml_io_info)); /* Set model info */ memset(info, 0, sizeof(struct rte_ml_model_info)); @@ -572,39 +557,37 @@ cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cnxk_ml_model *model) metadata->model.version[1], metadata->model.version[2], metadata->model.version[3]); info->model_id = model->model_id; - info->device_id = dev->data->dev_id; + info->device_id = cnxk_mldev->mldev->data->dev_id; info->io_layout = RTE_ML_IO_LAYOUT_PACKED; info->min_batches = model->batch_size; info->max_batches = cnxk_mldev->cn10k_mldev.fw.req->cn10k_req.jd.fw_load.cap.s.max_num_batches / model->batch_size; - info->nb_inputs = metadata->model.num_input; + info->nb_inputs = io_info->nb_inputs; info->input_info = input; - info->nb_outputs = metadata->model.num_output; + info->nb_outputs = io_info->nb_outputs; info->output_info = output; info->wb_size = metadata->weights_bias.file_size; /* Set input info */ - layer = &model->layer[0]; for (i = 0; i < info->nb_inputs; i++) { - rte_memcpy(input[i].name, layer->info.input[i].name, MRVL_ML_INPUT_NAME_LEN); - input[i].nb_dims = layer->info.input[i].nb_dims; - input[i].shape = &layer->info.input[i].shape[0]; - input[i].type = layer->info.input[i].qtype; - input[i].nb_elements = layer->info.input[i].nb_elements; - input[i].size = layer->info.input[i].nb_elements * - rte_ml_io_type_size_get(layer->info.input[i].qtype); + rte_memcpy(input[i].name, io_info->input[i].name, MRVL_ML_INPUT_NAME_LEN); + input[i].nb_dims = io_info->input[i].nb_dims; + input[i].shape = &io_info->input[i].shape[0]; + input[i].type = io_info->input[i].qtype; + input[i].nb_elements = io_info->input[i].nb_elements; + input[i].size = io_info->input[i].nb_elements * + rte_ml_io_type_size_get(io_info->input[i].qtype); } /* Set output info */ - layer = &model->layer[0]; for (i = 0; i < info->nb_outputs; i++) { - rte_memcpy(output[i].name, layer->info.output[i].name, MRVL_ML_INPUT_NAME_LEN); - output[i].nb_dims = layer->info.output[i].nb_dims; - output[i].shape = &layer->info.output[i].shape[0]; - output[i].type = layer->info.output[i].qtype; - output[i].nb_elements = layer->info.output[i].nb_elements; - output[i].size = layer->info.output[i].nb_elements * - rte_ml_io_type_size_get(layer->info.output[i].qtype); + rte_memcpy(output[i].name, io_info->output[i].name, MRVL_ML_INPUT_NAME_LEN); + output[i].nb_dims = io_info->output[i].nb_dims; + output[i].shape = &io_info->output[i].shape[0]; + output[i].type = io_info->output[i].qtype; + output[i].nb_elements = io_info->output[i].nb_elements; + output[i].size = io_info->output[i].nb_elements * + rte_ml_io_type_size_get(io_info->output[i].qtype); } } diff --git a/drivers/ml/cnxk/cn10k_ml_model.h b/drivers/ml/cnxk/cn10k_ml_model.h index 5c32f48c68f..45290b84cef 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.h +++ b/drivers/ml/cnxk/cn10k_ml_model.h @@ -9,9 +9,11 @@ #include -#include "cn10k_ml_dev.h" #include "cn10k_ml_ocm.h" +#include "cnxk_ml_io.h" + +struct cnxk_ml_dev; struct cnxk_ml_model; struct cnxk_ml_layer; struct cnxk_ml_req; @@ -366,27 +368,15 @@ struct cn10k_ml_layer_addr { /* Base DMA address for load */ void *base_dma_addr_load; - /* Base DMA address for run */ - void *base_dma_addr_run; - /* Init section load address */ void *init_load_addr; - /* Init section run address */ - void *init_run_addr; - /* Main section load address */ void *main_load_addr; - /* Main section run address */ - void *main_run_addr; - /* Finish section load address */ void *finish_load_addr; - /* Finish section run address */ - void *finish_run_addr; - /* Weights and Bias base address */ void *wb_base_addr; @@ -462,9 +452,12 @@ int cn10k_ml_model_metadata_check(uint8_t *buffer, uint64_t size); void cn10k_ml_model_metadata_update(struct cn10k_ml_model_metadata *metadata); void cn10k_ml_layer_addr_update(struct cnxk_ml_layer *layer, uint8_t *buffer, uint8_t *base_dma_addr); -void cn10k_ml_layer_info_update(struct cnxk_ml_layer *layer); -int cn10k_ml_model_ocm_pages_count(struct cn10k_ml_dev *cn10k_mldev, uint16_t model_id, +void cn10k_ml_layer_io_info_update(struct cnxk_ml_io_info *io_info, + struct cn10k_ml_model_metadata *metadata); +int cn10k_ml_model_ocm_pages_count(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, uint8_t *buffer, uint16_t *wb_pages, uint16_t *scratch_pages); -void cn10k_ml_model_info_set(struct rte_ml_dev *dev, struct cnxk_ml_model *model); +void cn10k_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, + struct cnxk_ml_io_info *io_info, + struct cn10k_ml_model_metadata *metadata); #endif /* _CN10K_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 330cb050cbd..3bfc63d9d40 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -19,6 +19,9 @@ /* ML model macros */ #define CN10K_ML_MODEL_MEMZONE_NAME "ml_cn10k_model_mz" +/* ML layer macros */ +#define CN10K_ML_LAYER_MEMZONE_NAME "ml_cn10k_layer_mz" + /* Debug print width */ #define STR_LEN 12 #define FIELD_LEN 16 @@ -277,7 +280,7 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml req->cn10k_req.jd.model_start.extended_args = PLT_U64_CAST( roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->cn10k_req.extended_args)); req->cn10k_req.jd.model_start.model_dst_ddr_addr = - PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, addr->init_run_addr)); + PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, addr->init_load_addr)); req->cn10k_req.jd.model_start.model_init_offset = 0x0; req->cn10k_req.jd.model_start.model_main_offset = metadata->init_model.file_size; req->cn10k_req.jd.model_start.model_finish_offset = @@ -1265,85 +1268,171 @@ cn10k_ml_dev_selftest(struct rte_ml_dev *dev) } int -cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, uint16_t *model_id) +cn10k_ml_layer_load(void *device, uint16_t model_id, const char *layer_name, uint8_t *buffer, + size_t size, uint16_t *index) { struct cn10k_ml_model_metadata *metadata; struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; char str[RTE_MEMZONE_NAMESIZE]; const struct plt_memzone *mz; - size_t model_scratch_size; - size_t model_stats_size; - size_t model_data_size; - size_t model_info_size; + size_t layer_object_size = 0; + size_t layer_scratch_size; + size_t layer_xstats_size; uint8_t *base_dma_addr; uint16_t scratch_pages; + uint16_t layer_id = 0; uint16_t wb_pages; uint64_t mz_size; uint16_t idx; - bool found; int qp_id; int ret; - ret = cn10k_ml_model_metadata_check(params->addr, params->size); + PLT_SET_USED(size); + PLT_SET_USED(layer_name); + + cnxk_mldev = (struct cnxk_ml_dev *)device; + if (cnxk_mldev == NULL) { + plt_err("Invalid device = %p", device); + return -EINVAL; + } + + model = cnxk_mldev->mldev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + + layer = &model->layer[layer_id]; + + ret = cn10k_ml_model_metadata_check(buffer, size); if (ret != 0) return ret; - cnxk_mldev = dev->data->dev_private; - - /* Find model ID */ - found = false; - for (idx = 0; idx < dev->data->nb_models; idx++) { - if (dev->data->models[idx] == NULL) { - found = true; + /* Get index */ + for (idx = 0; idx < cnxk_mldev->max_nb_layers; idx++) { + if (!cnxk_mldev->index_map[idx].active) { + layer->index = idx; break; } } - if (!found) { - plt_err("No slots available to load new model"); - return -ENOMEM; + if (idx >= cnxk_mldev->max_nb_layers) { + plt_err("No slots available for model layers, model_id = %u, layer_id = %u", + model->model_id, layer_id); + return -1; } + layer->model = model; + /* Get WB and scratch pages, check if model can be loaded. */ - ret = cn10k_ml_model_ocm_pages_count(&cnxk_mldev->cn10k_mldev, idx, params->addr, &wb_pages, - &scratch_pages); + ret = cn10k_ml_model_ocm_pages_count(cnxk_mldev, layer, buffer, &wb_pages, &scratch_pages); if (ret < 0) return ret; - /* Compute memzone size */ - metadata = (struct cn10k_ml_model_metadata *)params->addr; - model_data_size = metadata->init_model.file_size + metadata->main_model.file_size + - metadata->finish_model.file_size + metadata->weights_bias.file_size; - model_scratch_size = PLT_ALIGN_CEIL(metadata->model.ddr_scratch_range_end - + /* Compute layer memzone size */ + metadata = (struct cn10k_ml_model_metadata *)buffer; + layer_object_size = metadata->init_model.file_size + metadata->main_model.file_size + + metadata->finish_model.file_size + metadata->weights_bias.file_size; + layer_object_size = PLT_ALIGN_CEIL(layer_object_size, ML_CN10K_ALIGN_SIZE); + layer_scratch_size = PLT_ALIGN_CEIL(metadata->model.ddr_scratch_range_end - metadata->model.ddr_scratch_range_start + 1, ML_CN10K_ALIGN_SIZE); - model_data_size = PLT_ALIGN_CEIL(model_data_size, ML_CN10K_ALIGN_SIZE); - model_info_size = sizeof(struct rte_ml_model_info) + - metadata->model.num_input * sizeof(struct rte_ml_io_info) + - metadata->model.num_output * sizeof(struct rte_ml_io_info); - model_info_size = PLT_ALIGN_CEIL(model_info_size, ML_CN10K_ALIGN_SIZE); - model_stats_size = (dev->data->nb_queue_pairs + 1) * sizeof(struct cn10k_ml_layer_xstats); - - mz_size = PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), ML_CN10K_ALIGN_SIZE) + - 2 * model_data_size + model_scratch_size + model_info_size + - PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_req), ML_CN10K_ALIGN_SIZE) + - model_stats_size; + layer_xstats_size = (cnxk_mldev->mldev->data->nb_queue_pairs + 1) * + sizeof(struct cn10k_ml_layer_xstats); - /* Allocate memzone for model object and model data */ - snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", CN10K_ML_MODEL_MEMZONE_NAME, idx); + /* Allocate memzone for model data */ + mz_size = layer_object_size + layer_scratch_size + + PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_req), ML_CN10K_ALIGN_SIZE) + + layer_xstats_size; + snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u_%u", CN10K_ML_LAYER_MEMZONE_NAME, + model->model_id, layer_id); mz = plt_memzone_reserve_aligned(str, mz_size, 0, ML_CN10K_ALIGN_SIZE); if (!mz) { plt_err("plt_memzone_reserve failed : %s", str); return -ENOMEM; } - model = mz->addr; - model->cnxk_mldev = cnxk_mldev; - model->model_id = idx; - dev->data->models[idx] = model; + /* Copy metadata to internal buffer */ + rte_memcpy(&layer->glow.metadata, buffer, sizeof(struct cn10k_ml_model_metadata)); + cn10k_ml_model_metadata_update(&layer->glow.metadata); + + /* Set layer name */ + rte_memcpy(layer->name, layer->glow.metadata.model.name, MRVL_ML_MODEL_NAME_LEN); + + /* Enable support for batch_size of 256 */ + if (layer->glow.metadata.model.batch_size == 0) + layer->batch_size = 256; + else + layer->batch_size = layer->glow.metadata.model.batch_size; + + /* Set DMA base address */ + base_dma_addr = mz->addr; + cn10k_ml_layer_addr_update(layer, buffer, base_dma_addr); + + /* Set scratch base address */ + layer->glow.addr.scratch_base_addr = PLT_PTR_ADD(base_dma_addr, layer_object_size); + + /* Update internal I/O data structure */ + cn10k_ml_layer_io_info_update(&layer->info, &layer->glow.metadata); + + /* Initialize model_mem_map */ + memset(&layer->glow.ocm_map, 0, sizeof(struct cn10k_ml_ocm_layer_map)); + layer->glow.ocm_map.ocm_reserved = false; + layer->glow.ocm_map.tilemask = 0; + layer->glow.ocm_map.wb_page_start = -1; + layer->glow.ocm_map.wb_pages = wb_pages; + layer->glow.ocm_map.scratch_pages = scratch_pages; + + /* Set slow-path request address and state */ + layer->glow.req = PLT_PTR_ADD(mz->addr, layer_object_size + layer_scratch_size); + + /* Reset burst and sync stats */ + layer->glow.burst_xstats = PLT_PTR_ADD( + layer->glow.req, PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_req), ML_CN10K_ALIGN_SIZE)); + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs + 1; qp_id++) { + layer->glow.burst_xstats[qp_id].hw_latency_tot = 0; + layer->glow.burst_xstats[qp_id].hw_latency_min = UINT64_MAX; + layer->glow.burst_xstats[qp_id].hw_latency_max = 0; + layer->glow.burst_xstats[qp_id].fw_latency_tot = 0; + layer->glow.burst_xstats[qp_id].fw_latency_min = UINT64_MAX; + layer->glow.burst_xstats[qp_id].fw_latency_max = 0; + layer->glow.burst_xstats[qp_id].hw_reset_count = 0; + layer->glow.burst_xstats[qp_id].fw_reset_count = 0; + layer->glow.burst_xstats[qp_id].dequeued_count = 0; + } + + layer->glow.sync_xstats = + PLT_PTR_ADD(layer->glow.burst_xstats, cnxk_mldev->mldev->data->nb_queue_pairs * + sizeof(struct cn10k_ml_layer_xstats)); + + /* Update xstats names */ + cn10k_ml_xstats_model_name_update(cnxk_mldev->mldev, idx); + + layer->state = ML_CNXK_LAYER_STATE_LOADED; + cnxk_mldev->index_map[idx].model_id = model->model_id; + cnxk_mldev->index_map[idx].layer_id = layer_id; + cnxk_mldev->index_map[idx].active = true; + *index = idx; + + return 0; +} + +int +cn10k_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, + struct cnxk_ml_model *model) +{ + struct cnxk_ml_layer *layer; + int ret; + + /* Metadata check */ + ret = cn10k_ml_model_metadata_check(params->addr, params->size); + if (ret != 0) + return ret; + /* Copy metadata to internal buffer */ rte_memcpy(&model->glow.metadata, params->addr, sizeof(struct cn10k_ml_model_metadata)); cn10k_ml_model_metadata_update(&model->glow.metadata); @@ -1362,99 +1451,62 @@ cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, */ model->nb_layers = 1; - /* Copy metadata to internal buffer */ - rte_memcpy(&model->layer[0].glow.metadata, params->addr, - sizeof(struct cn10k_ml_model_metadata)); - cn10k_ml_model_metadata_update(&model->layer[0].glow.metadata); - model->layer[0].model = model; - - /* Set DMA base address */ - base_dma_addr = PLT_PTR_ADD( - mz->addr, PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), ML_CN10K_ALIGN_SIZE)); - cn10k_ml_layer_addr_update(&model->layer[0], params->addr, base_dma_addr); - model->layer[0].glow.addr.scratch_base_addr = - PLT_PTR_ADD(base_dma_addr, 2 * model_data_size); - - /* Copy data from load to run. run address to be used by MLIP */ - rte_memcpy(model->layer[0].glow.addr.base_dma_addr_run, - model->layer[0].glow.addr.base_dma_addr_load, model_data_size); - - /* Update internal I/O data structure */ - cn10k_ml_layer_info_update(&model->layer[0]); - - /* Initialize model_mem_map */ - memset(&model->layer[0].glow.ocm_map, 0, sizeof(struct cn10k_ml_ocm_layer_map)); - model->layer[0].glow.ocm_map.ocm_reserved = false; - model->layer[0].glow.ocm_map.tilemask = 0; - model->layer[0].glow.ocm_map.wb_page_start = -1; - model->layer[0].glow.ocm_map.wb_pages = wb_pages; - model->layer[0].glow.ocm_map.scratch_pages = scratch_pages; - - /* Set model info */ - model->info = PLT_PTR_ADD(model->layer[0].glow.addr.scratch_base_addr, model_scratch_size); - cn10k_ml_model_info_set(dev, model); - - /* Set slow-path request address and state */ - model->layer[0].glow.req = PLT_PTR_ADD(model->info, model_info_size); - - /* Reset burst and sync stats */ - model->layer[0].glow.burst_xstats = - PLT_PTR_ADD(model->layer[0].glow.req, - PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_req), ML_CN10K_ALIGN_SIZE)); - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs + 1; qp_id++) { - model->layer[0].glow.burst_xstats[qp_id].hw_latency_tot = 0; - model->layer[0].glow.burst_xstats[qp_id].hw_latency_min = UINT64_MAX; - model->layer[0].glow.burst_xstats[qp_id].hw_latency_max = 0; - model->layer[0].glow.burst_xstats[qp_id].fw_latency_tot = 0; - model->layer[0].glow.burst_xstats[qp_id].fw_latency_min = UINT64_MAX; - model->layer[0].glow.burst_xstats[qp_id].fw_latency_max = 0; - model->layer[0].glow.burst_xstats[qp_id].hw_reset_count = 0; - model->layer[0].glow.burst_xstats[qp_id].fw_reset_count = 0; - model->layer[0].glow.burst_xstats[qp_id].dequeued_count = 0; + /* Load layer and get the index */ + layer = &model->layer[0]; + ret = cn10k_ml_layer_load(cnxk_mldev, model->model_id, NULL, params->addr, params->size, + &layer->index); + if (ret != 0) { + plt_err("Model layer load failed: model_id = %u, layer_id = %u", model->model_id, + 0); + return ret; } - model->layer[0].glow.sync_xstats = - PLT_PTR_ADD(model->layer[0].glow.burst_xstats, - dev->data->nb_queue_pairs * sizeof(struct cn10k_ml_layer_xstats)); - - plt_spinlock_init(&model->lock); - model->state = ML_CNXK_MODEL_STATE_LOADED; - dev->data->models[idx] = model; - cnxk_mldev->nb_models_loaded++; - - /* Update xstats names */ - cn10k_ml_xstats_model_name_update(dev, idx); - - *model_id = idx; + cn10k_ml_model_info_set(cnxk_mldev, model, &model->layer[0].info, &model->glow.metadata); return 0; } int -cn10k_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) +cn10k_ml_layer_unload(void *device, uint16_t model_id, const char *layer_name) { - char str[RTE_MEMZONE_NAMESIZE]; - struct cnxk_ml_model *model; struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; - cnxk_mldev = dev->data->dev_private; - model = dev->data->models[model_id]; + char str[RTE_MEMZONE_NAMESIZE]; + uint16_t layer_id = 0; + int ret; + PLT_SET_USED(layer_name); + + cnxk_mldev = (struct cnxk_ml_dev *)device; + if (cnxk_mldev == NULL) { + plt_err("Invalid device = %p", device); + return -EINVAL; + } + + model = cnxk_mldev->mldev->data->models[model_id]; if (model == NULL) { plt_err("Invalid model_id = %u", model_id); return -EINVAL; } - if (model->state != ML_CNXK_MODEL_STATE_LOADED) { - plt_err("Cannot unload. Model in use."); - return -EBUSY; - } + layer = &model->layer[layer_id]; - dev->data->models[model_id] = NULL; - cnxk_mldev->nb_models_unloaded++; + snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u_%u", CN10K_ML_LAYER_MEMZONE_NAME, + model->model_id, layer_id); + ret = plt_memzone_free(plt_memzone_lookup(str)); - snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", CN10K_ML_MODEL_MEMZONE_NAME, model_id); - return plt_memzone_free(plt_memzone_lookup(str)); + layer->state = ML_CNXK_LAYER_STATE_UNKNOWN; + cnxk_mldev->index_map[layer->index].active = false; + + return ret; +} + +int +cn10k_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model) +{ + return cn10k_ml_layer_unload(cnxk_mldev, model->model_id, NULL); } int @@ -1752,7 +1804,6 @@ int cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buffer) { struct cnxk_ml_model *model; - size_t size; model = dev->data->models[model_id]; @@ -1766,19 +1817,10 @@ cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *bu else if (model->state != ML_CNXK_MODEL_STATE_LOADED) return -EBUSY; - size = model->layer[0].glow.metadata.init_model.file_size + - model->layer[0].glow.metadata.main_model.file_size + - model->layer[0].glow.metadata.finish_model.file_size + - model->layer[0].glow.metadata.weights_bias.file_size; - /* Update model weights & bias */ rte_memcpy(model->layer[0].glow.addr.wb_load_addr, buffer, model->layer[0].glow.metadata.weights_bias.file_size); - /* Copy data from load to run. run address to be used by MLIP */ - rte_memcpy(model->layer[0].glow.addr.base_dma_addr_run, - model->layer[0].glow.addr.base_dma_addr_load, size); - return 0; } diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 2d0a49d5cdf..677219dfdf7 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -12,6 +12,7 @@ struct cnxk_ml_dev; struct cnxk_ml_qp; +struct cnxk_ml_model; /* Firmware version string length */ #define MLDEV_FIRMWARE_VERSION_LENGTH 32 @@ -311,9 +312,9 @@ int cn10k_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mod int32_t model_id, const uint16_t stat_ids[], uint16_t nb_ids); /* Slow-path ops */ -int cn10k_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, - uint16_t *model_id); -int cn10k_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id); +int cn10k_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, + struct cnxk_ml_model *model); +int cn10k_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); int cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id); int cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id); int cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, @@ -339,4 +340,9 @@ __rte_hot int cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op * /* Misc ops */ void cn10k_ml_qp_initialize(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_qp *qp); +/* Layer ops */ +int cn10k_ml_layer_load(void *device, uint16_t model_id, const char *layer_name, uint8_t *buffer, + size_t size, uint16_t *index); +int cn10k_ml_layer_unload(void *device, uint16_t model_id, const char *layer_name); + #endif /* _CN10K_ML_OPS_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_dev.h b/drivers/ml/cnxk/cnxk_ml_dev.h index 02605fa28fc..1590249abd1 100644 --- a/drivers/ml/cnxk/cnxk_ml_dev.h +++ b/drivers/ml/cnxk/cnxk_ml_dev.h @@ -31,6 +31,18 @@ enum cnxk_ml_dev_state { ML_CNXK_DEV_STATE_CLOSED }; +/* Index to model and layer ID map */ +struct cnxk_ml_index_map { + /* Model ID */ + uint16_t model_id; + + /* Layer ID */ + uint16_t layer_id; + + /* Layer status */ + bool active; +}; + /* Device private data */ struct cnxk_ml_dev { /* RTE device */ @@ -56,6 +68,9 @@ struct cnxk_ml_dev { /* Maximum number of layers */ uint64_t max_nb_layers; + + /* Index map */ + struct cnxk_ml_index_map *index_map; }; #endif /* _CNXK_ML_DEV_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 1767a8a3dbc..3d9d5f9d78c 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -12,6 +12,9 @@ #include "cnxk_ml_model.h" #include "cnxk_ml_ops.h" +/* ML model macros */ +#define CNXK_ML_MODEL_MEMZONE_NAME "ml_cnxk_model_mz" + static void qp_memzone_name_get(char *name, int size, int dev_id, int qp_id) { @@ -139,6 +142,7 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co uint16_t model_id; uint32_t mz_size; uint16_t qp_id; + uint64_t i; int ret; if (dev == NULL) @@ -242,7 +246,7 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co plt_err("Could not stop model %u", model_id); } if (model->state == ML_CNXK_MODEL_STATE_LOADED) { - if (cn10k_ml_model_unload(dev, model_id) != 0) + if (cnxk_ml_model_unload(dev, model_id) != 0) plt_err("Could not unload model %u", model_id); } dev->data->models[model_id] = NULL; @@ -273,6 +277,23 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co cnxk_mldev->max_nb_layers = cnxk_mldev->cn10k_mldev.fw.req->cn10k_req.jd.fw_load.cap.s.max_models; + /* Allocate and initialize index_map */ + if (cnxk_mldev->index_map == NULL) { + cnxk_mldev->index_map = + rte_zmalloc("cnxk_ml_index_map", + sizeof(struct cnxk_ml_index_map) * cnxk_mldev->max_nb_layers, + RTE_CACHE_LINE_SIZE); + if (cnxk_mldev->index_map == NULL) { + plt_err("Failed to get memory for index_map, nb_layers %" PRIu64, + cnxk_mldev->max_nb_layers); + ret = -ENOMEM; + goto error; + } + } + + for (i = 0; i < cnxk_mldev->max_nb_layers; i++) + cnxk_mldev->index_map[i].active = false; + cnxk_mldev->nb_models_loaded = 0; cnxk_mldev->nb_models_started = 0; cnxk_mldev->nb_models_stopped = 0; @@ -305,6 +326,9 @@ cnxk_ml_dev_close(struct rte_ml_dev *dev) if (cn10k_ml_dev_close(cnxk_mldev) != 0) plt_err("Failed to close CN10K ML Device"); + if (cnxk_mldev->index_map) + rte_free(cnxk_mldev->index_map); + /* Stop and unload all models */ for (model_id = 0; model_id < dev->data->nb_models; model_id++) { model = dev->data->models[model_id]; @@ -314,7 +338,7 @@ cnxk_ml_dev_close(struct rte_ml_dev *dev) plt_err("Could not stop model %u", model_id); } if (model->state == ML_CNXK_MODEL_STATE_LOADED) { - if (cn10k_ml_model_unload(dev, model_id) != 0) + if (cnxk_ml_model_unload(dev, model_id) != 0) plt_err("Could not unload model %u", model_id); } dev->data->models[model_id] = NULL; @@ -430,6 +454,118 @@ cnxk_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, return 0; } +static int +cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, uint16_t *model_id) +{ + struct rte_ml_dev_info dev_info; + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + + char str[RTE_MEMZONE_NAMESIZE]; + const struct plt_memzone *mz; + uint64_t model_info_size; + uint16_t lcl_model_id; + uint64_t mz_size; + bool found; + int ret; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + /* Find model ID */ + found = false; + for (lcl_model_id = 0; lcl_model_id < dev->data->nb_models; lcl_model_id++) { + if (dev->data->models[lcl_model_id] == NULL) { + found = true; + break; + } + } + + if (!found) { + plt_err("No slots available to load new model"); + return -ENOMEM; + } + + /* Compute memzone size */ + cnxk_ml_dev_info_get(dev, &dev_info); + mz_size = PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), dev_info.align_size); + model_info_size = sizeof(struct rte_ml_model_info) + + ML_CNXK_MODEL_MAX_INPUT_OUTPUT * sizeof(struct rte_ml_io_info) + + ML_CNXK_MODEL_MAX_INPUT_OUTPUT * sizeof(struct rte_ml_io_info); + model_info_size = PLT_ALIGN_CEIL(model_info_size, dev_info.align_size); + mz_size += model_info_size; + + /* Allocate memzone for model object */ + snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", CNXK_ML_MODEL_MEMZONE_NAME, lcl_model_id); + mz = plt_memzone_reserve_aligned(str, mz_size, 0, dev_info.align_size); + if (!mz) { + plt_err("Failed to allocate memory for cnxk_ml_model: %s", str); + return -ENOMEM; + } + + model = mz->addr; + model->cnxk_mldev = cnxk_mldev; + model->model_id = lcl_model_id; + model->info = PLT_PTR_ADD( + model, PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), dev_info.align_size)); + dev->data->models[lcl_model_id] = model; + + ret = cn10k_ml_model_load(cnxk_mldev, params, model); + if (ret != 0) + goto error; + + plt_spinlock_init(&model->lock); + model->state = ML_CNXK_MODEL_STATE_LOADED; + cnxk_mldev->nb_models_loaded++; + + *model_id = lcl_model_id; + + return 0; + +error: + rte_memzone_free(mz); + + return ret; +} + +int +cnxk_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + + char str[RTE_MEMZONE_NAMESIZE]; + int ret; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + model = dev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + + if (model->state != ML_CNXK_MODEL_STATE_LOADED) { + plt_err("Cannot unload. Model in use."); + return -EBUSY; + } + + ret = cn10k_ml_model_unload(cnxk_mldev, model); + if (ret != 0) + return ret; + + dev->data->models[model_id] = NULL; + cnxk_mldev->nb_models_unloaded++; + + snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", CNXK_ML_MODEL_MEMZONE_NAME, model_id); + return plt_memzone_free(plt_memzone_lookup(str)); +} + struct rte_ml_dev_ops cnxk_ml_ops = { /* Device control ops */ .dev_info_get = cnxk_ml_dev_info_get, @@ -453,8 +589,8 @@ struct rte_ml_dev_ops cnxk_ml_ops = { .dev_xstats_reset = cn10k_ml_dev_xstats_reset, /* Model ops */ - .model_load = cn10k_ml_model_load, - .model_unload = cn10k_ml_model_unload, + .model_load = cnxk_ml_model_load, + .model_unload = cnxk_ml_model_unload, .model_start = cn10k_ml_model_start, .model_stop = cn10k_ml_model_stop, .model_info_get = cn10k_ml_model_info_get, diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h index a925c075809..bc14f6e5b9e 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.h +++ b/drivers/ml/cnxk/cnxk_ml_ops.h @@ -62,4 +62,6 @@ struct cnxk_ml_qp { extern struct rte_ml_dev_ops cnxk_ml_ops; +int cnxk_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id); + #endif /* _CNXK_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130894 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E0D6A41FD1; Wed, 30 Aug 2023 18:01:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 56716402A9; Wed, 30 Aug 2023 17:59:50 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id D05FB402A8 for ; Wed, 30 Aug 2023 17:59:38 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7p4004368 for ; Wed, 30 Aug 2023 08:59:38 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=FsdbnBkiKQiAlC5t2tQKRwnSai9x0ssoTSIGMehAKHU=; b=j+lizAHG2in62uhS+iJ7hTAdaKzKKKBIx54AajwPuu3Zzsy5l+uFmL44vifiEdN4ghLr q83XNL7R9zTHvsuU6OsQ4JHivmRE7vg8Cf+RDNce9Xe5HRJ6IG33+S0d+V9H7UA+JHXt dno6OXkZoADNrnlBhKznLEsWEZMPft1Wr/6rEfoGrLrmiBpJG2yc7IYvi9GnAvdkiOIX JjyC9IG4FutPOcMdFKA91VmU9bUAfYfb/mGcfsAxSsH7dXtv98PiUj4orehcVVW2IpEG qurK5Sf295/uEpdzTrLuosru1Spe/GGdd4kXseMxyPpCEDOOwaQ/58NfOmAGODQ6xzOa kA== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysst5-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:37 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:35 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:35 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 786B73F707D; Wed, 30 Aug 2023 08:59:35 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 11/34] ml/cnxk: update model start and stop functions Date: Wed, 30 Aug 2023 08:59:01 -0700 Message-ID: <20230830155927.3566-12-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: HobL2nrhPpaKI-ZnkEzGKIrvAQe0QCBV X-Proofpoint-ORIG-GUID: HobL2nrhPpaKI-ZnkEzGKIrvAQe0QCBV X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implemented cnxk wrapper functions to start and stop ML models. Wrapper functions would invoke the cn10k model start and stop functions. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ocm.c | 28 ++-- drivers/ml/cnxk/cn10k_ml_ocm.h | 12 +- drivers/ml/cnxk/cn10k_ml_ops.c | 282 ++++++++++++++++++++------------- drivers/ml/cnxk/cn10k_ml_ops.h | 8 +- drivers/ml/cnxk/cnxk_ml_ops.c | 48 +++++- drivers/ml/cnxk/cnxk_ml_ops.h | 1 + 6 files changed, 240 insertions(+), 139 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_ocm.c b/drivers/ml/cnxk/cn10k_ml_ocm.c index 639f329f8aa..6a8400b7763 100644 --- a/drivers/ml/cnxk/cn10k_ml_ocm.c +++ b/drivers/ml/cnxk/cn10k_ml_ocm.c @@ -217,11 +217,10 @@ cn10k_ml_ocm_tilecount(uint64_t tilemask, int *start, int *end) * scratch & WB pages and OCM allocation mode. */ int -cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t wb_pages, +cn10k_ml_ocm_tilemask_find(struct cnxk_ml_dev *cnxk_mldev, uint8_t num_tiles, uint16_t wb_pages, uint16_t scratch_pages, uint64_t *tilemask) { struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_ocm *ocm; uint16_t used_scratch_pages_max; @@ -240,7 +239,6 @@ cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t w int max_slot_sz; int page_id; - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; ocm = &cn10k_mldev->ocm; @@ -335,12 +333,10 @@ cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t w } void -cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t layer_id, +cn10k_ml_ocm_reserve_pages(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id, uint16_t layer_id, uint64_t tilemask, int wb_page_start, uint16_t wb_pages, uint16_t scratch_pages) { - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; struct cnxk_ml_layer *layer; struct cn10k_ml_ocm *ocm; @@ -353,10 +349,8 @@ cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t l int tile_id; int page_id; - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - ocm = &cn10k_mldev->ocm; - model = dev->data->models[model_id]; + ocm = &cnxk_mldev->cn10k_mldev.ocm; + model = cnxk_mldev->mldev->data->models[model_id]; layer = &model->layer[layer_id]; /* Get first set bit, tile_start */ @@ -398,12 +392,10 @@ cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t l } void -cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t layer_id) +cn10k_ml_ocm_free_pages(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id, uint16_t layer_id) { struct cnxk_ml_model *local_model; struct cnxk_ml_layer *local_layer; - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; struct cnxk_ml_layer *layer; struct cn10k_ml_ocm *ocm; @@ -418,10 +410,8 @@ cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t laye uint16_t i; uint16_t j; - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - ocm = &cn10k_mldev->ocm; - model = dev->data->models[model_id]; + ocm = &cnxk_mldev->cn10k_mldev.ocm; + model = cnxk_mldev->mldev->data->models[model_id]; layer = &model->layer[layer_id]; /* Update OCM info for WB memory */ @@ -440,8 +430,8 @@ cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t laye /* Get max scratch pages required, excluding the current model */ scratch_resize_pages = 0; - for (i = 0; i < dev->data->nb_models; i++) { - local_model = dev->data->models[i]; + for (i = 0; i < cnxk_mldev->mldev->data->nb_models; i++) { + local_model = cnxk_mldev->mldev->data->models[i]; if (local_model == NULL) continue; diff --git a/drivers/ml/cnxk/cn10k_ml_ocm.h b/drivers/ml/cnxk/cn10k_ml_ocm.h index 720f8caf766..97b723a56a5 100644 --- a/drivers/ml/cnxk/cn10k_ml_ocm.h +++ b/drivers/ml/cnxk/cn10k_ml_ocm.h @@ -8,6 +8,8 @@ #include #include +struct cnxk_ml_dev; + /* Number of OCM tiles. */ #define ML_CN10K_OCM_NUMTILES 0x8 @@ -75,12 +77,12 @@ struct cn10k_ml_ocm { }; int cn10k_ml_ocm_tilecount(uint64_t tilemask, int *start, int *end); -int cn10k_ml_ocm_tilemask_find(struct rte_ml_dev *dev, uint8_t num_tiles, uint16_t wb_pages, +int cn10k_ml_ocm_tilemask_find(struct cnxk_ml_dev *cnxk_mldev, uint8_t num_tiles, uint16_t wb_pages, uint16_t scratch_pages, uint64_t *tilemask); -void cn10k_ml_ocm_reserve_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t layer_id, - uint64_t tilemask, int wb_page_start, uint16_t wb_pages, - uint16_t scratch_pages); -void cn10k_ml_ocm_free_pages(struct rte_ml_dev *dev, uint16_t model_id, uint16_t layer_id); +void cn10k_ml_ocm_reserve_pages(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id, + uint16_t layer_id, uint64_t tilemask, int wb_page_start, + uint16_t wb_pages, uint16_t scratch_pages); +void cn10k_ml_ocm_free_pages(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id, uint16_t layer_id); void cn10k_ml_ocm_print(struct rte_ml_dev *dev, FILE *fp); #endif /* _CN10K_ML_OCM_H_ */ diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 3bfc63d9d40..e5b9837ed73 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -252,26 +252,28 @@ cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) } static void -cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml_model *model, +cn10k_ml_prep_sp_job_descriptor(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, struct cnxk_ml_req *req, enum cn10k_ml_job_type job_type) { struct cn10k_ml_model_metadata *metadata; struct cn10k_ml_layer_addr *addr; + struct cn10k_ml_dev *cn10k_mldev; - metadata = &model->glow.metadata; - addr = &model->layer[0].glow.addr; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + metadata = &layer->glow.metadata; + addr = &layer->glow.addr; memset(&req->cn10k_req.jd, 0, sizeof(struct cn10k_ml_jd)); req->cn10k_req.jd.hdr.jce.w0.u64 = 0; req->cn10k_req.jd.hdr.jce.w1.u64 = PLT_U64_CAST(&req->cn10k_req.status); - req->cn10k_req.jd.hdr.model_id = model->model_id; + req->cn10k_req.jd.hdr.model_id = layer->index; req->cn10k_req.jd.hdr.job_type = job_type; req->cn10k_req.jd.hdr.fp_flags = 0x0; req->cn10k_req.jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->cn10k_req.result); if (job_type == ML_CN10K_JOB_TYPE_MODEL_START) { - if (!model->glow.metadata.model.ocm_relocatable) + if (!layer->glow.metadata.model.ocm_relocatable) req->cn10k_req.jd.hdr.sp_flags = ML_CN10K_SP_FLAGS_OCM_NONRELOCATABLE; else req->cn10k_req.jd.hdr.sp_flags = 0x0; @@ -295,7 +297,7 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml req->cn10k_req.jd.model_start.num_gather_entries = 0; req->cn10k_req.jd.model_start.num_scatter_entries = 0; req->cn10k_req.jd.model_start.tilemask = 0; /* Updated after reserving pages */ - req->cn10k_req.jd.model_start.batch_size = model->batch_size; + req->cn10k_req.jd.model_start.batch_size = layer->batch_size; req->cn10k_req.jd.model_start.ocm_wb_base_address = 0; /* Updated after reserving pages */ req->cn10k_req.jd.model_start.ocm_wb_range_start = @@ -327,9 +329,13 @@ cn10k_ml_prep_sp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml } static __rte_always_inline void -cn10k_ml_prep_fp_job_descriptor(struct cn10k_ml_dev *cn10k_mldev, struct cnxk_ml_req *req, +cn10k_ml_prep_fp_job_descriptor(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_req *req, struct rte_ml_op *op) { + struct cn10k_ml_dev *cn10k_mldev; + + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + req->cn10k_req.jd.hdr.jce.w0.u64 = 0; req->cn10k_req.jd.hdr.jce.w1.u64 = PLT_U64_CAST(req->status); req->cn10k_req.jd.hdr.model_id = op->model_id; @@ -718,10 +724,8 @@ cn10k_ml_model_xstats_reset(struct rte_ml_dev *dev, int32_t model_id, const uint } static int -cn10k_ml_cache_model_data(struct rte_ml_dev *dev, uint16_t model_id) +cn10k_ml_cache_model_data(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer) { - struct rte_ml_model_info *info; - struct cnxk_ml_model *model; struct rte_ml_buff_seg seg[2]; struct rte_ml_buff_seg *inp; struct rte_ml_buff_seg *out; @@ -734,22 +738,20 @@ cn10k_ml_cache_model_data(struct rte_ml_dev *dev, uint16_t model_id) int ret = 0; uint32_t i; - model = dev->data->models[model_id]; - info = (struct rte_ml_model_info *)model->info; inp = &seg[0]; out = &seg[1]; /* Create input and output buffers. */ - for (i = 0; i < info->nb_inputs; i++) - isize += info->input_info[i].size; + for (i = 0; i < layer->info.nb_inputs; i++) + isize += layer->info.input[i].sz_q; - for (i = 0; i < info->nb_outputs; i++) - osize += info->output_info[i].size; + for (i = 0; i < layer->info.nb_outputs; i++) + osize += layer->info.output[i].sz_q; - isize = model->batch_size * isize; - osize = model->batch_size * osize; + isize = layer->batch_size * isize; + osize = layer->batch_size * osize; - snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", "ml_dummy_io", model_id); + snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", "ml_dummy_io", layer->index); mz = plt_memzone_reserve_aligned(str, isize + osize, 0, ML_CN10K_ALIGN_SIZE); if (mz == NULL) return -ENOMEM; @@ -765,15 +767,15 @@ cn10k_ml_cache_model_data(struct rte_ml_dev *dev, uint16_t model_id) seg[1].length = osize; seg[1].next = NULL; - op.model_id = model_id; - op.nb_batches = model->batch_size; + op.model_id = layer->index; + op.nb_batches = layer->batch_size; op.mempool = NULL; op.input = &inp; op.output = &out; - memset(model->layer[0].glow.req, 0, sizeof(struct cnxk_ml_req)); - ret = cn10k_ml_inference_sync(dev, &op); + memset(layer->glow.req, 0, sizeof(struct cnxk_ml_req)); + ret = cn10k_ml_inference_sync(cnxk_mldev, &op); plt_memzone_free(mz); return ret; @@ -1510,14 +1512,16 @@ cn10k_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mode } int -cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) +cn10k_ml_layer_start(void *device, uint16_t model_id, const char *layer_name) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; struct cn10k_ml_ocm *ocm; struct cnxk_ml_req *req; + uint16_t layer_id = 0; bool job_enqueued; bool job_dequeued; uint8_t num_tiles; @@ -1528,85 +1532,89 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) bool locked; int ret = 0; - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - ocm = &cn10k_mldev->ocm; - model = dev->data->models[model_id]; + PLT_SET_USED(layer_name); + cnxk_mldev = (struct cnxk_ml_dev *)device; + if (cnxk_mldev == NULL) { + plt_err("Invalid device = %p", device); + return -EINVAL; + } + + model = cnxk_mldev->mldev->data->models[model_id]; if (model == NULL) { plt_err("Invalid model_id = %u", model_id); return -EINVAL; } + layer = &model->layer[layer_id]; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + ocm = &cn10k_mldev->ocm; + /* Prepare JD */ - req = model->layer[0].glow.req; - cn10k_ml_prep_sp_job_descriptor(cn10k_mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_START); + req = layer->glow.req; + cn10k_ml_prep_sp_job_descriptor(cnxk_mldev, layer, req, ML_CN10K_JOB_TYPE_MODEL_START); req->cn10k_req.result.error_code = 0x0; req->cn10k_req.result.user_ptr = NULL; plt_write64(ML_CNXK_POLL_JOB_START, &req->cn10k_req.status); plt_wmb(); - num_tiles = model->layer[0].glow.metadata.model.tile_end - - model->layer[0].glow.metadata.model.tile_start + 1; + num_tiles = layer->glow.metadata.model.tile_end - layer->glow.metadata.model.tile_start + 1; locked = false; while (!locked) { if (plt_spinlock_trylock(&model->lock) != 0) { - if (model->state == ML_CNXK_MODEL_STATE_STARTED) { - plt_ml_dbg("Model already started, model = 0x%016lx", - PLT_U64_CAST(model)); + if (layer->state == ML_CNXK_LAYER_STATE_STARTED) { + plt_ml_dbg("Layer already started, model_id = %u, layer_id = %u", + model->model_id, layer_id); plt_spinlock_unlock(&model->lock); return 1; } - if (model->state == ML_CNXK_MODEL_STATE_JOB_ACTIVE) { - plt_err("A slow-path job is active for the model = 0x%016lx", - PLT_U64_CAST(model)); + if (layer->state == ML_CNXK_LAYER_STATE_JOB_ACTIVE) { + plt_err("A slow-path job is active for the model_id = %u", + model->model_id); plt_spinlock_unlock(&model->lock); return -EBUSY; } - model->state = ML_CNXK_MODEL_STATE_JOB_ACTIVE; + layer->state = ML_CNXK_LAYER_STATE_JOB_ACTIVE; plt_spinlock_unlock(&model->lock); locked = true; } } - while (!model->layer[0].glow.ocm_map.ocm_reserved) { + while (!layer->glow.ocm_map.ocm_reserved) { if (plt_spinlock_trylock(&ocm->lock) != 0) { wb_page_start = cn10k_ml_ocm_tilemask_find( - dev, num_tiles, model->layer[0].glow.ocm_map.wb_pages, - model->layer[0].glow.ocm_map.scratch_pages, &tilemask); + cnxk_mldev, num_tiles, layer->glow.ocm_map.wb_pages, + layer->glow.ocm_map.scratch_pages, &tilemask); if (wb_page_start == -1) { plt_err("Free pages not available on OCM tiles"); - plt_err("Failed to start model = 0x%016lx, name = %s", - PLT_U64_CAST(model), - model->layer[0].glow.metadata.model.name); - + plt_err("Failed to start layer, model_id = %u, layer_id = %u", + model->model_id, layer_id); plt_spinlock_unlock(&ocm->lock); return -ENOMEM; } - model->layer[0].glow.ocm_map.tilemask = tilemask; - model->layer[0].glow.ocm_map.wb_page_start = wb_page_start; + layer->glow.ocm_map.tilemask = tilemask; + layer->glow.ocm_map.wb_page_start = wb_page_start; - cn10k_ml_ocm_reserve_pages(dev, model->model_id, 0, - model->layer[0].glow.ocm_map.tilemask, - model->layer[0].glow.ocm_map.wb_page_start, - model->layer[0].glow.ocm_map.wb_pages, - model->layer[0].glow.ocm_map.scratch_pages); - model->layer[0].glow.ocm_map.ocm_reserved = true; + cn10k_ml_ocm_reserve_pages( + cnxk_mldev, model->model_id, layer_id, layer->glow.ocm_map.tilemask, + layer->glow.ocm_map.wb_page_start, layer->glow.ocm_map.wb_pages, + layer->glow.ocm_map.scratch_pages); + layer->glow.ocm_map.ocm_reserved = true; plt_spinlock_unlock(&ocm->lock); } } /* Update JD */ - cn10k_ml_ocm_tilecount(model->layer[0].glow.ocm_map.tilemask, &tile_start, &tile_end); + cn10k_ml_ocm_tilecount(layer->glow.ocm_map.tilemask, &tile_start, &tile_end); req->cn10k_req.jd.model_start.tilemask = GENMASK_ULL(tile_end, tile_start); req->cn10k_req.jd.model_start.ocm_wb_base_address = - model->layer[0].glow.ocm_map.wb_page_start * ocm->page_size; + layer->glow.ocm_map.wb_page_start * ocm->page_size; job_enqueued = false; job_dequeued = false; @@ -1640,66 +1648,94 @@ cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) locked = false; while (!locked) { if (plt_spinlock_trylock(&model->lock) != 0) { - if (ret == 0) { - model->state = ML_CNXK_MODEL_STATE_STARTED; - cnxk_mldev->nb_models_started++; - } else { - model->state = ML_CNXK_MODEL_STATE_UNKNOWN; - } + if (ret == 0) + layer->state = ML_CNXK_LAYER_STATE_STARTED; + else + layer->state = ML_CNXK_LAYER_STATE_UNKNOWN; plt_spinlock_unlock(&model->lock); locked = true; } } - if (model->state == ML_CNXK_MODEL_STATE_UNKNOWN) { - while (model->layer[0].glow.ocm_map.ocm_reserved) { + if (layer->state == ML_CNXK_LAYER_STATE_UNKNOWN) { + while (layer->glow.ocm_map.ocm_reserved) { if (plt_spinlock_trylock(&ocm->lock) != 0) { - cn10k_ml_ocm_free_pages(dev, model->model_id, 0); - model->layer[0].glow.ocm_map.ocm_reserved = false; - model->layer[0].glow.ocm_map.tilemask = 0x0; + cn10k_ml_ocm_free_pages(cnxk_mldev, model->model_id, layer_id); + layer->glow.ocm_map.ocm_reserved = false; + layer->glow.ocm_map.tilemask = 0x0; plt_spinlock_unlock(&ocm->lock); } } } - if (ret < 0) { /* Call unload to update model and FW state, ignore error */ - rte_ml_model_stop(dev->data->dev_id, model_id); + if (ret < 0) { + cn10k_ml_layer_stop(device, model_id, layer_name); } else { - if (cn10k_mldev->cache_model_data && roc_model_is_cn10ka()) - ret = cn10k_ml_cache_model_data(dev, model_id); + if (cn10k_mldev->cache_model_data) + ret = cn10k_ml_cache_model_data(cnxk_mldev, layer); } return ret; } int -cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) +cn10k_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model) +{ + struct cnxk_ml_layer *layer; + int ret; + + layer = &model->layer[0]; + ret = cn10k_ml_layer_start(cnxk_mldev, model->model_id, layer->name); + if (ret != 0) { + plt_err("CN10K Model start failed, model_id = %u, error = %d", model->model_id, + ret); + return ret; + } + + cnxk_mldev->nb_models_started++; + model->state = ML_CNXK_MODEL_STATE_STARTED; + + return 0; +} + +int +cn10k_ml_layer_stop(void *device, uint16_t model_id, const char *layer_name) { struct cn10k_ml_dev *cn10k_mldev; struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; struct cn10k_ml_ocm *ocm; struct cnxk_ml_req *req; + uint16_t layer_id = 0; bool job_enqueued; bool job_dequeued; bool locked; int ret = 0; - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - ocm = &cn10k_mldev->ocm; - model = dev->data->models[model_id]; + PLT_SET_USED(layer_name); + + cnxk_mldev = (struct cnxk_ml_dev *)device; + if (cnxk_mldev == NULL) { + plt_err("Invalid device = %p", device); + return -EINVAL; + } + model = cnxk_mldev->mldev->data->models[model_id]; if (model == NULL) { plt_err("Invalid model_id = %u", model_id); return -EINVAL; } + layer = &model->layer[layer_id]; + cn10k_mldev = &cnxk_mldev->cn10k_mldev; + ocm = &cn10k_mldev->ocm; + /* Prepare JD */ - req = model->layer[0].glow.req; - cn10k_ml_prep_sp_job_descriptor(cn10k_mldev, model, req, ML_CN10K_JOB_TYPE_MODEL_STOP); + req = layer->glow.req; + cn10k_ml_prep_sp_job_descriptor(cnxk_mldev, layer, req, ML_CN10K_JOB_TYPE_MODEL_STOP); req->cn10k_req.result.error_code = 0x0; req->cn10k_req.result.user_ptr = NULL; @@ -1709,31 +1745,31 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) locked = false; while (!locked) { if (plt_spinlock_trylock(&model->lock) != 0) { - if (model->state == ML_CNXK_MODEL_STATE_LOADED) { - plt_ml_dbg("Model not started, model = 0x%016lx", - PLT_U64_CAST(model)); + if (layer->state == ML_CNXK_LAYER_STATE_LOADED) { + plt_ml_dbg("Layer not started, model_id = %u, layer_id = %u", + model->model_id, layer_id); plt_spinlock_unlock(&model->lock); return 1; } - if (model->state == ML_CNXK_MODEL_STATE_JOB_ACTIVE) { - plt_err("A slow-path job is active for the model = 0x%016lx", - PLT_U64_CAST(model)); + if (layer->state == ML_CNXK_LAYER_STATE_JOB_ACTIVE) { + plt_err("A slow-path job is active for the layer, model_id = %u, layer_id = %u", + model->model_id, layer_id); plt_spinlock_unlock(&model->lock); return -EBUSY; } - model->state = ML_CNXK_MODEL_STATE_JOB_ACTIVE; + layer->state = ML_CNXK_LAYER_STATE_JOB_ACTIVE; plt_spinlock_unlock(&model->lock); locked = true; } } - while (model->layer[0].glow.ocm_map.ocm_reserved) { + while (layer->glow.ocm_map.ocm_reserved) { if (plt_spinlock_trylock(&ocm->lock) != 0) { - cn10k_ml_ocm_free_pages(dev, model->model_id, 0); - model->layer[0].glow.ocm_map.ocm_reserved = false; - model->layer[0].glow.ocm_map.tilemask = 0x0; + cn10k_ml_ocm_free_pages(cnxk_mldev, model->model_id, layer_id); + layer->glow.ocm_map.ocm_reserved = false; + layer->glow.ocm_map.tilemask = 0x0; plt_spinlock_unlock(&ocm->lock); } } @@ -1770,8 +1806,11 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) locked = false; while (!locked) { if (plt_spinlock_trylock(&model->lock) != 0) { - cnxk_mldev->nb_models_stopped++; - model->state = ML_CNXK_MODEL_STATE_LOADED; + if (ret == 0) + layer->state = ML_CNXK_LAYER_STATE_LOADED; + else + layer->state = ML_CNXK_LAYER_STATE_UNKNOWN; + plt_spinlock_unlock(&model->lock); locked = true; } @@ -1780,6 +1819,25 @@ cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) return ret; } +int +cn10k_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model) +{ + struct cnxk_ml_layer *layer; + int ret; + + layer = &model->layer[0]; + ret = cn10k_ml_layer_stop(cnxk_mldev, model->model_id, layer->name); + if (ret != 0) { + plt_err("CN10K Model stop failed, model_id = %u, error = %d", model->model_id, ret); + return ret; + } + + cnxk_mldev->nb_models_stopped++; + model->state = ML_CNXK_MODEL_STATE_LOADED; + + return 0; +} + int cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_model_info *model_info) @@ -2007,30 +2065,35 @@ queue_free_count(uint64_t head, uint64_t tail, uint64_t nb_desc) } static __rte_always_inline void -cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cnxk_ml_req *req) +cn10k_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, struct cnxk_ml_req *req) { union cn10k_ml_error_code *error_code; struct cn10k_ml_layer_xstats *xstats; struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_result *result; struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; struct cnxk_ml_qp *qp; struct rte_ml_op *op; uint64_t hw_latency; uint64_t fw_latency; + uint16_t model_id; + uint16_t layer_id; result = &req->cn10k_req.result; op = req->op; if (likely(result->error_code == 0)) { - model = dev->data->models[op->model_id]; + model_id = cnxk_mldev->index_map[op->model_id].model_id; + layer_id = cnxk_mldev->index_map[op->model_id].layer_id; + model = cnxk_mldev->mldev->data->models[model_id]; + layer = &model->layer[layer_id]; if (likely(qp_id >= 0)) { - qp = dev->data->queue_pairs[qp_id]; + qp = cnxk_mldev->mldev->data->queue_pairs[qp_id]; qp->stats.dequeued_count++; - xstats = &model->layer[0].glow.burst_xstats[qp_id]; + xstats = &layer->glow.burst_xstats[qp_id]; } else { - xstats = model->layer[0].glow.sync_xstats; + xstats = layer->glow.sync_xstats; } if (unlikely(xstats->dequeued_count == xstats->hw_reset_count)) { @@ -2058,14 +2121,13 @@ cn10k_ml_result_update(struct rte_ml_dev *dev, int qp_id, struct cnxk_ml_req *re op->status = RTE_ML_OP_STATUS_SUCCESS; } else { if (likely(qp_id >= 0)) { - qp = dev->data->queue_pairs[qp_id]; + qp = cnxk_mldev->mldev->data->queue_pairs[qp_id]; qp->stats.dequeue_err_count++; } /* Handle driver error */ error_code = (union cn10k_ml_error_code *)&result->error_code; if (error_code->s.etype == ML_ETYPE_DRIVER) { - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; /* Check for exception */ @@ -2120,7 +2182,7 @@ cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op req = &queue->reqs[head]; cn10k_mldev->set_poll_addr(req); - cn10k_ml_prep_fp_job_descriptor(cn10k_mldev, req, op); + cn10k_ml_prep_fp_job_descriptor(cnxk_mldev, req, op); memset(&req->cn10k_req.result, 0, sizeof(struct cn10k_ml_result)); error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; @@ -2187,7 +2249,7 @@ cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op } } - cn10k_ml_result_update(dev, qp_id, req); + cn10k_ml_result_update(cnxk_mldev, qp_id, req); ops[count] = req->op; queue_index_advance(&tail, qp->nb_desc); @@ -2236,23 +2298,27 @@ cn10k_ml_op_error_get(struct rte_ml_dev *dev, struct rte_ml_op *op, struct rte_m } __rte_hot int -cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) +cn10k_ml_inference_sync(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op) { union cn10k_ml_error_code *error_code; struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; struct cnxk_ml_req *req; + uint16_t model_id; + uint16_t layer_id; bool timeout; int ret = 0; - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; - model = dev->data->models[op->model_id]; - req = model->layer[0].glow.req; + model_id = cnxk_mldev->index_map[op->model_id].model_id; + layer_id = cnxk_mldev->index_map[op->model_id].layer_id; + model = cnxk_mldev->mldev->data->models[model_id]; + layer = &model->layer[layer_id]; + req = layer->glow.req; cn10k_ml_set_poll_addr(req); - cn10k_ml_prep_fp_job_descriptor(cn10k_mldev, req, op); + cn10k_ml_prep_fp_job_descriptor(cnxk_mldev, req, op); memset(&req->cn10k_req.result, 0, sizeof(struct cn10k_ml_result)); error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; @@ -2288,7 +2354,7 @@ cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op) if (timeout) ret = -ETIME; else - cn10k_ml_result_update(dev, -1, req); + cn10k_ml_result_update(cnxk_mldev, -1, req); error_enqueue: return ret; diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 677219dfdf7..a222a43d552 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -315,8 +315,8 @@ int cn10k_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mod int cn10k_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, struct cnxk_ml_model *model); int cn10k_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); -int cn10k_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id); -int cn10k_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id); +int cn10k_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); +int cn10k_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); int cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_model_info *model_info); int cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buffer); @@ -335,7 +335,7 @@ __rte_hot uint16_t cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id struct rte_ml_op **ops, uint16_t nb_ops); __rte_hot int cn10k_ml_op_error_get(struct rte_ml_dev *dev, struct rte_ml_op *op, struct rte_ml_op_error *error); -__rte_hot int cn10k_ml_inference_sync(struct rte_ml_dev *dev, struct rte_ml_op *op); +__rte_hot int cn10k_ml_inference_sync(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op); /* Misc ops */ void cn10k_ml_qp_initialize(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_qp *qp); @@ -344,5 +344,7 @@ void cn10k_ml_qp_initialize(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_qp *q int cn10k_ml_layer_load(void *device, uint16_t model_id, const char *layer_name, uint8_t *buffer, size_t size, uint16_t *index); int cn10k_ml_layer_unload(void *device, uint16_t model_id, const char *layer_name); +int cn10k_ml_layer_start(void *device, uint16_t model_id, const char *layer_name); +int cn10k_ml_layer_stop(void *device, uint16_t model_id, const char *layer_name); #endif /* _CN10K_ML_OPS_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 3d9d5f9d78c..915309168d8 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -242,7 +242,7 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co model = dev->data->models[model_id]; if (model != NULL) { if (model->state == ML_CNXK_MODEL_STATE_STARTED) { - if (cn10k_ml_model_stop(dev, model_id) != 0) + if (cnxk_ml_model_stop(dev, model_id) != 0) plt_err("Could not stop model %u", model_id); } if (model->state == ML_CNXK_MODEL_STATE_LOADED) { @@ -334,7 +334,7 @@ cnxk_ml_dev_close(struct rte_ml_dev *dev) model = dev->data->models[model_id]; if (model != NULL) { if (model->state == ML_CNXK_MODEL_STATE_STARTED) { - if (cn10k_ml_model_stop(dev, model_id) != 0) + if (cnxk_ml_model_stop(dev, model_id) != 0) plt_err("Could not stop model %u", model_id); } if (model->state == ML_CNXK_MODEL_STATE_LOADED) { @@ -566,6 +566,46 @@ cnxk_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) return plt_memzone_free(plt_memzone_lookup(str)); } +static int +cnxk_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + model = dev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + + return cn10k_ml_model_start(cnxk_mldev, model); +} + +int +cnxk_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + model = dev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + + return cn10k_ml_model_stop(cnxk_mldev, model); +} + struct rte_ml_dev_ops cnxk_ml_ops = { /* Device control ops */ .dev_info_get = cnxk_ml_dev_info_get, @@ -591,8 +631,8 @@ struct rte_ml_dev_ops cnxk_ml_ops = { /* Model ops */ .model_load = cnxk_ml_model_load, .model_unload = cnxk_ml_model_unload, - .model_start = cn10k_ml_model_start, - .model_stop = cn10k_ml_model_stop, + .model_start = cnxk_ml_model_start, + .model_stop = cnxk_ml_model_stop, .model_info_get = cn10k_ml_model_info_get, .model_params_update = cn10k_ml_model_params_update, diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h index bc14f6e5b9e..d27ca0d0cb2 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.h +++ b/drivers/ml/cnxk/cnxk_ml_ops.h @@ -63,5 +63,6 @@ struct cnxk_ml_qp { extern struct rte_ml_dev_ops cnxk_ml_ops; int cnxk_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id); +int cnxk_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id); #endif /* _CNXK_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130891 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 75C6B41FD1; Wed, 30 Aug 2023 18:01:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 39135402C7; Wed, 30 Aug 2023 17:59:47 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 15901402A7 for ; Wed, 30 Aug 2023 17:59:37 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I6u012554 for ; Wed, 30 Aug 2023 08:59:37 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=7WBUlhuujU3wyX16x3i6bY6EsZzpDhIHVR75vBsCXZM=; b=WhvvzAHVAyTPLv5JOOK4lxTiQaGc9y2ZLm1Ao33s3Dwwhwsl/Y5GE1yvdkBzggKP8KY7 LpjPL61U/FytR4Ab0FnK626kvi6/0Yv6iIjqcfyE0odtHk/uiMHRGFVkfXuLzD5KIVKl J1ix5DRPvA0uBX2/H+BWexKJh377pspCXFumbZCkMryZsFgPRz/vLomj9+8ogGqUNvIA g+LJYhtkuoQ6+906uFq51jI6N+RB4uuQZwdJqqJkEs/QIDFmFJz8PRef0GbSY7xpVWYg m1jhwqjvaCyRsHeQumxpmngTBpgW9EFKfIsZXo9amDREYtx5uKgBzQ2DA1FBWEbFGXyU BA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:37 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:36 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:36 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id C42403F7080; Wed, 30 Aug 2023 08:59:35 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 12/34] ml/cnxk: update model utility functions Date: Wed, 30 Aug 2023 08:59:02 -0700 Message-ID: <20230830155927.3566-13-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: MlnqxMxntRNh9_qz6alwEG-neX7ak-9b X-Proofpoint-GUID: MlnqxMxntRNh9_qz6alwEG-neX7ak-9b X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added cnxk wrapper function to update model params and fetch model info. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ops.c | 38 ++++++--------------------- drivers/ml/cnxk/cn10k_ml_ops.h | 5 ++-- drivers/ml/cnxk/cnxk_ml_ops.c | 48 ++++++++++++++++++++++++++++++++-- 3 files changed, 56 insertions(+), 35 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index e5b9837ed73..0eebefee5fc 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -1839,45 +1839,23 @@ cn10k_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model) } int -cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, - struct rte_ml_model_info *model_info) +cn10k_ml_model_params_update(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, + void *buffer) { - struct cnxk_ml_model *model; - - model = dev->data->models[model_id]; - - if (model == NULL) { - plt_err("Invalid model_id = %u", model_id); - return -EINVAL; - } - - rte_memcpy(model_info, model->info, sizeof(struct rte_ml_model_info)); - model_info->input_info = ((struct rte_ml_model_info *)model->info)->input_info; - model_info->output_info = ((struct rte_ml_model_info *)model->info)->output_info; - - return 0; -} - -int -cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buffer) -{ - struct cnxk_ml_model *model; - - model = dev->data->models[model_id]; + struct cnxk_ml_layer *layer; - if (model == NULL) { - plt_err("Invalid model_id = %u", model_id); - return -EINVAL; - } + RTE_SET_USED(cnxk_mldev); if (model->state == ML_CNXK_MODEL_STATE_UNKNOWN) return -1; else if (model->state != ML_CNXK_MODEL_STATE_LOADED) return -EBUSY; + layer = &model->layer[0]; + /* Update model weights & bias */ - rte_memcpy(model->layer[0].glow.addr.wb_load_addr, buffer, - model->layer[0].glow.metadata.weights_bias.file_size); + rte_memcpy(layer->glow.addr.wb_load_addr, buffer, + layer->glow.metadata.weights_bias.file_size); return 0; } diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index a222a43d552..ef12069f0df 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -317,9 +317,8 @@ int cn10k_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_para int cn10k_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); int cn10k_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); int cn10k_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); -int cn10k_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, - struct rte_ml_model_info *model_info); -int cn10k_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buffer); +int cn10k_ml_model_params_update(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, + void *buffer); /* I/O ops */ int cn10k_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 915309168d8..5ad0ea8c3ce 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -606,6 +606,50 @@ cnxk_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) return cn10k_ml_model_stop(cnxk_mldev, model); } +static int +cnxk_ml_model_info_get(struct rte_ml_dev *dev, uint16_t model_id, + struct rte_ml_model_info *model_info) +{ + struct rte_ml_model_info *info; + struct cnxk_ml_model *model; + + if ((dev == NULL) || (model_info == NULL)) + return -EINVAL; + + model = dev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + + info = (struct rte_ml_model_info *)model->info; + rte_memcpy(model_info, info, sizeof(struct rte_ml_model_info)); + model_info->input_info = info->input_info; + model_info->output_info = info->output_info; + + return 0; +} + +static int +cnxk_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buffer) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + + if ((dev == NULL) || (buffer == NULL)) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + model = dev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + + return cn10k_ml_model_params_update(cnxk_mldev, model, buffer); +} + struct rte_ml_dev_ops cnxk_ml_ops = { /* Device control ops */ .dev_info_get = cnxk_ml_dev_info_get, @@ -633,8 +677,8 @@ struct rte_ml_dev_ops cnxk_ml_ops = { .model_unload = cnxk_ml_model_unload, .model_start = cnxk_ml_model_start, .model_stop = cnxk_ml_model_stop, - .model_info_get = cn10k_ml_model_info_get, - .model_params_update = cn10k_ml_model_params_update, + .model_info_get = cnxk_ml_model_info_get, + .model_params_update = cnxk_ml_model_params_update, /* I/O ops */ .io_quantize = cn10k_ml_io_quantize, From patchwork Wed Aug 30 15:59:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130893 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7384F41FD1; Wed, 30 Aug 2023 18:01:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3DFF5402D1; Wed, 30 Aug 2023 17:59:49 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 99451402A7 for ; Wed, 30 Aug 2023 17:59:38 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I6v012554 for ; Wed, 30 Aug 2023 08:59:38 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=hLXYBdw03Mb7EQlGYFwzsGvY5hfFdiZg6z9o3AADuro=; b=JbqVsh1gvKbaHbuNe4iMdYdEJzx7nFL7p8FlUOMS0GOZIfMx3/EpYpEUlYl4zPC5lWS4 oniCkuW7Y+nTlAr871nrZcdwkVM2n0mnG3dE76OZJRMHy3LXutVy5eWQkh20/ixhO2S5 /sRpxcJMqy+CJTJKeVlUuNaO+edmNWYUVEA5B/i//ZcX/HWEUYTAOzuq4JhjUq+0H9n6 ojWNBddiYkEjTlEUhS043IBG+HYQixjkNv4YgHRC7iqrWhCWr+3O5SntqZtN6PJCJy8N ZirwJjlKuWgS1tQijZWLlT3SlnB/FSaxg9IHfvUmgogHIT6ICGuP18gR4esJXqesMmWX /A== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:37 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:36 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:36 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 227253F707E; Wed, 30 Aug 2023 08:59:36 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 13/34] ml/cnxk: update data quantization functions Date: Wed, 30 Aug 2023 08:59:03 -0700 Message-ID: <20230830155927.3566-14-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: Bjeqk7vGwp2nWc1xSQ7A3_IKiH-8t3Y- X-Proofpoint-GUID: Bjeqk7vGwp2nWc1xSQ7A3_IKiH-8t3Y- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added cnxk wrapper functions to quantize input data and dequantize output data. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ops.c | 164 --------------------------------- drivers/ml/cnxk/cn10k_ml_ops.h | 7 -- drivers/ml/cnxk/cnxk_ml_io.c | 95 +++++++++++++++++++ drivers/ml/cnxk/cnxk_ml_io.h | 3 + drivers/ml/cnxk/cnxk_ml_ops.c | 78 +++++++++++++++- drivers/ml/cnxk/meson.build | 1 + 6 files changed, 175 insertions(+), 173 deletions(-) create mode 100644 drivers/ml/cnxk/cnxk_ml_io.c diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 0eebefee5fc..1e6aee818c7 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -1860,170 +1860,6 @@ cn10k_ml_model_params_update(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_mode return 0; } -int -cn10k_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buff_seg **dbuffer, - struct rte_ml_buff_seg **qbuffer) -{ - struct cnxk_ml_model *model; - uint8_t model_input_type; - uint8_t *lcl_dbuffer; - uint8_t *lcl_qbuffer; - uint8_t input_type; - float qscale; - uint32_t i; - uint32_t j; - int ret; - - model = dev->data->models[model_id]; - - if (model == NULL) { - plt_err("Invalid model_id = %u", model_id); - return -EINVAL; - } - - lcl_dbuffer = dbuffer[0]->addr; - lcl_qbuffer = qbuffer[0]->addr; - - for (i = 0; i < model->layer[0].glow.metadata.model.num_input; i++) { - if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - input_type = model->layer[0].glow.metadata.input1[i].input_type; - model_input_type = model->layer[0].glow.metadata.input1[i].model_input_type; - qscale = model->layer[0].glow.metadata.input1[i].qscale; - } else { - j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - input_type = model->layer[0].glow.metadata.input2[j].input_type; - model_input_type = model->layer[0].glow.metadata.input2[j].model_input_type; - qscale = model->layer[0].glow.metadata.input2[j].qscale; - } - - if (input_type == model_input_type) { - rte_memcpy(lcl_qbuffer, lcl_dbuffer, model->layer[0].info.input[i].sz_d); - } else { - switch (model->layer[0].glow.metadata.input1[i].model_input_type) { - case RTE_ML_IO_TYPE_INT8: - ret = rte_ml_io_float32_to_int8( - qscale, model->layer[0].info.input[i].nb_elements, - lcl_dbuffer, lcl_qbuffer); - break; - case RTE_ML_IO_TYPE_UINT8: - ret = rte_ml_io_float32_to_uint8( - qscale, model->layer[0].info.input[i].nb_elements, - lcl_dbuffer, lcl_qbuffer); - break; - case RTE_ML_IO_TYPE_INT16: - ret = rte_ml_io_float32_to_int16( - qscale, model->layer[0].info.input[i].nb_elements, - lcl_dbuffer, lcl_qbuffer); - break; - case RTE_ML_IO_TYPE_UINT16: - ret = rte_ml_io_float32_to_uint16( - qscale, model->layer[0].info.input[i].nb_elements, - lcl_dbuffer, lcl_qbuffer); - break; - case RTE_ML_IO_TYPE_FP16: - ret = rte_ml_io_float32_to_float16( - model->layer[0].info.input[i].nb_elements, lcl_dbuffer, - lcl_qbuffer); - break; - default: - plt_err("Unsupported model_input_type[%u] : %u", i, - model->layer[0].glow.metadata.input1[i].model_input_type); - ret = -ENOTSUP; - } - if (ret < 0) - return ret; - } - - lcl_dbuffer += model->layer[0].info.input[i].sz_d; - lcl_qbuffer += model->layer[0].info.input[i].sz_q; - } - - return 0; -} - -int -cn10k_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buff_seg **qbuffer, - struct rte_ml_buff_seg **dbuffer) -{ - struct cnxk_ml_model *model; - uint8_t model_output_type; - uint8_t *lcl_qbuffer; - uint8_t *lcl_dbuffer; - uint8_t output_type; - float dscale; - uint32_t i; - uint32_t j; - int ret; - - model = dev->data->models[model_id]; - - if (model == NULL) { - plt_err("Invalid model_id = %u", model_id); - return -EINVAL; - } - - lcl_dbuffer = dbuffer[0]->addr; - lcl_qbuffer = qbuffer[0]->addr; - - for (i = 0; i < model->layer[0].glow.metadata.model.num_output; i++) { - if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - output_type = model->layer[0].glow.metadata.output1[i].output_type; - model_output_type = - model->layer[0].glow.metadata.output1[i].model_output_type; - dscale = model->layer[0].glow.metadata.output1[i].dscale; - } else { - j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - output_type = model->layer[0].glow.metadata.output2[j].output_type; - model_output_type = - model->layer[0].glow.metadata.output2[j].model_output_type; - dscale = model->layer[0].glow.metadata.output2[j].dscale; - } - - if (output_type == model_output_type) { - rte_memcpy(lcl_dbuffer, lcl_qbuffer, model->layer[0].info.output[i].sz_q); - } else { - switch (model->layer[0].glow.metadata.output1[i].model_output_type) { - case RTE_ML_IO_TYPE_INT8: - ret = rte_ml_io_int8_to_float32( - dscale, model->layer[0].info.output[i].nb_elements, - lcl_qbuffer, lcl_dbuffer); - break; - case RTE_ML_IO_TYPE_UINT8: - ret = rte_ml_io_uint8_to_float32( - dscale, model->layer[0].info.output[i].nb_elements, - lcl_qbuffer, lcl_dbuffer); - break; - case RTE_ML_IO_TYPE_INT16: - ret = rte_ml_io_int16_to_float32( - dscale, model->layer[0].info.output[i].nb_elements, - lcl_qbuffer, lcl_dbuffer); - break; - case RTE_ML_IO_TYPE_UINT16: - ret = rte_ml_io_uint16_to_float32( - dscale, model->layer[0].info.output[i].nb_elements, - lcl_qbuffer, lcl_dbuffer); - break; - case RTE_ML_IO_TYPE_FP16: - ret = rte_ml_io_float16_to_float32( - model->layer[0].info.output[i].nb_elements, lcl_qbuffer, - lcl_dbuffer); - break; - default: - plt_err("Unsupported model_output_type[%u] : %u", i, - model->layer[0].glow.metadata.output1[i].model_output_type); - ret = -ENOTSUP; - } - if (ret < 0) - return ret; - } - - lcl_qbuffer += model->layer[0].info.output[i].sz_q; - lcl_dbuffer += model->layer[0].info.output[i].sz_d; - } - - return 0; -} - static __rte_always_inline void queue_index_advance(uint64_t *index, uint64_t nb_desc) { diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index ef12069f0df..780e2a9f9c3 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -320,13 +320,6 @@ int cn10k_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mo int cn10k_ml_model_params_update(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, void *buffer); -/* I/O ops */ -int cn10k_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, - struct rte_ml_buff_seg **dbuffer, struct rte_ml_buff_seg **qbuffer); - -int cn10k_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, - struct rte_ml_buff_seg **qbuffer, struct rte_ml_buff_seg **dbuffer); - /* Fast-path ops */ __rte_hot uint16_t cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, uint16_t nb_ops); diff --git a/drivers/ml/cnxk/cnxk_ml_io.c b/drivers/ml/cnxk/cnxk_ml_io.c new file mode 100644 index 00000000000..c78009ab0cd --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_io.c @@ -0,0 +1,95 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#include + +#include + +#include + +#include "cnxk_ml_io.h" + +inline int +cnxk_ml_io_quantize_single(struct cnxk_ml_io *input, uint8_t *dbuffer, uint8_t *qbuffer) +{ + enum rte_ml_io_type qtype; + enum rte_ml_io_type dtype; + uint32_t nb_elements; + float qscale; + int ret = 0; + + dtype = input->dtype; + qtype = input->qtype; + qscale = input->scale; + nb_elements = input->nb_elements; + + if (dtype == qtype) { + rte_memcpy(qbuffer, dbuffer, input->sz_d); + } else { + switch (qtype) { + case RTE_ML_IO_TYPE_INT8: + ret = rte_ml_io_float32_to_int8(qscale, nb_elements, dbuffer, qbuffer); + break; + case RTE_ML_IO_TYPE_UINT8: + ret = rte_ml_io_float32_to_uint8(qscale, nb_elements, dbuffer, qbuffer); + break; + case RTE_ML_IO_TYPE_INT16: + ret = rte_ml_io_float32_to_int16(qscale, nb_elements, dbuffer, qbuffer); + break; + case RTE_ML_IO_TYPE_UINT16: + ret = rte_ml_io_float32_to_uint16(qscale, nb_elements, dbuffer, qbuffer); + break; + case RTE_ML_IO_TYPE_FP16: + ret = rte_ml_io_float32_to_float16(nb_elements, dbuffer, qbuffer); + break; + default: + plt_err("Unsupported qtype : %u", qtype); + ret = -ENOTSUP; + } + } + + return ret; +} + +inline int +cnxk_ml_io_dequantize_single(struct cnxk_ml_io *output, uint8_t *qbuffer, uint8_t *dbuffer) +{ + enum rte_ml_io_type qtype; + enum rte_ml_io_type dtype; + uint32_t nb_elements; + float dscale; + int ret = 0; + + dtype = output->dtype; + qtype = output->qtype; + dscale = output->scale; + nb_elements = output->nb_elements; + + if (dtype == qtype) { + rte_memcpy(dbuffer, qbuffer, output->sz_q); + } else { + switch (qtype) { + case RTE_ML_IO_TYPE_INT8: + ret = rte_ml_io_int8_to_float32(dscale, nb_elements, qbuffer, dbuffer); + break; + case RTE_ML_IO_TYPE_UINT8: + ret = rte_ml_io_uint8_to_float32(dscale, nb_elements, qbuffer, dbuffer); + break; + case RTE_ML_IO_TYPE_INT16: + ret = rte_ml_io_int16_to_float32(dscale, nb_elements, qbuffer, dbuffer); + break; + case RTE_ML_IO_TYPE_UINT16: + ret = rte_ml_io_uint16_to_float32(dscale, nb_elements, qbuffer, dbuffer); + break; + case RTE_ML_IO_TYPE_FP16: + ret = rte_ml_io_float16_to_float32(nb_elements, qbuffer, dbuffer); + break; + default: + plt_err("Unsupported qtype: %u", qtype); + ret = -ENOTSUP; + } + } + + return ret; +} diff --git a/drivers/ml/cnxk/cnxk_ml_io.h b/drivers/ml/cnxk/cnxk_ml_io.h index 29ec7ec5112..5de166c2520 100644 --- a/drivers/ml/cnxk/cnxk_ml_io.h +++ b/drivers/ml/cnxk/cnxk_ml_io.h @@ -76,4 +76,7 @@ struct cnxk_ml_io_info { uint32_t total_output_sz_d; }; +int cnxk_ml_io_quantize_single(struct cnxk_ml_io *input, uint8_t *dbuffer, uint8_t *qbuffer); +int cnxk_ml_io_dequantize_single(struct cnxk_ml_io *output, uint8_t *qbuffer, uint8_t *dbuffer); + #endif /* _CNXK_ML_IO_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 5ad0ea8c3ce..ed71a551327 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -5,6 +5,8 @@ #include #include +#include + #include "cn10k_ml_ops.h" #include "cnxk_ml_dev.h" @@ -650,6 +652,78 @@ cnxk_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buf return cn10k_ml_model_params_update(cnxk_mldev, model, buffer); } +static int +cnxk_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buff_seg **dbuffer, + struct rte_ml_buff_seg **qbuffer) +{ + struct cnxk_ml_io_info *info = NULL; + struct cnxk_ml_model *model; + uint8_t *lcl_dbuffer; + uint8_t *lcl_qbuffer; + uint32_t i; + int ret; + + if ((dev == NULL) || (dbuffer == NULL) || (qbuffer == NULL)) + return -EINVAL; + + model = dev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + + info = &model->layer[0].info; + + lcl_dbuffer = dbuffer[0]->addr; + lcl_qbuffer = qbuffer[0]->addr; + for (i = 0; i < info->nb_inputs; i++) { + ret = cnxk_ml_io_quantize_single(&info->input[i], lcl_dbuffer, lcl_qbuffer); + if (ret < 0) + return ret; + + lcl_dbuffer += info->input[i].sz_d; + lcl_qbuffer += info->input[i].sz_q; + } + + return 0; +} + +static int +cnxk_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buff_seg **qbuffer, + struct rte_ml_buff_seg **dbuffer) +{ + struct cnxk_ml_io_info *info = NULL; + struct cnxk_ml_model *model; + uint8_t *lcl_qbuffer; + uint8_t *lcl_dbuffer; + uint32_t i; + int ret; + + if ((dev == NULL) || (qbuffer == NULL) || (dbuffer == NULL)) + return -EINVAL; + + model = dev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + + info = &model->layer[model->nb_layers - 1].info; + + lcl_qbuffer = qbuffer[0]->addr; + lcl_dbuffer = dbuffer[0]->addr; + for (i = 0; i < info->nb_outputs; i++) { + ret = cnxk_ml_io_dequantize_single(&info->output[i], lcl_qbuffer, lcl_dbuffer); + if (ret < 0) + return ret; + + lcl_qbuffer += info->output[i].sz_q; + lcl_dbuffer += info->output[i].sz_d; + } + + return 0; +} + struct rte_ml_dev_ops cnxk_ml_ops = { /* Device control ops */ .dev_info_get = cnxk_ml_dev_info_get, @@ -681,6 +755,6 @@ struct rte_ml_dev_ops cnxk_ml_ops = { .model_params_update = cnxk_ml_model_params_update, /* I/O ops */ - .io_quantize = cn10k_ml_io_quantize, - .io_dequantize = cn10k_ml_io_dequantize, + .io_quantize = cnxk_ml_io_quantize, + .io_dequantize = cnxk_ml_io_dequantize, }; diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 6385ac45481..9cc4ddec702 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -25,6 +25,7 @@ sources = files( 'cn10k_ml_model.c', 'cn10k_ml_ocm.c', 'cnxk_ml_dev.c', + 'cnxk_ml_io.c', 'cnxk_ml_model.c', 'cnxk_ml_ops.c', ) From patchwork Wed Aug 30 15:59:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130895 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 83D9A41FD1; Wed, 30 Aug 2023 18:02:18 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D918A40A6D; Wed, 30 Aug 2023 17:59:53 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 34767402A3 for ; Wed, 30 Aug 2023 17:59:39 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I6w012554 for ; Wed, 30 Aug 2023 08:59:38 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=5mJuwvam7TC1qVw1wHr5FVKv2RZjfCJoVMftmf94veY=; b=DTsSO2+lecgYIRbNdvr/Dl2O29ljU+KYnFtV7kQ5SGJtFYZTxoX1unf+aoN0kc+yzPnK v7lCtKTc7pvoLgcd/RfDtZTqOJ4jRWKF0btcg8VcMfuPMSw9M9WQlzJ2picIlSt4gqIM KSSWEu4l6jfBU9T20Mjtf87Yq0dN7vPnSkc/DSGEM5adGI0oxPCdLTKOS1PjzDWoVwZP LQc8zwtEDb9FDlEJfDOWpO7Xc0/ptNtdEOJy3cwH3BmfejaiwfuBsb95cuh3FuiZqgs+ kC1WNB9tAfdyKf/+SEUkY/hzXax3tBPMJ1Tpy8rK/gUOcpxgmfVkGqJFoPK5nLN4tWHk mQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:38 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:36 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:36 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 7FE663F707F; Wed, 30 Aug 2023 08:59:36 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 14/34] ml/cnxk: update device debug functions Date: Wed, 30 Aug 2023 08:59:04 -0700 Message-ID: <20230830155927.3566-15-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: zCD1j-gZGW1o1uUfNf6JHAp83tO_dmLK X-Proofpoint-GUID: zCD1j-gZGW1o1uUfNf6JHAp83tO_dmLK X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added cnxk wrapper for device dump and selftest debug functions. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_model.c | 118 +++++++++++++++++++++ drivers/ml/cnxk/cn10k_ml_model.h | 1 + drivers/ml/cnxk/cn10k_ml_ocm.c | 8 +- drivers/ml/cnxk/cn10k_ml_ocm.h | 2 +- drivers/ml/cnxk/cn10k_ml_ops.c | 176 ++----------------------------- drivers/ml/cnxk/cn10k_ml_ops.h | 4 +- drivers/ml/cnxk/cnxk_ml_model.c | 33 ++++++ drivers/ml/cnxk/cnxk_ml_model.h | 2 + drivers/ml/cnxk/cnxk_ml_ops.c | 39 ++++++- drivers/ml/cnxk/cnxk_ml_utils.c | 15 +++ drivers/ml/cnxk/cnxk_ml_utils.h | 17 +++ drivers/ml/cnxk/meson.build | 2 + 12 files changed, 236 insertions(+), 181 deletions(-) create mode 100644 drivers/ml/cnxk/cnxk_ml_utils.c create mode 100644 drivers/ml/cnxk/cnxk_ml_utils.h diff --git a/drivers/ml/cnxk/cn10k_ml_model.c b/drivers/ml/cnxk/cn10k_ml_model.c index 9a336cd18f9..9e92d4acf36 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.c +++ b/drivers/ml/cnxk/cn10k_ml_model.c @@ -12,6 +12,7 @@ #include "cnxk_ml_dev.h" #include "cnxk_ml_model.h" #include "cnxk_ml_ops.h" +#include "cnxk_ml_utils.h" static enum rte_ml_io_type cn10k_ml_io_type_map(uint8_t type) @@ -591,3 +592,120 @@ cn10k_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mo rte_ml_io_type_size_get(io_info->output[i].qtype); } } + +void +cn10k_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp) +{ + struct cn10k_ml_ocm *ocm; + char str[STR_LEN]; + uint8_t i; + uint8_t j; + + ocm = &cnxk_mldev->cn10k_mldev.ocm; + + /* Print debug info */ + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, " Layer Information (Layer ID: %u, Name: %s)\n", + cnxk_mldev->index_map[layer->index].layer_id, layer->name); + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "index", layer->index); + fprintf(fp, "%*s : %s\n", FIELD_LEN, "name", layer->name); + fprintf(fp, "%*s : %u.%u.%u.%u\n", FIELD_LEN, "version", + layer->glow.metadata.model.version[0], layer->glow.metadata.model.version[1], + layer->glow.metadata.model.version[2], layer->glow.metadata.model.version[3]); + fprintf(fp, "%*s : 0x%016lx\n", FIELD_LEN, "layer", PLT_U64_CAST(layer)); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "batch_size", layer->batch_size); + + /* Print model state */ + if (layer->state == ML_CNXK_LAYER_STATE_LOADED) + fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "loaded"); + if (layer->state == ML_CNXK_LAYER_STATE_JOB_ACTIVE) + fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "job_active"); + if (layer->state == ML_CNXK_LAYER_STATE_STARTED) + fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "started"); + + /* Print OCM status */ + fprintf(fp, "%*s : %" PRIu64 " bytes\n", FIELD_LEN, "wb_size", + layer->glow.metadata.model.ocm_wb_range_end - + layer->glow.metadata.model.ocm_wb_range_start + 1); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "wb_pages", layer->glow.ocm_map.wb_pages); + fprintf(fp, "%*s : %" PRIu64 " bytes\n", FIELD_LEN, "scratch_size", + ocm->size_per_tile - layer->glow.metadata.model.ocm_tmp_range_floor); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "scratch_pages", layer->glow.ocm_map.scratch_pages); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_tiles", + layer->glow.metadata.model.tile_end - layer->glow.metadata.model.tile_start + 1); + + if (layer->state == ML_CNXK_LAYER_STATE_STARTED) { + fprintf(fp, "%*s : 0x%0*" PRIx64 "\n", FIELD_LEN, "tilemask", + ML_CN10K_OCM_NUMTILES / 4, layer->glow.ocm_map.tilemask); + fprintf(fp, "%*s : 0x%" PRIx64 "\n", FIELD_LEN, "ocm_wb_start", + layer->glow.ocm_map.wb_page_start * ocm->page_size); + } + + fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_inputs", layer->glow.metadata.model.num_input); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_outputs", layer->glow.metadata.model.num_output); + fprintf(fp, "\n"); + + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "%8s %16s %12s %18s\n", "input", "input_name", "input_type", + "model_input_type"); + cnxk_ml_print_line(fp, LINE_LEN); + for (i = 0; i < layer->glow.metadata.model.num_input; i++) { + if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { + fprintf(fp, "%8u ", i); + fprintf(fp, "%*s ", 16, layer->glow.metadata.input1[i].input_name); + rte_ml_io_type_to_str(layer->glow.metadata.input1[i].input_type, str, + STR_LEN); + fprintf(fp, "%*s ", 12, str); + rte_ml_io_type_to_str(layer->glow.metadata.input1[i].model_input_type, str, + STR_LEN); + fprintf(fp, "%*s ", 18, str); + fprintf(fp, "\n"); + } else { + j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; + + fprintf(fp, "%8u ", i); + fprintf(fp, "%*s ", 16, layer->glow.metadata.input2[j].input_name); + rte_ml_io_type_to_str(layer->glow.metadata.input2[j].input_type, str, + STR_LEN); + fprintf(fp, "%*s ", 12, str); + rte_ml_io_type_to_str(layer->glow.metadata.input2[j].model_input_type, str, + STR_LEN); + fprintf(fp, "%*s ", 18, str); + fprintf(fp, "\n"); + } + } + fprintf(fp, "\n"); + + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "%8s %16s %12s %18s\n", "output", "output_name", "output_type", + "model_output_type"); + cnxk_ml_print_line(fp, LINE_LEN); + for (i = 0; i < layer->glow.metadata.model.num_output; i++) { + if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { + fprintf(fp, "%8u ", i); + fprintf(fp, "%*s ", 16, layer->glow.metadata.output1[i].output_name); + rte_ml_io_type_to_str(layer->glow.metadata.output1[i].output_type, str, + STR_LEN); + fprintf(fp, "%*s ", 12, str); + rte_ml_io_type_to_str(layer->glow.metadata.output1[i].model_output_type, + str, STR_LEN); + fprintf(fp, "%*s ", 18, str); + fprintf(fp, "\n"); + } else { + j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; + fprintf(fp, "%8u ", i); + fprintf(fp, "%*s ", 16, layer->glow.metadata.output2[j].output_name); + rte_ml_io_type_to_str(layer->glow.metadata.output2[j].output_type, str, + STR_LEN); + fprintf(fp, "%*s ", 12, str); + rte_ml_io_type_to_str(layer->glow.metadata.output2[j].model_output_type, + str, STR_LEN); + fprintf(fp, "%*s ", 18, str); + fprintf(fp, "\n"); + } + } + fprintf(fp, "\n"); + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "\n"); +} diff --git a/drivers/ml/cnxk/cn10k_ml_model.h b/drivers/ml/cnxk/cn10k_ml_model.h index 45290b84cef..8717e7de3ec 100644 --- a/drivers/ml/cnxk/cn10k_ml_model.h +++ b/drivers/ml/cnxk/cn10k_ml_model.h @@ -459,5 +459,6 @@ int cn10k_ml_model_ocm_pages_count(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_m void cn10k_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, struct cnxk_ml_io_info *io_info, struct cn10k_ml_model_metadata *metadata); +void cn10k_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp); #endif /* _CN10K_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/cn10k_ml_ocm.c b/drivers/ml/cnxk/cn10k_ml_ocm.c index 6a8400b7763..7d4b1efad13 100644 --- a/drivers/ml/cnxk/cn10k_ml_ocm.c +++ b/drivers/ml/cnxk/cn10k_ml_ocm.c @@ -483,19 +483,15 @@ cn10k_ml_ocm_pagemask_to_str(struct cn10k_ml_ocm_tile_info *tile_info, uint16_t } void -cn10k_ml_ocm_print(struct rte_ml_dev *dev, FILE *fp) +cn10k_ml_ocm_print(struct cnxk_ml_dev *cnxk_mldev, FILE *fp) { - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; struct cn10k_ml_ocm *ocm; uint8_t tile_id; uint8_t word_id; int wb_pages; char *str; - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - ocm = &cn10k_mldev->ocm; + ocm = &cnxk_mldev->cn10k_mldev.ocm; /* Nibbles + prefix '0x' */ str = rte_zmalloc("ocm_mask_str", ocm->num_pages / 4 + 2, RTE_CACHE_LINE_SIZE); diff --git a/drivers/ml/cnxk/cn10k_ml_ocm.h b/drivers/ml/cnxk/cn10k_ml_ocm.h index 97b723a56a5..bf8944f8eee 100644 --- a/drivers/ml/cnxk/cn10k_ml_ocm.h +++ b/drivers/ml/cnxk/cn10k_ml_ocm.h @@ -83,6 +83,6 @@ void cn10k_ml_ocm_reserve_pages(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_i uint16_t layer_id, uint64_t tilemask, int wb_page_start, uint16_t wb_pages, uint16_t scratch_pages); void cn10k_ml_ocm_free_pages(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id, uint16_t layer_id); -void cn10k_ml_ocm_print(struct rte_ml_dev *dev, FILE *fp); +void cn10k_ml_ocm_print(struct cnxk_ml_dev *cnxk_mldev, FILE *fp); #endif /* _CN10K_ML_OCM_H_ */ diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 1e6aee818c7..c3608eec99e 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -22,11 +22,6 @@ /* ML layer macros */ #define CN10K_ML_LAYER_MEMZONE_NAME "ml_cn10k_layer_mz" -/* Debug print width */ -#define STR_LEN 12 -#define FIELD_LEN 16 -#define LINE_LEN 90 - /* ML Job descriptor flags */ #define ML_FLAGS_POLL_COMPL BIT(0) #define ML_FLAGS_SSO_COMPL BIT(1) @@ -74,16 +69,6 @@ static const struct cn10k_ml_stype_db_driver { {ML_DRIVER_ERR_FW_ERROR, "UNKNOWN FIRMWARE ERROR"}, }; -static void -print_line(FILE *fp, int len) -{ - int i; - - for (i = 0; i < len; i++) - fprintf(fp, "-"); - fprintf(fp, "\n"); -} - static inline void cn10k_ml_set_poll_addr(struct cnxk_ml_req *req) { @@ -117,140 +102,6 @@ cn10k_ml_qp_initialize(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_qp *qp) } } -static void -cn10k_ml_model_print(struct rte_ml_dev *dev, uint16_t model_id, FILE *fp) -{ - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - struct cnxk_ml_model *model; - struct cn10k_ml_ocm *ocm; - char str[STR_LEN]; - uint8_t i; - uint8_t j; - - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - ocm = &cn10k_mldev->ocm; - model = dev->data->models[model_id]; - - /* Print debug info */ - print_line(fp, LINE_LEN); - fprintf(fp, " Model Information (%s)\n", model->glow.metadata.model.name); - print_line(fp, LINE_LEN); - fprintf(fp, "%*s : %s\n", FIELD_LEN, "name", model->glow.metadata.model.name); - fprintf(fp, "%*s : %u.%u.%u.%u\n", FIELD_LEN, "version", - model->glow.metadata.model.version[0], model->glow.metadata.model.version[1], - model->glow.metadata.model.version[2], model->glow.metadata.model.version[3]); - if (strlen(model->name) != 0) - fprintf(fp, "%*s : %s\n", FIELD_LEN, "debug_name", model->name); - fprintf(fp, "%*s : 0x%016lx\n", FIELD_LEN, "model", PLT_U64_CAST(model)); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "model_id", model->model_id); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "batch_size", model->glow.metadata.model.batch_size); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_layers", model->glow.metadata.model.num_layers); - - /* Print model state */ - if (model->state == ML_CNXK_MODEL_STATE_LOADED) - fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "loaded"); - if (model->state == ML_CNXK_MODEL_STATE_JOB_ACTIVE) - fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "job_active"); - if (model->state == ML_CNXK_MODEL_STATE_STARTED) - fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "started"); - - /* Print OCM status */ - fprintf(fp, "%*s : %" PRIu64 " bytes\n", FIELD_LEN, "wb_size", - model->glow.metadata.model.ocm_wb_range_end - - model->glow.metadata.model.ocm_wb_range_start + 1); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "wb_pages", model->layer[0].glow.ocm_map.wb_pages); - fprintf(fp, "%*s : %" PRIu64 " bytes\n", FIELD_LEN, "scratch_size", - ocm->size_per_tile - model->glow.metadata.model.ocm_tmp_range_floor); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "scratch_pages", - model->layer[0].glow.ocm_map.scratch_pages); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_tiles", - model->glow.metadata.model.tile_end - model->glow.metadata.model.tile_start + 1); - - if (model->state == ML_CNXK_MODEL_STATE_STARTED) { - fprintf(fp, "%*s : 0x%0*" PRIx64 "\n", FIELD_LEN, "tilemask", - ML_CN10K_OCM_NUMTILES / 4, model->layer[0].glow.ocm_map.tilemask); - fprintf(fp, "%*s : 0x%" PRIx64 "\n", FIELD_LEN, "ocm_wb_start", - model->layer[0].glow.ocm_map.wb_page_start * cn10k_mldev->ocm.page_size); - } - - fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_inputs", model->glow.metadata.model.num_input); - fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_outputs", model->glow.metadata.model.num_output); - fprintf(fp, "\n"); - - print_line(fp, LINE_LEN); - fprintf(fp, "%8s %16s %12s %18s %12s\n", "input", "input_name", "input_type", - "model_input_type", "quantize"); - print_line(fp, LINE_LEN); - for (i = 0; i < model->glow.metadata.model.num_input; i++) { - if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - fprintf(fp, "%8u ", i); - fprintf(fp, "%*s ", 16, model->glow.metadata.input1[i].input_name); - rte_ml_io_type_to_str(model->glow.metadata.input1[i].input_type, str, - STR_LEN); - fprintf(fp, "%*s ", 12, str); - rte_ml_io_type_to_str(model->glow.metadata.input1[i].model_input_type, str, - STR_LEN); - fprintf(fp, "%*s ", 18, str); - fprintf(fp, "%*s", 12, - (model->glow.metadata.input1[i].quantize == 1 ? "Yes" : "No")); - fprintf(fp, "\n"); - } else { - j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - - fprintf(fp, "%8u ", i); - fprintf(fp, "%*s ", 16, model->glow.metadata.input2[j].input_name); - rte_ml_io_type_to_str(model->glow.metadata.input2[j].input_type, str, - STR_LEN); - fprintf(fp, "%*s ", 12, str); - rte_ml_io_type_to_str(model->glow.metadata.input2[j].model_input_type, str, - STR_LEN); - fprintf(fp, "%*s ", 18, str); - fprintf(fp, "%*s", 12, - (model->glow.metadata.input2[j].quantize == 1 ? "Yes" : "No")); - fprintf(fp, "\n"); - } - } - fprintf(fp, "\n"); - - print_line(fp, LINE_LEN); - fprintf(fp, "%8s %16s %12s %18s %12s\n", "output", "output_name", "output_type", - "model_output_type", "dequantize"); - print_line(fp, LINE_LEN); - for (i = 0; i < model->glow.metadata.model.num_output; i++) { - if (i < MRVL_ML_NUM_INPUT_OUTPUT_1) { - fprintf(fp, "%8u ", i); - fprintf(fp, "%*s ", 16, model->glow.metadata.output1[i].output_name); - rte_ml_io_type_to_str(model->glow.metadata.output1[i].output_type, str, - STR_LEN); - fprintf(fp, "%*s ", 12, str); - rte_ml_io_type_to_str(model->glow.metadata.output1[i].model_output_type, - str, STR_LEN); - fprintf(fp, "%*s ", 18, str); - fprintf(fp, "%*s", 12, - (model->glow.metadata.output1[i].dequantize == 1 ? "Yes" : "No")); - fprintf(fp, "\n"); - } else { - j = i - MRVL_ML_NUM_INPUT_OUTPUT_1; - fprintf(fp, "%8u ", i); - fprintf(fp, "%*s ", 16, model->glow.metadata.output2[j].output_name); - rte_ml_io_type_to_str(model->glow.metadata.output2[j].output_type, str, - STR_LEN); - fprintf(fp, "%*s ", 12, str); - rte_ml_io_type_to_str(model->glow.metadata.output2[j].model_output_type, - str, STR_LEN); - fprintf(fp, "%*s ", 18, str); - fprintf(fp, "%*s", 12, - (model->glow.metadata.output2[j].dequantize == 1 ? "Yes" : "No")); - fprintf(fp, "\n"); - } - } - fprintf(fp, "\n"); - print_line(fp, LINE_LEN); - fprintf(fp, "\n"); -} - static void cn10k_ml_prep_sp_job_descriptor(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, struct cnxk_ml_req *req, enum cn10k_ml_job_type job_type) @@ -1124,38 +975,25 @@ cn10k_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mo } int -cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) +cn10k_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp) { struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - struct cnxk_ml_model *model; struct cn10k_ml_fw *fw; uint32_t head_loc; uint32_t tail_loc; - uint16_t model_id; uint32_t bufsize; char *head_ptr; int core_id; - if (roc_env_is_asim()) - return 0; - - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; fw = &cn10k_mldev->fw; - /* Dump model info */ - for (model_id = 0; model_id < dev->data->nb_models; model_id++) { - model = dev->data->models[model_id]; - if (model != NULL) { - cn10k_ml_model_print(dev, model_id, fp); - fprintf(fp, "\n"); - } - } - /* Dump OCM state */ - cn10k_ml_ocm_print(dev, fp); + cn10k_ml_ocm_print(cnxk_mldev, fp); + + if (roc_env_is_asim()) + return 0; /* Dump debug buffer */ for (core_id = 0; core_id <= 1; core_id++) { @@ -1211,17 +1049,15 @@ cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) } int -cn10k_ml_dev_selftest(struct rte_ml_dev *dev) +cn10k_ml_dev_selftest(struct cnxk_ml_dev *cnxk_mldev) { struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; const struct plt_memzone *mz; struct cnxk_ml_req *req; uint64_t timeout_cycle; bool timeout; int ret; - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; mz = plt_memzone_reserve_aligned("dev_selftest", sizeof(struct cnxk_ml_req), 0, ML_CN10K_ALIGN_SIZE); diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 780e2a9f9c3..5fda98ae88f 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -295,8 +295,8 @@ int cn10k_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_d int cn10k_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev); int cn10k_ml_dev_start(struct cnxk_ml_dev *cnxk_mldev); int cn10k_ml_dev_stop(struct cnxk_ml_dev *cnxk_mldev); -int cn10k_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp); -int cn10k_ml_dev_selftest(struct rte_ml_dev *dev); +int cn10k_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp); +int cn10k_ml_dev_selftest(struct cnxk_ml_dev *cnxk_mldev); int cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats); void cn10k_ml_dev_stats_reset(struct rte_ml_dev *dev); diff --git a/drivers/ml/cnxk/cnxk_ml_model.c b/drivers/ml/cnxk/cnxk_ml_model.c index 3d735ced3e1..b069d4e3a5a 100644 --- a/drivers/ml/cnxk/cnxk_ml_model.c +++ b/drivers/ml/cnxk/cnxk_ml_model.c @@ -5,3 +5,36 @@ #include #include "cnxk_ml_model.h" +#include "cnxk_ml_utils.h" + +void +cnxk_ml_model_dump(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, FILE *fp) +{ + struct cnxk_ml_layer *layer; + uint16_t layer_id; + + /* Print debug info */ + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, " Model Information (Model ID: %u, Name: %s)\n", model->model_id, model->name); + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "model_id", model->model_id); + fprintf(fp, "%*s : %s\n", FIELD_LEN, "name", model->name); + fprintf(fp, "%*s : 0x%016lx\n", FIELD_LEN, "model", PLT_U64_CAST(model)); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "batch_size", model->batch_size); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "nb_layers", model->nb_layers); + + /* Print model state */ + if (model->state == ML_CNXK_MODEL_STATE_LOADED) + fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "loaded"); + if (model->state == ML_CNXK_MODEL_STATE_JOB_ACTIVE) + fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "job_active"); + if (model->state == ML_CNXK_MODEL_STATE_STARTED) + fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "started"); + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "\n"); + + for (layer_id = 0; layer_id < model->nb_layers; layer_id++) { + layer = &model->layer[layer_id]; + cn10k_ml_layer_print(cnxk_mldev, layer, fp); + } +} diff --git a/drivers/ml/cnxk/cnxk_ml_model.h b/drivers/ml/cnxk/cnxk_ml_model.h index a2994dbb71e..66d979dd3ca 100644 --- a/drivers/ml/cnxk/cnxk_ml_model.h +++ b/drivers/ml/cnxk/cnxk_ml_model.h @@ -108,4 +108,6 @@ struct cnxk_ml_model { plt_spinlock_t lock; }; +void cnxk_ml_model_dump(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, FILE *fp); + #endif /* _CNXK_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index ed71a551327..b49ab597984 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -411,6 +411,41 @@ cnxk_ml_dev_stop(struct rte_ml_dev *dev) return 0; } +static int +cnxk_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + uint16_t model_id; + + if ((dev == NULL) || (fp == NULL)) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + /* Dump model info */ + for (model_id = 0; model_id < cnxk_mldev->mldev->data->nb_models; model_id++) { + model = cnxk_mldev->mldev->data->models[model_id]; + if (model != NULL) + cnxk_ml_model_dump(cnxk_mldev, model, fp); + } + + return cn10k_ml_dev_dump(cnxk_mldev, fp); +} + +static int +cnxk_ml_dev_selftest(struct rte_ml_dev *dev) +{ + struct cnxk_ml_dev *cnxk_mldev; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + return cn10k_ml_dev_selftest(cnxk_mldev); +} + static int cnxk_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, const struct rte_ml_dev_qp_conf *qp_conf, int socket_id) @@ -731,8 +766,8 @@ struct rte_ml_dev_ops cnxk_ml_ops = { .dev_close = cnxk_ml_dev_close, .dev_start = cnxk_ml_dev_start, .dev_stop = cnxk_ml_dev_stop, - .dev_dump = cn10k_ml_dev_dump, - .dev_selftest = cn10k_ml_dev_selftest, + .dev_dump = cnxk_ml_dev_dump, + .dev_selftest = cnxk_ml_dev_selftest, /* Queue-pair handling ops */ .dev_queue_pair_setup = cnxk_ml_dev_queue_pair_setup, diff --git a/drivers/ml/cnxk/cnxk_ml_utils.c b/drivers/ml/cnxk/cnxk_ml_utils.c new file mode 100644 index 00000000000..ca3670a9e83 --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_utils.c @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#include "cnxk_ml_utils.h" + +void +cnxk_ml_print_line(FILE *fp, int len) +{ + int i; + + for (i = 0; i < len; i++) + fprintf(fp, "-"); + fprintf(fp, "\n"); +} diff --git a/drivers/ml/cnxk/cnxk_ml_utils.h b/drivers/ml/cnxk/cnxk_ml_utils.h new file mode 100644 index 00000000000..ed2ab213469 --- /dev/null +++ b/drivers/ml/cnxk/cnxk_ml_utils.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#ifndef _CNXK_ML_UTILS_H_ +#define _CNXK_ML_UTILS_H_ + +#include + +/* Debug print width */ +#define STR_LEN 12 +#define FIELD_LEN 16 +#define LINE_LEN 72 + +void cnxk_ml_print_line(FILE *fp, int len); + +#endif /* _CNXK_ML_UTILS_H_ */ diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 9cc4ddec702..575f08f9c09 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -17,6 +17,7 @@ driver_sdk_headers = files( 'cnxk_ml_model.h', 'cnxk_ml_ops.h', 'cnxk_ml_xstats.h', + 'cnxk_ml_utils.h', ) sources = files( @@ -28,6 +29,7 @@ sources = files( 'cnxk_ml_io.c', 'cnxk_ml_model.c', 'cnxk_ml_ops.c', + 'cnxk_ml_utils.c', ) deps += ['mldev', 'common_cnxk', 'kvargs', 'hash'] From patchwork Wed Aug 30 15:59:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130896 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 775C141FD1; Wed, 30 Aug 2023 18:02:28 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F1294402DA; Wed, 30 Aug 2023 17:59:54 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id B0375402A9 for ; Wed, 30 Aug 2023 17:59:39 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I6x012554 for ; Wed, 30 Aug 2023 08:59:39 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=FAg+HrXWY3I7mtJZrFUl2ENX1X8VERGXmRZjrc41Ris=; b=RekOkqxKCOPx4kSKXsGhhyHQsvnCqSrSUiP75g6z8UNagqL869VcovC/fJxS/onvAyNT xVTHCRnvjQPQwiukdej12ermjqhNu+9zn1gxCEF6cUOnuITr0T76H0W3b+byuR+KyIYr DeM19Ne7r4vhIUPNzaVSgYkWPjIe3/OklBeOZHzA8t0KE7NzI/Y20TomZjLUQWxydU4/ LlSAp9y5x9DFmgXQYwxiihCfYcdMOcCxA7qqTRUGy7RfTTc6qGouj7qPEp0U5pk7SdFq eyPfsw/YlQZzpDCBgDinD05Lctg1kVXdpMy9jhhP1V71YuZVWSEN1g+yqwCZAzYaUjDT hQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:39 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:37 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:37 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id CC70D3F707D; Wed, 30 Aug 2023 08:59:36 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 15/34] ml/cnxk: update device stats functions Date: Wed, 30 Aug 2023 08:59:05 -0700 Message-ID: <20230830155927.3566-16-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: IPuY_otEcfteylzQlhcmYDYjvgHnhrZ3 X-Proofpoint-GUID: IPuY_otEcfteylzQlhcmYDYjvgHnhrZ3 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added cnxk wrapper function to handle ML device stats Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ops.c | 32 ------------------------------ drivers/ml/cnxk/cn10k_ml_ops.h | 2 -- drivers/ml/cnxk/cnxk_ml_ops.c | 36 ++++++++++++++++++++++++++++++++-- 3 files changed, 34 insertions(+), 36 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index c3608eec99e..59cd3bb9b34 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -774,38 +774,6 @@ cn10k_ml_dev_stop(struct cnxk_ml_dev *cnxk_mldev) return 0; } -int -cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats) -{ - struct cnxk_ml_qp *qp; - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - qp = dev->data->queue_pairs[qp_id]; - stats->enqueued_count += qp->stats.enqueued_count; - stats->dequeued_count += qp->stats.dequeued_count; - stats->enqueue_err_count += qp->stats.enqueue_err_count; - stats->dequeue_err_count += qp->stats.dequeue_err_count; - } - - return 0; -} - -void -cn10k_ml_dev_stats_reset(struct rte_ml_dev *dev) -{ - struct cnxk_ml_qp *qp; - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - qp = dev->data->queue_pairs[qp_id]; - qp->stats.enqueued_count = 0; - qp->stats.dequeued_count = 0; - qp->stats.enqueue_err_count = 0; - qp->stats.dequeue_err_count = 0; - } -} - int cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, int32_t model_id, struct rte_ml_dev_xstats_map *xstats_map, diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 5fda98ae88f..47e7cb12af4 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -298,8 +298,6 @@ int cn10k_ml_dev_stop(struct cnxk_ml_dev *cnxk_mldev); int cn10k_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp); int cn10k_ml_dev_selftest(struct cnxk_ml_dev *cnxk_mldev); -int cn10k_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats); -void cn10k_ml_dev_stats_reset(struct rte_ml_dev *dev); int cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, int32_t model_id, struct rte_ml_dev_xstats_map *xstats_map, uint32_t size); diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index b49ab597984..ffeb3f44523 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -491,6 +491,38 @@ cnxk_ml_dev_queue_pair_setup(struct rte_ml_dev *dev, uint16_t queue_pair_id, return 0; } +static int +cnxk_ml_dev_stats_get(struct rte_ml_dev *dev, struct rte_ml_dev_stats *stats) +{ + struct cnxk_ml_qp *qp; + int qp_id; + + for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { + qp = dev->data->queue_pairs[qp_id]; + stats->enqueued_count += qp->stats.enqueued_count; + stats->dequeued_count += qp->stats.dequeued_count; + stats->enqueue_err_count += qp->stats.enqueue_err_count; + stats->dequeue_err_count += qp->stats.dequeue_err_count; + } + + return 0; +} + +static void +cnxk_ml_dev_stats_reset(struct rte_ml_dev *dev) +{ + struct cnxk_ml_qp *qp; + int qp_id; + + for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { + qp = dev->data->queue_pairs[qp_id]; + qp->stats.enqueued_count = 0; + qp->stats.dequeued_count = 0; + qp->stats.enqueue_err_count = 0; + qp->stats.dequeue_err_count = 0; + } +} + static int cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, uint16_t *model_id) { @@ -774,8 +806,8 @@ struct rte_ml_dev_ops cnxk_ml_ops = { .dev_queue_pair_release = cnxk_ml_dev_queue_pair_release, /* Stats ops */ - .dev_stats_get = cn10k_ml_dev_stats_get, - .dev_stats_reset = cn10k_ml_dev_stats_reset, + .dev_stats_get = cnxk_ml_dev_stats_get, + .dev_stats_reset = cnxk_ml_dev_stats_reset, .dev_xstats_names_get = cn10k_ml_dev_xstats_names_get, .dev_xstats_by_name_get = cn10k_ml_dev_xstats_by_name_get, .dev_xstats_get = cn10k_ml_dev_xstats_get, From patchwork Wed Aug 30 15:59:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130897 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 761B741FD1; Wed, 30 Aug 2023 18:02:38 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F0E95402E9; Wed, 30 Aug 2023 17:59:55 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 5A10940293 for ; Wed, 30 Aug 2023 17:59:40 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I70012554 for ; Wed, 30 Aug 2023 08:59:39 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=JZDMTVI3kwLDVS23RmYGeQ5Tv667lnV5XHsiXP3xcH4=; b=g/qq7EAisKWDihWUQPCt092DgsY/t95FHJyBD6nQnPE4dFRcAt75GWIcEbWe6K9xdLie JNmyiAUGWE7pY3YE77up76Z4hASHzt1GP0OpJLyPR3NV6gL4jSZY6G2I2xMbfGwY82P7 +SavR+Fo/3xFqX/DFCyuvQ1ShqXUW7W3tks27OnxgIAOygjMYEIpJLtXFQacYsN5wDuJ BWfEyLjK75AR0+6uCxPi3oL9yvOUrJsSnPnq9Iw2PeLrqZ/nfTSAriKA6sGAA9HpxBDd +WuI47JsflKxVywLZLRLrq/pil6Ai5rUA2r9a46oEP4hYjOzVPH/mIkjwzF1lYV6Wd50 QA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:39 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:37 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:37 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 2B4DF3F7080; Wed, 30 Aug 2023 08:59:37 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 16/34] ml/cnxk: update device and model xstats functions Date: Wed, 30 Aug 2023 08:59:06 -0700 Message-ID: <20230830155927.3566-17-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: ERWnU6v2EUNLccuKPrjYBSbxenS78yH0 X-Proofpoint-GUID: ERWnU6v2EUNLccuKPrjYBSbxenS78yH0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added cnxk wrapper function to handle ML device and model extended stats. Handling resources for the xstats is done in the cnxk layer. Introduced internal xstats group. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.h | 4 - drivers/ml/cnxk/cn10k_ml_ops.c | 542 +------------------------------ drivers/ml/cnxk/cn10k_ml_ops.h | 11 - drivers/ml/cnxk/cnxk_ml_dev.h | 5 + drivers/ml/cnxk/cnxk_ml_ops.c | 540 +++++++++++++++++++++++++++++- drivers/ml/cnxk/cnxk_ml_xstats.h | 21 +- 6 files changed, 571 insertions(+), 552 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_dev.h b/drivers/ml/cnxk/cn10k_ml_dev.h index be989e0a207..bde9d089015 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.h +++ b/drivers/ml/cnxk/cn10k_ml_dev.h @@ -10,7 +10,6 @@ #include "cn10k_ml_ocm.h" #include "cnxk_ml_io.h" -#include "cnxk_ml_xstats.h" /* Dummy Device ops */ extern struct rte_ml_dev_ops ml_dev_dummy_ops; @@ -133,9 +132,6 @@ struct cn10k_ml_dev { /* OCM info */ struct cn10k_ml_ocm ocm; - /* Extended stats data */ - struct cnxk_ml_xstats xstats; - /* Enable / disable model data caching */ int cache_model_data; diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 59cd3bb9b34..f1431b89a2d 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -202,107 +202,21 @@ cn10k_ml_prep_fp_job_descriptor(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_r req->cn10k_req.jd.model_run.num_batches = op->nb_batches; } -static int -cn10k_ml_xstats_init(struct rte_ml_dev *dev) -{ - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - uint16_t nb_stats; - uint16_t stat_id; - uint16_t model; - uint16_t i; - - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - - /* Allocate memory for xstats entries. Don't allocate during reconfigure */ - nb_stats = RTE_DIM(device_xstats) + ML_CNXK_MAX_MODELS * RTE_DIM(layer_xstats); - if (cn10k_mldev->xstats.entries == NULL) - cn10k_mldev->xstats.entries = rte_zmalloc( - "cn10k_ml_xstats", sizeof(struct cnxk_ml_xstats_entry) * nb_stats, - PLT_CACHE_LINE_SIZE); - - if (cn10k_mldev->xstats.entries == NULL) - return -ENOMEM; - - /* Initialize device xstats */ - stat_id = 0; - for (i = 0; i < RTE_DIM(device_xstats); i++) { - cn10k_mldev->xstats.entries[stat_id].map.id = stat_id; - snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, - sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), "%s", - device_xstats[i].name); - - cn10k_mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_DEVICE; - cn10k_mldev->xstats.entries[stat_id].type = device_xstats[i].type; - cn10k_mldev->xstats.entries[stat_id].fn_id = CNXK_ML_XSTATS_FN_DEVICE; - cn10k_mldev->xstats.entries[stat_id].obj_idx = 0; - cn10k_mldev->xstats.entries[stat_id].reset_allowed = device_xstats[i].reset_allowed; - stat_id++; - } - cn10k_mldev->xstats.count_mode_device = stat_id; - - /* Initialize model xstats */ - for (model = 0; model < ML_CNXK_MAX_MODELS; model++) { - cn10k_mldev->xstats.offset_for_model[model] = stat_id; - - for (i = 0; i < RTE_DIM(layer_xstats); i++) { - cn10k_mldev->xstats.entries[stat_id].map.id = stat_id; - cn10k_mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_MODEL; - cn10k_mldev->xstats.entries[stat_id].type = layer_xstats[i].type; - cn10k_mldev->xstats.entries[stat_id].fn_id = CNXK_ML_XSTATS_FN_MODEL; - cn10k_mldev->xstats.entries[stat_id].obj_idx = model; - cn10k_mldev->xstats.entries[stat_id].reset_allowed = - layer_xstats[i].reset_allowed; - - /* Name of xstat is updated during model load */ - snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, - sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), - "Model-%u-%s", model, layer_xstats[i].name); - - stat_id++; - } - - cn10k_mldev->xstats.count_per_model[model] = RTE_DIM(layer_xstats); - } - - cn10k_mldev->xstats.count_mode_model = stat_id - cn10k_mldev->xstats.count_mode_device; - cn10k_mldev->xstats.count = stat_id; - - return 0; -} - static void -cn10k_ml_xstats_uninit(struct rte_ml_dev *dev) +cn10k_ml_xstats_layer_name_update(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id, + uint16_t layer_id) { - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - - rte_free(cn10k_mldev->xstats.entries); - cn10k_mldev->xstats.entries = NULL; - - cn10k_mldev->xstats.count = 0; -} - -static void -cn10k_ml_xstats_model_name_update(struct rte_ml_dev *dev, uint16_t model_id) -{ - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; uint16_t rclk_freq; uint16_t sclk_freq; uint16_t stat_id; char suffix[8]; uint16_t i; - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - model = dev->data->models[model_id]; - stat_id = RTE_DIM(device_xstats) + model_id * RTE_DIM(layer_xstats); + model = cnxk_mldev->mldev->data->models[model_id]; + layer = &model->layer[layer_id]; + stat_id = cnxk_mldev->xstats.offset_for_layer[model_id][layer_id]; roc_clk_freq_get(&rclk_freq, &sclk_freq); if (sclk_freq == 0) @@ -310,270 +224,15 @@ cn10k_ml_xstats_model_name_update(struct rte_ml_dev *dev, uint16_t model_id) else strcpy(suffix, "ns"); - /* Update xstat name based on model name and sclk availability */ + /* Update xstat name based on layer name and sclk availability */ for (i = 0; i < RTE_DIM(layer_xstats); i++) { - snprintf(cn10k_mldev->xstats.entries[stat_id].map.name, - sizeof(cn10k_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", - model->layer[0].glow.metadata.model.name, layer_xstats[i].name, suffix); + snprintf(cnxk_mldev->xstats.entries[stat_id].map.name, + sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", + layer->glow.metadata.model.name, layer_xstats[i].name, suffix); stat_id++; } } -static uint64_t -cn10k_ml_dev_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx __rte_unused, - enum cnxk_ml_xstats_type type) -{ - struct cnxk_ml_dev *cnxk_mldev; - - cnxk_mldev = dev->data->dev_private; - - switch (type) { - case nb_models_loaded: - return cnxk_mldev->nb_models_loaded; - case nb_models_unloaded: - return cnxk_mldev->nb_models_unloaded; - case nb_models_started: - return cnxk_mldev->nb_models_started; - case nb_models_stopped: - return cnxk_mldev->nb_models_stopped; - default: - return -1; - } - - return 0; -} - -#define ML_AVG_FOREACH_QP(dev, model, qp_id, str, value, count) \ - do { \ - value = 0; \ - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - value += model->layer[0].glow.burst_xstats[qp_id].str##_latency_tot; \ - count += model->layer[0].glow.burst_xstats[qp_id].dequeued_count - \ - model->layer[0].glow.burst_xstats[qp_id].str##_reset_count; \ - } \ - if (count != 0) \ - value = value / count; \ - } while (0) - -#define ML_MIN_FOREACH_QP(dev, model, qp_id, str, value, count) \ - do { \ - value = UINT64_MAX; \ - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - value = PLT_MIN( \ - value, \ - model->layer[0].glow.burst_xstats[qp_id].str##_latency_min); \ - count += model->layer[0].glow.burst_xstats[qp_id].dequeued_count - \ - model->layer[0].glow.burst_xstats[qp_id].str##_reset_count; \ - } \ - if (count == 0) \ - value = 0; \ - } while (0) - -#define ML_MAX_FOREACH_QP(dev, model, qp_id, str, value, count) \ - do { \ - value = 0; \ - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - value = PLT_MAX( \ - value, \ - model->layer[0].glow.burst_xstats[qp_id].str##_latency_max); \ - count += model->layer[0].glow.burst_xstats[qp_id].dequeued_count - \ - model->layer[0].glow.burst_xstats[qp_id].str##_reset_count; \ - } \ - if (count == 0) \ - value = 0; \ - } while (0) - -static uint64_t -cn10k_ml_model_xstat_get(struct rte_ml_dev *dev, uint16_t obj_idx, enum cnxk_ml_xstats_type type) -{ - struct cnxk_ml_model *model; - uint16_t rclk_freq; /* MHz */ - uint16_t sclk_freq; /* MHz */ - uint64_t count = 0; - uint64_t value; - uint32_t qp_id; - - model = dev->data->models[obj_idx]; - if (model == NULL) - return 0; - - switch (type) { - case avg_hw_latency: - ML_AVG_FOREACH_QP(dev, model, qp_id, hw, value, count); - break; - case min_hw_latency: - ML_MIN_FOREACH_QP(dev, model, qp_id, hw, value, count); - break; - case max_hw_latency: - ML_MAX_FOREACH_QP(dev, model, qp_id, hw, value, count); - break; - case avg_fw_latency: - ML_AVG_FOREACH_QP(dev, model, qp_id, fw, value, count); - break; - case min_fw_latency: - ML_MIN_FOREACH_QP(dev, model, qp_id, fw, value, count); - break; - case max_fw_latency: - ML_MAX_FOREACH_QP(dev, model, qp_id, fw, value, count); - break; - default: - value = 0; - } - - roc_clk_freq_get(&rclk_freq, &sclk_freq); - if (sclk_freq != 0) /* return in ns */ - value = (value * 1000ULL) / sclk_freq; - - return value; -} - -static int -cn10k_ml_device_xstats_reset(struct rte_ml_dev *dev, const uint16_t stat_ids[], uint16_t nb_ids) -{ - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_xstats_entry *xs; - struct cnxk_ml_dev *cnxk_mldev; - uint16_t nb_stats; - uint16_t stat_id; - uint32_t i; - - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - - if (stat_ids == NULL) - nb_stats = cn10k_mldev->xstats.count_mode_device; - else - nb_stats = nb_ids; - - for (i = 0; i < nb_stats; i++) { - if (stat_ids == NULL) - stat_id = i; - else - stat_id = stat_ids[i]; - - if (stat_id >= cn10k_mldev->xstats.count_mode_device) - return -EINVAL; - - xs = &cn10k_mldev->xstats.entries[stat_id]; - if (!xs->reset_allowed) - continue; - - xs->reset_value = cn10k_ml_dev_xstat_get(dev, xs->obj_idx, xs->type); - } - - return 0; -} - -#define ML_AVG_RESET_FOREACH_QP(dev, model, qp_id, str) \ - do { \ - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { \ - model->layer[0].glow.burst_xstats[qp_id].str##_latency_tot = 0; \ - model->layer[0].glow.burst_xstats[qp_id].str##_reset_count = \ - model->layer[0].glow.burst_xstats[qp_id].dequeued_count; \ - } \ - } while (0) - -#define ML_MIN_RESET_FOREACH_QP(dev, model, qp_id, str) \ - do { \ - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) \ - model->layer[0].glow.burst_xstats[qp_id].str##_latency_min = UINT64_MAX; \ - } while (0) - -#define ML_MAX_RESET_FOREACH_QP(dev, model, qp_id, str) \ - do { \ - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) \ - model->layer[0].glow.burst_xstats[qp_id].str##_latency_max = 0; \ - } while (0) - -static void -cn10k_ml_reset_model_stat(struct rte_ml_dev *dev, uint16_t model_id, enum cnxk_ml_xstats_type type) -{ - struct cnxk_ml_model *model; - uint32_t qp_id; - - model = dev->data->models[model_id]; - - switch (type) { - case avg_hw_latency: - ML_AVG_RESET_FOREACH_QP(dev, model, qp_id, hw); - break; - case min_hw_latency: - ML_MIN_RESET_FOREACH_QP(dev, model, qp_id, hw); - break; - case max_hw_latency: - ML_MAX_RESET_FOREACH_QP(dev, model, qp_id, hw); - break; - case avg_fw_latency: - ML_AVG_RESET_FOREACH_QP(dev, model, qp_id, fw); - break; - case min_fw_latency: - ML_MIN_RESET_FOREACH_QP(dev, model, qp_id, fw); - break; - case max_fw_latency: - ML_MAX_RESET_FOREACH_QP(dev, model, qp_id, fw); - break; - default: - return; - } -} - -static int -cn10k_ml_model_xstats_reset(struct rte_ml_dev *dev, int32_t model_id, const uint16_t stat_ids[], - uint16_t nb_ids) -{ - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_xstats_entry *xs; - struct cnxk_ml_dev *cnxk_mldev; - struct cnxk_ml_model *model; - int32_t lcl_model_id = 0; - uint16_t start_id; - uint16_t end_id; - int32_t i; - int32_t j; - - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - for (i = 0; i < ML_CNXK_MAX_MODELS; i++) { - if (model_id == -1) { - model = dev->data->models[i]; - if (model == NULL) /* Skip inactive models */ - continue; - } else { - if (model_id != i) - continue; - - model = dev->data->models[model_id]; - if (model == NULL) { - plt_err("Invalid model_id = %d\n", model_id); - return -EINVAL; - } - } - - start_id = cn10k_mldev->xstats.offset_for_model[i]; - end_id = cn10k_mldev->xstats.offset_for_model[i] + - cn10k_mldev->xstats.count_per_model[i] - 1; - - if (stat_ids == NULL) { - for (j = start_id; j <= end_id; j++) { - xs = &cn10k_mldev->xstats.entries[j]; - cn10k_ml_reset_model_stat(dev, i, xs->type); - } - } else { - for (j = 0; j < nb_ids; j++) { - if (stat_ids[j] < start_id || stat_ids[j] > end_id) { - plt_err("Invalid stat_ids[%d] = %d for model_id = %d\n", j, - stat_ids[j], lcl_model_id); - return -EINVAL; - } - xs = &cn10k_mldev->xstats.entries[stat_ids[j]]; - cn10k_ml_reset_model_stat(dev, i, xs->type); - } - } - } - - return 0; -} - static int cn10k_ml_cache_model_data(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer) { @@ -658,7 +317,6 @@ cn10k_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_c struct cn10k_ml_dev *cn10k_mldev; struct cn10k_ml_ocm *ocm; uint16_t tile_id; - int ret; RTE_SET_USED(conf); @@ -686,13 +344,6 @@ cn10k_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_c rte_spinlock_init(&ocm->lock); - /* Initialize xstats */ - ret = cn10k_ml_xstats_init(cnxk_mldev->mldev); - if (ret != 0) { - plt_err("Failed to initialize xstats"); - return ret; - } - /* Set JCMDQ enqueue function */ if (cn10k_mldev->hw_queue_lock == 1) cn10k_mldev->ml_jcmdq_enqueue = roc_ml_jcmdq_enqueue_sl; @@ -721,9 +372,6 @@ cn10k_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev) /* Release ocm_mask memory */ rte_free(cn10k_mldev->ocm.ocm_mask); - /* Un-initialize xstats */ - cn10k_ml_xstats_uninit(cnxk_mldev->mldev); - /* Unload firmware */ cn10k_ml_fw_unload(cnxk_mldev); @@ -774,174 +422,6 @@ cn10k_ml_dev_stop(struct cnxk_ml_dev *cnxk_mldev) return 0; } -int -cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, - int32_t model_id, struct rte_ml_dev_xstats_map *xstats_map, - uint32_t size) -{ - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - uint32_t xstats_mode_count; - uint32_t idx = 0; - uint32_t i; - - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - - xstats_mode_count = 0; - switch (mode) { - case RTE_ML_DEV_XSTATS_DEVICE: - xstats_mode_count = cn10k_mldev->xstats.count_mode_device; - break; - case RTE_ML_DEV_XSTATS_MODEL: - if (model_id >= ML_CNXK_MAX_MODELS) - break; - xstats_mode_count = cn10k_mldev->xstats.count_per_model[model_id]; - break; - default: - return -EINVAL; - }; - - if (xstats_mode_count > size || xstats_map == NULL) - return xstats_mode_count; - - for (i = 0; i < cn10k_mldev->xstats.count && idx < size; i++) { - if (cn10k_mldev->xstats.entries[i].mode != mode) - continue; - - if (mode != RTE_ML_DEV_XSTATS_DEVICE && - model_id != cn10k_mldev->xstats.entries[i].obj_idx) - continue; - - strncpy(xstats_map[idx].name, cn10k_mldev->xstats.entries[i].map.name, - RTE_ML_STR_MAX); - xstats_map[idx].id = cn10k_mldev->xstats.entries[i].map.id; - idx++; - } - - return idx; -} - -int -cn10k_ml_dev_xstats_by_name_get(struct rte_ml_dev *dev, const char *name, uint16_t *stat_id, - uint64_t *value) -{ - struct cnxk_ml_xstats_entry *xs; - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - cnxk_ml_xstats_fn fn; - uint32_t i; - - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - for (i = 0; i < cn10k_mldev->xstats.count; i++) { - xs = &cn10k_mldev->xstats.entries[i]; - if (strncmp(xs->map.name, name, RTE_ML_STR_MAX) == 0) { - if (stat_id != NULL) - *stat_id = xs->map.id; - - switch (xs->fn_id) { - case CNXK_ML_XSTATS_FN_DEVICE: - fn = cn10k_ml_dev_xstat_get; - break; - case CNXK_ML_XSTATS_FN_MODEL: - fn = cn10k_ml_model_xstat_get; - break; - default: - plt_err("Unexpected xstat fn_id = %d", xs->fn_id); - return -EINVAL; - } - - *value = fn(dev, xs->obj_idx, xs->type) - xs->reset_value; - - return 0; - } - } - - if (stat_id != NULL) - *stat_id = (uint16_t)-1; - - return -EINVAL; -} - -int -cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, int32_t model_id, - const uint16_t stat_ids[], uint64_t values[], uint16_t nb_ids) -{ - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_xstats_entry *xs; - struct cnxk_ml_dev *cnxk_mldev; - uint32_t xstats_mode_count; - cnxk_ml_xstats_fn fn; - uint64_t val; - uint32_t idx; - uint32_t i; - - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - xstats_mode_count = 0; - - switch (mode) { - case RTE_ML_DEV_XSTATS_DEVICE: - xstats_mode_count = cn10k_mldev->xstats.count_mode_device; - break; - case RTE_ML_DEV_XSTATS_MODEL: - if (model_id >= ML_CNXK_MAX_MODELS) - return -EINVAL; - xstats_mode_count = cn10k_mldev->xstats.count_per_model[model_id]; - break; - default: - return -EINVAL; - }; - - idx = 0; - for (i = 0; i < nb_ids && idx < xstats_mode_count; i++) { - xs = &cn10k_mldev->xstats.entries[stat_ids[i]]; - if (stat_ids[i] > cn10k_mldev->xstats.count || xs->mode != mode) - continue; - - if (mode == RTE_ML_DEV_XSTATS_MODEL && model_id != xs->obj_idx) { - plt_err("Invalid stats_id[%d] = %d for model_id = %d\n", i, stat_ids[i], - model_id); - return -EINVAL; - } - - switch (xs->fn_id) { - case CNXK_ML_XSTATS_FN_DEVICE: - fn = cn10k_ml_dev_xstat_get; - break; - case CNXK_ML_XSTATS_FN_MODEL: - fn = cn10k_ml_model_xstat_get; - break; - default: - plt_err("Unexpected xstat fn_id = %d", xs->fn_id); - return -EINVAL; - } - - val = fn(dev, xs->obj_idx, xs->type); - if (values) - values[idx] = val; - - idx++; - } - - return idx; -} - -int -cn10k_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, - int32_t model_id, const uint16_t stat_ids[], uint16_t nb_ids) -{ - switch (mode) { - case RTE_ML_DEV_XSTATS_DEVICE: - return cn10k_ml_device_xstats_reset(dev, stat_ids, nb_ids); - case RTE_ML_DEV_XSTATS_MODEL: - return cn10k_ml_model_xstats_reset(dev, model_id, stat_ids, nb_ids); - }; - - return 0; -} - int cn10k_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp) { @@ -1215,7 +695,7 @@ cn10k_ml_layer_load(void *device, uint16_t model_id, const char *layer_name, uin sizeof(struct cn10k_ml_layer_xstats)); /* Update xstats names */ - cn10k_ml_xstats_model_name_update(cnxk_mldev->mldev, idx); + cn10k_ml_xstats_layer_name_update(cnxk_mldev, model_id, layer_id); layer->state = ML_CNXK_LAYER_STATE_LOADED; cnxk_mldev->index_map[idx].model_id = model->model_id; diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 47e7cb12af4..8a090a31592 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -298,17 +298,6 @@ int cn10k_ml_dev_stop(struct cnxk_ml_dev *cnxk_mldev); int cn10k_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp); int cn10k_ml_dev_selftest(struct cnxk_ml_dev *cnxk_mldev); -int cn10k_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, - int32_t model_id, struct rte_ml_dev_xstats_map *xstats_map, - uint32_t size); -int cn10k_ml_dev_xstats_by_name_get(struct rte_ml_dev *dev, const char *name, uint16_t *stat_id, - uint64_t *value); -int cn10k_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, - int32_t model_id, const uint16_t stat_ids[], uint64_t values[], - uint16_t nb_ids); -int cn10k_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, - int32_t model_id, const uint16_t stat_ids[], uint16_t nb_ids); - /* Slow-path ops */ int cn10k_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, struct cnxk_ml_model *model); diff --git a/drivers/ml/cnxk/cnxk_ml_dev.h b/drivers/ml/cnxk/cnxk_ml_dev.h index 1590249abd1..3ce9338f1f1 100644 --- a/drivers/ml/cnxk/cnxk_ml_dev.h +++ b/drivers/ml/cnxk/cnxk_ml_dev.h @@ -9,6 +9,8 @@ #include "cn10k_ml_dev.h" +#include "cnxk_ml_xstats.h" + /* ML command timeout in seconds */ #define ML_CNXK_CMD_TIMEOUT 5 @@ -51,6 +53,9 @@ struct cnxk_ml_dev { /* Configuration state */ enum cnxk_ml_dev_state state; + /* Extended stats data */ + struct cnxk_ml_xstats xstats; + /* Number of models loaded */ uint16_t nb_models_loaded; diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index ffeb3f44523..96d40029b36 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -117,6 +117,344 @@ cnxk_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_desc return NULL; } +static int +cnxk_ml_xstats_init(struct cnxk_ml_dev *cnxk_mldev) +{ + uint16_t nb_stats; + uint16_t stat_id; + uint16_t model; + uint16_t layer; + uint16_t i; + + /* Allocate memory for xstats entries. Don't allocate during reconfigure */ + nb_stats = RTE_DIM(device_xstats) + + RTE_DIM(layer_xstats) * ML_CNXK_MAX_MODELS * ML_CNXK_MODEL_MAX_LAYERS; + if (cnxk_mldev->xstats.entries == NULL) + cnxk_mldev->xstats.entries = rte_zmalloc( + "cnxk_ml_xstats", sizeof(struct cnxk_ml_xstats_entry) * nb_stats, + PLT_CACHE_LINE_SIZE); + + if (cnxk_mldev->xstats.entries == NULL) + return -ENOMEM; + + /* Initialize device xstats */ + stat_id = 0; + for (i = 0; i < RTE_DIM(device_xstats); i++) { + cnxk_mldev->xstats.entries[stat_id].map.id = stat_id; + snprintf(cnxk_mldev->xstats.entries[stat_id].map.name, + sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), "%s", + device_xstats[i].name); + + cnxk_mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_DEVICE; + cnxk_mldev->xstats.entries[stat_id].group = CNXK_ML_XSTATS_GROUP_DEVICE; + cnxk_mldev->xstats.entries[stat_id].type = device_xstats[i].type; + cnxk_mldev->xstats.entries[stat_id].fn_id = CNXK_ML_XSTATS_FN_DEVICE; + cnxk_mldev->xstats.entries[stat_id].obj_idx = 0; + cnxk_mldev->xstats.entries[stat_id].reset_allowed = device_xstats[i].reset_allowed; + stat_id++; + } + cnxk_mldev->xstats.count_mode_device = stat_id; + + /* Initialize model xstats */ + for (model = 0; model < ML_CNXK_MAX_MODELS; model++) { + cnxk_mldev->xstats.offset_for_model[model] = stat_id; + + for (layer = 0; layer < ML_CNXK_MODEL_MAX_LAYERS; layer++) { + cnxk_mldev->xstats.offset_for_layer[model][layer] = stat_id; + + for (i = 0; i < RTE_DIM(layer_xstats); i++) { + cnxk_mldev->xstats.entries[stat_id].map.id = stat_id; + cnxk_mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_MODEL; + cnxk_mldev->xstats.entries[stat_id].group = + CNXK_ML_XSTATS_GROUP_LAYER; + cnxk_mldev->xstats.entries[stat_id].type = layer_xstats[i].type; + cnxk_mldev->xstats.entries[stat_id].fn_id = CNXK_ML_XSTATS_FN_MODEL; + cnxk_mldev->xstats.entries[stat_id].obj_idx = model; + cnxk_mldev->xstats.entries[stat_id].layer_id = layer; + cnxk_mldev->xstats.entries[stat_id].reset_allowed = + layer_xstats[i].reset_allowed; + + /* Name of xstat is updated during model load */ + snprintf(cnxk_mldev->xstats.entries[stat_id].map.name, + sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), + "Layer-%u-%u-%s", model, layer, layer_xstats[i].name); + + stat_id++; + } + + cnxk_mldev->xstats.count_per_layer[model][layer] = RTE_DIM(layer_xstats); + } + + cnxk_mldev->xstats.count_per_model[model] = RTE_DIM(layer_xstats); + } + + cnxk_mldev->xstats.count_mode_model = stat_id - cnxk_mldev->xstats.count_mode_device; + cnxk_mldev->xstats.count = stat_id; + + return 0; +} + +static void +cnxk_ml_xstats_uninit(struct cnxk_ml_dev *cnxk_mldev) +{ + rte_free(cnxk_mldev->xstats.entries); + cnxk_mldev->xstats.entries = NULL; + + cnxk_mldev->xstats.count = 0; +} + +static uint64_t +cnxk_ml_dev_xstat_get(struct cnxk_ml_dev *cnxk_mldev, uint16_t obj_idx __rte_unused, + int32_t layer_id __rte_unused, enum cnxk_ml_xstats_type type) +{ + switch (type) { + case nb_models_loaded: + return cnxk_mldev->nb_models_loaded; + case nb_models_unloaded: + return cnxk_mldev->nb_models_unloaded; + case nb_models_started: + return cnxk_mldev->nb_models_started; + case nb_models_stopped: + return cnxk_mldev->nb_models_stopped; + default: + return -1; + } + + return 0; +} + +#define ML_AVG_FOREACH_QP(cnxk_mldev, layer, qp_id, str, value, count) \ + do { \ + value = 0; \ + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \ + value += layer->glow.burst_xstats[qp_id].str##_latency_tot; \ + count += layer->glow.burst_xstats[qp_id].dequeued_count - \ + layer->glow.burst_xstats[qp_id].str##_reset_count; \ + } \ + if (count != 0) \ + value = value / count; \ + } while (0) + +#define ML_MIN_FOREACH_QP(cnxk_mldev, layer, qp_id, str, value, count) \ + do { \ + value = UINT64_MAX; \ + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \ + value = PLT_MIN(value, layer->glow.burst_xstats[qp_id].str##_latency_min); \ + count += layer->glow.burst_xstats[qp_id].dequeued_count - \ + layer->glow.burst_xstats[qp_id].str##_reset_count; \ + } \ + if (count == 0) \ + value = 0; \ + } while (0) + +#define ML_MAX_FOREACH_QP(cnxk_mldev, layer, qp_id, str, value, count) \ + do { \ + value = 0; \ + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \ + value = PLT_MAX(value, layer->glow.burst_xstats[qp_id].str##_latency_max); \ + count += layer->glow.burst_xstats[qp_id].dequeued_count - \ + layer->glow.burst_xstats[qp_id].str##_reset_count; \ + } \ + if (count == 0) \ + value = 0; \ + } while (0) + +static uint64_t +cnxk_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, uint16_t obj_idx, int32_t layer_id, + enum cnxk_ml_xstats_type type) +{ + struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; + uint16_t rclk_freq; /* MHz */ + uint16_t sclk_freq; /* MHz */ + uint64_t count = 0; + uint64_t value; + uint32_t qp_id; + + model = cnxk_mldev->mldev->data->models[obj_idx]; + if (model == NULL) + return 0; + + if (layer_id >= 0) + layer = &model->layer[layer_id]; + else + layer = NULL; + + switch (type) { + case avg_hw_latency: + ML_AVG_FOREACH_QP(cnxk_mldev, layer, qp_id, hw, value, count); + break; + case min_hw_latency: + ML_MIN_FOREACH_QP(cnxk_mldev, layer, qp_id, hw, value, count); + break; + case max_hw_latency: + ML_MAX_FOREACH_QP(cnxk_mldev, layer, qp_id, hw, value, count); + break; + case avg_fw_latency: + ML_AVG_FOREACH_QP(cnxk_mldev, layer, qp_id, fw, value, count); + break; + case min_fw_latency: + ML_MIN_FOREACH_QP(cnxk_mldev, layer, qp_id, fw, value, count); + break; + case max_fw_latency: + ML_MAX_FOREACH_QP(cnxk_mldev, layer, qp_id, fw, value, count); + break; + default: + value = 0; + } + + roc_clk_freq_get(&rclk_freq, &sclk_freq); + if (sclk_freq != 0) /* return in ns */ + value = (value * 1000ULL) / sclk_freq; + + return value; +} + +static int +cnxk_ml_device_xstats_reset(struct cnxk_ml_dev *cnxk_mldev, const uint16_t stat_ids[], + uint16_t nb_ids) +{ + struct cnxk_ml_xstats_entry *xs; + uint16_t nb_stats; + uint16_t stat_id; + uint32_t i; + + if (stat_ids == NULL) + nb_stats = cnxk_mldev->xstats.count_mode_device; + else + nb_stats = nb_ids; + + for (i = 0; i < nb_stats; i++) { + if (stat_ids == NULL) + stat_id = i; + else + stat_id = stat_ids[i]; + + if (stat_id >= cnxk_mldev->xstats.count_mode_device) + return -EINVAL; + + xs = &cnxk_mldev->xstats.entries[stat_id]; + if (!xs->reset_allowed) + continue; + + xs->reset_value = + cnxk_ml_dev_xstat_get(cnxk_mldev, xs->obj_idx, xs->layer_id, xs->type); + } + + return 0; +} + +#define ML_AVG_RESET_FOREACH_QP(cnxk_mldev, layer, qp_id, str) \ + do { \ + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \ + layer->glow.burst_xstats[qp_id].str##_latency_tot = 0; \ + layer->glow.burst_xstats[qp_id].str##_reset_count = \ + layer->glow.burst_xstats[qp_id].dequeued_count; \ + } \ + } while (0) + +#define ML_MIN_RESET_FOREACH_QP(cnxk_mldev, layer, qp_id, str) \ + do { \ + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) \ + layer->glow.burst_xstats[qp_id].str##_latency_min = UINT64_MAX; \ + } while (0) + +#define ML_MAX_RESET_FOREACH_QP(cnxk_mldev, layer, qp_id, str) \ + do { \ + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) \ + layer->glow.burst_xstats[qp_id].str##_latency_max = 0; \ + } while (0) + +static void +cnxk_ml_reset_model_stat(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id, + enum cnxk_ml_xstats_type type) +{ + struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; + uint16_t layer_id = 0; + uint32_t qp_id; + + model = cnxk_mldev->mldev->data->models[model_id]; + layer = &model->layer[layer_id]; + + switch (type) { + case avg_hw_latency: + ML_AVG_RESET_FOREACH_QP(cnxk_mldev, layer, qp_id, hw); + break; + case min_hw_latency: + ML_MIN_RESET_FOREACH_QP(cnxk_mldev, layer, qp_id, hw); + break; + case max_hw_latency: + ML_MAX_RESET_FOREACH_QP(cnxk_mldev, layer, qp_id, hw); + break; + case avg_fw_latency: + ML_AVG_RESET_FOREACH_QP(cnxk_mldev, layer, qp_id, fw); + break; + case min_fw_latency: + ML_MIN_RESET_FOREACH_QP(cnxk_mldev, layer, qp_id, fw); + break; + case max_fw_latency: + ML_MAX_RESET_FOREACH_QP(cnxk_mldev, layer, qp_id, fw); + break; + default: + return; + } +} + +static int +cnxk_ml_model_xstats_reset(struct cnxk_ml_dev *cnxk_mldev, int32_t model_id, + const uint16_t stat_ids[], uint16_t nb_ids) +{ + struct cnxk_ml_xstats_entry *xs; + struct cnxk_ml_model *model; + int32_t lcl_model_id = 0; + uint16_t layer_id = 0; + uint16_t start_id; + uint16_t end_id; + int32_t i; + int32_t j; + + for (i = 0; i < ML_CNXK_MAX_MODELS; i++) { + if (model_id == -1) { + model = cnxk_mldev->mldev->data->models[i]; + if (model == NULL) /* skip inactive models */ + continue; + } else { + if (model_id != i) + continue; + + model = cnxk_mldev->mldev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %d\n", model_id); + return -EINVAL; + } + } + + start_id = cnxk_mldev->xstats.offset_for_layer[i][layer_id]; + end_id = cnxk_mldev->xstats.offset_for_layer[i][layer_id] + + cnxk_mldev->xstats.count_per_layer[i][layer_id] - 1; + + if (stat_ids == NULL) { + for (j = start_id; j <= end_id; j++) { + xs = &cnxk_mldev->xstats.entries[j]; + cnxk_ml_reset_model_stat(cnxk_mldev, i, xs->type); + } + } else { + for (j = 0; j < nb_ids; j++) { + if (stat_ids[j] < start_id || stat_ids[j] > end_id) { + plt_err("Invalid stat_ids[%d] = %d for model_id = %d\n", j, + stat_ids[j], lcl_model_id); + return -EINVAL; + } + xs = &cnxk_mldev->xstats.entries[stat_ids[j]]; + cnxk_ml_reset_model_stat(cnxk_mldev, i, xs->type); + } + } + } + + return 0; +} + static int cnxk_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) { @@ -296,6 +634,13 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co for (i = 0; i < cnxk_mldev->max_nb_layers; i++) cnxk_mldev->index_map[i].active = false; + /* Initialize xstats */ + ret = cnxk_ml_xstats_init(cnxk_mldev); + if (ret != 0) { + plt_err("Failed to initialize xstats"); + goto error; + } + cnxk_mldev->nb_models_loaded = 0; cnxk_mldev->nb_models_started = 0; cnxk_mldev->nb_models_stopped = 0; @@ -325,6 +670,9 @@ cnxk_ml_dev_close(struct rte_ml_dev *dev) cnxk_mldev = dev->data->dev_private; + /* Un-initialize xstats */ + cnxk_ml_xstats_uninit(cnxk_mldev); + if (cn10k_ml_dev_close(cnxk_mldev) != 0) plt_err("Failed to close CN10K ML Device"); @@ -523,6 +871,190 @@ cnxk_ml_dev_stats_reset(struct rte_ml_dev *dev) } } +static int +cnxk_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, + int32_t model_id, struct rte_ml_dev_xstats_map *xstats_map, + uint32_t size) +{ + struct cnxk_ml_xstats_entry *xs; + struct cnxk_ml_dev *cnxk_mldev; + uint32_t xstats_mode_count; + uint16_t layer_id = 0; + uint32_t idx = 0; + uint32_t i; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + xstats_mode_count = 0; + + switch (mode) { + case RTE_ML_DEV_XSTATS_DEVICE: + xstats_mode_count = cnxk_mldev->xstats.count_mode_device; + break; + case RTE_ML_DEV_XSTATS_MODEL: + if (model_id >= ML_CNXK_MAX_MODELS) + break; + xstats_mode_count = cnxk_mldev->xstats.count_per_layer[model_id][layer_id]; + break; + default: + return -EINVAL; + }; + + if (xstats_mode_count > size || xstats_map == NULL) + return xstats_mode_count; + + for (i = 0; i < cnxk_mldev->xstats.count && idx < size; i++) { + xs = &cnxk_mldev->xstats.entries[i]; + if (xs->mode != mode) + continue; + + if (mode == RTE_ML_DEV_XSTATS_MODEL && + (model_id != xs->obj_idx || layer_id != xs->layer_id)) + continue; + + strncpy(xstats_map[idx].name, xs->map.name, RTE_ML_STR_MAX); + xstats_map[idx].id = xs->map.id; + idx++; + } + + return idx; +} + +static int +cnxk_ml_dev_xstats_by_name_get(struct rte_ml_dev *dev, const char *name, uint16_t *stat_id, + uint64_t *value) +{ + struct cnxk_ml_xstats_entry *xs; + struct cnxk_ml_dev *cnxk_mldev; + cnxk_ml_xstats_fn fn; + uint32_t i; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + for (i = 0; i < cnxk_mldev->xstats.count; i++) { + xs = &cnxk_mldev->xstats.entries[i]; + if (strncmp(xs->map.name, name, RTE_ML_STR_MAX) == 0) { + if (stat_id != NULL) + *stat_id = xs->map.id; + + switch (xs->fn_id) { + case CNXK_ML_XSTATS_FN_DEVICE: + fn = cnxk_ml_dev_xstat_get; + break; + case CNXK_ML_XSTATS_FN_MODEL: + fn = cnxk_ml_model_xstat_get; + break; + default: + plt_err("Unexpected xstat fn_id = %d", xs->fn_id); + return -EINVAL; + } + + *value = fn(cnxk_mldev, xs->obj_idx, xs->layer_id, xs->type) - + xs->reset_value; + + return 0; + } + } + + if (stat_id != NULL) + *stat_id = (uint16_t)-1; + + return -EINVAL; +} + +static int +cnxk_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, int32_t model_id, + const uint16_t stat_ids[], uint64_t values[], uint16_t nb_ids) +{ + struct cnxk_ml_xstats_entry *xs; + struct cnxk_ml_dev *cnxk_mldev; + uint32_t xstats_mode_count; + uint16_t layer_id = 0; + cnxk_ml_xstats_fn fn; + uint64_t val; + uint32_t idx; + uint32_t i; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + xstats_mode_count = 0; + + switch (mode) { + case RTE_ML_DEV_XSTATS_DEVICE: + xstats_mode_count = cnxk_mldev->xstats.count_mode_device; + break; + case RTE_ML_DEV_XSTATS_MODEL: + if (model_id >= ML_CNXK_MAX_MODELS) + return -EINVAL; + xstats_mode_count = cnxk_mldev->xstats.count_per_layer[model_id][layer_id]; + break; + default: + return -EINVAL; + }; + + idx = 0; + for (i = 0; i < nb_ids && idx < xstats_mode_count; i++) { + xs = &cnxk_mldev->xstats.entries[stat_ids[i]]; + if (stat_ids[i] > cnxk_mldev->xstats.count || xs->mode != mode) + continue; + + if (mode == RTE_ML_DEV_XSTATS_MODEL && + (model_id != xs->obj_idx || layer_id != xs->layer_id)) { + plt_err("Invalid stats_id[%d] = %d for model_id = %d\n", i, stat_ids[i], + model_id); + return -EINVAL; + } + + switch (xs->fn_id) { + case CNXK_ML_XSTATS_FN_DEVICE: + fn = cnxk_ml_dev_xstat_get; + break; + case CNXK_ML_XSTATS_FN_MODEL: + fn = cnxk_ml_model_xstat_get; + break; + default: + plt_err("Unexpected xstat fn_id = %d", xs->fn_id); + return -EINVAL; + } + + val = fn(cnxk_mldev, xs->obj_idx, xs->layer_id, xs->type); + if (values) + values[idx] = val; + + idx++; + } + + return idx; +} + +static int +cnxk_ml_dev_xstats_reset(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, int32_t model_id, + const uint16_t stat_ids[], uint16_t nb_ids) +{ + struct cnxk_ml_dev *cnxk_mldev; + + if (dev == NULL) + return -EINVAL; + + cnxk_mldev = dev->data->dev_private; + + switch (mode) { + case RTE_ML_DEV_XSTATS_DEVICE: + return cnxk_ml_device_xstats_reset(cnxk_mldev, stat_ids, nb_ids); + case RTE_ML_DEV_XSTATS_MODEL: + return cnxk_ml_model_xstats_reset(cnxk_mldev, model_id, stat_ids, nb_ids); + }; + + return 0; +} + static int cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, uint16_t *model_id) { @@ -808,10 +1340,10 @@ struct rte_ml_dev_ops cnxk_ml_ops = { /* Stats ops */ .dev_stats_get = cnxk_ml_dev_stats_get, .dev_stats_reset = cnxk_ml_dev_stats_reset, - .dev_xstats_names_get = cn10k_ml_dev_xstats_names_get, - .dev_xstats_by_name_get = cn10k_ml_dev_xstats_by_name_get, - .dev_xstats_get = cn10k_ml_dev_xstats_get, - .dev_xstats_reset = cn10k_ml_dev_xstats_reset, + .dev_xstats_names_get = cnxk_ml_dev_xstats_names_get, + .dev_xstats_by_name_get = cnxk_ml_dev_xstats_by_name_get, + .dev_xstats_get = cnxk_ml_dev_xstats_get, + .dev_xstats_reset = cnxk_ml_dev_xstats_reset, /* Model ops */ .model_load = cnxk_ml_model_load, diff --git a/drivers/ml/cnxk/cnxk_ml_xstats.h b/drivers/ml/cnxk/cnxk_ml_xstats.h index 0d405679caa..5e02bb876ca 100644 --- a/drivers/ml/cnxk/cnxk_ml_xstats.h +++ b/drivers/ml/cnxk/cnxk_ml_xstats.h @@ -7,6 +7,8 @@ #include "cnxk_ml_io.h" +struct cnxk_ml_dev; + /* Extended stats types enum */ enum cnxk_ml_xstats_type { /* Number of models loaded */ @@ -58,9 +60,21 @@ enum cnxk_ml_xstats_fn_type { CNXK_ML_XSTATS_FN_MODEL, }; +/* Extended stats group */ +enum cnxk_ml_xstats_group { + /* Device stats */ + CNXK_ML_XSTATS_GROUP_DEVICE, + + /* Model stats */ + CNXK_ML_XSTATS_GROUP_MODEL, + + /* Layer stats */ + CNXK_ML_XSTATS_GROUP_LAYER, +}; + /* Function pointer to get xstats for a type */ -typedef uint64_t (*cnxk_ml_xstats_fn)(struct rte_ml_dev *cnxk_mldev, uint16_t obj_idx, - enum cnxk_ml_xstats_type stat); +typedef uint64_t (*cnxk_ml_xstats_fn)(struct cnxk_ml_dev *cnxk_mldev, uint16_t obj_idx, + int32_t layer_id, enum cnxk_ml_xstats_type stat); /* Extended stats entry structure */ struct cnxk_ml_xstats_entry { @@ -70,6 +84,9 @@ struct cnxk_ml_xstats_entry { /* xstats mode, device or model */ enum rte_ml_dev_xstats_mode mode; + /* xstats group */ + enum cnxk_ml_xstats_group group; + /* Type of xstats */ enum cnxk_ml_xstats_type type; From patchwork Wed Aug 30 15:59:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130898 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AE17941FD1; Wed, 30 Aug 2023 18:02:49 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 272044067B; Wed, 30 Aug 2023 17:59:57 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id E9DD640293 for ; Wed, 30 Aug 2023 17:59:40 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I71012554 for ; Wed, 30 Aug 2023 08:59:40 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=UstRXdOR0xDFm58JyoxaXUlLW4iQ7r++D//LpoGbKEo=; b=EAtrKRAeYw2x74SqlW1md3ptqHLGK0ofuoKbUX9DbyZUlePYSNF1vRE9icCT9ZRPMjW0 cRR9Q4H2uB5sCGI8Q6IsMiImhCz8Qm0yMwt+aD9CLVWNl2sd61pWOK3d4jsH8QYdL2ez LjeyWkI/3gErZdavRMPkqTw6+xukwprIcSghWykM/O59G7e7xJvd7GnbmsmNsSio8Agn SZLIIOiSBWdGA63plGP+PrwdHXzRw/sMd191a6ID9AZLhMMWHovqVvjo+C62M/aWcS7D WdFA5ChDQpbZnd99qW3/ijV4ya9ASlevzEzUY4+Q7zXY25k2UDxeeQcZAx5tBfyhnTvF cw== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:40 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:37 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:37 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id A82E13F707E; Wed, 30 Aug 2023 08:59:37 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 17/34] ml/cnxk: update fast path functions Date: Wed, 30 Aug 2023 08:59:07 -0700 Message-ID: <20230830155927.3566-18-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: _B6iiomduQLkBUwVxU3-8u5-MEDz-RXK X-Proofpoint-GUID: _B6iiomduQLkBUwVxU3-8u5-MEDz-RXK X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implemented cnxk layer fast-path functions and added support for model specific fast-path functions. CNXK layer functions would invoke model specific fast-path functions. Added support for model specific poll handling functions and updated internal inference sync function. Drop use of rte_ml_op as argument. Updated function arguments to enable the function to be used as callback by TVM HW runtime. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.h | 5 - drivers/ml/cnxk/cn10k_ml_ops.c | 241 ++++++++------------------------ drivers/ml/cnxk/cn10k_ml_ops.h | 13 +- drivers/ml/cnxk/cnxk_ml_model.h | 14 ++ drivers/ml/cnxk/cnxk_ml_ops.c | 128 +++++++++++++++++ drivers/ml/cnxk/cnxk_ml_ops.h | 7 + 6 files changed, 216 insertions(+), 192 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_dev.h b/drivers/ml/cnxk/cn10k_ml_dev.h index bde9d089015..94a94d996f8 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.h +++ b/drivers/ml/cnxk/cn10k_ml_dev.h @@ -143,11 +143,6 @@ struct cn10k_ml_dev { /* JCMD enqueue function handler */ bool (*ml_jcmdq_enqueue)(struct roc_ml *roc_ml, struct ml_job_cmd_s *job_cmd); - - /* Poll handling function pointers */ - void (*set_poll_addr)(struct cnxk_ml_req *req); - void (*set_poll_ptr)(struct cnxk_ml_req *req); - uint64_t (*get_poll_ptr)(struct cnxk_ml_req *req); }; uint64_t cn10k_ml_fw_flags_get(struct cn10k_ml_fw *fw); diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index f1431b89a2d..7d809d25ae0 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -69,24 +69,12 @@ static const struct cn10k_ml_stype_db_driver { {ML_DRIVER_ERR_FW_ERROR, "UNKNOWN FIRMWARE ERROR"}, }; -static inline void +__rte_hot void cn10k_ml_set_poll_addr(struct cnxk_ml_req *req) { req->status = &req->cn10k_req.status; } -static inline void -cn10k_ml_set_poll_ptr(struct cnxk_ml_req *req) -{ - plt_write64(ML_CNXK_POLL_JOB_START, req->status); -} - -static inline uint64_t -cn10k_ml_get_poll_ptr(struct cnxk_ml_req *req) -{ - return plt_read64(req->status); -} - void cn10k_ml_qp_initialize(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_qp *qp) { @@ -181,7 +169,7 @@ cn10k_ml_prep_sp_job_descriptor(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_l static __rte_always_inline void cn10k_ml_prep_fp_job_descriptor(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_req *req, - struct rte_ml_op *op) + uint16_t index, void *input, void *output, uint16_t nb_batches) { struct cn10k_ml_dev *cn10k_mldev; @@ -189,17 +177,17 @@ cn10k_ml_prep_fp_job_descriptor(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_r req->cn10k_req.jd.hdr.jce.w0.u64 = 0; req->cn10k_req.jd.hdr.jce.w1.u64 = PLT_U64_CAST(req->status); - req->cn10k_req.jd.hdr.model_id = op->model_id; + req->cn10k_req.jd.hdr.model_id = index; req->cn10k_req.jd.hdr.job_type = ML_CN10K_JOB_TYPE_MODEL_RUN; req->cn10k_req.jd.hdr.fp_flags = ML_FLAGS_POLL_COMPL; req->cn10k_req.jd.hdr.sp_flags = 0x0; req->cn10k_req.jd.hdr.result = roc_ml_addr_ap2mlip(&cn10k_mldev->roc, &req->cn10k_req.result); req->cn10k_req.jd.model_run.input_ddr_addr = - PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, op->input[0]->addr)); + PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, input)); req->cn10k_req.jd.model_run.output_ddr_addr = - PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, op->output[0]->addr)); - req->cn10k_req.jd.model_run.num_batches = op->nb_batches; + PLT_U64_CAST(roc_ml_addr_ap2mlip(&cn10k_mldev->roc, output)); + req->cn10k_req.jd.model_run.num_batches = nb_batches; } static void @@ -236,30 +224,15 @@ cn10k_ml_xstats_layer_name_update(struct cnxk_ml_dev *cnxk_mldev, uint16_t model static int cn10k_ml_cache_model_data(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer) { - struct rte_ml_buff_seg seg[2]; - struct rte_ml_buff_seg *inp; - struct rte_ml_buff_seg *out; - struct rte_ml_op op; - char str[RTE_MEMZONE_NAMESIZE]; const struct plt_memzone *mz; uint64_t isize = 0; uint64_t osize = 0; int ret = 0; - uint32_t i; - - inp = &seg[0]; - out = &seg[1]; /* Create input and output buffers. */ - for (i = 0; i < layer->info.nb_inputs; i++) - isize += layer->info.input[i].sz_q; - - for (i = 0; i < layer->info.nb_outputs; i++) - osize += layer->info.output[i].sz_q; - - isize = layer->batch_size * isize; - osize = layer->batch_size * osize; + isize = layer->info.total_input_sz_q; + osize = layer->info.total_output_sz_q; snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", "ml_dummy_io", layer->index); mz = plt_memzone_reserve_aligned(str, isize + osize, 0, ML_CN10K_ALIGN_SIZE); @@ -267,25 +240,9 @@ cn10k_ml_cache_model_data(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer * return -ENOMEM; memset(mz->addr, 0, isize + osize); - seg[0].addr = mz->addr; - seg[0].iova_addr = mz->iova; - seg[0].length = isize; - seg[0].next = NULL; - - seg[1].addr = PLT_PTR_ADD(mz->addr, isize); - seg[1].iova_addr = mz->iova + isize; - seg[1].length = osize; - seg[1].next = NULL; - - op.model_id = layer->index; - op.nb_batches = layer->batch_size; - op.mempool = NULL; - - op.input = &inp; - op.output = &out; - memset(layer->glow.req, 0, sizeof(struct cnxk_ml_req)); - ret = cn10k_ml_inference_sync(cnxk_mldev, &op); + ret = cn10k_ml_inference_sync(cnxk_mldev, layer->index, mz->addr, + PLT_PTR_ADD(mz->addr, isize), 1); plt_memzone_free(mz); return ret; @@ -350,13 +307,8 @@ cn10k_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_c else cn10k_mldev->ml_jcmdq_enqueue = roc_ml_jcmdq_enqueue_lf; - /* Set polling function pointers */ - cn10k_mldev->set_poll_addr = cn10k_ml_set_poll_addr; - cn10k_mldev->set_poll_ptr = cn10k_ml_set_poll_ptr; - cn10k_mldev->get_poll_ptr = cn10k_ml_get_poll_ptr; - - cnxk_mldev->mldev->enqueue_burst = cn10k_ml_enqueue_burst; - cnxk_mldev->mldev->dequeue_burst = cn10k_ml_dequeue_burst; + cnxk_mldev->mldev->enqueue_burst = cnxk_ml_enqueue_burst; + cnxk_mldev->mldev->dequeue_burst = cnxk_ml_dequeue_burst; cnxk_mldev->mldev->op_error_get = cn10k_ml_op_error_get; return 0; @@ -749,6 +701,12 @@ cn10k_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * cn10k_ml_model_info_set(cnxk_mldev, model, &model->layer[0].info, &model->glow.metadata); + /* Set fast-path functions */ + model->enqueue_single = cn10k_ml_enqueue_single; + model->result_update = cn10k_ml_result_update; + model->set_error_code = cn10k_ml_set_error_code; + model->set_poll_addr = cn10k_ml_set_poll_addr; + return 0; } @@ -1144,26 +1102,8 @@ cn10k_ml_model_params_update(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_mode return 0; } -static __rte_always_inline void -queue_index_advance(uint64_t *index, uint64_t nb_desc) -{ - *index = (*index + 1) % nb_desc; -} - -static __rte_always_inline uint64_t -queue_pending_count(uint64_t head, uint64_t tail, uint64_t nb_desc) -{ - return (nb_desc + head - tail) % nb_desc; -} - -static __rte_always_inline uint64_t -queue_free_count(uint64_t head, uint64_t tail, uint64_t nb_desc) -{ - return nb_desc - queue_pending_count(head, tail, nb_desc) - 1; -} - -static __rte_always_inline void -cn10k_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, struct cnxk_ml_req *req) +__rte_hot void +cn10k_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request) { union cn10k_ml_error_code *error_code; struct cn10k_ml_layer_xstats *xstats; @@ -1171,6 +1111,7 @@ cn10k_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, struct cnxk_ml struct cn10k_ml_result *result; struct cnxk_ml_model *model; struct cnxk_ml_layer *layer; + struct cnxk_ml_req *req; struct cnxk_ml_qp *qp; struct rte_ml_op *op; uint64_t hw_latency; @@ -1178,9 +1119,9 @@ cn10k_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, struct cnxk_ml uint16_t model_id; uint16_t layer_id; + req = (struct cnxk_ml_req *)request; result = &req->cn10k_req.result; op = req->op; - if (likely(result->error_code == 0)) { model_id = cnxk_mldev->index_map[op->model_id].model_id; layer_id = cnxk_mldev->index_map[op->model_id].layer_id; @@ -1247,119 +1188,48 @@ cn10k_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, struct cnxk_ml op->user_ptr = result->user_ptr; } -__rte_hot uint16_t -cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, - uint16_t nb_ops) +__rte_hot void +cn10k_ml_set_error_code(struct cnxk_ml_req *req, uint64_t etype, uint64_t stype) +{ + union cn10k_ml_error_code *error_code; + + error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; + error_code->s.etype = etype; + error_code->s.stype = stype; +} + +__rte_hot bool +cn10k_ml_enqueue_single(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op, uint16_t layer_id, + struct cnxk_ml_qp *qp, uint64_t head) { union cn10k_ml_error_code *error_code; struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; struct cnxk_ml_queue *queue; struct cnxk_ml_req *req; - struct cnxk_ml_qp *qp; - struct rte_ml_op *op; - - uint16_t count; - uint64_t head; - bool enqueued; - cnxk_mldev = dev->data->dev_private; cn10k_mldev = &cnxk_mldev->cn10k_mldev; - qp = dev->data->queue_pairs[qp_id]; queue = &qp->queue; - - head = queue->head; - nb_ops = PLT_MIN(nb_ops, queue_free_count(head, queue->tail, qp->nb_desc)); - count = 0; - - if (unlikely(nb_ops == 0)) - return 0; - -enqueue_req: - op = ops[count]; req = &queue->reqs[head]; - cn10k_mldev->set_poll_addr(req); - cn10k_ml_prep_fp_job_descriptor(cnxk_mldev, req, op); + model = cnxk_mldev->mldev->data->models[op->model_id]; + model->set_poll_addr(req); + cn10k_ml_prep_fp_job_descriptor(cnxk_mldev, req, model->layer[layer_id].index, + op->input[0]->addr, op->output[0]->addr, op->nb_batches); memset(&req->cn10k_req.result, 0, sizeof(struct cn10k_ml_result)); error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; error_code->s.etype = ML_ETYPE_UNKNOWN; req->cn10k_req.result.user_ptr = op->user_ptr; - cn10k_mldev->set_poll_ptr(req); - enqueued = cn10k_mldev->ml_jcmdq_enqueue(&cn10k_mldev->roc, &req->cn10k_req.jcmd); - if (unlikely(!enqueued)) - goto jcmdq_full; + cnxk_ml_set_poll_ptr(req); + if (unlikely(!cn10k_mldev->ml_jcmdq_enqueue(&cn10k_mldev->roc, &req->cn10k_req.jcmd))) + return false; req->timeout = plt_tsc_cycles() + queue->wait_cycles; req->op = op; - queue_index_advance(&head, qp->nb_desc); - count++; - - if (count < nb_ops) - goto enqueue_req; - -jcmdq_full: - queue->head = head; - qp->stats.enqueued_count += count; - - return count; -} - -__rte_hot uint16_t -cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, - uint16_t nb_ops) -{ - union cn10k_ml_error_code *error_code; - struct cn10k_ml_dev *cn10k_mldev; - struct cnxk_ml_dev *cnxk_mldev; - struct cnxk_ml_queue *queue; - struct cnxk_ml_req *req; - struct cnxk_ml_qp *qp; - - uint64_t status; - uint16_t count; - uint64_t tail; - - cnxk_mldev = dev->data->dev_private; - cn10k_mldev = &cnxk_mldev->cn10k_mldev; - qp = dev->data->queue_pairs[qp_id]; - queue = &qp->queue; - - tail = queue->tail; - nb_ops = PLT_MIN(nb_ops, queue_pending_count(queue->head, tail, qp->nb_desc)); - count = 0; - - if (unlikely(nb_ops == 0)) - goto empty_or_active; - -dequeue_req: - req = &queue->reqs[tail]; - status = cn10k_mldev->get_poll_ptr(req); - if (unlikely(status != ML_CNXK_POLL_JOB_FINISH)) { - if (plt_tsc_cycles() < req->timeout) { - goto empty_or_active; - } else { /* Timeout, set indication of driver error */ - error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; - error_code->s.etype = ML_ETYPE_DRIVER; - } - } - - cn10k_ml_result_update(cnxk_mldev, qp_id, req); - ops[count] = req->op; - - queue_index_advance(&tail, qp->nb_desc); - count++; - - if (count < nb_ops) - goto dequeue_req; - -empty_or_active: - queue->tail = tail; - - return count; + return true; } __rte_hot int @@ -1396,41 +1266,48 @@ cn10k_ml_op_error_get(struct rte_ml_dev *dev, struct rte_ml_op *op, struct rte_m } __rte_hot int -cn10k_ml_inference_sync(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op) +cn10k_ml_inference_sync(void *device, uint16_t index, void *input, void *output, + uint16_t nb_batches) { union cn10k_ml_error_code *error_code; struct cn10k_ml_dev *cn10k_mldev; + struct cnxk_ml_dev *cnxk_mldev; struct cnxk_ml_model *model; struct cnxk_ml_layer *layer; struct cnxk_ml_req *req; + struct rte_ml_op op; uint16_t model_id; uint16_t layer_id; bool timeout; int ret = 0; + cnxk_mldev = (struct cnxk_ml_dev *)device; cn10k_mldev = &cnxk_mldev->cn10k_mldev; - model_id = cnxk_mldev->index_map[op->model_id].model_id; - layer_id = cnxk_mldev->index_map[op->model_id].layer_id; + model_id = cnxk_mldev->index_map[index].model_id; + layer_id = cnxk_mldev->index_map[index].layer_id; model = cnxk_mldev->mldev->data->models[model_id]; layer = &model->layer[layer_id]; req = layer->glow.req; + op.model_id = index; + op.impl_opaque = 0; + cn10k_ml_set_poll_addr(req); - cn10k_ml_prep_fp_job_descriptor(cnxk_mldev, req, op); + cn10k_ml_prep_fp_job_descriptor(cnxk_mldev, req, index, input, output, nb_batches); memset(&req->cn10k_req.result, 0, sizeof(struct cn10k_ml_result)); error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; error_code->s.etype = ML_ETYPE_UNKNOWN; - req->cn10k_req.result.user_ptr = op->user_ptr; + req->cn10k_req.result.user_ptr = NULL; - cn10k_mldev->set_poll_ptr(req); + cnxk_ml_set_poll_ptr(req); req->cn10k_req.jcmd.w1.s.jobptr = PLT_U64_CAST(&req->cn10k_req.jd); timeout = true; req->timeout = plt_tsc_cycles() + ML_CNXK_CMD_TIMEOUT * plt_tsc_hz(); do { if (cn10k_mldev->ml_jcmdq_enqueue(&cn10k_mldev->roc, &req->cn10k_req.jcmd)) { - req->op = op; + req->op = &op; timeout = false; break; } @@ -1443,7 +1320,7 @@ cn10k_ml_inference_sync(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op) timeout = true; do { - if (cn10k_mldev->get_poll_ptr(req) == ML_CNXK_POLL_JOB_FINISH) { + if (cnxk_ml_get_poll_ptr(req) == ML_CNXK_POLL_JOB_FINISH) { timeout = false; break; } diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 8a090a31592..3e75cae65a3 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -13,6 +13,7 @@ struct cnxk_ml_dev; struct cnxk_ml_qp; struct cnxk_ml_model; +struct cnxk_ml_req; /* Firmware version string length */ #define MLDEV_FIRMWARE_VERSION_LENGTH 32 @@ -308,13 +309,15 @@ int cn10k_ml_model_params_update(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_ void *buffer); /* Fast-path ops */ -__rte_hot uint16_t cn10k_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, - struct rte_ml_op **ops, uint16_t nb_ops); -__rte_hot uint16_t cn10k_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, - struct rte_ml_op **ops, uint16_t nb_ops); +__rte_hot bool cn10k_ml_enqueue_single(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op, + uint16_t layer_id, struct cnxk_ml_qp *qp, uint64_t head); __rte_hot int cn10k_ml_op_error_get(struct rte_ml_dev *dev, struct rte_ml_op *op, struct rte_ml_op_error *error); -__rte_hot int cn10k_ml_inference_sync(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op); +__rte_hot int cn10k_ml_inference_sync(void *device, uint16_t index, void *input, void *output, + uint16_t nb_batches); +__rte_hot void cn10k_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request); +__rte_hot void cn10k_ml_set_error_code(struct cnxk_ml_req *req, uint64_t etype, uint64_t stype); +__rte_hot void cn10k_ml_set_poll_addr(struct cnxk_ml_req *req); /* Misc ops */ void cn10k_ml_qp_initialize(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_qp *qp); diff --git a/drivers/ml/cnxk/cnxk_ml_model.h b/drivers/ml/cnxk/cnxk_ml_model.h index 66d979dd3ca..f618e5aa5fc 100644 --- a/drivers/ml/cnxk/cnxk_ml_model.h +++ b/drivers/ml/cnxk/cnxk_ml_model.h @@ -15,6 +15,8 @@ struct cnxk_ml_dev; struct cnxk_ml_model; +struct cnxk_ml_qp; +struct cnxk_ml_req; /* Model state */ enum cnxk_ml_model_state { @@ -70,6 +72,12 @@ struct cnxk_ml_layer { struct cn10k_ml_layer_data glow; }; +typedef bool (*enqueue_single_t)(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op, + uint16_t layer_id, struct cnxk_ml_qp *qp, uint64_t head); +typedef void (*result_update_t)(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request); +typedef void (*set_error_code_t)(struct cnxk_ml_req *req, uint64_t etype, uint64_t stype); +typedef void (*set_poll_addr_t)(struct cnxk_ml_req *req); + /* Model Object */ struct cnxk_ml_model { /* Device reference */ @@ -106,6 +114,12 @@ struct cnxk_ml_model { /* Spinlock, used to update model state */ plt_spinlock_t lock; + + /* Fast-path functions */ + enqueue_single_t enqueue_single; + result_update_t result_update; + set_error_code_t set_error_code; + set_poll_addr_t set_poll_addr; }; void cnxk_ml_model_dump(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, FILE *fp); diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 96d40029b36..4bf8bd25457 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -17,6 +17,18 @@ /* ML model macros */ #define CNXK_ML_MODEL_MEMZONE_NAME "ml_cnxk_model_mz" +__rte_hot void +cnxk_ml_set_poll_ptr(struct cnxk_ml_req *req) +{ + plt_write64(ML_CNXK_POLL_JOB_START, req->status); +} + +__rte_hot uint64_t +cnxk_ml_get_poll_ptr(struct cnxk_ml_req *req) +{ + return plt_read64(req->status); +} + static void qp_memzone_name_get(char *name, int size, int dev_id, int qp_id) { @@ -1323,6 +1335,122 @@ cnxk_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_b return 0; } +static __rte_always_inline void +queue_index_advance(uint64_t *index, uint64_t nb_desc) +{ + *index = (*index + 1) % nb_desc; +} + +static __rte_always_inline uint64_t +queue_pending_count(uint64_t head, uint64_t tail, uint64_t nb_desc) +{ + return (nb_desc + head - tail) % nb_desc; +} + +static __rte_always_inline uint64_t +queue_free_count(uint64_t head, uint64_t tail, uint64_t nb_desc) +{ + return nb_desc - queue_pending_count(head, tail, nb_desc) - 1; +} + +__rte_hot uint16_t +cnxk_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, + uint16_t nb_ops) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + struct cnxk_ml_queue *queue; + struct cnxk_ml_qp *qp; + struct rte_ml_op *op; + + uint16_t layer_id = 0; + uint16_t count; + uint64_t head; + + cnxk_mldev = dev->data->dev_private; + qp = dev->data->queue_pairs[qp_id]; + queue = &qp->queue; + + head = queue->head; + nb_ops = PLT_MIN(nb_ops, queue_free_count(head, queue->tail, qp->nb_desc)); + count = 0; + + if (unlikely(nb_ops == 0)) + return 0; + +enqueue_req: + op = ops[count]; + model = cnxk_mldev->mldev->data->models[op->model_id]; + + if (unlikely(!model->enqueue_single(cnxk_mldev, op, layer_id, qp, head))) + goto jcmdq_full; + + queue_index_advance(&head, qp->nb_desc); + count++; + + if (count < nb_ops) + goto enqueue_req; + +jcmdq_full: + queue->head = head; + qp->stats.enqueued_count += count; + + return count; +} + +__rte_hot uint16_t +cnxk_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, + uint16_t nb_ops) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_queue *queue; + struct cnxk_ml_model *model; + struct cnxk_ml_req *req; + struct cnxk_ml_qp *qp; + + uint64_t status; + uint16_t count; + uint64_t tail; + + cnxk_mldev = dev->data->dev_private; + qp = dev->data->queue_pairs[qp_id]; + queue = &qp->queue; + + tail = queue->tail; + nb_ops = PLT_MIN(nb_ops, queue_pending_count(queue->head, tail, qp->nb_desc)); + count = 0; + + if (unlikely(nb_ops == 0)) + goto empty_or_active; + +dequeue_req: + + req = &queue->reqs[tail]; + model = cnxk_mldev->mldev->data->models[req->op->model_id]; + + status = cnxk_ml_get_poll_ptr(req); + if (unlikely(status != ML_CNXK_POLL_JOB_FINISH)) { + if (plt_tsc_cycles() < req->timeout) + goto empty_or_active; + else /* Timeout, set indication of driver error */ + model->set_error_code(req, ML_ETYPE_DRIVER, 0); + } + + model->result_update(cnxk_mldev, qp->id, req); + + ops[count] = req->op; + queue_index_advance(&tail, qp->nb_desc); + count++; + + if (count < nb_ops) + goto dequeue_req; + +empty_or_active: + queue->tail = tail; + + return count; +} + struct rte_ml_dev_ops cnxk_ml_ops = { /* Device control ops */ .dev_info_get = cnxk_ml_dev_info_get, diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h index d27ca0d0cb2..d0c126f34b7 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.h +++ b/drivers/ml/cnxk/cnxk_ml_ops.h @@ -65,4 +65,11 @@ extern struct rte_ml_dev_ops cnxk_ml_ops; int cnxk_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id); int cnxk_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id); +__rte_hot uint16_t cnxk_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, + struct rte_ml_op **ops, uint16_t nb_ops); +__rte_hot uint16_t cnxk_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, + struct rte_ml_op **ops, uint16_t nb_ops); +__rte_hot void cnxk_ml_set_poll_ptr(struct cnxk_ml_req *req); +__rte_hot uint64_t cnxk_ml_get_poll_ptr(struct cnxk_ml_req *req); + #endif /* _CNXK_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130900 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9AA8041FD1; Wed, 30 Aug 2023 18:03:14 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 225CE40A73; Wed, 30 Aug 2023 17:59:59 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 79D3C402AE for ; Wed, 30 Aug 2023 17:59:41 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P93M004408 for ; Wed, 30 Aug 2023 08:59:40 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=taxDAuKfHyztP/DLC3hzlJcadJd5iiBq1iHSOOdN8wo=; b=FQyCa7E8c4g0IaayXkdLgmYMfu2pQWQJd4CbjwYteDUV/lTwQlefM8rwr5AQSxO5rd/W ygKZd5cVlGv7DTpl0OqcROzD8MG0HxzBwfwBhfP9UCJvucCi5UNJKapIW8JHNt0jZU1N FcmtxfWDSJQJx2zf+lJ/Sjww0A83xzVh8hzWlsTVPyIueqz+9MCjSVm9HagvVrR4ZKBQ KjockFtbmpNs1uD1RO8aaDvVXbg0BD/U9QuXHP7vqFjhMhlurHaY8a0qaJmhWIwkc6u/ 7FU2bu6Fk/xmQHHqdPId4DqnlL9drjtcPSK/kHg9a/CYT3om6nrryuSVqBRjMUo+Ofrj eQ== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysstm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:39 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:38 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:38 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 037B73F707D; Wed, 30 Aug 2023 08:59:37 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 18/34] ml/cnxk: move error handling to cnxk layer Date: Wed, 30 Aug 2023 08:59:08 -0700 Message-ID: <20230830155927.3566-19-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: X8IIzcc_m2vtsxtBPFHsk_oNZuewWose X-Proofpoint-ORIG-GUID: X8IIzcc_m2vtsxtBPFHsk_oNZuewWose X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Move error type structures to cnxk layer. cn10k layer to handle fw and hw error sub-types only. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.h | 41 ++++++--------- drivers/ml/cnxk/cn10k_ml_ops.c | 93 +++++++++++++--------------------- drivers/ml/cnxk/cnxk_ml_dev.c | 8 +++ drivers/ml/cnxk/cnxk_ml_dev.h | 18 +++++++ drivers/ml/cnxk/cnxk_ml_ops.c | 2 +- 5 files changed, 78 insertions(+), 84 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_dev.h b/drivers/ml/cnxk/cn10k_ml_dev.h index 94a94d996f8..2e7eb6c9ef9 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.h +++ b/drivers/ml/cnxk/cn10k_ml_dev.h @@ -52,38 +52,27 @@ struct cnxk_ml_dev; struct cnxk_ml_req; struct cnxk_ml_qp; -/* Error types enumeration */ -enum cn10k_ml_error_etype { - /* 0x0 */ ML_ETYPE_NO_ERROR = 0, /* No error */ - /* 0x1 */ ML_ETYPE_FW_NONFATAL, /* Firmware non-fatal error */ - /* 0x2 */ ML_ETYPE_HW_NONFATAL, /* Hardware non-fatal error */ - /* 0x3 */ ML_ETYPE_HW_FATAL, /* Hardware fatal error */ - /* 0x4 */ ML_ETYPE_HW_WARNING, /* Hardware warning */ - /* 0x5 */ ML_ETYPE_DRIVER, /* Driver specific error */ - /* 0x6 */ ML_ETYPE_UNKNOWN, /* Unknown error */ -}; - /* Firmware non-fatal error sub-type */ enum cn10k_ml_error_stype_fw_nf { - /* 0x0 */ ML_FW_ERR_NOERR = 0, /* No error */ - /* 0x1 */ ML_FW_ERR_UNLOAD_ID_NOT_FOUND, /* Model ID not found during load */ - /* 0x2 */ ML_FW_ERR_LOAD_LUT_OVERFLOW, /* Lookup table overflow at load */ - /* 0x3 */ ML_FW_ERR_ID_IN_USE, /* Model ID already in use */ - /* 0x4 */ ML_FW_ERR_INVALID_TILEMASK, /* Invalid OCM tilemask */ - /* 0x5 */ ML_FW_ERR_RUN_LUT_OVERFLOW, /* Lookup table overflow at run */ - /* 0x6 */ ML_FW_ERR_RUN_ID_NOT_FOUND, /* Model ID not found during run */ - /* 0x7 */ ML_FW_ERR_COMMAND_NOTSUP, /* Unsupported command */ - /* 0x8 */ ML_FW_ERR_DDR_ADDR_RANGE, /* DDR address out of range */ - /* 0x9 */ ML_FW_ERR_NUM_BATCHES_INVALID, /* Invalid number of batches */ - /* 0xA */ ML_FW_ERR_INSSYNC_TIMEOUT, /* INS sync timeout */ + /* 0x0 */ ML_CN10K_FW_ERR_NOERR = 0, /* No error */ + /* 0x1 */ ML_CN10K_FW_ERR_UNLOAD_ID_NOT_FOUND, /* Model ID not found during load */ + /* 0x2 */ ML_CN10K_FW_ERR_LOAD_LUT_OVERFLOW, /* Lookup table overflow at load */ + /* 0x3 */ ML_CN10K_FW_ERR_ID_IN_USE, /* Model ID already in use */ + /* 0x4 */ ML_CN10K_FW_ERR_INVALID_TILEMASK, /* Invalid OCM tilemask */ + /* 0x5 */ ML_CN10K_FW_ERR_RUN_LUT_OVERFLOW, /* Lookup table overflow at run */ + /* 0x6 */ ML_CN10K_FW_ERR_RUN_ID_NOT_FOUND, /* Model ID not found during run */ + /* 0x7 */ ML_CN10K_FW_ERR_COMMAND_NOTSUP, /* Unsupported command */ + /* 0x8 */ ML_CN10K_FW_ERR_DDR_ADDR_RANGE, /* DDR address out of range */ + /* 0x9 */ ML_CN10K_FW_ERR_NUM_BATCHES_INVALID, /* Invalid number of batches */ + /* 0xA */ ML_CN10K_FW_ERR_INSSYNC_TIMEOUT, /* INS sync timeout */ }; /* Driver error sub-type */ enum cn10k_ml_error_stype_driver { - /* 0x0 */ ML_DRIVER_ERR_NOERR = 0, /* No error */ - /* 0x1 */ ML_DRIVER_ERR_UNKNOWN, /* Unable to determine error sub-type */ - /* 0x2 */ ML_DRIVER_ERR_EXCEPTION, /* Firmware exception */ - /* 0x3 */ ML_DRIVER_ERR_FW_ERROR, /* Unknown firmware error */ + /* 0x0 */ ML_CN10K_DRIVER_ERR_NOERR = 0, /* No error */ + /* 0x1 */ ML_CN10K_DRIVER_ERR_UNKNOWN, /* Unable to determine error sub-type */ + /* 0x2 */ ML_CN10K_DRIVER_ERR_EXCEPTION, /* Firmware exception */ + /* 0x3 */ ML_CN10K_DRIVER_ERR_FW_ERROR, /* Unknown firmware error */ }; /* Error structure */ diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 7d809d25ae0..daeb3b712c5 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -26,47 +26,27 @@ #define ML_FLAGS_POLL_COMPL BIT(0) #define ML_FLAGS_SSO_COMPL BIT(1) -/* Error message length */ -#define ERRMSG_LEN 32 - -/* Error type database */ -static const struct cn10k_ml_etype_db { - enum cn10k_ml_error_etype etype; - char name[ERRMSG_LEN]; -} ml_etype_db[] = { - {ML_ETYPE_NO_ERROR, "NO_ERROR"}, {ML_ETYPE_FW_NONFATAL, "FW_NON_FATAL"}, - {ML_ETYPE_HW_NONFATAL, "HW_NON_FATAL"}, {ML_ETYPE_HW_FATAL, "HW_FATAL"}, - {ML_ETYPE_HW_WARNING, "HW_WARNING"}, {ML_ETYPE_DRIVER, "DRIVER_ERROR"}, - {ML_ETYPE_UNKNOWN, "UNKNOWN_ERROR"}, -}; - /* Hardware non-fatal error subtype database */ -static const struct cn10k_ml_stype_db_hw_nf { - enum cn10k_ml_error_stype_fw_nf stype; - char msg[ERRMSG_LEN]; -} ml_stype_db_hw_nf[] = { - {ML_FW_ERR_NOERR, "NO ERROR"}, - {ML_FW_ERR_UNLOAD_ID_NOT_FOUND, "UNLOAD MODEL ID NOT FOUND"}, - {ML_FW_ERR_LOAD_LUT_OVERFLOW, "LOAD LUT OVERFLOW"}, - {ML_FW_ERR_ID_IN_USE, "MODEL ID IN USE"}, - {ML_FW_ERR_INVALID_TILEMASK, "INVALID TILEMASK"}, - {ML_FW_ERR_RUN_LUT_OVERFLOW, "RUN LUT OVERFLOW"}, - {ML_FW_ERR_RUN_ID_NOT_FOUND, "RUN MODEL ID NOT FOUND"}, - {ML_FW_ERR_COMMAND_NOTSUP, "COMMAND NOT SUPPORTED"}, - {ML_FW_ERR_DDR_ADDR_RANGE, "DDR ADDRESS OUT OF RANGE"}, - {ML_FW_ERR_NUM_BATCHES_INVALID, "INVALID BATCHES"}, - {ML_FW_ERR_INSSYNC_TIMEOUT, "INSSYNC TIMEOUT"}, +static struct cnxk_ml_error_db ml_stype_db_hw_nf[] = { + {ML_CN10K_FW_ERR_NOERR, "NO ERROR"}, + {ML_CN10K_FW_ERR_UNLOAD_ID_NOT_FOUND, "UNLOAD MODEL ID NOT FOUND"}, + {ML_CN10K_FW_ERR_LOAD_LUT_OVERFLOW, "LOAD LUT OVERFLOW"}, + {ML_CN10K_FW_ERR_ID_IN_USE, "MODEL ID IN USE"}, + {ML_CN10K_FW_ERR_INVALID_TILEMASK, "INVALID TILEMASK"}, + {ML_CN10K_FW_ERR_RUN_LUT_OVERFLOW, "RUN LUT OVERFLOW"}, + {ML_CN10K_FW_ERR_RUN_ID_NOT_FOUND, "RUN MODEL ID NOT FOUND"}, + {ML_CN10K_FW_ERR_COMMAND_NOTSUP, "COMMAND NOT SUPPORTED"}, + {ML_CN10K_FW_ERR_DDR_ADDR_RANGE, "DDR ADDRESS OUT OF RANGE"}, + {ML_CN10K_FW_ERR_NUM_BATCHES_INVALID, "INVALID BATCHES"}, + {ML_CN10K_FW_ERR_INSSYNC_TIMEOUT, "INSSYNC TIMEOUT"}, }; /* Driver error subtype database */ -static const struct cn10k_ml_stype_db_driver { - enum cn10k_ml_error_stype_driver stype; - char msg[ERRMSG_LEN]; -} ml_stype_db_driver[] = { - {ML_DRIVER_ERR_NOERR, "NO ERROR"}, - {ML_DRIVER_ERR_UNKNOWN, "UNKNOWN ERROR"}, - {ML_DRIVER_ERR_EXCEPTION, "FW EXCEPTION"}, - {ML_DRIVER_ERR_FW_ERROR, "UNKNOWN FIRMWARE ERROR"}, +static struct cnxk_ml_error_db ml_stype_db_driver[] = { + {ML_CN10K_DRIVER_ERR_NOERR, "NO ERROR"}, + {ML_CN10K_DRIVER_ERR_UNKNOWN, "UNKNOWN ERROR"}, + {ML_CN10K_DRIVER_ERR_EXCEPTION, "FW EXCEPTION"}, + {ML_CN10K_DRIVER_ERR_FW_ERROR, "UNKNOWN FIRMWARE ERROR"}, }; __rte_hot void @@ -1166,19 +1146,19 @@ cn10k_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request) /* Handle driver error */ error_code = (union cn10k_ml_error_code *)&result->error_code; - if (error_code->s.etype == ML_ETYPE_DRIVER) { + if (error_code->s.etype == ML_CNXK_ETYPE_DRIVER) { cn10k_mldev = &cnxk_mldev->cn10k_mldev; /* Check for exception */ if ((roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C0) != 0) || (roc_ml_reg_read64(&cn10k_mldev->roc, ML_SCRATCH_EXCEPTION_SP_C1) != 0)) - error_code->s.stype = ML_DRIVER_ERR_EXCEPTION; + error_code->s.stype = ML_CN10K_DRIVER_ERR_EXCEPTION; else if ((roc_ml_reg_read64(&cn10k_mldev->roc, ML_CORE_INT_LO) != 0) || (roc_ml_reg_read64(&cn10k_mldev->roc, ML_CORE_INT_HI) != 0)) - error_code->s.stype = ML_DRIVER_ERR_FW_ERROR; + error_code->s.stype = ML_CN10K_DRIVER_ERR_FW_ERROR; else - error_code->s.stype = ML_DRIVER_ERR_UNKNOWN; + error_code->s.stype = ML_CN10K_DRIVER_ERR_UNKNOWN; } op->impl_opaque = result->error_code; @@ -1219,7 +1199,7 @@ cn10k_ml_enqueue_single(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op, ui memset(&req->cn10k_req.result, 0, sizeof(struct cn10k_ml_result)); error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; - error_code->s.etype = ML_ETYPE_UNKNOWN; + error_code->s.etype = ML_CNXK_ETYPE_UNKNOWN; req->cn10k_req.result.user_ptr = op->user_ptr; cnxk_ml_set_poll_ptr(req); @@ -1236,30 +1216,29 @@ __rte_hot int cn10k_ml_op_error_get(struct rte_ml_dev *dev, struct rte_ml_op *op, struct rte_ml_op_error *error) { union cn10k_ml_error_code *error_code; - char msg[RTE_ML_STR_MAX]; PLT_SET_USED(dev); error_code = (union cn10k_ml_error_code *)&op->impl_opaque; - /* Copy error message */ - plt_strlcpy(msg, ml_etype_db[error_code->s.etype].name, sizeof(msg)); - /* Copy sub error message */ - if (error_code->s.etype == ML_ETYPE_HW_NONFATAL) { - strcat(msg, " : "); + if (error_code->s.etype == ML_CNXK_ETYPE_HW_NONFATAL) { if (error_code->s.stype < PLT_DIM(ml_stype_db_hw_nf)) - strcat(msg, ml_stype_db_hw_nf[error_code->s.stype].msg); + snprintf(error->message, RTE_ML_STR_MAX, "%s : %s", + ml_etype_db[error_code->s.etype].str, + ml_stype_db_hw_nf[error_code->s.stype].str); else - strcat(msg, "UNKNOWN ERROR"); - } - - if (error_code->s.etype == ML_ETYPE_DRIVER) { - strcat(msg, " : "); - strcat(msg, ml_stype_db_driver[error_code->s.stype].msg); + snprintf(error->message, RTE_ML_STR_MAX, "%s : UNKNOWN ERROR", + ml_etype_db[error_code->s.etype].str); + } else if (error_code->s.etype == ML_CNXK_ETYPE_DRIVER) { + snprintf(error->message, RTE_ML_STR_MAX, "%s : %s", + ml_etype_db[error_code->s.etype].str, + ml_stype_db_driver[error_code->s.stype].str); + } else { + snprintf(error->message, RTE_ML_STR_MAX, "%s", + ml_etype_db[error_code->s.etype].str); } - plt_strlcpy(error->message, msg, sizeof(error->message)); error->errcode = error_code->u64; return 0; @@ -1297,7 +1276,7 @@ cn10k_ml_inference_sync(void *device, uint16_t index, void *input, void *output, memset(&req->cn10k_req.result, 0, sizeof(struct cn10k_ml_result)); error_code = (union cn10k_ml_error_code *)&req->cn10k_req.result.error_code; - error_code->s.etype = ML_ETYPE_UNKNOWN; + error_code->s.etype = ML_CNXK_ETYPE_UNKNOWN; req->cn10k_req.result.user_ptr = NULL; cnxk_ml_set_poll_ptr(req); diff --git a/drivers/ml/cnxk/cnxk_ml_dev.c b/drivers/ml/cnxk/cnxk_ml_dev.c index 2a5c17c973b..63d1c9e417b 100644 --- a/drivers/ml/cnxk/cnxk_ml_dev.c +++ b/drivers/ml/cnxk/cnxk_ml_dev.c @@ -9,3 +9,11 @@ /* Dummy operations for ML device */ struct rte_ml_dev_ops ml_dev_dummy_ops = {0}; + +/* Error type database */ +struct cnxk_ml_error_db ml_etype_db[] = { + {ML_CNXK_ETYPE_NO_ERROR, "NO_ERROR"}, {ML_CNXK_ETYPE_FW_NONFATAL, "FW_NON_FATAL"}, + {ML_CNXK_ETYPE_HW_NONFATAL, "HW_NON_FATAL"}, {ML_CNXK_ETYPE_HW_FATAL, "HW_FATAL"}, + {ML_CNXK_ETYPE_HW_WARNING, "HW_WARNING"}, {ML_CNXK_ETYPE_DRIVER, "DRIVER_ERROR"}, + {ML_CNXK_ETYPE_UNKNOWN, "UNKNOWN_ERROR"}, +}; diff --git a/drivers/ml/cnxk/cnxk_ml_dev.h b/drivers/ml/cnxk/cnxk_ml_dev.h index 3ce9338f1f1..382fca64bea 100644 --- a/drivers/ml/cnxk/cnxk_ml_dev.h +++ b/drivers/ml/cnxk/cnxk_ml_dev.h @@ -18,6 +18,22 @@ #define ML_CNXK_POLL_JOB_START 0 #define ML_CNXK_POLL_JOB_FINISH 1 +/* Error types enumeration */ +enum cnxk_ml_error_etype { + /* 0x0 */ ML_CNXK_ETYPE_NO_ERROR = 0, /* No error */ + /* 0x1 */ ML_CNXK_ETYPE_FW_NONFATAL, /* Firmware non-fatal error */ + /* 0x2 */ ML_CNXK_ETYPE_HW_NONFATAL, /* Hardware non-fatal error */ + /* 0x3 */ ML_CNXK_ETYPE_HW_FATAL, /* Hardware fatal error */ + /* 0x4 */ ML_CNXK_ETYPE_HW_WARNING, /* Hardware warning */ + /* 0x5 */ ML_CNXK_ETYPE_DRIVER, /* Driver specific error */ + /* 0x6 */ ML_CNXK_ETYPE_UNKNOWN, /* Unknown error */ +}; + +struct cnxk_ml_error_db { + uint64_t code; + char str[RTE_ML_STR_MAX]; +}; + /* Device configuration state enum */ enum cnxk_ml_dev_state { /* Probed and not configured */ @@ -78,4 +94,6 @@ struct cnxk_ml_dev { struct cnxk_ml_index_map *index_map; }; +extern struct cnxk_ml_error_db ml_etype_db[]; + #endif /* _CNXK_ML_DEV_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 4bf8bd25457..b2eb4bd0d9a 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -1433,7 +1433,7 @@ cnxk_ml_dequeue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op * if (plt_tsc_cycles() < req->timeout) goto empty_or_active; else /* Timeout, set indication of driver error */ - model->set_error_code(req, ML_ETYPE_DRIVER, 0); + model->set_error_code(req, ML_CNXK_ETYPE_DRIVER, 0); } model->result_update(cnxk_mldev, qp->id, req); From patchwork Wed Aug 30 15:59:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130902 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B990841FD1; Wed, 30 Aug 2023 18:04:18 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6F72F40E96; Wed, 30 Aug 2023 18:00:03 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 41C8D402AE for ; Wed, 30 Aug 2023 17:59:42 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P93O004408 for ; Wed, 30 Aug 2023 08:59:41 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=DGSs8MbByPyhTcKHN/9qsDopr4or0e4onim1/rCXtzk=; b=M6zPUGOyWCF3NUMTJv4l0fUPweb0Whzc3aFn8SVPUNY5tAzc6zlV05Y9JYJbzh3CilI9 rieOzfhShCfRVy3xOtUIpbTH/vMfKjlVUOUGEVDd+P7CuIznruZiP7W7nAQcCbP8eCQx lW0Edak7JhdsCE1VK55Oj0JOwyuRyM9qesaPklce2GrVL+rjw+3siz7aQILllfKMz9F+ ARL7K5hC0M7DQ+6gBRH/hY88mM7ghvNs87BPmh33ysyXUtbwK/kM5Uq2vxCNsWlbiNnQ g2CTpqEOXNMfyuphU1P6CAfPspS1Z6CCG8wjxXMZwWBe/lUnMFOhVbyuD27cDCoNYpX7 Cw== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysstm-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:41 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:38 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:38 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 535BE3F707F; Wed, 30 Aug 2023 08:59:38 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 19/34] ml/cnxk: support config and close of tvmdp library Date: Wed, 30 Aug 2023 08:59:09 -0700 Message-ID: <20230830155927.3566-20-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: JsRLh8OZ4bD44_jbg5sW366pwVyUhR_V X-Proofpoint-ORIG-GUID: JsRLh8OZ4bD44_jbg5sW366pwVyUhR_V X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added support to configure and close TVMDP library based on ML device configuration options. Updated meson build to enable Jansson, TVM runtime, TVMDP library as build dependencies. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cnxk_ml_ops.c | 15 ++++++++++++ drivers/ml/cnxk/meson.build | 45 ++++++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_ops.c | 44 +++++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_ops.h | 15 ++++++++++++ 4 files changed, 119 insertions(+) create mode 100644 drivers/ml/cnxk/mvtvm_ml_ops.c create mode 100644 drivers/ml/cnxk/mvtvm_ml_ops.h diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index b2eb4bd0d9a..454fec33234 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -9,6 +9,10 @@ #include "cn10k_ml_ops.h" +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM +#include "mvtvm_ml_ops.h" +#endif + #include "cnxk_ml_dev.h" #include "cnxk_ml_io.h" #include "cnxk_ml_model.h" @@ -625,6 +629,12 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co goto error; } +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + ret = mvtvm_ml_dev_configure(cnxk_mldev, conf); + if (ret != 0) + goto error; +#endif + /* Set device capabilities */ cnxk_mldev->max_nb_layers = cnxk_mldev->cn10k_mldev.fw.req->cn10k_req.jd.fw_load.cap.s.max_models; @@ -685,6 +695,11 @@ cnxk_ml_dev_close(struct rte_ml_dev *dev) /* Un-initialize xstats */ cnxk_ml_xstats_uninit(cnxk_mldev); +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + if (mvtvm_ml_dev_close(cnxk_mldev) != 0) + plt_err("Failed to close MVTVM ML Device"); +#endif + if (cn10k_ml_dev_close(cnxk_mldev) != 0) plt_err("Failed to close CN10K ML Device"); diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 575f08f9c09..29dad0b0e33 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -7,6 +7,27 @@ if not is_linux or not dpdk_conf.get('RTE_ARCH_64') subdir_done() endif +enable_mvtvm = true + +if not jansson_dep.found() + message('drivers/ml/cnxk: jansson not found') + enable_mvtvm = false +endif + +tvmrt_lib = cc.find_library('tvm_runtime', required: false) +if tvmrt_lib.found() + tvmrt_dep = declare_dependency(dependencies: tvmrt_lib) +else + message('drivers/ml/cnxk: tvm_runtime not found') + enable_mvtvm = false +endif + +tvmdp_dep = dependency('tvmdp', required: false) +if not tvmdp_dep.found() + message('drivers/ml/cnxk: tvmdp not found') + enable_mvtvm = false +endif + driver_sdk_headers = files( 'cn10k_ml_dev.h', 'cn10k_ml_ops.h', @@ -34,6 +55,30 @@ sources = files( deps += ['mldev', 'common_cnxk', 'kvargs', 'hash'] +if enable_mvtvm + +dpdk_conf.set('RTE_MLDEV_CNXK_ENABLE_MVTVM', true) + +driver_sdk_headers += files( + 'mvtvm_ml_ops.h', +) + +sources += files( + 'mvtvm_ml_ops.c', +) + +ext_deps += tvmrt_dep +ext_deps += tvmdp_dep +ext_deps += cc.find_library('stdc++', required: true) +ext_deps += jansson_dep + +deps += ['bus_vdev'] + +message('drivers/ml/cnxk: Enabled TVM model support') +else +message('drivers/ml/cnxk: Disabled TVM model support') +endif + require_iova_in_mbuf = false if get_option('buildtype').contains('debug') diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c new file mode 100644 index 00000000000..0e1fc527daa --- /dev/null +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#include + +#include +#include +#include +#include + +#include "mvtvm_ml_ops.h" + +#include "cnxk_ml_dev.h" + +int +mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf) +{ + int ret; + + RTE_SET_USED(conf); + + /* Configure TVMDP library */ + ret = tvmdp_configure(cnxk_mldev->mldev->data->nb_models, rte_get_tsc_cycles); + if (ret != 0) + plt_err("TVMDP configuration failed, error = %d\n", ret); + + return ret; +} + +int +mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev) +{ + int ret; + + RTE_SET_USED(cnxk_mldev); + + /* Close TVMDP library configuration */ + ret = tvmdp_close(); + if (ret != 0) + plt_err("TVMDP close failed, error = %d\n", ret); + + return ret; +} diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.h b/drivers/ml/cnxk/mvtvm_ml_ops.h new file mode 100644 index 00000000000..988f3a1fd5e --- /dev/null +++ b/drivers/ml/cnxk/mvtvm_ml_ops.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#ifndef _MVTVM_ML_OPS_H_ +#define _MVTVM_ML_OPS_H_ + +#include + +struct cnxk_ml_dev; + +int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf); +int mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev); + +#endif /* _MVTVM_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130904 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2E01141FD1; Wed, 30 Aug 2023 18:04:45 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B46744064A; Wed, 30 Aug 2023 18:00:05 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id A788D402B5 for ; Wed, 30 Aug 2023 17:59:42 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P93P004408 for ; Wed, 30 Aug 2023 08:59:41 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=29FFLZaRTWpm+jfcgRt/1fehwhYh+Jp5vjBWJOfNQTs=; b=HCFGp4NSkcsR+m5vu62MHJgMBd8uErqmPFCwCAUULkYqeDxU57Cl8TjfnB+aycoyl3hn RzFX2a7HTUyZyuXQnj/qaUgRrw0+5d15/GRgoNjNF0oo2edp2k1UO4k0U9dUTDY8wzFv LgS5gEUw757G0L0sQ3HJ9+yzEmYfgmZp4GBc+WHrzQhwdH3s4ydfy7HsxpJyKYljXNac u8DsonjD8TOVlyGQe/OrL+9erPmrFwmU7xKNBgGygTWd4kYWdUI4fmw/9PtFEkg/dqrS teIU4TiB51kG+dW/KiGEB8HphIVj63K8JVl7HVDN+w7Xe5NYU/fhF8T8PN+43FiNiwqZ EA== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysstm-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:41 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:38 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:38 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id AA1423F7080; Wed, 30 Aug 2023 08:59:38 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 20/34] ml/cnxk: add structures to support TVM model type Date: Wed, 30 Aug 2023 08:59:10 -0700 Message-ID: <20230830155927.3566-21-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: 6iXNecTGuUkLYlvxAN4GPlFBLB_os8yg X-Proofpoint-ORIG-GUID: 6iXNecTGuUkLYlvxAN4GPlFBLB_os8yg X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Introduced model type, sub-type and layer type. Added internal structures for TVM model objects. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ocm.c | 3 ++ drivers/ml/cnxk/cn10k_ml_ops.c | 6 ++- drivers/ml/cnxk/cnxk_ml_model.h | 63 +++++++++++++++++++++++++++++++- drivers/ml/cnxk/cnxk_ml_ops.c | 54 ++++++++++++++++++++++----- drivers/ml/cnxk/meson.build | 1 + drivers/ml/cnxk/mvtvm_ml_model.h | 46 +++++++++++++++++++++++ 6 files changed, 160 insertions(+), 13 deletions(-) create mode 100644 drivers/ml/cnxk/mvtvm_ml_model.h diff --git a/drivers/ml/cnxk/cn10k_ml_ocm.c b/drivers/ml/cnxk/cn10k_ml_ocm.c index 7d4b1efad13..c665e2cf661 100644 --- a/drivers/ml/cnxk/cn10k_ml_ocm.c +++ b/drivers/ml/cnxk/cn10k_ml_ocm.c @@ -437,6 +437,9 @@ cn10k_ml_ocm_free_pages(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id, uint1 for (j = 0; j < local_model->nb_layers; j++) { local_layer = &local_model->layer[j]; + if (local_layer->type != ML_CNXK_LAYER_TYPE_MRVL) + continue; + if (local_layer != layer && local_layer->glow.ocm_map.ocm_reserved) { if (IS_BIT_SET(local_layer->glow.ocm_map.tilemask, tile_id)) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index daeb3b712c5..db18f320527 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -650,6 +650,9 @@ cn10k_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * if (ret != 0) return ret; + /* Set model sub type */ + model->subtype = ML_CNXK_MODEL_SUBTYPE_GLOW_MRVL; + /* Copy metadata to internal buffer */ rte_memcpy(&model->glow.metadata, params->addr, sizeof(struct cn10k_ml_model_metadata)); cn10k_ml_model_metadata_update(&model->glow.metadata); @@ -671,6 +674,7 @@ cn10k_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * /* Load layer and get the index */ layer = &model->layer[0]; + layer->type = ML_CNXK_LAYER_TYPE_MRVL; ret = cn10k_ml_layer_load(cnxk_mldev, model->model_id, NULL, params->addr, params->size, &layer->index); if (ret != 0) { @@ -894,7 +898,7 @@ cn10k_ml_layer_start(void *device, uint16_t model_id, const char *layer_name) if (ret < 0) { cn10k_ml_layer_stop(device, model_id, layer_name); } else { - if (cn10k_mldev->cache_model_data) + if (cn10k_mldev->cache_model_data && model->type == ML_CNXK_MODEL_TYPE_GLOW) ret = cn10k_ml_cache_model_data(cnxk_mldev, layer); } diff --git a/drivers/ml/cnxk/cnxk_ml_model.h b/drivers/ml/cnxk/cnxk_ml_model.h index f618e5aa5fc..b5d6ab2b1e2 100644 --- a/drivers/ml/cnxk/cnxk_ml_model.h +++ b/drivers/ml/cnxk/cnxk_ml_model.h @@ -11,6 +11,10 @@ #include "cn10k_ml_model.h" +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM +#include "mvtvm_ml_model.h" +#endif + #include "cnxk_ml_io.h" struct cnxk_ml_dev; @@ -18,6 +22,45 @@ struct cnxk_ml_model; struct cnxk_ml_qp; struct cnxk_ml_req; +/* Model type */ +enum cnxk_ml_model_type { + /* Invalid model type */ + ML_CNXK_MODEL_TYPE_INVALID, + + /* Glow compiled model, for MLIP target */ + ML_CNXK_MODEL_TYPE_GLOW, + + /* TVM compiled model, for ARM64 / ARM64 + MLIP target */ + ML_CNXK_MODEL_TYPE_TVM, +}; + +/* Model subtype */ +enum cnxk_ml_model_subtype { + /* Marvell Glow model */ + ML_CNXK_MODEL_SUBTYPE_GLOW_MRVL, + + /* TVM model with single MRVL region */ + ML_CNXK_MODEL_SUBTYPE_TVM_MRVL, + + /* TVM model with LLVM regions only */ + ML_CNXK_MODEL_SUBTYPE_TVM_LLVM, + + /* TVM hybrid model, with both MRVL and LLVM regions or (> 1) MRVL regions*/ + ML_CNXK_MODEL_SUBTYPE_TVM_HYBRID, +}; + +/* Layer type */ +enum cnxk_ml_layer_type { + /* MRVL layer, for MLIP target*/ + ML_CNXK_LAYER_TYPE_UNKNOWN = 0, + + /* MRVL layer, for MLIP target*/ + ML_CNXK_LAYER_TYPE_MRVL, + + /* LLVM layer, for ARM64 target*/ + ML_CNXK_LAYER_TYPE_LLVM, +}; + /* Model state */ enum cnxk_ml_model_state { /* Unknown state */ @@ -53,6 +96,9 @@ struct cnxk_ml_layer { /* Name*/ char name[RTE_ML_STR_MAX]; + /* Type */ + enum cnxk_ml_layer_type type; + /* Model handle */ struct cnxk_ml_model *model; @@ -83,14 +129,27 @@ struct cnxk_ml_model { /* Device reference */ struct cnxk_ml_dev *cnxk_mldev; + /* Type */ + enum cnxk_ml_model_type type; + + /* Model subtype */ + enum cnxk_ml_model_subtype subtype; + /* ID */ uint16_t model_id; /* Name */ char name[RTE_ML_STR_MAX]; - /* Model specific data - glow */ - struct cn10k_ml_model_data glow; + union { + /* Model specific data - glow */ + struct cn10k_ml_model_data glow; + +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + /* Model type specific data - mvtvm */ + struct mvtvm_ml_model_data mvtvm; +#endif + }; /* Batch size */ uint32_t batch_size; diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 454fec33234..4a5b054975d 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -1286,6 +1286,8 @@ cnxk_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buf struct cnxk_ml_model *model; uint8_t *lcl_dbuffer; uint8_t *lcl_qbuffer; + uint64_t d_offset; + uint64_t q_offset; uint32_t i; int ret; @@ -1298,17 +1300,32 @@ cnxk_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buf return -EINVAL; } - info = &model->layer[0].info; + if (model->type == ML_CNXK_MODEL_TYPE_GLOW) + info = &model->layer[0].info; +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + else + info = &model->mvtvm.info; +#endif - lcl_dbuffer = dbuffer[0]->addr; - lcl_qbuffer = qbuffer[0]->addr; + d_offset = 0; + q_offset = 0; for (i = 0; i < info->nb_inputs; i++) { + if (model->type == ML_CNXK_MODEL_TYPE_TVM) { + lcl_dbuffer = dbuffer[i]->addr; + lcl_qbuffer = qbuffer[i]->addr; + } else { + lcl_dbuffer = RTE_PTR_ADD(dbuffer[0]->addr, d_offset); + lcl_qbuffer = RTE_PTR_ADD(qbuffer[0]->addr, q_offset); + } + ret = cnxk_ml_io_quantize_single(&info->input[i], lcl_dbuffer, lcl_qbuffer); if (ret < 0) return ret; - lcl_dbuffer += info->input[i].sz_d; - lcl_qbuffer += info->input[i].sz_q; + if (model->type == ML_CNXK_MODEL_TYPE_GLOW) { + d_offset += info->input[i].sz_d; + q_offset += info->input[i].sz_q; + } } return 0; @@ -1322,6 +1339,8 @@ cnxk_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_b struct cnxk_ml_model *model; uint8_t *lcl_qbuffer; uint8_t *lcl_dbuffer; + uint64_t q_offset; + uint64_t d_offset; uint32_t i; int ret; @@ -1334,17 +1353,32 @@ cnxk_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_b return -EINVAL; } - info = &model->layer[model->nb_layers - 1].info; + if (model->type == ML_CNXK_MODEL_TYPE_GLOW) + info = &model->layer[model->nb_layers - 1].info; +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + else + info = &model->mvtvm.info; +#endif - lcl_qbuffer = qbuffer[0]->addr; - lcl_dbuffer = dbuffer[0]->addr; + q_offset = 0; + d_offset = 0; for (i = 0; i < info->nb_outputs; i++) { + if (model->type == ML_CNXK_MODEL_TYPE_TVM) { + lcl_qbuffer = qbuffer[i]->addr; + lcl_dbuffer = dbuffer[i]->addr; + } else { + lcl_qbuffer = RTE_PTR_ADD(qbuffer[0]->addr, q_offset); + lcl_dbuffer = RTE_PTR_ADD(dbuffer[0]->addr, d_offset); + } + ret = cnxk_ml_io_dequantize_single(&info->output[i], lcl_qbuffer, lcl_dbuffer); if (ret < 0) return ret; - lcl_qbuffer += info->output[i].sz_q; - lcl_dbuffer += info->output[i].sz_d; + if (model->type == ML_CNXK_MODEL_TYPE_GLOW) { + q_offset += info->output[i].sz_q; + d_offset += info->output[i].sz_d; + } } return 0; diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 29dad0b0e33..9579cdf7867 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -61,6 +61,7 @@ dpdk_conf.set('RTE_MLDEV_CNXK_ENABLE_MVTVM', true) driver_sdk_headers += files( 'mvtvm_ml_ops.h', + 'mvtvm_ml_model.h', ) sources += files( diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h new file mode 100644 index 00000000000..1f6b435be02 --- /dev/null +++ b/drivers/ml/cnxk/mvtvm_ml_model.h @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#ifndef _MVTVM_ML_MODEL_H_ +#define _MVTVM_ML_MODEL_H_ + +#include + +#include + +#include "cnxk_ml_io.h" + +/* Maximum number of objects per model */ +#define ML_MVTVM_MODEL_OBJECT_MAX 3 + +/* Objects list */ +extern char mvtvm_object_list[ML_MVTVM_MODEL_OBJECT_MAX][RTE_ML_STR_MAX]; + +/* Model object structure */ +struct mvtvm_ml_model_object { + /* Name */ + char name[RTE_ML_STR_MAX]; + + /* Temporary buffer */ + uint8_t *buffer; + + /* Buffer size */ + int64_t size; +}; + +struct mvtvm_ml_model_data { + /* Model metadata */ + struct tvmdp_model_metadata metadata; + + /* Model objects */ + struct tvmdp_model_object object; + + /* TVM runtime callbacks */ + struct tvmrt_glow_callback cb; + + /* Model I/O info */ + struct cnxk_ml_io_info info; +}; + +#endif /* _MVTVM_ML_MODEL_H_ */ From patchwork Wed Aug 30 15:59:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130899 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C17E341FD1; Wed, 30 Aug 2023 18:03:02 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 35FB8402DB; Wed, 30 Aug 2023 17:59:58 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 6EC2040293 for ; Wed, 30 Aug 2023 17:59:41 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I72012554 for ; Wed, 30 Aug 2023 08:59:40 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=bctiBbMNb5Q9FkB94zXDhAVtUIH7e0j5KkgCi0UQSxA=; b=cmChm6954LEacTd9MjrK1kjHGKpzEEZbgEcxUmnoDfJZBmJgxwGc3RnrSnFcOoP6ubmc nvoGZNpbFjgBu3aVBukIaeJQxshMYwtV9yIp3GFRp96Sw1SCodfuoW+P4snHRXOVdtkJ nj9I5YqnRML9rUphN4+eH2IAZWHkk8cSoD/7I+wIDKVGzt4ilH3hnTzY+yJszXd7YULk Vt8l8wGitDO8+6jxsNKo3Vb9inDgx+xI1S4RVDmkbJOwLWvS4omxRw1nr+RBm36aP3d8 cWiilz/7UN8+vPhyg8Vv8jy6YmvR5NzwYwfgkts2+dmM/oVO62YxJd7Zbp800aSS6ZEl Jg== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:40 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:39 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:39 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 09FAA3F707E; Wed, 30 Aug 2023 08:59:39 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 21/34] ml/cnxk: add support for identify model type Date: Wed, 30 Aug 2023 08:59:11 -0700 Message-ID: <20230830155927.3566-22-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 5YipQyzVBADoo4BT9zf96Eekp4yZ2q0c X-Proofpoint-GUID: 5YipQyzVBADoo4BT9zf96Eekp4yZ2q0c X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable support to parse model buffer to identify the model type and model sub-type. Enabled basic checks for Glow model type buffer. Signed-off-by: Srikanth Yalavarthi Signed-off-by: Anup Prabhu --- drivers/ml/cnxk/cnxk_ml_model.c | 96 ++++++++++++++++++++++++++++++++ drivers/ml/cnxk/cnxk_ml_model.h | 1 + drivers/ml/cnxk/cnxk_ml_ops.c | 9 +++ drivers/ml/cnxk/meson.build | 6 ++ drivers/ml/cnxk/mvtvm_ml_model.c | 11 ++++ 5 files changed, 123 insertions(+) create mode 100644 drivers/ml/cnxk/mvtvm_ml_model.c diff --git a/drivers/ml/cnxk/cnxk_ml_model.c b/drivers/ml/cnxk/cnxk_ml_model.c index b069d4e3a5a..746d3ca5a95 100644 --- a/drivers/ml/cnxk/cnxk_ml_model.c +++ b/drivers/ml/cnxk/cnxk_ml_model.c @@ -2,11 +2,107 @@ * Copyright (c) 2023 Marvell. */ +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM +#include +#include +#endif + +#include #include +#include "cn10k_ml_model.h" + +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM +#include "mvtvm_ml_model.h" +#endif + #include "cnxk_ml_model.h" #include "cnxk_ml_utils.h" +enum cnxk_ml_model_type +cnxk_ml_model_get_type(struct rte_ml_model_params *params) +{ + struct cn10k_ml_model_metadata_header *metadata_header; + uint32_t payload_crc32c; + uint32_t header_crc32c; + +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + struct archive_entry *entry; + struct archive *a; + uint8_t i; + int ret; + + /* Assume as archive and check for read status */ + a = archive_read_new(); + archive_read_support_filter_all(a); + archive_read_support_format_all(a); + + ret = archive_read_open_memory(a, params->addr, params->size); + if (ret == ARCHIVE_OK) + goto check_tvm; + else + goto check_glow; + +check_tvm: + bool object_found[ML_MVTVM_MODEL_OBJECT_MAX] = {false, false, false}; + + /* Parse buffer for available objects */ + while (archive_read_next_header(a, &entry) == ARCHIVE_OK) { + for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) { + if (!object_found[i] && + (strcmp(archive_entry_pathname(entry), mvtvm_object_list[i]) == 0)) + object_found[i] = true; + } + archive_read_data_skip(a); + } + + /* Check if all objects are available */ + for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) { + if (!object_found[i]) { + plt_err("Object %s not found in archive!\n", mvtvm_object_list[i]); + return ML_CNXK_MODEL_TYPE_INVALID; + } + } + + return ML_CNXK_MODEL_TYPE_TVM; + +check_glow: +#endif + + /* Check model magic string */ + metadata_header = (struct cn10k_ml_model_metadata_header *)params->addr; + if (strncmp((char *)metadata_header->magic, MRVL_ML_MODEL_MAGIC_STRING, 4) != 0) { + plt_err("Invalid Glow model, magic = %s", metadata_header->magic); + return ML_CNXK_MODEL_TYPE_INVALID; + } + + /* Header CRC check */ + if (metadata_header->header_crc32c != 0) { + header_crc32c = rte_hash_crc( + params->addr, + sizeof(struct cn10k_ml_model_metadata_header) - sizeof(uint32_t), 0); + + if (header_crc32c != metadata_header->header_crc32c) { + plt_err("Invalid Glow model, Header CRC mismatch"); + return ML_CNXK_MODEL_TYPE_INVALID; + } + } + + /* Payload CRC check */ + if (metadata_header->payload_crc32c != 0) { + payload_crc32c = rte_hash_crc( + PLT_PTR_ADD(params->addr, sizeof(struct cn10k_ml_model_metadata_header)), + params->size - sizeof(struct cn10k_ml_model_metadata_header), 0); + + if (payload_crc32c != metadata_header->payload_crc32c) { + plt_err("Invalid Glow model, Payload CRC mismatch"); + return ML_CNXK_MODEL_TYPE_INVALID; + } + } + + return ML_CNXK_MODEL_TYPE_GLOW; +} + void cnxk_ml_model_dump(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, FILE *fp) { diff --git a/drivers/ml/cnxk/cnxk_ml_model.h b/drivers/ml/cnxk/cnxk_ml_model.h index b5d6ab2b1e2..577a96dc265 100644 --- a/drivers/ml/cnxk/cnxk_ml_model.h +++ b/drivers/ml/cnxk/cnxk_ml_model.h @@ -181,6 +181,7 @@ struct cnxk_ml_model { set_poll_addr_t set_poll_addr; }; +enum cnxk_ml_model_type cnxk_ml_model_get_type(struct rte_ml_model_params *params); void cnxk_ml_model_dump(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, FILE *fp); #endif /* _CNXK_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 4a5b054975d..cbb701f20bb 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -10,6 +10,7 @@ #include "cn10k_ml_ops.h" #ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM +#include "mvtvm_ml_model.h" #include "mvtvm_ml_ops.h" #endif @@ -1087,6 +1088,7 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u { struct rte_ml_dev_info dev_info; struct cnxk_ml_dev *cnxk_mldev; + enum cnxk_ml_model_type type; struct cnxk_ml_model *model; char str[RTE_MEMZONE_NAMESIZE]; @@ -1102,6 +1104,12 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u cnxk_mldev = dev->data->dev_private; + type = cnxk_ml_model_get_type(params); + if (type == ML_CNXK_MODEL_TYPE_INVALID) { + plt_err("Invalid / unsupported model type"); + return -EINVAL; + } + /* Find model ID */ found = false; for (lcl_model_id = 0; lcl_model_id < dev->data->nb_models; lcl_model_id++) { @@ -1135,6 +1143,7 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u model = mz->addr; model->cnxk_mldev = cnxk_mldev; + model->type = type; model->model_id = lcl_model_id; model->info = PLT_PTR_ADD( model, PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), dev_info.align_size)); diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 9579cdf7867..db175b0834d 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -9,6 +9,11 @@ endif enable_mvtvm = true +if not libarchive.found() + message('drivers/ml/cnxk: libarchive not found') + enable_mvtvm = false +endif + if not jansson_dep.found() message('drivers/ml/cnxk: jansson not found') enable_mvtvm = false @@ -66,6 +71,7 @@ driver_sdk_headers += files( sources += files( 'mvtvm_ml_ops.c', + 'mvtvm_ml_model.c', ) ext_deps += tvmrt_dep diff --git a/drivers/ml/cnxk/mvtvm_ml_model.c b/drivers/ml/cnxk/mvtvm_ml_model.c new file mode 100644 index 00000000000..64622675345 --- /dev/null +++ b/drivers/ml/cnxk/mvtvm_ml_model.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#include + +#include "mvtvm_ml_model.h" + +/* Objects list */ +char mvtvm_object_list[ML_MVTVM_MODEL_OBJECT_MAX][RTE_ML_STR_MAX] = {"mod.so", "mod.json", + "mod.params"}; From patchwork Wed Aug 30 15:59:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130906 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BFF2841FD1; Wed, 30 Aug 2023 18:05:08 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C4E8140EF0; Wed, 30 Aug 2023 18:00:07 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 17C4C402B9 for ; Wed, 30 Aug 2023 17:59:42 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P93Q004408 for ; Wed, 30 Aug 2023 08:59:42 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=TFYbOF3LUb8AXDch2GvuQD3IbmhnFBUtmJPrRB7DShA=; b=QFfx8ph6Cb1zLDCfaSVgUjWCyUThfhc3WGZdh+QAVbMahc32xB2D7rtZTcfU6+LBMdWS K9dL9cf4+k6wPG0PL7lwaT2x92skHVATmLi9E45v0aX2425vDVKCFpY2BSAcnfqhQ4bt JZjBlx2B68uTTBn7kDLy85CzTe6M+ScmGMYySDe6C3CskexydzTavcQse4Y+2gc+s/ww FhAZYN5nAl7Azg3Y9eqoCqKUI1dTvnR/16UsHyDINNXN4fkKgjR42I0+crxI4kRJfAxd nai1usFEVUVVYLXbn5ynZs4yGqzt0dG3zt09ww266L4qnCpZV/fEGnVWwQNhvC7KXjPP ZQ== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysstm-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:42 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:39 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:39 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 6D6F23F707D; Wed, 30 Aug 2023 08:59:39 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 22/34] ml/cnxk: add support to parse TVM model objects Date: Wed, 30 Aug 2023 08:59:12 -0700 Message-ID: <20230830155927.3566-23-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: 7ThTb-TAVKUnL7Q3nqFQRjRF_q_dzrPv X-Proofpoint-ORIG-GUID: 7ThTb-TAVKUnL7Q3nqFQRjRF_q_dzrPv X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added support to parse TVM model objects from the model archive buffer. Added support to check for all expected objects and copy TVM model objects to internal buffers. Signed-off-by: Srikanth Yalavarthi Signed-off-by: Anup Prabhu --- drivers/ml/cnxk/cnxk_ml_ops.c | 14 +++++-- drivers/ml/cnxk/mvtvm_ml_model.c | 62 +++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_model.h | 3 ++ drivers/ml/cnxk/mvtvm_ml_ops.c | 63 ++++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_ops.h | 3 ++ 5 files changed, 142 insertions(+), 3 deletions(-) diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index cbb701f20bb..a99367089b4 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -1149,9 +1149,17 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u model, PLT_ALIGN_CEIL(sizeof(struct cnxk_ml_model), dev_info.align_size)); dev->data->models[lcl_model_id] = model; - ret = cn10k_ml_model_load(cnxk_mldev, params, model); - if (ret != 0) - goto error; + if (type == ML_CNXK_MODEL_TYPE_GLOW) { + ret = cn10k_ml_model_load(cnxk_mldev, params, model); + if (ret != 0) + goto error; +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + } else { + ret = mvtvm_ml_model_load(cnxk_mldev, params, model); + if (ret != 0) + goto error; +#endif + } plt_spinlock_init(&model->lock); model->state = ML_CNXK_MODEL_STATE_LOADED; diff --git a/drivers/ml/cnxk/mvtvm_ml_model.c b/drivers/ml/cnxk/mvtvm_ml_model.c index 64622675345..425a682209f 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.c +++ b/drivers/ml/cnxk/mvtvm_ml_model.c @@ -2,10 +2,72 @@ * Copyright (c) 2023 Marvell. */ +#include +#include + #include +#include + #include "mvtvm_ml_model.h" /* Objects list */ char mvtvm_object_list[ML_MVTVM_MODEL_OBJECT_MAX][RTE_ML_STR_MAX] = {"mod.so", "mod.json", "mod.params"}; + +int +mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_model_object *object) +{ + bool object_found[ML_MVTVM_MODEL_OBJECT_MAX] = {false, false, false}; + struct archive_entry *entry; + struct archive *a; + uint8_t i; + int ret; + + /* Open archive */ + a = archive_read_new(); + archive_read_support_filter_all(a); + archive_read_support_format_all(a); + + ret = archive_read_open_memory(a, params->addr, params->size); + if (ret != ARCHIVE_OK) + return archive_errno(a); + + /* Read archive */ + while (archive_read_next_header(a, &entry) == ARCHIVE_OK) { + for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) { + if (!object_found[i] && + (strcmp(archive_entry_pathname(entry), mvtvm_object_list[i]) == 0)) { + memcpy(object[i].name, mvtvm_object_list[i], RTE_ML_STR_MAX); + object[i].size = archive_entry_size(entry); + object[i].buffer = rte_malloc(NULL, object[i].size, 0); + + if (archive_read_data(a, object[i].buffer, object[i].size) != + object[i].size) { + plt_err("Failed to read object from model archive: %s", + object[i].name); + goto error; + } + object_found[i] = true; + } + } + archive_read_data_skip(a); + } + + /* Check if all objects are parsed */ + for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) { + if (!object_found[i]) { + plt_err("Object %s not found in archive!\n", mvtvm_object_list[i]); + goto error; + } + } + return 0; + +error: + for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) { + if (object[i].buffer != NULL) + rte_free(object[i].buffer); + } + + return -EINVAL; +} diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h index 1f6b435be02..73a45a91d66 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.h +++ b/drivers/ml/cnxk/mvtvm_ml_model.h @@ -43,4 +43,7 @@ struct mvtvm_ml_model_data { struct cnxk_ml_io_info info; }; +int mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, + struct mvtvm_ml_model_object *object); + #endif /* _MVTVM_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index 0e1fc527daa..1bdd4515771 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -9,9 +9,14 @@ #include #include +#include "mvtvm_ml_model.h" #include "mvtvm_ml_ops.h" #include "cnxk_ml_dev.h" +#include "cnxk_ml_model.h" + +/* ML model macros */ +#define MVTVM_ML_MODEL_MEMZONE_NAME "ml_mvtvm_model_mz" int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf) @@ -42,3 +47,61 @@ mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev) return ret; } + +int +mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, + struct cnxk_ml_model *model) +{ + struct mvtvm_ml_model_object object[ML_MVTVM_MODEL_OBJECT_MAX]; + char str[RTE_MEMZONE_NAMESIZE]; + const struct plt_memzone *mz; + size_t model_object_size = 0; + uint64_t mz_size = 0; + int ret; + + RTE_SET_USED(cnxk_mldev); + + ret = mvtvm_ml_model_blob_parse(params, object); + if (ret != 0) + return ret; + + model_object_size = RTE_ALIGN_CEIL(object[0].size, RTE_CACHE_LINE_MIN_SIZE) + + RTE_ALIGN_CEIL(object[1].size, RTE_CACHE_LINE_MIN_SIZE) + + RTE_ALIGN_CEIL(object[2].size, RTE_CACHE_LINE_MIN_SIZE); + mz_size += model_object_size; + + /* Allocate memzone for model object */ + snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", MVTVM_ML_MODEL_MEMZONE_NAME, model->model_id); + mz = plt_memzone_reserve_aligned(str, mz_size, 0, ML_CN10K_ALIGN_SIZE); + if (!mz) { + plt_err("plt_memzone_reserve failed : %s", str); + return -ENOMEM; + } + + /* Copy mod.so */ + model->mvtvm.object.so.addr = mz->addr; + model->mvtvm.object.so.size = object[0].size; + rte_memcpy(model->mvtvm.object.so.name, object[0].name, TVMDP_NAME_STRLEN); + rte_memcpy(model->mvtvm.object.so.addr, object[0].buffer, object[0].size); + rte_free(object[0].buffer); + + /* Copy mod.json */ + model->mvtvm.object.json.addr = + RTE_PTR_ADD(model->mvtvm.object.so.addr, + RTE_ALIGN_CEIL(model->mvtvm.object.so.size, RTE_CACHE_LINE_MIN_SIZE)); + model->mvtvm.object.json.size = object[1].size; + rte_memcpy(model->mvtvm.object.json.name, object[1].name, TVMDP_NAME_STRLEN); + rte_memcpy(model->mvtvm.object.json.addr, object[1].buffer, object[1].size); + rte_free(object[1].buffer); + + /* Copy mod.params */ + model->mvtvm.object.params.addr = + RTE_PTR_ADD(model->mvtvm.object.json.addr, + RTE_ALIGN_CEIL(model->mvtvm.object.json.size, RTE_CACHE_LINE_MIN_SIZE)); + model->mvtvm.object.params.size = object[2].size; + rte_memcpy(model->mvtvm.object.params.name, object[2].name, TVMDP_NAME_STRLEN); + rte_memcpy(model->mvtvm.object.params.addr, object[2].buffer, object[2].size); + rte_free(object[2].buffer); + + return 0; +} diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.h b/drivers/ml/cnxk/mvtvm_ml_ops.h index 988f3a1fd5e..ca8f57992da 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.h +++ b/drivers/ml/cnxk/mvtvm_ml_ops.h @@ -8,8 +8,11 @@ #include struct cnxk_ml_dev; +struct cnxk_ml_model; int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf); int mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev); +int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, + struct cnxk_ml_model *model); #endif /* _MVTVM_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130901 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0CF7141FD1; Wed, 30 Aug 2023 18:04:01 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EC94E40E72; Wed, 30 Aug 2023 18:00:01 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id E5F66402B4 for ; Wed, 30 Aug 2023 17:59:41 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I73012554 for ; Wed, 30 Aug 2023 08:59:41 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=uudmush/oxpbGoH/zjziUY1/QqqLDF9rBkKwNIqZ62Y=; b=PRLeblUIhrNET0K01cQGU6/BXBBjttPlkUnjC7hL/+Q+kn4zgeI9BT7SMJoV4pyTHUvy Nyi2L4S/yOVVdsnHmbf0xHM+zuauVi9mG7+ywmKfnKtrx4qNJnBe+SP7h1tov+MpM80e xGVbaSLkCVerWBBo3b6GgQTHmDKzV+jocZg0EsjYx70tcMA0I4/BVltjIvDIPmpmM+GV nyFNfZ627iYDZ8rfrKvwBHNTR9Dozp3MNwgbyDfSp12ZanXZtHYBTv/DforEFKqI0wm2 SSgBOzlrg6FOiFf2cr5KPIdWi21teCQcxrctntturhZoX3+yWlIqZlNpGB5jNDvqI7US qg== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-12 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:41 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:39 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:39 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id B9C483F707F; Wed, 30 Aug 2023 08:59:39 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 23/34] ml/cnxk: fetch layer info and load TVM model Date: Wed, 30 Aug 2023 08:59:13 -0700 Message-ID: <20230830155927.3566-24-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: PTWUBdskfr4dXRHYnogCMAu-mWVCPC1o X-Proofpoint-GUID: PTWUBdskfr4dXRHYnogCMAu-mWVCPC1o X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added support to fetch TVM model layer information and update internal structures based on the layer information Set callback functions for layer load and unload and enable model loading using TVMDP library. Added support to fetch full metadata after model load. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ops.c | 22 ++++++++- drivers/ml/cnxk/mvtvm_ml_model.h | 2 + drivers/ml/cnxk/mvtvm_ml_ops.c | 83 ++++++++++++++++++++++++++++++++ 3 files changed, 106 insertions(+), 1 deletion(-) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index db18f320527..79217165cd5 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -508,8 +508,10 @@ cn10k_ml_layer_load(void *device, uint16_t model_id, const char *layer_name, uin int qp_id; int ret; - PLT_SET_USED(size); +#ifndef RTE_MLDEV_CNXK_ENABLE_MVTVM PLT_SET_USED(layer_name); +#endif + PLT_SET_USED(size); cnxk_mldev = (struct cnxk_ml_dev *)device; if (cnxk_mldev == NULL) { @@ -523,6 +525,24 @@ cn10k_ml_layer_load(void *device, uint16_t model_id, const char *layer_name, uin return -EINVAL; } +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + if (model->type == ML_CNXK_MODEL_TYPE_TVM) { + for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) { + if (strcmp(model->layer[layer_id].name, layer_name) == 0) + break; + } + + if (layer_id == model->mvtvm.metadata.model.nb_layers) { + plt_err("Invalid layer name: %s", layer_name); + return -EINVAL; + } + + if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) { + plt_err("Invalid layer name / type: %s", layer_name); + return -EINVAL; + } + } +#endif layer = &model->layer[layer_id]; ret = cn10k_ml_model_metadata_check(buffer, size); diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h index 73a45a91d66..6c38217c158 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.h +++ b/drivers/ml/cnxk/mvtvm_ml_model.h @@ -11,6 +11,8 @@ #include "cnxk_ml_io.h" +struct cnxk_ml_model; + /* Maximum number of objects per model */ #define ML_MVTVM_MODEL_OBJECT_MAX 3 diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index 1bdd4515771..5c30bbf6b89 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -9,6 +9,8 @@ #include #include +#include "cn10k_ml_ops.h" + #include "mvtvm_ml_model.h" #include "mvtvm_ml_ops.h" @@ -53,9 +55,13 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * struct cnxk_ml_model *model) { struct mvtvm_ml_model_object object[ML_MVTVM_MODEL_OBJECT_MAX]; + struct tvmrt_glow_callback *callback; char str[RTE_MEMZONE_NAMESIZE]; const struct plt_memzone *mz; size_t model_object_size = 0; + uint16_t nb_mrvl_layers; + uint16_t nb_llvm_layers; + uint8_t layer_id = 0; uint64_t mz_size = 0; int ret; @@ -103,5 +109,82 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * rte_memcpy(model->mvtvm.object.params.addr, object[2].buffer, object[2].size); rte_free(object[2].buffer); + /* Get metadata - stage 1 */ + ret = tvmdp_model_metadata_get_stage1(model->mvtvm.object.json.addr, + model->mvtvm.object.json.size, + &model->mvtvm.metadata); + if (ret != 0) { + plt_err("TVMDP: Failed to parse metadata - stage 1, model_id = %u, error = %d", + model->model_id, ret); + goto error; + } + + /* Set model fields */ + plt_strlcpy(model->name, model->mvtvm.metadata.model.name, TVMDP_NAME_STRLEN); + model->batch_size = 1; + model->nb_layers = model->mvtvm.metadata.model.nb_layers; + + /* Update layer info */ + nb_mrvl_layers = 0; + nb_llvm_layers = 0; + for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) { + strncpy(model->layer[layer_id].name, + model->mvtvm.metadata.model.layer[layer_id].name, TVMDP_NAME_STRLEN); + if (strcmp(model->mvtvm.metadata.model.layer[layer_id].type, "mrvl") == 0 || + strcmp(model->mvtvm.metadata.model.layer[layer_id].type, "MRVL") == 0) { + model->layer[layer_id].type = ML_CNXK_LAYER_TYPE_MRVL; + nb_mrvl_layers++; + } else if (strcmp(model->mvtvm.metadata.model.layer[layer_id].type, "llvm") == 0 || + strcmp(model->mvtvm.metadata.model.layer[layer_id].type, "LLVM") == 0) { + model->layer[layer_id].type = ML_CNXK_LAYER_TYPE_LLVM; + nb_llvm_layers++; + } + } + + if ((nb_llvm_layers == 0) && (nb_mrvl_layers == 0)) { + plt_err("Invalid model, nb_llvm_layers = %u, nb_mrvl_layers = %u", nb_llvm_layers, + nb_mrvl_layers); + goto error; + } + + /* Set model subtype */ + if ((nb_llvm_layers == 0) && (nb_mrvl_layers == 1)) + model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_MRVL; + else if ((nb_llvm_layers > 0) && (nb_mrvl_layers == 0)) + model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_LLVM; + else + model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_HYBRID; + + /* Set callback function array */ + if (model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_LLVM) { + callback = &model->mvtvm.cb; + callback->tvmrt_glow_layer_load = cn10k_ml_layer_load; + callback->tvmrt_glow_layer_unload = cn10k_ml_layer_unload; + } else { + callback = NULL; + } + + /* Initialize model in TVMDP */ + ret = tvmdp_model_load(cnxk_mldev, model->model_id, (void *)(&model->mvtvm.object), + callback); + if (ret != 0) { + plt_err("TVMDP: Model load failed, model_id = %u, error = %d", model->model_id, + ret); + goto error; + } + + /* Get model metadata - stage 2 */ + ret = tvmdp_model_metadata_get_stage2(model->model_id, &model->mvtvm.metadata); + if (ret != 0) { + plt_err("TVMDP: Failed to get metadata, model_id = %u, error = %d\n", + model->model_id, ret); + goto error; + } + return 0; + +error: + rte_memzone_free(mz); + + return ret; } From patchwork Wed Aug 30 15:59:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130903 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D37BC41FD1; Wed, 30 Aug 2023 18:04:34 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8DB1340ED2; Wed, 30 Aug 2023 18:00:04 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 6B760402B4 for ; Wed, 30 Aug 2023 17:59:42 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I74012554 for ; Wed, 30 Aug 2023 08:59:41 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=b9Goicxb871KLXyUYnm4kFeVjkCtjPxJ/0YG1HY95rQ=; b=hPQZa+plV6gM+MvU2qVHSwqRksTf9QNRcXYK9O/zYS2ijWajh41Zq+A78VyEWCY7wI5v SVloj1dGhFYrQj7qY9DQhkGC9Y0RFZkjU4TyQBxVWMkhODvBct12Zop8YrHVqMx9Ctoi wjpDdUVjFnqBlnKCFQjZw73Q5GutiCvCIOCjWhKkAEkTdjXVjpnmlYRJeH48K3hVSY0Q 0i0zfOohk40sZWkBW0zjRDSEGFKgmNbRxcYf52PaNWtF0/gN4xgNZxvSqUlJaLSunMxz LW48bRg25ejD3hC6xpp0uhwsQRZ5XQkwjI8It+HfhSxiqEq2joSG5aFZlinrNuq9c1SE sw== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-13 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:41 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:40 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:40 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 1396F3F7080; Wed, 30 Aug 2023 08:59:40 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 24/34] ml/cnxk: update internal info for TVM model Date: Wed, 30 Aug 2023 08:59:14 -0700 Message-ID: <20230830155927.3566-25-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: WPeEhVmDhbKGUFbFLDs1OdeIYEgx4qRd X-Proofpoint-GUID: WPeEhVmDhbKGUFbFLDs1OdeIYEgx4qRd X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enabled updating internal IO info structures for TVM model. Compute static fields related to the model I/O. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/mvtvm_ml_model.c | 105 +++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_model.h | 1 + drivers/ml/cnxk/mvtvm_ml_ops.c | 3 + 3 files changed, 109 insertions(+) diff --git a/drivers/ml/cnxk/mvtvm_ml_model.c b/drivers/ml/cnxk/mvtvm_ml_model.c index 425a682209f..86f465a645f 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.c +++ b/drivers/ml/cnxk/mvtvm_ml_model.c @@ -7,10 +7,14 @@ #include +#include + #include #include "mvtvm_ml_model.h" +#include "cnxk_ml_model.h" + /* Objects list */ char mvtvm_object_list[ML_MVTVM_MODEL_OBJECT_MAX][RTE_ML_STR_MAX] = {"mod.so", "mod.json", "mod.params"}; @@ -71,3 +75,104 @@ mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_mo return -EINVAL; } + +static enum rte_ml_io_type +mvtvm_ml_io_type_map(uint8_t type) +{ + switch (type) { + case kDLInt: + return RTE_ML_IO_TYPE_INT32; + case kDLUInt: + return RTE_ML_IO_TYPE_UINT32; + case kDLFloat: + return RTE_ML_IO_TYPE_FP32; + case kDLBfloat: + return RTE_ML_IO_TYPE_BFLOAT16; + } + + return RTE_ML_IO_TYPE_UNKNOWN; +} + +void +mvtvm_ml_model_io_info_update(struct cnxk_ml_model *model) +{ + struct tvmdp_model_metadata *metadata; + int32_t i; + int32_t j; + + if (model->subtype == ML_CNXK_MODEL_SUBTYPE_TVM_MRVL) + goto tvm_mrvl_model; + + metadata = &model->mvtvm.metadata; + + /* Inputs, set for layer_id = 0 */ + model->mvtvm.info.nb_inputs = metadata->model.num_input; + model->mvtvm.info.total_input_sz_d = 0; + model->mvtvm.info.total_input_sz_q = 0; + for (i = 0; i < metadata->model.num_input; i++) { + strncpy(model->mvtvm.info.input[i].name, metadata->input[i].name, + TVMDP_NAME_STRLEN); + model->mvtvm.info.input[i].dtype = + mvtvm_ml_io_type_map(metadata->input[i].datatype.code); + model->mvtvm.info.input[i].qtype = + mvtvm_ml_io_type_map(metadata->input[i].model_datatype.code); + model->mvtvm.info.input[i].nb_dims = metadata->input[i].ndim; + + model->mvtvm.info.input[i].nb_elements = 1; + for (j = 0; j < metadata->input[i].ndim; j++) { + model->mvtvm.info.input[i].shape[j] = metadata->input[i].shape[j]; + model->mvtvm.info.input[i].nb_elements *= metadata->input[i].shape[j]; + } + + model->mvtvm.info.input[i].sz_d = + model->mvtvm.info.input[i].nb_elements * + rte_ml_io_type_size_get(model->mvtvm.info.input[i].dtype); + model->mvtvm.info.input[i].sz_q = + model->mvtvm.info.input[i].nb_elements * + rte_ml_io_type_size_get(model->mvtvm.info.input[i].qtype); + + model->mvtvm.info.total_input_sz_d += model->mvtvm.info.input[i].sz_d; + model->mvtvm.info.total_input_sz_q += model->mvtvm.info.input[i].sz_q; + + plt_ml_dbg("model_id = %u, input[%u] - sz_d = %u sz_q = %u", model->model_id, i, + model->mvtvm.info.input[i].sz_d, model->mvtvm.info.input[i].sz_q); + } + + /* Outputs, set for nb_layers - 1 */ + model->mvtvm.info.nb_outputs = metadata->model.num_output; + model->mvtvm.info.total_output_sz_d = 0; + model->mvtvm.info.total_output_sz_q = 0; + for (i = 0; i < metadata->model.num_output; i++) { + strncpy(model->mvtvm.info.output[i].name, metadata->output[i].name, + TVMDP_NAME_STRLEN); + model->mvtvm.info.output[i].dtype = + mvtvm_ml_io_type_map(metadata->output[i].datatype.code); + model->mvtvm.info.output[i].qtype = + mvtvm_ml_io_type_map(metadata->output[i].model_datatype.code); + model->mvtvm.info.output[i].nb_dims = metadata->output[i].ndim; + + model->mvtvm.info.output[i].nb_elements = 1; + for (j = 0; j < metadata->output[i].ndim; j++) { + model->mvtvm.info.output[i].shape[j] = metadata->output[i].shape[j]; + model->mvtvm.info.output[i].nb_elements *= metadata->output[i].shape[j]; + } + + model->mvtvm.info.output[i].sz_d = + model->mvtvm.info.output[i].nb_elements * + rte_ml_io_type_size_get(model->mvtvm.info.output[i].dtype); + model->mvtvm.info.output[i].sz_q = + model->mvtvm.info.output[i].nb_elements * + rte_ml_io_type_size_get(model->mvtvm.info.output[i].qtype); + + model->mvtvm.info.total_output_sz_d += model->mvtvm.info.output[i].sz_d; + model->mvtvm.info.total_output_sz_q += model->mvtvm.info.output[i].sz_q; + + plt_ml_dbg("model_id = %u, output[%u] - sz_d = %u sz_q = %u", model->model_id, i, + model->mvtvm.info.output[i].sz_d, model->mvtvm.info.output[i].sz_q); + } + + return; + +tvm_mrvl_model: + cn10k_ml_layer_io_info_update(&model->mvtvm.info, &model->layer[0].glow.metadata); +} diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h index 6c38217c158..2b25a7b568e 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.h +++ b/drivers/ml/cnxk/mvtvm_ml_model.h @@ -47,5 +47,6 @@ struct mvtvm_ml_model_data { int mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_model_object *object); +void mvtvm_ml_model_io_info_update(struct cnxk_ml_model *model); #endif /* _MVTVM_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index 5c30bbf6b89..a783e16e6eb 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -181,6 +181,9 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * goto error; } + /* Update model I/O data */ + mvtvm_ml_model_io_info_update(model); + return 0; error: From patchwork Wed Aug 30 15:59:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130905 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C979341FD1; Wed, 30 Aug 2023 18:04:56 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C39DA40EE2; Wed, 30 Aug 2023 18:00:06 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 02927402B8 for ; Wed, 30 Aug 2023 17:59:42 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I75012554 for ; Wed, 30 Aug 2023 08:59:42 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=UhddiT7Q5mYTGnYakJ6Jdzxyo/K8T13kjTYXu86nXPo=; b=eKFagoCCnMSWICS2lvdq6vvN3opxg5TI1Q29dIPGuvKhfTdxb0sHr8aXs0mBZ1ijP6uF 4pGC5sOEG91qDOl1s9i1UxdplpJl7u0PraU+yti/bAuMKB6BiiyV+DD2KtMR8l5J3ifd s5Xs88+QKVvCcN3kwmq9OYmJSYHWn0WvfNd3/qnsHJQ6ps6+Z1nIqfcVInOc+r3ItEXE UfifFyPtl4/FZBJrnKetFVtSHKiAV/bL6VlCUMzK4R+VxMFLqmQyBE+XMYSXXv4i3BWe ShKL22YUqf68jEUyRzz81mjbXKi08ogZwPfJ9sNVKSGXN8lnoWcyz1BBDA94MqTWD6Xi SQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-14 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:42 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:40 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:40 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 646483F707E; Wed, 30 Aug 2023 08:59:40 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 25/34] ml/cnxk: enable model unload in tvmdp library Date: Wed, 30 Aug 2023 08:59:15 -0700 Message-ID: <20230830155927.3566-26-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: PYpFucofcc-wBCM1-Uf9KmXnXtt9uiUD X-Proofpoint-GUID: PYpFucofcc-wBCM1-Uf9KmXnXtt9uiUD X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable unloading model using external tvmdp library. Updated layer unload callback to support multiple layers. Signed-off-by: Srikanth Yalavarthi Signed-off-by: Anup Prabhu --- drivers/ml/cnxk/cn10k_ml_ops.c | 20 ++++++++++++++++++++ drivers/ml/cnxk/cnxk_ml_ops.c | 9 +++++++-- drivers/ml/cnxk/mvtvm_ml_ops.c | 28 ++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_ops.h | 1 + 4 files changed, 56 insertions(+), 2 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 79217165cd5..85d0a9e18bb 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -725,7 +725,9 @@ cn10k_ml_layer_unload(void *device, uint16_t model_id, const char *layer_name) uint16_t layer_id = 0; int ret; +#ifndef RTE_MLDEV_CNXK_ENABLE_MVTVM PLT_SET_USED(layer_name); +#endif cnxk_mldev = (struct cnxk_ml_dev *)device; if (cnxk_mldev == NULL) { @@ -739,6 +741,24 @@ cn10k_ml_layer_unload(void *device, uint16_t model_id, const char *layer_name) return -EINVAL; } +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + if (model->type == ML_CNXK_MODEL_TYPE_TVM) { + for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) { + if (strcmp(model->layer[layer_id].name, layer_name) == 0) + break; + } + + if (layer_id == model->mvtvm.metadata.model.nb_layers) { + plt_err("Invalid layer name: %s", layer_name); + return -EINVAL; + } + + if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) { + plt_err("Invalid layer name / type: %s", layer_name); + return -EINVAL; + } + } +#endif layer = &model->layer[layer_id]; snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u_%u", CN10K_ML_LAYER_MEMZONE_NAME, diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index a99367089b4..d8eadcb8121 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -1182,7 +1182,7 @@ cnxk_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) struct cnxk_ml_model *model; char str[RTE_MEMZONE_NAMESIZE]; - int ret; + int ret = 0; if (dev == NULL) return -EINVAL; @@ -1200,7 +1200,12 @@ cnxk_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id) return -EBUSY; } - ret = cn10k_ml_model_unload(cnxk_mldev, model); + if (model->type == ML_CNXK_MODEL_TYPE_GLOW) + ret = cn10k_ml_model_unload(cnxk_mldev, model); +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + else + ret = mvtvm_ml_model_unload(cnxk_mldev, model); +#endif if (ret != 0) return ret; diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index a783e16e6eb..1edbfb0dcc3 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -191,3 +191,31 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * return ret; } + +int +mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model) +{ + char str[RTE_MEMZONE_NAMESIZE]; + const struct plt_memzone *mz; + int ret; + + RTE_SET_USED(cnxk_mldev); + + /* Initialize model in TVMDP */ + ret = tvmdp_model_unload(model->model_id); + if (ret != 0) { + plt_err("TVMDP: Model unload failed, model_id = %u, error = %d", model->model_id, + ret); + return ret; + } + + snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", MVTVM_ML_MODEL_MEMZONE_NAME, model->model_id); + mz = rte_memzone_lookup(str); + if (mz == NULL) { + plt_err("Memzone lookup failed for TVM model: model_id = %u, mz = %s", + model->model_id, str); + return -EINVAL; + } + + return plt_memzone_free(mz); +} diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.h b/drivers/ml/cnxk/mvtvm_ml_ops.h index ca8f57992da..8b4db20fe94 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.h +++ b/drivers/ml/cnxk/mvtvm_ml_ops.h @@ -14,5 +14,6 @@ int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_d int mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev); int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, struct cnxk_ml_model *model); +int mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); #endif /* _MVTVM_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130907 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ECF9E41FD1; Wed, 30 Aug 2023 18:05:20 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B00E4410F6; Wed, 30 Aug 2023 18:00:08 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 89D0D4028C for ; Wed, 30 Aug 2023 17:59:43 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I76012554 for ; Wed, 30 Aug 2023 08:59:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=HmgUp1LCw6DlYZ8CMXrID6x2oxonKRlkk7VDvlJXyiY=; b=kzsUw4BXfrwsuEh8VuiYCsTIaNmqcquRmwQCKZvgQAzyfpLHLMuzpgHZTU4W3TKQo32K fnr1PVA28nCAVQn2vv6knkAOb8xmW5cPnS8wbJNMLFK3lMFpwkFGOnxVIo5VYeQ1RT2U Ahu6AiWd9AFsOwat+krli8PG65p41hXeFDIK7xatIcgVoYJizaTVnULNKMgYn92npfLM mgZzq7qDMWHGN6s9Ip8oLD4a/9vTyCY4T0KprDdGN9RqCB0bSNCgS47iJofvJwq378tj rIucoCKRCKaU1Y+nwDMR9lGB1UDptVA0D55U0aHhPuwP+9Lhft5yLkjpDY6mLGIaHEbe 6Q== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-15 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:42 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:40 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:40 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id BA3F33F707D; Wed, 30 Aug 2023 08:59:40 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 26/34] ml/cnxk: support start and stop for TVM models Date: Wed, 30 Aug 2023 08:59:16 -0700 Message-ID: <20230830155927.3566-27-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 6JvfiyibXZ01q9VWyw1CNx0yDPV_WjPO X-Proofpoint-GUID: 6JvfiyibXZ01q9VWyw1CNx0yDPV_WjPO X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added support to start and stop TVM models. TVM model start would invoke layer start for all Glow layers part of the model. TVM model stop would invoke layer stop for all Glow layers part of the model. Signed-off-by: Srikanth Yalavarthi Signed-off-by: Anup Prabhu --- drivers/ml/cnxk/cn10k_ml_ops.c | 42 +++++++++++++++++++++++++++ drivers/ml/cnxk/cnxk_ml_ops.c | 18 ++++++++++-- drivers/ml/cnxk/mvtvm_ml_ops.c | 52 ++++++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_ops.h | 2 ++ 4 files changed, 112 insertions(+), 2 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 85d0a9e18bb..f70383b1281 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -798,7 +798,9 @@ cn10k_ml_layer_start(void *device, uint16_t model_id, const char *layer_name) bool locked; int ret = 0; +#ifndef RTE_MLDEV_CNXK_ENABLE_MVTVM PLT_SET_USED(layer_name); +#endif cnxk_mldev = (struct cnxk_ml_dev *)device; if (cnxk_mldev == NULL) { @@ -812,6 +814,25 @@ cn10k_ml_layer_start(void *device, uint16_t model_id, const char *layer_name) return -EINVAL; } +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + if (model->type == ML_CNXK_MODEL_TYPE_TVM) { + for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) { + if (strcmp(model->layer[layer_id].name, layer_name) == 0) + break; + } + + if (layer_id == model->mvtvm.metadata.model.nb_layers) { + plt_err("Invalid layer name: %s", layer_name); + return -EINVAL; + } + + if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) { + plt_err("Invalid layer name / type: %s", layer_name); + return -EINVAL; + } + } +#endif + layer = &model->layer[layer_id]; cn10k_mldev = &cnxk_mldev->cn10k_mldev; ocm = &cn10k_mldev->ocm; @@ -981,7 +1002,9 @@ cn10k_ml_layer_stop(void *device, uint16_t model_id, const char *layer_name) bool locked; int ret = 0; +#ifndef RTE_MLDEV_CNXK_ENABLE_MVTVM PLT_SET_USED(layer_name); +#endif cnxk_mldev = (struct cnxk_ml_dev *)device; if (cnxk_mldev == NULL) { @@ -995,6 +1018,25 @@ cn10k_ml_layer_stop(void *device, uint16_t model_id, const char *layer_name) return -EINVAL; } +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + if (model->type == ML_CNXK_MODEL_TYPE_TVM) { + for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) { + if (strcmp(model->layer[layer_id].name, layer_name) == 0) + break; + } + + if (layer_id == model->mvtvm.metadata.model.nb_layers) { + plt_err("Invalid layer name: %s", layer_name); + return -EINVAL; + } + + if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) { + plt_err("Invalid layer name / type: %s", layer_name); + return -EINVAL; + } + } +#endif + layer = &model->layer[layer_id]; cn10k_mldev = &cnxk_mldev->cn10k_mldev; ocm = &cn10k_mldev->ocm; diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index d8eadcb8121..45c3d61a9b6 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -1233,7 +1233,14 @@ cnxk_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id) return -EINVAL; } - return cn10k_ml_model_start(cnxk_mldev, model); + if (model->type == ML_CNXK_MODEL_TYPE_GLOW) + return cn10k_ml_model_start(cnxk_mldev, model); +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + else + return mvtvm_ml_model_start(cnxk_mldev, model); +#endif + + return 0; } int @@ -1253,7 +1260,14 @@ cnxk_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id) return -EINVAL; } - return cn10k_ml_model_stop(cnxk_mldev, model); + if (model->type == ML_CNXK_MODEL_TYPE_GLOW) + return cn10k_ml_model_stop(cnxk_mldev, model); +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + else + return mvtvm_ml_model_stop(cnxk_mldev, model); +#endif + + return 0; } static int diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index 1edbfb0dcc3..8d25b5d4b87 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -219,3 +219,55 @@ mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mode return plt_memzone_free(mz); } + +int +mvtvm_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model) +{ + struct cnxk_ml_layer *layer; + + uint16_t layer_id = 0; + int ret = 0; + +next_layer: + layer = &model->layer[layer_id]; + if (layer->type == ML_CNXK_LAYER_TYPE_MRVL) { + ret = cn10k_ml_layer_start(cnxk_mldev, model->model_id, layer->name); + if (ret != 0) { + plt_err("Layer start failed, model_id = %u, layer_name = %s, error = %d", + model->model_id, layer->name, ret); + return ret; + } + } + layer_id++; + + if (layer_id < model->nb_layers) + goto next_layer; + + return 0; +} + +int +mvtvm_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model) +{ + struct cnxk_ml_layer *layer; + + uint16_t layer_id = 0; + int ret = 0; + +next_layer: + layer = &model->layer[layer_id]; + if (layer->type == ML_CNXK_LAYER_TYPE_MRVL) { + ret = cn10k_ml_layer_stop(cnxk_mldev, model->model_id, layer->name); + if (ret != 0) { + plt_err("Layer stop failed, model_id = %u, layer_name = %s, error = %d", + model->model_id, layer->name, ret); + return ret; + } + } + layer_id++; + + if (layer_id < model->nb_layers) + goto next_layer; + + return 0; +} diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.h b/drivers/ml/cnxk/mvtvm_ml_ops.h index 8b4db20fe94..f6ede6229f4 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.h +++ b/drivers/ml/cnxk/mvtvm_ml_ops.h @@ -15,5 +15,7 @@ int mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev); int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, struct cnxk_ml_model *model); int mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); +int mvtvm_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); +int mvtvm_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); #endif /* _MVTVM_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130908 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4D8D141FD1; Wed, 30 Aug 2023 18:05:33 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A6C004113D; Wed, 30 Aug 2023 18:00:09 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 14372402B9 for ; Wed, 30 Aug 2023 17:59:43 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I77012554 for ; Wed, 30 Aug 2023 08:59:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=DPk8AVg7E6DjxR6C8TeUOaySHLtI4IQJrTT71ZjZqeg=; b=kAy1bFvcHmdKpGHrOjVklYncCaUVdJk1Almf02ENsXIMYdq3PdB5LlMTPrWoqdukM6M5 EVesLlIHo9SX40rippqVXCeM2qh61lW+GM5Z36At3CthS/1anVZICeUIMoR90ELDxhZZ nEckfG4LLCXF9DLqtiVtM5Sdi+IAs3BEyODfznXRaQb9tACrlMfnyIsoEnGzKZbRe6B+ cypa7yd5kqCHUqac0Vg1uD0ACk3Le8U0AJqu2uM4VMGdGok3B2VYelDRV43qWqcriTZ6 RJe+gFxvr30ni9TxvAU8pKu4b8CsLx9OXF8LQA3Fw6W7/6/AtYMqAGwbAlnX3w0DEkH8 Cw== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-16 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:43 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:41 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:41 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 147FF3F707F; Wed, 30 Aug 2023 08:59:41 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 27/34] ml/cnxk: update internal TVM model info structure Date: Wed, 30 Aug 2023 08:59:17 -0700 Message-ID: <20230830155927.3566-28-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 61dX_Iv4UDiek1ywq3AMWTIvZeEpe9V- X-Proofpoint-GUID: 61dX_Iv4UDiek1ywq3AMWTIvZeEpe9V- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Prince Takkar Added support to update internal model info structure for TVM models. Signed-off-by: Prince Takkar Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/mvtvm_ml_model.c | 65 ++++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_model.h | 2 + drivers/ml/cnxk/mvtvm_ml_ops.c | 3 ++ 3 files changed, 70 insertions(+) diff --git a/drivers/ml/cnxk/mvtvm_ml_model.c b/drivers/ml/cnxk/mvtvm_ml_model.c index 86f465a645f..6d72a5255e2 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.c +++ b/drivers/ml/cnxk/mvtvm_ml_model.c @@ -13,6 +13,7 @@ #include "mvtvm_ml_model.h" +#include "cnxk_ml_dev.h" #include "cnxk_ml_model.h" /* Objects list */ @@ -176,3 +177,67 @@ mvtvm_ml_model_io_info_update(struct cnxk_ml_model *model) tvm_mrvl_model: cn10k_ml_layer_io_info_update(&model->mvtvm.info, &model->layer[0].glow.metadata); } + +void +mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model) +{ + struct tvmdp_model_metadata *metadata; + struct rte_ml_model_info *info; + struct rte_ml_io_info *output; + struct rte_ml_io_info *input; + uint8_t i; + + info = PLT_PTR_CAST(model->info); + input = PLT_PTR_ADD(info, sizeof(struct rte_ml_model_info)); + output = PLT_PTR_ADD(input, ML_CNXK_MODEL_MAX_INPUT_OUTPUT * sizeof(struct rte_ml_io_info)); + + /* Reset model info */ + memset(info, 0, sizeof(struct rte_ml_model_info)); + + if (model->subtype == ML_CNXK_MODEL_SUBTYPE_TVM_MRVL) + goto tvm_mrvl_model; + + metadata = &model->mvtvm.metadata; + rte_memcpy(info->name, metadata->model.name, MRVL_ML_MODEL_NAME_LEN); + snprintf(info->version, RTE_ML_STR_MAX, "%u.%u.%u.%u", metadata->model.version[0], + metadata->model.version[1], metadata->model.version[2], + metadata->model.version[3]); + info->model_id = model->model_id; + info->device_id = cnxk_mldev->mldev->data->dev_id; + info->io_layout = RTE_ML_IO_LAYOUT_SPLIT; + info->min_batches = model->batch_size; + info->max_batches = model->batch_size; + info->nb_inputs = metadata->model.num_input; + info->input_info = input; + info->nb_outputs = metadata->model.num_output; + info->output_info = output; + info->wb_size = 0; + + /* Set input info */ + for (i = 0; i < info->nb_inputs; i++) { + rte_memcpy(input[i].name, metadata->input[i].name, MRVL_ML_INPUT_NAME_LEN); + input[i].nb_dims = metadata->input[i].ndim; + input[i].shape = &model->mvtvm.info.input[i].shape[0]; + input[i].type = model->mvtvm.info.input[i].qtype; + input[i].nb_elements = model->mvtvm.info.input[i].nb_elements; + input[i].size = model->mvtvm.info.input[i].nb_elements * + rte_ml_io_type_size_get(model->mvtvm.info.input[i].qtype); + } + + /* Set output info */ + for (i = 0; i < info->nb_outputs; i++) { + rte_memcpy(output[i].name, metadata->output[i].name, MRVL_ML_OUTPUT_NAME_LEN); + output[i].nb_dims = metadata->output[i].ndim; + output[i].shape = &model->mvtvm.info.output[i].shape[0]; + output[i].type = model->mvtvm.info.output[i].qtype; + output[i].nb_elements = model->mvtvm.info.output[i].nb_elements; + output[i].size = model->mvtvm.info.output[i].nb_elements * + rte_ml_io_type_size_get(model->mvtvm.info.output[i].qtype); + } + + return; + +tvm_mrvl_model: + cn10k_ml_model_info_set(cnxk_mldev, model, &model->mvtvm.info, + &model->layer[0].glow.metadata); +} diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h index 2b25a7b568e..eef424b5c2a 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.h +++ b/drivers/ml/cnxk/mvtvm_ml_model.h @@ -11,6 +11,7 @@ #include "cnxk_ml_io.h" +struct cnxk_ml_dev; struct cnxk_ml_model; /* Maximum number of objects per model */ @@ -48,5 +49,6 @@ struct mvtvm_ml_model_data { int mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_model_object *object); void mvtvm_ml_model_io_info_update(struct cnxk_ml_model *model); +void mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); #endif /* _MVTVM_ML_MODEL_H_ */ diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index 8d25b5d4b87..3fae25f6d2d 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -184,6 +184,9 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * /* Update model I/O data */ mvtvm_ml_model_io_info_update(model); + /* Set model info */ + mvtvm_ml_model_info_set(cnxk_mldev, model); + return 0; error: From patchwork Wed Aug 30 15:59:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130909 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A58A641FD1; Wed, 30 Aug 2023 18:05:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B3EF341148; Wed, 30 Aug 2023 18:00:10 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 19D12402BB for ; Wed, 30 Aug 2023 17:59:43 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7K2004364 for ; Wed, 30 Aug 2023 08:59:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=ZzjEthiRNI8jxounMVghvDisdgRiDNceDgCIbttm7Sg=; b=KFarGqaUHMbfZt9zu6SsgjRfRHcGwAJv2H2sgKrqIaNbsnyEvi8GAdUFl6DjiPDqXzar 6+IKa8+2FvRVoImlcon48p+ruvoox7Wp9Rn3HAjxNpyhe+rtHKcH9M3cezvQCosE8xlp ss4gwuKgXfRGUxBlLvQ/tVPrSF7xyuSwjVdCl9yJ6fh2FFoR1RqnpgjeLc88z3uo9tq9 XdMwhJ0HRmbBFTlZrNZA3VlBfPe57ecwEysU8+DNAGK/7bilbmyvqYWwoaP/cmgqGJan 9jliJSsrrRnOtN3uBFkqqZLX2B627NYZSwcr8Kx4f1UZCA+C4WYfvIzyaqE42PqXUqIN xA== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysstv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:43 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:41 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:41 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 6327F3F7080; Wed, 30 Aug 2023 08:59:41 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 28/34] ml/cnxk: support device dump for TVM models Date: Wed, 30 Aug 2023 08:59:18 -0700 Message-ID: <20230830155927.3566-29-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: qWpzHJKM73znVPucirhG05c4chBMGcfe X-Proofpoint-ORIG-GUID: qWpzHJKM73znVPucirhG05c4chBMGcfe X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enabled support to print TVM model layer info. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cnxk_ml_model.c | 9 ++++- drivers/ml/cnxk/cnxk_ml_ops.c | 1 + drivers/ml/cnxk/mvtvm_ml_model.c | 59 ++++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_model.h | 2 ++ 4 files changed, 70 insertions(+), 1 deletion(-) diff --git a/drivers/ml/cnxk/cnxk_ml_model.c b/drivers/ml/cnxk/cnxk_ml_model.c index 746d3ca5a95..e63ee58ab27 100644 --- a/drivers/ml/cnxk/cnxk_ml_model.c +++ b/drivers/ml/cnxk/cnxk_ml_model.c @@ -115,6 +115,8 @@ cnxk_ml_model_dump(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, cnxk_ml_print_line(fp, LINE_LEN); fprintf(fp, "%*s : %u\n", FIELD_LEN, "model_id", model->model_id); fprintf(fp, "%*s : %s\n", FIELD_LEN, "name", model->name); + fprintf(fp, "%*s : %d\n", FIELD_LEN, "type", model->type); + fprintf(fp, "%*s : %d\n", FIELD_LEN, "subtype", model->subtype); fprintf(fp, "%*s : 0x%016lx\n", FIELD_LEN, "model", PLT_U64_CAST(model)); fprintf(fp, "%*s : %u\n", FIELD_LEN, "batch_size", model->batch_size); fprintf(fp, "%*s : %u\n", FIELD_LEN, "nb_layers", model->nb_layers); @@ -131,6 +133,11 @@ cnxk_ml_model_dump(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model, for (layer_id = 0; layer_id < model->nb_layers; layer_id++) { layer = &model->layer[layer_id]; - cn10k_ml_layer_print(cnxk_mldev, layer, fp); + if (layer->type == ML_CNXK_LAYER_TYPE_MRVL) + cn10k_ml_layer_print(cnxk_mldev, layer, fp); +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + else + mvtvm_ml_layer_print(cnxk_mldev, layer, fp); +#endif } } diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index 45c3d61a9b6..f933a2b846f 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -18,6 +18,7 @@ #include "cnxk_ml_io.h" #include "cnxk_ml_model.h" #include "cnxk_ml_ops.h" +#include "cnxk_ml_utils.h" /* ML model macros */ #define CNXK_ML_MODEL_MEMZONE_NAME "ml_cnxk_model_mz" diff --git a/drivers/ml/cnxk/mvtvm_ml_model.c b/drivers/ml/cnxk/mvtvm_ml_model.c index 6d72a5255e2..24dc862d685 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.c +++ b/drivers/ml/cnxk/mvtvm_ml_model.c @@ -15,6 +15,7 @@ #include "cnxk_ml_dev.h" #include "cnxk_ml_model.h" +#include "cnxk_ml_utils.h" /* Objects list */ char mvtvm_object_list[ML_MVTVM_MODEL_OBJECT_MAX][RTE_ML_STR_MAX] = {"mod.so", "mod.json", @@ -241,3 +242,61 @@ mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mo cn10k_ml_model_info_set(cnxk_mldev, model, &model->mvtvm.info, &model->layer[0].glow.metadata); } + +void +mvtvm_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp) +{ + char str[STR_LEN]; + uint8_t i; + + /* Print debug info */ + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, " Layer Information (Layer ID: %u, Name: %s)\n", + cnxk_mldev->index_map[layer->index].layer_id, layer->name); + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "layer_id", + cnxk_mldev->index_map[layer->index].layer_id); + fprintf(fp, "%*s : %s\n", FIELD_LEN, "name", layer->name); + fprintf(fp, "%*s : %d\n", FIELD_LEN, "type", layer->type); + fprintf(fp, "%*s : 0x%016lx\n", FIELD_LEN, "layer", PLT_U64_CAST(layer)); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "batch_size", layer->batch_size); + + /* Print model state */ + if (layer->state == ML_CNXK_LAYER_STATE_LOADED) + fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "loaded"); + if (layer->state == ML_CNXK_LAYER_STATE_JOB_ACTIVE) + fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "job_active"); + if (layer->state == ML_CNXK_LAYER_STATE_STARTED) + fprintf(fp, "%*s : %s\n", FIELD_LEN, "state", "started"); + + fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_inputs", layer->info.nb_inputs); + fprintf(fp, "%*s : %u\n", FIELD_LEN, "num_outputs", layer->info.nb_outputs); + fprintf(fp, "\n"); + + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "%8s %16s %12s\n", "input", "input_name", "input_type"); + cnxk_ml_print_line(fp, LINE_LEN); + for (i = 0; i < layer->info.nb_inputs; i++) { + fprintf(fp, "%8u ", i); + fprintf(fp, "%*s ", 16, layer->info.input[i].name); + rte_ml_io_type_to_str(layer->info.input[i].qtype, str, STR_LEN); + fprintf(fp, "%*s ", 12, str); + } + fprintf(fp, "\n"); + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "\n"); + + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "%8s %16s %12s\n", "output", "output_name", "output_type"); + cnxk_ml_print_line(fp, LINE_LEN); + for (i = 0; i < layer->info.nb_outputs; i++) { + fprintf(fp, "%8u ", i); + fprintf(fp, "%*s ", 16, layer->info.output[i].name); + rte_ml_io_type_to_str(layer->info.output[i].qtype, str, STR_LEN); + fprintf(fp, "%*s ", 12, str); + fprintf(fp, "\n"); + } + fprintf(fp, "\n"); + cnxk_ml_print_line(fp, LINE_LEN); + fprintf(fp, "\n"); +} diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h index eef424b5c2a..fa7735cfaa0 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.h +++ b/drivers/ml/cnxk/mvtvm_ml_model.h @@ -13,6 +13,7 @@ struct cnxk_ml_dev; struct cnxk_ml_model; +struct cnxk_ml_layer; /* Maximum number of objects per model */ #define ML_MVTVM_MODEL_OBJECT_MAX 3 @@ -50,5 +51,6 @@ int mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_model_object *object); void mvtvm_ml_model_io_info_update(struct cnxk_ml_model *model); void mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); +void mvtvm_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp); #endif /* _MVTVM_ML_MODEL_H_ */ From patchwork Wed Aug 30 15:59:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130911 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 395F641FD1; Wed, 30 Aug 2023 18:06:05 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E5E1C4161A; Wed, 30 Aug 2023 18:00:12 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id BC49F402B9 for ; Wed, 30 Aug 2023 17:59:44 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I78012554 for ; Wed, 30 Aug 2023 08:59:44 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=MupgVzzsozKHHqPwsM+OclWzuM1XLJmozdV9sg2YfDY=; b=ZS++8x/DQY2kfvTgQJMTgEV7ol6q8TkFdV3NEK0WzaB47V2rQGrUeU45bZNWhvc9/h7s mquEmS0T+Bz0P1XzZAAWRQcKpyaCXaiSG9ybLxt7lUUNWhZanDOKFM5Oy2Vxauh1xSbY elg8TbVyGerx/6itfptK9ZNnTeNC7GxuNcBuhkrMicrwhHiigQmEA0L6C/YBh12lyZqa GZehTieqkOmnzqXoW8YOfULB2kIRH3htrERUEDFWke2IgmFKuwnZoTwjBMeQ7OYDLyeu hPGI+7r7+EKaNOy7/crhjN4IfggZSQ6kUKgRJRMGnKFwja+PVIsvAbXdw/svTSvYLsnu gw== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-17 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:43 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:41 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:41 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id B49DA3F707D; Wed, 30 Aug 2023 08:59:41 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 29/34] ml/cnxk: enable reporting model runtime as xstats Date: Wed, 30 Aug 2023 08:59:19 -0700 Message-ID: <20230830155927.3566-30-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: zV_7fMdvF98xqHrfXe3Wrfij0qdfou23 X-Proofpoint-GUID: zV_7fMdvF98xqHrfXe3Wrfij0qdfou23 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added model xstats entries to compute runtime latency. Allocated internal resources for TVM model xstats. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cnxk_ml_ops.c | 182 ++++++++++++++++++++++++++++--- drivers/ml/cnxk/cnxk_ml_ops.h | 1 + drivers/ml/cnxk/cnxk_ml_xstats.h | 7 ++ drivers/ml/cnxk/mvtvm_ml_model.h | 24 ++++ drivers/ml/cnxk/mvtvm_ml_ops.c | 24 +++- 5 files changed, 223 insertions(+), 15 deletions(-) diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index f933a2b846f..ff9ecd3c941 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -146,7 +146,8 @@ cnxk_ml_xstats_init(struct cnxk_ml_dev *cnxk_mldev) /* Allocate memory for xstats entries. Don't allocate during reconfigure */ nb_stats = RTE_DIM(device_xstats) + - RTE_DIM(layer_xstats) * ML_CNXK_MAX_MODELS * ML_CNXK_MODEL_MAX_LAYERS; + RTE_DIM(layer_xstats) * ML_CNXK_MAX_MODELS * ML_CNXK_MODEL_MAX_LAYERS + + RTE_DIM(model_xstats) * ML_CNXK_MAX_MODELS; if (cnxk_mldev->xstats.entries == NULL) cnxk_mldev->xstats.entries = rte_zmalloc( "cnxk_ml_xstats", sizeof(struct cnxk_ml_xstats_entry) * nb_stats, @@ -177,6 +178,25 @@ cnxk_ml_xstats_init(struct cnxk_ml_dev *cnxk_mldev) for (model = 0; model < ML_CNXK_MAX_MODELS; model++) { cnxk_mldev->xstats.offset_for_model[model] = stat_id; + for (i = 0; i < RTE_DIM(model_xstats); i++) { + cnxk_mldev->xstats.entries[stat_id].map.id = stat_id; + cnxk_mldev->xstats.entries[stat_id].mode = RTE_ML_DEV_XSTATS_MODEL; + cnxk_mldev->xstats.entries[stat_id].group = CNXK_ML_XSTATS_GROUP_MODEL; + cnxk_mldev->xstats.entries[stat_id].type = model_xstats[i].type; + cnxk_mldev->xstats.entries[stat_id].fn_id = CNXK_ML_XSTATS_FN_MODEL; + cnxk_mldev->xstats.entries[stat_id].obj_idx = model; + cnxk_mldev->xstats.entries[stat_id].layer_id = -1; + cnxk_mldev->xstats.entries[stat_id].reset_allowed = + model_xstats[i].reset_allowed; + + /* Name of xstat is updated during model load */ + snprintf(cnxk_mldev->xstats.entries[stat_id].map.name, + sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), + "Model-%u-%s", model, model_xstats[i].name); + + stat_id++; + } + for (layer = 0; layer < ML_CNXK_MODEL_MAX_LAYERS; layer++) { cnxk_mldev->xstats.offset_for_layer[model][layer] = stat_id; @@ -203,7 +223,8 @@ cnxk_ml_xstats_init(struct cnxk_ml_dev *cnxk_mldev) cnxk_mldev->xstats.count_per_layer[model][layer] = RTE_DIM(layer_xstats); } - cnxk_mldev->xstats.count_per_model[model] = RTE_DIM(layer_xstats); + cnxk_mldev->xstats.count_per_model[model] = + RTE_DIM(layer_xstats) + ML_CNXK_MODEL_MAX_LAYERS * RTE_DIM(model_xstats); } cnxk_mldev->xstats.count_mode_model = stat_id - cnxk_mldev->xstats.count_mode_device; @@ -212,6 +233,42 @@ cnxk_ml_xstats_init(struct cnxk_ml_dev *cnxk_mldev) return 0; } +void +cnxk_ml_xstats_model_name_update(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id) +{ + struct cnxk_ml_model *model; + uint16_t rclk_freq; + uint16_t sclk_freq; + uint16_t stat_id; + char suffix[8]; + uint16_t i; + + model = cnxk_mldev->mldev->data->models[model_id]; + stat_id = cnxk_mldev->xstats.offset_for_model[model_id]; + + roc_clk_freq_get(&rclk_freq, &sclk_freq); + if (sclk_freq == 0) + strcpy(suffix, "cycles"); + else + strcpy(suffix, "ns"); + + /* Update xstat name based on layer name and sclk availability */ + for (i = 0; i < RTE_DIM(model_xstats); i++) { + if (model->type == ML_CNXK_MODEL_TYPE_GLOW) + snprintf(cnxk_mldev->xstats.entries[stat_id].map.name, + sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", + model->glow.metadata.model.name, model_xstats[i].name, suffix); +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + else + snprintf(cnxk_mldev->xstats.entries[stat_id].map.name, + sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", + model->mvtvm.metadata.model.name, model_xstats[i].name, suffix); +#endif + + stat_id++; + } +} + static void cnxk_ml_xstats_uninit(struct cnxk_ml_dev *cnxk_mldev) { @@ -249,6 +306,9 @@ cnxk_ml_dev_xstat_get(struct cnxk_ml_dev *cnxk_mldev, uint16_t obj_idx __rte_unu count += layer->glow.burst_xstats[qp_id].dequeued_count - \ layer->glow.burst_xstats[qp_id].str##_reset_count; \ } \ + value += layer->glow.sync_xstats->str##_latency_tot; \ + count += layer->glow.sync_xstats->dequeued_count - \ + layer->glow.sync_xstats->str##_reset_count; \ if (count != 0) \ value = value / count; \ } while (0) @@ -261,6 +321,9 @@ cnxk_ml_dev_xstat_get(struct cnxk_ml_dev *cnxk_mldev, uint16_t obj_idx __rte_unu count += layer->glow.burst_xstats[qp_id].dequeued_count - \ layer->glow.burst_xstats[qp_id].str##_reset_count; \ } \ + value = PLT_MIN(value, layer->glow.sync_xstats->str##_latency_min); \ + count += layer->glow.sync_xstats->dequeued_count - \ + layer->glow.sync_xstats->str##_reset_count; \ if (count == 0) \ value = 0; \ } while (0) @@ -273,10 +336,53 @@ cnxk_ml_dev_xstat_get(struct cnxk_ml_dev *cnxk_mldev, uint16_t obj_idx __rte_unu count += layer->glow.burst_xstats[qp_id].dequeued_count - \ layer->glow.burst_xstats[qp_id].str##_reset_count; \ } \ + value = PLT_MAX(value, layer->glow.sync_xstats->str##_latency_max); \ + count += layer->glow.sync_xstats->dequeued_count - \ + layer->glow.sync_xstats->str##_reset_count; \ if (count == 0) \ value = 0; \ } while (0) +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM +#define ML_AVG_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count) \ + do { \ + value = 0; \ + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \ + value += model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_tot; \ + count += model->mvtvm.burst_xstats[qp_id].dequeued_count - \ + model->mvtvm.burst_xstats[qp_id].tvm_rt_reset_count; \ + } \ + if (count != 0) \ + value = value / count; \ + } while (0) + +#define ML_MIN_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count) \ + do { \ + value = UINT64_MAX; \ + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \ + value = PLT_MIN(value, \ + model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_min); \ + count += model->mvtvm.burst_xstats[qp_id].dequeued_count - \ + model->mvtvm.burst_xstats[qp_id].tvm_rt_reset_count; \ + } \ + if (count == 0) \ + value = 0; \ + } while (0) + +#define ML_MAX_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count) \ + do { \ + value = 0; \ + for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \ + value = PLT_MAX(value, \ + model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_max); \ + count += model->mvtvm.burst_xstats[qp_id].dequeued_count - \ + model->mvtvm.burst_xstats[qp_id].tvm_rt_reset_count; \ + } \ + if (count == 0) \ + value = 0; \ + } while (0) +#endif + static uint64_t cnxk_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, uint16_t obj_idx, int32_t layer_id, enum cnxk_ml_xstats_type type) @@ -317,6 +423,17 @@ cnxk_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, uint16_t obj_idx, int32_ case max_fw_latency: ML_MAX_FOREACH_QP(cnxk_mldev, layer, qp_id, fw, value, count); break; +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + case avg_rt_latency: + ML_AVG_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count); + break; + case min_rt_latency: + ML_MIN_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count); + break; + case max_rt_latency: + ML_MAX_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count); + break; +#endif default: value = 0; } @@ -907,8 +1024,9 @@ cnxk_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode { struct cnxk_ml_xstats_entry *xs; struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; uint32_t xstats_mode_count; - uint16_t layer_id = 0; + uint16_t layer_id; uint32_t idx = 0; uint32_t i; @@ -925,7 +1043,17 @@ cnxk_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode case RTE_ML_DEV_XSTATS_MODEL: if (model_id >= ML_CNXK_MAX_MODELS) break; - xstats_mode_count = cnxk_mldev->xstats.count_per_layer[model_id][layer_id]; + + model = cnxk_mldev->mldev->data->models[model_id]; + for (layer_id = 0; layer_id < model->nb_layers; layer_id++) { + if (model->layer[layer_id].type == ML_CNXK_LAYER_TYPE_MRVL) + xstats_mode_count += + cnxk_mldev->xstats.count_per_layer[model_id][layer_id]; + } + + if ((model->type == ML_CNXK_MODEL_TYPE_TVM) && + (model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_MRVL)) + xstats_mode_count += RTE_DIM(model_xstats); break; default: return -EINVAL; @@ -939,9 +1067,20 @@ cnxk_ml_dev_xstats_names_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode if (xs->mode != mode) continue; - if (mode == RTE_ML_DEV_XSTATS_MODEL && - (model_id != xs->obj_idx || layer_id != xs->layer_id)) - continue; + if (mode == RTE_ML_DEV_XSTATS_MODEL) { + if (model_id != xs->obj_idx) + continue; + + model = cnxk_mldev->mldev->data->models[model_id]; + if ((model->type == ML_CNXK_MODEL_TYPE_GLOW || + model->subtype == ML_CNXK_MODEL_SUBTYPE_TVM_MRVL) && + xs->group == CNXK_ML_XSTATS_GROUP_MODEL) + continue; + + if (model->type == ML_CNXK_MODEL_TYPE_TVM && + model->layer[xs->layer_id].type == ML_CNXK_LAYER_TYPE_LLVM) + continue; + } strncpy(xstats_map[idx].name, xs->map.name, RTE_ML_STR_MAX); xstats_map[idx].id = xs->map.id; @@ -1002,9 +1141,10 @@ cnxk_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, { struct cnxk_ml_xstats_entry *xs; struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; uint32_t xstats_mode_count; - uint16_t layer_id = 0; cnxk_ml_xstats_fn fn; + uint16_t layer_id; uint64_t val; uint32_t idx; uint32_t i; @@ -1022,7 +1162,14 @@ cnxk_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, case RTE_ML_DEV_XSTATS_MODEL: if (model_id >= ML_CNXK_MAX_MODELS) return -EINVAL; - xstats_mode_count = cnxk_mldev->xstats.count_per_layer[model_id][layer_id]; + + model = cnxk_mldev->mldev->data->models[model_id]; + for (layer_id = 0; layer_id < model->nb_layers; layer_id++) + xstats_mode_count += cnxk_mldev->xstats.count_per_layer[model_id][layer_id]; + + if ((model->type == ML_CNXK_MODEL_TYPE_TVM) && + (model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_MRVL)) + xstats_mode_count += RTE_DIM(model_xstats); break; default: return -EINVAL; @@ -1034,11 +1181,18 @@ cnxk_ml_dev_xstats_get(struct rte_ml_dev *dev, enum rte_ml_dev_xstats_mode mode, if (stat_ids[i] > cnxk_mldev->xstats.count || xs->mode != mode) continue; - if (mode == RTE_ML_DEV_XSTATS_MODEL && - (model_id != xs->obj_idx || layer_id != xs->layer_id)) { - plt_err("Invalid stats_id[%d] = %d for model_id = %d\n", i, stat_ids[i], - model_id); - return -EINVAL; + if (mode == RTE_ML_DEV_XSTATS_MODEL) { + if (model_id != xs->obj_idx) + continue; + + model = cnxk_mldev->mldev->data->models[xs->obj_idx]; + if ((model->type == ML_CNXK_MODEL_TYPE_GLOW || + model->subtype == ML_CNXK_MODEL_SUBTYPE_TVM_MRVL) && + xs->group == CNXK_ML_XSTATS_GROUP_MODEL) + continue; + + if (xs->layer_id == -1 && xs->group == CNXK_ML_XSTATS_GROUP_LAYER) + continue; } switch (xs->fn_id) { diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h index d0c126f34b7..2575f4c6e10 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.h +++ b/drivers/ml/cnxk/cnxk_ml_ops.h @@ -64,6 +64,7 @@ extern struct rte_ml_dev_ops cnxk_ml_ops; int cnxk_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id); int cnxk_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id); +void cnxk_ml_xstats_model_name_update(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_id); __rte_hot uint16_t cnxk_ml_enqueue_burst(struct rte_ml_dev *dev, uint16_t qp_id, struct rte_ml_op **ops, uint16_t nb_ops); diff --git a/drivers/ml/cnxk/cnxk_ml_xstats.h b/drivers/ml/cnxk/cnxk_ml_xstats.h index 5e02bb876ca..a2c9adfe4ab 100644 --- a/drivers/ml/cnxk/cnxk_ml_xstats.h +++ b/drivers/ml/cnxk/cnxk_ml_xstats.h @@ -142,4 +142,11 @@ static const struct cnxk_ml_xstat_info layer_xstats[] = { {"Min-FW-Latency", min_fw_latency, 1}, {"Max-FW-Latency", max_fw_latency, 1}, }; +/* Model xstats */ +static const struct cnxk_ml_xstat_info model_xstats[] = { + {"Avg-RT-Latency", avg_rt_latency, 1}, + {"Min-RT-Latency", min_rt_latency, 1}, + {"Max-RT-Latency", max_rt_latency, 1}, +}; + #endif /* _CNXK_ML_XSTATS_H_ */ diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h index fa7735cfaa0..d71df36f5a5 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.h +++ b/drivers/ml/cnxk/mvtvm_ml_model.h @@ -33,6 +33,27 @@ struct mvtvm_ml_model_object { int64_t size; }; +/* Model fast-path stats */ +struct mvtvm_ml_model_xstats { + /* Total TVM runtime latency, sum of all inferences */ + uint64_t tvm_rt_latency_tot; + + /* TVM runtime latency */ + uint64_t tvm_rt_latency; + + /* Minimum TVM runtime latency */ + uint64_t tvm_rt_latency_min; + + /* Maximum TVM runtime latency */ + uint64_t tvm_rt_latency_max; + + /* Total jobs dequeued */ + uint64_t dequeued_count; + + /* Hardware stats reset index */ + uint64_t tvm_rt_reset_count; +}; + struct mvtvm_ml_model_data { /* Model metadata */ struct tvmdp_model_metadata metadata; @@ -45,6 +66,9 @@ struct mvtvm_ml_model_data { /* Model I/O info */ struct cnxk_ml_io_info info; + + /* Stats for burst ops */ + struct mvtvm_ml_model_xstats *burst_xstats; }; int mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index 3fae25f6d2d..c251579668c 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -16,6 +16,7 @@ #include "cnxk_ml_dev.h" #include "cnxk_ml_model.h" +#include "cnxk_ml_ops.h" /* ML model macros */ #define MVTVM_ML_MODEL_MEMZONE_NAME "ml_mvtvm_model_mz" @@ -59,6 +60,7 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * char str[RTE_MEMZONE_NAMESIZE]; const struct plt_memzone *mz; size_t model_object_size = 0; + size_t model_xstats_size = 0; uint16_t nb_mrvl_layers; uint16_t nb_llvm_layers; uint8_t layer_id = 0; @@ -74,7 +76,11 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * model_object_size = RTE_ALIGN_CEIL(object[0].size, RTE_CACHE_LINE_MIN_SIZE) + RTE_ALIGN_CEIL(object[1].size, RTE_CACHE_LINE_MIN_SIZE) + RTE_ALIGN_CEIL(object[2].size, RTE_CACHE_LINE_MIN_SIZE); - mz_size += model_object_size; + + model_xstats_size = + cnxk_mldev->mldev->data->nb_queue_pairs * sizeof(struct mvtvm_ml_model_xstats); + + mz_size += model_object_size + model_xstats_size; /* Allocate memzone for model object */ snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", MVTVM_ML_MODEL_MEMZONE_NAME, model->model_id); @@ -187,6 +193,22 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * /* Set model info */ mvtvm_ml_model_info_set(cnxk_mldev, model); + /* Update model xstats name */ + cnxk_ml_xstats_model_name_update(cnxk_mldev, model->model_id); + + model->mvtvm.burst_xstats = RTE_PTR_ADD( + model->mvtvm.object.params.addr, + RTE_ALIGN_CEIL(model->mvtvm.object.params.size, RTE_CACHE_LINE_MIN_SIZE)); + + for (int qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { + model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_tot = 0; + model->mvtvm.burst_xstats[qp_id].tvm_rt_latency = 0; + model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_min = UINT64_MAX; + model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_max = 0; + model->mvtvm.burst_xstats[qp_id].tvm_rt_reset_count = 0; + model->mvtvm.burst_xstats[qp_id].dequeued_count = 0; + } + return 0; error: From patchwork Wed Aug 30 15:59:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130914 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BF72D41FD1; Wed, 30 Aug 2023 18:06:37 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DDF87427E5; Wed, 30 Aug 2023 18:00:15 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 5BB8F402C0 for ; Wed, 30 Aug 2023 17:59:45 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U86I79012554 for ; Wed, 30 Aug 2023 08:59:44 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=AhCzCM6/a6MNDUq1/jcm9aWSKFzojHmbRPaTEcutU6Y=; b=MB/73s9GIBqwWvyD4bFIWTY9Disxm7jdswnlHXcGufCSR1TFcpVoWB0SuTTxCEhjs2fJ 3Xgvz0W3zrO2j/D6d6Qd2tz1cvKReOGW2j0zDL+5ZVlGeGYm00yVfCLmYU3YucimCWjj ya9Mu6j12FGaoBYKqTJ6lGsVcI74ef/xmxKL5MyjH/uzT8SoeZinTUxLPIej//qsjAB8 L8KiOxZjq2XF/i0hJVQhjPg5dw+Su/Dn7sBOB07JFfo8yE6VA9bUg07W5snUMvMHrFGb ojCL45POrmZV3a3Au+qtkOcjuTh+irJK2T65L93yyeYlsXP4xdu0XBAjuvBwOgOsPeTN NQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3st1y61ga5-18 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:44 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:42 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:42 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 0E8F13F707E; Wed, 30 Aug 2023 08:59:42 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 30/34] ml/cnxk: implement I/O alloc and free callbacks Date: Wed, 30 Aug 2023 08:59:20 -0700 Message-ID: <20230830155927.3566-31-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: uzf2NPtbAIv9gfrGBtJTjJPXxsxE8CDz X-Proofpoint-GUID: uzf2NPtbAIv9gfrGBtJTjJPXxsxE8CDz X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implemented callback functions for IO allocation and free for Glow layers. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ops.c | 123 +++++++++++++++++++++++++++++++++ drivers/ml/cnxk/cn10k_ml_ops.h | 3 + drivers/ml/cnxk/mvtvm_ml_ops.c | 2 + 3 files changed, 128 insertions(+) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index f70383b1281..23e98b96c59 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -1399,3 +1399,126 @@ cn10k_ml_inference_sync(void *device, uint16_t index, void *input, void *output, error_enqueue: return ret; } + +int +cn10k_ml_io_alloc(void *device, uint16_t model_id, const char *layer_name, uint64_t **input_qbuffer, + uint64_t **output_qbuffer) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + struct cnxk_ml_layer *layer; + + char str[RTE_MEMZONE_NAMESIZE]; + const struct plt_memzone *mz; + uint16_t layer_id = 0; + uint64_t output_size; + uint64_t input_size; + +#ifndef RTE_MLDEV_CNXK_ENABLE_MVTVM + PLT_SET_USED(layer_name); +#endif + + cnxk_mldev = (struct cnxk_ml_dev *)device; + if (cnxk_mldev == NULL) { + plt_err("Invalid device = %p", cnxk_mldev); + return -EINVAL; + } + + model = cnxk_mldev->mldev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + if (model->type == ML_CNXK_MODEL_TYPE_TVM) { + for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) { + if (strcmp(model->layer[layer_id].name, layer_name) == 0) + break; + } + + if (layer_id == model->mvtvm.metadata.model.nb_layers) { + plt_err("Invalid layer name: %s", layer_name); + return -EINVAL; + } + + if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) { + plt_err("Invalid layer name / type: %s", layer_name); + return -EINVAL; + } + } +#endif + + layer = &model->layer[layer_id]; + input_size = PLT_ALIGN_CEIL(layer->info.total_input_sz_q, ML_CN10K_ALIGN_SIZE); + output_size = PLT_ALIGN_CEIL(layer->info.total_output_sz_q, ML_CN10K_ALIGN_SIZE); + + sprintf(str, "cn10k_ml_io_mz_%u_%u", model_id, layer_id); + mz = plt_memzone_reserve_aligned(str, input_size + output_size, 0, ML_CN10K_ALIGN_SIZE); + if (mz == NULL) { + plt_err("io_alloc failed: Unable to allocate memory: model_id = %u, layer_name = %s", + model_id, layer_name); + return -ENOMEM; + } + + *input_qbuffer = mz->addr; + *output_qbuffer = PLT_PTR_ADD(mz->addr, input_size); + + return 0; +} + +int +cn10k_ml_io_free(void *device, uint16_t model_id, const char *layer_name) +{ + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + + char str[RTE_MEMZONE_NAMESIZE]; + const struct plt_memzone *mz; + uint16_t layer_id = 0; + +#ifndef RTE_MLDEV_CNXK_ENABLE_MVTVM + PLT_SET_USED(layer_name); +#endif + + cnxk_mldev = (struct cnxk_ml_dev *)device; + if (cnxk_mldev == NULL) { + plt_err("Invalid device = %p", cnxk_mldev); + return -EINVAL; + } + + model = cnxk_mldev->mldev->data->models[model_id]; + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } + +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + if (model->type == ML_CNXK_MODEL_TYPE_TVM) { + for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) { + if (strcmp(model->layer[layer_id].name, layer_name) == 0) + break; + } + + if (layer_id == model->mvtvm.metadata.model.nb_layers) { + plt_err("Invalid layer name: %s", layer_name); + return -EINVAL; + } + + if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) { + plt_err("Invalid layer name / type: %s", layer_name); + return -EINVAL; + } + } +#endif + + sprintf(str, "cn10k_ml_io_mz_%u_%u", model_id, layer_id); + mz = plt_memzone_lookup(str); + if (mz == NULL) { + plt_err("io_free failed: Memzone not found: model_id = %u, layer_name = %s", + model_id, layer_name); + return -EINVAL; + } + + return plt_memzone_free(mz); +} diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 3e75cae65a3..055651eaa24 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -328,5 +328,8 @@ int cn10k_ml_layer_load(void *device, uint16_t model_id, const char *layer_name, int cn10k_ml_layer_unload(void *device, uint16_t model_id, const char *layer_name); int cn10k_ml_layer_start(void *device, uint16_t model_id, const char *layer_name); int cn10k_ml_layer_stop(void *device, uint16_t model_id, const char *layer_name); +int cn10k_ml_io_alloc(void *device, uint16_t model_id, const char *layer_name, + uint64_t **input_qbuffer, uint64_t **output_qbuffer); +int cn10k_ml_io_free(void *device, uint16_t model_id, const char *layer_name); #endif /* _CN10K_ML_OPS_H_ */ diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index c251579668c..821b3e8f3c9 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -166,6 +166,8 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * callback = &model->mvtvm.cb; callback->tvmrt_glow_layer_load = cn10k_ml_layer_load; callback->tvmrt_glow_layer_unload = cn10k_ml_layer_unload; + callback->tvmrt_io_alloc = cn10k_ml_io_alloc; + callback->tvmrt_io_free = cn10k_ml_io_free; } else { callback = NULL; } From patchwork Wed Aug 30 15:59:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130910 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7CB8841FD1; Wed, 30 Aug 2023 18:05:54 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CFB7441156; Wed, 30 Aug 2023 18:00:11 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 78F1C4028D for ; Wed, 30 Aug 2023 17:59:44 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7K3004364 for ; Wed, 30 Aug 2023 08:59:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=H9uaPiRSQOPYS+rIVMzx/7WGlu3yPlIEXPNoC3Uup3Q=; b=ahFdqkp90E1m6U1QGCS+DgsE54mfGYm7k+rWmQ9XO6B1kKp0lvYdKv5sf9mofseBPiVR qNMrdO2BC1Izt9s0qQJ5p2WaJv7ToIyVKA86S+/p0KUA6JRMpI6fF+NlH+W3ZGISd12M xGaZ0PC4zwA3snWNxXDn4SQQclnlA2IrhOOhp7ojJ1ayj6lwtr3A6s1fLyTgGtN0vfrD O494aJq+gm34YpF1RuSbyvq+inWQnoqv2SbDgLHOFuHPnQKrci5ll8MN6RchoOMXVxUQ Eh0iDz2iw8tobrCOJPbhV+r6rKReew4D7kFChsY26bTiX5uOci9fymTDVlYVSfme/SZS ig== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysstv-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:43 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:42 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:42 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 5B02C3F7081; Wed, 30 Aug 2023 08:59:42 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 31/34] ml/cnxk: add generic ML malloc and free callback Date: Wed, 30 Aug 2023 08:59:21 -0700 Message-ID: <20230830155927.3566-32-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: gWArrxN_WGlOQnnOExMJ7xtIiRrT3C6J X-Proofpoint-ORIG-GUID: gWArrxN_WGlOQnnOExMJ7xtIiRrT3C6J X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implemented generic ML malloc and free callbacks Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ops.c | 30 ++++++++++++++++++++++++++++++ drivers/ml/cnxk/cn10k_ml_ops.h | 3 +++ drivers/ml/cnxk/mvtvm_ml_ops.c | 2 ++ 3 files changed, 35 insertions(+) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 23e98b96c59..140f7a343f9 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -1522,3 +1522,33 @@ cn10k_ml_io_free(void *device, uint16_t model_id, const char *layer_name) return plt_memzone_free(mz); } + +int +cn10k_ml_malloc(const char *name, size_t size, uint32_t align, void **addr) +{ + const struct plt_memzone *mz; + + mz = plt_memzone_reserve_aligned(name, size, 0, align); + if (mz == NULL) { + plt_err("ml_malloc failed: Unable to allocate memory: name = %s", name); + return -ENOMEM; + } + + *addr = mz->addr; + + return 0; +} + +int +cn10k_ml_free(const char *name) +{ + const struct plt_memzone *mz; + + mz = plt_memzone_lookup(name); + if (mz == NULL) { + plt_err("ml_free failed: Memzone not found: name = %s", name); + return -EINVAL; + } + + return plt_memzone_free(mz); +} diff --git a/drivers/ml/cnxk/cn10k_ml_ops.h b/drivers/ml/cnxk/cn10k_ml_ops.h index 055651eaa24..d7df1d003aa 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.h +++ b/drivers/ml/cnxk/cn10k_ml_ops.h @@ -332,4 +332,7 @@ int cn10k_ml_io_alloc(void *device, uint16_t model_id, const char *layer_name, uint64_t **input_qbuffer, uint64_t **output_qbuffer); int cn10k_ml_io_free(void *device, uint16_t model_id, const char *layer_name); +int cn10k_ml_malloc(const char *name, size_t size, uint32_t align, void **addr); +int cn10k_ml_free(const char *name); + #endif /* _CN10K_ML_OPS_H_ */ diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index 821b3e8f3c9..36616ece964 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -168,6 +168,8 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * callback->tvmrt_glow_layer_unload = cn10k_ml_layer_unload; callback->tvmrt_io_alloc = cn10k_ml_io_alloc; callback->tvmrt_io_free = cn10k_ml_io_free; + callback->tvmrt_malloc = cn10k_ml_malloc; + callback->tvmrt_free = cn10k_ml_free; } else { callback = NULL; } From patchwork Wed Aug 30 15:59:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130912 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 71E9341FD1; Wed, 30 Aug 2023 18:06:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E5999427D8; Wed, 30 Aug 2023 18:00:13 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id E35A9402BD for ; Wed, 30 Aug 2023 17:59:44 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7K4004364 for ; Wed, 30 Aug 2023 08:59:44 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=SzvV2crAZvHZxE2qCJ8GBs7FYNpI/y/NgZrWzp90wSQ=; b=SdWuljYEqSbrFMNegTLGUqmFFFgE+PO34itaybdZ4Yy4uOb8OcNqYB5QAkBImQ2OLX/t FQi7rVZMGX/M+Hp4vzeT0s+xVtDs0jVIRWoDnAcrwv7WZa9JcV+PUM822urFkUe6LaP5 UcAVS06QWJawkVEOY8jtP26X8T4vWjeZO2Bz8n776P2zCqeWZf1p4MeGni+qAst8gKHw HmsmcLi6JEe69z5mfsvNl6NP0visgZ2Hh0AZr0fk/+b2bVF92E2CRUmf7HXnH6dehCPU khoJtg8clbETrC8qZ3gyuyp//thPNTZ/aM3UEhgJFBmT7qya/5sModwELHaLFjSCOTKI BQ== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysstv-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:43 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:42 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:42 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id AC2913F7080; Wed, 30 Aug 2023 08:59:42 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 32/34] ml/cnxk: support quantize and dequantize callback Date: Wed, 30 Aug 2023 08:59:22 -0700 Message-ID: <20230830155927.3566-33-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: pjXSAdQ8j4JE92qqkmdaYEw3ulQULWem X-Proofpoint-ORIG-GUID: pjXSAdQ8j4JE92qqkmdaYEw3ulQULWem X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Prince Takkar Added support for quantize and dequantize callback functions for TVM models. Signed-off-by: Prince Takkar --- drivers/ml/cnxk/meson.build | 5 ++ drivers/ml/cnxk/mvtvm_ml_model.h | 2 + drivers/ml/cnxk/mvtvm_ml_ops.c | 127 +++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_ops.h | 6 ++ 4 files changed, 140 insertions(+) diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index db175b0834d..09a62b5c55a 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -19,6 +19,11 @@ if not jansson_dep.found() enable_mvtvm = false endif +if not cc.check_header('dlpack/dlpack.h') + message('drivers/ml/cnxk: dlpack.h not found') + enable_mvtvm = false +endif + tvmrt_lib = cc.find_library('tvm_runtime', required: false) if tvmrt_lib.found() tvmrt_dep = declare_dependency(dependencies: tvmrt_lib) diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h index d71df36f5a5..57a6ce0bb1a 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.h +++ b/drivers/ml/cnxk/mvtvm_ml_model.h @@ -5,6 +5,8 @@ #ifndef _MVTVM_ML_MODEL_H_ #define _MVTVM_ML_MODEL_H_ +#include + #include #include diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index 36616ece964..0bee5884640 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -9,6 +9,8 @@ #include #include +#include + #include "cn10k_ml_ops.h" #include "mvtvm_ml_model.h" @@ -170,6 +172,8 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * callback->tvmrt_io_free = cn10k_ml_io_free; callback->tvmrt_malloc = cn10k_ml_malloc; callback->tvmrt_free = cn10k_ml_free; + callback->tvmrt_quantize = mvtvm_ml_io_quantize; + callback->tvmrt_dequantize = mvtvm_ml_io_dequantize; } else { callback = NULL; } @@ -300,3 +304,126 @@ mvtvm_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model) return 0; } + +int +mvtvm_ml_io_quantize(void *device, uint16_t model_id, const char *layer_name, + const DLTensor **deq_tensor, void *qbuffer) +{ + struct cnxk_ml_io_info *info = NULL; + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + uint16_t layer_id = 0; + uint8_t *lcl_dbuffer; + uint8_t *lcl_qbuffer; + uint32_t i; + int ret; + +#ifdef CNXK_ML_DEV_DEBUG + if ((device == NULL) || (deq_tensor == NULL) || (qbuffer == NULL)) + return -EINVAL; +#endif + + cnxk_mldev = (struct cnxk_ml_dev *)device; + + model = cnxk_mldev->mldev->data->models[model_id]; +#ifdef CNXK_ML_DEV_DEBUG + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } +#endif + + /* Get layer id */ + for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) { + if (strcmp(model->layer[layer_id].name, layer_name) == 0) + break; + } + +#ifdef CNXK_ML_DEV_DEBUG + if (layer_id == model->mvtvm.metadata.model.nb_layers) { + plt_err("Invalid layer name: %s", layer_name); + return -EINVAL; + } + + if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) { + plt_err("Invalid layer name / type: %s", layer_name); + return -EINVAL; + } +#endif + + info = &model->layer[layer_id].info; + lcl_qbuffer = (uint8_t *)qbuffer; + + for (i = 0; i < info->nb_inputs; i++) { + lcl_dbuffer = PLT_PTR_ADD(deq_tensor[i]->data, deq_tensor[i]->byte_offset); + + ret = cnxk_ml_io_quantize_single(&info->input[i], lcl_dbuffer, lcl_qbuffer); + if (ret < 0) + return ret; + + lcl_qbuffer += info->input[i].sz_q; + } + + return 0; +} + +int +mvtvm_ml_io_dequantize(void *device, uint16_t model_id, const char *layer_name, void *qbuffer, + const DLTensor **deq_tensor) +{ + struct cnxk_ml_io_info *info = NULL; + struct cnxk_ml_dev *cnxk_mldev; + struct cnxk_ml_model *model; + uint16_t layer_id = 0; + uint8_t *lcl_dbuffer; + uint8_t *lcl_qbuffer; + uint32_t i; + int ret; + +#ifdef CNXK_ML_DEV_DEBUG + if ((device == NULL) || (deq_tensor == NULL) || (qbuffer == NULL)) + return -EINVAL; +#endif + + cnxk_mldev = (struct cnxk_ml_dev *)device; + + model = cnxk_mldev->mldev->data->models[model_id]; +#ifdef CNXK_ML_DEV_DEBUG + if (model == NULL) { + plt_err("Invalid model_id = %u", model_id); + return -EINVAL; + } +#endif + + for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) { + if (strcmp(model->layer[layer_id].name, layer_name) == 0) + break; + } + +#ifdef CNXK_ML_DEV_DEBUG + if (layer_id == model->mvtvm.metadata.model.nb_layers) { + plt_err("Invalid layer name: %s", layer_name); + return -EINVAL; + } + + if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) { + plt_err("Invalid layer name / type: %s", layer_name); + return -EINVAL; + } +#endif + + info = &model->layer[layer_id].info; + lcl_qbuffer = (uint8_t *)qbuffer; + + for (i = 0; i < info->nb_outputs; i++) { + lcl_dbuffer = PLT_PTR_ADD(deq_tensor[i]->data, deq_tensor[i]->byte_offset); + + ret = cnxk_ml_io_dequantize_single(&info->output[i], lcl_qbuffer, lcl_dbuffer); + if (ret < 0) + return ret; + + lcl_qbuffer += info->output[i].sz_q; + } + + return 0; +} diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.h b/drivers/ml/cnxk/mvtvm_ml_ops.h index f6ede6229f4..3a1e97a7a08 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.h +++ b/drivers/ml/cnxk/mvtvm_ml_ops.h @@ -5,6 +5,8 @@ #ifndef _MVTVM_ML_OPS_H_ #define _MVTVM_ML_OPS_H_ +#include + #include struct cnxk_ml_dev; @@ -17,5 +19,9 @@ int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_para int mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); int mvtvm_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); int mvtvm_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model); +int mvtvm_ml_io_quantize(void *device, uint16_t model_id, const char *layer_name, + const DLTensor **deq_tensor, void *qbuffer); +int mvtvm_ml_io_dequantize(void *device, uint16_t model_id, const char *layer_name, void *qbuffer, + const DLTensor **deq_tensor); #endif /* _MVTVM_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130915 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BAF4041FD1; Wed, 30 Aug 2023 18:06:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D676C427E9; Wed, 30 Aug 2023 18:00:16 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 6644D402C2 for ; Wed, 30 Aug 2023 17:59:45 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7K5004364 for ; Wed, 30 Aug 2023 08:59:44 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=HAYabDzDImDdm/CHCbdwT3e4FequM7pkb1J+jEtV6DQ=; b=SyfuVHsUtqFCFg2p6+eczaQ3PH391LCmd3Idstmr/QxME+jLQFsTiUJmviiEPSM7Q6DB WiNGvcJr/Qi5EMRaCbVS9kHj8gL9kECXXBLKFi4mR+73mj2zo8yhSJB/yQDptjLBmdsq R162uwzsxhHYTCvTBzHli7+7E6Z4mbLrnhynnFQTKyH/ssW+Ak3+pai9WUR+z94P9reR bNf8MENN04ysoloT8oCTB5PdmycwrrLzF0qQCQBh5ey08PadWPKMbPDfF/r2KIOK7WeJ b+Ijm0cCJ1B6floeC6aYOC01RatCKEr3MYH7WPEfSPIDYLH3rWjyasoo3vM3dIX95ryO sw== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fysstv-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:44 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:43 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:43 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 0C1BB3F707D; Wed, 30 Aug 2023 08:59:43 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 33/34] ml/cnxk: enable fast-path ops for TVM models Date: Wed, 30 Aug 2023 08:59:23 -0700 Message-ID: <20230830155927.3566-34-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: QkilcD0o7fiAysM2y0S4lbRarbG3cKJh X-Proofpoint-ORIG-GUID: QkilcD0o7fiAysM2y0S4lbRarbG3cKJh X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Anup Prabhu Enable fast-path ops support for TVM models. Models would use TVMDP library function calls to execute inference operations for Hybrid and LLVM model sub-types. For TVM MRVL model subtypes that have a single MRVL layer, the inference requests are directly enqueued to hardware by the driver. Signed-off-by: Anup Prabhu Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_ops.c | 4 - drivers/ml/cnxk/cnxk_ml_io.h | 6 ++ drivers/ml/cnxk/cnxk_ml_ops.c | 4 + drivers/ml/cnxk/cnxk_ml_ops.h | 9 +++ drivers/ml/cnxk/mvtvm_ml_model.c | 20 +++++ drivers/ml/cnxk/mvtvm_ml_model.h | 6 ++ drivers/ml/cnxk/mvtvm_ml_ops.c | 124 +++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_ops.h | 43 +++++++++++ 8 files changed, 212 insertions(+), 4 deletions(-) diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c index 140f7a343f9..c1353fb0c81 100644 --- a/drivers/ml/cnxk/cn10k_ml_ops.c +++ b/drivers/ml/cnxk/cn10k_ml_ops.c @@ -287,10 +287,6 @@ cn10k_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_c else cn10k_mldev->ml_jcmdq_enqueue = roc_ml_jcmdq_enqueue_lf; - cnxk_mldev->mldev->enqueue_burst = cnxk_ml_enqueue_burst; - cnxk_mldev->mldev->dequeue_burst = cnxk_ml_dequeue_burst; - cnxk_mldev->mldev->op_error_get = cn10k_ml_op_error_get; - return 0; } diff --git a/drivers/ml/cnxk/cnxk_ml_io.h b/drivers/ml/cnxk/cnxk_ml_io.h index 5de166c2520..6d5d25a7c9c 100644 --- a/drivers/ml/cnxk/cnxk_ml_io.h +++ b/drivers/ml/cnxk/cnxk_ml_io.h @@ -47,6 +47,12 @@ struct cnxk_ml_io { /* Scale */ float scale; + + /* Dequantized offset */ + uint32_t offset_d; + + /* Quantized offset */ + uint32_t offset_q; }; /* Model / Layer IO structure */ diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index ff9ecd3c941..c8491646da9 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -758,6 +758,10 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co cnxk_mldev->max_nb_layers = cnxk_mldev->cn10k_mldev.fw.req->cn10k_req.jd.fw_load.cap.s.max_models; + cnxk_mldev->mldev->enqueue_burst = cnxk_ml_enqueue_burst; + cnxk_mldev->mldev->dequeue_burst = cnxk_ml_dequeue_burst; + cnxk_mldev->mldev->op_error_get = cn10k_ml_op_error_get; + /* Allocate and initialize index_map */ if (cnxk_mldev->index_map == NULL) { cnxk_mldev->index_map = diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h index 2575f4c6e10..62e2b17e35b 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.h +++ b/drivers/ml/cnxk/cnxk_ml_ops.h @@ -12,12 +12,21 @@ #include "cn10k_ml_ops.h" +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM +#include "mvtvm_ml_ops.h" +#endif + /* Request structure */ struct cnxk_ml_req { /* Device specific request */ union { /* CN10K */ struct cn10k_ml_req cn10k_req; + +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + /* MVTVM */ + struct mvtvm_ml_req mvtvm_req; +#endif }; /* Address of status field */ diff --git a/drivers/ml/cnxk/mvtvm_ml_model.c b/drivers/ml/cnxk/mvtvm_ml_model.c index 24dc862d685..4ac053408e2 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.c +++ b/drivers/ml/cnxk/mvtvm_ml_model.c @@ -136,6 +136,16 @@ mvtvm_ml_model_io_info_update(struct cnxk_ml_model *model) model->mvtvm.info.total_input_sz_d += model->mvtvm.info.input[i].sz_d; model->mvtvm.info.total_input_sz_q += model->mvtvm.info.input[i].sz_q; + model->mvtvm.info.input[i].offset_d = model->mvtvm.info.total_input_sz_d; + model->mvtvm.info.input[i].offset_q = model->mvtvm.info.total_input_sz_q; + + model->mvtvm.input_tensor[i].device = metadata->input[i].device; + model->mvtvm.input_tensor[i].ndim = metadata->input[i].ndim; + model->mvtvm.input_tensor[i].dtype = metadata->input[i].datatype; + model->mvtvm.input_tensor[i].shape = metadata->input[i].shape; + model->mvtvm.input_tensor[i].strides = NULL; + model->mvtvm.input_tensor[i].byte_offset = model->mvtvm.info.input[i].offset_q; + plt_ml_dbg("model_id = %u, input[%u] - sz_d = %u sz_q = %u", model->model_id, i, model->mvtvm.info.input[i].sz_d, model->mvtvm.info.input[i].sz_q); } @@ -169,6 +179,16 @@ mvtvm_ml_model_io_info_update(struct cnxk_ml_model *model) model->mvtvm.info.total_output_sz_d += model->mvtvm.info.output[i].sz_d; model->mvtvm.info.total_output_sz_q += model->mvtvm.info.output[i].sz_q; + model->mvtvm.info.output[i].offset_d = model->mvtvm.info.total_output_sz_d; + model->mvtvm.info.output[i].offset_q = model->mvtvm.info.total_output_sz_q; + + model->mvtvm.output_tensor[i].device = metadata->output[i].device; + model->mvtvm.output_tensor[i].ndim = metadata->output[i].ndim; + model->mvtvm.output_tensor[i].dtype = metadata->output[i].datatype; + model->mvtvm.output_tensor[i].shape = metadata->output[i].shape; + model->mvtvm.output_tensor[i].strides = NULL; + model->mvtvm.output_tensor[i].byte_offset = model->mvtvm.info.output[i].offset_q; + plt_ml_dbg("model_id = %u, output[%u] - sz_d = %u sz_q = %u", model->model_id, i, model->mvtvm.info.output[i].sz_d, model->mvtvm.info.output[i].sz_q); } diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h index 57a6ce0bb1a..08e101bbe74 100644 --- a/drivers/ml/cnxk/mvtvm_ml_model.h +++ b/drivers/ml/cnxk/mvtvm_ml_model.h @@ -71,6 +71,12 @@ struct mvtvm_ml_model_data { /* Stats for burst ops */ struct mvtvm_ml_model_xstats *burst_xstats; + + /* Input Tensor */ + DLTensor input_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT]; + + /* Output Tensor */ + DLTensor output_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT]; }; int mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index 0bee5884640..e8484b3bd92 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -23,6 +23,12 @@ /* ML model macros */ #define MVTVM_ML_MODEL_MEMZONE_NAME "ml_mvtvm_model_mz" +__rte_hot static void +mvtvm_ml_set_poll_addr(struct cnxk_ml_req *req) +{ + req->status = &req->mvtvm_req.status; +} + int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf) { @@ -174,6 +180,7 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * callback->tvmrt_free = cn10k_ml_free; callback->tvmrt_quantize = mvtvm_ml_io_quantize; callback->tvmrt_dequantize = mvtvm_ml_io_dequantize; + callback->tvmrt_inference = cn10k_ml_inference_sync; } else { callback = NULL; } @@ -217,6 +224,19 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * model->mvtvm.burst_xstats[qp_id].dequeued_count = 0; } + /* Set model specific fast path functions */ + if (model->subtype == ML_CNXK_MODEL_SUBTYPE_TVM_MRVL) { + model->enqueue_single = cn10k_ml_enqueue_single; + model->result_update = cn10k_ml_result_update; + model->set_error_code = cn10k_ml_set_error_code; + model->set_poll_addr = cn10k_ml_set_poll_addr; + } else { + model->enqueue_single = mvtvm_ml_enqueue_single; + model->result_update = mvtvm_ml_result_update; + model->set_error_code = mvtvm_ml_set_error_code; + model->set_poll_addr = mvtvm_ml_set_poll_addr; + } + return 0; error: @@ -427,3 +447,107 @@ mvtvm_ml_io_dequantize(void *device, uint16_t model_id, const char *layer_name, return 0; } + +static int +mvtvm_ml_model_run(struct cnxk_ml_model *model, struct rte_ml_op *op, struct cnxk_ml_req *req) +{ + uint8_t i; + + rte_memcpy(req->mvtvm_req.input_tensor, model->mvtvm.input_tensor, + model->mvtvm.metadata.model.num_input * sizeof(DLTensor)); + for (i = 0; i < model->mvtvm.metadata.model.num_input; i++) { + req->mvtvm_req.input_tensor[i].data = op->input[i]->addr; + req->mvtvm_req.input_tensor[i].byte_offset = 0; + } + + rte_memcpy(req->mvtvm_req.output_tensor, model->mvtvm.output_tensor, + model->mvtvm.metadata.model.num_output * sizeof(DLTensor)); + for (i = 0; i < model->mvtvm.metadata.model.num_output; i++) { + req->mvtvm_req.output_tensor[i].data = op->output[i]->addr; + req->mvtvm_req.output_tensor[i].byte_offset = 0; + } + + tvmdp_model_run(model->model_id, model->mvtvm.metadata.model.num_input, + req->mvtvm_req.input_tensor, model->mvtvm.metadata.model.num_output, + req->mvtvm_req.output_tensor, &req->mvtvm_req.result, + &req->mvtvm_req.status); + + plt_write64(ML_CNXK_POLL_JOB_FINISH, req->status); + + return 0; +} + +__rte_hot void +mvtvm_ml_set_error_code(struct cnxk_ml_req *req, uint64_t etype, uint64_t stype) +{ + RTE_SET_USED(stype); + + req->mvtvm_req.result.error_code = etype; +} + +__rte_hot bool +mvtvm_ml_enqueue_single(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op, uint16_t layer_id, + struct cnxk_ml_qp *qp, uint64_t head) +{ + struct cnxk_ml_model *model; + struct cnxk_ml_queue *queue; + struct cnxk_ml_req *req; + + RTE_SET_USED(layer_id); + + queue = &qp->queue; + req = &queue->reqs[head]; + model = cnxk_mldev->mldev->data->models[op->model_id]; + + model->set_poll_addr(req); + memset(&req->mvtvm_req.result, 0, sizeof(struct mvtvm_ml_result)); + req->mvtvm_req.result.error_code = 0x0; + req->mvtvm_req.result.user_ptr = op->user_ptr; + + cnxk_ml_set_poll_ptr(req); + mvtvm_ml_model_run(model, op, req); + req->timeout = plt_tsc_cycles() + queue->wait_cycles; + req->op = op; + + return true; +} + +__rte_hot void +mvtvm_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request) +{ + struct mvtvm_ml_model_xstats *xstats; + struct mvtvm_ml_result *result; + struct cnxk_ml_model *model; + struct cnxk_ml_req *req; + uint64_t tvm_rt_latency; + struct cnxk_ml_qp *qp; + struct rte_ml_op *op; + + req = (struct cnxk_ml_req *)request; + result = &req->mvtvm_req.result; + op = req->op; + qp = cnxk_mldev->mldev->data->queue_pairs[qp_id]; + op->impl_opaque = result->error_code; + + if (likely(result->error_code == 0)) { + qp->stats.dequeued_count++; + op->status = RTE_ML_OP_STATUS_SUCCESS; + + model = cnxk_mldev->mldev->data->models[op->model_id]; + xstats = &model->mvtvm.burst_xstats[qp_id]; + + if (unlikely(xstats->dequeued_count == xstats->tvm_rt_reset_count)) { + xstats->tvm_rt_latency_min = UINT64_MAX; + xstats->tvm_rt_latency_max = 0; + } + tvm_rt_latency = result->stats.end_ns - result->stats.start_ns; + xstats->tvm_rt_latency = tvm_rt_latency; + xstats->tvm_rt_latency_tot += tvm_rt_latency; + xstats->tvm_rt_latency_min = RTE_MIN(xstats->tvm_rt_latency_min, tvm_rt_latency); + xstats->tvm_rt_latency_max = RTE_MAX(xstats->tvm_rt_latency_max, tvm_rt_latency); + xstats->dequeued_count++; + } else { + qp->stats.dequeue_err_count++; + op->status = RTE_ML_OP_STATUS_ERROR; + } +} diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.h b/drivers/ml/cnxk/mvtvm_ml_ops.h index 3a1e97a7a08..dba055c22e7 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.h +++ b/drivers/ml/cnxk/mvtvm_ml_ops.h @@ -11,6 +11,44 @@ struct cnxk_ml_dev; struct cnxk_ml_model; +struct cnxk_ml_qp; +struct cnxk_ml_req; + +/* Inference stats */ +struct mvtvm_ml_stats { + /* Start ns */ + uint64_t start_ns; + + /* Start ns */ + uint64_t end_ns; +}; + +/* Result structure */ +struct mvtvm_ml_result { + /* Job error code */ + uint64_t error_code; + + /* Inference stats */ + struct mvtvm_ml_stats stats; + + /* User context pointer */ + void *user_ptr; +}; + +/* MVTVM specific request */ +struct mvtvm_ml_req { + /* Input tensors */ + DLTensor input_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT]; + + /* Output tensors */ + DLTensor output_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT]; + + /* Status field for poll mode requests */ + volatile uint64_t status; + + /* Result */ + struct mvtvm_ml_result result; +}; int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf); int mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev); @@ -24,4 +62,9 @@ int mvtvm_ml_io_quantize(void *device, uint16_t model_id, const char *layer_name int mvtvm_ml_io_dequantize(void *device, uint16_t model_id, const char *layer_name, void *qbuffer, const DLTensor **deq_tensor); +__rte_hot bool mvtvm_ml_enqueue_single(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op, + uint16_t layer_id, struct cnxk_ml_qp *qp, uint64_t head); +__rte_hot void mvtvm_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request); +__rte_hot void mvtvm_ml_set_error_code(struct cnxk_ml_req *req, uint64_t etype, uint64_t stype); + #endif /* _MVTVM_ML_OPS_H_ */ From patchwork Wed Aug 30 15:59:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikanth Yalavarthi X-Patchwork-Id: 130916 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E7B3841FD1; Wed, 30 Aug 2023 18:06:55 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CAADC42C24; Wed, 30 Aug 2023 18:00:17 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 85193402C4 for ; Wed, 30 Aug 2023 17:59:46 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37U6P7K6004364 for ; Wed, 30 Aug 2023 08:59:45 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=isORdApt7E0oH04xIyz/TYZtqcSsnQcN7F+WZUlz9Jw=; b=ZtlMLJNdrATcH3E5N1c/X3GwrOXjlAhNYfVmqQR9E5YeUlCicI8IHrLSQTgphiqYMlmF gELYj25vqAEZ6SBBH82Udv6P1FJWwmceATE73quwKgiKwHapYZTU/032NFm7ClWJNVbx /18HKfT9Bw2USvR27yI+BDoNJH4Ex+Xs+7kAxG8EX/lPeXItCv9nVbeh52nEzDLw7j1l Q81X459i8DpiGCDJd5XRGlljpLphwBC1TSbmwaQkqdUGkKmmYl11Ea/rLzu5R8sgavHh /v4NO554F6vtJR/iDi8TAS0lSUpt3B2NMkB3TsXTKn9OpfZ1dHXmobvCq0EIceiVkZg/ 7Q== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3st0fyssu0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 30 Aug 2023 08:59:45 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 30 Aug 2023 08:59:43 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Wed, 30 Aug 2023 08:59:43 -0700 Received: from ml-host-33.caveonetworks.com (unknown [10.110.143.233]) by maili.marvell.com (Postfix) with ESMTP id 6420E3F707E; Wed, 30 Aug 2023 08:59:43 -0700 (PDT) From: Srikanth Yalavarthi To: Srikanth Yalavarthi CC: , , , Subject: [PATCH v1 34/34] ml/cnxk: enable creation of mvtvm virtual device Date: Wed, 30 Aug 2023 08:59:24 -0700 Message-ID: <20230830155927.3566-35-syalavarthi@marvell.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230830155927.3566-1-syalavarthi@marvell.com> References: <20230830155927.3566-1-syalavarthi@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: n3BZ55akUIgAqOEMtwweQLaVNqfRvVXC X-Proofpoint-ORIG-GUID: n3BZ55akUIgAqOEMtwweQLaVNqfRvVXC X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_12,2023-08-29_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable support to create a mvtvm virtual device on system's without a PCI based ML HW accelerator. Signed-off-by: Srikanth Yalavarthi --- drivers/ml/cnxk/cn10k_ml_dev.c | 8 ++ drivers/ml/cnxk/cn10k_ml_dev.h | 3 + drivers/ml/cnxk/cnxk_ml_dev.c | 3 + drivers/ml/cnxk/cnxk_ml_dev.h | 21 ++++ drivers/ml/cnxk/cnxk_ml_ops.c | 86 ++++++++++---- drivers/ml/cnxk/meson.build | 2 + drivers/ml/cnxk/mvtvm_ml_dev.c | 198 +++++++++++++++++++++++++++++++++ drivers/ml/cnxk/mvtvm_ml_dev.h | 40 +++++++ drivers/ml/cnxk/mvtvm_ml_ops.c | 34 +++++- drivers/ml/cnxk/mvtvm_ml_ops.h | 2 + 10 files changed, 372 insertions(+), 25 deletions(-) create mode 100644 drivers/ml/cnxk/mvtvm_ml_dev.c create mode 100644 drivers/ml/cnxk/mvtvm_ml_dev.h diff --git a/drivers/ml/cnxk/cn10k_ml_dev.c b/drivers/ml/cnxk/cn10k_ml_dev.c index 20c114b8bf7..e6dc87e3530 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.c +++ b/drivers/ml/cnxk/cn10k_ml_dev.c @@ -368,6 +368,12 @@ cn10k_ml_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_de PLT_SET_USED(pci_drv); + if (cnxk_ml_dev_initialized == 1) { + plt_err("ML CNXK device already initialized!"); + plt_err("Cannot initialize CN10K PCI dev"); + rte_exit(-EINVAL, "Invalid EAL arguments "); + } + init_params = (struct rte_ml_dev_pmd_init_params){ .socket_id = rte_socket_id(), .private_data_size = sizeof(struct cnxk_ml_dev)}; @@ -414,6 +420,8 @@ cn10k_ml_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_de dev->dequeue_burst = NULL; dev->op_error_get = NULL; + cnxk_ml_dev_initialized = 1; + cnxk_mldev->type = CNXK_ML_DEV_TYPE_PCI; cnxk_mldev->state = ML_CNXK_DEV_STATE_PROBED; return 0; diff --git a/drivers/ml/cnxk/cn10k_ml_dev.h b/drivers/ml/cnxk/cn10k_ml_dev.h index 2e7eb6c9ef9..cee405f3f5b 100644 --- a/drivers/ml/cnxk/cn10k_ml_dev.h +++ b/drivers/ml/cnxk/cn10k_ml_dev.h @@ -11,6 +11,9 @@ #include "cnxk_ml_io.h" +/* Device status */ +extern int cnxk_ml_dev_initialized; + /* Dummy Device ops */ extern struct rte_ml_dev_ops ml_dev_dummy_ops; diff --git a/drivers/ml/cnxk/cnxk_ml_dev.c b/drivers/ml/cnxk/cnxk_ml_dev.c index 63d1c9e417b..dc4512223ca 100644 --- a/drivers/ml/cnxk/cnxk_ml_dev.c +++ b/drivers/ml/cnxk/cnxk_ml_dev.c @@ -7,6 +7,9 @@ #include "cnxk_ml_dev.h" +/* Device status */ +int cnxk_ml_dev_initialized; + /* Dummy operations for ML device */ struct rte_ml_dev_ops ml_dev_dummy_ops = {0}; diff --git a/drivers/ml/cnxk/cnxk_ml_dev.h b/drivers/ml/cnxk/cnxk_ml_dev.h index 382fca64bea..491c4c4aea5 100644 --- a/drivers/ml/cnxk/cnxk_ml_dev.h +++ b/drivers/ml/cnxk/cnxk_ml_dev.h @@ -9,6 +9,10 @@ #include "cn10k_ml_dev.h" +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM +#include "mvtvm_ml_dev.h" +#endif + #include "cnxk_ml_xstats.h" /* ML command timeout in seconds */ @@ -34,6 +38,15 @@ struct cnxk_ml_error_db { char str[RTE_ML_STR_MAX]; }; +/* Device type */ +enum cnxk_ml_dev_type { + /* PCI based Marvell's ML HW accelerator device */ + CNXK_ML_DEV_TYPE_PCI, + + /* Generic Virtual device */ + CNXK_ML_DEV_TYPE_VDEV, +}; + /* Device configuration state enum */ enum cnxk_ml_dev_state { /* Probed and not configured */ @@ -66,6 +79,9 @@ struct cnxk_ml_dev { /* RTE device */ struct rte_ml_dev *mldev; + /* Device type */ + enum cnxk_ml_dev_type type; + /* Configuration state */ enum cnxk_ml_dev_state state; @@ -87,6 +103,11 @@ struct cnxk_ml_dev { /* CN10K device structure */ struct cn10k_ml_dev cn10k_mldev; +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + /* MVTVM device structure */ + struct mvtvm_ml_dev mvtvm_mldev; +#endif + /* Maximum number of layers */ uint64_t max_nb_layers; diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c index c8491646da9..3525215e716 100644 --- a/drivers/ml/cnxk/cnxk_ml_ops.c +++ b/drivers/ml/cnxk/cnxk_ml_ops.c @@ -125,7 +125,8 @@ cnxk_ml_qp_create(const struct rte_ml_dev *dev, uint16_t qp_id, uint32_t nb_desc qp->stats.enqueue_err_count = 0; qp->stats.dequeue_err_count = 0; - cn10k_ml_qp_initialize(cnxk_mldev, qp); + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) + cn10k_ml_qp_initialize(cnxk_mldev, qp); return qp; @@ -604,7 +605,14 @@ cnxk_ml_dev_info_get(struct rte_ml_dev *dev, struct rte_ml_dev_info *dev_info) dev_info->driver_name = dev->device->driver->name; dev_info->max_models = ML_CNXK_MAX_MODELS; - return cn10k_ml_dev_info_get(cnxk_mldev, dev_info); + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) + return cn10k_ml_dev_info_get(cnxk_mldev, dev_info); +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + else + return mvtvm_ml_dev_info_get(cnxk_mldev, dev_info); +#endif + + return 0; } static int @@ -642,9 +650,11 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co conf->nb_queue_pairs, conf->nb_models); /* Load firmware */ - ret = cn10k_ml_fw_load(cnxk_mldev); - if (ret != 0) - return ret; + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) { + ret = cn10k_ml_fw_load(cnxk_mldev); + if (ret != 0) + return ret; + } } else if (cnxk_mldev->state == ML_CNXK_DEV_STATE_CONFIGURED) { plt_ml_dbg("Re-configuring ML device, nb_queue_pairs = %u, nb_models = %u", conf->nb_queue_pairs, conf->nb_models); @@ -742,10 +752,12 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co } dev->data->nb_models = conf->nb_models; - ret = cn10k_ml_dev_configure(cnxk_mldev, conf); - if (ret != 0) { - plt_err("Failed to configure CN10K ML Device"); - goto error; + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) { + ret = cn10k_ml_dev_configure(cnxk_mldev, conf); + if (ret != 0) { + plt_err("Failed to configure CN10K ML Device"); + goto error; + } } #ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM @@ -755,12 +767,17 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co #endif /* Set device capabilities */ - cnxk_mldev->max_nb_layers = - cnxk_mldev->cn10k_mldev.fw.req->cn10k_req.jd.fw_load.cap.s.max_models; + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) + cnxk_mldev->max_nb_layers = + cnxk_mldev->cn10k_mldev.fw.req->cn10k_req.jd.fw_load.cap.s.max_models; + else + cnxk_mldev->max_nb_layers = ML_CNXK_MAX_MODELS; cnxk_mldev->mldev->enqueue_burst = cnxk_ml_enqueue_burst; cnxk_mldev->mldev->dequeue_burst = cnxk_ml_dequeue_burst; - cnxk_mldev->mldev->op_error_get = cn10k_ml_op_error_get; + + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) + cnxk_mldev->mldev->op_error_get = cn10k_ml_op_error_get; /* Allocate and initialize index_map */ if (cnxk_mldev->index_map == NULL) { @@ -823,8 +840,10 @@ cnxk_ml_dev_close(struct rte_ml_dev *dev) plt_err("Failed to close MVTVM ML Device"); #endif - if (cn10k_ml_dev_close(cnxk_mldev) != 0) - plt_err("Failed to close CN10K ML Device"); + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) { + if (cn10k_ml_dev_close(cnxk_mldev) != 0) + plt_err("Failed to close CN10K ML Device"); + } if (cnxk_mldev->index_map) rte_free(cnxk_mldev->index_map); @@ -876,10 +895,12 @@ cnxk_ml_dev_start(struct rte_ml_dev *dev) cnxk_mldev = dev->data->dev_private; - ret = cn10k_ml_dev_start(cnxk_mldev); - if (ret != 0) { - plt_err("Failed to start CN10K ML Device"); - return ret; + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) { + ret = cn10k_ml_dev_start(cnxk_mldev); + if (ret != 0) { + plt_err("Failed to start CN10K ML Device"); + return ret; + } } cnxk_mldev->state = ML_CNXK_DEV_STATE_STARTED; @@ -898,10 +919,12 @@ cnxk_ml_dev_stop(struct rte_ml_dev *dev) cnxk_mldev = dev->data->dev_private; - ret = cn10k_ml_dev_stop(cnxk_mldev); - if (ret != 0) { - plt_err("Failed to stop CN10K ML Device"); - return ret; + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) { + ret = cn10k_ml_dev_stop(cnxk_mldev); + if (ret != 0) { + plt_err("Failed to stop CN10K ML Device"); + return ret; + } } cnxk_mldev->state = ML_CNXK_DEV_STATE_CONFIGURED; @@ -928,7 +951,14 @@ cnxk_ml_dev_dump(struct rte_ml_dev *dev, FILE *fp) cnxk_ml_model_dump(cnxk_mldev, model, fp); } - return cn10k_ml_dev_dump(cnxk_mldev, fp); + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) + return cn10k_ml_dev_dump(cnxk_mldev, fp); +#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM + else + return mvtvm_ml_dev_dump(cnxk_mldev, fp); +#endif + + return 0; } static int @@ -941,6 +971,9 @@ cnxk_ml_dev_selftest(struct rte_ml_dev *dev) cnxk_mldev = dev->data->dev_private; + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_VDEV) + return -ENOTSUP; + return cn10k_ml_dev_selftest(cnxk_mldev); } @@ -1269,6 +1302,11 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u return -EINVAL; } + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_VDEV && type != ML_CNXK_MODEL_TYPE_TVM) { + plt_err("Unsupported model type"); + return -ENOTSUP; + } + /* Find model ID */ found = false; for (lcl_model_id = 0; lcl_model_id < dev->data->nb_models; lcl_model_id++) { @@ -1463,6 +1501,8 @@ cnxk_ml_model_params_update(struct rte_ml_dev *dev, uint16_t model_id, void *buf return -EINVAL; cnxk_mldev = dev->data->dev_private; + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_VDEV) + return -ENOTSUP; model = dev->data->models[model_id]; if (model == NULL) { diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build index 09a62b5c55a..f5989c5cafe 100644 --- a/drivers/ml/cnxk/meson.build +++ b/drivers/ml/cnxk/meson.build @@ -70,11 +70,13 @@ if enable_mvtvm dpdk_conf.set('RTE_MLDEV_CNXK_ENABLE_MVTVM', true) driver_sdk_headers += files( + 'mvtvm_ml_dev.h', 'mvtvm_ml_ops.h', 'mvtvm_ml_model.h', ) sources += files( + 'mvtvm_ml_dev.c', 'mvtvm_ml_ops.c', 'mvtvm_ml_model.c', ) diff --git a/drivers/ml/cnxk/mvtvm_ml_dev.c b/drivers/ml/cnxk/mvtvm_ml_dev.c new file mode 100644 index 00000000000..8ca0e959e35 --- /dev/null +++ b/drivers/ml/cnxk/mvtvm_ml_dev.c @@ -0,0 +1,198 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#include +#include +#include + +#include + +#include + +#include "mvtvm_ml_dev.h" + +#include "cnxk_ml_dev.h" + +#define MVTVM_ML_DEV_MAX_QPS "max_qps" +#define MVTVM_ML_DEV_CACHE_MODEL_DATA "cache_model_data" + +#define MVTVM_ML_DEV_MAX_QPS_DEFAULT 32 +#define CN10K_ML_DEV_CACHE_MODEL_DATA_DEFAULT 1 + +static const char *const valid_args[] = {MVTVM_ML_DEV_MAX_QPS, MVTVM_ML_DEV_CACHE_MODEL_DATA, NULL}; + +static int +parse_integer_arg(const char *key __rte_unused, const char *value, void *extra_args) +{ + int *i = (int *)extra_args; + + *i = atoi(value); + if (*i < 0) { + plt_err("Argument has to be positive."); + return -EINVAL; + } + + return 0; +} + +static int +parse_uint_arg(const char *key __rte_unused, const char *value, void *extra_args) +{ + int i; + char *end; + errno = 0; + + i = strtol(value, &end, 10); + if (*end != 0 || errno != 0 || i < 0) + return -EINVAL; + + *((uint32_t *)extra_args) = i; + + return 0; +} + +static int +mvtvm_mldev_parse_devargs(const char *args, struct mvtvm_ml_dev *mvtvm_mldev) +{ + bool cache_model_data_set = false; + struct rte_kvargs *kvlist = NULL; + bool max_qps_set = false; + int ret = 0; + + if (args == NULL) + goto check_args; + + kvlist = rte_kvargs_parse(args, valid_args); + if (kvlist == NULL) { + plt_err("Error parsing %s devargs\n", "MLDEV_NAME_MVTVM_PMD"); + return -EINVAL; + } + + if (rte_kvargs_count(kvlist, MVTVM_ML_DEV_MAX_QPS) == 1) { + ret = rte_kvargs_process(kvlist, MVTVM_ML_DEV_MAX_QPS, &parse_uint_arg, + &mvtvm_mldev->max_nb_qpairs); + if (ret < 0) { + plt_err("Error processing arguments, key = %s\n", MVTVM_ML_DEV_MAX_QPS); + ret = -EINVAL; + goto exit; + } + max_qps_set = true; + } + + if (rte_kvargs_count(kvlist, MVTVM_ML_DEV_CACHE_MODEL_DATA) == 1) { + ret = rte_kvargs_process(kvlist, MVTVM_ML_DEV_CACHE_MODEL_DATA, &parse_integer_arg, + &mvtvm_mldev->cache_model_data); + if (ret < 0) { + plt_err("Error processing arguments, key = %s\n", + MVTVM_ML_DEV_CACHE_MODEL_DATA); + ret = -EINVAL; + goto exit; + } + cache_model_data_set = true; + } + +check_args: + if (!max_qps_set) + mvtvm_mldev->max_nb_qpairs = MVTVM_ML_DEV_MAX_QPS_DEFAULT; + plt_ml_dbg("ML: %s = %u", MVTVM_ML_DEV_MAX_QPS, mvtvm_mldev->max_nb_qpairs); + + if (!cache_model_data_set) { + mvtvm_mldev->cache_model_data = CN10K_ML_DEV_CACHE_MODEL_DATA_DEFAULT; + } else { + if ((mvtvm_mldev->cache_model_data < 0) || (mvtvm_mldev->cache_model_data > 1)) { + plt_err("Invalid argument, %s = %d\n", MVTVM_ML_DEV_CACHE_MODEL_DATA, + mvtvm_mldev->cache_model_data); + ret = -EINVAL; + goto exit; + } + } + plt_ml_dbg("ML: %s = %d", MVTVM_ML_DEV_CACHE_MODEL_DATA, mvtvm_mldev->cache_model_data); + +exit: + if (kvlist) + rte_kvargs_free(kvlist); + + return ret; +} + +static int +mvtvm_ml_vdev_probe(struct rte_vdev_device *vdev) +{ + struct rte_ml_dev_pmd_init_params init_params; + struct mvtvm_ml_dev *mvtvm_mldev; + struct cnxk_ml_dev *cnxk_mldev; + struct rte_ml_dev *dev; + const char *input_args; + const char *name; + int ret = 0; + + if (cnxk_ml_dev_initialized == 1) { + plt_err("ML CNXK device already initialized!"); + plt_err("Cannot initialize MVTVM vdev"); + rte_exit(-EINVAL, "Invalid EAL arguments "); + } + + init_params = (struct rte_ml_dev_pmd_init_params){ + .socket_id = rte_socket_id(), .private_data_size = sizeof(struct cnxk_ml_dev)}; + + name = rte_vdev_device_name(vdev); + if (name == NULL) + return -EINVAL; + input_args = rte_vdev_device_args(vdev); + + dev = rte_ml_dev_pmd_create(name, &vdev->device, &init_params); + if (dev == NULL) { + ret = -EFAULT; + goto error_exit; + } + + cnxk_mldev = dev->data->dev_private; + cnxk_mldev->mldev = dev; + mvtvm_mldev = &cnxk_mldev->mvtvm_mldev; + mvtvm_mldev->vdev = vdev; + + ret = mvtvm_mldev_parse_devargs(input_args, mvtvm_mldev); + if (ret < 0) + goto error_exit; + + dev->dev_ops = &cnxk_ml_ops; + dev->enqueue_burst = NULL; + dev->dequeue_burst = NULL; + dev->op_error_get = NULL; + + cnxk_ml_dev_initialized = 1; + cnxk_mldev->type = CNXK_ML_DEV_TYPE_VDEV; + + return 0; + +error_exit: + plt_err("Could not create device: ml_mvtvm"); + + return ret; +} + +static int +mvtvm_ml_vdev_remove(struct rte_vdev_device *vdev) +{ + struct rte_ml_dev *dev; + const char *name; + + name = rte_vdev_device_name(vdev); + if (name == NULL) + return -EINVAL; + + dev = rte_ml_dev_pmd_get_named_dev(name); + if (dev == NULL) + return -ENODEV; + + return rte_ml_dev_pmd_destroy(dev); +} + +static struct rte_vdev_driver mvtvm_mldev_pmd = {.probe = mvtvm_ml_vdev_probe, + .remove = mvtvm_ml_vdev_remove}; + +RTE_PMD_REGISTER_VDEV(MLDEV_NAME_MVTVM_PMD, mvtvm_mldev_pmd); + +RTE_PMD_REGISTER_PARAM_STRING(MLDEV_NAME_MVTVM_PMD, + MVTVM_ML_DEV_MAX_QPS "=" MVTVM_ML_DEV_CACHE_MODEL_DATA "=<0|1>"); diff --git a/drivers/ml/cnxk/mvtvm_ml_dev.h b/drivers/ml/cnxk/mvtvm_ml_dev.h new file mode 100644 index 00000000000..6922c193372 --- /dev/null +++ b/drivers/ml/cnxk/mvtvm_ml_dev.h @@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Marvell. + */ + +#ifndef _MVTVM_ML_DEV_H_ +#define _MVTVM_ML_DEV_H_ + +#include + +/* Device status */ +extern int cnxk_ml_dev_initialized; + +/* CNXK Device ops */ +extern struct rte_ml_dev_ops cnxk_ml_ops; + +/* Marvell MVTVM ML PMD device name */ +#define MLDEV_NAME_MVTVM_PMD ml_mvtvm + +/* Maximum number of descriptors per queue-pair */ +#define ML_MVTVM_MAX_DESC_PER_QP 1024 + +/* Maximum number of inputs / outputs per model */ +#define ML_MVTVM_MAX_INPUT_OUTPUT 32 + +/* Maximum number of segments for IO data */ +#define ML_MVTVM_MAX_SEGMENTS 1 + +/* Device private data */ +struct mvtvm_ml_dev { + /* Virtual device */ + struct rte_vdev_device *vdev; + + /* Maximum number of queue pairs */ + uint16_t max_nb_qpairs; + + /* Enable / disable model data caching */ + int cache_model_data; +}; + +#endif /* _MVTVM_ML_DEV_H_ */ diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c index e8484b3bd92..dc9beb86e2e 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.c +++ b/drivers/ml/cnxk/mvtvm_ml_ops.c @@ -11,8 +11,7 @@ #include -#include "cn10k_ml_ops.h" - +#include "mvtvm_ml_dev.h" #include "mvtvm_ml_model.h" #include "mvtvm_ml_ops.h" @@ -29,6 +28,22 @@ mvtvm_ml_set_poll_addr(struct cnxk_ml_req *req) req->status = &req->mvtvm_req.status; } +int +mvtvm_ml_dev_info_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_dev_info *dev_info) +{ + struct mvtvm_ml_dev *mvtvm_mldev; + + mvtvm_mldev = &cnxk_mldev->mvtvm_mldev; + + dev_info->max_queue_pairs = mvtvm_mldev->max_nb_qpairs; + dev_info->max_desc = ML_MVTVM_MAX_DESC_PER_QP; + dev_info->max_io = ML_MVTVM_MAX_INPUT_OUTPUT; + dev_info->max_segments = ML_MVTVM_MAX_SEGMENTS; + dev_info->align_size = RTE_CACHE_LINE_SIZE; + + return 0; +} + int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf) { @@ -59,6 +74,15 @@ mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev) return ret; } +int +mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp) +{ + RTE_SET_USED(cnxk_mldev); + RTE_SET_USED(fp); + + return 0; +} + int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, struct cnxk_ml_model *model) @@ -169,6 +193,12 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params * else model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_HYBRID; + if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_VDEV && + model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_LLVM) { + plt_err("Unsupported model sub-type"); + return -ENOTSUP; + } + /* Set callback function array */ if (model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_LLVM) { callback = &model->mvtvm.cb; diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.h b/drivers/ml/cnxk/mvtvm_ml_ops.h index dba055c22e7..6cb8db92030 100644 --- a/drivers/ml/cnxk/mvtvm_ml_ops.h +++ b/drivers/ml/cnxk/mvtvm_ml_ops.h @@ -50,8 +50,10 @@ struct mvtvm_ml_req { struct mvtvm_ml_result result; }; +int mvtvm_ml_dev_info_get(struct cnxk_ml_dev *mldev, struct rte_ml_dev_info *dev_info); int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf); int mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev); +int mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp); int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params, struct cnxk_ml_model *model); int mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);