From patchwork Wed Feb 7 17:13:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hernan Vargas X-Patchwork-Id: 136503 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E86C543AA1; Wed, 7 Feb 2024 18:17:50 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 530B642E2D; Wed, 7 Feb 2024 18:17:20 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) by mails.dpdk.org (Postfix) with ESMTP id ED10E402E4 for ; Wed, 7 Feb 2024 18:17:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707326234; x=1738862234; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3f1AiyVU/ZGQmcTeKriCjLuzkBieCDdSIIgfHU8JIR8=; b=VYfAY8AxagCdhw0+yv/DnUAvvA4a8oet42ie6XYf0Ih6lNodEJ+wXj18 VGNTBQksPYG0qSFqrZXV9ylNycf0JW7K2vUoKd7oOdYkQSMhzz2sPVDsk 8C8ZjlsB+046jTFEyfgJdOOb7vdPwOm5P5ed5zg/jILwVed3MK7TIQg5n X/finmXCN286Ok0c64MQsrH63Crb1RgBRVN9knA9uKGig9m/jxA3VmI13 rSF+8JQ7s6+EN5q6qkhWMzU9WUZr56I1Jd1zcHx+qyi1DW/37Q/McQuf7 c4fVop5gloLe7zMNof1H56O2Ymgh2d9lqiB/ecSPZsnoNwhQjIZnn/b+r w==; X-IronPort-AV: E=McAfee;i="6600,9927,10977"; a="1185798" X-IronPort-AV: E=Sophos;i="6.05,251,1701158400"; d="scan'208";a="1185798" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Feb 2024 09:17:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,251,1701158400"; d="scan'208";a="1410901" Received: from unknown (HELO csl-npg-qt0.la.intel.com) ([10.233.181.103]) by fmviesa007.fm.intel.com with ESMTP; 07 Feb 2024 09:17:12 -0800 From: Hernan Vargas To: dev@dpdk.org, gakhil@marvell.com, trix@redhat.com, maxime.coquelin@redhat.com Cc: nicolas.chautru@intel.com, qi.z.zhang@intel.com, Hernan Vargas Subject: [PATCH v6 5/6] baseband/fpga_5gnr_fec: add AGX100 support Date: Wed, 7 Feb 2024 09:13:49 -0800 Message-Id: <20240207171350.242156-6-hernan.vargas@intel.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20240207171350.242156-1-hernan.vargas@intel.com> References: <20240207171350.242156-1-hernan.vargas@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for new FPGA variant AGX100 (on Arrow Creek N6000). Signed-off-by: Hernan Vargas Reviewed-by: Maxime Coquelin --- doc/guides/bbdevs/fpga_5gnr_fec.rst | 69 +- drivers/baseband/fpga_5gnr_fec/agx100_pmd.h | 273 ++++ .../baseband/fpga_5gnr_fec/fpga_5gnr_fec.h | 10 +- .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 1199 +++++++++++++++-- drivers/baseband/fpga_5gnr_fec/vc_5gnr_pmd.h | 1 - 5 files changed, 1438 insertions(+), 114 deletions(-) create mode 100644 drivers/baseband/fpga_5gnr_fec/agx100_pmd.h diff --git a/doc/guides/bbdevs/fpga_5gnr_fec.rst b/doc/guides/bbdevs/fpga_5gnr_fec.rst index 99fc936829a8..1ae192a86b25 100644 --- a/doc/guides/bbdevs/fpga_5gnr_fec.rst +++ b/doc/guides/bbdevs/fpga_5gnr_fec.rst @@ -6,12 +6,13 @@ Intel(R) FPGA 5GNR FEC Poll Mode Driver The BBDEV FPGA 5GNR FEC poll mode driver (PMD) supports an FPGA implementation of a VRAN LDPC Encode / Decode 5GNR wireless acceleration function, using Intel's PCI-e and FPGA -based Vista Creek device. +based Vista Creek (N3000, referred to as VC_5GNR in the code) as well as Arrow Creek (N6000, +referred to as AGX100 in the code). Features -------- -FPGA 5GNR FEC PMD supports the following features: +FPGA 5GNR FEC PMD supports the following BBDEV capabilities: - LDPC Encode in the DL - LDPC Decode in the UL @@ -67,10 +68,18 @@ Initialization When the device first powers up, its PCI Physical Functions (PF) can be listed through this command: +Vista Creek (N3000) + .. code-block:: console sudo lspci -vd8086:0d8f +Arrow Creek (N6000) + +.. code-block:: console + + sudo lspci -vd8086:5799 + The physical and virtual functions are compatible with Linux UIO drivers: ``vfio_pci`` and ``igb_uio``. However, in order to work the FPGA 5GNR FEC device firstly needs to be bound to one of these linux drivers through DPDK. @@ -78,6 +87,7 @@ to be bound to one of these linux drivers through DPDK. For more details on how to bind the PF device and create VF devices, see :ref:`linux_gsg_binding_kernel`. + Configure the VFs through PF ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -110,12 +120,12 @@ parameters defined in ``rte_fpga_5gnr_fec_conf`` structure: - ``vf_*l_queues_number``: defines the hardware queue mapping for every VF. -- ``*l_bandwidth``: in case of congestion on PCIe interface. The device - allocates different bandwidth to UL and DL. The weight is configured by this - setting. The unit of weight is 3 code blocks. For example, if the code block - cbps (code block per second) ratio between UL and DL is 12:1, then the - configuration value should be set to 36:3. The schedule algorithm is based - on code block regardless the length of each block. +- ``*l_bandwidth``: Only used for the Vista Creek schedule algorithm in case of + congestion on PCIe interface. The device allocates different bandwidth to UL + and DL. The weight is configured by this setting. The unit of weight is 3 code + blocks. For example, if the code block cbps (code block per second) ratio between + UL and DL is 12:1, then the configuration value should be set to 36:3. + The schedule algorithm is based on code block regardless the length of each block. - ``*l_load_balance``: hardware queues are load-balanced in a round-robin fashion. Queues get filled first-in first-out until they reach a pre-defined @@ -159,8 +169,38 @@ Test Application BBDEV provides a test application, ``test-bbdev.py`` and range of test data for testing the functionality of the device, depending on the device's capabilities. -For more details on how to use the test application, -see :ref:`test_bbdev_application`. +.. code-block:: console + + "-p", "--testapp-path": specifies path to the bbdev test app. + "-e", "--eal-params" : EAL arguments which are passed to the test app. + "-t", "--timeout" : Timeout in seconds (default=300). + "-c", "--test-cases" : Defines test cases to run. Run all if not specified. + "-v", "--test-vector" : Test vector path (default=dpdk_path+/app/test-bbdev/test_vectors/bbdev_null.data). + "-n", "--num-ops" : Number of operations to process on device (default=32). + "-b", "--burst-size" : Operations enqueue/dequeue burst size (default=32). + "-l", "--num-lcores" : Number of lcores to run (default=16). + "-i", "--init-device" : Initialise PF device with default values. + + +To execute the test application tool using simple decode or encode data, +type one of the following: + +.. code-block:: console + + ./test-bbdev.py -c validation -n 64 -b 1 -v ./ldpc_dec_default.data + ./test-bbdev.py -c validation -n 64 -b 1 -v ./ldpc_enc_default.data + + +The test application ``test-bbdev.py``, supports the ability to configure the PF device with +a default set of values, if the "-i" or "- -init-device" option is included. The default values +are defined in test_bbdev_perf.c as: + +- VF_UL_QUEUE_VALUE 4 +- VF_DL_QUEUE_VALUE 4 +- UL_BANDWIDTH 3 +- DL_BANDWIDTH 3 +- UL_LOAD_BALANCE 128 +- DL_LOAD_BALANCE 128 Test Vectors @@ -184,7 +224,16 @@ See for more details: https://github.com/intel/pf-bb-config Specifically for the BBDEV FPGA 5GNR FEC PMD, the command below can be used: +Vista Creek (N3000) + .. code-block:: console ./pf_bb_config FPGA_5GNR -c fpga_5gnr/fpga_5gnr_config_vf.cfg ./test-bbdev.py -e="-c 0xff0 -a${VF_PCI_ADDR}" -c validation -n 64 -b 32 -l 1 -v ./ldpc_dec_default.data + +Arrow Creek (N6000) + +.. code-block:: console + + ./pf_bb_config AGX100 -c agx100/agx100_config_1vf.cfg + ./test-bbdev.py -e="-c 0xff0 -a${VF_PCI_ADDR}" -c validation -n 64 -b 32 -l 1 -v ./ldpc_dec_default.data diff --git a/drivers/baseband/fpga_5gnr_fec/agx100_pmd.h b/drivers/baseband/fpga_5gnr_fec/agx100_pmd.h new file mode 100644 index 000000000000..5e562376c966 --- /dev/null +++ b/drivers/baseband/fpga_5gnr_fec/agx100_pmd.h @@ -0,0 +1,273 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Intel Corporation + */ + +#ifndef _AGX100_H_ +#define _AGX100_H_ + +#include +#include + +/* AGX100 PCI vendor & device IDs. */ +#define AGX100_VENDOR_ID (0x8086) +#define AGX100_PF_DEVICE_ID (0x5799) +#define AGX100_VF_DEVICE_ID (0x579A) + +/* Maximum number of possible queues supported on device. */ +#define AGX100_MAXIMUM_QUEUES_SUPPORTED (64) + +/* AGX100 Ring size is in 256 bits (64 bytes) units. */ +#define AGX100_RING_DESC_LEN_UNIT_BYTES (64) + +/* Align DMA descriptors to 256 bytes - cache-aligned. */ +#define AGX100_RING_DESC_ENTRY_LENGTH (8) + +/* AGX100 Register mapping on BAR0. */ +enum { + AGX100_FLR_TIME_OUT = 0x0000000E, /* len: 2B. */ + AGX100_QUEUE_MAP = 0x00000100 /* len: 256B. */ +}; + +/* AGX100 DESCRIPTOR ERROR. */ +enum { + AGX100_DESC_ERR_NO_ERR = 0x00, /**< 4'b0000 2'b00. */ + AGX100_DESC_ERR_E_NOT_LEGAL = 0x11, /**< 4'b0001 2'b01. */ + AGX100_DESC_ERR_K_P_OUT_OF_RANGE = 0x21, /**< 4'b0010 2'b01. */ + AGX100_DESC_ERR_NCB_OUT_OF_RANGE = 0x31, /**< 4'b0011 2'b01. */ + AGX100_DESC_ERR_Z_C_NOT_LEGAL = 0x41, /**< 4'b0100 2'b01. */ + AGX100_DESC_ERR_DESC_INDEX_ERR = 0x03, /**< 4'b0000 2'b11. */ + AGX100_DESC_ERR_HARQ_INPUT_LEN_A = 0x51, /**< 4'b0101 2'b01. */ + AGX100_DESC_ERR_HARQ_INPUT_LEN_B = 0x61, /**< 4'b0110 2'b01. */ + AGX100_DESC_ERR_HBSTORE_OFFSET_ERR = 0x71, /**< 4'b0111 2'b01. */ + AGX100_DESC_ERR_TB_CBG_ERR = 0x81, /**< 4'b1000 2'b01. */ + AGX100_DESC_ERR_CBG_OUT_OF_RANGE = 0x91, /**< 4'b1001 2'b01. */ + AGX100_DESC_ERR_CW_RM_NOT_LEGAL = 0xA1, /**< 4'b1010 2'b01. */ + AGX100_DESC_ERR_UNSUPPORTED_REQ = 0x12, /**< 4'b0000 2'b10. */ + AGX100_DESC_ERR_RESERVED = 0x22, /**< 4'b0010 2'b10. */ + AGX100_DESC_ERR_DESC_ABORT = 0x42, /**< 4'b0100 2'b10. */ + AGX100_DESC_ERR_DESC_READ_TLP_POISONED = 0x82 /**< 4'b1000 2'b10. */ +}; + +/* AGX100 TX Slice Descriptor. */ +struct __rte_packed agx100_input_slice_desc { + uint32_t input_start_addr_lo; + uint32_t input_start_addr_hi; + uint32_t input_slice_length:21, + rsrvd0:9, + end_of_pkt:1, + start_of_pkt:1; + uint32_t input_slice_time_stamp:31, + input_c:1; +}; + +/* AGX100 RX Slice Descriptor. */ +struct __rte_packed agx100_output_slice_desc { + uint32_t output_start_addr_lo; + uint32_t output_start_addr_hi; + uint32_t output_slice_length:21, + rsrvd0:9, + end_of_pkt:1, + start_of_pkt:1; + uint32_t output_slice_time_stamp:31, + output_c:1; +}; + +/* AGX100 DL DMA Encoding Request Descriptor. */ +struct __rte_packed agx100_dma_enc_desc { + uint32_t done:1, /**< 0: not completed 1: completed. */ + rsrvd0:17, + error_msg:2, + error_code:4, + rsrvd1:8; + uint32_t ncb:16, /**< Limited circular buffer size. */ + bg_idx:1, /**< Base Graph 0: BG1 1: BG2.*/ + qm_idx:3, /**< 0: BPSK; 1: QPSK; 2: 16QAM; 3: 64QAM; 4: 256QAM. */ + zc:9, /**< Lifting size. */ + rv:2, /**< Redundancy version number. */ + int_en:1; /**< Interrupt enable. */ + uint32_t max_cbg:4, /**< Only valid when workload is TB or CBGs. */ + rsrvd2:4, + cbgti:8, /**< CBG bitmap. */ + rsrvd3:4, + cbgs:1, /**< 0: TB or CB 1: CBGs. */ + desc_idx:11; /**< Sequence number of the descriptor. */ + uint32_t ca:10, /**< Code block number with Ea in TB or CBG. */ + c:10, /**< Total code block number in TB or CBG. */ + rsrvd4:2, + num_null:10; /**< Number of null bits. */ + uint32_t ea:21, /**< Value of E when workload is CB. */ + rsrvd5:11; + uint32_t eb:21, /**< Only valid when workload is TB or CBGs. */ + rsrvd6:11; + uint32_t k_:16, /**< Code block length without null bits. */ + rsrvd7:8, + en_slice_ts:1, /**< Enable slice descriptor timestamp. */ + en_host_ts:1, /**< Enable host descriptor timestamp. */ + en_cb_wr_status:1, /**< Enable code block write back status. */ + en_output_sg:1, /**< Enable RX scatter-gather. */ + en_input_sg:1, /**< Enable TX scatter-gather. */ + tb_cb:1, /**< 2'b10: the descriptor is for a TrBlk. + * 2'b00: the descriptor is for a CBlk. + * 2'b11 or 01: the descriptor is for a CBGs. + */ + crc_en:1, /**< 1: CB CRC enabled 0: CB CRC disabled. + * Only valid when workload is CB or CBGs. + */ + rsrvd8:1; + uint32_t rsrvd9; + union { + uint32_t input_slice_table_addr_lo; /** #include +#include "agx100_pmd.h" #include "vc_5gnr_pmd.h" /* Helper macro for logging */ @@ -133,12 +134,19 @@ struct fpga_5gnr_fec_device { bool pf_device; /** Maximum number of possible queues for this device. */ uint8_t total_num_queues; + /** FPGA Variant. VC_5GNR_FPGA_VARIANT = 0; AGX100_FPGA_VARIANT = 1. */ + uint8_t fpga_variant; }; /** Structure associated with each queue. */ struct __rte_cache_aligned fpga_5gnr_queue { struct fpga_5gnr_ring_ctrl_reg ring_ctrl_reg; /**< Ring Control Register */ - union vc_5gnr_dma_desc *vc_5gnr_ring_addr; /**< Virtual address of VC 5GNR software ring. */ + union { + /** Virtual address of VC 5GNR software ring. */ + union vc_5gnr_dma_desc *vc_5gnr_ring_addr; + /** Virtual address of AGX100 software ring. */ + union agx100_dma_desc *agx100_ring_addr; + }; uint64_t *ring_head_addr; /* Virtual address of completion_head */ uint64_t shadow_completion_head; /* Shadow completion head value */ uint16_t head_free_desc; /* Ring head */ diff --git a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c index 3fb505775f61..6beb10e546c4 100644 --- a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c +++ b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c @@ -18,8 +18,8 @@ #include #include -#include "fpga_5gnr_fec.h" #include "rte_pmd_fpga_5gnr_fec.h" +#include "fpga_5gnr_fec.h" #ifdef RTE_LIBRTE_BBDEV_DEBUG RTE_LOG_REGISTER_DEFAULT(fpga_5gnr_fec_logtype, DEBUG); @@ -71,24 +71,28 @@ print_ring_reg_debug_info(void *mmio_base, uint32_t offset) /* Read Static Register of Vista Creek device. */ static inline void -print_static_reg_debug_info(void *mmio_base) +print_static_reg_debug_info(void *mmio_base, uint8_t fpga_variant) { - uint16_t config = fpga_5gnr_reg_read_16(mmio_base, VC_5GNR_CONFIGURATION); - uint8_t qmap_done = fpga_5gnr_reg_read_8(mmio_base, - FPGA_5GNR_FEC_QUEUE_PF_VF_MAP_DONE); - uint16_t lb_factor = fpga_5gnr_reg_read_16(mmio_base, - FPGA_5GNR_FEC_LOAD_BALANCE_FACTOR); - uint16_t ring_desc_len = fpga_5gnr_reg_read_16(mmio_base, - FPGA_5GNR_FEC_RING_DESC_LEN); - - rte_bbdev_log_debug("UL.DL Weights = %u.%u", - ((uint8_t)config), ((uint8_t)(config >> 8))); + uint16_t config; + uint8_t qmap_done = fpga_5gnr_reg_read_8(mmio_base, FPGA_5GNR_FEC_QUEUE_PF_VF_MAP_DONE); + uint16_t lb_factor = fpga_5gnr_reg_read_16(mmio_base, FPGA_5GNR_FEC_LOAD_BALANCE_FACTOR); + uint16_t ring_desc_len = fpga_5gnr_reg_read_16(mmio_base, FPGA_5GNR_FEC_RING_DESC_LEN); + if (fpga_variant == VC_5GNR_FPGA_VARIANT) + config = fpga_5gnr_reg_read_16(mmio_base, VC_5GNR_CONFIGURATION); + + if (fpga_variant == VC_5GNR_FPGA_VARIANT) + rte_bbdev_log_debug("UL.DL Weights = %u.%u", + ((uint8_t)config), ((uint8_t)(config >> 8))); rte_bbdev_log_debug("UL.DL Load Balance = %u.%u", ((uint8_t)lb_factor), ((uint8_t)(lb_factor >> 8))); rte_bbdev_log_debug("Queue-PF/VF Mapping Table = %s", (qmap_done > 0) ? "READY" : "NOT-READY"); - rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", - ring_desc_len*VC_5GNR_RING_DESC_LEN_UNIT_BYTES); + if (fpga_variant == VC_5GNR_FPGA_VARIANT) + rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", + ring_desc_len * VC_5GNR_RING_DESC_LEN_UNIT_BYTES); + else + rte_bbdev_log_debug("Ring Descriptor Size = %u bytes", + ring_desc_len * AGX100_RING_DESC_LEN_UNIT_BYTES); } /* Print decode DMA Descriptor of Vista Creek Decoder device. */ @@ -142,6 +146,108 @@ vc_5gnr_print_dma_dec_desc_debug_info(union vc_5gnr_dma_desc *desc) word[4], word[5], word[6], word[7]); } +/* Print decode DMA Descriptor of AGX100 Decoder device. */ +static void +agx100_print_dma_dec_desc_debug_info(union agx100_dma_desc *desc) +{ + rte_bbdev_log_debug("DMA response desc %p\n" + "\t-- done(%"PRIu32") | tb_crc_pass(%"PRIu32") | cb_crc_all_pass(%"PRIu32")" + " | cb_all_et_pass(%"PRIu32") | max_iter_ret(%"PRIu32") |" + "cgb_crc_bitmap(%"PRIu32") | error_msg(%"PRIu32") | error_code(%"PRIu32") |" + "et_dis (%"PRIu32") | harq_in_en(%"PRIu32") | max_iter(%"PRIu32")\n" + "\t-- ncb(%"PRIu32") | bg_idx (%"PRIu32") | qm_idx (%"PRIu32")" + "| zc(%"PRIu32") | rv(%"PRIu32") | int_en(%"PRIu32")\n" + "\t-- max_cbg(%"PRIu32") | cbgti(%"PRIu32") | cbgfi(%"PRIu32") |" + "cbgs(%"PRIu32") | desc_idx(%"PRIu32")\n" + "\t-- ca(%"PRIu32") | c(%"PRIu32") | llr_pckg(%"PRIu32") |" + "syndrome_check_mode(%"PRIu32") | num_null(%"PRIu32")\n" + "\t-- ea(%"PRIu32") | eba(%"PRIu32")\n" + "\t-- hbstore_offset_out(%"PRIu32")\n" + "\t-- hbstore_offset_in(%"PRIu32") | en_slice_ts(%"PRIu32") |" + "en_host_ts(%"PRIu32") | en_cb_wr_status(%"PRIu32")" + " | en_output_sg(%"PRIu32") | en_input_sg(%"PRIu32") | tb_cb(%"PRIu32")" + " | crc24b_ind(%"PRIu32")| drop_crc24b(%"PRIu32")\n" + "\t-- harq_input_length_a(%"PRIu32") | harq_input_length_b(%"PRIu32")\n" + "\t-- input_slice_table_addr_lo(%"PRIu32")" + " | input_start_addr_lo(%"PRIu32")\n" + "\t-- input_slice_table_addr_hi(%"PRIu32")" + " | input_start_addr_hi(%"PRIu32")\n" + "\t-- input_slice_num(%"PRIu32") | input_length(%"PRIu32")\n" + "\t-- output_slice_table_addr_lo(%"PRIu32")" + " | output_start_addr_lo(%"PRIu32")\n" + "\t-- output_slice_table_addr_hi(%"PRIu32")" + " | output_start_addr_hi(%"PRIu32")\n" + "\t-- output_slice_num(%"PRIu32") | output_length(%"PRIu32")\n" + "\t-- enqueue_timestamp(%"PRIu32")\n" + "\t-- completion_timestamp(%"PRIu32")\n", + desc, + (uint32_t)desc->dec_req.done, + (uint32_t)desc->dec_req.tb_crc_pass, + (uint32_t)desc->dec_req.cb_crc_all_pass, + (uint32_t)desc->dec_req.cb_all_et_pass, + (uint32_t)desc->dec_req.max_iter_ret, + (uint32_t)desc->dec_req.cgb_crc_bitmap, + (uint32_t)desc->dec_req.error_msg, + (uint32_t)desc->dec_req.error_code, + (uint32_t)desc->dec_req.et_dis, + (uint32_t)desc->dec_req.harq_in_en, + (uint32_t)desc->dec_req.max_iter, + (uint32_t)desc->dec_req.ncb, + (uint32_t)desc->dec_req.bg_idx, + (uint32_t)desc->dec_req.qm_idx, + (uint32_t)desc->dec_req.zc, + (uint32_t)desc->dec_req.rv, + (uint32_t)desc->dec_req.int_en, + (uint32_t)desc->dec_req.max_cbg, + (uint32_t)desc->dec_req.cbgti, + (uint32_t)desc->dec_req.cbgfi, + (uint32_t)desc->dec_req.cbgs, + (uint32_t)desc->dec_req.desc_idx, + (uint32_t)desc->dec_req.ca, + (uint32_t)desc->dec_req.c, + (uint32_t)desc->dec_req.llr_pckg, + (uint32_t)desc->dec_req.syndrome_check_mode, + (uint32_t)desc->dec_req.num_null, + (uint32_t)desc->dec_req.ea, + (uint32_t)desc->dec_req.eba, + (uint32_t)desc->dec_req.hbstore_offset_out, + (uint32_t)desc->dec_req.hbstore_offset_in, + (uint32_t)desc->dec_req.en_slice_ts, + (uint32_t)desc->dec_req.en_host_ts, + (uint32_t)desc->dec_req.en_cb_wr_status, + (uint32_t)desc->dec_req.en_output_sg, + (uint32_t)desc->dec_req.en_input_sg, + (uint32_t)desc->dec_req.tb_cb, + (uint32_t)desc->dec_req.crc24b_ind, + (uint32_t)desc->dec_req.drop_crc24b, + (uint32_t)desc->dec_req.harq_input_length_a, + (uint32_t)desc->dec_req.harq_input_length_b, + (uint32_t)desc->dec_req.input_slice_table_addr_lo, + (uint32_t)desc->dec_req.input_start_addr_lo, + (uint32_t)desc->dec_req.input_slice_table_addr_hi, + (uint32_t)desc->dec_req.input_start_addr_hi, + (uint32_t)desc->dec_req.input_slice_num, + (uint32_t)desc->dec_req.input_length, + (uint32_t)desc->dec_req.output_slice_table_addr_lo, + (uint32_t)desc->dec_req.output_start_addr_lo, + (uint32_t)desc->dec_req.output_slice_table_addr_hi, + (uint32_t)desc->dec_req.output_start_addr_hi, + (uint32_t)desc->dec_req.output_slice_num, + (uint32_t)desc->dec_req.output_length, + (uint32_t)desc->dec_req.enqueue_timestamp, + (uint32_t)desc->dec_req.completion_timestamp); + + uint32_t *word = (uint32_t *) desc; + rte_bbdev_log_debug("%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n" + "%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n" + "%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n" + "%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n", + word[0], word[1], word[2], word[3], + word[4], word[5], word[6], word[7], + word[8], word[9], word[10], word[11], + word[12], word[13], word[14], word[15]); +} + /* Print decode DMA Descriptor of Vista Creek encoder device. */ static void vc_5gnr_print_dma_enc_desc_debug_info(union vc_5gnr_dma_desc *desc) @@ -175,8 +281,102 @@ vc_5gnr_print_dma_enc_desc_debug_info(union vc_5gnr_dma_desc *desc) word[4], word[5], word[6], word[7]); } +/* Print decode DMA Descriptor of AGX100 encoder device. */ +static void +agx100_print_dma_enc_desc_debug_info(union agx100_dma_desc *desc) +{ + rte_bbdev_log_debug("DMA response desc %p\n" + "\t-- done(%"PRIu32") | error_msg(%"PRIu32") | error_code(%"PRIu32")\n" + "\t-- ncb(%"PRIu32") | bg_idx (%"PRIu32") | qm_idx (%"PRIu32")" + "| zc(%"PRIu32") | rv(%"PRIu32") | int_en(%"PRIu32")\n" + "\t-- max_cbg(%"PRIu32") | cbgti(%"PRIu32") | cbgs(%"PRIu32") | " + "desc_idx(%"PRIu32")\n" + "\t-- ca(%"PRIu32") | c(%"PRIu32") | num_null(%"PRIu32")\n" + "\t-- ea(%"PRIu32")\n" + "\t-- eb(%"PRIu32")\n" + "\t-- k_(%"PRIu32") | en_slice_ts(%"PRIu32") | en_host_ts(%"PRIu32") | " + "en_cb_wr_status(%"PRIu32") | en_output_sg(%"PRIu32") | " + "en_input_sg(%"PRIu32") | tb_cb(%"PRIu32") | crc_en(%"PRIu32")\n" + "\t-- input_slice_table_addr_lo(%"PRIu32")" + " | input_start_addr_lo(%"PRIu32")\n" + "\t-- input_slice_table_addr_hi(%"PRIu32")" + " | input_start_addr_hi(%"PRIu32")\n" + "\t-- input_slice_num(%"PRIu32") | input_length(%"PRIu32")\n" + "\t-- output_slice_table_addr_lo(%"PRIu32")" + " | output_start_addr_lo(%"PRIu32")\n" + "\t-- output_slice_table_addr_hi(%"PRIu32")" + " | output_start_addr_hi(%"PRIu32")\n" + "\t-- output_slice_num(%"PRIu32") | output_length(%"PRIu32")\n" + "\t-- enqueue_timestamp(%"PRIu32")\n" + "\t-- completion_timestamp(%"PRIu32")\n", + desc, + (uint32_t)desc->enc_req.done, + (uint32_t)desc->enc_req.error_msg, + (uint32_t)desc->enc_req.error_code, + (uint32_t)desc->enc_req.ncb, + (uint32_t)desc->enc_req.bg_idx, + (uint32_t)desc->enc_req.qm_idx, + (uint32_t)desc->enc_req.zc, + (uint32_t)desc->enc_req.rv, + (uint32_t)desc->enc_req.int_en, + (uint32_t)desc->enc_req.max_cbg, + (uint32_t)desc->enc_req.cbgti, + (uint32_t)desc->enc_req.cbgs, + (uint32_t)desc->enc_req.desc_idx, + (uint32_t)desc->enc_req.ca, + (uint32_t)desc->enc_req.c, + (uint32_t)desc->enc_req.num_null, + (uint32_t)desc->enc_req.ea, + (uint32_t)desc->enc_req.eb, + (uint32_t)desc->enc_req.k_, + (uint32_t)desc->enc_req.en_slice_ts, + (uint32_t)desc->enc_req.en_host_ts, + (uint32_t)desc->enc_req.en_cb_wr_status, + (uint32_t)desc->enc_req.en_output_sg, + (uint32_t)desc->enc_req.en_input_sg, + (uint32_t)desc->enc_req.tb_cb, + (uint32_t)desc->enc_req.crc_en, + (uint32_t)desc->enc_req.input_slice_table_addr_lo, + (uint32_t)desc->enc_req.input_start_addr_lo, + (uint32_t)desc->enc_req.input_slice_table_addr_hi, + (uint32_t)desc->enc_req.input_start_addr_hi, + (uint32_t)desc->enc_req.input_slice_num, + (uint32_t)desc->enc_req.input_length, + (uint32_t)desc->enc_req.output_slice_table_addr_lo, + (uint32_t)desc->enc_req.output_start_addr_lo, + (uint32_t)desc->enc_req.output_slice_table_addr_hi, + (uint32_t)desc->enc_req.output_start_addr_hi, + (uint32_t)desc->enc_req.output_slice_num, + (uint32_t)desc->enc_req.output_length, + (uint32_t)desc->enc_req.enqueue_timestamp, + (uint32_t)desc->enc_req.completion_timestamp); + + uint32_t *word = (uint32_t *) desc; + rte_bbdev_log_debug("%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n" + "%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n" + "%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n" + "%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n%08"PRIx32"\n", + word[0], word[1], word[2], word[3], + word[4], word[5], word[6], word[7], + word[8], word[9], word[10], word[11], + word[12], word[13], word[14], word[15]); +} + #endif +/** + * Helper function that returns queue ID if queue is valid + * or FPGA_5GNR_INVALID_HW_QUEUE_ID otherwise. + */ +static inline uint32_t +fpga_5gnr_get_queue_map(struct fpga_5gnr_fec_device *d, uint32_t q_id) +{ + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) + return fpga_5gnr_reg_read_32(d->mmio_base, VC_5GNR_QUEUE_MAP + (q_id << 2)); + else + return fpga_5gnr_reg_read_32(d->mmio_base, AGX100_QUEUE_MAP + (q_id << 2)); +} + static int fpga_5gnr_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id) { @@ -204,8 +404,7 @@ fpga_5gnr_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id * FPGA_5GNR_INVALID_HW_QUEUE_ID is returned. */ for (q_id = 0; q_id < d->total_num_queues; ++q_id) { - uint32_t hw_q_id = fpga_5gnr_reg_read_32(d->mmio_base, - VC_5GNR_QUEUE_MAP + (q_id << 2)); + uint32_t hw_q_id = fpga_5gnr_get_queue_map(d, q_id); rte_bbdev_log_debug("%s: queue ID: %u, registry queue ID: %u", dev->device->name, q_id, hw_q_id); @@ -231,8 +430,10 @@ fpga_5gnr_setup_queues(struct rte_bbdev *dev, uint16_t num_queues, int socket_id dev->device->name, num_queues, hw_q_num); return -EINVAL; } - - ring_size = FPGA_5GNR_RING_MAX_SIZE * sizeof(struct vc_5gnr_dma_dec_desc); + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) + ring_size = FPGA_5GNR_RING_MAX_SIZE * sizeof(struct vc_5gnr_dma_dec_desc); + else + ring_size = FPGA_5GNR_RING_MAX_SIZE * sizeof(struct agx100_dma_dec_desc); /* Enforce 32 byte alignment */ RTE_BUILD_BUG_ON((RTE_CACHE_LINE_SIZE % 32) != 0); @@ -293,7 +494,7 @@ fpga_5gnr_dev_info_get(struct rte_bbdev *dev, struct rte_bbdev_driver_info *dev_ struct fpga_5gnr_fec_device *d = dev->data->dev_private; uint32_t q_id = 0; - static const struct rte_bbdev_op_cap bbdev_capabilities[] = { + static const struct rte_bbdev_op_cap vc_5gnr_bbdev_capabilities[] = { { .type = RTE_BBDEV_OP_LDPC_ENC, .cap.ldpc_enc = { @@ -333,6 +534,44 @@ fpga_5gnr_dev_info_get(struct rte_bbdev *dev, struct rte_bbdev_driver_info *dev_ RTE_BBDEV_END_OF_CAPABILITIES_LIST() }; + static const struct rte_bbdev_op_cap agx100_bbdev_capabilities[] = { + { + .type = RTE_BBDEV_OP_LDPC_ENC, + .cap.ldpc_enc = { + .capability_flags = + RTE_BBDEV_LDPC_RATE_MATCH | + RTE_BBDEV_LDPC_CRC_24B_ATTACH, + .num_buffers_src = + RTE_BBDEV_LDPC_MAX_CODE_BLOCKS, + .num_buffers_dst = + RTE_BBDEV_LDPC_MAX_CODE_BLOCKS, + } + }, + { + .type = RTE_BBDEV_OP_LDPC_DEC, + .cap.ldpc_dec = { + .capability_flags = + RTE_BBDEV_LDPC_CRC_TYPE_24B_CHECK | + RTE_BBDEV_LDPC_CRC_TYPE_24B_DROP | + RTE_BBDEV_LDPC_HQ_COMBINE_IN_ENABLE | + RTE_BBDEV_LDPC_HQ_COMBINE_OUT_ENABLE | + RTE_BBDEV_LDPC_ITERATION_STOP_ENABLE | + RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_IN_ENABLE | + RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_OUT_ENABLE | + RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_LOOPBACK | + RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_FILLERS, + .llr_size = 6, + .llr_decimals = 2, + .num_buffers_src = + RTE_BBDEV_LDPC_MAX_CODE_BLOCKS, + .num_buffers_hard_out = + RTE_BBDEV_LDPC_MAX_CODE_BLOCKS, + .num_buffers_soft_out = 0, + } + }, + RTE_BBDEV_END_OF_CAPABILITIES_LIST() + }; + /* Check the HARQ DDR size available */ uint8_t timeout_counter = 0; uint32_t harq_buf_ready = fpga_5gnr_reg_read_32(d->mmio_base, @@ -357,10 +596,16 @@ fpga_5gnr_dev_info_get(struct rte_bbdev *dev, struct rte_bbdev_driver_info *dev_ dev_info->driver_name = dev->device->driver->name; dev_info->queue_size_lim = FPGA_5GNR_RING_MAX_SIZE; dev_info->hardware_accelerated = true; - dev_info->min_alignment = 64; - dev_info->harq_buffer_size = (harq_buf_size >> 10) + 1; + dev_info->min_alignment = 1; + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) + dev_info->harq_buffer_size = (harq_buf_size >> 10) + 1; + else + dev_info->harq_buffer_size = harq_buf_size << 10; dev_info->default_queue_conf = default_queue_conf; - dev_info->capabilities = bbdev_capabilities; + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) + dev_info->capabilities = vc_5gnr_bbdev_capabilities; + else + dev_info->capabilities = agx100_bbdev_capabilities; dev_info->cpu_flag_reqs = NULL; dev_info->data_endianness = RTE_LITTLE_ENDIAN; dev_info->device_status = RTE_BBDEV_DEV_NOT_SUPPORTED; @@ -368,8 +613,8 @@ fpga_5gnr_dev_info_get(struct rte_bbdev *dev, struct rte_bbdev_driver_info *dev_ /* Calculates number of queues assigned to device */ dev_info->max_num_queues = 0; for (q_id = 0; q_id < d->total_num_queues; ++q_id) { - uint32_t hw_q_id = fpga_5gnr_reg_read_32(d->mmio_base, - VC_5GNR_QUEUE_MAP + (q_id << 2)); + uint32_t hw_q_id = fpga_5gnr_get_queue_map(d, q_id); + if (hw_q_id != FPGA_5GNR_INVALID_HW_QUEUE_ID) dev_info->max_num_queues++; } @@ -445,7 +690,11 @@ fpga_5gnr_queue_setup(struct rte_bbdev *dev, uint16_t queue_id, q->q_idx = q_idx; /* Set ring_base_addr */ - q->vc_5gnr_ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) + q->vc_5gnr_ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + else + q->agx100_ring_addr = RTE_PTR_ADD(d->sw_rings, (d->sw_ring_size * queue_id)); + q->ring_ctrl_reg.ring_base_addr = d->sw_rings_phys + (d->sw_ring_size * queue_id); /* Allocate memory for Completion Head variable*/ @@ -781,6 +1030,48 @@ static const struct rte_bbdev_ops fpga_5gnr_ops = { .queue_intr_disable = fpga_5gnr_queue_intr_disable }; +/* Provide the descriptor index on a given queue */ +static inline uint16_t +fpga_5gnr_desc_idx(struct fpga_5gnr_queue *q, uint16_t offset) +{ + return (q->head_free_desc + offset) & q->sw_ring_wrap_mask; +} + +/* Provide the VC 5GNR descriptor pointer on a given queue */ +static inline union vc_5gnr_dma_desc* +vc_5gnr_get_desc(struct fpga_5gnr_queue *q, uint16_t offset) +{ + return q->vc_5gnr_ring_addr + fpga_5gnr_desc_idx(q, offset); +} + +/* Provide the AGX100 descriptor pointer on a given queue */ +static inline union agx100_dma_desc* +agx100_get_desc(struct fpga_5gnr_queue *q, uint16_t offset) +{ + return q->agx100_ring_addr + fpga_5gnr_desc_idx(q, offset); +} + +/* Provide the descriptor index for the tail of a given queue */ +static inline uint16_t +fpga_5gnr_desc_idx_tail(struct fpga_5gnr_queue *q, uint16_t offset) +{ + return (q->tail + offset) & q->sw_ring_wrap_mask; +} + +/* Provide the descriptor tail pointer on a given queue */ +static inline union vc_5gnr_dma_desc* +vc_5gnr_get_desc_tail(struct fpga_5gnr_queue *q, uint16_t offset) +{ + return q->vc_5gnr_ring_addr + fpga_5gnr_desc_idx_tail(q, offset); +} + +/* Provide the descriptor tail pointer on a given queue */ +static inline union agx100_dma_desc* +agx100_get_desc_tail(struct fpga_5gnr_queue *q, uint16_t offset) +{ + return q->agx100_ring_addr + fpga_5gnr_desc_idx_tail(q, offset); +} + static inline void fpga_5gnr_dma_enqueue(struct fpga_5gnr_queue *q, uint16_t num_desc, struct rte_bbdev_stats *queue_stats) @@ -789,7 +1080,7 @@ fpga_5gnr_dma_enqueue(struct fpga_5gnr_queue *q, uint16_t num_desc, queue_stats->acc_offload_cycles = 0; /* Update tail and shadow_tail register */ - q->tail = (q->tail + num_desc) & q->sw_ring_wrap_mask; + q->tail = fpga_5gnr_desc_idx_tail(q, num_desc); rte_wmb(); @@ -859,6 +1150,72 @@ vc_5gnr_check_desc_error(uint32_t error_code) { return 1; } +/* AGX100 FPGA descriptor errors + * Print an error if a descriptor error has occurred. + * Return 0 on success, 1 on failure + */ +static inline int +agx100_check_desc_error(uint32_t error_code, uint32_t error_msg) { + uint8_t error = error_code << 4 | error_msg; + switch (error) { + case AGX100_DESC_ERR_NO_ERR: + return 0; + case AGX100_DESC_ERR_E_NOT_LEGAL: + rte_bbdev_log(ERR, "Invalid output length of rate matcher E"); + break; + case AGX100_DESC_ERR_K_P_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Encode block size K' is out of range"); + break; + case AGX100_DESC_ERR_NCB_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Ncb circular buffer size is out of range"); + break; + case AGX100_DESC_ERR_Z_C_NOT_LEGAL: + rte_bbdev_log(ERR, "Zc is illegal"); + break; + case AGX100_DESC_ERR_DESC_INDEX_ERR: + rte_bbdev_log(ERR, + "Desc_index received does not meet the expectation in the AGX100" + ); + break; + case AGX100_DESC_ERR_HARQ_INPUT_LEN_A: + rte_bbdev_log(ERR, "HARQ input length A is invalid."); + break; + case AGX100_DESC_ERR_HARQ_INPUT_LEN_B: + rte_bbdev_log(ERR, "HARQ input length B is invalid."); + break; + case AGX100_DESC_ERR_HBSTORE_OFFSET_ERR: + rte_bbdev_log(ERR, "Hbstore exceeds HARQ buffer size."); + break; + case AGX100_DESC_ERR_TB_CBG_ERR: + rte_bbdev_log(ERR, "Total CB number C=0 or CB number with Ea Ca=0 or Ca>C."); + break; + case AGX100_DESC_ERR_CBG_OUT_OF_RANGE: + rte_bbdev_log(ERR, "Cbgti or max_cbg is out of range"); + break; + case AGX100_DESC_ERR_CW_RM_NOT_LEGAL: + rte_bbdev_log(ERR, "Cw_rm is illegal"); + break; + case AGX100_DESC_ERR_UNSUPPORTED_REQ: + rte_bbdev_log(ERR, "Unsupported request for descriptor"); + break; + case AGX100_DESC_ERR_RESERVED: + rte_bbdev_log(ERR, "Reserved"); + break; + case AGX100_DESC_ERR_DESC_ABORT: + rte_bbdev_log(ERR, "Completed abort for descriptor"); + break; + case AGX100_DESC_ERR_DESC_READ_TLP_POISONED: + rte_bbdev_log(ERR, "Descriptor read TLP poisoned"); + break; + default: + rte_bbdev_log(ERR, + "Descriptor error unknown error code %u error msg %u", + error_code, error_msg); + break; + } + return 1; +} + /* Compute value of k0. * Based on 3GPP 38.212 Table 5.4.2.1-2 * Starting position of different redundancy versions, k0 @@ -956,6 +1313,88 @@ vc_5gnr_dma_desc_te_fill(struct rte_bbdev_enc_op *op, return 0; } +/** + * AGX100 FPGA + * Set DMA descriptor for encode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param e + * E value (length of output in bits). + * @param ncb + * Ncb value (size of the soft buffer). + * @param out_length + * Length of output buffer + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +agx100_dma_desc_le_fill(struct rte_bbdev_enc_op *op, + struct agx100_dma_enc_desc *desc, struct rte_mbuf *input, + struct rte_mbuf *output, uint16_t k_, uint32_t e, + uint32_t in_offset, uint32_t out_offset, uint16_t desc_offset, + uint8_t cbs_in_op) +{ + /* reset. */ + desc->done = 0; + desc->error_msg = 0; + desc->error_code = 0; + desc->ncb = op->ldpc_enc.n_cb; + desc->bg_idx = op->ldpc_enc.basegraph - 1; + desc->qm_idx = op->ldpc_enc.q_m >> 1; + desc->zc = op->ldpc_enc.z_c; + desc->rv = op->ldpc_enc.rv_index; + desc->int_en = 0; /**< Set by device externally. */ + desc->max_cbg = 0; /**< TODO: CBG specific. */ + desc->cbgti = 0; /**< TODO: CBG specific. */ + desc->cbgs = 0; /**< TODO: CBG specific. */ + desc->desc_idx = desc_offset; + desc->ca = 0; /**< TODO: CBG specific. */ + desc->c = 0; /**< TODO: CBG specific. */ + desc->num_null = op->ldpc_enc.n_filler; + desc->ea = e; + desc->eb = e; /**< TODO: TB/CBG specific. */ + desc->k_ = k_; + desc->en_slice_ts = 0; /**< TODO: Slice specific. */ + desc->en_host_ts = 0; /**< TODO: Slice specific. */ + desc->en_cb_wr_status = 0; /**< TODO: Event Queue specific. */ + desc->en_output_sg = 0; /**< TODO: Slice specific. */ + desc->en_input_sg = 0; /**< TODO: Slice specific. */ + desc->tb_cb = 0; /**< Descriptor for CB. TODO: Add TB and CBG logic. */ + desc->crc_en = check_bit(op->ldpc_enc.op_flags, + RTE_BBDEV_LDPC_CRC_24B_ATTACH); + + /* Set inbound/outbound data buffer address. */ + /* TODO: add logic for input_slice. */ + desc->output_start_addr_hi = (uint32_t)( + rte_pktmbuf_iova_offset(output, out_offset) >> 32); + desc->output_start_addr_lo = (uint32_t)( + rte_pktmbuf_iova_offset(output, out_offset)); + desc->input_start_addr_hi = (uint32_t)( + rte_pktmbuf_iova_offset(input, in_offset) >> 32); + desc->input_start_addr_lo = (uint32_t)( + rte_pktmbuf_iova_offset(input, in_offset)); + desc->output_length = (e + 7) >> 3; /* in bytes. */ + desc->input_length = input->data_len; + desc->enqueue_timestamp = 0; + desc->completion_timestamp = 0; + /* Save software context needed for dequeue. */ + desc->op_addr = op; + /* Set total number of CBs in an op. */ + desc->cbs_in_op = cbs_in_op; + return 0; +} + /** * Vista Creek 5GNR FPGA * Set DMA descriptor for decode operation (1 Code Block) @@ -1024,6 +1463,105 @@ vc_5gnr_dma_desc_ld_fill(struct rte_bbdev_dec_op *op, return 0; } +/** + * AGX100 FPGA + * Set DMA descriptor for decode operation (1 Code Block) + * + * @param op + * Pointer to a single encode operation. + * @param desc + * Pointer to DMA descriptor. + * @param input + * Pointer to pointer to input data which will be decoded. + * @param in_offset + * Input offset in rte_mbuf structure. It is used for calculating the point + * where data is starting. + * @param out_offset + * Output offset in rte_mbuf structure. It is used for calculating the point + * where hard output data will be stored. + * @param cbs_in_op + * Number of CBs contained in one operation. + */ +static inline int +agx100_dma_desc_ld_fill(struct rte_bbdev_dec_op *op, + struct agx100_dma_dec_desc *desc, + struct rte_mbuf *input, struct rte_mbuf *output, + uint16_t harq_in_length, + uint32_t in_offset, uint32_t out_offset, + uint32_t harq_in_offset, + uint32_t harq_out_offset, + uint16_t desc_offset, + uint8_t cbs_in_op) +{ + /* reset. */ + desc->done = 0; + desc->tb_crc_pass = 0; + desc->cb_crc_all_pass = 0; + desc->cb_all_et_pass = 0; + desc->max_iter_ret = 0; + desc->cgb_crc_bitmap = 0; /**< TODO: CBG specific. */ + desc->error_msg = 0; + desc->error_code = 0; + desc->et_dis = !check_bit(op->ldpc_dec.op_flags, + RTE_BBDEV_LDPC_ITERATION_STOP_ENABLE); + desc->harq_in_en = check_bit(op->ldpc_dec.op_flags, + RTE_BBDEV_LDPC_HQ_COMBINE_IN_ENABLE); + desc->max_iter = op->ldpc_dec.iter_max; + desc->ncb = op->ldpc_dec.n_cb; + desc->bg_idx = op->ldpc_dec.basegraph - 1; + desc->qm_idx = op->ldpc_dec.q_m >> 1; + desc->zc = op->ldpc_dec.z_c; + desc->rv = op->ldpc_dec.rv_index; + desc->int_en = 0; /**< Set by device externally. */ + desc->max_cbg = 0; /**< TODO: CBG specific. */ + desc->cbgti = 0; /**< TODO: CBG specific. */ + desc->cbgfi = 0; /**< TODO: CBG specific. */ + desc->cbgs = 0; /**< TODO: CBG specific. */ + desc->desc_idx = desc_offset; + desc->ca = 0; /**< TODO: CBG specific. */ + desc->c = 0; /**< TODO: CBG specific. */ + desc->llr_pckg = 0; /**< TODO: Not implemented yet. */ + desc->syndrome_check_mode = 1; /**< TODO: Make it configurable. */ + desc->num_null = op->ldpc_dec.n_filler; + desc->ea = op->ldpc_dec.cb_params.e; /**< TODO: TB/CBG specific. */ + desc->eba = 0; /**< TODO: TB/CBG specific. */ + desc->hbstore_offset_out = harq_out_offset >> 10; + desc->hbstore_offset_in = harq_in_offset >> 10; + desc->en_slice_ts = 0; /**< TODO: Slice specific. */ + desc->en_host_ts = 0; /**< TODO: Slice specific. */ + desc->en_cb_wr_status = 0; /**< TODO: Event Queue specific. */ + desc->en_output_sg = 0; /**< TODO: Slice specific. */ + desc->en_input_sg = 0; /**< TODO: Slice specific. */ + desc->tb_cb = 0; /**< Descriptor for CB. TODO: Add TB and CBG logic. */ + desc->crc24b_ind = check_bit(op->ldpc_dec.op_flags, + RTE_BBDEV_LDPC_CRC_TYPE_24B_CHECK); + desc->drop_crc24b = check_bit(op->ldpc_dec.op_flags, + RTE_BBDEV_LDPC_CRC_TYPE_24B_DROP); + desc->harq_input_length_a = + harq_in_length; /**< Descriptor for CB. TODO: Add TB and CBG logic. */ + desc->harq_input_length_b = 0; /**< Descriptor for CB. TODO: Add TB and CBG logic. */ + /* Set inbound/outbound data buffer address. */ + /* TODO: add logic for input_slice. */ + desc->output_start_addr_hi = (uint32_t)( + rte_pktmbuf_iova_offset(output, out_offset) >> 32); + desc->output_start_addr_lo = (uint32_t)( + rte_pktmbuf_iova_offset(output, out_offset)); + desc->input_start_addr_hi = (uint32_t)( + rte_pktmbuf_iova_offset(input, in_offset) >> 32); + desc->input_start_addr_lo = (uint32_t)( + rte_pktmbuf_iova_offset(input, in_offset)); + desc->output_length = (((op->ldpc_dec.basegraph == 1) ? 22 : 10) * op->ldpc_dec.z_c + - op->ldpc_dec.n_filler - desc->drop_crc24b * 24) >> 3; + desc->input_length = op->ldpc_dec.cb_params.e; /**< TODO: TB/CBG specific. */ + desc->enqueue_timestamp = 0; + desc->completion_timestamp = 0; + /* Save software context needed for dequeue. */ + desc->op_addr = op; + /* Set total number of CBs in an op. */ + desc->cbs_in_op = cbs_in_op; + return 0; +} + /* Validates LDPC encoder parameters for VC 5GNR FPGA. */ static inline int vc_5gnr_validate_ldpc_enc_op(struct rte_bbdev_enc_op *op) @@ -1487,27 +2025,35 @@ fpga_5gnr_harq_write_loopback(struct fpga_5gnr_queue *q, uint64_t *input = NULL; uint32_t last_transaction = left_length % FPGA_5GNR_DDR_WR_DATA_LEN_IN_BYTES; uint64_t last_word; + struct fpga_5gnr_fec_device *d = q->d; if (last_transaction > 0) left_length -= last_transaction; - - /* - * Get HARQ buffer size for each VF/PF: When 0x00, there is no - * available DDR space for the corresponding VF/PF. - */ - reg_32 = fpga_5gnr_reg_read_32(q->d->mmio_base, FPGA_5GNR_FEC_HARQ_BUF_SIZE_REGS); - if (reg_32 < harq_in_length) { - left_length = reg_32; - rte_bbdev_log(ERR, "HARQ in length > HARQ buffer size\n"); + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) { + /* + * Get HARQ buffer size for each VF/PF: When 0x00, there is no + * available DDR space for the corresponding VF/PF. + */ + reg_32 = fpga_5gnr_reg_read_32(q->d->mmio_base, FPGA_5GNR_FEC_HARQ_BUF_SIZE_REGS); + if (reg_32 < harq_in_length) { + left_length = reg_32; + rte_bbdev_log(ERR, "HARQ in length > HARQ buffer size\n"); + } } input = (uint64_t *)rte_pktmbuf_mtod_offset(harq_input, uint8_t *, in_offset); while (left_length > 0) { if (fpga_5gnr_reg_read_8(q->d->mmio_base, FPGA_5GNR_FEC_DDR4_ADDR_RDY_REGS) == 1) { - fpga_5gnr_reg_write_32(q->d->mmio_base, - FPGA_5GNR_FEC_DDR4_WR_ADDR_REGS, - out_offset); + if (d->fpga_variant == AGX100_FPGA_VARIANT) { + fpga_5gnr_reg_write_32(q->d->mmio_base, + FPGA_5GNR_FEC_DDR4_WR_ADDR_REGS, + out_offset >> 3); + } else { + fpga_5gnr_reg_write_32(q->d->mmio_base, + FPGA_5GNR_FEC_DDR4_WR_ADDR_REGS, + out_offset); + } fpga_5gnr_reg_write_64(q->d->mmio_base, FPGA_5GNR_FEC_DDR4_WR_DATA_REGS, input[increment]); @@ -1519,12 +2065,17 @@ fpga_5gnr_harq_write_loopback(struct fpga_5gnr_queue *q, } while (last_transaction > 0) { if (fpga_5gnr_reg_read_8(q->d->mmio_base, FPGA_5GNR_FEC_DDR4_ADDR_RDY_REGS) == 1) { - fpga_5gnr_reg_write_32(q->d->mmio_base, - FPGA_5GNR_FEC_DDR4_WR_ADDR_REGS, - out_offset); + if (d->fpga_variant == AGX100_FPGA_VARIANT) { + fpga_5gnr_reg_write_32(q->d->mmio_base, + FPGA_5GNR_FEC_DDR4_WR_ADDR_REGS, + out_offset >> 3); + } else { + fpga_5gnr_reg_write_32(q->d->mmio_base, + FPGA_5GNR_FEC_DDR4_WR_ADDR_REGS, + out_offset); + } last_word = input[increment]; - last_word &= (uint64_t)(1 << (last_transaction * 4)) - - 1; + last_word &= (uint64_t)(1ULL << (last_transaction * 4)) - 1; fpga_5gnr_reg_write_64(q->d->mmio_base, FPGA_5GNR_FEC_DDR4_WR_DATA_REGS, last_word); @@ -1547,14 +2098,17 @@ fpga_5gnr_harq_read_loopback(struct fpga_5gnr_queue *q, uint32_t increment = 0; uint64_t *input = NULL; uint32_t last_transaction = harq_in_length % FPGA_5GNR_DDR_WR_DATA_LEN_IN_BYTES; + struct fpga_5gnr_fec_device *d = q->d; if (last_transaction > 0) harq_in_length += (8 - last_transaction); - reg = fpga_5gnr_reg_read_32(q->d->mmio_base, FPGA_5GNR_FEC_HARQ_BUF_SIZE_REGS); - if (reg < harq_in_length) { - harq_in_length = reg; - rte_bbdev_log(ERR, "HARQ in length > HARQ buffer size\n"); + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) { + reg = fpga_5gnr_reg_read_32(q->d->mmio_base, FPGA_5GNR_FEC_HARQ_BUF_SIZE_REGS); + if (reg < harq_in_length) { + harq_in_length = reg; + rte_bbdev_log(ERR, "HARQ in length > HARQ buffer size\n"); + } } if (!mbuf_append(harq_output, harq_output, harq_in_length)) { @@ -1573,9 +2127,15 @@ fpga_5gnr_harq_read_loopback(struct fpga_5gnr_queue *q, input = (uint64_t *)rte_pktmbuf_mtod_offset(harq_output, uint8_t *, harq_out_offset); while (left_length > 0) { - fpga_5gnr_reg_write_32(q->d->mmio_base, - FPGA_5GNR_FEC_DDR4_RD_ADDR_REGS, - in_offset); + if (d->fpga_variant == AGX100_FPGA_VARIANT) { + fpga_5gnr_reg_write_32(q->d->mmio_base, + FPGA_5GNR_FEC_DDR4_RD_ADDR_REGS, + in_offset >> 3); + } else { + fpga_5gnr_reg_write_32(q->d->mmio_base, + FPGA_5GNR_FEC_DDR4_RD_ADDR_REGS, + in_offset); + } fpga_5gnr_reg_write_8(q->d->mmio_base, FPGA_5GNR_FEC_DDR4_RD_DONE_REGS, 1); reg = fpga_5gnr_reg_read_8(q->d->mmio_base, FPGA_5GNR_FEC_DDR4_RD_RDY_REGS); while (reg != 1) { @@ -1590,7 +2150,10 @@ fpga_5gnr_harq_read_loopback(struct fpga_5gnr_queue *q, left_length -= FPGA_5GNR_DDR_RD_DATA_LEN_IN_BYTES; in_offset += FPGA_5GNR_DDR_WR_DATA_LEN_IN_BYTES; increment++; - fpga_5gnr_reg_write_8(q->d->mmio_base, FPGA_5GNR_FEC_DDR4_RD_DONE_REGS, 0); + if (d->fpga_variant == AGX100_FPGA_VARIANT) + fpga_5gnr_reg_write_8(q->d->mmio_base, FPGA_5GNR_FEC_DDR4_RD_RDY_REGS, 0); + else + fpga_5gnr_reg_write_8(q->d->mmio_base, FPGA_5GNR_FEC_DDR4_RD_DONE_REGS, 0); } fpga_5gnr_mutex_free(q); return 1; @@ -1601,6 +2164,7 @@ enqueue_ldpc_enc_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_enc_op *o uint16_t desc_offset) { union vc_5gnr_dma_desc *vc_5gnr_desc; + union agx100_dma_desc *agx100_desc; int ret; uint8_t c, crc24_bits = 0; struct rte_bbdev_op_ldpc_enc *enc = &op->ldpc_enc; @@ -1613,10 +2177,13 @@ enqueue_ldpc_enc_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_enc_op *o uint16_t total_left = enc->input.length; uint16_t ring_offset; uint16_t K, k_; + struct fpga_5gnr_fec_device *d = q->d; - if (vc_5gnr_validate_ldpc_enc_op(op) == -1) { - rte_bbdev_log(ERR, "LDPC encoder validation rejected"); - return -EINVAL; + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) { + if (vc_5gnr_validate_ldpc_enc_op(op) == -1) { + rte_bbdev_log(ERR, "LDPC encoder validation rejected"); + return -EINVAL; + } } /* Clear op status */ @@ -1632,14 +2199,13 @@ enqueue_ldpc_enc_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_enc_op *o crc24_bits = 24; if (enc->code_block_mode == RTE_BBDEV_TRANSPORT_BLOCK) { - /* For Transport Block mode */ - /* FIXME */ - c = enc->tb_params.c; - e = enc->tb_params.ea; - } else { /* For Code Block mode */ - c = 1; - e = enc->cb_params.e; + /* TODO: For Transport Block mode. */ + rte_bbdev_log(ERR, "Transport Block not supported yet"); + return -1; } + /* For Code Block mode. */ + c = 1; + e = enc->cb_params.e; /* Update total_left */ K = (enc->basegraph == 1 ? 22 : 10) * enc->z_c; @@ -1659,13 +2225,22 @@ enqueue_ldpc_enc_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_enc_op *o mbuf_append(m_out_head, m_out, out_length); - /* Offset into the ring */ - ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); - /* Setup DMA Descriptor */ - vc_5gnr_desc = q->vc_5gnr_ring_addr + ring_offset; - ret = vc_5gnr_dma_desc_te_fill(op, &vc_5gnr_desc->enc_req, m_in, m_out, - k_, e, in_offset, out_offset, ring_offset, c); - if (unlikely(ret < 0)) + /* Offset into the ring. */ + ring_offset = fpga_5gnr_desc_idx_tail(q, desc_offset); + + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) { + /* Setup DMA Descriptor. */ + vc_5gnr_desc = vc_5gnr_get_desc_tail(q, desc_offset); + ret = vc_5gnr_dma_desc_te_fill(op, &vc_5gnr_desc->enc_req, m_in, m_out, + k_, e, in_offset, out_offset, ring_offset, c); + } else { + /* Setup DMA Descriptor. */ + agx100_desc = agx100_get_desc_tail(q, desc_offset); + ret = agx100_dma_desc_le_fill(op, &agx100_desc->enc_req, m_in, m_out, + k_, e, in_offset, out_offset, ring_offset, c); + } + + if (unlikely(ret < 0)) return ret; /* Update lengths */ @@ -1680,7 +2255,10 @@ enqueue_ldpc_enc_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_enc_op *o } #ifdef RTE_LIBRTE_BBDEV_DEBUG - vc_5gnr_print_dma_enc_desc_debug_info(vc_5gnr_desc); + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) + vc_5gnr_print_dma_enc_desc_debug_info(vc_5gnr_desc); + else + agx100_print_dma_enc_desc_debug_info(agx100_desc); #endif return 1; } @@ -1713,8 +2291,8 @@ vc_5gnr_enqueue_ldpc_dec_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_d op->status = 0; /* Setup DMA Descriptor */ - ring_offset = ((q->tail + desc_offset) & q->sw_ring_wrap_mask); - desc = q->vc_5gnr_ring_addr + ring_offset; + ring_offset = fpga_5gnr_desc_idx_tail(q, desc_offset); + desc = vc_5gnr_get_desc_tail(q, desc_offset); if (check_bit(dec->op_flags, RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_LOOPBACK)) { struct rte_mbuf *harq_in = dec->harq_combined_input.data; @@ -1820,6 +2398,128 @@ vc_5gnr_enqueue_ldpc_dec_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_d return 1; } +static inline int +agx100_enqueue_ldpc_dec_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_dec_op *op, + uint16_t desc_offset) +{ + union agx100_dma_desc *desc; + int ret; + uint16_t ring_offset; + uint8_t c; + uint16_t e, in_length, out_length, k0, l, seg_total_left, sys_cols; + uint16_t K, parity_offset, harq_in_length = 0, harq_out_length = 0; + uint16_t crc24_overlap = 0; + struct rte_bbdev_op_ldpc_dec *dec = &op->ldpc_dec; + struct rte_mbuf *m_in = dec->input.data; + struct rte_mbuf *m_out = dec->hard_output.data; + struct rte_mbuf *m_out_head = dec->hard_output.data; + uint16_t in_offset = dec->input.offset; + uint16_t out_offset = dec->hard_output.offset; + uint32_t harq_in_offset = 0; + uint32_t harq_out_offset = 0; + + /* Clear op status. */ + op->status = 0; + + /* Setup DMA Descriptor. */ + ring_offset = fpga_5gnr_desc_idx_tail(q, desc_offset); + desc = agx100_get_desc_tail(q, desc_offset); + + if (check_bit(dec->op_flags, RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_LOOPBACK)) { + struct rte_mbuf *harq_in = dec->harq_combined_input.data; + struct rte_mbuf *harq_out = dec->harq_combined_output.data; + harq_in_length = dec->harq_combined_input.length; + uint32_t harq_in_offset = dec->harq_combined_input.offset; + uint32_t harq_out_offset = dec->harq_combined_output.offset; + + if (check_bit(dec->op_flags, RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_OUT_ENABLE)) { + ret = fpga_5gnr_harq_write_loopback(q, harq_in, + harq_in_length, harq_in_offset, + harq_out_offset); + } else if (check_bit(dec->op_flags, + RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_IN_ENABLE)) { + ret = fpga_5gnr_harq_read_loopback(q, harq_out, + harq_in_length, harq_in_offset, + harq_out_offset); + dec->harq_combined_output.length = harq_in_length; + } else { + rte_bbdev_log(ERR, "OP flag Err!"); + ret = -1; + } + + /* Set descriptor for dequeue. */ + desc->dec_req.done = 1; + desc->dec_req.error_code = 0; + desc->dec_req.error_msg = 0; + desc->dec_req.op_addr = op; + desc->dec_req.cbs_in_op = 1; + + /* Mark this dummy descriptor to be dropped by HW. */ + desc->dec_req.desc_idx = (ring_offset + 1) & q->sw_ring_wrap_mask; + + return ret; /* Error or number of CB. */ + } + + if (m_in == NULL || m_out == NULL) { + rte_bbdev_log(ERR, "Invalid mbuf pointer"); + op->status = 1 << RTE_BBDEV_DATA_ERROR; + return -1; + } + + c = 1; + e = dec->cb_params.e; + + if (check_bit(dec->op_flags, RTE_BBDEV_LDPC_CRC_TYPE_24B_DROP)) + crc24_overlap = 24; + + sys_cols = (dec->basegraph == 1) ? 22 : 10; + K = sys_cols * dec->z_c; + parity_offset = K - 2 * dec->z_c; + + out_length = ((K - crc24_overlap - dec->n_filler) >> 3); + in_length = e; + seg_total_left = dec->input.length; + + if (check_bit(dec->op_flags, RTE_BBDEV_LDPC_HQ_COMBINE_IN_ENABLE)) + harq_in_length = RTE_MIN(dec->harq_combined_input.length, (uint32_t)dec->n_cb); + + if (check_bit(dec->op_flags, RTE_BBDEV_LDPC_HQ_COMBINE_OUT_ENABLE)) { + k0 = get_k0(dec->n_cb, dec->z_c, dec->basegraph, dec->rv_index); + if (k0 > parity_offset) + l = k0 + e; + else + l = k0 + e + dec->n_filler; + harq_out_length = RTE_MIN(RTE_MAX(harq_in_length, l), dec->n_cb); + dec->harq_combined_output.length = harq_out_length; + } + + mbuf_append(m_out_head, m_out, out_length); + harq_in_offset = dec->harq_combined_input.offset; + harq_out_offset = dec->harq_combined_output.offset; + + ret = agx100_dma_desc_ld_fill(op, &desc->dec_req, m_in, m_out, + harq_in_length, in_offset, out_offset, harq_in_offset, + harq_out_offset, ring_offset, c); + + if (unlikely(ret < 0)) + return ret; + /* Update lengths. */ + seg_total_left -= in_length; + op->ldpc_dec.hard_output.length += out_length; + if (seg_total_left > 0) { + rte_bbdev_log(ERR, + "Mismatch between mbuf length and included CB sizes: mbuf len %u, cb len %u", + seg_total_left, in_length); + return -1; + } + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + agx100_print_dma_dec_desc_debug_info(desc); +#endif + + return 1; +} + static uint16_t fpga_5gnr_enqueue_ldpc_enc(struct rte_bbdev_queue_data *q_data, struct rte_bbdev_enc_op **ops, uint16_t num) @@ -1829,9 +2529,11 @@ fpga_5gnr_enqueue_ldpc_enc(struct rte_bbdev_queue_data *q_data, int enqueued_cbs; struct fpga_5gnr_queue *q = q_data->queue_private; union vc_5gnr_dma_desc *vc_5gnr_desc; + union agx100_dma_desc *agx100_desc; + struct fpga_5gnr_fec_device *d = q->d; /* Check if queue is not full */ - if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == q->head_free_desc)) + if (unlikely((fpga_5gnr_desc_idx_tail(q, 1)) == q->head_free_desc)) return 0; /* Calculates available space */ @@ -1861,9 +2563,13 @@ fpga_5gnr_enqueue_ldpc_enc(struct rte_bbdev_queue_data *q_data, /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt * only when all previous CBs were already processed. */ - vc_5gnr_desc = q->vc_5gnr_ring_addr + - ((q->tail + total_enqueued_cbs - 1) & q->sw_ring_wrap_mask); - vc_5gnr_desc->enc_req.irq_en = q->irq_enable; + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) { + vc_5gnr_desc = vc_5gnr_get_desc_tail(q, total_enqueued_cbs - 1); + vc_5gnr_desc->enc_req.irq_en = q->irq_enable; + } else { + agx100_desc = agx100_get_desc_tail(q, total_enqueued_cbs - 1); + agx100_desc->enc_req.int_en = q->irq_enable; + } fpga_5gnr_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); @@ -1883,9 +2589,11 @@ fpga_5gnr_enqueue_ldpc_dec(struct rte_bbdev_queue_data *q_data, int enqueued_cbs; struct fpga_5gnr_queue *q = q_data->queue_private; union vc_5gnr_dma_desc *vc_5gnr_desc; + union agx100_dma_desc *agx100_desc; + struct fpga_5gnr_fec_device *d = q->d; /* Check if queue is not full */ - if (unlikely(((q->tail + 1) & q->sw_ring_wrap_mask) == q->head_free_desc)) + if (unlikely((fpga_5gnr_desc_idx_tail(q, 1)) == q->head_free_desc)) return 0; /* Calculates available space */ @@ -1901,8 +2609,13 @@ fpga_5gnr_enqueue_ldpc_dec(struct rte_bbdev_queue_data *q_data, if (unlikely(avail - 1 < 0)) break; avail -= 1; - enqueued_cbs = vc_5gnr_enqueue_ldpc_dec_one_op_cb(q, ops[i], - total_enqueued_cbs); + if (q->d->fpga_variant == VC_5GNR_FPGA_VARIANT) { + enqueued_cbs = vc_5gnr_enqueue_ldpc_dec_one_op_cb(q, ops[i], + total_enqueued_cbs); + } else { + enqueued_cbs = agx100_enqueue_ldpc_dec_one_op_cb(q, ops[i], + total_enqueued_cbs); + } if (enqueued_cbs < 0) break; @@ -1921,9 +2634,14 @@ fpga_5gnr_enqueue_ldpc_dec(struct rte_bbdev_queue_data *q_data, /* Set interrupt bit for last CB in enqueued ops. FPGA issues interrupt * only when all previous CBs were already processed. */ - vc_5gnr_desc = q->vc_5gnr_ring_addr + - ((q->tail + total_enqueued_cbs - 1) & q->sw_ring_wrap_mask); - vc_5gnr_desc->enc_req.irq_en = q->irq_enable; + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) { + vc_5gnr_desc = vc_5gnr_get_desc_tail(q, total_enqueued_cbs - 1); + vc_5gnr_desc->enc_req.irq_en = q->irq_enable; + } else { + agx100_desc = agx100_get_desc_tail(q, total_enqueued_cbs - 1); + agx100_desc->enc_req.int_en = q->irq_enable; + } + fpga_5gnr_dma_enqueue(q, total_enqueued_cbs, &q_data->queue_stats); return i; } @@ -1936,7 +2654,7 @@ vc_5gnr_dequeue_ldpc_enc_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_e union vc_5gnr_dma_desc *desc; int desc_error; /* Set current desc */ - desc = q->vc_5gnr_ring_addr + ((q->head_free_desc + desc_offset) & q->sw_ring_wrap_mask); + desc = vc_5gnr_get_desc(q, desc_offset); /*check if done */ if (desc->enc_req.done == 0) @@ -1958,6 +2676,36 @@ vc_5gnr_dequeue_ldpc_enc_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_e return 1; } +static inline int +agx100_dequeue_ldpc_enc_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_enc_op **op, + uint16_t desc_offset) +{ + union agx100_dma_desc *desc; + int desc_error; + + /* Set current desc. */ + desc = agx100_get_desc(q, desc_offset); + /*check if done */ + if (desc->enc_req.done == 0) + return -1; + + /* make sure the response is read atomically. */ + rte_smp_rmb(); + + rte_bbdev_log_debug("DMA response desc %p", desc); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + agx100_print_dma_enc_desc_debug_info(desc); +#endif + *op = desc->enc_req.op_addr; + /* Check the descriptor error field, return 1 on error. */ + desc_error = agx100_check_desc_error(desc->enc_req.error_code, + desc->enc_req.error_msg); + + (*op)->status = desc_error << RTE_BBDEV_DATA_ERROR; + + return 1; +} static inline int vc_5gnr_dequeue_ldpc_dec_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_dec_op **op, @@ -1967,7 +2715,7 @@ vc_5gnr_dequeue_ldpc_dec_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_d int desc_error; /* Set descriptor */ - desc = q->vc_5gnr_ring_addr + ((q->head_free_desc + desc_offset) & q->sw_ring_wrap_mask); + desc = vc_5gnr_get_desc(q, desc_offset); /* Verify done bit is set */ if (desc->dec_req.done == 0) @@ -2006,6 +2754,51 @@ vc_5gnr_dequeue_ldpc_dec_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_d return 1; } +static inline int +agx100_dequeue_ldpc_dec_one_op_cb(struct fpga_5gnr_queue *q, struct rte_bbdev_dec_op **op, + uint16_t desc_offset) +{ + union agx100_dma_desc *desc; + int desc_error; + + /* Set descriptor. */ + desc = agx100_get_desc(q, desc_offset); + /* Verify done bit is set. */ + if (desc->dec_req.done == 0) + return -1; + + /* make sure the response is read atomically. */ + rte_smp_rmb(); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + agx100_print_dma_dec_desc_debug_info(desc); +#endif + + *op = desc->dec_req.op_addr; + + if (check_bit((*op)->ldpc_dec.op_flags, RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_LOOPBACK)) { + (*op)->status = 0; + return 1; + } + + /* FPGA reports iterations based on round-up minus 1. */ + (*op)->ldpc_dec.iter_count = desc->dec_req.max_iter_ret + 1; + + /* CRC Check criteria. */ + if (desc->dec_req.crc24b_ind && !(desc->dec_req.cb_crc_all_pass)) + (*op)->status = 1 << RTE_BBDEV_CRC_ERROR; + + /* et_pass = 0 when decoder fails. */ + (*op)->status |= !(desc->dec_req.cb_all_et_pass) << RTE_BBDEV_SYNDROME_ERROR; + + /* Check the descriptor error field, return 1 on error. */ + desc_error = agx100_check_desc_error(desc->dec_req.error_code, + desc->dec_req.error_msg); + + (*op)->status |= desc_error << RTE_BBDEV_DATA_ERROR; + return 1; +} + static uint16_t fpga_5gnr_dequeue_ldpc_enc(struct rte_bbdev_queue_data *q_data, struct rte_bbdev_enc_op **ops, uint16_t num) @@ -2017,7 +2810,10 @@ fpga_5gnr_dequeue_ldpc_enc(struct rte_bbdev_queue_data *q_data, int ret; for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { - ret = vc_5gnr_dequeue_ldpc_enc_one_op_cb(q, &ops[i], dequeued_cbs); + if (q->d->fpga_variant == VC_5GNR_FPGA_VARIANT) + ret = vc_5gnr_dequeue_ldpc_enc_one_op_cb(q, &ops[i], dequeued_cbs); + else + ret = agx100_dequeue_ldpc_enc_one_op_cb(q, &ops[i], dequeued_cbs); if (ret < 0) break; @@ -2029,8 +2825,7 @@ fpga_5gnr_dequeue_ldpc_enc(struct rte_bbdev_queue_data *q_data, } /* Update head */ - q->head_free_desc = (q->head_free_desc + dequeued_cbs) & - q->sw_ring_wrap_mask; + q->head_free_desc = fpga_5gnr_desc_idx(q, dequeued_cbs); /* Update stats */ q_data->queue_stats.dequeued_count += i; @@ -2049,7 +2844,10 @@ fpga_5gnr_dequeue_ldpc_dec(struct rte_bbdev_queue_data *q_data, int ret; for (i = 0; (i < num) && (dequeued_cbs < avail); ++i) { - ret = vc_5gnr_dequeue_ldpc_dec_one_op_cb(q, &ops[i], dequeued_cbs); + if (q->d->fpga_variant == VC_5GNR_FPGA_VARIANT) + ret = vc_5gnr_dequeue_ldpc_dec_one_op_cb(q, &ops[i], dequeued_cbs); + else + ret = agx100_dequeue_ldpc_dec_one_op_cb(q, &ops[i], dequeued_cbs); if (ret < 0) break; @@ -2061,7 +2859,7 @@ fpga_5gnr_dequeue_ldpc_dec(struct rte_bbdev_queue_data *q_data, } /* Update head */ - q->head_free_desc = (q->head_free_desc + dequeued_cbs) & q->sw_ring_wrap_mask; + q->head_free_desc = fpga_5gnr_desc_idx(q, dequeued_cbs); /* Update stats */ q_data->queue_stats.dequeued_count += i; @@ -2082,12 +2880,29 @@ fpga_5gnr_fec_init(struct rte_bbdev *dev, struct rte_pci_driver *drv) dev->dequeue_ldpc_enc_ops = fpga_5gnr_dequeue_ldpc_enc; dev->dequeue_ldpc_dec_ops = fpga_5gnr_dequeue_ldpc_dec; - ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->pf_device = - !strcmp(drv->driver.name, RTE_STR(FPGA_5GNR_FEC_PF_DRIVER_NAME)); - ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->mmio_base = - pci_dev->mem_resource[0].addr; - ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->total_num_queues = - VC_5GNR_TOTAL_NUM_QUEUES; + /* Device variant specific handling. */ + if ((pci_dev->id.device_id == AGX100_PF_DEVICE_ID) || + (pci_dev->id.device_id == AGX100_VF_DEVICE_ID)) { + ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->fpga_variant = + AGX100_FPGA_VARIANT; + ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->pf_device = + !strcmp(drv->driver.name, RTE_STR(FPGA_5GNR_FEC_PF_DRIVER_NAME)); + ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->mmio_base = + pci_dev->mem_resource[0].addr; + /* Maximum number of queues possible for this device. */ + ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->total_num_queues = + fpga_5gnr_reg_read_32(pci_dev->mem_resource[0].addr, + FPGA_5GNR_FEC_VERSION_ID) >> 24; + } else { + ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->fpga_variant = + VC_5GNR_FPGA_VARIANT; + ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->pf_device = + !strcmp(drv->driver.name, RTE_STR(FPGA_5GNR_FEC_PF_DRIVER_NAME)); + ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->mmio_base = + pci_dev->mem_resource[0].addr; + ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->total_num_queues = + VC_5GNR_TOTAL_NUM_QUEUES; + } rte_bbdev_log_debug( "Init device %s [%s] @ virtaddr %p phyaddr %#"PRIx64, @@ -2102,6 +2917,7 @@ fpga_5gnr_fec_probe(struct rte_pci_driver *pci_drv, { struct rte_bbdev *bbdev = NULL; char dev_name[RTE_BBDEV_NAME_MAX_LEN]; + struct fpga_5gnr_fec_device *d; if (pci_dev == NULL) { rte_bbdev_log(ERR, "NULL PCI device"); @@ -2140,15 +2956,24 @@ fpga_5gnr_fec_probe(struct rte_pci_driver *pci_drv, rte_bbdev_log_debug("bbdev id = %u [%s]", bbdev->data->dev_id, dev_name); - struct fpga_5gnr_fec_device *d = bbdev->data->dev_private; - uint32_t version_id = fpga_5gnr_reg_read_32(d->mmio_base, FPGA_5GNR_FEC_VERSION_ID); - rte_bbdev_log(INFO, "Vista Creek FPGA RTL v%u.%u", - ((uint16_t)(version_id >> 16)), ((uint16_t)version_id)); + d = bbdev->data->dev_private; + if (d->fpga_variant == VC_5GNR_FPGA_VARIANT) { + uint32_t version_id = fpga_5gnr_reg_read_32(d->mmio_base, FPGA_5GNR_FEC_VERSION_ID); + rte_bbdev_log(INFO, "Vista Creek FPGA RTL v%u.%u", + ((uint16_t)(version_id >> 16)), ((uint16_t)version_id)); + } else { + uint32_t version_num_queues = fpga_5gnr_reg_read_32(d->mmio_base, + FPGA_5GNR_FEC_VERSION_ID); + uint8_t major_version_id = version_num_queues >> 16; + uint8_t minor_version_id = version_num_queues >> 8; + uint8_t patch_id = version_num_queues; + + rte_bbdev_log(INFO, "AGX100 RTL v%u.%u.%u", + major_version_id, minor_version_id, patch_id); + } #ifdef RTE_LIBRTE_BBDEV_DEBUG - if (!strcmp(pci_drv->driver.name, - RTE_STR(FPGA_5GNR_FEC_PF_DRIVER_NAME))) - print_static_reg_debug_info(d->mmio_base); + print_static_reg_debug_info(d->mmio_base, d->fpga_variant); #endif return 0; } @@ -2374,7 +3199,169 @@ static int vc_5gnr_configure(const char *dev_name, const struct rte_fpga_5gnr_fe rte_bbdev_log_debug("PF Vista Creek 5GNR FPGA configuration complete for %s", dev_name); #ifdef RTE_LIBRTE_BBDEV_DEBUG - print_static_reg_debug_info(d->mmio_base); + print_static_reg_debug_info(d->mmio_base, d->fpga_variant); +#endif + return 0; +} + +/* Initial configuration of AGX100 device. */ +static int agx100_configure(const char *dev_name, const struct rte_fpga_5gnr_fec_conf *conf) +{ + uint32_t payload_32, address; + uint16_t payload_16; + uint8_t payload_8; + uint16_t q_id, vf_id, total_q_id, total_ul_q_id, total_dl_q_id; + struct rte_bbdev *bbdev = rte_bbdev_get_named_dev(dev_name); + struct rte_fpga_5gnr_fec_conf def_conf; + + if (bbdev == NULL) { + rte_bbdev_log(ERR, + "Invalid dev_name (%s), or device is not yet initialised", + dev_name); + return -ENODEV; + } + + struct fpga_5gnr_fec_device *d = bbdev->data->dev_private; + + if (conf == NULL) { + rte_bbdev_log(ERR, "AGX100 Configuration was not provided."); + rte_bbdev_log(ERR, "Default configuration will be loaded."); + fpga_5gnr_set_default_conf(&def_conf); + conf = &def_conf; + } + + uint8_t total_num_queues = d->total_num_queues; + uint8_t num_ul_queues = total_num_queues >> 1; + uint8_t num_dl_queues = total_num_queues >> 1; + + /* Clear all queues registers */ + payload_32 = FPGA_5GNR_INVALID_HW_QUEUE_ID; + for (q_id = 0; q_id < total_num_queues; ++q_id) { + address = (q_id << 2) + AGX100_QUEUE_MAP; + fpga_5gnr_reg_write_32(d->mmio_base, address, payload_32); + } + + /* + * If PF mode is enabled allocate all queues for PF only. + * + * For VF mode each VF can have different number of UL and DL queues. + * Total number of queues to configure cannot exceed AGX100 + * capabilities - 64 queues - 32 queues for UL and 32 queues for DL. + * Queues mapping is done according to configuration: + * + * UL queues: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | ... | 0 | + * | conf->vf_dl_queues_number[0] - 1 | 0 | + * | conf->vf_dl_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_dl_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_dl_queues_number[7] - 1 | 7 | + * + * DL queues: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | ... | 0 | + * | conf->vf_ul_queues_number[0] - 1 | 0 | + * | conf->vf_ul_queues_number[0] | 1 | + * | ... | 1 | + * | conf->vf_ul_queues_number[1] - 1 | 1 | + * | ... | ... | + * | conf->vf_ul_queues_number[7] - 1 | 7 | + * + * Example of configuration: + * conf->vf_ul_queues_number[0] = 4; -> 4 UL queues for VF0 + * conf->vf_dl_queues_number[0] = 4; -> 4 DL queues for VF0 + * conf->vf_ul_queues_number[1] = 2; -> 2 UL queues for VF1 + * conf->vf_dl_queues_number[1] = 2; -> 2 DL queues for VF1 + * + * UL: + * | Q_ID | VF_ID | + * | 0 | 0 | + * | 1 | 0 | + * | 2 | 0 | + * | 3 | 0 | + * | 4 | 1 | + * | 5 | 1 | + * + * DL: + * | Q_ID | VF_ID | + * | 32 | 0 | + * | 33 | 0 | + * | 34 | 0 | + * | 35 | 0 | + * | 36 | 1 | + * | 37 | 1 | + */ + if (conf->pf_mode_en) { + payload_32 = 0x1; + for (q_id = 0; q_id < total_num_queues; ++q_id) { + address = (q_id << 2) + AGX100_QUEUE_MAP; + fpga_5gnr_reg_write_32(d->mmio_base, address, payload_32); + } + } else { + /* Calculate total number of UL and DL queues to configure. */ + total_ul_q_id = total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_5GNR_FEC_NUM_VFS; ++vf_id) { + total_ul_q_id += conf->vf_ul_queues_number[vf_id]; + total_dl_q_id += conf->vf_dl_queues_number[vf_id]; + } + total_q_id = total_dl_q_id + total_ul_q_id; + /* + * Check if total number of queues to configure does not exceed + * AGX100 capabilities (64 queues - 32 UL and 32 DL queues) + */ + if ((total_ul_q_id > num_ul_queues) || + (total_dl_q_id > num_dl_queues) || + (total_q_id > total_num_queues)) { + rte_bbdev_log(ERR, + "AGX100 Configuration failed. Too many queues to configure: UL_Q %u, DL_Q %u, AGX100_Q %u", + total_ul_q_id, total_dl_q_id, + total_num_queues); + return -EINVAL; + } + total_ul_q_id = 0; + for (vf_id = 0; vf_id < FPGA_5GNR_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_ul_queues_number[vf_id]; + ++q_id, ++total_ul_q_id) { + address = (total_ul_q_id << 2) + AGX100_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_5gnr_reg_write_32(d->mmio_base, address, payload_32); + } + } + total_dl_q_id = 0; + for (vf_id = 0; vf_id < FPGA_5GNR_FEC_NUM_VFS; ++vf_id) { + for (q_id = 0; q_id < conf->vf_dl_queues_number[vf_id]; + ++q_id, ++total_dl_q_id) { + address = ((total_dl_q_id + num_ul_queues) + << 2) + AGX100_QUEUE_MAP; + payload_32 = ((0x80 + vf_id) << 16) | 0x1; + fpga_5gnr_reg_write_32(d->mmio_base, address, payload_32); + } + } + } + + /* Setting Load Balance Factor. */ + payload_16 = (conf->dl_load_balance << 8) | (conf->ul_load_balance); + address = FPGA_5GNR_FEC_LOAD_BALANCE_FACTOR; + fpga_5gnr_reg_write_16(d->mmio_base, address, payload_16); + + /* Setting length of ring descriptor entry. */ + payload_16 = FPGA_5GNR_RING_DESC_ENTRY_LENGTH; + address = FPGA_5GNR_FEC_RING_DESC_LEN; + fpga_5gnr_reg_write_16(d->mmio_base, address, payload_16); + + /* Queue PF/VF mapping table is ready. */ + payload_8 = 0x1; + address = FPGA_5GNR_FEC_QUEUE_PF_VF_MAP_DONE; + fpga_5gnr_reg_write_8(d->mmio_base, address, payload_8); + + rte_bbdev_log_debug("PF AGX100 configuration complete for %s", dev_name); + +#ifdef RTE_LIBRTE_BBDEV_DEBUG + print_static_reg_debug_info(d->mmio_base, d->fpga_variant); #endif return 0; } @@ -2391,6 +3378,8 @@ int rte_fpga_5gnr_fec_configure(const char *dev_name, const struct rte_fpga_5gnr printf("Configure dev id %x\n", pci_dev->id.device_id); if (pci_dev->id.device_id == VC_5GNR_PF_DEVICE_ID) return vc_5gnr_configure(dev_name, conf); + else if (pci_dev->id.device_id == AGX100_PF_DEVICE_ID) + return agx100_configure(dev_name, conf); rte_bbdev_log(ERR, "Invalid device_id (%d)", pci_dev->id.device_id); return -ENODEV; @@ -2398,6 +3387,9 @@ int rte_fpga_5gnr_fec_configure(const char *dev_name, const struct rte_fpga_5gnr /* FPGA 5GNR FEC PCI PF address map */ static struct rte_pci_id pci_id_fpga_5gnr_fec_pf_map[] = { + { + RTE_PCI_DEVICE(AGX100_VENDOR_ID, AGX100_PF_DEVICE_ID) + }, { RTE_PCI_DEVICE(VC_5GNR_VENDOR_ID, VC_5GNR_PF_DEVICE_ID) }, @@ -2413,6 +3405,9 @@ static struct rte_pci_driver fpga_5gnr_fec_pci_pf_driver = { /* FPGA 5GNR FEC PCI VF address map */ static struct rte_pci_id pci_id_fpga_5gnr_fec_vf_map[] = { + { + RTE_PCI_DEVICE(AGX100_VENDOR_ID, AGX100_VF_DEVICE_ID) + }, { RTE_PCI_DEVICE(VC_5GNR_VENDOR_ID, VC_5GNR_VF_DEVICE_ID) }, diff --git a/drivers/baseband/fpga_5gnr_fec/vc_5gnr_pmd.h b/drivers/baseband/fpga_5gnr_fec/vc_5gnr_pmd.h index 47fb43199f86..9a488ae8d6d1 100644 --- a/drivers/baseband/fpga_5gnr_fec/vc_5gnr_pmd.h +++ b/drivers/baseband/fpga_5gnr_fec/vc_5gnr_pmd.h @@ -16,7 +16,6 @@ #define VC_5GNR_NUM_UL_QUEUES (32) #define VC_5GNR_NUM_DL_QUEUES (32) #define VC_5GNR_TOTAL_NUM_QUEUES (VC_5GNR_NUM_UL_QUEUES + VC_5GNR_NUM_DL_QUEUES) -#define VC_5GNR_NUM_INTR_VEC (VC_5GNR_TOTAL_NUM_QUEUES - RTE_INTR_VEC_RXTX_OFFSET) /* VC 5GNR Ring size is in 256 bits (32 bytes) units. */ #define VC_5GNR_RING_DESC_LEN_UNIT_BYTES (32)