[02/10] net/nfp: add TLVs capability parsing

Message ID 20230925060644.1458598-3-chaoyong.he@corigine.com (mailing list archive)
State Superseded, archived
Delegated to: Ferruh Yigit
Headers
Series add the support of ipsec offload |

Checks

Context Check Description
ci/checkpatch success coding style OK

Commit Message

Chaoyong He Sept. 25, 2023, 6:06 a.m. UTC
  From: Shihong Wang <shihong.wang@corigine.com>

Add TLV capabilities to the BAR, TLVs is fit for expressing
capabilities of applications running on programmable hardware.
Declares a TLV capability start at offset 0x58, up to 0x0d90.
The used space can be wrapped with RESERVED.

Signed-off-by: Shihong Wang <shihong.wang@corigine.com>
Signed-off-by: Chang Miao <chang.miao@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
---
 drivers/net/nfp/meson.build  |   1 +
 drivers/net/nfp/nfp_common.h |  13 +++
 drivers/net/nfp/nfp_ctrl.c   | 153 +++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_ctrl.h   | 121 +++++++++++++++++++++++++++
 drivers/net/nfp/nfp_ethdev.c |   6 ++
 5 files changed, 294 insertions(+)
 create mode 100644 drivers/net/nfp/nfp_ctrl.c
  

Comments

Ferruh Yigit Sept. 27, 2023, 2:19 p.m. UTC | #1
On 9/25/2023 7:06 AM, Chaoyong He wrote:
> From: Shihong Wang <shihong.wang@corigine.com>
> 
> Add TLV capabilities to the BAR, TLVs is fit for expressing
> capabilities of applications running on programmable hardware.
>

Here application referred is bitstream or FW, right?

And PCIe BAR is used to exchange the capability information.
Not for this patch, but I wonder is there a value to add this kind of
parsing to the PCI code, if it is a generic usage, Chaoyong what do you
think?

> Declares a TLV capability start at offset 0x58, up to 0x0d90.
> The used space can be wrapped with RESERVED.
> 
> Signed-off-by: Shihong Wang <shihong.wang@corigine.com>
> Signed-off-by: Chang Miao <chang.miao@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> 

<...>
  
Chaoyong He Sept. 28, 2023, 2:02 a.m. UTC | #2
> On 9/25/2023 7:06 AM, Chaoyong He wrote:
> > From: Shihong Wang <shihong.wang@corigine.com>
> >
> > Add TLV capabilities to the BAR, TLVs is fit for expressing
> > capabilities of applications running on programmable hardware.
> >
> 
> Here application referred is bitstream or FW, right?
Yes, it is.

> And PCIe BAR is used to exchange the capability information.
> Not for this patch, but I wonder is there a value to add this kind of parsing to the
> PCI code, if it is a generic usage, Chaoyong what do you think?
Basically, it does have value to do that, the kernel driver already have these logics for a long time and using it for some features.
For our PMD, it's the first time we import this and the main reason is we need to support the mailbox channel between driver and CoreNIC firmware.
The mailbox channel is needed for IPsec and other features we will try to merge in the near future.

I have to admit the ` nfp_net_tlv_caps_parse()` function now seems does have some unrelated logics, I will try to remove them in the next version patch.

Do you think it's okay?

> > Declares a TLV capability start at offset 0x58, up to 0x0d90.
> > The used space can be wrapped with RESERVED.
> >
> > Signed-off-by: Shihong Wang <shihong.wang@corigine.com>
> > Signed-off-by: Chang Miao <chang.miao@corigine.com>
> > Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> >
> 
> <...>
  
Ferruh Yigit Sept. 28, 2023, 9:30 a.m. UTC | #3
On 9/28/2023 3:02 AM, Chaoyong He wrote:
>> On 9/25/2023 7:06 AM, Chaoyong He wrote:
>>> From: Shihong Wang <shihong.wang@corigine.com>
>>>
>>> Add TLV capabilities to the BAR, TLVs is fit for expressing
>>> capabilities of applications running on programmable hardware.
>>>
>>
>> Here application referred is bitstream or FW, right?
> Yes, it is.
> 
>> And PCIe BAR is used to exchange the capability information.
>> Not for this patch, but I wonder is there a value to add this kind of parsing to the
>> PCI code, if it is a generic usage, Chaoyong what do you think?
> Basically, it does have value to do that, the kernel driver already have these logics for a long time and using it for some features.
> For our PMD, it's the first time we import this and the main reason is we need to support the mailbox channel between driver and CoreNIC firmware.
> The mailbox channel is needed for IPsec and other features we will try to merge in the near future.
> 
> I have to admit the ` nfp_net_tlv_caps_parse()` function now seems does have some unrelated logics, I will try to remove them in the next version patch.
> 
> Do you think it's okay?
> 

That is OK, thanks.

And we can consider to move some of the functionality to bus layer when
there are more users of it, I can ask for your help at that time if it
is OK.

>>> Declares a TLV capability start at offset 0x58, up to 0x0d90.
>>> The used space can be wrapped with RESERVED.
>>>
>>> Signed-off-by: Shihong Wang <shihong.wang@corigine.com>
>>> Signed-off-by: Chang Miao <chang.miao@corigine.com>
>>> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
>>>
>>
>> <...>
>
  

Patch

diff --git a/drivers/net/nfp/meson.build b/drivers/net/nfp/meson.build
index d422269c4b..e78bcb8b75 100644
--- a/drivers/net/nfp/meson.build
+++ b/drivers/net/nfp/meson.build
@@ -27,6 +27,7 @@  sources = files(
         'nfpcore/nfp_target.c',
         'nfpcore/nfp6000_pcie.c',
         'nfp_common.c',
+        'nfp_ctrl.c',
         'nfp_rxtx.c',
         'nfp_cpp_bridge.c',
         'nfp_ethdev_vf.c',
diff --git a/drivers/net/nfp/nfp_common.h b/drivers/net/nfp/nfp_common.h
index b0372c3dc6..4a19c18466 100644
--- a/drivers/net/nfp/nfp_common.h
+++ b/drivers/net/nfp/nfp_common.h
@@ -64,6 +64,17 @@  enum nfp_net_meta_format {
 	NFP_NET_METAFORMAT_CHAINED,
 };
 
+/* Parsed control BAR TLV capabilities */
+struct nfp_net_tlv_caps {
+	uint32_t me_freq_mhz;            /**< ME clock_freq (MHz) */
+	uint32_t mbox_off;               /**< VNIC mailbox area offset */
+	uint32_t mbox_len;               /**< VNIC mailbox area length */
+	uint32_t repr_cap;               /**< Capabilities for representors */
+	uint32_t mbox_cmsg_types;        /**< Cmsgs which can be passed through the mailbox */
+	uint32_t vnic_stats_off;         /**< Offset of vNIC stats area */
+	uint32_t vnic_stats_cnt;         /**< Number of vNIC stats */
+};
+
 struct nfp_pf_dev {
 	/* Backpointer to associated pci device */
 	struct rte_pci_device *pci_dev;
@@ -163,6 +174,8 @@  struct nfp_net_hw {
 	uint8_t idx;
 	/* Internal port number as seen from NFP */
 	uint8_t nfp_idx;
+
+	struct nfp_net_tlv_caps tlv_caps;
 };
 
 struct nfp_net_adapter {
diff --git a/drivers/net/nfp/nfp_ctrl.c b/drivers/net/nfp/nfp_ctrl.c
new file mode 100644
index 0000000000..c61d3fd158
--- /dev/null
+++ b/drivers/net/nfp/nfp_ctrl.c
@@ -0,0 +1,153 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2023 Corigine Systems, Inc.
+ * All rights reserved.
+ */
+
+#include "nfp_ctrl.h"
+
+#include <ethdev_pci.h>
+
+#include "nfpcore/nfp_platform.h"
+
+#include "nfp_common.h"
+#include "nfp_logs.h"
+
+/* Mailbox Me clock freq */
+#define NFP_NET_CFG_MBOX_FREQ_MHZ             1200
+
+static void
+nfp_net_tlv_caps_reset(struct nfp_net_tlv_caps *caps)
+{
+	memset(caps, 0, sizeof(*caps));
+	caps->me_freq_mhz = NFP_NET_CFG_MBOX_FREQ_MHZ;
+	caps->mbox_off = NFP_NET_CFG_MBOX_BASE;
+	caps->mbox_len = NFP_NET_CFG_MBOX_VAL_MAX_SZ;
+}
+
+int
+nfp_net_tlv_caps_parse(struct rte_eth_dev *dev)
+{
+	uint32_t hdr;
+	uint8_t *end;
+	uint8_t *data;
+	uint32_t length;
+	uint32_t offset;
+	uint32_t tlv_type;
+	uint32_t tlv_offset;
+	struct nfp_net_hw *hw;
+	struct nfp_net_tlv_caps *caps;
+
+	hw = NFP_NET_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	caps = &hw->tlv_caps;
+	nfp_net_tlv_caps_reset(caps);
+
+	data = hw->ctrl_bar + NFP_NET_CFG_TLV_BASE;
+	end = hw->ctrl_bar + NFP_NET_CFG_BAR_SZ;
+
+	hdr = rte_read32(data);
+	if (hdr == 0) {
+		PMD_DRV_LOG(INFO, "TLV is empty!");
+		return 0;
+	}
+
+	for (; ; data += length) {
+		offset = data - hw->ctrl_bar;
+
+		if (data + NFP_NET_CFG_TLV_VALUE > end) {
+			PMD_DRV_LOG(ERR, "Reached end of BAR without END TLV");
+			return -EINVAL;
+		}
+
+		hdr = rte_read32(data);
+
+		length = FIELD_GET(NFP_NET_CFG_TLV_HEADER_LENGTH, hdr);
+		if ((length & (NFP_NET_CFG_TLV_LENGTH_INC - 1)) != 0) {
+			PMD_DRV_LOG(ERR, "TLV size not multiple of 4B len: %u", length);
+			return -EINVAL;
+		}
+
+		/* Advance past the header */
+		data += NFP_NET_CFG_TLV_VALUE;
+		if (data + length > end) {
+			PMD_DRV_LOG(ERR, "Oversized TLV offset: %u len: %u",
+					offset, length);
+			return -EINVAL;
+		}
+
+		tlv_type = FIELD_GET(NFP_NET_CFG_TLV_HEADER_TYPE, hdr);
+
+		switch (tlv_type) {
+		case NFP_NET_CFG_TLV_TYPE_UNKNOWN:
+			PMD_DRV_LOG(ERR, "Unknown TLV at offset: %u", offset);
+			return -EINVAL;
+		case NFP_NET_CFG_TLV_TYPE_RESERVED:
+			break;
+		case NFP_NET_CFG_TLV_TYPE_END:
+			if (length == 0)
+				return 0;
+
+			PMD_DRV_LOG(ERR, "END TLV should be empty, has len: %u", length);
+			return -EINVAL;
+		case NFP_NET_CFG_TLV_TYPE_ME_FREQ:
+			if (length != NFP_NET_CFG_TLV_LENGTH_INC) {
+				PMD_DRV_LOG(ERR, "ME FREQ TLV should be 4B, is %u", length);
+				return -EINVAL;
+			}
+
+			caps->me_freq_mhz = rte_read32(data);
+			break;
+		case NFP_NET_CFG_TLV_TYPE_MBOX:
+			caps->mbox_len = length;
+
+			if (length != 0)
+				caps->mbox_off = data - hw->ctrl_bar;
+			else
+				caps->mbox_off = 0;
+			break;
+		case NFP_NET_CFG_TLV_TYPE_EXPERIMENTAL0:
+			/* FALLTHROUGH */
+		case NFP_NET_CFG_TLV_TYPE_EXPERIMENTAL1:
+			PMD_DRV_LOG(ERR, "Experimental TLV type: %u offset: %u len: %u",
+					tlv_type, offset, length);
+			 /* Skip, not process */
+			break;
+		case NFP_NET_CFG_TLV_TYPE_REPR_CAP:
+			if (length == 0) {
+				PMD_DRV_LOG(ERR, "REPR CAP TLV with length 0");
+				return -EINVAL;
+			}
+
+			caps->repr_cap = rte_read32(data);
+			break;
+		case NFP_NET_CFG_TLV_TYPE_MBOX_CMSG_TYPES:
+			if (length != 0)
+				caps->mbox_cmsg_types = rte_read32(data);
+			break;
+		case NFP_NET_CFG_TLV_TYPE_CRYPTO_OPS:
+			break;
+		case NFP_NET_CFG_TLV_TYPE_VNIC_STATS:
+			tlv_offset = data - hw->ctrl_bar;
+			if ((tlv_offset & (NFP_NET_CFG_TLV_STATS_OFFSET - 1)) != 0) {
+				PMD_DRV_LOG(ERR, "VNIC STATS TLV misaligned, ignoring offset: %u len: %u",
+						offset, length);
+				break;
+			}
+
+			caps->vnic_stats_off = tlv_offset;
+			caps->vnic_stats_cnt = length / 10;
+			break;
+		case NFP_NET_CFG_TLV_TYPE_CRYPTO_OPS_RX_SCAN:
+			break;
+		default:
+			if (FIELD_GET(NFP_NET_CFG_TLV_HEADER_REQUIRED, hdr) == 0)
+				break;
+
+			PMD_DRV_LOG(ERR, "Unknown TLV type: %u offset: %u len: %u",
+					tlv_type, offset, length);
+			return -EINVAL;
+		}
+	}
+
+	PMD_DRV_LOG(ERR, "Reached end of BAR without END TLV");
+	return -EINVAL;
+}
diff --git a/drivers/net/nfp/nfp_ctrl.h b/drivers/net/nfp/nfp_ctrl.h
index 53727992a9..80a4196bea 100644
--- a/drivers/net/nfp/nfp_ctrl.h
+++ b/drivers/net/nfp/nfp_ctrl.h
@@ -8,6 +8,8 @@ 
 
 #include <stdint.h>
 
+#include <ethdev_driver.h>
+
 /*
  * Configuration BAR size.
  *
@@ -207,6 +209,9 @@  struct nfp_net_fw_ver {
 #define NFP_NET_CFG_RX_OFFSET		0x0050
 #define NFP_NET_CFG_RX_OFFSET_DYNAMIC		0	/* Prepend mode */
 
+/* Start anchor of the TLV area */
+#define NFP_NET_CFG_TLV_BASE            0x0058
+
 /**
  * Reuse spare address to contain the offset from the start of
  * the host buffer where the first byte of the received frame
@@ -434,6 +439,122 @@  struct nfp_net_fw_ver {
 
 #define NFP_PF_CSR_SLICE_SIZE	(32 * 1024)
 
+/*
+ * General use mailbox area (0x1800 - 0x19ff)
+ * 4B used for update command and 4B return code followed by
+ * a max of 504B of variable length value.
+ */
+#define NFP_NET_CFG_MBOX_BASE                 0x1800
+#define NFP_NET_CFG_MBOX_VAL                  0x1808
+#define NFP_NET_CFG_MBOX_VAL_MAX_SZ           0x1F8
+
+/*
+ * TLV capabilities
+ * @NFP_NET_CFG_TLV_TYPE:          Offset of type within the TLV
+ * @NFP_NET_CFG_TLV_TYPE_REQUIRED: Driver must be able to parse the TLV
+ * @NFP_NET_CFG_TLV_LENGTH:        Offset of length within the TLV
+ * @NFP_NET_CFG_TLV_LENGTH_INC:    TLV length increments
+ * @NFP_NET_CFG_TLV_VALUE:         Offset of value with the TLV
+ * @NFP_NET_CFG_TLV_STATS_OFFSET:  Length of TLV stats offset
+ *
+ * List of simple TLV structures, first one starts at @NFP_NET_CFG_TLV_BASE.
+ * Last structure must be of type @NFP_NET_CFG_TLV_TYPE_END. Presence of TLVs
+ * is indicated by @NFP_NET_CFG_TLV_BASE being non-zero. TLV structures may
+ * fill the entire remainder of the BAR or be shorter. FW must make sure TLVs
+ * don't conflict with other features which allocate space beyond
+ * @NFP_NET_CFG_TLV_BASE. @NFP_NET_CFG_TLV_TYPE_RESERVED should be used to wrap
+ * space used by such features.
+ *
+ * Note that the 4 byte TLV header is not counted in %NFP_NET_CFG_TLV_LENGTH.
+ */
+#define NFP_NET_CFG_TLV_TYPE                  0x00
+#define NFP_NET_CFG_TLV_TYPE_REQUIRED         0x8000
+#define NFP_NET_CFG_TLV_LENGTH                0x02
+#define NFP_NET_CFG_TLV_LENGTH_INC            4
+#define NFP_NET_CFG_TLV_VALUE                 0x04
+#define NFP_NET_CFG_TLV_STATS_OFFSET          0x08
+
+#define NFP_NET_CFG_TLV_HEADER_REQUIRED       0x80000000
+#define NFP_NET_CFG_TLV_HEADER_TYPE           0x7fff0000
+#define NFP_NET_CFG_TLV_HEADER_LENGTH         0x0000ffff
+
+/*
+ * Capability TLV types
+ *
+ * @NFP_NET_CFG_TLV_TYPE_UNKNOWN:
+ * Special TLV type to catch bugs, should never be encountered. Drivers should
+ * treat encountering this type as error and refuse to probe.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_RESERVED:
+ * Reserved space, may contain legacy fixed-offset fields, or be used for
+ * padding. The use of this type should be otherwise avoided.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_END:
+ * Empty, end of TLV list. Must be the last TLV. Drivers will stop processing
+ * further TLVs when encountered.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_ME_FREQ:
+ * Single word, ME frequency in MHz as used in calculation for
+ * @NFP_NET_CFG_RXR_IRQ_MOD and @NFP_NET_CFG_TXR_IRQ_MOD.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_MBOX:
+ * Variable, mailbox area. Overwrites the default location which is
+ * @NFP_NET_CFG_MBOX_BASE and length @NFP_NET_CFG_MBOX_VAL_MAX_SZ.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_EXPERIMENTAL0:
+ * @NFP_NET_CFG_TLV_TYPE_EXPERIMENTAL1:
+ * Variable, experimental IDs. IDs designated for internal development and
+ * experiments before a stable TLV ID has been allocated to a feature. Should
+ * never be present in production FW.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_REPR_CAP:
+ * Single word, equivalent of %NFP_NET_CFG_CAP for representors, features which
+ * can be used on representors.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_MBOX_CMSG_TYPES:
+ * Variable, bitmap of control message types supported by the mailbox handler.
+ * Bit 0 corresponds to message type 0, bit 1 to 1, etc. Control messages are
+ * encapsulated into simple TLVs, with an end TLV and written to the Mailbox.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_CRYPTO_OPS:
+ * 8 words, bitmaps of supported and enabled crypto operations.
+ * First 16B (4 words) contains a bitmap of supported crypto operations,
+ * and next 16B contain the enabled operations.
+ * This capability is obsoleted by ones with better sync methods.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_VNIC_STATS:
+ * Variable, per-vNIC statistics, data should be 8B aligned (FW should insert
+ * zero-length RESERVED TLV to pad).
+ * TLV data has two sections. First is an array of statistics' IDs (2B each).
+ * Second 8B statistics themselves. Statistics are 8B aligned, meaning there
+ * may be a padding between sections.
+ * Number of statistics can be determined as floor(tlv.length / (2 + 8)).
+ * This TLV overwrites %NFP_NET_CFG_STATS_* values (statistics in this TLV
+ * duplicate the old ones, so driver should be careful not to unnecessarily
+ * render both).
+ *
+ * @NFP_NET_CFG_TLV_TYPE_CRYPTO_OPS_RX_SCAN:
+ * Same as %NFP_NET_CFG_TLV_TYPE_CRYPTO_OPS, but crypto TLS does stream scan
+ * RX sync, rather than kernel-assisted sync.
+ *
+ * @NFP_NET_CFG_TLV_TYPE_CRYPTO_OPS_LENGTH:
+ * CRYPTO OPS TLV should be at least 32B.
+ */
+#define NFP_NET_CFG_TLV_TYPE_UNKNOWN            0
+#define NFP_NET_CFG_TLV_TYPE_RESERVED           1
+#define NFP_NET_CFG_TLV_TYPE_END                2
+#define NFP_NET_CFG_TLV_TYPE_ME_FREQ            3
+#define NFP_NET_CFG_TLV_TYPE_MBOX               4
+#define NFP_NET_CFG_TLV_TYPE_EXPERIMENTAL0      5
+#define NFP_NET_CFG_TLV_TYPE_EXPERIMENTAL1      6
+#define NFP_NET_CFG_TLV_TYPE_REPR_CAP           7
+#define NFP_NET_CFG_TLV_TYPE_MBOX_CMSG_TYPES    10
+#define NFP_NET_CFG_TLV_TYPE_CRYPTO_OPS         11
+#define NFP_NET_CFG_TLV_TYPE_VNIC_STATS         12
+#define NFP_NET_CFG_TLV_TYPE_CRYPTO_OPS_RX_SCAN 13
+
+int nfp_net_tlv_caps_parse(struct rte_eth_dev *dev);
+
 /*
  * nfp_net_cfg_ctrl_rss() - Get RSS flag based on firmware's capability
  * @hw_cap: The firmware's capabilities
diff --git a/drivers/net/nfp/nfp_ethdev.c b/drivers/net/nfp/nfp_ethdev.c
index 241595be9d..7dc93f7c43 100644
--- a/drivers/net/nfp/nfp_ethdev.c
+++ b/drivers/net/nfp/nfp_ethdev.c
@@ -549,6 +549,12 @@  nfp_net_init(struct rte_eth_dev *eth_dev)
 	if (err != 0)
 		return err;
 
+	err = nfp_net_tlv_caps_parse(eth_dev);
+	if (err != 0) {
+		PMD_INIT_LOG(ERR, "Failed to parser TLV caps");
+		return err;
+	}
+
 	nfp_net_ethdev_ops_mount(hw, eth_dev);
 
 	hw->eth_xstats_base = rte_malloc("rte_eth_xstat", sizeof(struct rte_eth_xstat) *