[3/3] net/ionic: add vdev support for embedded applications

  Add support for running DPDK applications directly on AMD Pensando
embedded HW. The platform exposes the device BARs through UIO. The
UIO code in the common/ionic library walks the sysfs filesystem
to identify the relevant BARs and map them into process memory.

The SoCs are named 'Capri' and 'Elba'.

The vdev device interface code is located in ionic_dev_vdev.c.

Some datapath operations are #ifdef-ed out to save on resources when
running in embedded mode.

Some controlpath operations are skipped by the ionic_is_embedded()
helper function.

Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
barrier inside rte_write64() is insufficient on these devices due to
a chip errata.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>
---
 config/arm/arm64_capri_linux_gcc   |  16 +++
 config/arm/arm64_elba_linux_gcc    |  16 +++
 config/arm/meson.build             |  42 ++++++++
 drivers/net/ionic/ionic.h          |   2 +-
 drivers/net/ionic/ionic_dev.h      |  17 ++++
 drivers/net/ionic/ionic_dev_vdev.c | 156 +++++++++++++++++++++++++++++
 drivers/net/ionic/ionic_ethdev.c   |   7 ++
 drivers/net/ionic/ionic_lif.c      |  19 ++++
 drivers/net/ionic/ionic_rxtx.h     |   4 +
 drivers/net/ionic/meson.build      |   1 +
 10 files changed, 279 insertions(+), 1 deletion(-)
 create mode 100644 config/arm/arm64_capri_linux_gcc
 create mode 100644 config/arm/arm64_elba_linux_gcc
 create mode 100644 drivers/net/ionic/ionic_dev_vdev.c
  

On 2/16/2024 5:07 PM, Andrew Boyer wrote:
> Add support for running DPDK applications directly on AMD Pensando
> embedded HW. The platform exposes the device BARs through UIO. The
> UIO code in the common/ionic library walks the sysfs filesystem
> to identify the relevant BARs and map them into process memory.
> 
> The SoCs are named 'Capri' and 'Elba'.
> 
> The vdev device interface code is located in ionic_dev_vdev.c.
> 
> Some datapath operations are #ifdef-ed out to save on resources when
> running in embedded mode.
> 
> Some controlpath operations are skipped by the ionic_is_embedded()
> helper function.
> 
> Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
> barrier inside rte_write64() is insufficient on these devices due to
> a chip errata.
> 
> Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
> Signed-off-by: Neel Patel <neel.patel@amd.com>
> Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
> Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>

<...>

> +static struct rte_vdev_driver rte_vdev_ionic_pmd = {
> +	.probe = eth_ionic_vdev_probe,
> +	.remove = eth_ionic_vdev_remove,
> +};
> +
> +RTE_PMD_REGISTER_VDEV(net_ionic, rte_vdev_ionic_pmd);
> +
> +static void
> +vdev_ionic_scan_cb(__rte_unused void *arg)
> +{
> +	ionic_uio_scan_mnet_devices();
> +}
> +
> +RTE_INIT(vdev_ionic_custom_add)
> +{
> +	rte_vdev_add_custom_scan(vdev_ionic_scan_cb, NULL);
> +}

Hi Andrew,

My understanding is 'rte_vdev_add_custom_scan()' to add a vdev
automatically (via rte_devargs_add()) before vdev scan starts.

As far as I can see you are not doing this, why callback is added?
Why not call 'ionic_uio_scan_mnet_devices()' within the
'eth_ionic_vdev_probe()'?
  

On Feb 19, 2024, at 10:24 AM, Yigit, Ferruh <Ferruh.Yigit@amd.com> wrote:

On 2/16/2024 5:07 PM, Andrew Boyer wrote:
Add support for running DPDK applications directly on AMD Pensando
embedded HW. The platform exposes the device BARs through UIO. The
UIO code in the common/ionic library walks the sysfs filesystem
to identify the relevant BARs and map them into process memory.

The SoCs are named 'Capri' and 'Elba'.

The vdev device interface code is located in ionic_dev_vdev.c.

Some datapath operations are #ifdef-ed out to save on resources when
running in embedded mode.

Some controlpath operations are skipped by the ionic_is_embedded()
helper function.

Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
barrier inside rte_write64() is insufficient on these devices due to
a chip errata.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>

<...>

+static struct rte_vdev_driver rte_vdev_ionic_pmd = {
+ .probe = eth_ionic_vdev_probe,
+ .remove = eth_ionic_vdev_remove,
+};
+
+RTE_PMD_REGISTER_VDEV(net_ionic, rte_vdev_ionic_pmd);
+
+static void
+vdev_ionic_scan_cb(__rte_unused void *arg)
+{
+ ionic_uio_scan_mnet_devices();
+}
+
+RTE_INIT(vdev_ionic_custom_add)
+{
+ rte_vdev_add_custom_scan(vdev_ionic_scan_cb, NULL);
+}

Hi Andrew,

My understanding is 'rte_vdev_add_custom_scan()' to add a vdev
automatically (via rte_devargs_add()) before vdev scan starts.

As far as I can see you are not doing this, why callback is added?
Why not call 'ionic_uio_scan_mnet_devices()' within the
'eth_ionic_vdev_probe()'?

I think you are correct and our scan should be restricted to the vdev_probe function. Otherwise it will run in all cases, even on irrelevant hardware.

Thanks!
Andrew
  

> On Feb 16, 2024, at 11:07 AM, Andrew Boyer <andrew.boyer@amd.com> wrote:
> 
> Add support for running DPDK applications directly on AMD Pensando
> embedded HW. The platform exposes the device BARs through UIO. The
> UIO code in the common/ionic library walks the sysfs filesystem
> to identify the relevant BARs and map them into process memory.
> 
> The SoCs are named 'Capri' and 'Elba'.
> 
> The vdev device interface code is located in ionic_dev_vdev.c.
> 
> Some datapath operations are #ifdef-ed out to save on resources when
> running in embedded mode.
> 
> Some controlpath operations are skipped by the ionic_is_embedded()
> helper function.
> 
> Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
> barrier inside rte_write64() is insufficient on these devices due to
> a chip errata.
> 
> Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
> Signed-off-by: Neel Patel <neel.patel@amd.com>
> Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
> Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>
> ---
> config/arm/arm64_capri_linux_gcc   |  16 +++
> config/arm/arm64_elba_linux_gcc    |  16 +++
> config/arm/meson.build             |  42 ++++++++
> drivers/net/ionic/ionic.h          |   2 +-
> drivers/net/ionic/ionic_dev.h      |  17 ++++
> drivers/net/ionic/ionic_dev_vdev.c | 156 +++++++++++++++++++++++++++++
> drivers/net/ionic/ionic_ethdev.c   |   7 ++
> drivers/net/ionic/ionic_lif.c      |  19 ++++
> drivers/net/ionic/ionic_rxtx.h     |   4 +
> drivers/net/ionic/meson.build      |   1 +
> 10 files changed, 279 insertions(+), 1 deletion(-)
> create mode 100644 config/arm/arm64_capri_linux_gcc
> create mode 100644 config/arm/arm64_elba_linux_gcc
> create mode 100644 drivers/net/ionic/ionic_dev_vdev.c
> 
> diff --git a/config/arm/arm64_capri_linux_gcc b/config/arm/arm64_capri_linux_gcc
> new file mode 100644
> index 0000000000..1a6313e684
> --- /dev/null
> +++ b/config/arm/arm64_capri_linux_gcc
> @@ -0,0 +1,16 @@
> +[binaries]
> +c = ['ccache', 'aarch64-linux-gnu-gcc']
> +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> +ar = 'aarch64-linux-gnu-gcc-ar'
> +strip = 'aarch64-linux-gnu-strip'
> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> +pcap-config = ''
> +
> +[host_machine]
> +system = 'linux'
> +cpu_family = 'aarch64'
> +cpu = 'armv8-a'
> +endian = 'little'
> +
> +[properties]
> +platform = 'capri'
> diff --git a/config/arm/arm64_elba_linux_gcc b/config/arm/arm64_elba_linux_gcc
> new file mode 100644
> index 0000000000..4d891bd5a7
> --- /dev/null
> +++ b/config/arm/arm64_elba_linux_gcc
> @@ -0,0 +1,16 @@
> +[binaries]
> +c = ['ccache', 'aarch64-linux-gnu-gcc']
> +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> +ar = 'aarch64-linux-gnu-gcc-ar'
> +strip = 'aarch64-linux-gnu-strip'
> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> +pcap-config = ''
> +
> +[host_machine]
> +system = 'linux'
> +cpu_family = 'aarch64'
> +cpu = 'armv8-a'
> +endian = 'little'
> +
> +[properties]
> +platform = 'elba'
> diff --git a/config/arm/meson.build b/config/arm/meson.build
> index 36f21d2259..2326021fed 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -165,6 +165,33 @@ implementer_cavium = {
>     }
> }
> 
> +implementer_ionic = {
> +    'description': 'AMD Pensando',
> +    'flags': [
> +        ['RTE_MAX_NUMA_NODES', 1],
> +        ['RTE_CACHE_LINE_SIZE', 64],
> +        ['RTE_LIBRTE_VHOST_NUMA', false],
> +        ['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
> +        ['RTE_LIBRTE_IONIC_PMD_EMBEDDED', true],
> +    ],
> +    'part_number_config': {
> +        '0xc1': {
> +            'compiler_options':  ['-mcpu=cortex-a72'],
> +            'flags': [
> +                ['RTE_MAX_LCORE', 4],
> +                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
> +            ]
> +        },
> +        '0xc2': {
> +            'compiler_options':  ['-mcpu=cortex-a72'],
> +            'flags': [
> +                ['RTE_MAX_LCORE', 16],
> +                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
> +            ]
> +        }
> +    }
> +}

Can you place it such that it is ordered alphabetically? (I understand that currently things are not ordered alphabetically, I have plans to fix that)

> +
> implementer_ampere = {
>     'description': 'Ampere Computing',
>     'flags': [
> @@ -294,6 +321,7 @@ implementers = {
>     '0x50': implementer_ampere,
>     '0x51': implementer_qualcomm,
>     '0x70': implementer_phytium,
> +    '0x75': implementer_ionic,
>     '0xc0': implementer_ampere,
> }
> 
> @@ -517,6 +545,18 @@ soc_bluefield3 = {
>    'numa': false
> }
> 
> +soc_ionic_capri = {
> +    'description': 'AMD Pensando Capri',
> +    'implementer': '0x75',
> +    'part_number': '0xc1'
> +}
> +
> +soc_ionic_elba = {
> +    'description': 'AMD Pensando Elba',
> +    'implementer': '0x75',
> +    'part_number': '0xc2'
> +}
> +
> '''
> Start of SoCs list
> generic:         Generic un-optimized build for armv8 aarch64 execution mode.
> @@ -576,6 +616,8 @@ socs = {
>     'thunderx2': soc_thunderx2,
>     'thunderxt88': soc_thunderxt88,
>     'thunderxt83': soc_thunderxt83,
> +    'capri': soc_ionic_capri,
> +    'elba': soc_ionic_elba,
Same here, place them in alphabetical order with respect to existing SOCs

> }
> 
> dpdk_conf.set('RTE_ARCH_ARM', 1)
> diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
> index a4a2e2756d..baa1322186 100644
> --- a/drivers/net/ionic/ionic.h
> +++ b/drivers/net/ionic/ionic.h
> @@ -15,7 +15,7 @@
> 
> #define IONIC_DRV_NAME "ionic"
> #define IONIC_DRV_DESCRIPTION "AMD Pensando Ethernet NIC Driver"
> -#define IONIC_DRV_VERSION "0.11.0-49"
> +#define IONIC_DRV_VERSION "1.3.0-112"
> 
> /* Vendor ID */
> #define IONIC_PENSANDO_VENDOR_ID 0x1dd8
> diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
> index b8eebcd181..3ec6aa5f6d 100644
> --- a/drivers/net/ionic/ionic_dev.h
> +++ b/drivers/net/ionic/ionic_dev.h
> @@ -171,6 +171,7 @@ struct ionic_dev_intf {
> struct rte_eth_dev *eth_dev);
> int  (*configure_intr)(struct ionic_adapter *adapter);
> void (*unconfigure_intr)(struct ionic_adapter *adapter);
> + void (*poll)(struct ionic_adapter *adapter);
> void (*unmap_bars)(struct ionic_adapter *adapter);
> };
> 
> @@ -245,7 +246,23 @@ ionic_q_flush(struct ionic_queue *q)
> {
> uint64_t val = IONIC_DBELL_QID(q->hw_index) | q->head_idx;
> 
> +#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA)
> + /* On some devices the standard 'dmb' barrier is insufficient */
> + asm volatile("dsb st" : : : "memory”);
Thanks for keeping this private to the driver.

> + rte_write64_relaxed(rte_cpu_to_le_64(val), q->db);
> +#else
> rte_write64(rte_cpu_to_le_64(val), q->db);
> +#endif
> +}
> +
> +static inline bool
> +ionic_is_embedded(void)
> +{
> +#if defined(RTE_LIBRTE_IONIC_PMD_EMBEDDED)
> + return true;
> +#else
> + return false;
> +#endif
> }
> 
> #endif /* _IONIC_DEV_H_ */
> diff --git a/drivers/net/ionic/ionic_dev_vdev.c b/drivers/net/ionic/ionic_dev_vdev.c
> new file mode 100644
> index 0000000000..31824c6c89
> --- /dev/null
> +++ b/drivers/net/ionic/ionic_dev_vdev.c
> @@ -0,0 +1,156 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2019-2024 Advanced Micro Devices, Inc.
> + */
> +
> +#include <stdint.h>
> +#include <stdlib.h>
> +#include <errno.h>
> +
> +#include <rte_errno.h>
> +#include <rte_common.h>
> +#include <rte_log.h>
> +#include <rte_eal.h>
> +#include <bus_vdev_driver.h>
> +#include <rte_dev.h>
> +#include <rte_string_fns.h>
> +#include <rte_kvargs.h>
> +
> +#include "ionic.h"
> +#include "ionic_common.h"
> +#include "ionic_logs.h"
> +#include "ionic_ethdev.h"
> +
> +#define IONIC_VDEV_DEV_BAR          0
> +#define IONIC_VDEV_INTR_CTL_BAR     1
> +#define IONIC_VDEV_INTR_CFG_BAR     2
> +#define IONIC_VDEV_DB_BAR           3
> +#define IONIC_VDEV_BARS_MAX         4
> +
> +#define IONIC_VDEV_DEV_INFO_REGS_OFFSET      0x0000
> +#define IONIC_VDEV_DEV_CMD_REGS_OFFSET       0x0800
> +
> +#define IONIC_VDEV_FW_WAIT_US       1000     /* 1ms */
> +#define IONIC_VDEV_FW_WAIT_MAX      5000     /* 5s */
> +
> +static int
> +ionic_vdev_setup(struct ionic_adapter *adapter)
> +{
> + struct ionic_bars *bars = &adapter->bars;
> + struct ionic_dev *idev = &adapter->idev;
> + uint8_t *bar0_base;
> + uint32_t sig;
> + uint32_t fw_waits = 0;
> + uint8_t fw;
> +
> + IONIC_PRINT_CALL();
> +
> + /* BAR0: dev_cmd and interrupts */
> + if (bars->num_bars < 1) {
> + IONIC_PRINT(ERR, "No bars found, aborting");
> + return -EFAULT;
> + }
> +
> + bar0_base = bars->bar[IONIC_VDEV_DEV_BAR].vaddr;
> + idev->dev_info = (union ionic_dev_info_regs *)
> + &bar0_base[IONIC_VDEV_DEV_INFO_REGS_OFFSET];
> + idev->dev_cmd = (union ionic_dev_cmd_regs *)
> + &bar0_base[IONIC_VDEV_DEV_CMD_REGS_OFFSET];
> + idev->intr_ctrl = (void *)bars->bar[IONIC_VDEV_INTR_CTL_BAR].vaddr;
> + idev->db_pages = (void *)bars->bar[IONIC_VDEV_DB_BAR].vaddr;
> +
> + sig = ioread32(&idev->dev_info->signature);
> + if (sig != IONIC_DEV_INFO_SIGNATURE) {
> + IONIC_PRINT(ERR, "Incompatible firmware signature %x", sig);
> + return -EFAULT;
> + }
> +
> + /* Wait for the FW to indicate readiness */
> + while (1) {
> + fw = ioread8(&idev->dev_info->fw_status);
> + if ((fw & IONIC_FW_STS_F_RUNNING) != 0)
> + break;
> +
> + if (fw_waits > IONIC_VDEV_FW_WAIT_MAX) {
> + IONIC_PRINT(ERR, "Firmware readiness bit not set");
> + return -ETIMEDOUT;
> + }
> +
> + fw_waits++;
> + rte_delay_us_block(IONIC_VDEV_FW_WAIT_US);
> + }
> + IONIC_PRINT(DEBUG, "Firmware ready (%u waits)", fw_waits);
> +
> + adapter->name = rte_vdev_device_name(adapter->bus_dev);
> +
> + return 0;
> +}
> +
> +static void
> +ionic_vdev_poll(struct ionic_adapter *adapter)
> +{
> + ionic_dev_interrupt_handler(adapter);
> +}
> +
> +static void
> +ionic_vdev_unmap_bars(struct ionic_adapter *adapter)
> +{
> + struct ionic_bars *bars = &adapter->bars;
> + uint32_t i;
> +
> + for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
> + ionic_uio_rel_rsrc(adapter->name, i, &bars->bar[i]);
> +}
> +
> +static const struct ionic_dev_intf ionic_vdev_intf = {
> + .setup = ionic_vdev_setup,
> + .poll = ionic_vdev_poll,
> + .unmap_bars = ionic_vdev_unmap_bars,
> +};
> +
> +static int
> +eth_ionic_vdev_probe(struct rte_vdev_device *vdev)
> +{
> + struct ionic_bars bars = {};
> + const char *name = rte_vdev_device_name(vdev);
> + unsigned int i;
> +
> + IONIC_PRINT(NOTICE, "Initializing device %s",
> + rte_eal_process_type() == RTE_PROC_SECONDARY ?
> + "[SECONDARY]" : "");
> +
> + for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
> + ionic_uio_get_rsrc(name, i, &bars.bar[i]);
> +
> + bars.num_bars = IONIC_VDEV_BARS_MAX;
> +
> + return eth_ionic_dev_probe((void *)vdev,
> + &vdev->device,
> + &bars,
> + &ionic_vdev_intf,
> + IONIC_DEV_ID_ETH_VF,
> + IONIC_PENSANDO_VENDOR_ID);
> +}
> +
> +static int
> +eth_ionic_vdev_remove(struct rte_vdev_device *vdev)
> +{
> + return eth_ionic_dev_remove(&vdev->device);
> +}
> +
> +static struct rte_vdev_driver rte_vdev_ionic_pmd = {
> + .probe = eth_ionic_vdev_probe,
> + .remove = eth_ionic_vdev_remove,
> +};
> +
> +RTE_PMD_REGISTER_VDEV(net_ionic, rte_vdev_ionic_pmd);
> +
> +static void
> +vdev_ionic_scan_cb(__rte_unused void *arg)
> +{
> + ionic_uio_scan_mnet_devices();
> +}
> +
> +RTE_INIT(vdev_ionic_custom_add)
> +{
> + rte_vdev_add_custom_scan(vdev_ionic_scan_cb, NULL);
> +}
> diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
> index 7e80751846..aa22b6a70d 100644
> --- a/drivers/net/ionic/ionic_ethdev.c
> +++ b/drivers/net/ionic/ionic_ethdev.c
> @@ -300,6 +300,13 @@ ionic_dev_link_update(struct rte_eth_dev *eth_dev,
> 
> IONIC_PRINT_CALL();
> 
> + /*
> + * There is no way to hook up the device interrupts in the vdev
> + * framework. Instead, poll for updates on the adapter.
> + */
> + if (adapter->intf && adapter->intf->poll)
> + (*adapter->intf->poll)(adapter);
> +
> /* Initialize */
> memset(&link, 0, sizeof(link));
> 
> diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
> index 93a1011772..7f02b67610 100644
> --- a/drivers/net/ionic/ionic_lif.c
> +++ b/drivers/net/ionic/ionic_lif.c
> @@ -540,6 +540,10 @@ ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu)
> },
> };
> 
> + /* Not needed for embedded applications */
> + if (ionic_is_embedded())
> + return 0;
> +
> return ionic_adminq_post_wait(lif, &ctx);
> }
> 
> @@ -975,6 +979,13 @@ ionic_lif_queue_identify(struct ionic_lif *lif)
> 
> memset(qti, 0, sizeof(*qti));
> 
> + if (ionic_is_embedded()) {
> + /* When embedded, FW will always match the driver */
> + qti->version = ionic_qtype_vers[qtype];
> + continue;
> + }
> +
> + /* On the host, query the FW for info */
> ionic_dev_cmd_queue_identify(idev, IONIC_LIF_TYPE_CLASSIC,
> qtype, ionic_qtype_vers[qtype]);
> err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
> @@ -1246,6 +1257,10 @@ ionic_lif_rss_setup(struct ionic_lif *lif)
> static void
> ionic_lif_rss_teardown(struct ionic_lif *lif)
> {
> + /* Not needed for embedded applications */
> + if (ionic_is_embedded())
> + return;
> +
> if (lif->rss_ind_tbl) {
> lif->rss_ind_tbl = NULL;
> lif->rss_ind_tbl_pa = 0;
> @@ -1770,6 +1785,10 @@ ionic_lif_set_name(struct ionic_lif *lif)
> },
> };
> 
> + /* Not needed for embedded applications */
> + if (ionic_is_embedded())
> + return;
> +
> memcpy(ctx.cmd.lif_setattr.name, lif->name,
> sizeof(ctx.cmd.lif_setattr.name) - 1);
> 
> diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
> index 63dffb7866..b3e4e5e5b3 100644
> --- a/drivers/net/ionic/ionic_rxtx.h
> +++ b/drivers/net/ionic/ionic_rxtx.h
> @@ -101,6 +101,7 @@ int ionic_rx_fill_sg(struct ionic_rx_qcq *rxq);
> static inline void
> ionic_rxq_flush(struct ionic_queue *q)
> {
> +#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
> struct ionic_rxq_desc *desc_base = q->base;
> struct ionic_rxq_desc *cmb_desc_base = q->cmb_base;
> 
> @@ -122,6 +123,7 @@ ionic_rxq_flush(struct ionic_queue *q)
> }
> q->cmb_head_idx = q->head_idx;
> }
> +#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
> 
> ionic_q_flush(q);
> }
> @@ -129,6 +131,7 @@ ionic_rxq_flush(struct ionic_queue *q)
> static inline void
> ionic_txq_flush(struct ionic_queue *q)
> {
> +#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
> struct ionic_txq_desc *desc_base = q->base;
> struct ionic_txq_desc *cmb_desc_base = q->cmb_base;
> 
> @@ -150,6 +153,7 @@ ionic_txq_flush(struct ionic_queue *q)
> }
> q->cmb_head_idx = q->head_idx;
> }
> +#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
> 
> ionic_q_flush(q);
> }
> diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
> index 9f735e353e..cc6d5ce4db 100644
> --- a/drivers/net/ionic/meson.build
> +++ b/drivers/net/ionic/meson.build
> @@ -12,6 +12,7 @@ deps += ['common_ionic']
> sources = files(
>         'ionic_dev.c',
>         'ionic_dev_pci.c',
> +        'ionic_dev_vdev.c',
>         'ionic_ethdev.c',
>         'ionic_lif.c',
>         'ionic_mac_api.c',
> -- 
> 2.17.1
>
  

On 2/16/2024 5:07 PM, Andrew Boyer wrote:
> +#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA)
> +	/* On some devices the standard 'dmb' barrier is insufficient */
> +	asm volatile("dsb st" : : : "memory");
> +	rte_write64_relaxed(rte_cpu_to_le_64(val), q->db);
> +#else
>  	rte_write64(rte_cpu_to_le_64(val), q->db);
> +#endif
>

Hi Honnappa, Ruifeng,

Should we have an API for the above asm instruction? What do you think?
  

> On Feb 20, 2024, at 2:28 AM, Ferruh Yigit <ferruh.yigit@amd.com> wrote:
> 
> On 2/16/2024 5:07 PM, Andrew Boyer wrote:
>> +#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA)
>> + /* On some devices the standard 'dmb' barrier is insufficient */
>> + asm volatile("dsb st" : : : "memory");
>> + rte_write64_relaxed(rte_cpu_to_le_64(val), q->db);
>> +#else
>> rte_write64(rte_cpu_to_le_64(val), q->db);
>> +#endif
>> 
> 
> Hi Honnappa, Ruifeng,
> 
> Should we have an API for the above asm instruction? What do you think?
We do not expect “dsb st” to be used as per the architecture in DPDK. This is being used here due to a hardware errata. Hence, we do not need an API for this.

>
  

On Feb 19, 2024, at 9:02 PM, Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


On Feb 16, 2024, at 11:07 AM, Andrew Boyer <andrew.boyer@amd.com> wrote:

Add support for running DPDK applications directly on AMD Pensando
embedded HW. The platform exposes the device BARs through UIO. The
UIO code in the common/ionic library walks the sysfs filesystem
to identify the relevant BARs and map them into process memory.

The SoCs are named 'Capri' and 'Elba'.

The vdev device interface code is located in ionic_dev_vdev.c.

Some datapath operations are #ifdef-ed out to save on resources when
running in embedded mode.

Some controlpath operations are skipped by the ionic_is_embedded()
helper function.

Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
barrier inside rte_write64() is insufficient on these devices due to
a chip errata.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>
---
config/arm/arm64_capri_linux_gcc   |  16 +++
config/arm/arm64_elba_linux_gcc    |  16 +++
config/arm/meson.build             |  42 ++++++++
drivers/net/ionic/ionic.h          |   2 +-
drivers/net/ionic/ionic_dev.h      |  17 ++++
drivers/net/ionic/ionic_dev_vdev.c | 156 +++++++++++++++++++++++++++++
drivers/net/ionic/ionic_ethdev.c   |   7 ++
drivers/net/ionic/ionic_lif.c      |  19 ++++
drivers/net/ionic/ionic_rxtx.h     |   4 +
drivers/net/ionic/meson.build      |   1 +
10 files changed, 279 insertions(+), 1 deletion(-)
create mode 100644 config/arm/arm64_capri_linux_gcc
create mode 100644 config/arm/arm64_elba_linux_gcc
create mode 100644 drivers/net/ionic/ionic_dev_vdev.c

diff --git a/config/arm/arm64_capri_linux_gcc b/config/arm/arm64_capri_linux_gcc
new file mode 100644
index 0000000000..1a6313e684
--- /dev/null
+++ b/config/arm/arm64_capri_linux_gcc
@@ -0,0 +1,16 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'capri'
diff --git a/config/arm/arm64_elba_linux_gcc b/config/arm/arm64_elba_linux_gcc
new file mode 100644
index 0000000000..4d891bd5a7
--- /dev/null
+++ b/config/arm/arm64_elba_linux_gcc
@@ -0,0 +1,16 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'elba'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 36f21d2259..2326021fed 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -165,6 +165,33 @@ implementer_cavium = {
   }
}

+implementer_ionic = {
+    'description': 'AMD Pensando',
+    'flags': [
+        ['RTE_MAX_NUMA_NODES', 1],
+        ['RTE_CACHE_LINE_SIZE', 64],
+        ['RTE_LIBRTE_VHOST_NUMA', false],
+        ['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
+        ['RTE_LIBRTE_IONIC_PMD_EMBEDDED', true],
+    ],
+    'part_number_config': {
+        '0xc1': {
+            'compiler_options':  ['-mcpu=cortex-a72'],
+            'flags': [
+                ['RTE_MAX_LCORE', 4],
+                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
+            ]
+        },
+        '0xc2': {
+            'compiler_options':  ['-mcpu=cortex-a72'],
+            'flags': [
+                ['RTE_MAX_LCORE', 16],
+                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
+            ]
+        }
+    }
+}

Can you place it such that it is ordered alphabetically? (I understand that currently things are not ordered alphabetically, I have plans to fix that)

I switched from 'soc_ionic_capri' and 'soc_ionic_elba' to the shorter 'soc_capri' and 'soc_elba', to better match other vendors, and sorted my entries. Please see V2 and let me know if anything else needs to change.

Thanks!
Andrew