List comments

GET /api/patches/73440/comments/
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "id": 115379,
        "web_url": "https://patches.dpdk.org/comment/115379/",
        "msgid": "<20200707152304.GL5869@platinum>",
        "date": "2020-07-07T15:23:04",
        "subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet Tx\n\tscheduling",
        "submitter": {
            "id": 8,
            "url": "https://patches.dpdk.org/api/people/8/",
            "name": "Olivier Matz",
            "email": "olivier.matz@6wind.com"
        },
        "content": "On Tue, Jul 07, 2020 at 02:57:11PM +0000, Viacheslav Ovsiienko wrote:\n> There is the requirement on some networks for precise traffic timing\n> management. The ability to send (and, generally speaking, receive)\n> the packets at the very precisely specified moment of time provides\n> the opportunity to support the connections with Time Division\n> Multiplexing using the contemporary general purpose NIC without involving\n> an auxiliary hardware. For example, the supporting of O-RAN Fronthaul\n> interface is one of the promising features for potentially usage of the\n> precise time management for the egress packets.\n> \n> The main objective of this RFC is to specify the way how applications\n> can provide the moment of time at what the packet transmission must be\n> started and to describe in preliminary the supporting this feature from\n> mlx5 PMD side.\n> \n> The new dynamic timestamp field is proposed, it provides some timing\n> information, the units and time references (initial phase) are not\n> explicitly defined but are maintained always the same for a given port.\n> Some devices allow to query rte_eth_read_clock() that will return\n> the current device timestamp. The dynamic timestamp flag tells whether\n> the field contains actual timestamp value. For the packets being sent\n> this value can be used by PMD to schedule packet sending.\n> \n> After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation\n> and obsoleting, these dynamic flag and field will be used to manage\n> the timestamps on receiving datapath as well.\n> \n> When PMD sees the \"rte_dynfield_timestamp\" set on the packet being sent\n> it tries to synchronize the time of packet appearing on the wire with\n> the specified packet timestamp. If the specified one is in the past it\n> should be ignored, if one is in the distant future it should be capped\n> with some reasonable value (in range of seconds). These specific cases\n> (\"too late\" and \"distant future\") can be optionally reported via\n> device xstats to assist applications to detect the time-related\n> problems.\n> \n> There is no any packet reordering according timestamps is supposed,\n> neither within packet burst, nor between packets, it is an entirely\n> application responsibility to generate packets and its timestamps\n> in desired order. The timestamps can be put only in the first packet\n> in the burst providing the entire burst scheduling.\n> \n> PMD reports the ability to synchronize packet sending on timestamp\n> with new offload flag:\n> \n> This is palliative and is going to be replaced with new eth_dev API\n> about reporting/managing the supported dynamic flags and its related\n> features. This API would break ABI compatibility and can't be introduced\n> at the moment, so is postponed to 20.11.\n> \n> For testing purposes it is proposed to update testpmd \"txonly\"\n> forwarding mode routine. With this update testpmd application generates\n> the packets and sets the dynamic timestamps according to specified time\n> pattern if it sees the \"rte_dynfield_timestamp\" is registered.\n> \n> The new testpmd command is proposed to configure sending pattern:\n> \n> set tx_times <burst_gap>,<intra_gap>\n> \n> <intra_gap> - the delay between the packets within the burst\n>               specified in the device clock units. The number\n>               of packets in the burst is defined by txburst parameter\n> \n> <burst_gap> - the delay between the bursts in the device clock units\n> \n> As the result the bursts of packet will be transmitted with specific\n> delays between the packets within the burst and specific delay between\n> the bursts. The rte_eth_get_clock is supposed to be engaged to get the\n> current device clock value and provide the reference for the timestamps.\n> \n> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>\n> ---\n>  v1->v4:\n>     - dedicated dynamic Tx timestamp flag instead of shared with Rx\n>     - Doxygen-style comment\n>     - comments update\n> \n> ---\n>  lib/librte_ethdev/rte_ethdev.c |  1 +\n>  lib/librte_ethdev/rte_ethdev.h |  4 ++++\n>  lib/librte_mbuf/rte_mbuf_dyn.h | 31 +++++++++++++++++++++++++++++++\n>  3 files changed, 36 insertions(+)\n> \n> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c\n> index 8e10a6f..02157d5 100644\n> --- a/lib/librte_ethdev/rte_ethdev.c\n> +++ b/lib/librte_ethdev/rte_ethdev.c\n> @@ -162,6 +162,7 @@ struct rte_eth_xstats_name_off {\n>  \tRTE_TX_OFFLOAD_BIT2STR(UDP_TNL_TSO),\n>  \tRTE_TX_OFFLOAD_BIT2STR(IP_TNL_TSO),\n>  \tRTE_TX_OFFLOAD_BIT2STR(OUTER_UDP_CKSUM),\n> +\tRTE_TX_OFFLOAD_BIT2STR(SEND_ON_TIMESTAMP),\n>  };\n>  \n>  #undef RTE_TX_OFFLOAD_BIT2STR\n> diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h\n> index a49242b..6f6454c 100644\n> --- a/lib/librte_ethdev/rte_ethdev.h\n> +++ b/lib/librte_ethdev/rte_ethdev.h\n> @@ -1178,6 +1178,10 @@ struct rte_eth_conf {\n>  /** Device supports outer UDP checksum */\n>  #define DEV_TX_OFFLOAD_OUTER_UDP_CKSUM  0x00100000\n>  \n> +/** Device supports send on timestamp */\n> +#define DEV_TX_OFFLOAD_SEND_ON_TIMESTAMP 0x00200000\n> +\n> +\n>  #define RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP 0x00000001\n>  /**< Device supports Rx queue setup after device started*/\n>  #define RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP 0x00000002\n> diff --git a/lib/librte_mbuf/rte_mbuf_dyn.h b/lib/librte_mbuf/rte_mbuf_dyn.h\n> index 96c3631..7e9f7d2 100644\n> --- a/lib/librte_mbuf/rte_mbuf_dyn.h\n> +++ b/lib/librte_mbuf/rte_mbuf_dyn.h\n> @@ -250,4 +250,35 @@ int rte_mbuf_dynflag_lookup(const char *name,\n>  #define RTE_MBUF_DYNFIELD_METADATA_NAME \"rte_flow_dynfield_metadata\"\n>  #define RTE_MBUF_DYNFLAG_METADATA_NAME \"rte_flow_dynflag_metadata\"\n>  \n> +/**\n> + * The timestamp dynamic field provides some timing information, the\n> + * units and time references (initial phase) are not explicitly defined\n> + * but are maintained always the same for a given port. Some devices allow4\n\nallow4 -> allow\n\n> + * to query rte_eth_read_clock() that will return the current device\n> + * timestamp. The dynamic Tx timestamp flag tells whether the field contains\n> + * actual timestamp value. For the packets being sent this value can be\n> + * used by PMD to schedule packet sending.\n> + *\n> + * After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation\n> + * and obsoleting, the dedicated Rx timestamp flag is supposed to be\n> + * introduced and the shared dynamic timestamp field will be used\n> + * to handle the timestamps on receiving datapath as well.\n> + */\n> +#define RTE_MBUF_DYNFIELD_TIMESTAMP_NAME \"rte_dynfield_timestamp\"\n> +\n> +/**\n> + * When PMD sees the RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME flag set on the\n> + * packet being sent it tries to synchronize the time of packet appearing\n> + * on the wire with the specified packet timestamp. If the specified one\n> + * is in the past it should be ignored, if one is in the distant future\n> + * it should be capped with some reasonable value (in range of seconds).\n> + *\n> + * There is no any packet reordering according to timestamps is supposed,\n> + * neither for packet within the burst, nor for the whole bursts, it is\n> + * an entirely application responsibility to generate packets and its\n> + * timestamps in desired order. The timestamps might be put only in\n> + * the first packet in the burst providing the entire burst scheduling.\n> + */\n> +#define RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME \"rte_dynflag_tx_timestamp\"\n> +\n\nAcked-by: Olivier Matz <olivier.matz@6wind.com>",
        "headers": {
            "Return-Path": "<dev-bounces@dpdk.org>",
            "References": "<1591771085-24959-1-git-send-email-viacheslavo@mellanox.com>\n <1594133832-12785-1-git-send-email-viacheslavo@mellanox.com>",
            "X-Mailman-Version": "2.1.15",
            "From": "Olivier Matz <olivier.matz@6wind.com>",
            "User-Agent": "Mutt/1.10.1 (2018-07-13)",
            "List-Post": "<mailto:dev@dpdk.org>",
            "X-BeenThere": "dev@dpdk.org",
            "Content-Type": "text/plain; charset=us-ascii",
            "Delivered-To": "patchwork@inbox.dpdk.org",
            "X-Original-To": "patchwork@inbox.dpdk.org",
            "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20161025;\n h=x-gm-message-state:date:from:to:cc:subject:message-id:references\n :mime-version:content-disposition:in-reply-to:user-agent;\n bh=D69hku9Wehgqg1K7ZoqFsb/qsqDHKVNQhpdimdy5HWo=;\n b=aAuTYP0CJLRYqyk12+Wa2TgvZVWx/QGo03pGlbQIJ3CRVdC7ihQcXI0iVXSRNc/nXa\n MuW7gxEzpBHmLlExZwoierZN/qRZwl+nqQDtFMX6RSfGlv5us3jtQHKwKknrZ+DwybP2\n fzBct4ZRHuEb9p/NiRA37w1zaEY9sFlBhG6EYYfh6ICrTV2Ii/RDGNQPgPqiSzrvK3cf\n FCMfU9PJhZLIcfules4lLAcs4cMDSv84JhSlbyQSUeZYV1zs4TaQ6E4MLMIAt3Aj4QXU\n PMFh1KgdP/Gu0CR8IerZiZtECeXaoy151bgS/t8P420pDNug7M0fJxqDK3y0rowe0vfU\n ojkQ==",
            "Received": [
                "from dpdk.org (dpdk.org [92.243.14.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id EC4EFA00BE;\n\tTue,  7 Jul 2020 17:23:07 +0200 (CEST)",
                "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id C11EB1DE6E;\n\tTue,  7 Jul 2020 17:23:07 +0200 (CEST)",
                "from mail-wr1-f65.google.com (mail-wr1-f65.google.com\n [209.85.221.65]) by dpdk.org (Postfix) with ESMTP id DDE7B1DE0F\n for <dev@dpdk.org>; Tue,  7 Jul 2020 17:23:05 +0200 (CEST)",
                "by mail-wr1-f65.google.com with SMTP id f7so42579844wrw.1\n for <dev@dpdk.org>; Tue, 07 Jul 2020 08:23:05 -0700 (PDT)",
                "from 6wind.com\n (2a01cb0c0005a600345636f7e65ed1a0.ipv6.abo.wanadoo.fr.\n [2a01:cb0c:5:a600:3456:36f7:e65e:d1a0])\n by smtp.gmail.com with ESMTPSA id c20sm1873796wrb.65.2020.07.07.08.23.04\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Tue, 07 Jul 2020 08:23:05 -0700 (PDT)"
            ],
            "Subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet Tx\n\tscheduling",
            "X-Received": "by 2002:a5d:5310:: with SMTP id\n e16mr52078136wrv.289.1594135385577;\n Tue, 07 Jul 2020 08:23:05 -0700 (PDT)",
            "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
            "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google;\n h=date:from:to:cc:subject:message-id:references:mime-version\n :content-disposition:in-reply-to:user-agent;\n bh=D69hku9Wehgqg1K7ZoqFsb/qsqDHKVNQhpdimdy5HWo=;\n b=cKfOsi48rQ14xFOxFb62h3LOnNOe6o5rTkgZtsey9SLhE7Ftfyx0LiXHZx6ILEtOXh\n 8Yep27SckYBGW3dYaqSLflfEduus9PnJEOdVHjyUOdCYx+nCjMUes6Gn9GlxFkI8pWT5\n zCdeMOU+bhCImKYE8+obGLPbd2J1MRhQ38ij+E7EAAPkqSh//4xo/gIWZl4k+xAvmpRp\n a6qZx+cy0ECKaOqQt90/NFKDbb3Wo+qPvwi8U1y0kj+Yr2fEaRXksdwnmvuR7d4GH9mX\n qrFhh3SH726kgnf1Ej93b/Z5WaMg1X+SqQ7qvNMNRttnyWfK9nwkSD9cd5riHVWytNSp\n 13wg==",
            "Message-ID": "<20200707152304.GL5869@platinum>",
            "Precedence": "list",
            "X-Gm-Message-State": "AOAM530CF3bw17nkCD7x0eQFCM5IZx6nrgjyBGXqAkoehE4Sn71ohjHR\n zmyy/taP6MxB9W9Tt375x+nj0g==",
            "Date": "Tue, 7 Jul 2020 17:23:04 +0200",
            "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
            "Sender": "\"dev\" <dev-bounces@dpdk.org>",
            "Errors-To": "dev-bounces@dpdk.org",
            "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>",
            "X-Google-Smtp-Source": "\n ABdhPJwLOc2sIJnzhClWcCpIVym7tdQkGidA/dyWlnLAol8RaBbMqYT6ud+5AR8H2rFNJrG2Cmohtw==",
            "Cc": "dev@dpdk.org, matan@mellanox.com, rasland@mellanox.com,\n bernard.iremonger@intel.com, thomas@mellanox.net",
            "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
            "In-Reply-To": "<1594133832-12785-1-git-send-email-viacheslavo@mellanox.com>",
            "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>",
            "MIME-Version": "1.0",
            "To": "Viacheslav Ovsiienko <viacheslavo@mellanox.com>",
            "Content-Disposition": "inline"
        }
    },
    {
        "id": 115534,
        "web_url": "https://patches.dpdk.org/comment/115534/",
        "msgid": "<98CBD80474FA8B44BF855DF32C47DC35C61109@smartserver.smartshare.dk>",
        "date": "2020-07-08T14:16:00",
        "subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n\tTxscheduling",
        "submitter": {
            "id": 591,
            "url": "https://patches.dpdk.org/api/people/591/",
            "name": "Morten Brørup",
            "email": "mb@smartsharesystems.com"
        },
        "content": "> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Viacheslav\n> Ovsiienko\n> Sent: Tuesday, July 7, 2020 4:57 PM\n> \n> There is the requirement on some networks for precise traffic timing\n> management. The ability to send (and, generally speaking, receive)\n> the packets at the very precisely specified moment of time provides\n> the opportunity to support the connections with Time Division\n> Multiplexing using the contemporary general purpose NIC without\n> involving\n> an auxiliary hardware. For example, the supporting of O-RAN Fronthaul\n> interface is one of the promising features for potentially usage of the\n> precise time management for the egress packets.\n> \n> The main objective of this RFC is to specify the way how applications\n> can provide the moment of time at what the packet transmission must be\n> started and to describe in preliminary the supporting this feature from\n> mlx5 PMD side.\n> \n> The new dynamic timestamp field is proposed, it provides some timing\n> information, the units and time references (initial phase) are not\n> explicitly defined but are maintained always the same for a given port.\n> Some devices allow to query rte_eth_read_clock() that will return\n> the current device timestamp. The dynamic timestamp flag tells whether\n> the field contains actual timestamp value. For the packets being sent\n> this value can be used by PMD to schedule packet sending.\n> \n> After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation\n> and obsoleting, these dynamic flag and field will be used to manage\n> the timestamps on receiving datapath as well.\n> \n> When PMD sees the \"rte_dynfield_timestamp\" set on the packet being sent\n> it tries to synchronize the time of packet appearing on the wire with\n> the specified packet timestamp. If the specified one is in the past it\n> should be ignored, if one is in the distant future it should be capped\n> with some reasonable value (in range of seconds). These specific cases\n> (\"too late\" and \"distant future\") can be optionally reported via\n> device xstats to assist applications to detect the time-related\n> problems.\n> \n> There is no any packet reordering according timestamps is supposed,\n> neither within packet burst, nor between packets, it is an entirely\n> application responsibility to generate packets and its timestamps\n> in desired order. The timestamps can be put only in the first packet\n> in the burst providing the entire burst scheduling.\n> \n> PMD reports the ability to synchronize packet sending on timestamp\n> with new offload flag:\n> \n> This is palliative and is going to be replaced with new eth_dev API\n> about reporting/managing the supported dynamic flags and its related\n> features. This API would break ABI compatibility and can't be\n> introduced\n> at the moment, so is postponed to 20.11.\n> \n> For testing purposes it is proposed to update testpmd \"txonly\"\n> forwarding mode routine. With this update testpmd application generates\n> the packets and sets the dynamic timestamps according to specified time\n> pattern if it sees the \"rte_dynfield_timestamp\" is registered.\n> \n> The new testpmd command is proposed to configure sending pattern:\n> \n> set tx_times <burst_gap>,<intra_gap>\n> \n> <intra_gap> - the delay between the packets within the burst\n>               specified in the device clock units. The number\n>               of packets in the burst is defined by txburst parameter\n> \n> <burst_gap> - the delay between the bursts in the device clock units\n> \n> As the result the bursts of packet will be transmitted with specific\n> delays between the packets within the burst and specific delay between\n> the bursts. The rte_eth_get_clock is supposed to be engaged to get the\n> current device clock value and provide the reference for the\n> timestamps.\n> \n> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>\n> ---\n>  v1->v4:\n>     - dedicated dynamic Tx timestamp flag instead of shared with Rx\n\nThe detailed description above should be updated to reflect that it is now two flags.\n\n>     - Doxygen-style comment\n>     - comments update\n> \n> ---\n>  lib/librte_ethdev/rte_ethdev.c |  1 +\n>  lib/librte_ethdev/rte_ethdev.h |  4 ++++\n>  lib/librte_mbuf/rte_mbuf_dyn.h | 31 +++++++++++++++++++++++++++++++\n>  3 files changed, 36 insertions(+)\n> \n> diff --git a/lib/librte_ethdev/rte_ethdev.c\n> b/lib/librte_ethdev/rte_ethdev.c\n> index 8e10a6f..02157d5 100644\n> --- a/lib/librte_ethdev/rte_ethdev.c\n> +++ b/lib/librte_ethdev/rte_ethdev.c\n> @@ -162,6 +162,7 @@ struct rte_eth_xstats_name_off {\n>  \tRTE_TX_OFFLOAD_BIT2STR(UDP_TNL_TSO),\n>  \tRTE_TX_OFFLOAD_BIT2STR(IP_TNL_TSO),\n>  \tRTE_TX_OFFLOAD_BIT2STR(OUTER_UDP_CKSUM),\n> +\tRTE_TX_OFFLOAD_BIT2STR(SEND_ON_TIMESTAMP),\n>  };\n> \n>  #undef RTE_TX_OFFLOAD_BIT2STR\n> diff --git a/lib/librte_ethdev/rte_ethdev.h\n> b/lib/librte_ethdev/rte_ethdev.h\n> index a49242b..6f6454c 100644\n> --- a/lib/librte_ethdev/rte_ethdev.h\n> +++ b/lib/librte_ethdev/rte_ethdev.h\n> @@ -1178,6 +1178,10 @@ struct rte_eth_conf {\n>  /** Device supports outer UDP checksum */\n>  #define DEV_TX_OFFLOAD_OUTER_UDP_CKSUM  0x00100000\n> \n> +/** Device supports send on timestamp */\n> +#define DEV_TX_OFFLOAD_SEND_ON_TIMESTAMP 0x00200000\n> +\n> +\n>  #define RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP 0x00000001\n>  /**< Device supports Rx queue setup after device started*/\n>  #define RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP 0x00000002\n> diff --git a/lib/librte_mbuf/rte_mbuf_dyn.h\n> b/lib/librte_mbuf/rte_mbuf_dyn.h\n> index 96c3631..7e9f7d2 100644\n> --- a/lib/librte_mbuf/rte_mbuf_dyn.h\n> +++ b/lib/librte_mbuf/rte_mbuf_dyn.h\n> @@ -250,4 +250,35 @@ int rte_mbuf_dynflag_lookup(const char *name,\n>  #define RTE_MBUF_DYNFIELD_METADATA_NAME \"rte_flow_dynfield_metadata\"\n>  #define RTE_MBUF_DYNFLAG_METADATA_NAME \"rte_flow_dynflag_metadata\"\n> \n> +/**\n> + * The timestamp dynamic field provides some timing information, the\n> + * units and time references (initial phase) are not explicitly\n> defined\n> + * but are maintained always the same for a given port. Some devices\n> allow4\n> + * to query rte_eth_read_clock() that will return the current device\n> + * timestamp. The dynamic Tx timestamp flag tells whether the field\n> contains\n> + * actual timestamp value. For the packets being sent this value can\n> be\n> + * used by PMD to schedule packet sending.\n> + *\n> + * After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation\n> + * and obsoleting, the dedicated Rx timestamp flag is supposed to be\n> + * introduced and the shared dynamic timestamp field will be used\n> + * to handle the timestamps on receiving datapath as well.\n> + */\n> +#define RTE_MBUF_DYNFIELD_TIMESTAMP_NAME \"rte_dynfield_timestamp\"\n\nThe description above should not say anything about the dynamic TX timestamp flag.\n\nPlease elaborate \"some timing information\", e.g. add \"... about when the packet was received\".\n\n> +\n> +/**\n> + * When PMD sees the RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME flag set on\n> the\n> + * packet being sent it tries to synchronize the time of packet\n> appearing\n> + * on the wire with the specified packet timestamp. If the specified\n> one\n> + * is in the past it should be ignored, if one is in the distant\n> future\n> + * it should be capped with some reasonable value (in range of\n> seconds).\n> + *\n> + * There is no any packet reordering according to timestamps is\n> supposed,\n> + * neither for packet within the burst, nor for the whole bursts, it\n> is\n> + * an entirely application responsibility to generate packets and its\n> + * timestamps in desired order. The timestamps might be put only in\n> + * the first packet in the burst providing the entire burst\n> scheduling.\n> + */\n> +#define RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME \"rte_dynflag_tx_timestamp\"\n> +\n>  #endif\n> --\n> 1.8.3.1\n> \n\nIt may be worth adding some documentation about how the clocks of the NICs are out of sync with the clock of the CPU, and are all drifting relatively.\n\nAnd those clocks are also out of sync with the actual time (NTP clock).\n\nPreferably, some sort of cookbook for handling this should be provided. PCAP could be used as an example.",
        "headers": {
            "Return-Path": "<dev-bounces@dpdk.org>",
            "MIME-Version": "1.0",
            "X-Mailman-Version": "2.1.15",
            "From": "=?utf-8?q?Morten_Br=C3=B8rup?= <mb@smartsharesystems.com>",
            "List-Post": "<mailto:dev@dpdk.org>",
            "Thread-Index": "AdZUbusvfEPPTCcdTXCH1i2M/xgKLgAwSpJQ",
            "Message-ID": "<98CBD80474FA8B44BF855DF32C47DC35C61109@smartserver.smartshare.dk>",
            "References": "<1591771085-24959-1-git-send-email-viacheslavo@mellanox.com>\n <1594133832-12785-1-git-send-email-viacheslavo@mellanox.com>",
            "X-Original-To": "patchwork@inbox.dpdk.org",
            "Received": [
                "from dpdk.org (dpdk.org [92.243.14.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 0A62DA0526;\n\tWed,  8 Jul 2020 16:16:05 +0200 (CEST)",
                "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 3D3291DD8A;\n\tWed,  8 Jul 2020 16:16:04 +0200 (CEST)",
                "from smartserver.smartsharesystems.com\n (smartserver.smartsharesystems.com [77.243.40.215])\n by dpdk.org (Postfix) with ESMTP id 7B50C1DB32\n for <dev@dpdk.org>; Wed,  8 Jul 2020 16:16:02 +0200 (CEST)"
            ],
            "Subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n\tTxscheduling",
            "Sender": "\"dev\" <dev-bounces@dpdk.org>",
            "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
            "Content-class": "urn:content-classes:message",
            "Content-Type": "text/plain;\n\tcharset=\"UTF-8\"",
            "X-MS-Has-Attach": "",
            "X-BeenThere": "dev@dpdk.org",
            "Date": "Wed, 8 Jul 2020 16:16:00 +0200",
            "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
            "Content-Transfer-Encoding": "base64",
            "Errors-To": "dev-bounces@dpdk.org",
            "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>",
            "Cc": "<matan@mellanox.com>, <rasland@mellanox.com>, <olivier.matz@6wind.com>,\n <bernard.iremonger@intel.com>, <thomas@mellanox.net>",
            "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
            "Precedence": "list",
            "Delivered-To": "patchwork@inbox.dpdk.org",
            "In-Reply-To": "<1594133832-12785-1-git-send-email-viacheslavo@mellanox.com>",
            "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>",
            "X-MimeOLE": "Produced By Microsoft Exchange V6.5",
            "To": "\"Viacheslav Ovsiienko\" <viacheslavo@mellanox.com>,\n\t<dev@dpdk.org>",
            "Thread-Topic": "[dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n Txscheduling",
            "X-MS-TNEF-Correlator": ""
        }
    },
    {
        "id": 115544,
        "web_url": "https://patches.dpdk.org/comment/115544/",
        "msgid": "<AM4PR05MB3265F08B5462C8278296A60DD2670@AM4PR05MB3265.eurprd05.prod.outlook.com>",
        "date": "2020-07-08T14:54:15",
        "subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n Txscheduling",
        "submitter": {
            "id": 1102,
            "url": "https://patches.dpdk.org/api/people/1102/",
            "name": "Slava Ovsiienko",
            "email": "viacheslavo@mellanox.com"
        },
        "content": "Hi, Morten\n\nThank you for the comments. Please, see below.\n\n> -----Original Message-----\n> From: Morten Brørup <mb@smartsharesystems.com>\n> Sent: Wednesday, July 8, 2020 17:16\n> To: Slava Ovsiienko <viacheslavo@mellanox.com>; dev@dpdk.org\n> Cc: Matan Azrad <matan@mellanox.com>; Raslan Darawsheh\n> <rasland@mellanox.com>; olivier.matz@6wind.com;\n> bernard.iremonger@intel.com; thomas@mellanox.net\n> Subject: RE: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n> Txscheduling\n> \n> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Viacheslav\n> > Ovsiienko\n> > Sent: Tuesday, July 7, 2020 4:57 PM\n> >\n> > There is the requirement on some networks for precise traffic timing\n> > management. The ability to send (and, generally speaking, receive) the\n> > packets at the very precisely specified moment of time provides the\n> > opportunity to support the connections with Time Division Multiplexing\n> > using the contemporary general purpose NIC without involving an\n> > auxiliary hardware. For example, the supporting of O-RAN Fronthaul\n> > interface is one of the promising features for potentially usage of\n> > the precise time management for the egress packets.\n> >\n> > The main objective of this RFC is to specify the way how applications\n> > can provide the moment of time at what the packet transmission must be\n> > started and to describe in preliminary the supporting this feature\n> > from\n> > mlx5 PMD side.\n> >\n> > The new dynamic timestamp field is proposed, it provides some timing\n> > information, the units and time references (initial phase) are not\n> > explicitly defined but are maintained always the same for a given port.\n> > Some devices allow to query rte_eth_read_clock() that will return the\n> > current device timestamp. The dynamic timestamp flag tells whether the\n> > field contains actual timestamp value. For the packets being sent this\n> > value can be used by PMD to schedule packet sending.\n> >\n> > After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation and\n> > obsoleting, these dynamic flag and field will be used to manage the\n> > timestamps on receiving datapath as well.\n> >\n> > When PMD sees the \"rte_dynfield_timestamp\" set on the packet being\n> > sent it tries to synchronize the time of packet appearing on the wire\n> > with the specified packet timestamp. If the specified one is in the\n> > past it should be ignored, if one is in the distant future it should\n> > be capped with some reasonable value (in range of seconds). These\n> > specific cases (\"too late\" and \"distant future\") can be optionally\n> > reported via device xstats to assist applications to detect the\n> > time-related problems.\n> >\n> > There is no any packet reordering according timestamps is supposed,\n> > neither within packet burst, nor between packets, it is an entirely\n> > application responsibility to generate packets and its timestamps in\n> > desired order. The timestamps can be put only in the first packet in\n> > the burst providing the entire burst scheduling.\n> >\n> > PMD reports the ability to synchronize packet sending on timestamp\n> > with new offload flag:\n> >\n> > This is palliative and is going to be replaced with new eth_dev API\n> > about reporting/managing the supported dynamic flags and its related\n> > features. This API would break ABI compatibility and can't be\n> > introduced at the moment, so is postponed to 20.11.\n> >\n> > For testing purposes it is proposed to update testpmd \"txonly\"\n> > forwarding mode routine. With this update testpmd application\n> > generates the packets and sets the dynamic timestamps according to\n> > specified time pattern if it sees the \"rte_dynfield_timestamp\" is registered.\n> >\n> > The new testpmd command is proposed to configure sending pattern:\n> >\n> > set tx_times <burst_gap>,<intra_gap>\n> >\n> > <intra_gap> - the delay between the packets within the burst\n> >               specified in the device clock units. The number\n> >               of packets in the burst is defined by txburst parameter\n> >\n> > <burst_gap> - the delay between the bursts in the device clock units\n> >\n> > As the result the bursts of packet will be transmitted with specific\n> > delays between the packets within the burst and specific delay between\n> > the bursts. The rte_eth_get_clock is supposed to be engaged to get the\n> > current device clock value and provide the reference for the\n> > timestamps.\n> >\n> > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>\n> > ---\n> >  v1->v4:\n> >     - dedicated dynamic Tx timestamp flag instead of shared with Rx\n> \n> The detailed description above should be updated to reflect that it is now\n> two flags.\nOK\n\n> \n> >     - Doxygen-style comment\n> >     - comments update\n> >\n> > ---\n> >  lib/librte_ethdev/rte_ethdev.c |  1 +  lib/librte_ethdev/rte_ethdev.h\n> > |  4 ++++  lib/librte_mbuf/rte_mbuf_dyn.h | 31\n> > +++++++++++++++++++++++++++++++\n> >  3 files changed, 36 insertions(+)\n> >\n> > diff --git a/lib/librte_ethdev/rte_ethdev.c\n> > b/lib/librte_ethdev/rte_ethdev.c index 8e10a6f..02157d5 100644\n> > --- a/lib/librte_ethdev/rte_ethdev.c\n> > +++ b/lib/librte_ethdev/rte_ethdev.c\n> > @@ -162,6 +162,7 @@ struct rte_eth_xstats_name_off {\n> >  \tRTE_TX_OFFLOAD_BIT2STR(UDP_TNL_TSO),\n> >  \tRTE_TX_OFFLOAD_BIT2STR(IP_TNL_TSO),\n> >  \tRTE_TX_OFFLOAD_BIT2STR(OUTER_UDP_CKSUM),\n> > +\tRTE_TX_OFFLOAD_BIT2STR(SEND_ON_TIMESTAMP),\n> >  };\n> >\n> >  #undef RTE_TX_OFFLOAD_BIT2STR\n> > diff --git a/lib/librte_ethdev/rte_ethdev.h\n> > b/lib/librte_ethdev/rte_ethdev.h index a49242b..6f6454c 100644\n> > --- a/lib/librte_ethdev/rte_ethdev.h\n> > +++ b/lib/librte_ethdev/rte_ethdev.h\n> > @@ -1178,6 +1178,10 @@ struct rte_eth_conf {\n> >  /** Device supports outer UDP checksum */  #define\n> > DEV_TX_OFFLOAD_OUTER_UDP_CKSUM  0x00100000\n> >\n> > +/** Device supports send on timestamp */ #define\n> > +DEV_TX_OFFLOAD_SEND_ON_TIMESTAMP 0x00200000\n> > +\n> > +\n> >  #define RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP 0x00000001\n> /**<\n> > Device supports Rx queue setup after device started*/  #define\n> > RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP 0x00000002 diff --git\n> > a/lib/librte_mbuf/rte_mbuf_dyn.h b/lib/librte_mbuf/rte_mbuf_dyn.h\n> > index 96c3631..7e9f7d2 100644\n> > --- a/lib/librte_mbuf/rte_mbuf_dyn.h\n> > +++ b/lib/librte_mbuf/rte_mbuf_dyn.h\n> > @@ -250,4 +250,35 @@ int rte_mbuf_dynflag_lookup(const char *name,\n> > #define RTE_MBUF_DYNFIELD_METADATA_NAME\n> \"rte_flow_dynfield_metadata\"\n> >  #define RTE_MBUF_DYNFLAG_METADATA_NAME\n> \"rte_flow_dynflag_metadata\"\n> >\n> > +/**\n> > + * The timestamp dynamic field provides some timing information, the\n> > + * units and time references (initial phase) are not explicitly\n> > defined\n> > + * but are maintained always the same for a given port. Some devices\n> > allow4\n> > + * to query rte_eth_read_clock() that will return the current device\n> > + * timestamp. The dynamic Tx timestamp flag tells whether the field\n> > contains\n> > + * actual timestamp value. For the packets being sent this value can\n> > be\n> > + * used by PMD to schedule packet sending.\n> > + *\n> > + * After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation\n> > + * and obsoleting, the dedicated Rx timestamp flag is supposed to be\n> > + * introduced and the shared dynamic timestamp field will be used\n> > + * to handle the timestamps on receiving datapath as well.\n> > + */\n> > +#define RTE_MBUF_DYNFIELD_TIMESTAMP_NAME\n> \"rte_dynfield_timestamp\"\n> \n> The description above should not say anything about the dynamic TX\n> timestamp flag.\nIt does not. Or do you mean RX? \nNot sure, field and flag are tightly coupled,\nit is nice to mention this relation for better understanding. \nAnd mentioning the RX explains why it is not like this:\nRTE_MBUF_DYNFIELD_[TX]_TIMESTAMP_NAME\n\n> \n> Please elaborate \"some timing information\", e.g. add \"... about when the\n> packet was received\".\n\nSorry, I do not follow,  currently the dynamic field is not\n\"about when the packet was received\". Now it is introduced for Tx\nonly and just the opportunity to be shared with Rx one in coming releases\nis mentioned. \"Some\" means - not specified (herein) exactly.\nAnd it is elaborated what Is not specified and how it is supposed\nto use Tx timestamp.\n\n> \n> > +\n> > +/**\n> > + * When PMD sees the RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME flag\n> set on\n> > the\n> > + * packet being sent it tries to synchronize the time of packet\n> > appearing\n> > + * on the wire with the specified packet timestamp. If the specified\n> > one\n> > + * is in the past it should be ignored, if one is in the distant\n> > future\n> > + * it should be capped with some reasonable value (in range of\n> > seconds).\n> > + *\n> > + * There is no any packet reordering according to timestamps is\n> > supposed,\n> > + * neither for packet within the burst, nor for the whole bursts, it\n> > is\n> > + * an entirely application responsibility to generate packets and its\n> > + * timestamps in desired order. The timestamps might be put only in\n> > + * the first packet in the burst providing the entire burst\n> > scheduling.\n> > + */\n> > +#define RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME\n> \"rte_dynflag_tx_timestamp\"\n> > +\n> >  #endif\n> > --\n> > 1.8.3.1\n> >\n> \n> It may be worth adding some documentation about how the clocks of the\n> NICs are out of sync with the clock of the CPU, and are all drifting relatively.\n> \n> And those clocks are also out of sync with the actual time (NTP clock).\n\nIMO, It is out of scope of this very generic patch.  As for mlx NICs - the internal device\nclock might be (or might be not) synchronized with PTP, it can provide timestamps\nin real nanoseconds in various formats or just some free running counter.\nOn some systems the NIC and CPU might share the same clock source (for their PLL inputs\nfor example) and there will be no any drifts. As we can see - it is a wide and interesting \nopic to discuss, but, IMO,  the comment in header file might be not the most relevant\nplace to do. As for mlx5 devices clock specifics - it will be documented in PMD chapter.\n\nOK, will add few generic words, the few ones - in order not to make comment wordy, just\npoint the direction for further thinking.\n\n> \n> Preferably, some sort of cookbook for handling this should be provided.\n> PCAP could be used as an example.\n> \ntestpmd example is included in series, mlx5 PMD patch is prepared and coming soon.\n\nWith best regards, Slava",
        "headers": {
            "X-OriginatorOrg": "Mellanox.com",
            "x-ms-office365-filtering-correlation-id": "e7ead967-61e8-4f9a-661e-08d8234ec8bc",
            "References": "<1591771085-24959-1-git-send-email-viacheslavo@mellanox.com>\n <1594133832-12785-1-git-send-email-viacheslavo@mellanox.com>\n <98CBD80474FA8B44BF855DF32C47DC35C61109@smartserver.smartshare.dk>",
            "X-Mailman-Version": "2.1.15",
            "X-MS-Exchange-CrossTenant-mailboxtype": "HOSTED",
            "X-MS-Exchange-CrossTenant-AuthAs": "Internal",
            "X-MS-Exchange-CrossTenant-userprincipalname": "\n 7uG6oxDTwq6WgQULHelYvn0LOzHB2OKdScNZ4jA7ZKErCgHba4tK+bbotZZdL/m9wZFugH9/5YrFcp5HJFFkQhI1zFz9FqmqbgKGlUAvEx4=",
            "x-ms-oob-tlc-oobclassifiers": "OLM:10000;",
            "Thread-Index": "AQHWVTJRtpVhbms7BUGtcj8QA0ZT26j9vBYw",
            "Content-Type": "text/plain; charset=\"utf-8\"",
            "CC": "Matan Azrad <matan@mellanox.com>, Raslan Darawsheh <rasland@mellanox.com>,\n \"olivier.matz@6wind.com\" <olivier.matz@6wind.com>,\n \"bernard.iremonger@intel.com\" <bernard.iremonger@intel.com>,\n \"thomas@mellanox.net\" <thomas@mellanox.net>",
            "Thread-Topic": "[dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n Txscheduling",
            "Received": [
                "from dpdk.org (dpdk.org [92.243.14.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 0BB8AA0527;\n\tWed,  8 Jul 2020 16:54:20 +0200 (CEST)",
                "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id A18EC1D9AB;\n\tWed,  8 Jul 2020 16:54:19 +0200 (CEST)",
                "from EUR04-DB3-obe.outbound.protection.outlook.com\n (mail-eopbgr60072.outbound.protection.outlook.com [40.107.6.72])\n by dpdk.org (Postfix) with ESMTP id D6DE01D72D\n for <dev@dpdk.org>; Wed,  8 Jul 2020 16:54:17 +0200 (CEST)",
                "from AM4PR05MB3265.eurprd05.prod.outlook.com (2603:10a6:205:8::26)\n by AM0PR05MB6083.eurprd05.prod.outlook.com (2603:10a6:208:127::10) with\n Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.24; Wed, 8 Jul\n 2020 14:54:15 +0000",
                "from AM4PR05MB3265.eurprd05.prod.outlook.com\n ([fe80::194e:dc46:7543:50ed]) by AM4PR05MB3265.eurprd05.prod.outlook.com\n ([fe80::194e:dc46:7543:50ed%2]) with mapi id 15.20.3174.022; Wed, 8 Jul 2020\n 14:54:15 +0000"
            ],
            "Subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n Txscheduling",
            "ARC-Seal": "i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;\n b=TVp4LIlmuT09UqyDaWqArxGB5vpBQ6uWdm/kYUJ/K0m1o8Tz3vYAc5f9n1rocevqdWt5UUdl88K9LH/tki/R5NU8StAs1W353wPmicc02U6sYFsEXPjXo9j8PLDII5BMQ7MqJT9tNxyXmwgbM6X7PBWfWiVAzNGTjgToBX5C4/AuJY5o2LZ65bUezOyu+RIy/c68ZXj5kwBAWpREaRTYX6/Lrtv+T1Jiln1DY8rSOnc+EXEkMwlmEJYfaghHyoTZYd2UKCL+01rZ4cGKQ91IsH1sZJ9sFBWFBdFiTHIIII2nSPMHSNKQVJORK/CqERMFX0RvBWjS94D7dIUsrSTXWA==",
            "authentication-results": "smartsharesystems.com; dkim=none (message not signed)\n header.d=none; smartsharesystems.com;\n dmarc=none action=none header.from=mellanox.com;",
            "x-ms-traffictypediagnostic": "AM0PR05MB6083:",
            "X-MS-Exchange-CrossTenant-originalarrivaltime": "08 Jul 2020 14:54:15.3827 (UTC)",
            "Precedence": "list",
            "Date": "Wed, 8 Jul 2020 14:54:15 +0000",
            "x-ms-office365-filtering-ht": "Tenant",
            "x-forefront-antispam-report": "CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:AM4PR05MB3265.eurprd05.prod.outlook.com; PTR:; CAT:NONE;\n SFTY:;\n SFS:(4636009)(396003)(136003)(376002)(39860400002)(346002)(366004)(26005)(83380400001)(86362001)(5660300002)(6506007)(9686003)(186003)(54906003)(8936002)(110136005)(2906002)(55016002)(76116006)(316002)(66946007)(8676002)(53546011)(30864003)(52536014)(4326008)(478600001)(66574015)(66556008)(66476007)(64756008)(71200400001)(66446008)(7696005)(33656002);\n DIR:OUT; SFP:1101;",
            "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>",
            "x-ms-exchange-transport-forked": "True",
            "x-microsoft-antispam-prvs": "\n <AM0PR05MB6083400F19BE2E7685BE7528D2670@AM0PR05MB6083.eurprd05.prod.outlook.com>",
            "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>",
            "X-BeenThere": "dev@dpdk.org",
            "x-forefront-prvs": "04583CED1A",
            "Content-Transfer-Encoding": "base64",
            "To": "=?utf-8?q?Morten_Br=C3=B8rup?= <mb@smartsharesystems.com>,\n \"dev@dpdk.org\" <dev@dpdk.org>",
            "X-MS-TNEF-Correlator": "",
            "Return-Path": "<dev-bounces@dpdk.org>",
            "x-microsoft-antispam-message-info": "\n jBin3UZj/YLMlVkZvf02BbNa0FGz7k5WG8ZElu05ftxHiDqwiXjv5qG/JSzbJsILzENacr2gAyV0OM2GieO8k3te65TCtufX4yW9AmUDDGesKWmGuonHtBmzcytDnfMfoIdqfzVc9Riyxjlx51VngVyHOQmwAc1amgXq64aCbwOpQhwojyBj4m5IqplER1ttVFpeuQRRIAH3SQRKM6W7d76wN4JxROpdjCswzCPLqTzZmG3uGfJv+dWQMAhFxsrrOyPPox7UeyA+iCvSofyII92n5XEQj0Kut9rluIdYhVNzZlxcUxLRKT8DJM0BXXndxdvijaXvf5XVOM7wT0IiGg==",
            "x-ld-processed": "a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtFwd",
            "MIME-Version": "1.0",
            "List-Post": "<mailto:dev@dpdk.org>",
            "X-MS-Exchange-CrossTenant-AuthSource": "AM4PR05MB3265.eurprd05.prod.outlook.com",
            "From": "Slava Ovsiienko <viacheslavo@mellanox.com>",
            "x-originating-ip": "[95.164.10.10]",
            "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
            "x-ms-publictraffictype": "Email",
            "X-MS-Exchange-CrossTenant-fromentityheader": "Hosted",
            "Accept-Language": "en-US",
            "Message-ID": "\n <AM4PR05MB3265F08B5462C8278296A60DD2670@AM4PR05MB3265.eurprd05.prod.outlook.com>",
            "x-ms-exchange-senderadcheck": "1",
            "Delivered-To": "patchwork@inbox.dpdk.org",
            "X-Original-To": "patchwork@inbox.dpdk.org",
            "X-MS-Exchange-CrossTenant-Network-Message-Id": "\n e7ead967-61e8-4f9a-661e-08d8234ec8bc",
            "ARC-Authentication-Results": "i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com;\n dkim=pass header.d=mellanox.com; arc=none",
            "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
            "Sender": "\"dev\" <dev-bounces@dpdk.org>",
            "X-MS-Exchange-CrossTenant-id": "a652971c-7d2e-4d9b-a6a4-d149256f461b",
            "Content-Language": "en-US",
            "X-MS-Has-Attach": "",
            "X-MS-Exchange-Transport-CrossTenantHeadersStamped": "AM0PR05MB6083",
            "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;\n s=selector1;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=iWEfmlBxWoIVK4wME7gSl0K7Wu8gxDe+C6q+FJWAzS0=;\n b=XaEDASg395KsINgnxG+aoCIrVMchX89dCyDUSrMomPyWAd5G2x3hxGlfVRrWtdjLLCn5LkD70Uh88TxteuBr7UMHDX/bja9OcVTYZuLFklPcUVdPgbVkVSFj3x8BxgHSydyBEuk/XvCT8u5eFkvu82aUjZStsqtnlkwURX5EMKU=",
            "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
            "Errors-To": "dev-bounces@dpdk.org",
            "ARC-Message-Signature": "i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector9901;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=iWEfmlBxWoIVK4wME7gSl0K7Wu8gxDe+C6q+FJWAzS0=;\n b=kKgafDhv0t33hHoqBB98x1EzLUduomf42bKY/e6OlZfh1Rqw5XE2GLdqyNHwCdwtQL5a5g+yY/oYtPH69elmGkrLLcFSnREHeGvh1TyN/C5GRMP4ObKEliS3D/eZqtn8KRrzbjuS/DXbiihqoNK5PetHagXQKGo7uPDYPkOT2ktp/Vq6UeDH7lQfgjCaJ11foqRZTtgXV5dj7XvmPtO+pJWNPmbghPvKxPM8JbHmteFzVFWws1RsBUSqRY6IzaBlmmhOHOJnfIHNXJKP2jZorOvwaf4oNqFxyXLKmSR1KQtp4Gg2YeZxo0B6uPhuq9LaEuzHhPtJ70hgGokiM0qd/Q==",
            "x-microsoft-antispam": "BCL:0;",
            "In-Reply-To": "\n <98CBD80474FA8B44BF855DF32C47DC35C61109@smartserver.smartshare.dk>",
            "x-ms-exchange-antispam-messagedata": "\n IpFueX44c0LfdWvoGCpmjAGatxl4ZYo8oOYNQV98kWzyZ4LsuuyNaXnsCThrNeGl9BNHAtqoQagmPUQ7HOjVG7tIwUQtmwOv0NyZj4fxShcOWyyC4cPa0KHXT7mkP6qa9hZ/w0X4jSAKuWWdnr3EEoRXK8H2oT1GHPIoo/QJiZajU59H8lgbGskVCYDEPsHPsr2RgOeJZSdOaGQhsMArS/QhK6ugJfTDp8HIppNL7tfpFu0NLD5MmAxumdx1h4OY7wxOXfqCaxL90x4W6bZY+w1WUGW4Va++EvQAOkRmPDMuT5u+8oWr3kg+pNfJ2kvsQC8enshlB005ncK3wkEfEy47eyPwRPtjKKSmyLxTXAfWHke0J3UHguXctFs7CkDrGq9dCXsbDZUwDdJw61OslKjF3YsLXpxJ8d2qvQOGdAjZaz+PP9OXe4iv7XfB7/lCyE5TZh9y4GoN/BIJ1xYV5LzHtId4yqA7hlhUTP1u1mc="
        }
    },
    {
        "id": 115551,
        "web_url": "https://patches.dpdk.org/comment/115551/",
        "msgid": "<98CBD80474FA8B44BF855DF32C47DC35C6110B@smartserver.smartshare.dk>",
        "date": "2020-07-08T15:27:19",
        "subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n\tTxscheduling",
        "submitter": {
            "id": 591,
            "url": "https://patches.dpdk.org/api/people/591/",
            "name": "Morten Brørup",
            "email": "mb@smartsharesystems.com"
        },
        "content": "> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Slava Ovsiienko\n> Sent: Wednesday, July 8, 2020 4:54 PM\n> \n> Hi, Morten\n> \n> Thank you for the comments. Please, see below.\n> \n> > -----Original Message-----\n> > From: Morten Brørup <mb@smartsharesystems.com>\n> > Sent: Wednesday, July 8, 2020 17:16\n> > To: Slava Ovsiienko <viacheslavo@mellanox.com>; dev@dpdk.org\n> > Cc: Matan Azrad <matan@mellanox.com>; Raslan Darawsheh\n> > <rasland@mellanox.com>; olivier.matz@6wind.com;\n> > bernard.iremonger@intel.com; thomas@mellanox.net\n> > Subject: RE: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate\n> packet\n> > Txscheduling\n> >\n> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Viacheslav\n> > > Ovsiienko\n> > > Sent: Tuesday, July 7, 2020 4:57 PM\n> > >\n> > > There is the requirement on some networks for precise traffic\n> timing\n> > > management. The ability to send (and, generally speaking, receive)\n> the\n> > > packets at the very precisely specified moment of time provides the\n> > > opportunity to support the connections with Time Division\n> Multiplexing\n> > > using the contemporary general purpose NIC without involving an\n> > > auxiliary hardware. For example, the supporting of O-RAN Fronthaul\n> > > interface is one of the promising features for potentially usage of\n> > > the precise time management for the egress packets.\n> > >\n> > > The main objective of this RFC is to specify the way how\n> applications\n> > > can provide the moment of time at what the packet transmission must\n> be\n> > > started and to describe in preliminary the supporting this feature\n> > > from\n> > > mlx5 PMD side.\n> > >\n> > > The new dynamic timestamp field is proposed, it provides some\n> timing\n> > > information, the units and time references (initial phase) are not\n> > > explicitly defined but are maintained always the same for a given\n> port.\n> > > Some devices allow to query rte_eth_read_clock() that will return\n> the\n> > > current device timestamp. The dynamic timestamp flag tells whether\n> the\n> > > field contains actual timestamp value. For the packets being sent\n> this\n> > > value can be used by PMD to schedule packet sending.\n> > >\n> > > After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation\n> and\n> > > obsoleting, these dynamic flag and field will be used to manage the\n> > > timestamps on receiving datapath as well.\n> > >\n> > > When PMD sees the \"rte_dynfield_timestamp\" set on the packet being\n> > > sent it tries to synchronize the time of packet appearing on the\n> wire\n> > > with the specified packet timestamp. If the specified one is in the\n> > > past it should be ignored, if one is in the distant future it\n> should\n> > > be capped with some reasonable value (in range of seconds). These\n> > > specific cases (\"too late\" and \"distant future\") can be optionally\n> > > reported via device xstats to assist applications to detect the\n> > > time-related problems.\n> > >\n> > > There is no any packet reordering according timestamps is supposed,\n> > > neither within packet burst, nor between packets, it is an entirely\n> > > application responsibility to generate packets and its timestamps\n> in\n> > > desired order. The timestamps can be put only in the first packet\n> in\n> > > the burst providing the entire burst scheduling.\n> > >\n> > > PMD reports the ability to synchronize packet sending on timestamp\n> > > with new offload flag:\n> > >\n> > > This is palliative and is going to be replaced with new eth_dev API\n> > > about reporting/managing the supported dynamic flags and its\n> related\n> > > features. This API would break ABI compatibility and can't be\n> > > introduced at the moment, so is postponed to 20.11.\n> > >\n> > > For testing purposes it is proposed to update testpmd \"txonly\"\n> > > forwarding mode routine. With this update testpmd application\n> > > generates the packets and sets the dynamic timestamps according to\n> > > specified time pattern if it sees the \"rte_dynfield_timestamp\" is\n> registered.\n> > >\n> > > The new testpmd command is proposed to configure sending pattern:\n> > >\n> > > set tx_times <burst_gap>,<intra_gap>\n> > >\n> > > <intra_gap> - the delay between the packets within the burst\n> > >               specified in the device clock units. The number\n> > >               of packets in the burst is defined by txburst\n> parameter\n> > >\n> > > <burst_gap> - the delay between the bursts in the device clock\n> units\n> > >\n> > > As the result the bursts of packet will be transmitted with\n> specific\n> > > delays between the packets within the burst and specific delay\n> between\n> > > the bursts. The rte_eth_get_clock is supposed to be engaged to get\n> the\n> > > current device clock value and provide the reference for the\n> > > timestamps.\n> > >\n> > > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>\n> > > ---\n> > >  v1->v4:\n> > >     - dedicated dynamic Tx timestamp flag instead of shared with Rx\n> >\n> > The detailed description above should be updated to reflect that it\n> is now\n> > two flags.\n> OK\n> \n> >\n> > >     - Doxygen-style comment\n> > >     - comments update\n> > >\n> > > ---\n> > >  lib/librte_ethdev/rte_ethdev.c |  1 +\n> lib/librte_ethdev/rte_ethdev.h\n> > > |  4 ++++  lib/librte_mbuf/rte_mbuf_dyn.h | 31\n> > > +++++++++++++++++++++++++++++++\n> > >  3 files changed, 36 insertions(+)\n> > >\n> > > diff --git a/lib/librte_ethdev/rte_ethdev.c\n> > > b/lib/librte_ethdev/rte_ethdev.c index 8e10a6f..02157d5 100644\n> > > --- a/lib/librte_ethdev/rte_ethdev.c\n> > > +++ b/lib/librte_ethdev/rte_ethdev.c\n> > > @@ -162,6 +162,7 @@ struct rte_eth_xstats_name_off {\n> > >  \tRTE_TX_OFFLOAD_BIT2STR(UDP_TNL_TSO),\n> > >  \tRTE_TX_OFFLOAD_BIT2STR(IP_TNL_TSO),\n> > >  \tRTE_TX_OFFLOAD_BIT2STR(OUTER_UDP_CKSUM),\n> > > +\tRTE_TX_OFFLOAD_BIT2STR(SEND_ON_TIMESTAMP),\n> > >  };\n> > >\n> > >  #undef RTE_TX_OFFLOAD_BIT2STR\n> > > diff --git a/lib/librte_ethdev/rte_ethdev.h\n> > > b/lib/librte_ethdev/rte_ethdev.h index a49242b..6f6454c 100644\n> > > --- a/lib/librte_ethdev/rte_ethdev.h\n> > > +++ b/lib/librte_ethdev/rte_ethdev.h\n> > > @@ -1178,6 +1178,10 @@ struct rte_eth_conf {\n> > >  /** Device supports outer UDP checksum */  #define\n> > > DEV_TX_OFFLOAD_OUTER_UDP_CKSUM  0x00100000\n> > >\n> > > +/** Device supports send on timestamp */ #define\n> > > +DEV_TX_OFFLOAD_SEND_ON_TIMESTAMP 0x00200000\n> > > +\n> > > +\n> > >  #define RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP 0x00000001\n> > /**<\n> > > Device supports Rx queue setup after device started*/  #define\n> > > RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP 0x00000002 diff --git\n> > > a/lib/librte_mbuf/rte_mbuf_dyn.h b/lib/librte_mbuf/rte_mbuf_dyn.h\n> > > index 96c3631..7e9f7d2 100644\n> > > --- a/lib/librte_mbuf/rte_mbuf_dyn.h\n> > > +++ b/lib/librte_mbuf/rte_mbuf_dyn.h\n> > > @@ -250,4 +250,35 @@ int rte_mbuf_dynflag_lookup(const char *name,\n> > > #define RTE_MBUF_DYNFIELD_METADATA_NAME\n> > \"rte_flow_dynfield_metadata\"\n> > >  #define RTE_MBUF_DYNFLAG_METADATA_NAME\n> > \"rte_flow_dynflag_metadata\"\n> > >\n> > > +/**\n> > > + * The timestamp dynamic field provides some timing information,\n> the\n> > > + * units and time references (initial phase) are not explicitly\n> > > defined\n> > > + * but are maintained always the same for a given port. Some\n> devices\n> > > allow4\n> > > + * to query rte_eth_read_clock() that will return the current\n> device\n> > > + * timestamp. The dynamic Tx timestamp flag tells whether the\n> field\n> > > contains\n> > > + * actual timestamp value. For the packets being sent this value\n> can\n> > > be\n> > > + * used by PMD to schedule packet sending.\n> > > + *\n> > > + * After PKT_RX_TIMESTAMP flag and fixed timestamp field\n> deprecation\n> > > + * and obsoleting, the dedicated Rx timestamp flag is supposed to\n> be\n> > > + * introduced and the shared dynamic timestamp field will be used\n> > > + * to handle the timestamps on receiving datapath as well.\n> > > + */\n> > > +#define RTE_MBUF_DYNFIELD_TIMESTAMP_NAME\n> > \"rte_dynfield_timestamp\"\n> >\n> > The description above should not say anything about the dynamic TX\n> > timestamp flag.\n> It does not. Or do you mean RX?\n> Not sure, field and flag are tightly coupled,\n> it is nice to mention this relation for better understanding.\n> And mentioning the RX explains why it is not like this:\n> RTE_MBUF_DYNFIELD_[TX]_TIMESTAMP_NAME\n\nSorry. I misunderstood its purpose!\nIt's the name of the field, and the field will not only be used for RX, but in the future also for RX.\n(I thought it was the name of the RX flag, reserved for future use.)\n\n> \n> >\n> > Please elaborate \"some timing information\", e.g. add \"... about when\n> the\n> > packet was received\".\n> \n> Sorry, I do not follow,  currently the dynamic field is not\n> \"about when the packet was received\". Now it is introduced for Tx\n> only and just the opportunity to be shared with Rx one in coming\n> releases\n> is mentioned. \"Some\" means - not specified (herein) exactly.\n> And it is elaborated what Is not specified and how it is supposed\n> to use Tx timestamp.\n\nIt should be described when it is valid, and how it is being used, e.g. by adding a reference to the \"rte_dynflag_tx_timestamp\" flag.\n\n> >\n> > > +\n> > > +/**\n> > > + * When PMD sees the RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME flag\n> > set on\n> > > the\n> > > + * packet being sent it tries to synchronize the time of packet\n> > > appearing\n> > > + * on the wire with the specified packet timestamp. If the\n> specified\n> > > one\n> > > + * is in the past it should be ignored, if one is in the distant\n> > > future\n> > > + * it should be capped with some reasonable value (in range of\n> > > seconds).\n> > > + *\n> > > + * There is no any packet reordering according to timestamps is\n> > > supposed,\n> > > + * neither for packet within the burst, nor for the whole bursts,\n> it\n> > > is\n> > > + * an entirely application responsibility to generate packets and\n> its\n> > > + * timestamps in desired order. The timestamps might be put only\n> in\n> > > + * the first packet in the burst providing the entire burst\n> > > scheduling.\n> > > + */\n> > > +#define RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME\n> > \"rte_dynflag_tx_timestamp\"\n> > > +\n> > >  #endif\n> > > --\n> > > 1.8.3.1\n> > >\n> >\n> > It may be worth adding some documentation about how the clocks of the\n> > NICs are out of sync with the clock of the CPU, and are all drifting\n> relatively.\n> >\n> > And those clocks are also out of sync with the actual time (NTP\n> clock).\n> \n> IMO, It is out of scope of this very generic patch.  As for mlx NICs -\n> the internal device\n> clock might be (or might be not) synchronized with PTP, it can provide\n> timestamps\n> in real nanoseconds in various formats or just some free running\n> counter.\n\nCool!\n\n> On some systems the NIC and CPU might share the same clock source (for\n> their PLL inputs\n> for example) and there will be no any drifts. As we can see - it is a\n> wide and interesting\n> opic to discuss, but, IMO,  the comment in header file might be not the\n> most relevant\n> place to do. As for mlx5 devices clock specifics - it will be\n> documented in PMD chapter.\n> \n> OK, will add few generic words, the few ones - in order not to make\n> comment wordy, just\n> point the direction for further thinking.\n\nI agree - we don't want cookbooks in the header files. Only enough description to avoid the worst misunderstandings.\n\n> \n> >\n> > Preferably, some sort of cookbook for handling this should be\n> provided.\n> > PCAP could be used as an example.\n> >\n> testpmd example is included in series, mlx5 PMD patch is prepared and\n> coming soon.\n\nGreat.\n\nAnd I suppose that the more detailed cookbook/example - regarding offset and drift of various clocks - is probably more relevant for the RX side (for various PCAP applications), and thus completely unrelated to this patch.\n\n> \n> With best regards, Slava",
        "headers": {
            "Return-Path": "<dev-bounces@dpdk.org>",
            "MIME-Version": "1.0",
            "X-Mailman-Version": "2.1.15",
            "From": "=?utf-8?q?Morten_Br=C3=B8rup?= <mb@smartsharesystems.com>",
            "List-Post": "<mailto:dev@dpdk.org>",
            "Thread-Index": "AQHWVTJRtpVhbms7BUGtcj8QA0ZT26j9vBYwgAAKtNA=",
            "Message-ID": "<98CBD80474FA8B44BF855DF32C47DC35C6110B@smartserver.smartshare.dk>",
            "References": "<1591771085-24959-1-git-send-email-viacheslavo@mellanox.com>\n <1594133832-12785-1-git-send-email-viacheslavo@mellanox.com>\n <98CBD80474FA8B44BF855DF32C47DC35C61109@smartserver.smartshare.dk>\n <AM4PR05MB3265F08B5462C8278296A60DD2670@AM4PR05MB3265.eurprd05.prod.outlook.com>",
            "X-Original-To": "patchwork@inbox.dpdk.org",
            "Received": [
                "from dpdk.org (dpdk.org [92.243.14.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id A6DBBA0527;\n\tWed,  8 Jul 2020 17:27:21 +0200 (CEST)",
                "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 7A4A31E4C1;\n\tWed,  8 Jul 2020 17:27:21 +0200 (CEST)",
                "from smartserver.smartsharesystems.com\n (smartserver.smartsharesystems.com [77.243.40.215])\n by dpdk.org (Postfix) with ESMTP id E1B881E4C0\n for <dev@dpdk.org>; Wed,  8 Jul 2020 17:27:20 +0200 (CEST)"
            ],
            "Subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n\tTxscheduling",
            "Sender": "\"dev\" <dev-bounces@dpdk.org>",
            "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
            "Content-class": "urn:content-classes:message",
            "Content-Type": "text/plain;\n\tcharset=\"UTF-8\"",
            "X-MS-Has-Attach": "",
            "X-BeenThere": "dev@dpdk.org",
            "Date": "Wed, 8 Jul 2020 17:27:19 +0200",
            "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
            "Content-Transfer-Encoding": "base64",
            "Errors-To": "dev-bounces@dpdk.org",
            "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>",
            "Cc": "\"Matan Azrad\" <matan@mellanox.com>,\n \"Raslan Darawsheh\" <rasland@mellanox.com>, <olivier.matz@6wind.com>,\n <bernard.iremonger@intel.com>, <thomas@mellanox.net>",
            "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
            "Precedence": "list",
            "Delivered-To": "patchwork@inbox.dpdk.org",
            "In-Reply-To": "\n <AM4PR05MB3265F08B5462C8278296A60DD2670@AM4PR05MB3265.eurprd05.prod.outlook.com>",
            "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>",
            "X-MimeOLE": "Produced By Microsoft Exchange V6.5",
            "To": "\"Slava Ovsiienko\" <viacheslavo@mellanox.com>,\n\t<dev@dpdk.org>",
            "Thread-Topic": "[dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n Txscheduling",
            "X-MS-TNEF-Correlator": ""
        }
    },
    {
        "id": 115553,
        "web_url": "https://patches.dpdk.org/comment/115553/",
        "msgid": "<AM4PR05MB32651CBBEA2629D04BA81898D2670@AM4PR05MB3265.eurprd05.prod.outlook.com>",
        "date": "2020-07-08T15:51:43",
        "subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n Txscheduling",
        "submitter": {
            "id": 1102,
            "url": "https://patches.dpdk.org/api/people/1102/",
            "name": "Slava Ovsiienko",
            "email": "viacheslavo@mellanox.com"
        },
        "content": "Hi, Morten\n\nAddressed most of your comments in the v5 commit message.\nHeader file comments are close to become too wordy,\nand I did not dare to elaborate ones more.\n\nWith best regards, Slava\n\n> -----Original Message-----\n> From: Morten Brørup <mb@smartsharesystems.com>\n> Sent: Wednesday, July 8, 2020 18:27\n> To: Slava Ovsiienko <viacheslavo@mellanox.com>; dev@dpdk.org\n> Cc: Matan Azrad <matan@mellanox.com>; Raslan Darawsheh\n> <rasland@mellanox.com>; olivier.matz@6wind.com;\n> bernard.iremonger@intel.com; thomas@mellanox.net\n> Subject: RE: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n> Txscheduling\n> \n> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Slava Ovsiienko\n> > Sent: Wednesday, July 8, 2020 4:54 PM\n> >\n> > Hi, Morten\n> >\n> > Thank you for the comments. Please, see below.\n> >\n> > > -----Original Message-----\n> > > From: Morten Brørup <mb@smartsharesystems.com>\n> > > Sent: Wednesday, July 8, 2020 17:16\n> > > To: Slava Ovsiienko <viacheslavo@mellanox.com>; dev@dpdk.org\n> > > Cc: Matan Azrad <matan@mellanox.com>; Raslan Darawsheh\n> > > <rasland@mellanox.com>; olivier.matz@6wind.com;\n> > > bernard.iremonger@intel.com; thomas@mellanox.net\n> > > Subject: RE: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate\n> > packet\n> > > Txscheduling\n> > >\n> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Viacheslav\n> > > > Ovsiienko\n> > > > Sent: Tuesday, July 7, 2020 4:57 PM\n> > > >\n> > > > There is the requirement on some networks for precise traffic\n> > timing\n> > > > management. The ability to send (and, generally speaking, receive)\n> > the\n> > > > packets at the very precisely specified moment of time provides\n> > > > the opportunity to support the connections with Time Division\n> > Multiplexing\n> > > > using the contemporary general purpose NIC without involving an\n> > > > auxiliary hardware. For example, the supporting of O-RAN Fronthaul\n> > > > interface is one of the promising features for potentially usage\n> > > > of the precise time management for the egress packets.\n> > > >\n> > > > The main objective of this RFC is to specify the way how\n> > applications\n> > > > can provide the moment of time at what the packet transmission\n> > > > must\n> > be\n> > > > started and to describe in preliminary the supporting this feature\n> > > > from\n> > > > mlx5 PMD side.\n> > > >\n> > > > The new dynamic timestamp field is proposed, it provides some\n> > timing\n> > > > information, the units and time references (initial phase) are not\n> > > > explicitly defined but are maintained always the same for a given\n> > port.\n> > > > Some devices allow to query rte_eth_read_clock() that will return\n> > the\n> > > > current device timestamp. The dynamic timestamp flag tells whether\n> > the\n> > > > field contains actual timestamp value. For the packets being sent\n> > this\n> > > > value can be used by PMD to schedule packet sending.\n> > > >\n> > > > After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation\n> > and\n> > > > obsoleting, these dynamic flag and field will be used to manage\n> > > > the timestamps on receiving datapath as well.\n> > > >\n> > > > When PMD sees the \"rte_dynfield_timestamp\" set on the packet being\n> > > > sent it tries to synchronize the time of packet appearing on the\n> > wire\n> > > > with the specified packet timestamp. If the specified one is in\n> > > > the past it should be ignored, if one is in the distant future it\n> > should\n> > > > be capped with some reasonable value (in range of seconds). These\n> > > > specific cases (\"too late\" and \"distant future\") can be optionally\n> > > > reported via device xstats to assist applications to detect the\n> > > > time-related problems.\n> > > >\n> > > > There is no any packet reordering according timestamps is\n> > > > supposed, neither within packet burst, nor between packets, it is\n> > > > an entirely application responsibility to generate packets and its\n> > > > timestamps\n> > in\n> > > > desired order. The timestamps can be put only in the first packet\n> > in\n> > > > the burst providing the entire burst scheduling.\n> > > >\n> > > > PMD reports the ability to synchronize packet sending on timestamp\n> > > > with new offload flag:\n> > > >\n> > > > This is palliative and is going to be replaced with new eth_dev\n> > > > API about reporting/managing the supported dynamic flags and its\n> > related\n> > > > features. This API would break ABI compatibility and can't be\n> > > > introduced at the moment, so is postponed to 20.11.\n> > > >\n> > > > For testing purposes it is proposed to update testpmd \"txonly\"\n> > > > forwarding mode routine. With this update testpmd application\n> > > > generates the packets and sets the dynamic timestamps according to\n> > > > specified time pattern if it sees the \"rte_dynfield_timestamp\" is\n> > registered.\n> > > >\n> > > > The new testpmd command is proposed to configure sending pattern:\n> > > >\n> > > > set tx_times <burst_gap>,<intra_gap>\n> > > >\n> > > > <intra_gap> - the delay between the packets within the burst\n> > > >               specified in the device clock units. The number\n> > > >               of packets in the burst is defined by txburst\n> > parameter\n> > > >\n> > > > <burst_gap> - the delay between the bursts in the device clock\n> > units\n> > > >\n> > > > As the result the bursts of packet will be transmitted with\n> > specific\n> > > > delays between the packets within the burst and specific delay\n> > between\n> > > > the bursts. The rte_eth_get_clock is supposed to be engaged to get\n> > the\n> > > > current device clock value and provide the reference for the\n> > > > timestamps.\n> > > >\n> > > > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>\n> > > > ---\n> > > >  v1->v4:\n> > > >     - dedicated dynamic Tx timestamp flag instead of shared with\n> > > > Rx\n> > >\n> > > The detailed description above should be updated to reflect that it\n> > is now\n> > > two flags.\n> > OK\n> >\n> > >\n> > > >     - Doxygen-style comment\n> > > >     - comments update\n> > > >\n> > > > ---\n> > > >  lib/librte_ethdev/rte_ethdev.c |  1 +\n> > lib/librte_ethdev/rte_ethdev.h\n> > > > |  4 ++++  lib/librte_mbuf/rte_mbuf_dyn.h | 31\n> > > > +++++++++++++++++++++++++++++++\n> > > >  3 files changed, 36 insertions(+)\n> > > >\n> > > > diff --git a/lib/librte_ethdev/rte_ethdev.c\n> > > > b/lib/librte_ethdev/rte_ethdev.c index 8e10a6f..02157d5 100644\n> > > > --- a/lib/librte_ethdev/rte_ethdev.c\n> > > > +++ b/lib/librte_ethdev/rte_ethdev.c\n> > > > @@ -162,6 +162,7 @@ struct rte_eth_xstats_name_off {\n> > > >  \tRTE_TX_OFFLOAD_BIT2STR(UDP_TNL_TSO),\n> > > >  \tRTE_TX_OFFLOAD_BIT2STR(IP_TNL_TSO),\n> > > >  \tRTE_TX_OFFLOAD_BIT2STR(OUTER_UDP_CKSUM),\n> > > > +\tRTE_TX_OFFLOAD_BIT2STR(SEND_ON_TIMESTAMP),\n> > > >  };\n> > > >\n> > > >  #undef RTE_TX_OFFLOAD_BIT2STR\n> > > > diff --git a/lib/librte_ethdev/rte_ethdev.h\n> > > > b/lib/librte_ethdev/rte_ethdev.h index a49242b..6f6454c 100644\n> > > > --- a/lib/librte_ethdev/rte_ethdev.h\n> > > > +++ b/lib/librte_ethdev/rte_ethdev.h\n> > > > @@ -1178,6 +1178,10 @@ struct rte_eth_conf {\n> > > >  /** Device supports outer UDP checksum */  #define\n> > > > DEV_TX_OFFLOAD_OUTER_UDP_CKSUM  0x00100000\n> > > >\n> > > > +/** Device supports send on timestamp */ #define\n> > > > +DEV_TX_OFFLOAD_SEND_ON_TIMESTAMP 0x00200000\n> > > > +\n> > > > +\n> > > >  #define RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP\n> 0x00000001\n> > > /**<\n> > > > Device supports Rx queue setup after device started*/  #define\n> > > > RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP 0x00000002 diff --\n> git\n> > > > a/lib/librte_mbuf/rte_mbuf_dyn.h b/lib/librte_mbuf/rte_mbuf_dyn.h\n> > > > index 96c3631..7e9f7d2 100644\n> > > > --- a/lib/librte_mbuf/rte_mbuf_dyn.h\n> > > > +++ b/lib/librte_mbuf/rte_mbuf_dyn.h\n> > > > @@ -250,4 +250,35 @@ int rte_mbuf_dynflag_lookup(const char\n> *name,\n> > > > #define RTE_MBUF_DYNFIELD_METADATA_NAME\n> > > \"rte_flow_dynfield_metadata\"\n> > > >  #define RTE_MBUF_DYNFLAG_METADATA_NAME\n> > > \"rte_flow_dynflag_metadata\"\n> > > >\n> > > > +/**\n> > > > + * The timestamp dynamic field provides some timing information,\n> > the\n> > > > + * units and time references (initial phase) are not explicitly\n> > > > defined\n> > > > + * but are maintained always the same for a given port. Some\n> > devices\n> > > > allow4\n> > > > + * to query rte_eth_read_clock() that will return the current\n> > device\n> > > > + * timestamp. The dynamic Tx timestamp flag tells whether the\n> > field\n> > > > contains\n> > > > + * actual timestamp value. For the packets being sent this value\n> > can\n> > > > be\n> > > > + * used by PMD to schedule packet sending.\n> > > > + *\n> > > > + * After PKT_RX_TIMESTAMP flag and fixed timestamp field\n> > deprecation\n> > > > + * and obsoleting, the dedicated Rx timestamp flag is supposed to\n> > be\n> > > > + * introduced and the shared dynamic timestamp field will be used\n> > > > + * to handle the timestamps on receiving datapath as well.\n> > > > + */\n> > > > +#define RTE_MBUF_DYNFIELD_TIMESTAMP_NAME\n> > > \"rte_dynfield_timestamp\"\n> > >\n> > > The description above should not say anything about the dynamic TX\n> > > timestamp flag.\n> > It does not. Or do you mean RX?\n> > Not sure, field and flag are tightly coupled, it is nice to mention\n> > this relation for better understanding.\n> > And mentioning the RX explains why it is not like this:\n> > RTE_MBUF_DYNFIELD_[TX]_TIMESTAMP_NAME\n> \n> Sorry. I misunderstood its purpose!\n> It's the name of the field, and the field will not only be used for RX, but in the\n> future also for RX.\n> (I thought it was the name of the RX flag, reserved for future use.)\n> \n> >\n> > >\n> > > Please elaborate \"some timing information\", e.g. add \"... about when\n> > the\n> > > packet was received\".\n> >\n> > Sorry, I do not follow,  currently the dynamic field is not \"about\n> > when the packet was received\". Now it is introduced for Tx only and\n> > just the opportunity to be shared with Rx one in coming releases is\n> > mentioned. \"Some\" means - not specified (herein) exactly.\n> > And it is elaborated what Is not specified and how it is supposed to\n> > use Tx timestamp.\n> \n> It should be described when it is valid, and how it is being used, e.g. by\n> adding a reference to the \"rte_dynflag_tx_timestamp\" flag.\n> \n> > >\n> > > > +\n> > > > +/**\n> > > > + * When PMD sees the RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME\n> flag\n> > > set on\n> > > > the\n> > > > + * packet being sent it tries to synchronize the time of packet\n> > > > appearing\n> > > > + * on the wire with the specified packet timestamp. If the\n> > specified\n> > > > one\n> > > > + * is in the past it should be ignored, if one is in the distant\n> > > > future\n> > > > + * it should be capped with some reasonable value (in range of\n> > > > seconds).\n> > > > + *\n> > > > + * There is no any packet reordering according to timestamps is\n> > > > supposed,\n> > > > + * neither for packet within the burst, nor for the whole bursts,\n> > it\n> > > > is\n> > > > + * an entirely application responsibility to generate packets and\n> > its\n> > > > + * timestamps in desired order. The timestamps might be put only\n> > in\n> > > > + * the first packet in the burst providing the entire burst\n> > > > scheduling.\n> > > > + */\n> > > > +#define RTE_MBUF_DYNFLAG_TX_TIMESTAMP_NAME\n> > > \"rte_dynflag_tx_timestamp\"\n> > > > +\n> > > >  #endif\n> > > > --\n> > > > 1.8.3.1\n> > > >\n> > >\n> > > It may be worth adding some documentation about how the clocks of\n> > > the NICs are out of sync with the clock of the CPU, and are all\n> > > drifting\n> > relatively.\n> > >\n> > > And those clocks are also out of sync with the actual time (NTP\n> > clock).\n> >\n> > IMO, It is out of scope of this very generic patch.  As for mlx NICs -\n> > the internal device clock might be (or might be not) synchronized with\n> > PTP, it can provide timestamps in real nanoseconds in various formats\n> > or just some free running counter.\n> \n> Cool!\n> \n> > On some systems the NIC and CPU might share the same clock source (for\n> > their PLL inputs for example) and there will be no any drifts. As we\n> > can see - it is a wide and interesting opic to discuss, but, IMO,  the\n> > comment in header file might be not the most relevant place to do. As\n> > for mlx5 devices clock specifics - it will be documented in PMD\n> > chapter.\n> >\n> > OK, will add few generic words, the few ones - in order not to make\n> > comment wordy, just point the direction for further thinking.\n> \n> I agree - we don't want cookbooks in the header files. Only enough\n> description to avoid the worst misunderstandings.\n> \n> >\n> > >\n> > > Preferably, some sort of cookbook for handling this should be\n> > provided.\n> > > PCAP could be used as an example.\n> > >\n> > testpmd example is included in series, mlx5 PMD patch is prepared and\n> > coming soon.\n> \n> Great.\n> \n> And I suppose that the more detailed cookbook/example - regarding offset\n> and drift of various clocks - is probably more relevant for the RX side (for\n> various PCAP applications), and thus completely unrelated to this patch.\n> \n> >\n> > With best regards, Slava",
        "headers": {
            "X-OriginatorOrg": "Mellanox.com",
            "x-ms-office365-filtering-correlation-id": "247f3af4-1aae-45da-85cb-08d82356d008",
            "References": "<1591771085-24959-1-git-send-email-viacheslavo@mellanox.com>\n <1594133832-12785-1-git-send-email-viacheslavo@mellanox.com>\n <98CBD80474FA8B44BF855DF32C47DC35C61109@smartserver.smartshare.dk>\n <AM4PR05MB3265F08B5462C8278296A60DD2670@AM4PR05MB3265.eurprd05.prod.outlook.com>\n <98CBD80474FA8B44BF855DF32C47DC35C6110B@smartserver.smartshare.dk>",
            "X-Mailman-Version": "2.1.15",
            "X-MS-Exchange-CrossTenant-mailboxtype": "HOSTED",
            "X-MS-Exchange-CrossTenant-AuthAs": "Internal",
            "X-MS-Exchange-CrossTenant-userprincipalname": "\n URc7/lPoWR5Dwz2a6kfxxWJUMkv6QI6BxCi+cOOCe4K9/16U+YywznfQoy6fPj7UQC+87iTz4+D4s6EdzO8yGil5alswRxWEuJfexmD3K60=",
            "x-ms-oob-tlc-oobclassifiers": "OLM:10000;",
            "Thread-Index": "AQHWVTJRtpVhbms7BUGtcj8QA0ZT26j9vBYwgAAKtNCAAA1YAA==",
            "Content-Type": "text/plain; charset=\"utf-8\"",
            "CC": "Matan Azrad <matan@mellanox.com>, Raslan Darawsheh <rasland@mellanox.com>,\n \"olivier.matz@6wind.com\" <olivier.matz@6wind.com>,\n \"bernard.iremonger@intel.com\" <bernard.iremonger@intel.com>,\n \"thomas@mellanox.net\" <thomas@mellanox.net>",
            "Thread-Topic": "[dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n Txscheduling",
            "Received": [
                "from dpdk.org (dpdk.org [92.243.14.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 29F4DA0527;\n\tWed,  8 Jul 2020 17:51:48 +0200 (CEST)",
                "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 1F2A51E48D;\n\tWed,  8 Jul 2020 17:51:47 +0200 (CEST)",
                "from EUR03-AM5-obe.outbound.protection.outlook.com\n (mail-eopbgr30086.outbound.protection.outlook.com [40.107.3.86])\n by dpdk.org (Postfix) with ESMTP id 7E9E11DF90\n for <dev@dpdk.org>; Wed,  8 Jul 2020 17:51:45 +0200 (CEST)",
                "from AM4PR05MB3265.eurprd05.prod.outlook.com (2603:10a6:205:8::26)\n by AM0PR05MB6708.eurprd05.prod.outlook.com (2603:10a6:20b:144::8) with\n Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.20; Wed, 8 Jul\n 2020 15:51:43 +0000",
                "from AM4PR05MB3265.eurprd05.prod.outlook.com\n ([fe80::194e:dc46:7543:50ed]) by AM4PR05MB3265.eurprd05.prod.outlook.com\n ([fe80::194e:dc46:7543:50ed%2]) with mapi id 15.20.3174.022; Wed, 8 Jul 2020\n 15:51:43 +0000"
            ],
            "Subject": "Re: [dpdk-dev] [PATCH v4 1/2] mbuf: introduce accurate packet\n Txscheduling",
            "ARC-Seal": "i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;\n b=bGzLhxHUNiTFdTdk7OuqQNQmyjgGb9d58sNv9Ld4gdwIE0Ttv+ZNy93kZdj+0+OVUT5hYFL9juBfsGmoITgFNVbIZ3/uXGB9wKsholF3Isvh8vTFPiyn3AWZYFRFgYTtAEjtveNK7Xb6gqnzRxR9RsEai1kFvTcQElcwGAld63X2EypaXpy/ZH2qiV0kEPO5and9jSJLMzlEsZJTTBoTMj5u4tlmNxhxj/8hsr2tZvxl7tHDpa26ZoyPUIahiyy5KNkt/up7ksUDtLpXMNka/5hAHMTVFPI0wjdY4gcV1LsR+6so5GshRbKzjBCHAQC4IQFEXkc+JQr+IiXzZxdQww==",
            "authentication-results": "smartsharesystems.com; dkim=none (message not signed)\n header.d=none; smartsharesystems.com;\n dmarc=none action=none header.from=mellanox.com;",
            "x-ms-traffictypediagnostic": "AM0PR05MB6708:",
            "X-MS-Exchange-CrossTenant-originalarrivaltime": "08 Jul 2020 15:51:43.5715 (UTC)",
            "Precedence": "list",
            "Date": "Wed, 8 Jul 2020 15:51:43 +0000",
            "x-ms-office365-filtering-ht": "Tenant",
            "x-forefront-antispam-report": "CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:AM4PR05MB3265.eurprd05.prod.outlook.com; PTR:; CAT:NONE;\n SFTY:;\n SFS:(4636009)(136003)(376002)(346002)(39860400002)(366004)(396003)(186003)(66476007)(66556008)(64756008)(66446008)(55016002)(2906002)(7696005)(110136005)(316002)(54906003)(4326008)(66946007)(86362001)(76116006)(71200400001)(30864003)(6506007)(53546011)(26005)(83380400001)(66574015)(33656002)(52536014)(9686003)(5660300002)(8676002)(8936002)(478600001);\n DIR:OUT; SFP:1101;",
            "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>",
            "x-ms-exchange-transport-forked": "True",
            "x-microsoft-antispam-prvs": "\n <AM0PR05MB6708F90EE9BADE1042F0E829D2670@AM0PR05MB6708.eurprd05.prod.outlook.com>",
            "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>",
            "X-BeenThere": "dev@dpdk.org",
            "x-forefront-prvs": "04583CED1A",
            "Content-Transfer-Encoding": "base64",
            "To": "=?utf-8?q?Morten_Br=C3=B8rup?= <mb@smartsharesystems.com>,\n \"dev@dpdk.org\" <dev@dpdk.org>",
            "X-MS-TNEF-Correlator": "",
            "Return-Path": "<dev-bounces@dpdk.org>",
            "x-microsoft-antispam-message-info": "\n HbVjR2s50Xr2bb6VEGB8f9ZhhIJRtIAqixOD1vG5IbmgGto6AJMmcZ2kbP/79shHTtZ2LCh46ZGgDbqBifFxV80O6ZpJELuVPSVUvbjxWePe9PN49qkbfu8AIHK0NXxBIfjYB3myoKTkXlqvLeh9YU7vADbTlaQaEDZ+170iueITfKwZxA4yHN7UEFKaNIudoGbNDBwGnS58OJWn/e6hNbd/9eodgT4E77nX3Z2hOP11U/fnYPj5knWsb0IqJAzlJjSpG1OVFkHnOeUiQMpyuZcBRTyrOpmWzd+T3bVtCKMBW+7oXqhEouVOIP/m20daWnAlqJw79Mw5DU4s4agb8Q==",
            "x-ld-processed": "a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtFwd",
            "MIME-Version": "1.0",
            "List-Post": "<mailto:dev@dpdk.org>",
            "X-MS-Exchange-CrossTenant-AuthSource": "AM4PR05MB3265.eurprd05.prod.outlook.com",
            "From": "Slava Ovsiienko <viacheslavo@mellanox.com>",
            "x-originating-ip": "[95.164.10.10]",
            "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
            "x-ms-publictraffictype": "Email",
            "X-MS-Exchange-CrossTenant-fromentityheader": "Hosted",
            "Accept-Language": "en-US",
            "Message-ID": "\n <AM4PR05MB32651CBBEA2629D04BA81898D2670@AM4PR05MB3265.eurprd05.prod.outlook.com>",
            "x-ms-exchange-senderadcheck": "1",
            "Delivered-To": "patchwork@inbox.dpdk.org",
            "X-Original-To": "patchwork@inbox.dpdk.org",
            "X-MS-Exchange-CrossTenant-Network-Message-Id": "\n 247f3af4-1aae-45da-85cb-08d82356d008",
            "ARC-Authentication-Results": "i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com;\n dkim=pass header.d=mellanox.com; arc=none",
            "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
            "Sender": "\"dev\" <dev-bounces@dpdk.org>",
            "X-MS-Exchange-CrossTenant-id": "a652971c-7d2e-4d9b-a6a4-d149256f461b",
            "Content-Language": "en-US",
            "X-MS-Has-Attach": "",
            "X-MS-Exchange-Transport-CrossTenantHeadersStamped": "AM0PR05MB6708",
            "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;\n s=selector1;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=Z9snYFuJez5T7uTW1WmphRcPSKcF8E2LE7eetuhPKpg=;\n b=b/yQxQSAgoGaMU3HTuMkmroR44RpBehbEY/DajFXaBWy3UrFLtLfFYhE/3FE5IIQ+57usJ38VopswMCoqSHwGfVtgFyniCCFG8TQCwdQSy/vDHqP3cryuRgGYIw9RoFv0n+QFrMa0QhngbZRGRa2a087tLV1UNgvy8Qr2Eeqo2k=",
            "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
            "Errors-To": "dev-bounces@dpdk.org",
            "ARC-Message-Signature": "i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector9901;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=Z9snYFuJez5T7uTW1WmphRcPSKcF8E2LE7eetuhPKpg=;\n b=CoiJLPP869nTuob9HT3f16d7vaKyLlT4TNUidTKqoHxaGDxt+yv98myl5myV2a+i2PV7VR7wMtMgQ9/HQyZvjuw1y8LHQJpVbbCtQ7+V4O9vXh9eA9ae1nC0oxOMWmwjLmOcCXU3S+VrGm5m+9oZshqIgF3xU0BKV4wRbA4h+t0jQ9yvwpxnFCAoqJMAPURg04MMx0l3VMQfEu2UggJ/8gIcGNnYEAYUFxmX4V3pS8oeMC7Zs7IvKnxypu5H1zJp/P2LR6p2AwUAozRpeKI5PCJwts5zc9hqu6NQ1yudmG/sdB0wf6LQruvUqpE3VvybeOc9vK2l4D5QLyrhKqG4cQ==",
            "x-microsoft-antispam": "BCL:0;",
            "In-Reply-To": "\n <98CBD80474FA8B44BF855DF32C47DC35C6110B@smartserver.smartshare.dk>",
            "x-ms-exchange-antispam-messagedata": "\n bpA2fER//KQuz8PC2oW4eHJwpXggwpjA/eeWfH34QgqZUk+eJgXiLgaYYEWLBWOAAGL2cZQtxPHdSXuG3szsyUCuuD5mamzlGmYwS8LtwBhbfNHlcJd/eEsIBqZBUK+f5+gBlIbbXHwJMDL2q/iDd+rrr484K6wg2WwDij3ElfXkecyrV1jMkhrsl89uB1ep5YiGdAUxE9f40H4LkycmZefP5nCA5c7CsT+SJcX2BiHU7CRTfiE35jXT4tZN47+2FnFvfjDNZjtthTpKmNnt40vLvZ+dFdZyDEigdAlW/T//7vmozFzs1bpV86BwlOAk6w9YnUbMWQzba0rcUjhSOObJcpPw3kuLVmfH4aA1QxGaIKXeYLU8sqcdoUCqzxn6Gw1DjiPvmvDIgM7I/8AQOXEppRnGiIVZhYfhAB9gh03D9oEztsIBp9vKLDCg1epyoUXbIA7JLjNKtH52AytTkcko2G5q1eDC6amRKHaLfGU="
        }
    }
]