From patchwork Mon Jul 27 17:47:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Parav Pandit X-Patchwork-Id: 74860 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8CE92A053E; Mon, 27 Jul 2020 19:47:49 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0E78D2C01; Mon, 27 Jul 2020 19:47:48 +0200 (CEST) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140080.outbound.protection.outlook.com [40.107.14.80]) by dpdk.org (Postfix) with ESMTP id D1AE4A3 for ; Mon, 27 Jul 2020 19:47:45 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cj0KijO3ydA/1266siWk/R6+7zEEwEsvzgOFAN2ZvEuxRX5hqv8skS5o5ou4Qtw3l6/xN2KsAWqd967F8PUjIXjSUt2qGkbALVtD7try4cuALi+x3W0E9VlLe6aDttOscrh/inB2NIoULzTRRjR/BT9neUZofiwOUJclnimy16fvhNndN4XCdXuA+3m6nn5AdNn+4FxRby/wUL+ALouGozXxdgrfyIFfUo7WIb4tvBURgeAjdpPSwygpg55y3XC7yxRn86Rsp/YdkZPsiF8ueh1P7eeiVy9XY+wv8RQ4FAv0TBaFNOOQTwnmFAF3jYg+SLIGNTJUymwE1xazmht3Ag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1FxqL5hMKsv21aYvZfAty1y39sU1x2QzZkRkfbZbrnM=; b=PDX0aJLS9jXoPZPWCVqX2kYQmahssKNRwPYmo3e8MPGSpH/B7eJ3rpNnsKIHmfRdoBzrs8yE6KIcB7jASIgxcfcZfqJW/3uTPW5IPCB4ejcBX9tku8thQhQrUBZx6oBhqBQ7ezenxDqYP3P6GXn7pWCunGNeSgZ2NxYP9z2kSr2iGhTveQsgbExbjAoYm2chklhTSHjl40ahbeC0Ho+6/ke+NgUUgtW5DZCWIUPS1FdRqEPWimI/OWi5SqQdyMAxsUZiWiON0Z07ldg85WrkfieaElWO8Q/qIgk4tbWGu6CYbDqed9J12C5tw826Jc6dAzrzB9hFU6oc4533y0rKfA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1FxqL5hMKsv21aYvZfAty1y39sU1x2QzZkRkfbZbrnM=; b=YOj+VDxYzoBN6sKfMXRrn8xLk5I93VxvlYBFjl8MifIQQzV0s8U4CPB+N3FCrt5q1wlIJhEmNdzS7SrwbFM08kfGbXXzOoiJ81n/q6nMIj+YQoH9OTbKEY0LK8jvQhhq1bEro3IBV//H1m+kx8iyPaJhzVyJ+d07YZqqQ9irLn4= Authentication-Results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=mellanox.com; Received: from AM0PR05MB4866.eurprd05.prod.outlook.com (2603:10a6:208:c0::32) by AM0PR05MB6227.eurprd05.prod.outlook.com (2603:10a6:208:10f::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3216.23; Mon, 27 Jul 2020 17:47:43 +0000 Received: from AM0PR05MB4866.eurprd05.prod.outlook.com ([fe80::eccf:72b3:bacb:f09d]) by AM0PR05MB4866.eurprd05.prod.outlook.com ([fe80::eccf:72b3:bacb:f09d%5]) with mapi id 15.20.3216.033; Mon, 27 Jul 2020 17:47:42 +0000 From: Parav Pandit To: dev@dpdk.org, grive@u256.net, ferruh.yigit@intel.com, thomas@monjalon.net Cc: rasland@mellanox.com, orika@mellanox.com, matan@mellanox.com, joyce.kong@arm.com, Parav Pandit Date: Mon, 27 Jul 2020 20:47:05 +0300 Message-Id: <20200727174715.330117-1-parav@mellanox.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200610171728.89-2-parav@mellanox.com> References: <20200610171728.89-2-parav@mellanox.com> X-ClientProxiedBy: DM5PR15CA0049.namprd15.prod.outlook.com (2603:10b6:3:ae::11) To AM0PR05MB4866.eurprd05.prod.outlook.com (2603:10a6:208:c0::32) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from sw-mtx-036.mtx.labs.mlnx (208.176.44.194) by DM5PR15CA0049.namprd15.prod.outlook.com (2603:10b6:3:ae::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3216.20 via Frontend Transport; Mon, 27 Jul 2020 17:47:41 +0000 X-Mailer: git-send-email 2.26.2 X-Originating-IP: [208.176.44.194] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 038a6cf4-8909-43d4-09e8-08d8325529bf X-MS-TrafficTypeDiagnostic: AM0PR05MB6227: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtFwd,ExtAddr X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:469; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7awlGBN8NtnoCGz9o6VC2Slv9YL1wlRBlAU7fzVQ4u6x2TGs4sQMsnxMlJ9IEx2FNhVlvidO5UktBhvzh/aBzgJWhB/9NcEqsIsIQrnOioy5A0PkShDNOmc1yobTPN4qdq3bP0zfw9EmVK22qnb2gKPYFTbkJxE6T8sL+Mje4ay98Xu0Zl4j7NA/OGqljt4SgzaJVj7ecCwLs815yBeVIOOZpRuJhbNbcdLIStFq9ESqWLrhM4Xy5eYLExLlLC/IqEeiWllDOwJ6kkUYfMFSoZ5q4nATovb1vGyzyKtqmfTr54fRmy5eT5m60x8ySQf+5Kyikt8lauvtUppiAUbxzw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR05MB4866.eurprd05.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(376002)(136003)(366004)(396003)(39860400002)(346002)(52116002)(6506007)(2906002)(316002)(6486002)(1076003)(5660300002)(8936002)(8676002)(478600001)(6666004)(6512007)(26005)(16526019)(186003)(66946007)(66556008)(66476007)(86362001)(2616005)(956004)(107886003)(66574015)(4326008)(83380400001)(36756003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: i36lgdr+L6wtbyDP9dQLd6p460ysGWZnl9Mzv8pDG3YjU/fyexxctYlM/AOhlpb9QFKv/u2YzR7pJZO+ctk7vHL2kPUCi6Zdn4LgTiltm8yDvglZrW1cCEyCsj38/RL4upZHbF9Po1V29n/qc1OqQlxhW3ACItTb6e6XdAYBK2zyp/sN2Fku7aegZBoUm8ZD7ol7oJlJQeFkxjDaCujLaoEGBI/wOIheqYejc6ptsxdeGmS+VdL+opT9R6t+m2PYxbhHCOc0jfWBA9hH6EeRHQzUuQH1HkOoOd0Fx2QA6hBegBmy2JQhzkdViVcujUxtLnOoRHRr3S/tdJ7T5Ba4NkxnZt4j7bFGHyfvhaPCljvsHV3eJVYcBZvuSNOvVciS9ArmNlkVrWs9PbgLpEm6MBikgxREx7A9EWeOuUYwrzu8LPlaeLAkrfGOy3OJDWAGqv2rNiDbntXZ7fVa1wYjqbLI08iPTIoCgWD9OWcuhEQ= X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 038a6cf4-8909-43d4-09e8-08d8325529bf X-MS-Exchange-CrossTenant-AuthSource: AM0PR05MB4866.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jul 2020 17:47:42.8948 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Mb+uEIILQUeZqKJFTzvvXiZ1S1QDmS7lgRbSyvc7qDgsXKih98YpwL1q/0Bx65i4V6ZYErQvDD8kX8BfS+IaPQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR05MB6227 Subject: [dpdk-dev] [PATCH v12 00/10] Improve mlx5 PMD driver framework for multiple classes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This series introduces mlx5 common driver layer to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, mlx5 common module is extended as generic PCI PMD. This PMD enables multiple class PMDS (net, regex, vdpa) to be loaded on a single PCI Device. Change description ------------------ Patch-1 Introduces RTE_BIT() macro Patch-2 Fixed indentation for directory listing Patch-3 Relax dependency order Patch-4 Fixes compilation error Patch-5 Fixes segmentation fault in regex probe error unwind path Patch-6 Avoid using class constructor priority Patch-7 Change class values as bits Patch-8 Introduce mlx5 layer to support multiple class drivers Patch-9 Migrates mlx5 net and vdpa driver to use mlx5 PCI driver API instead of rte PCI bus API Patch-10 Removed class check code from class drivers Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ --------------- / \______| mlx5 common |____/ | pmd | --------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 common driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD define their specific PCI ID tables. mlx5 common driver probes each individual class driver(s) for maching and enabled classes. - mlx5 pci layer is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI common driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci common PMD, a common driver is implemented which exposes class registration API. However, common PMD model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common driver. It also eliminates peer blind zone problem as bottom pci layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w ,class=net:regex ./testpmd -w ,class=vdpa Changelog: v11->v12: - Define RTE_BIT32 macro as well - Use RTE_BIT32 and RTE_BIT64 macros in bitops as well - Updated release notes to cover mlx5 changes v10->v11: - Added vdpa + regex combination to class combination table - All class defintions are moved to single patch v9->v10: - Fixed checkpatch warnings related to spelling mistakes and alignment v8->v9: - Updated commit message - Fixed LD_FLAGS - Fixed white space to tab - Fixed Makefile to have dependency on common/mlx5 for parallel builds v7->v8: - Instead of dedicated mlx5 bus driver, merged the PMD to common_mlx5 - Avoided new RTE PRIORITY - Removed mlx5 common to use RTE_PRIORITY_CLASS v6->v7: - Updated release notes v5->v6: - Fixed compilation failure in parallel build for shared lib v4->v5: - Squash the maintainers update path with other patch which adds the bus - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Added RTE priority for common driver initialization - Addressed comments from Thomas and Asaf - Fixed compilation error in glue wrapper - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address commnts from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Deriving drv_flags based on the class drivers - Fixed alignment for id_table - Perform dma map on best effort basis for all supported drivers - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - Use uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (8): eal: introduce macro for bit definition common/mlx5: fix void parameters in glue wrappers regex/mlx5: fix segmentation fault during error unwinding common/mlx5: avoid using class constructor priority common/mlx5: change class values as bits common/mlx5: introduce layer to support multiple class drivers common/mlx5: register class drivers through common layer common/mlx5: remove class check from class drivers Thomas Monjalon (2): drivers: fix indent of directory list drivers: relax dependency order doc/guides/rel_notes/release_20_08.rst | 6 + drivers/Makefile | 10 +- drivers/common/Makefile | 4 - drivers/common/meson.build | 2 +- drivers/common/mlx5/Makefile | 2 + drivers/common/mlx5/linux/mlx5_glue.h | 4 +- drivers/common/mlx5/meson.build | 9 +- drivers/common/mlx5/mlx5_common.c | 52 +- drivers/common/mlx5/mlx5_common.h | 12 +- drivers/common/mlx5/mlx5_common_pci.c | 540 ++++++++++++++++++ drivers/common/mlx5/mlx5_common_pci.h | 77 +++ .../common/mlx5/rte_common_mlx5_version.map | 4 +- drivers/meson.build | 51 +- drivers/net/mlx5/Makefile | 1 - drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/mlx5.c | 25 +- drivers/net/mlx5/mlx5.h | 1 - drivers/regex/mlx5/meson.build | 2 +- drivers/regex/mlx5/mlx5_regex.c | 29 +- drivers/vdpa/mlx5/Makefile | 2 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 29 +- lib/librte_eal/include/rte_bitops.h | 34 +- 23 files changed, 772 insertions(+), 132 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_pci.c create mode 100644 drivers/common/mlx5/mlx5_common_pci.h