From patchwork Fri Aug 25 10:19:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shreyansh Jain X-Patchwork-Id: 27955 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id BE9A27D9F; Fri, 25 Aug 2017 12:10:42 +0200 (CEST) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on0041.outbound.protection.outlook.com [104.47.36.41]) by dpdk.org (Postfix) with ESMTP id 5470B7D6E for ; Fri, 25 Aug 2017 12:10:38 +0200 (CEST) Received: from MWHPR03CA0003.namprd03.prod.outlook.com (10.175.133.141) by CY1PR03MB2266.namprd03.prod.outlook.com (10.166.207.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1362.18; Fri, 25 Aug 2017 10:10:36 +0000 Received: from BL2FFO11FD028.protection.gbl (2a01:111:f400:7c09::145) by MWHPR03CA0003.outlook.office365.com (2603:10b6:300:117::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1385.8 via Frontend Transport; Fri, 25 Aug 2017 10:10:36 +0000 Authentication-Results: spf=fail (sender IP is 192.88.168.50) smtp.mailfrom=nxp.com; nxp.com; dkim=none (message not signed) header.d=none;nxp.com; dmarc=fail action=none header.from=nxp.com; Received-SPF: Fail (protection.outlook.com: domain of nxp.com does not designate 192.88.168.50 as permitted sender) receiver=protection.outlook.com; client-ip=192.88.168.50; helo=tx30smr01.am.freescale.net; Received: from tx30smr01.am.freescale.net (192.88.168.50) by BL2FFO11FD028.mail.protection.outlook.com (10.173.161.107) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1341.15 via Frontend Transport; Fri, 25 Aug 2017 10:10:35 +0000 Received: from Tophie.ap.freescale.net ([10.232.14.39]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id v7PAAHC4025398; Fri, 25 Aug 2017 03:10:33 -0700 From: Shreyansh Jain To: CC: , , , Shreyansh Jain Date: Fri, 25 Aug 2017 15:49:54 +0530 Message-ID: <20170825101954.28124-7-shreyansh.jain@nxp.com> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20170825101954.28124-1-shreyansh.jain@nxp.com> References: <20170818125127.22346-1-shreyansh.jain@nxp.com> <20170825101954.28124-1-shreyansh.jain@nxp.com> X-EOPAttributedMessage: 0 X-Matching-Connectors: 131481294361347314; (91ab9b29-cfa4-454e-5278-08d120cd25b8); () X-Forefront-Antispam-Report: CIP:192.88.168.50; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(336005)(39380400002)(39860400002)(2980300002)(1109001)(1110001)(339900001)(189002)(199003)(53936002)(106466001)(33646002)(6916009)(36756003)(2906002)(626005)(54906002)(50986999)(76176999)(77096006)(2351001)(2950100002)(6666003)(110136004)(8656003)(105606002)(48376002)(85426001)(4326008)(97736004)(498600001)(50466002)(8676002)(189998001)(50226002)(356003)(47776003)(1076002)(81156014)(8936002)(81166006)(5003940100001)(68736007)(305945005)(575784001)(86362001)(104016004)(5660300001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY1PR03MB2266; H:tx30smr01.am.freescale.net; FPR:; SPF:Fail; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BL2FFO11FD028; 1:7xVox4r7azV4xIH3uwWz7GwaEYZG00MW7X1V0v8DVe0I8HDj6lJtbKub/ib3Go0oj919rnOMDh/ZqkWJMASeAUlsMZ7WysFU4xPXHzK1mlZlEGdmfeQJBgKoU2woZI6T MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7b87f94e-9269-4215-e391-08d4eba18778 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(300000502095)(300135100095)(22001)(300000503095)(300135400095)(2017052603199)(201703131430075)(201703131517081)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:CY1PR03MB2266; X-Microsoft-Exchange-Diagnostics: 1; CY1PR03MB2266; 3:AWBglNUcocEhSyrk8qkFFNbxj4DdqaYWYjXOGZfun7ZRifXeH923LPYgCymdjLq2KO24+5gVMDw3HpogqrYEbuBqco4HwMkfy9QD4uDugI6vfz4v4Km1EiqPrwpr5VAO3g/W7s7KpXgCsc7F9/Ea2aH/lD6f1yV0uYw+3lX7Ox6e+C6JDBPOFOHgfDdmDgvMDNUfPbe7LGNw5Lhr8SeP+jBfFGEkbz/FhRkJ0HeRvgfbWvWcesOIY5y5RfHLOtqmmJ5yMz/dty6EHTPAL0U1/i1bCW8kcr7YEIAb21j5fmMdfLdGroGR7IsqwdY8ed0YQrPmJmfCzmAoMgUWW0VsylxoPCDrseJRnBxKPGx5CUI=; 25:UylunPaJlihduwCCCnMgbrMy3cMbpbCaCsIhNmJJfnSh7WvFJF4+eG0nsnc/TAKiTQqHehNh5xu3NpxhvjecjXR3+eKdoQR8RT5yGOnxiGAwT1PL9Oev1vPW1JXKlbr35tKswkYlE0jmdaCmJN+X8q5QAP0sMcVKTWikROq2vJH6Y94sEVJx22LoBxS4fxZ0NPTdYxAdfA6gsREvFMgFQ1FNnLZlBHvbuaeOxK+BvBgDeGLU7AuWx0X4Ho7z0QPhGKZ3vjkuJUNyq8hE1f8BvKL4Ax54fwn1Edeio508Rwq3F8H8ftdGztP6cHPZN11g6+gIdq7Bn3gw+fVA8hGj2Q== X-MS-TrafficTypeDiagnostic: CY1PR03MB2266: X-Microsoft-Exchange-Diagnostics: 1; CY1PR03MB2266; 31:mxXkLrERb/CB8SqdAIF8qT7EFbb6w43Jl+QxqLFFGEkbuvfJqELfXBFpIbxz7fKYQKgD5um7s1DP0a7Q9+4J273r/2LpHXHur3jMjtwEWjY2HlSo/EWhN55hy22iXH1SoX2ebb/UlCcfxlevYcC0HArI76QGNR7mcpsFCS1znhNlrDZr1Kvk7PqMYgiead96lXH3fjU81ptxHysmWX7VgOSL2Dt6kainygcAgKyzxK8=; 4:TnBuyQ19Wsx+Zq0tAj7y0XrWkMc1PLcY5xPyMiRExw/oEEqpF+Fkcgr8sAuith9QVXr/pmO8h45FHn/TD2SC6GD2IwPEFzaxlMO2jeHU0jpkXAiNyVX6pBL+EKwBsw93HJZulO9CQUs0y53G+B53DSL3Qt0HVvUNQ3ow+xLVd0cUyaBYU0FVb1x26jVaOVyfm9CUNitHgFIWGHsTP83K9/tLaIEzb2+5QPy7cP3VcGCf+JFNFBROVAw3KF3mREu4ZJ3QBAXL2FGFPorFmq1+ireaEHq3L1R2yMBRtI2SBeSGbLiWxv6bfS3So5lie1iWt9p8WtyzowwzS4rDmb85SNK7bD2F9IyL3HEBt6raMzBV6Ey7eljkqmISCUpn8+bA X-Exchange-Antispam-Report-Test: UriScan:(185117386973197)(275809806118684)(21532816269658); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6095135)(601004)(2401047)(13016025)(13018025)(5005006)(8121501046)(10201501046)(3002001)(100000703101)(100105400095)(93006095)(93001095)(6055026)(6096035)(20161123561025)(20161123559100)(201703131430075)(201703131433075)(201703131441075)(201703131448075)(201703161259150)(20161123556025)(20161123565025)(20161123563025)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CY1PR03MB2266; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(400006)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CY1PR03MB2266; X-Forefront-PRVS: 041032FF37 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY1PR03MB2266; 23:fsiNsMOasI04uqJWIAqtkEtEd1njwPYa3ETKyIdrL?= GFXTvXU/LJbLBF6Cc1p4DJX61wB2RUExPf8ZIxbkYZV6wlqrCTABarTcRpyHrxLFFSgRNePnRFhWHIoJMFV5NCpFqzCLCc8d7V9KiZfZqnFmqUDP7fLppFnDmKO/XlJvpwT4xUnCR85PnfNjd4L2RZrq7pi0wzFvE5NhzTgdp2z1xizBpDVhlOneaRt/N/RZ0cThEhfitVSRUZ9eAD5pBB3BIJFbBGX+AtngG+5Y3t3A5/XmrfUHzhpkFvKOxcnblMr79MQuuv3nOawkK5X2QYautHQdesf2222wUbyf8M1PSnuqUiWDJ9Qzc+NDjjKR7OADUCSC0YoLbYU2+wzJ2rmWpvXQszw1L93HIqbzwAfXVQJRfebSpI3kq6P0YZ/9X/7GemNA9jO2G6cAx61DUVGxWsVCBEwl/tN4ai70tpvX4ndQjuuGsUvLRLQsgX93D47V3Jfs7I1XioRMjD562SJBvYqc2kpxlJHsxJeyegcaPY/B7oO8VGQBc1v2yfpHO7mQL0gegptr8wPVABBaq8+ww6gXixexP/4zX5n1G18GWZ5ix7EjnpHdvhwRMLdDAUzd6tSfvZrRqc2vVtfwVzAvSK8s89RtFsClQRONZ1D2bctA4Ci2UIC2cwsAmOz2vKcswX5yapTOzXokBVwRiNYEtVDptb+WjDYLARNKowgPSGoUS1nC1cZbZoN/oeuq9CiZVYDaJ+JOaNcQyVrFkTF8nS2vU14837RDeq4yOunnsqrW8zj7mg1V+rB17zZ7nTSWEBui2aYf6plHIh5pW7Dzc400TfhIbYpiKxf1jj8N7q223mzc+TqPMX6kKdfPNGaUZY2APp/qLIo3Ju0CGS0+IYnNxJEW0ZbLPccOU5AySoTEqTDReki4fmIsufmzyWuJ4qfNPgzQnUIjJ6U4iLhThaf1e5eebC5DPMFj9RAcasx4wBMpK4ddKKvz1zypm4tmGWxesWdkQ9ERNiMck3/5U73CwowPGsi817Sdqzdvso17vmZhWWzsEOBnAckzmVAU1bkTSY2V+d4Epcpf+80b0lqK/I7WVL2CH/ob3fEkTX2VTXDbjPpYXMPmnW5xjC2je7B+zmU5/78bAfLJHac2t6g/OjOUTpAKuzSanOgHZziYxhRogPqk1RP3E7uwd8= X-Microsoft-Exchange-Diagnostics: 1; CY1PR03MB2266; 6:1Jj2vg8hnjPBCwHp8tJZY37oSKrFxQy6GvVYxa7CeHuC+F//Afy6sYQiMRZhFHUQQvWN/zWUfZISAGj2qZvGEI68YAVET/XGoBSYOcH7BtcA2e3IOvnaWFSgUJKWN2IVvIXKuTXIai9hXoQQJZRVnOc0Cy/P9AZTLy+lw/akn5pYl4UQ+rmRoO55ZMJoNv7HJNssTnSK3tNy19TMR60UWFtxo0w1H2AviFp24fcZM7rkmqNi08BPBj5/4RR7xNaXP+ru4DbwNYd3qJG1vi0/zpv12t+ud7oYhPYAZwbF6jvil3x1ztn882HxkQOycDP6L+EM30F5JQ/dKvvpWwA5kA==; 5:oAHi4XeBeymBdDPU89Kl0w0kKMzJMRpZljudwlb3HV7i8uHOS+0WhH7Df3Vf3+juFvtFWbRT1RWEzSZXSdvR2KRAnrHJw4JPyNDmwxqTSqn4stdZI4C4ai7A1Sh6Bf9POURA0qOS42H5HeRqJSsrWQ==; 24:3cMZhHAcwu/zziKw7S192WSY+o3F38bLXoTrNP0a+VHXXVu+UT0A0QZG6W9P4gMD4PIm/3L+qYC/0Gpf5mpYVgb+9HeE4gSFaxP4s9JsLSQ=; 7:bi+ZPrN4R2TfYJKY4QupHzvB3/KRCM/xp79XRH5mfHavFcbZdUnYTVuFVMrlJ2yjpVrKd3V7g6QV7awYyS4dtDqwV46Fc3nI/fBH5aY/saRbCO2c+VmGQBn8VtRaIrBI4alTDoufNgsWgMbhXOdA70pdAmeOu0FzS0OXW+EI8XKAhk8cz9Qr2hGw3k38Up9vMgOD7vTWYuv7lR0sG4l+t73K23rcWJYmtHd3cRnNb00= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2017 10:10:35.9475 (UTC) X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR03MB2266 Subject: [dpdk-dev] [PATCH v2 6/6] bus/fslmc: refactor scan and probe functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Scan now searches for devices through sysfs interface and adds them to a list for later initialization. During probe, each device is initialized accroding to its property. Signed-off-by: Shreyansh Jain --- drivers/bus/fslmc/fslmc_bus.c | 180 +++++++++++- drivers/bus/fslmc/fslmc_vfio.c | 455 ++++++++++++++----------------- drivers/bus/fslmc/fslmc_vfio.h | 4 +- drivers/bus/fslmc/portal/dpaa2_hw_dpio.c | 1 - drivers/bus/fslmc/rte_fslmc.h | 3 + 5 files changed, 381 insertions(+), 262 deletions(-) diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c index f71598d..d50c303 100644 --- a/drivers/bus/fslmc/fslmc_bus.c +++ b/drivers/bus/fslmc/fslmc_bus.c @@ -42,32 +42,170 @@ #include #include -#include "rte_fslmc.h" -#include "fslmc_vfio.h" +#include +#include #define FSLMC_BUS_LOG(level, fmt, args...) \ - RTE_LOG(level, EAL, "%s(): " fmt "\n", __func__, ##args) + RTE_LOG(level, EAL, fmt "\n", ##args) + +#define VFIO_IOMMU_GROUP_PATH "/sys/kernel/iommu_groups" struct rte_fslmc_bus rte_fslmc_bus; +static void +cleanup_fslmc_device_list(void) +{ + struct rte_dpaa2_device *dev; + struct rte_dpaa2_device *t_dev; + + TAILQ_FOREACH_SAFE(dev, &rte_fslmc_bus.device_list, next, t_dev) { + TAILQ_REMOVE(&rte_fslmc_bus.device_list, dev, next); + free(dev); + dev = NULL; + } +} + +static int +scan_one_fslmc_device(char *dev_name) +{ + char *dup_dev_name, *t_ptr; + struct rte_dpaa2_device *dev; + + if (!dev_name) + return -1; + + /* Ignore the Container name itself */ + if (!strncmp("dprc", dev_name, 4)) + return 0; + + /* Creating a temporary copy to perform cut-parse over string */ + dup_dev_name = strdup(dev_name); + if (!dup_dev_name) { + FSLMC_BUS_LOG(ERR, "Out of memory."); + return -ENOMEM; + } + + /* For all other devices, we allocate rte_dpaa2_device. + * For those devices where there is no driver, probe would release + * the memory associated with the rte_dpaa2_device after necessary + * initialization. + */ + dev = calloc(1, sizeof(struct rte_dpaa2_device)); + if (!dev) { + FSLMC_BUS_LOG(ERR, "Out of memory."); + free(dup_dev_name); + return -ENOMEM; + } + + /* Parse the device name and ID */ + t_ptr = strtok(dup_dev_name, "."); + if (!t_ptr) { + FSLMC_BUS_LOG(ERR, "Incorrect device string observed."); + goto cleanup; + } + if (!strncmp("dpni", t_ptr, 4)) + dev->dev_type = DPAA2_ETH; + else if (!strncmp("dpseci", t_ptr, 6)) + dev->dev_type = DPAA2_CRYPTO; + else if (!strncmp("dpcon", t_ptr, 5)) + dev->dev_type = DPAA2_CON; + else if (!strncmp("dpbp", t_ptr, 4)) + dev->dev_type = DPAA2_BPOOL; + else if (!strncmp("dpio", t_ptr, 4)) + dev->dev_type = DPAA2_IO; + else if (!strncmp("dpci", t_ptr, 5)) + dev->dev_type = DPAA2_CI; + else if (!strncmp("dpmcp", t_ptr, 5)) + dev->dev_type = DPAA2_MPORTAL; + else + dev->dev_type = DPAA2_UNKNOWN; + + t_ptr = strtok(NULL, "."); + if (!t_ptr) { + FSLMC_BUS_LOG(ERR, "Incorrect device string observed (%s).", + t_ptr); + goto cleanup; + } + + sscanf(t_ptr, "%hu", &dev->object_id); + dev->device.name = strdup(dev_name); + if (!dev->device.name) { + FSLMC_BUS_LOG(ERR, "Out of memory."); + goto cleanup; + } + + /* Add device in the fslmc device list */ + TAILQ_INSERT_TAIL(&rte_fslmc_bus.device_list, dev, next); + + /* Don't need the duplicated device filesystem entry anymore */ + if (dup_dev_name) + free(dup_dev_name); + + return 0; +cleanup: + if (dup_dev_name) + free(dup_dev_name); + if (dev) + free(dev); + return -1; +} + static int rte_fslmc_scan(void) { int ret; + int device_count = 0; + char fslmc_dirpath[PATH_MAX]; + DIR *dir; + struct dirent *entry; + static int process_once; + int groupid; - ret = fslmc_vfio_setup_group(); - if (ret) { - FSLMC_BUS_LOG(ERR, "fslmc: Unable to setup VFIO"); - return ret; + if (process_once) { + FSLMC_BUS_LOG(DEBUG, + "Fslmc bus already scanned. Not rescanning"); + return 0; } + process_once = 1; - ret = fslmc_vfio_process_group(); - if (ret) { - FSLMC_BUS_LOG(ERR, "fslmc: Unable to setup devices"); - return -1; + ret = fslmc_get_container_group(&groupid); + if (ret != 0) + goto scan_fail; + + /* Scan devices on the group */ + sprintf(fslmc_dirpath, "%s/%d/devices", VFIO_IOMMU_GROUP_PATH, + groupid); + dir = opendir(fslmc_dirpath); + if (!dir) { + FSLMC_BUS_LOG(ERR, "Unable to open VFIO group dir."); + goto scan_fail; } - RTE_LOG(INFO, EAL, "fslmc: Bus scan completed\n"); + while ((entry = readdir(dir)) != NULL) { + if (entry->d_name[0] == '.' || entry->d_type != DT_LNK) + continue; + + ret = scan_one_fslmc_device(entry->d_name); + if (ret != 0) { + /* Error in parsing directory - exit gracefully */ + goto scan_fail_cleanup; + } + device_count += 1; + } + + FSLMC_BUS_LOG(INFO, "fslmc: Bus scan completed"); + + closedir(dir); + return 0; + +scan_fail_cleanup: + closedir(dir); + + /* Remove all devices in the list */ + cleanup_fslmc_device_list(); +scan_fail: + FSLMC_BUS_LOG(DEBUG, "FSLMC Bus Not Available. Skipping."); + /* Irrespective of failure, scan only return success */ return 0; } @@ -88,6 +226,21 @@ rte_fslmc_probe(void) struct rte_dpaa2_device *dev; struct rte_dpaa2_driver *drv; + if (TAILQ_EMPTY(&rte_fslmc_bus.device_list)) + return 0; + + ret = fslmc_vfio_setup_group(); + if (ret) { + FSLMC_BUS_LOG(ERR, "Unable to setup VFIO %d", ret); + return 0; + } + + ret = fslmc_vfio_process_group(); + if (ret) { + FSLMC_BUS_LOG(ERR, "Unable to setup devices %d", ret); + return 0; + } + TAILQ_FOREACH(dev, &rte_fslmc_bus.device_list, next) { TAILQ_FOREACH(drv, &rte_fslmc_bus.driver_list, next) { ret = rte_fslmc_match(drv, dev); @@ -103,7 +256,8 @@ rte_fslmc_probe(void) break; } } - return ret; + + return 0; } static struct rte_device * diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c index f0d69a9..a1309b3 100644 --- a/drivers/bus/fslmc/fslmc_vfio.c +++ b/drivers/bus/fslmc/fslmc_vfio.c @@ -63,18 +63,20 @@ #include "portal/dpaa2_hw_dpio.h" #define FSLMC_VFIO_LOG(level, fmt, args...) \ - RTE_LOG(level, EAL, "%s(): " fmt "\n", __func__, ##args) + RTE_LOG(level, EAL, fmt "\n", ##args) /** Pathname of FSL-MC devices directory. */ #define SYSFS_FSL_MC_DEVICES "/sys/bus/fsl-mc/devices" +#define FSLMC_CONTAINER_MAX_LEN 8 /**< Of the format dprc.XX */ + /* Number of VFIO containers & groups with in */ static struct fslmc_vfio_group vfio_group; static struct fslmc_vfio_container vfio_container; static int container_device_fd; +static char *g_container; static uint32_t *msi_intr_vaddr; void *(*rte_mcp_ptr_list); -static uint32_t mcp_id; static int is_dma_done; static struct rte_dpaa2_object_list dpaa2_obj_list = @@ -89,10 +91,56 @@ rte_fslmc_object_register(struct rte_dpaa2_object *object) TAILQ_INSERT_TAIL(&dpaa2_obj_list, object, next); } -static int vfio_connect_container(void) +int +fslmc_get_container_group(int *groupid) +{ + int ret; + char *container; + + if (!g_container) { + container = getenv("DPRC"); + if (container == NULL) { + RTE_LOG(WARNING, EAL, "DPAA2: DPRC not available\n"); + return -EINVAL; + } + + if (strlen(container) >= FSLMC_CONTAINER_MAX_LEN) { + FSLMC_VFIO_LOG(ERR, "Invalid container name: %s\n", + container); + return -1; + } + + g_container = strdup(container); + if (!g_container) { + FSLMC_VFIO_LOG(ERR, "Out of memory."); + return -ENOMEM; + } + } + + /* get group number */ + ret = vfio_get_group_no(SYSFS_FSL_MC_DEVICES, g_container, groupid); + if (ret <= 0) { + FSLMC_VFIO_LOG(ERR, "Unable to find %s IOMMU group", + g_container); + return -1; + } + + FSLMC_VFIO_LOG(DEBUG, "Container: %s has VFIO iommu group id = %d", + g_container, *groupid); + + return 0; +} + +static int +vfio_connect_container(void) { int fd, ret; + if (vfio_container.used) { + FSLMC_VFIO_LOG(DEBUG, "No container available."); + return -1; + } + /* Try connecting to vfio container if already created */ if (!ioctl(vfio_group.fd, VFIO_GROUP_SET_CONTAINER, &vfio_container.fd)) { @@ -284,48 +332,6 @@ static int64_t vfio_map_mcp_obj(struct fslmc_vfio_group *group, char *mcp_obj) return v_addr; } -static inline int -dpaa2_compare_dpaa2_dev(const struct rte_dpaa2_device *dev, - const struct rte_dpaa2_device *dev2) -{ - /*not the same family device */ - if (dev->dev_type != DPAA2_ETH || - dev->dev_type != DPAA2_CRYPTO) - return -1; - - if (dev->object_id == dev2->object_id) - return 0; - else - return 1; -} - -static void -fslmc_bus_add_device(struct rte_dpaa2_device *dev) -{ - struct rte_fslmc_device_list *dev_l; - - dev_l = &rte_fslmc_bus.device_list; - - /* device is valid, add in list (sorted) */ - if (TAILQ_EMPTY(dev_l)) { - TAILQ_INSERT_TAIL(dev_l, dev, next); - } else { - struct rte_dpaa2_device *dev2; - int ret; - - TAILQ_FOREACH(dev2, dev_l, next) { - ret = dpaa2_compare_dpaa2_dev(dev, dev2); - if (ret <= 0) - continue; - - TAILQ_INSERT_BEFORE(dev2, dev, next); - return; - } - - TAILQ_INSERT_TAIL(dev_l, dev, next); - } -} - #define IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + sizeof(int)) int rte_dpaa2_intr_enable(struct rte_intr_handle *intr_handle, @@ -360,283 +366,240 @@ int rte_dpaa2_intr_enable(struct rte_intr_handle *intr_handle, return 0; } -/* Following function shall fetch total available list of MC devices - * from VFIO container & populate private list of devices and other - * data structures +/* + * fslmc_process_iodevices for processing only IO (ETH, CRYPTO, and possibly + * EVENT) devices. */ -int fslmc_vfio_process_group(void) +static int +fslmc_process_iodevices(struct rte_dpaa2_device *dev) { - struct fslmc_vfio_device *vdev; + int dev_fd; struct vfio_device_info device_info = { .argsz = sizeof(device_info) }; - char *temp_obj, *object_type, *mcp_obj, *dev_name; - int32_t object_id, i, dev_fd, ret; - DIR *d; - struct dirent *dir; - char path[PATH_MAX]; - int64_t v_addr; - int ndev_count; - struct fslmc_vfio_group *group = &vfio_group; - static int process_once; + struct rte_dpaa2_object *object = NULL; - /* if already done once */ - if (process_once) { - FSLMC_VFIO_LOG(DEBUG, - "Already scanned once - re-scan not supported"); - return 0; + dev_fd = ioctl(vfio_group.fd, VFIO_GROUP_GET_DEVICE_FD, + dev->device.name); + if (dev_fd <= 0) { + FSLMC_VFIO_LOG(ERR, "Unable to obtain device FD for device:%s", + dev->device.name); + return -1; } - process_once = 0; - - sprintf(path, "/sys/kernel/iommu_groups/%d/devices", group->groupid); - d = opendir(path); - if (!d) { - FSLMC_VFIO_LOG(ERR, "Unable to open directory %s", path); + if (ioctl(dev_fd, VFIO_DEVICE_GET_INFO, &device_info)) { + FSLMC_VFIO_LOG(ERR, "DPAA2 VFIO_DEVICE_GET_INFO fail"); return -1; } - /*Counting the number of devices in a group and getting the mcp ID*/ - ndev_count = 0; - mcp_obj = NULL; - while ((dir = readdir(d)) != NULL) { - if (dir->d_type == DT_LNK) { - ndev_count++; - if (!strncmp("dpmcp", dir->d_name, 5)) { - if (mcp_obj) - free(mcp_obj); - mcp_obj = malloc(sizeof(dir->d_name)); - if (!mcp_obj) { - FSLMC_VFIO_LOG(ERR, - "mcp obj:alloc failed"); - closedir(d); - return -ENOMEM; - } - strcpy(mcp_obj, dir->d_name); - temp_obj = strtok(dir->d_name, "."); - temp_obj = strtok(NULL, "."); - sscanf(temp_obj, "%d", &mcp_id); - } + switch (dev->dev_type) { + case DPAA2_CON: + case DPAA2_IO: + case DPAA2_CI: + case DPAA2_BPOOL: + TAILQ_FOREACH(object, &dpaa2_obj_list, next) { + if (dev->dev_type == object->dev_type) + object->create(dev_fd, &device_info, + dev->object_id); + else + continue; } - } - closedir(d); - d = NULL; - if (!mcp_obj) { - FSLMC_VFIO_LOG(ERR, "DPAA2 MCP Object not Found"); - return -ENODEV; - } - RTE_LOG(INFO, EAL, "fslmc: DPRC contains = %d devices\n", ndev_count); - - /* Allocate the memory depends upon number of objects in a group*/ - group->vfio_device = (struct fslmc_vfio_device *)malloc(ndev_count * - sizeof(struct fslmc_vfio_device)); - if (!(group->vfio_device)) { - FSLMC_VFIO_LOG(ERR, "vfio device: Unable to allocate memory\n"); - free(mcp_obj); - return -ENOMEM; + break; + default: + break; } - /* Allocate memory for MC Portal list */ + FSLMC_VFIO_LOG(DEBUG, "Device (%s) abstracted from VFIO", + dev->device.name); + return 0; +} + +static int +fslmc_process_mcp(struct rte_dpaa2_device *dev) +{ + int64_t v_addr; + char *dev_name; + rte_mcp_ptr_list = malloc(sizeof(void *) * 1); if (!rte_mcp_ptr_list) { - FSLMC_VFIO_LOG(ERR, "portal list: Unable to allocate memory!"); - free(mcp_obj); - goto FAILURE; + FSLMC_VFIO_LOG(ERR, "Out of memory"); + return -ENOMEM; } - v_addr = vfio_map_mcp_obj(group, mcp_obj); - free(mcp_obj); - if (v_addr == (int64_t)MAP_FAILED) { - FSLMC_VFIO_LOG(ERR, "Error mapping region (errno = %d)", errno); - goto FAILURE; + dev_name = strdup(dev->device.name); + if (!dev_name) { + FSLMC_VFIO_LOG(ERR, "Out of memory."); + free(rte_mcp_ptr_list); + rte_mcp_ptr_list = NULL; + return -ENOMEM; } - rte_mcp_ptr_list[0] = (void *)v_addr; - - d = opendir(path); - if (!d) { - FSLMC_VFIO_LOG(ERR, "Unable to open %s Directory", path); - goto FAILURE; + v_addr = vfio_map_mcp_obj(&vfio_group, dev_name); + if (v_addr == (int64_t)MAP_FAILED) { + FSLMC_VFIO_LOG(ERR, "Error mapping region (errno = %d)", + errno); + free(rte_mcp_ptr_list); + rte_mcp_ptr_list = NULL; + return -1; } - i = 0; - /* Parsing each object and initiating them*/ - while ((dir = readdir(d)) != NULL) { - if (dir->d_type != DT_LNK) - continue; - if (!strncmp("dprc", dir->d_name, 4) || - !strncmp("dpmcp", dir->d_name, 5)) - continue; - dev_name = malloc(sizeof(dir->d_name)); - if (!dev_name) { - FSLMC_VFIO_LOG(ERR, "name: Unable to allocate memory"); - goto FAILURE; - } - strcpy(dev_name, dir->d_name); - object_type = strtok(dir->d_name, "."); - temp_obj = strtok(NULL, "."); - sscanf(temp_obj, "%d", &object_id); - - /* getting the device fd*/ - dev_fd = ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, dev_name); - if (dev_fd < 0) { - FSLMC_VFIO_LOG(ERR, - "GET_DEVICE_FD error fd: %s, Group: %d", - dev_name, group->fd); - free(dev_name); - goto FAILURE; - } + rte_mcp_ptr_list[0] = (void *)v_addr; - free(dev_name); - vdev = &group->vfio_device[group->object_index++]; - vdev->fd = dev_fd; - vdev->index = i; - i++; - /* Get Device inofrmation */ - if (ioctl(vdev->fd, VFIO_DEVICE_GET_INFO, &device_info)) { - FSLMC_VFIO_LOG(ERR, "DPAA2 VFIO_DEVICE_GET_INFO fail"); - goto FAILURE; - } - if (!strcmp(object_type, "dpni") || - !strcmp(object_type, "dpseci")) { - struct rte_dpaa2_device *dev; + return 0; +} - dev = malloc(sizeof(struct rte_dpaa2_device)); - if (dev == NULL) +int +fslmc_vfio_process_group(void) +{ + int ret; + int found_mportal = 0; + struct rte_dpaa2_device *dev, *dev_temp; + + /* Search the MCP as that should be initialized first. */ + TAILQ_FOREACH_SAFE(dev, &rte_fslmc_bus.device_list, next, dev_temp) { + if (dev->dev_type == DPAA2_MPORTAL) { + ret = fslmc_process_mcp(dev); + if (ret) { + FSLMC_VFIO_LOG(DEBUG, "Unable to map Portal."); return -1; - - memset(dev, 0, sizeof(*dev)); - /* store hw_id of dpni/dpseci device */ - dev->object_id = object_id; - dev->dev_type = (strcmp(object_type, "dpseci")) ? - DPAA2_ETH : DPAA2_CRYPTO; - - sprintf(dev->name, "%s.%d", object_type, object_id); - dev->device.name = dev->name; - - fslmc_bus_add_device(dev); - FSLMC_VFIO_LOG(DEBUG, "DPAA2: Added %s", dev->name); - } else { - /* Parse all other objects */ - struct rte_dpaa2_object *object; - - TAILQ_FOREACH(object, &dpaa2_obj_list, next) { - if (!strcmp(object_type, object->name)) - object->create(dev_fd, &device_info, - object_id); - else - continue; } + if (!found_mportal) + found_mportal = 1; + + TAILQ_REMOVE(&rte_fslmc_bus.device_list, dev, next); + free(dev); + dev = NULL; + /* Ideally there is only a single dpmcp, but in case + * multiple exists, looping on remaining devices. + */ } } - closedir(d); - ret = dpaa2_affine_qbman_swp(); - if (ret) - FSLMC_VFIO_LOG(DEBUG, "Error in affining qbman swp %d", ret); + /* Cannot continue if there is not even a single mportal */ + if (!found_mportal) { + FSLMC_VFIO_LOG(DEBUG, + "No MC Portal device found. Not continuing."); + return -1; + } - return 0; + TAILQ_FOREACH_SAFE(dev, &rte_fslmc_bus.device_list, next, dev_temp) { + if (!dev) + break; -FAILURE: - if (d) - closedir(d); - if (rte_mcp_ptr_list) { - free(rte_mcp_ptr_list); - rte_mcp_ptr_list = NULL; + switch (dev->dev_type) { + case DPAA2_ETH: + case DPAA2_CRYPTO: + ret = fslmc_process_iodevices(dev); + if (ret) { + FSLMC_VFIO_LOG(DEBUG, + "Dev (%s) init failed.", + dev->device.name); + return ret; + } + break; + case DPAA2_CON: + case DPAA2_IO: + case DPAA2_CI: + case DPAA2_BPOOL: + /* Call the object creation routine and remove the + * device entry from device list + */ + ret = fslmc_process_iodevices(dev); + if (ret) { + FSLMC_VFIO_LOG(DEBUG, + "Dev (%s) init failed.", + dev->device.name); + return -1; + } + + /* This device is not required to be in the DPDK + * exposed device list. + */ + TAILQ_REMOVE(&rte_fslmc_bus.device_list, dev, next); + free(dev); + dev = NULL; + break; + case DPAA2_UNKNOWN: + default: + /* Unknown - ignore */ + FSLMC_VFIO_LOG(DEBUG, "Found unknown device (%s).", + dev->device.name); + TAILQ_REMOVE(&rte_fslmc_bus.device_list, dev, next); + free(dev); + dev = NULL; + } } - free(group->vfio_device); - group->vfio_device = NULL; - return -1; + return 0; } -int fslmc_vfio_setup_group(void) +int +fslmc_vfio_setup_group(void) { - struct fslmc_vfio_group *group = NULL; int groupid; int ret; - char *container; struct vfio_group_status status = { .argsz = sizeof(status) }; /* if already done once */ if (container_device_fd) return 0; - container = getenv("DPRC"); - - if (container == NULL) { - FSLMC_VFIO_LOG(ERR, "VFIO container not set in env DPRC"); - return -EOPNOTSUPP; - } - - /* get group number */ - ret = vfio_get_group_no(SYSFS_FSL_MC_DEVICES, container, &groupid); - if (ret == 0) { - RTE_LOG(WARNING, EAL, "%s not managed by VFIO, skipping\n", - container); - return -EOPNOTSUPP; - } - - /* if negative, something failed */ - if (ret < 0) + ret = fslmc_get_container_group(&groupid); + if (ret) return ret; - FSLMC_VFIO_LOG(DEBUG, "VFIO iommu group id = %d", groupid); - - /* Check if group already exists */ - group = &vfio_group; - if (group->groupid == groupid) { - FSLMC_VFIO_LOG(ERR, "groupid already exists %d", - groupid); + /* In case this group was already opened, continue without any + * processing. + */ + if (vfio_group.groupid == groupid) { + FSLMC_VFIO_LOG(ERR, "groupid already exists %d", groupid); return 0; } - /* get the actual group fd */ + /* Get the actual group fd */ ret = vfio_get_group_fd(groupid); if (ret < 0) return ret; - group->fd = ret; + vfio_group.fd = ret; - /* - * at this point, we know that this group is viable (meaning, - * all devices are either bound to VFIO or not bound to anything) - */ - - ret = ioctl(group->fd, VFIO_GROUP_GET_STATUS, &status); + /* Check group viability */ + ret = ioctl(vfio_group.fd, VFIO_GROUP_GET_STATUS, &status); if (ret) { - FSLMC_VFIO_LOG(ERR, " VFIO error getting group status"); - close(group->fd); + FSLMC_VFIO_LOG(ERR, "VFIO error getting group status"); + close(vfio_group.fd); return ret; } if (!(status.flags & VFIO_GROUP_FLAGS_VIABLE)) { FSLMC_VFIO_LOG(ERR, "VFIO group not viable"); - close(group->fd); + close(vfio_group.fd); return -EPERM; } /* Since Group is VIABLE, Store the groupid */ - group->groupid = groupid; + vfio_group.groupid = groupid; /* check if group does not have a container yet */ if (!(status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) { /* Now connect this IOMMU group to given container */ ret = vfio_connect_container(); if (ret) { - FSLMC_VFIO_LOG(ERR, "VFIO error connecting container" - " with groupid %d", groupid); - close(group->fd); + FSLMC_VFIO_LOG(ERR, + "Error connecting container with groupid %d", + groupid); + close(vfio_group.fd); return ret; } } /* Get Device information */ - ret = ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, container); + ret = ioctl(vfio_group.fd, VFIO_GROUP_GET_DEVICE_FD, g_container); if (ret < 0) { - FSLMC_VFIO_LOG(ERR, "VFIO error getting device %s fd from" - " group %d", container, group->groupid); + FSLMC_VFIO_LOG(ERR, "Error getting device %s fd from group %d", + g_container, vfio_group.groupid); + close(vfio_group.fd); return ret; } container_device_fd = ret; FSLMC_VFIO_LOG(DEBUG, "VFIO Container FD is [0x%X]", - container_device_fd); + container_device_fd); return 0; } diff --git a/drivers/bus/fslmc/fslmc_vfio.h b/drivers/bus/fslmc/fslmc_vfio.h index 60a5743..edb86d0 100644 --- a/drivers/bus/fslmc/fslmc_vfio.h +++ b/drivers/bus/fslmc/fslmc_vfio.h @@ -36,7 +36,6 @@ #include "eal_vfio.h" -#define DPAA2_VENDOR_ID 0x1957 #define DPAA2_MC_DPNI_DEVID 7 #define DPAA2_MC_DPSECI_DEVID 3 #define DPAA2_MC_DPCON_DEVID 5 @@ -44,7 +43,6 @@ #define DPAA2_MC_DPBP_DEVID 10 #define DPAA2_MC_DPCI_DEVID 11 - typedef struct fslmc_vfio_device { int fd; /* fslmc root container device ?? */ int index; /*index of child object */ @@ -71,6 +69,8 @@ int rte_dpaa2_intr_enable(struct rte_intr_handle *intr_handle, int fslmc_vfio_setup_group(void); int fslmc_vfio_process_group(void); +char *fslmc_get_container(void); +int fslmc_get_container_group(int *gropuid); int rte_fslmc_vfio_dmamap(void); #endif /* _FSLMC_VFIO_H_ */ diff --git a/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c b/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c index da41849..7315555 100644 --- a/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c +++ b/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c @@ -59,7 +59,6 @@ #include #include -#include #include #include "dpaa2_hw_pvt.h" #include "dpaa2_hw_dpio.h" diff --git a/drivers/bus/fslmc/rte_fslmc.h b/drivers/bus/fslmc/rte_fslmc.h index 2f684d9..4c32db6 100644 --- a/drivers/bus/fslmc/rte_fslmc.h +++ b/drivers/bus/fslmc/rte_fslmc.h @@ -56,6 +56,9 @@ extern "C" { #include #include #include +#include + +#include #define FSLMC_OBJECT_MAX_LEN 32 /**< Length of each device on bus */