From patchwork Mon Dec 19 20:53:50 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hemant Agrawal X-Patchwork-Id: 18171 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 1590BFA6F; Mon, 19 Dec 2016 16:21:40 +0100 (CET) Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0042.outbound.protection.outlook.com [104.47.32.42]) by dpdk.org (Postfix) with ESMTP id A3B98FA38 for ; Mon, 19 Dec 2016 16:21:13 +0100 (CET) Received: from BY2PR03CA062.namprd03.prod.outlook.com (10.141.249.35) by DM2PR0301MB0752.namprd03.prod.outlook.com (10.160.97.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.771.8; Mon, 19 Dec 2016 15:21:10 +0000 Received: from BN1BFFO11FD034.protection.gbl (2a01:111:f400:7c10::1:154) by BY2PR03CA062.outlook.office365.com (2a01:111:e400:2c5d::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.789.14 via Frontend Transport; Mon, 19 Dec 2016 15:21:10 +0000 Authentication-Results: spf=fail (sender IP is 192.88.158.2) smtp.mailfrom=nxp.com; nxp.com; dkim=none (message not signed) header.d=none; nxp.com; dmarc=fail action=none header.from=nxp.com; nxp.com; dkim=none (message not signed) header.d=none; Received-SPF: Fail (protection.outlook.com: domain of nxp.com does not designate 192.88.158.2 as permitted sender) receiver=protection.outlook.com; client-ip=192.88.158.2; helo=az84smr01.freescale.net; Received: from az84smr01.freescale.net (192.88.158.2) by BN1BFFO11FD034.mail.protection.outlook.com (10.58.144.97) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.789.10 via Frontend Transport; Mon, 19 Dec 2016 15:21:10 +0000 Received: from bf-netperf1.idc ([10.232.134.28]) by az84smr01.freescale.net (8.14.3/8.14.0) with ESMTP id uBJFKMha029110; Mon, 19 Dec 2016 08:21:06 -0700 From: Hemant Agrawal To: CC: , , , , , , Hemant Agrawal Date: Tue, 20 Dec 2016 02:23:50 +0530 Message-ID: <1482180853-18823-12-git-send-email-hemant.agrawal@nxp.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1482180853-18823-1-git-send-email-hemant.agrawal@nxp.com> References: <1480875447-23680-1-git-send-email-hemant.agrawal@nxp.com> <1482180853-18823-1-git-send-email-hemant.agrawal@nxp.com> X-EOPAttributedMessage: 0 X-Matching-Connectors: 131266344704839775; (91ab9b29-cfa4-454e-5278-08d120cd25b8); () X-Forefront-Antispam-Report: CIP:192.88.158.2; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(7916002)(336005)(39450400003)(39860400002)(39850400002)(39840400002)(39400400002)(39410400002)(39380400002)(2980300002)(1110001)(1109001)(339900001)(199003)(189002)(50986999)(76176999)(2351001)(68736007)(106466001)(47776003)(36756003)(50466002)(97736004)(2906002)(48376002)(110136003)(189998001)(33646002)(356003)(2950100002)(6666003)(104016004)(8666005)(5660300001)(6916009)(5003940100001)(85426001)(8936002)(8676002)(50226002)(626004)(81156014)(38730400001)(81166006)(4326007)(92566002)(305945005)(69596002)(77096006)(105606002)(86362001)(7059030); DIR:OUT; SFP:1101; SCL:1; SRVR:DM2PR0301MB0752; H:az84smr01.freescale.net; FPR:; SPF:Fail; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BN1BFFO11FD034; 1:xl6cMs4lEk9bBi4VHMJxhUYGdxjrZRSk0vQBOXsZGsI4fIl+pMaLoOGvFYDH7XfJoMt3J2z31WJsLNFm3eU/1vp/8ri5YKUd1bANXXkoPyHkK3jbv2qKZfLWuueIoI41AXz77s/NRM4fMtudHbCfTboT+hARhf6UkB93nkKuRQC1LYRSJ+/zTibpBEVPOB0u/WBwY++2iBMtFcdQHgaBipehaBWCX2lwekf/9rduOBfQp0Wm+wWBHbRg8Pl5QvnlgfN8peuI9XwxFBFDXVtEad06GRBOnPEVTJjLWXyhyafVAme1EDWafxAYsA8ub/XvQfFGtDSizj6geatXla7kum+ppLKaGhOLGBC3jNk0bmNGdxRRXwmLkbB8EMg/CdF+SAS5eM1Ukdl4N0jQsO37TJW2Yb94sKpM+SHtcAsh1uKIx9QMjf2sQs/RMPQAtBvv0vV//Ou2OltgI7pe0iC/xZVZEJbwfR4yqxJ//bcLKN39eZROjplvl3xLmQtMhPtZB8BYUfCeNJa0wZOZjpu/KI1it0kWnz8kRuDocXb0jEMy+8GujtEbgAVKsNadiu1/Xr+RAv4f/sGPlWeD9pqIg7ErhuaYy3WXeU2+IOun6BufplnTZ0dRxKHaMWEfTgefC6ZxIVPr9Mhu8zWu+9oueiN1w+DbE81tpheLnhR1cTFZREho4Amz3BAFu8QW7S9/KhA/t8wvTAqj2KboE03n+9TcVYNbpnP78qQse3I9SZNjqeYdqpxstxZWkLN9zTnL6hky+Az0PxsXPXGezYTRcA== MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 36f1782d-85bb-44fa-2efd-08d42822a98d X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001); SRVR:DM2PR0301MB0752; X-Microsoft-Exchange-Diagnostics: 1; DM2PR0301MB0752; 3:dGbZ3JIUuRlKy63dFp3kdxylQJ3kIgds3gXJ6TQHjhuFxdnVpYyUOxslfbjmoDgh08VwpJ043Qk+BQDwGVs9bn8ijufZh/m74pZuiIAlMf/nHSNDJLcGzb8ANZDo6ZiTVbdbARRIxf5e1mqGmp+UCp9xns8zo1WpSyzm0inUlEzKuCAbw4lbLUgb5vm7TZ5/z3vYJQCGVhxJ5vPxA3ZmJRctkOniyI10C2ZsigAhRe1e8FvxQBS3SOUlhKzt+EQhiguEOmZaXZybgIdbVokbTvL6k1sV8Durb0h8H6r9DnNAAgyGKQfQQttSlrT586sZIwdE7cmVt2k5EKnVgYlsS1/x7OK2CZEzcIuR56UckBAnwH8qPoDWHbSZ5SaD24dP; 25:26h9LxWC1ob1wvtiw8yfJCIm32SdrQtICRnr5taDlUVrdeDBuBVa8Jrd2p3jYcssZNwsBNczEioVZcq87jOBwHBZxQlt03Nh/vYEs1mspEuESJBipCGt0F9vCjZTeMjXCXUaXnv2RUdIMXsuQk1mgnIuHqSUmb3sPIzDDfdTvVpIt/PJ2yUscRZaWp4ztKvKg8MVFIbUehYdHL6Ug7GYdyJtuBg4O0F4yVGfidMSce27NA/xX07bjVvH6b3MjPCRR7ClkAURPXZvxGWyjWRt81IKg8qRNdfz5i2peg/Y8IinyRiIEpDBtgdk6GRpwEI4dHAlpcJUoU4k7c+GTJDydu0O7O3bGDySv1mxIvbFnNB+v2OkwcP69XW/u7WdA0/jaOgmTScyvIA2OtGEI1uDco9oLKTRf+wdyYUYAyQSeFvdsBZzjr/7qfiSpFFE3HKML/0q49wnkA1jYe8Vlbu5Kw== X-Microsoft-Exchange-Diagnostics: 1; DM2PR0301MB0752; 31:Gmp9ZM097O5oTuVancxY/Hf2vBj8Ntx/8OllWOXQw9d3s2QbfR66sbN1oBbEuOMZrRSP1cJYeoB9pZHxYrMFaHr6eFU0vrChckM16n/9AW3bZ7LTHzqwnpQsNT6gE9Cwq1kYljYkfwbeFE47UvTG7By2Xo17rM3USHOsrVRuZjdlO5HLq08aKHKe+De60u4qCCz0ofwxpqo2kVUhO1eQqKXjQwO5rrK7CvFKcx5Mu0kMPWID7SvjyN1rgK9simMQ6LU16ZxQS0puic3DGfJj2w== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(185117386973197)(275809806118684)(21532816269658); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6095060)(601004)(2401047)(5005006)(8121501046)(13024025)(13017025)(13015025)(13023025)(13018025)(10201501046)(3002001)(6055026)(6096035)(20161123556025)(20161123563025)(20161123565025)(20161123559025)(20161123561025); SRVR:DM2PR0301MB0752; BCL:0; PCL:0; RULEID:(400006); SRVR:DM2PR0301MB0752; X-Microsoft-Exchange-Diagnostics: 1; DM2PR0301MB0752; 4:HW9DxQ9xM5JUppMqrnZnVqJp6e66J2j3voM359cdEsKGHT/vSh7ST2GOGV2tJgAe14q378KyEuJVrXRo01EcPrFvuAUWmEomTTzriKuFgUfyRxCmIlRcob7ro7bzzPfg6LtsE0ENtg1wsuWT9+aQa9+22u1SCNWOI/8i3QuPLhHLm4ftTnMfTimhA3uyrYfnelHTtxTLVfHRCMsTBmdbUNIG+m0z7yzeIMBYLBNfCbSMeej8m5jRfR8mkTPJ6Qy97+CK5QXL9N+ENhpMufc25r7JWthEHK9CrKdWzN/QAwo6w+5q/XkSGnLK6J0tM/YKUDRev6AYc2VcGZ+4dd+d5ukMIdjJT+/2mQmkHp1W0647mt9pNDmS6hL0GttS0a9YD7yDf14xsOzarnARFfgi74KW0v/jpF1OyuOrUGiuiLfk61exZcMvdukjaWau/DMoUa+oGE23h4Q0DXoJf6joCzVrYaJH5zbZSxmSw3tBWiOVx9NBZSSg4+GqzaqsdjTadUGQbwTBagTe+PPO25vKrI6/vJQjP7QIdBS2Jp3sP2uzSIw4afgMIIaak6m3NAxEvLyz62ACm64XQnxjdDSDNk4N4QKhn2Cma5jnxk27Qj4cM+CvlV4fb067nUhQNal3PULvX9V4l591j17VLWHzhI2txD+QWQ/fOXJNIUO032LsVWPQ+63SzAVR2uvQBTPe2QCjgWCk2MqqbUOsla3ObixIGkyMyjfhODt1jMy3HwwloG7Yt1+qSqghln8RrpZYK26MoOWjxvmwwASVfBvt8tb1R5tHNgw7vRRpa3U/qHk8T0txdEri0e10Nyd/J3CJ X-Forefront-PRVS: 01613DFDC8 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM2PR0301MB0752; 23:9p1XPUIgBnrTbbrVSBcJVpxcIPikWqa3wc9kq3I?= CNZaGFnvPJ7GmoglixDOiGx0bWHlCrp5B1m63JWYMWgNqowILjkoxPx0owNQkCfL56nbshftONISOjWnj56dVlBawUEn4rGzmTXP9EfSasSMZPplwyk1iYdxGzmA39aJZA+1XrKnXuRCaLg29B7njOkxolIB8No9mFzjLaWt8P8OL+Yqs3ASo6hJR+675m3UlNPuRhM5UDv34iW88RTmMSSLh2BeZoXdxXSMTeJoR1etx5YdNOzyCHbPNYeEXj6EdqvuWIKwN61nMPtbbeL7dMCpRsZFZZcCI8iF/UWJ8P+h0J6iM5RmPn9mzPxbtJwZUD5tdKYl4+YO/B2GOwhsnH/HKfDvoAQdxR87D3oZtMaCGBqpZTCOdCcRyiriyN4WsKBJF5PcQKx33IIrAZ5CHnSRB1GXkg3MVc8G/5bfSEBiKRLlCuCyrRQ77SnGKCxG0SMeK2HSgCNnra2J5m6iasyjXSvBN8902SAazxEB0ZRoTklBUfRtRrAOz0adKMegUMjMjZS0zAZyt0Okg7okPF7+dQ2w7nl3GWSj49VSX1MN/ve4YN3lHDx3t8pvv35vHTX2QZ1m0CiFPUqyu3yFh5rDtKFijfebzDnk8imtji/xGssFaRjYlvt8rf4N9xU9M7I0K8i811aSr9yB7BEGtaLR6T1yXnKKwtoRLkY7R3BUH484rcjgWPmp+mNd/+ZABNzK/CDRB0QUhDglXk8wxUEevQNrPdrnHeLb29AN01SBVgfYnr0fE6NO/bpYsHvI+yKRbKAtUOn8MuO5gTfbfTJUOW45+FKTebHR6UPcbYKFv1m59fv9UBmum/qR91zCc3LsXlMaUsALrj9eECKa1mC8IG+stxdCwXbN9XWK3+Oaxa7Cg2AmHDBvoKDsl+YaTMupR9gl0GcjsTmr0oRRDBYiHI+x60fHj3ZQlQCfbKCoQzofMxlywRKp0QV8diszP1dKOoYHq7e1D21Hp9AB2P2Qkf3iE2dz06uA61zGCqh1viZFhOBP6dqSS4vGCe2kltiutie0isFnoWHmL0uAk344FPDmQvNah/RUrc1sWcUvU5tDLyMwYazXYDA6g4E4AH3nOJrCPXqNmJmK8HMmWlj0CpILcYGKlty6p/sCqM6Ier+Zaj6UFtQmLYqeeTmzQeNuexmb9oceGuFU2VdwMSSwJiODhLt16Uqkm0mH89+IjkVtLZY61x3UF4m8Qzj9f8pULKD+T1eYHrMC0f+sZlDYvdPD62H8z5gYvF1szT81RKAIXmht5Uh7yiFS5c4tblX4= X-Microsoft-Exchange-Diagnostics: 1; DM2PR0301MB0752; 6:LsUvNSS0hEHoFYdNscXmKSnbtmsxrg9OQAQOPLlCrvlWtA8wSHNrM6khuAkBrDq9zMSbZZXwa/keYhpLiKKMp3bYJN90GTppLk4PtYk4tQfdbJAX4dObcphJBUx0WE4gImAp/orQWjPvmNgzQi8gIogRPoLk55uBRHmyl11GwJWdk6RM99z8zcX/NXykouUYWqKdeFxRuN9KeBneGlolSsSFnyLp+UYCYN69p1LIElgWyxzFNpgrt+1m1sBj8NF39vkbrtL/djzSGXeltndrOZ45+5uaf2LRzjK331uCmll8ewAftPbnQmzuemH7RXfFL0ioEoTcgIkTsLRpNE7vCC9kxd8/NquCyK9hNVtvyG8b/PaQ58mepplmL0VONT7apgPW1Y1DTRk/S9a9J/XgDNB0zeMU0n6Lgjf9H4sHKwuR6RkleP0sGiK1mfobsJ9R; 5:HYhLmNqN5A5/SnSfvl9oC4I5ZgoLbG7++7a37uj0Xajq9FQIpbjskxLMMFu5p/gB/7d3EFkqUGo0BKOxJ7P08OPqznAs7NYup31mUPBiHAd95H6qsE27nimKx65K+QAJab50MJmIhTqk0H77cMmA/+hF3T1wIQBPLgA/udDoCynBxVkNCw0r4LhYJNnQF3gd; 24:6F1DF0j/GaxyaFb7FFg7qLoJbqXerrK8H6XnB0KAC/FT3KkKag7l1rWWrMRPaEew840AxKp8Q/T1rxZxKHyv7FBD+YvXsgfr/3ZrCRLIDY8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DM2PR0301MB0752; 7:rF5/PBuLXVguL/iDL6k+6l0ym9ScEpMCK6ng/X4JGuo3zPrO9YxGNWTh97Vik3uCTgFftfWPZF36EEM7ut5unKjtUFYN7kWmUCrq7SVidhWvkSKYB6QUgEXI80SyZ6Q7tQ6nLHvHL9OXz1SpWGBMO+eioWpt5piP5D4qWP39RMr2g1tfHk12iXmjj1AuqLM2XlIKx5KbG988gjoECO4NaWYIypJTzp2fWjZHA0agIjRMCAVokjDhAbqApmnf7bcGpMD4NaXmkm01dZvmSZAbiBLfplghrFvng6DMfDPRk8HYDgR3t6gO7Ab1IJBb5NeAwR5quuo8cpe+inrDshOrTp7U0KH9GDSwspnZLMQcmFMkpZzSUOwvixUe1FG1Bu6WZp+Ws0rdqoR4GpxCqDeTipqg2B7RKvSdBbc8eG5rNefz/a/iH2aZuoMFGsU5p9XF+eyyYul8h7VBGANVebxKtQ== X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2016 15:21:10.2343 (UTC) X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e; Ip=[192.88.158.2]; Helo=[az84smr01.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR0301MB0752 Subject: [dpdk-dev] [PATCHv2 11/34] bus/fslmc: add vfio support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support for using VFIO for dpaa2 based fsl-mc bus. There are some differences in the way vfio used for fsl-mc bus from the eal vfio. - The scanning of bus for individual objects on the basis of the DPRC container. - The use and mapping of MC portal for object access With the evolution of bus model, they canbe further aligned with eal vfio code. Signed-off-by: Hemant Agrawal --- doc/guides/nics/features/dpaa2.ini | 1 + drivers/bus/fslmc/Makefile | 2 + drivers/bus/fslmc/fslmc_bus.c | 26 +- drivers/bus/fslmc/fslmc_vfio.c | 471 +++++++++++++++++++++++++++++++++++++ drivers/bus/fslmc/fslmc_vfio.h | 74 ++++++ 5 files changed, 570 insertions(+), 4 deletions(-) create mode 100644 drivers/bus/fslmc/fslmc_vfio.c create mode 100644 drivers/bus/fslmc/fslmc_vfio.h diff --git a/doc/guides/nics/features/dpaa2.ini b/doc/guides/nics/features/dpaa2.ini index b2ad6ec..b176208 100644 --- a/doc/guides/nics/features/dpaa2.ini +++ b/doc/guides/nics/features/dpaa2.ini @@ -4,5 +4,6 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Linux VFIO = Y ARMv8 = Y Usage doc = Y diff --git a/drivers/bus/fslmc/Makefile b/drivers/bus/fslmc/Makefile index f5da4e0..54f757b 100644 --- a/drivers/bus/fslmc/Makefile +++ b/drivers/bus/fslmc/Makefile @@ -41,6 +41,7 @@ CFLAGS += "-Wno-strict-aliasing" CFLAGS += -I$(RTE_SDK)/drivers/bus/fslmc CFLAGS += -I$(RTE_SDK)/drivers/bus/fslmc/mc +CFLAGS += -I$(RTE_SDK)/lib/librte_eal/linuxapp/eal # versioning export map EXPORT_MAP := rte_pmd_fslmcbus_version.map @@ -55,6 +56,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += \ mc/dpio.c \ mc/mc_sys.c +SRCS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc_vfio.c SRCS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc_bus.c # library dependencies diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c index cbf7600..ef38b3b 100644 --- a/drivers/bus/fslmc/fslmc_bus.c +++ b/drivers/bus/fslmc/fslmc_bus.c @@ -42,21 +42,39 @@ #include #include "rte_fslmc.h" +#include "fslmc_vfio.h" #define FSLMC_BUS_LOG(level, fmt, args...) \ RTE_LOG(level, EAL, "%s(): " fmt "\n", __func__, ##args) static -int rte_fslmc_scan(struct rte_bus *bus_d __rte_unused) +int rte_fslmc_scan(struct rte_bus *bus_d) { + if (fslmc_vfio_setup_group()) { + FSLMC_BUS_LOG(ERR, "fslmc: Unable to setup VFIO"); + return -1; + } + if (fslmc_vfio_process_group(bus_d)) { + FSLMC_BUS_LOG(ERR, "fslmc: Unable to setup devices"); + return -1; + } + RTE_LOG(INFO, EAL, "fslmc: Bus scan completed"); return 0; } static -int rte_fslmc_match(struct rte_driver *drv __rte_unused, - struct rte_device *dev __rte_unused) +int rte_fslmc_match(struct rte_driver *drv, struct rte_device *dev) { - return 0; + struct rte_dpaa2_driver *dpaa2_drv; + struct rte_dpaa2_device *dpaa2_dev; + + dpaa2_drv = container_of(drv, struct rte_dpaa2_driver, driver); + dpaa2_dev = container_of(dev, struct rte_dpaa2_device, device); + + if (dpaa2_drv->drv_type == dpaa2_dev->dev_type) + return 0; + + return 1; } struct rte_bus fslmc_bus = { diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c new file mode 100644 index 0000000..eae6495 --- /dev/null +++ b/drivers/bus/fslmc/fslmc_vfio.c @@ -0,0 +1,471 @@ +/*- + * BSD LICENSE + * + * Copyright (c) 2015-2016 Freescale Semiconductor, Inc. All rights reserved. + * Copyright (c) 2016 NXP. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Freescale Semiconductor, Inc nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "rte_fslmc.h" +#include "fslmc_vfio.h" + +#define VFIO_MAX_CONTAINERS 1 + +#define FSLMC_VFIO_LOG(level, fmt, args...) \ + RTE_LOG(level, EAL, "%s(): " fmt "\n", __func__, ##args) + +/** Pathname of FSL-MC devices directory. */ +#define SYSFS_DPAA2_MC_DEVICES "/sys/bus/fsl-mc/devices" + +/* Number of VFIO containers & groups with in */ +static struct fslmc_vfio_group vfio_groups[VFIO_MAX_GRP]; +static struct fslmc_vfio_container vfio_containers[VFIO_MAX_CONTAINERS]; +static int container_device_fd; +void *(*mcp_ptr_list); +static uint32_t mcp_id; + +static int vfio_connect_container(struct fslmc_vfio_group *vfio_group) +{ + struct fslmc_vfio_container *container; + int i, fd, ret; + + /* Try connecting to vfio container if already created */ + for (i = 0; i < VFIO_MAX_CONTAINERS; i++) { + container = &vfio_containers[i]; + if (!ioctl(vfio_group->fd, VFIO_GROUP_SET_CONTAINER, + &container->fd)) { + FSLMC_VFIO_LOG(INFO, "Container pre-exists with" + " FD[0x%x] for this group", + container->fd); + vfio_group->container = container; + return 0; + } + } + + /* Opens main vfio file descriptor which represents the "container" */ + fd = vfio_get_container_fd(); + if (fd < 0) { + FSLMC_VFIO_LOG(ERR, "Failed to open VFIO container"); + return -errno; + } + + /* Check whether support for SMMU type IOMMU present or not */ + if (ioctl(fd, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU)) { + /* Connect group to container */ + ret = ioctl(vfio_group->fd, VFIO_GROUP_SET_CONTAINER, &fd); + if (ret) { + FSLMC_VFIO_LOG(ERR, "Failed to setup group container"); + close(fd); + return -errno; + } + + ret = ioctl(fd, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU); + if (ret) { + FSLMC_VFIO_LOG(ERR, "Failed to setup VFIO iommu"); + close(fd); + return -errno; + } + } else { + FSLMC_VFIO_LOG(ERR, "No supported IOMMU available"); + close(fd); + return -EINVAL; + } + + container = NULL; + for (i = 0; i < VFIO_MAX_CONTAINERS; i++) { + if (vfio_containers[i].used) + continue; + FSLMC_VFIO_LOG(DEBUG, "Unused container at index %d", i); + container = &vfio_containers[i]; + } + if (!container) { + FSLMC_VFIO_LOG(ERR, "No free container found"); + close(fd); + return -ENOMEM; + } + + container->used = 1; + container->fd = fd; + container->group_list[container->index] = vfio_group; + vfio_group->container = container; + container->index++; + return 0; +} + +int vfio_dmamap_mem_region(uint64_t vaddr, + uint64_t iova, + uint64_t size) +{ + struct fslmc_vfio_group *group; + struct vfio_iommu_type1_dma_map dma_map = { + .argsz = sizeof(dma_map), + .flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE, + }; + + dma_map.vaddr = vaddr; + dma_map.size = size; + dma_map.iova = iova; + + /* SET DMA MAP for IOMMU */ + group = &vfio_groups[0]; + if (ioctl(group->container->fd, VFIO_IOMMU_MAP_DMA, &dma_map)) { + FSLMC_VFIO_LOG(ERR, "VFIO_IOMMU_MAP_DMA (errno = %d)", errno); + return -1; + } + return 0; +} +static int64_t vfio_map_mcp_obj(struct fslmc_vfio_group *group, char *mcp_obj) +{ + int64_t v_addr = (int64_t)MAP_FAILED; + int32_t ret, mc_fd; + + struct vfio_device_info d_info = { .argsz = sizeof(d_info) }; + struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) }; + + /* getting the mcp object's fd*/ + mc_fd = ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, mcp_obj); + if (mc_fd < 0) { + FSLMC_VFIO_LOG(ERR, "error in VFIO get device %s fd from group" + " %d", mcp_obj, group->fd); + return v_addr; + } + + /* getting device info*/ + ret = ioctl(mc_fd, VFIO_DEVICE_GET_INFO, &d_info); + if (ret < 0) { + FSLMC_VFIO_LOG(ERR, "error in VFIO getting DEVICE_INFO"); + goto MC_FAILURE; + } + + /* getting device region info*/ + ret = ioctl(mc_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info); + if (ret < 0) { + FSLMC_VFIO_LOG(ERR, "error in VFIO getting REGION_INFO"); + goto MC_FAILURE; + } + + FSLMC_VFIO_LOG(DEBUG, "region offset = %llx , region size = %llx", + reg_info.offset, reg_info.size); + + v_addr = (uint64_t)mmap(NULL, reg_info.size, + PROT_WRITE | PROT_READ, MAP_SHARED, + mc_fd, reg_info.offset); + +MC_FAILURE: + close(mc_fd); + + return v_addr; +} + +/* Following function shall fetch total available list of MC devices + * from VFIO container & populate private list of devices and other + * data structures + */ +int fslmc_vfio_process_group(struct rte_bus *bus __rte_unused) +{ + struct fslmc_vfio_device *vdev; + struct vfio_device_info device_info = { .argsz = sizeof(device_info) }; + char *temp_obj, *object_type __rte_unused, *mcp_obj, *dev_name; + int32_t object_id, i, dev_fd; + DIR *d; + struct dirent *dir; + char path[PATH_MAX]; + int64_t v_addr; + int ndev_count; + struct fslmc_vfio_group *group = &vfio_groups[0]; + static int process_once; + + /* if already done once */ + if (process_once) { + FSLMC_VFIO_LOG(DEBUG, "Already scanned once - re-scan " + "not supported"); + return 0; + } + process_once = 0; + + sprintf(path, "/sys/kernel/iommu_groups/%d/devices", group->groupid); + + d = opendir(path); + if (!d) { + FSLMC_VFIO_LOG(ERR, "Unable to open directory %s", path); + return -1; + } + + /*Counting the number of devices in a group and getting the mcp ID*/ + ndev_count = 0; + mcp_obj = NULL; + while ((dir = readdir(d)) != NULL) { + if (dir->d_type == DT_LNK) { + ndev_count++; + if (!strncmp("dpmcp", dir->d_name, 5)) { + if (mcp_obj) + free(mcp_obj); + mcp_obj = malloc(sizeof(dir->d_name)); + if (!mcp_obj) { + FSLMC_VFIO_LOG(ERR, "mcp obj:Unable to" + " allocate memory"); + return -ENOMEM; + } + strcpy(mcp_obj, dir->d_name); + temp_obj = strtok(dir->d_name, "."); + temp_obj = strtok(NULL, "."); + sscanf(temp_obj, "%d", &mcp_id); + } + } + } + closedir(d); + + if (!mcp_obj) { + FSLMC_VFIO_LOG(ERR, "DPAA2 MCP Object not Found"); + return -ENODEV; + } + RTE_LOG(INFO, EAL, "fslmc: DPRC contains = %d devices\n", ndev_count); + + /* Allocate the memory depends upon number of objects in a group*/ + group->vfio_device = (struct fslmc_vfio_device *)malloc(ndev_count * + sizeof(struct fslmc_vfio_device)); + if (!(group->vfio_device)) { + FSLMC_VFIO_LOG(ERR, "vfio device: Unable to allocate memory\n"); + free(mcp_obj); + return -ENOMEM; + } + + /* Allocate memory for MC Portal list */ + mcp_ptr_list = malloc(sizeof(void *) * 1); + if (!mcp_ptr_list) { + FSLMC_VFIO_LOG(ERR, "portal list: Unable to allocate memory!"); + free(mcp_obj); + goto FAILURE; + } + + v_addr = vfio_map_mcp_obj(group, mcp_obj); + free(mcp_obj); + if (v_addr == (int64_t)MAP_FAILED) { + FSLMC_VFIO_LOG(ERR, "Error mapping region (errno = %d)", errno); + goto FAILURE; + } + + FSLMC_VFIO_LOG(DEBUG, "DPAA2 MC has VIR_ADD = %ld", v_addr); + + mcp_ptr_list[0] = (void *)v_addr; + + d = opendir(path); + if (!d) { + FSLMC_VFIO_LOG(ERR, "Unable to open %s Directory", path); + goto FAILURE; + } + + i = 0; + FSLMC_VFIO_LOG(DEBUG, "DPAA2 - Parsing devices:"); + /* Parsing each object and initiating them*/ + while ((dir = readdir(d)) != NULL) { + if (dir->d_type != DT_LNK) + continue; + if (!strncmp("dprc", dir->d_name, 4) || + !strncmp("dpmcp", dir->d_name, 5)) + continue; + dev_name = malloc(sizeof(dir->d_name)); + if (!dev_name) { + FSLMC_VFIO_LOG(ERR, "name: Unable to allocate memory"); + goto FAILURE; + } + strcpy(dev_name, dir->d_name); + object_type = strtok(dir->d_name, "."); + temp_obj = strtok(NULL, "."); + sscanf(temp_obj, "%d", &object_id); + FSLMC_VFIO_LOG(DEBUG, " - %s ", dev_name); + + /* getting the device fd*/ + dev_fd = ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, dev_name); + if (dev_fd < 0) { + FSLMC_VFIO_LOG(ERR, "VFIO_GROUP_GET_DEVICE_FD error" + " Device fd: %s, Group: %d", + dev_name, group->fd); + free(dev_name); + goto FAILURE; + } + + free(dev_name); + vdev = &group->vfio_device[group->object_index++]; + vdev->fd = dev_fd; + vdev->index = i; + i++; + /* Get Device inofrmation */ + if (ioctl(vdev->fd, VFIO_DEVICE_GET_INFO, &device_info)) { + FSLMC_VFIO_LOG(ERR, "DPAA2 VFIO_DEVICE_GET_INFO fail"); + goto FAILURE; + } + } + closedir(d); + + return 0; + +FAILURE: + free(group->vfio_device); + group->vfio_device = NULL; + return -1; +} + + +int fslmc_vfio_setup_group(void) +{ + char path[PATH_MAX]; + char iommu_group_path[PATH_MAX], *group_name; + struct fslmc_vfio_group *group = NULL; + struct stat st; + int groupid; + int ret, len, i; + char *container; + struct vfio_group_status status = { .argsz = sizeof(status) }; + + /* if already done once */ + if (container_device_fd) + return 0; + + container = getenv("DPRC"); + + if (container == NULL) { + FSLMC_VFIO_LOG(ERR, "VFIO container not set in env DPRC"); + return -1; + } + RTE_LOG(INFO, PMD, "DPAA2: Processing Container = %s\n", container); + snprintf(path, sizeof(path), "%s/%s", + SYSFS_DPAA2_MC_DEVICES, container); + + /* Check whether fsl-mc container exists or not */ + FSLMC_VFIO_LOG(DEBUG, "container device path = %s", path); + if (stat(path, &st) < 0) { + FSLMC_VFIO_LOG(ERR, "Error (%d) getting FSL-MC dev (%s)", + errno, path); + return -errno; + } + + /* DPRC container exists. Now checkout the IOMMU Group */ + strncat(path, "/iommu_group", sizeof(path) - strlen(path) - 1); + + len = readlink(path, iommu_group_path, PATH_MAX); + if (len == -1) { + FSLMC_VFIO_LOG(ERR, "Error no iommu_group for device" + " %s: len = %d, (errno = %d)", + path, len, errno); + return -errno; + } + + iommu_group_path[len] = 0; + group_name = basename(iommu_group_path); + if (sscanf(group_name, "%d", &groupid) != 1) { + FSLMC_VFIO_LOG(ERR, "error in VFIO groupname reading %s", path); + return -errno; + } + + FSLMC_VFIO_LOG(DEBUG, "VFIO iommu group id = %d", groupid); + + /* Check if group already exists */ + for (i = 0; i < VFIO_MAX_GRP; i++) { + group = &vfio_groups[i]; + if (group->groupid == groupid) { + FSLMC_VFIO_LOG(ERR, "groupid already exists %d", + groupid); + return 0; + } + } + + /* Open the VFIO file corresponding to the IOMMU group */ + snprintf(path, sizeof(path), "/dev/vfio/%d", groupid); + + group->fd = open(path, O_RDWR); + if (group->fd < 0) { + FSLMC_VFIO_LOG(ERR, " VFIO error opening %s", path); + return -1; + } + + /* Test & Verify that group is VIABLE & AVAILABLE */ + if (ioctl(group->fd, VFIO_GROUP_GET_STATUS, &status)) { + FSLMC_VFIO_LOG(ERR, " VFIO error getting group status"); + close(group->fd); + return -1; + } + if (!(status.flags & VFIO_GROUP_FLAGS_VIABLE)) { + FSLMC_VFIO_LOG(ERR, "VFIO group not viable"); + close(group->fd); + return -1; + } + /* Since Group is VIABLE, Store the groupid */ + group->groupid = groupid; + + /* check if group does not have a container yet */ + if (!(status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) { + /* Now connect this IOMMU group to given container */ + if (vfio_connect_container(group)) { + FSLMC_VFIO_LOG(ERR, "VFIO error connecting container" + " with groupid %d", groupid); + close(group->fd); + return -1; + } + } + + /* Get Device information */ + ret = ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, container); + if (ret < 0) { + FSLMC_VFIO_LOG(ERR, "VFIO error getting device %s fd from" + " group %d", container, group->groupid); + return ret; + } + container_device_fd = ret; + FSLMC_VFIO_LOG(DEBUG, "VFIO Container FD is [0x%X]", + container_device_fd); + + return 0; +} diff --git a/drivers/bus/fslmc/fslmc_vfio.h b/drivers/bus/fslmc/fslmc_vfio.h new file mode 100644 index 0000000..c5a42fe --- /dev/null +++ b/drivers/bus/fslmc/fslmc_vfio.h @@ -0,0 +1,74 @@ +/*- + * BSD LICENSE + * + * Copyright (c) 2015-2016 Freescale Semiconductor, Inc. All rights reserved. + * Copyright (c) 2016 NXP. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Freescale Semiconductor, Inc nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _FSLMC_VFIO_H_ +#define _FSLMC_VFIO_H_ + +#include "eal_vfio.h" + +#define DPAA2_VENDOR_ID 0x1957 +#define DPAA2_MC_DPNI_DEVID 7 +#define DPAA2_MC_DPSECI_DEVID 3 + +#define VFIO_MAX_GRP 1 + +typedef struct fslmc_vfio_device { + int fd; /* fslmc root container device ?? */ + int index; /*index of child object */ + struct fslmc_vfio_device *child; /* Child object */ +} fslmc_vfio_device; + +typedef struct fslmc_vfio_group { + int fd; /* /dev/vfio/"groupid" */ + int groupid; + struct fslmc_vfio_container *container; + int object_index; + struct fslmc_vfio_device *vfio_device; +} fslmc_vfio_group; + +typedef struct fslmc_vfio_container { + int fd; /* /dev/vfio/vfio */ + int used; + int index; /* index in group list */ + struct fslmc_vfio_group *group_list[VFIO_MAX_GRP]; +} fslmc_vfio_container; + +int vfio_dmamap_mem_region( + uint64_t vaddr, + uint64_t iova, + uint64_t size); + +int fslmc_vfio_setup_group(void); +int fslmc_vfio_process_group(struct rte_bus *bus); + +#endif /* _FSLMC_VFIO_H_ */