From patchwork Thu Aug 31 06:37:12 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Shukla X-Patchwork-Id: 28113 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id A2190330D; Thu, 31 Aug 2017 08:38:08 +0200 (CEST) Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0069.outbound.protection.outlook.com [104.47.33.69]) by dpdk.org (Postfix) with ESMTP id 2EAB83251 for ; Thu, 31 Aug 2017 08:38:06 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=qYfnw6t+Hbp4XRMKHIPOPWbXFJb2Z58GOMiUVmtKtHs=; b=NYMYp/Qccyugpi08eouxsqv8frqIG6xukGgzRJdUuQSyccpEVlS+6lhsPbnVYlcwN/h7ij5GSIIDIxw+Fl7bt9RV3mjo3/RCQtXkBFtYMXMXzkezDembcEUXO07T0hK07k1ZHIDeNuvaC+tmo23d43faEsu4nTxaPmA/Iv7UfaM= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Santosh.Shukla@cavium.com; Received: from localhost.localdomain (111.93.218.67) by MWHPR07MB3104.namprd07.prod.outlook.com (10.172.95.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.13.10; Thu, 31 Aug 2017 06:38:01 +0000 From: Santosh Shukla To: dev@dpdk.org, olivier.matz@6wind.com Cc: jerin.jacob@caviumnetworks.com, john.mcnamara@intel.com, thomas@monjalon.net, hemant.agrawal@nxp.com, Santosh Shukla Date: Thu, 31 Aug 2017 12:07:12 +0530 Message-Id: <20170831063719.19273-4-santosh.shukla@caviumnetworks.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170831063719.19273-1-santosh.shukla@caviumnetworks.com> References: <20170824132903.32057-1-santosh.shukla@caviumnetworks.com> <20170831063719.19273-1-santosh.shukla@caviumnetworks.com> MIME-Version: 1.0 X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: PN1PR01CA0089.INDPRD01.PROD.OUTLOOK.COM (10.174.144.157) To MWHPR07MB3104.namprd07.prod.outlook.com (10.172.95.10) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f17c33b9-c64d-4f22-6cb0-08d4f03ad55a X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:MWHPR07MB3104; X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3104; 3:G6MjdM6UCf5TOLPJ8V+0Unk13Ce0F9EXjSL3dhJuke5AQz2FS2iQZczBYacSjK6R7eeIxKVJ7yJaQM19snaurctBz6D7CoFCa3mDjTRg9VJtW1w/ikN7P+CRWTxcLQcQcDLbJdHHOynOnPNvIbenFg+m3tcpRJ3mnUtcG2TtldiLtAZS6fWvoGRf9CPcDWvKm8gG7BBBGUVPorAiRUpejMJ3BiJbP1aqL4ZgNaiO1+zoLuAok79UTbKhCwnQ3SOd; 25:zf9HAvqCzKSS6mtW5gZbuq9RpWkwMkpWUtd3BGVv6pUK6PtF+KnUS1btvuhrCRMS+ao55q5r2SgnWywa3/WE/YnQOWKKYzGeYpKPJxEQiC+ldqz356U2HnD3vHgGHuqchEKDeMuGrsm7VFpcZoFZyms5YaN6zg5zyJ9mmyxsncB4WwBbyX/q6u8XJh5wRJ5OYmgqwm0UDInE5JNQ40kFaZFevY1pYIIkVyjzA34f6zyarmZZg0qC392S68x7Y0kzlvV6+oTbkAlttA8lKlFDcHb4t2LPBAwSmB1w/IhgbzN4ME5hFA6dhioDlfv7knlBszPjxwow83yVIytMihFaVA==; 31:YZbR5sjtnRfYApO7PHHYUEP3asrEAhd0W+GZnYlieLFQPns3R0MxArvzYH85OcC53510TwrY22MMltwlTF/Lao/MdMhSEqugH264czmqQI7Pu/tlojlRAfiz5ZPq8UsO1EjBOsd2q37RSj/6g1O+N3TrE152OVIokuhWDOm2h/4YAiRe/NRQzvZ4YAfdJRwXyBR+qNT3SN03S7s9whm0ms0jii1kqvuvpiXXwdqk+f0= X-MS-TrafficTypeDiagnostic: MWHPR07MB3104: X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3104; 20:iCJQbQ++B1flAZaJGPehIZv4oFT51V5rQQEaiwW+3kd6fPeL5RanMVVO4O7UUnL/fVcQv9HBRtD8ukxqK6IQADmgrBYkzDNz22Ntm3PkgdPTeJld+lZaht9ce9VttzdMqzi5A/z2UCWxvahKHSED7lsFJSV9BfCAJIx+RGbz4HwhX6Z1DtBsCvyzflxIT2RKWCkL5eqiI4+2jUrMK3FP4JAyDmUllhycu0z7NpWgDkR/m2DRYaAaRVxV0v9CVRNU+8hWN8X7D7mpqBoLA0FdjDa7399fSqNFd263rm6ZYEsLWXOILO5hs7ZMyVzpb11ywaEtOuqraZ9tf3Wh5+X8GTZ2HJ/GRqgRpKn/fBMr+7+AFwrIAxM8RzppsZfBbYiBTjmmqDThdI6ifewMLzHiXR6twZSBOQZjGo6RtQiu5ybZWLPRQD6nwa2rdMELXCseSJ/jvi9nSuEKAnhkePdTpATdg0sh73CctMj41bDIuI/NgCbsQrPDtm1VuYHdlx3TmmQT38X7964Ap0jWhQFK/jbsR6wtW+dsM2jYPn/MvVIOJ55X6q4K2Wgg0/ZR13bWYknoqJA3twrVDtPkgEE762X6+pAREfX5AtWRCD2gRAI=; 4:JW/EjOAFxiAixGS34D/Ye6dw0UzeAd8d+E7ZFusbkE/AeJ1eAiuTghd8xts2/i/Z4h0Ox/r3cvEesvWZZgpVWClfmBi4egZjoEVf1J+BSvwtPFq4/qwxTNHlnQGXvs0RWdDSQb0bJ5kPOZ1RAgI3Wi50TsnRd+N6rlKBd+J4Y1yi/SnycgOc/j7pWgBMLHi+Sh12BliVA297dgZYJLkibT+qw0NFub5zqvx4s3d2A1Z2WQPEtpm9LcCWcwi+/VG6 X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(10201501046)(100000703101)(100105400095)(93006095)(3002001)(6041248)(20161123564025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123562025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR07MB3104; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR07MB3104; X-Forefront-PRVS: 04163EF38A X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(6069001)(199003)(189002)(66066001)(6486002)(33646002)(50226002)(47776003)(6506006)(68736007)(36756003)(50986999)(101416001)(4326008)(76176999)(189998001)(25786009)(50466002)(48376002)(7736002)(5003940100001)(6512007)(105586002)(8656003)(478600001)(6666003)(2906002)(110136004)(107886003)(305945005)(53936002)(8676002)(42186005)(5009440100003)(72206003)(42882006)(2950100002)(1076002)(97736004)(81166006)(81156014)(8936002)(3846002)(6116002)(5660300001)(106356001); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR07MB3104; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR07MB3104; 23:Gw6A86U2W+FiZ7OnyTXnidlV+xLWRRuxmqA3xUqzG?= c2m04E0trBPrEsyPjPCQZFJTcqAobvpaDknX0whmiOZ/IwgKQMBSMs+czdB0hehOFnrRfHwrmWiKJdLv//kZ2zdOP5J9eWxg7Dg3CBhSUV+Xefd8n8BsaXRhEBOUKnTOxeRK39CKixWGuRNDbgPR9zZtks+zStoLUUolI5lLVwpWnRvyOWsitNJpxEskqwC+hES0onLJIblYa+5hinAVf/gVkg93HY2IA1S4oIE16MEVxsM4WDJsK5q9PUop1/S87anFfDPApK9GBJAnbTuJQTUtrnZQH8cI+MVS/WhuHTUFVE62f5Xdf4mB3X8kv2zXEwNJkDUK4yFcnQpdRh3jZzCQKibGR1YSuvLdlyn4vnxEGqd7OljrWHnahZ14nQXAsz4D7lpa3iFQbwauWCCTJ3L2PcNBdn4Mhvmx6NBZwCNxSWGj3AvmdqNFNWsk/LpyOGUzoEDFue8qzHInWcyc+HC+h5ImozVYXajdYt8G3H9mP2sEDAMiGgcO9gYa+7rjRMa4DoxZI8hsrGxPEYCPkfnlQ0ln/n8cRlq9K1l2dstR30dK+3PcH6iJUjd1nwPcP6kIjUHTV58a40zmrh5DcgwRPipiQAOICiJczMp13fM++oCJiRHvKh6Cb1IS9KuTVC5ZOIeTi8l2ILh/U2g9JXVQrTlrLyrgCa37bVemdVtiomhUVod+MEJAMTAkEpet/6g4GPqGwV7cALkyRW2KTSAf2Nsg0f7W0K2R5fu3rmqAja8vepCVll0OGoIg4Wfkp7UYdHX2t3jwOPYTQlhT6JjGJA8i13NBoj/VsPHWNIx2AV0J2hgtEDpibKvAFyCIfEcIAyZZPtsCsOawnPePVdHmR2sydwsF5pdVGHjsptxUePMfCxfyBLq2VZz3+41D9kBk70A1kgZV2lOZ8/u7zU/KBdJ+iB00XGy0AhmaCZ8OipwVzMzfqnLcl2iACiNmSY6Lm1GDOp9mwvqxsMGla5pOOXrZSbQ97Hxgb5CtQZD8tSbtBUuU3bCinjafmfjoqlbQKKe1Hyi0pMW2tRKvb61GKuz2aCisNaqyI8aXhQxWjakbWFrXn+PKXrIo4L11bsk9y7h4CbOjIvrGrQ5Pj8s X-Microsoft-Exchange-Diagnostics: 1; MWHPR07MB3104; 6:olAjmBRW0lHmeuXcUHFze5JGbtLk6q1srYINAnhKmy7PxLx4aLw7D6aVOK3drWdtBWnJR6TsXzSoI7z7iLmxMAT5vW06sd0TTJPXeIxrEUavwUr8HaSOiOCUhkSmOvuSBM7ahRYM+u0HipYxuqh62wNAKaYKE8hW2BNft6Qqju9gA+FcwG6S1kNo81SGe5XPmQriUCUBqF6PE1HwPKEMfDZ9YWydb+MaEtvzy2CkDrXsBR35rgYHqhXruTfvS2yXSyQGT+PucUQr3ZBeCrgW6kB5/GOUdsh2oTPNrLt18hcbrpdiA66BznUJrSuFvRnzHB5tTrS7d3apPHAL0m3ILw==; 5:RwIcsJmnu+zdyLh/nYbtISfVk+WGknF7sXZV01qFqMwqEO2D2TgXHoqzNodPmTDfk7nIhrxA8Huu7Hff7BZQDMs0fbHyQSQeNYayGMgnMSvTzz27XyFXhtrOeFuo6n7WBmee2N6EYhvAhjBBNKmzOg==; 24:uzXnpy2Mv/VDjf/Be1DuLQ8xld0Q9YeDy6/brfEHWpHoOzX47MGxbaxBdDhCyOqo0jl6mmPjmcFnb5/JBjkiq5+YWMYvXIe+efuUsc3dMhg=; 7:nGGPVXuCz3u/zB5MzZZVOSyE206lgy/0WBtyUTrBahBF5Kja6UlhaZGSe0PxZdUgBBarzJipD4NhPf5vVracTTuLDlrnu2Pu6Wm2SqdxjnKCjlRCOn02cVwKxUJc8BSf44ePmaNdHeLvXbo3M7hcrG/VaKOrGqCSSy89QTKGFQn1bJZQXrIolP0AId+QD7O+7jnvRM8wOI5ACLTtCPw+i1hGq3UbCysfZyZaJNVT/MQ= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Aug 2017 06:38:01.8074 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR07MB3104 Subject: [dpdk-dev] [PATCH v2 03/10] mempool/octeontx: probe fpavf pcie devices X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" A mempool device is set of PCIe vfs. On Octeontx HW, each mempool devices are enumerated as separate SRIOV VF PCIe device. In order to expose as a mempool device: On PCIe probe, the driver stores the information associated with the PCIe device and later upon application pool request (e.g. rte_mempool_create_empty), Infrastructure creates a pool device with earlier probed PCIe VF devices. Signed-off-by: Santosh Shukla Signed-off-by: Jerin Jacob --- drivers/mempool/octeontx/octeontx_fpavf.c | 151 ++++++++++++++++++++++++++++++ drivers/mempool/octeontx/octeontx_fpavf.h | 39 ++++++++ 2 files changed, 190 insertions(+) diff --git a/drivers/mempool/octeontx/octeontx_fpavf.c b/drivers/mempool/octeontx/octeontx_fpavf.c index 9bb7759c0..0b4a9357f 100644 --- a/drivers/mempool/octeontx/octeontx_fpavf.c +++ b/drivers/mempool/octeontx/octeontx_fpavf.c @@ -29,3 +29,154 @@ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "octeontx_fpavf.h" + +struct fpavf_res { + void *pool_stack_base; + void *bar0; + uint64_t stack_ln_ptr; + uint16_t domain_id; + uint16_t vf_id; /* gpool_id */ + uint16_t sz128; /* Block size in cache lines */ + bool is_inuse; +}; + +struct octeontx_fpadev { + rte_spinlock_t lock; + uint8_t total_gpool_cnt; + struct fpavf_res pool[FPA_VF_MAX]; +}; + +static struct octeontx_fpadev fpadev; + +static void +octeontx_fpavf_setup(void) +{ + uint8_t i; + static bool init_once; + + if (!init_once) { + rte_spinlock_init(&fpadev.lock); + fpadev.total_gpool_cnt = 0; + + for (i = 0; i < FPA_VF_MAX; i++) { + + fpadev.pool[i].domain_id = ~0; + fpadev.pool[i].stack_ln_ptr = 0; + fpadev.pool[i].sz128 = 0; + fpadev.pool[i].bar0 = NULL; + fpadev.pool[i].pool_stack_base = NULL; + fpadev.pool[i].is_inuse = false; + } + init_once = 1; + } +} + +static int +octeontx_fpavf_identify(void *bar0) +{ + uint64_t val; + uint16_t domain_id; + uint16_t vf_id; + uint64_t stack_ln_ptr; + + val = fpavf_read64((void *)((uintptr_t)bar0 + + FPA_VF_VHAURA_CNT_THRESHOLD(0))); + + domain_id = (val >> 8) & 0xffff; + vf_id = (val >> 24) & 0xffff; + + stack_ln_ptr = fpavf_read64((void *)((uintptr_t)bar0 + + FPA_VF_VHPOOL_THRESHOLD(0))); + if (vf_id >= FPA_VF_MAX) { + fpavf_log_err("vf_id(%d) greater than max vf (32)\n", vf_id); + return -1; + } + + if (fpadev.pool[vf_id].is_inuse) { + fpavf_log_err("vf_id %d is_inuse\n", vf_id); + return -1; + } + + fpadev.pool[vf_id].domain_id = domain_id; + fpadev.pool[vf_id].vf_id = vf_id; + fpadev.pool[vf_id].bar0 = bar0; + fpadev.pool[vf_id].stack_ln_ptr = stack_ln_ptr; + + /* SUCCESS */ + return vf_id; +} + +/* FPAVF pcie device aka mempool probe */ +static int +fpavf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) +{ + uint8_t *idreg; + int res; + struct fpavf_res *fpa; + + RTE_SET_USED(pci_drv); + RTE_SET_USED(fpa); + + /* For secondary processes, the primary has done all the work */ + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + if (pci_dev->mem_resource[0].addr == NULL) { + fpavf_log_err("Empty bars %p ", pci_dev->mem_resource[0].addr); + return -ENODEV; + } + idreg = pci_dev->mem_resource[0].addr; + + octeontx_fpavf_setup(); + + res = octeontx_fpavf_identify(idreg); + if (res < 0) + return -1; + + fpa = &fpadev.pool[res]; + fpadev.total_gpool_cnt++; + rte_wmb(); + + fpavf_log_dbg("total_fpavfs %d bar0 %p domain %d vf %d stk_ln_ptr 0x%x", + fpadev.total_gpool_cnt, fpa->bar0, fpa->domain_id, + fpa->vf_id, (unsigned int)fpa->stack_ln_ptr); + + return 0; +} + +static const struct rte_pci_id pci_fpavf_map[] = { + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, + PCI_DEVICE_ID_OCTEONTX_FPA_VF) + }, + { + .vendor_id = 0, + }, +}; + +static struct rte_pci_driver pci_fpavf = { + .id_table = pci_fpavf_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_IOVA_AS_VA, + .probe = fpavf_probe, +}; + +RTE_PMD_REGISTER_PCI(octeontx_fpavf, pci_fpavf); diff --git a/drivers/mempool/octeontx/octeontx_fpavf.h b/drivers/mempool/octeontx/octeontx_fpavf.h index 1c703725c..c43b1a7d2 100644 --- a/drivers/mempool/octeontx/octeontx_fpavf.h +++ b/drivers/mempool/octeontx/octeontx_fpavf.h @@ -34,6 +34,7 @@ #define __OCTEONTX_FPAVF_H__ #include +#include #ifdef RTE_LIBRTE_OCTEONTX_MEMPOOL_DEBUG #define fpavf_log_info(fmt, args...) \ @@ -87,4 +88,42 @@ #define FPA_VF0_APERTURE_SHIFT 22 #define FPA_AURA_SET_SIZE 16 + +/* + * In Cavium OcteonTX SoC, all accesses to the device registers are + * implictly strongly ordered. So, The relaxed version of IO operation is + * safe to use with out any IO memory barriers. + */ +#define fpavf_read64 rte_read64_relaxed +#define fpavf_write64 rte_write64_relaxed + +/* ARM64 specific functions */ +#if defined(RTE_ARCH_ARM64) +#define fpavf_load_pair(val0, val1, addr) ({ \ + asm volatile( \ + "ldp %x[x0], %x[x1], [%x[p1]]" \ + :[x0]"=r"(val0), [x1]"=r"(val1) \ + :[p1]"r"(addr) \ + ); }) + +#define fpavf_store_pair(val0, val1, addr) ({ \ + asm volatile( \ + "stp %x[x0], %x[x1], [%x[p1]]" \ + ::[x0]"r"(val0), [x1]"r"(val1), [p1]"r"(addr) \ + ); }) +#else /* Un optimized functions for building on non arm64 arch */ + +#define fpavf_load_pair(val0, val1, addr) \ +do { \ + val0 = rte_read64(addr); \ + val1 = rte_read64(((uint8_t *)addr) + 8); \ +} while (0) + +#define fpavf_store_pair(val0, val1, addr) \ +do { \ + rte_write64(val0, addr); \ + rte_write64(val1, (((uint8_t *)addr) + 8)); \ +} while (0) +#endif + #endif /* __OCTEONTX_FPAVF_H__ */