Cover Comment List
List cover comments
GET /api/covers/40744/comments/?format=api
[ { "id": 82264, "web_url": "http://patches.dpdk.org/comment/82264/", "msgid": "<afff840b-f76f-a008-5298-51f3303763b2@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/afff840b-f76f-a008-5298-51f3303763b2@intel.com", "date": "2018-06-15T15:16:00", "subject": "Re: [dpdk-dev] [PATCH 00/22] enable hotplug on multi-process", "submitter": { "id": 4, "url": "http://patches.dpdk.org/api/people/4/?format=api", "name": "Burakov, Anatoly", "email": "anatoly.burakov@intel.com" }, "content": "Hi Qi,\n\nI haven't read the code yet, and i'll be the first to admit that i'm not \ntoo well versed on how shared/private device data works, so my apologies \nin advance if all of below comments are addressed by implementation \ndetails or are way off base!\n\n\n\nOn 07-Jun-18 1:38 PM, Qi Zhang wrote:\n> \n> 1. Attach a share device from primary\n> 2. Detach a share device from primary\n> 3. Attach a share device from secondary\n> 4. Detach a share device from secondary\n> 5. Attach a private device from secondary\n> 6. Detach a private device from secondary\n> 7. Detach a share device from secondary privately\n> 8. Attach a share device from secondary privately\n> \n\n<...>\n\n> Case 7, 8:\n> Secondary process can also temporally to detach a share device\n> \"privately\" then attach it back later, this action also not impact other\n> processes.\n>\n\nDo we really need to implement these cases? It seems to me that this \n\"reattach it later\" introduces unnecessary complexity. If secondary has \ndetached the device, i think it is safer if we cannot reattach it, \nperiod, because it was a shared device. What if we try to attach it when \na handshake has already completed and all other processes expect to \ndetach it?\n\n(in fact, do we differentiate between non-existent device and shared \ndevice that has been \"privately detached\"? I would expect that we keep \nthe device as detached as opposed to forgetting about it, so that, come \nhandshake, we can safely reply \"yeah, we can detach the device\", but \nmaybe it's OK to not treat request to detach a non-existent device as an \nerror... thoughts? am i getting something wrong?)\n\n> APIs chenages:\n> ==============\n> \n> rte_eth_dev_attach and rte_eth_dev_attach are extended to support\n> share device attach/detach in primary-secondary process model, it will\n> be called in case 1,2,3,4.\n> \n> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are\n> introduced to cover case 5,6,7,8, this API can only be invoked in\n> secondary process.\n> \n> New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let\n> application lock or unlock on specific ethdev, a locked device\n> can't be detached. This help applicaiton to prevent unexpected\n> device detaching, especially in multi-process envrionment.\n> Aslo the new API let application to register a callback function\n> which will be invoked before a device is going to be detached,\n> the return value of the function will decide if device will continue\n> be detached or not, this support application to do condition check\n> at runtime.\n\nI assume that you've added device locking to avoid having to do \nhandshake before detach, correct? Is this a shared lock of some kind, or \nis it a private lock? If it's shared lock, what happens if the process \nholding that lock crashes?\n\n> \n> PMD Impact:\n> ===========\n> \n> Currently device removing is not handled well in secondary process on most\n> pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up\n> primary process since it reset all shared data. So we introduced new API\n> rte_eth_dev_release_port_local which only reset ethdev's state to unsued but\n> not touch shared data so other process will not be impacted.\n> Since not all device driver is target to support primary-secondary\n> process model, so the patch set only fix this on all Intel devices and\n> vdev, it can be refereneced by other driver when equevalent fix is required\n\nNitpick - why the naming mismatch between *_private() and *_local()?\n\n> \n> Limitation:\n> ===========\n> \n> The solution does not cover the case that primary process exit while\n> secondary processes still be active. Though this is not a typial use\n> case, but if this happens:\n> 1. secondary process can't attach / detach any shared device since no\n> primary exist.\n> 2. secondary process still can attach / detach private device.\n> 3. secondary process still can detach a share device privately but may\n> not attach it back, that ethdev slot will become zombie slot.\n\nI think this should be explicit and by design. Shared devices can only \nbe communicated to all secondaries through a primary process. No primary \n- no shared devices. I don't think we can do anything about it unless we \nimplement some kind of peer-to-peer IPC (which isn't happening as far as \ni'm aware).\n\n\n\nThanks for your work on this patchset!", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id D98531B5B1;\n\tFri, 15 Jun 2018 17:16:06 +0200 (CEST)", "from mga17.intel.com (mga17.intel.com [192.55.52.151])\n\tby dpdk.org (Postfix) with ESMTP id 03E551B53B\n\tfor <dev@dpdk.org>; Fri, 15 Jun 2018 17:16:04 +0200 (CEST)", "from fmsmga003.fm.intel.com ([10.253.24.29])\n\tby fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t15 Jun 2018 08:16:03 -0700", "from aburakov-mobl.ger.corp.intel.com (HELO [10.237.220.66])\n\t([10.237.220.66])\n\tby FMSMGA003.fm.intel.com with ESMTP; 15 Jun 2018 08:16:01 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.51,227,1526367600\"; d=\"scan'208\";a=\"57624380\"", "To": "Qi Zhang <qi.z.zhang@intel.com>, thomas@monjalon.net", "Cc": "konstantin.ananyev@intel.com, dev@dpdk.org, bruce.richardson@intel.com, \n\tferruh.yigit@intel.com, benjamin.h.shelton@intel.com,\n\tnarender.vangati@intel.com", "References": "<20180607123849.14439-1-qi.z.zhang@intel.com>", "From": "\"Burakov, Anatoly\" <anatoly.burakov@intel.com>", "Message-ID": "<afff840b-f76f-a008-5298-51f3303763b2@intel.com>", "Date": "Fri, 15 Jun 2018 16:16:00 +0100", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101\n\tThunderbird/52.8.0", "MIME-Version": "1.0", "In-Reply-To": "<20180607123849.14439-1-qi.z.zhang@intel.com>", "Content-Type": "text/plain; charset=utf-8; format=flowed", "Content-Language": "en-US", "Content-Transfer-Encoding": "7bit", "Subject": "Re: [dpdk-dev] [PATCH 00/22] enable hotplug on multi-process", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://dpdk.org/ml/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://dpdk.org/ml/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://dpdk.org/ml/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 82360, "web_url": "http://patches.dpdk.org/comment/82360/", "msgid": "<039ED4275CED7440929022BC67E7061153239F80@SHSMSX103.ccr.corp.intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/039ED4275CED7440929022BC67E7061153239F80@SHSMSX103.ccr.corp.intel.com", "date": "2018-06-19T02:43:34", "subject": "Re: [dpdk-dev] [PATCH 00/22] enable hotplug on multi-process", "submitter": { "id": 504, "url": "http://patches.dpdk.org/api/people/504/?format=api", "name": "Qi Zhang", "email": "qi.z.zhang@intel.com" }, "content": "Hi Anatoly:\n\n\tThanks for the review, see my reply in inline.\n\n> -----Original Message-----\n> From: Burakov, Anatoly\n> Sent: Friday, June 15, 2018 11:16 PM\n> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net\n> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;\n> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh\n> <ferruh.yigit@intel.com>; Shelton, Benjamin H\n> <benjamin.h.shelton@intel.com>; Vangati, Narender\n> <narender.vangati@intel.com>\n> Subject: Re: [PATCH 00/22] enable hotplug on multi-process\n> \n> Hi Qi,\n> \n> I haven't read the code yet, and i'll be the first to admit that i'm not too well\n> versed on how shared/private device data works, so my apologies in advance\n> if all of below comments are addressed by implementation details or are way\n> off base!\n> \n> \n> \n> On 07-Jun-18 1:38 PM, Qi Zhang wrote:\n> >\n> > 1. Attach a share device from primary\n> > 2. Detach a share device from primary\n> > 3. Attach a share device from secondary 4. Detach a share device from\n> > secondary 5. Attach a private device from secondary 6. Detach a\n> > private device from secondary 7. Detach a share device from secondary\n> > privately 8. Attach a share device from secondary privately\n> >\n> \n> <...>\n> \n> > Case 7, 8:\n> > Secondary process can also temporally to detach a share device\n> > \"privately\" then attach it back later, this action also not impact other\n> > processes.\n> >\n> \n> Do we really need to implement these cases? It seems to me that this\n> \"reattach it later\" introduces unnecessary complexity. \n\nI agree it's not necessary, but this looks like a free feature based on current implementation :)\n\n\n>If secondary has\n> detached the device, I think it is safer if we cannot reattach it,\n> period, because it was a shared device. What if we try to attach it when\n> a handshake has already completed and all other processes expect to\n> detach it?\n\nin the case: attach back a shared device already be detached will fail as expected.\n\nFor PCI devices, it will fail at driver probe.\nFor vdev, it will failed at rte_eth_dev_attach_secondary.\n\n> \n> (in fact, do we differentiate between non-existent device and shared\n> device that has been \"privately detached\"? I would expect that we keep\n> the device as detached as opposed to forgetting about it, so that, come\n> handshake, we can safely reply \"yeah, we can detach the device\", but\n> maybe it's OK to not treat request to detach a non-existent device as an\n> error... thoughts? am i getting something wrong?)\n> \n> > APIs chenages:\n> > ==============\n> >\n> > rte_eth_dev_attach and rte_eth_dev_attach are extended to support\n> > share device attach/detach in primary-secondary process model, it will\n> > be called in case 1,2,3,4.\n> >\n> > New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are\n> > introduced to cover case 5,6,7,8, this API can only be invoked in\n> > secondary process.\n> >\n> > New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let\n> > application lock or unlock on specific ethdev, a locked device\n> > can't be detached. This help applicaiton to prevent unexpected\n> > device detaching, especially in multi-process envrionment.\n> > Aslo the new API let application to register a callback function\n> > which will be invoked before a device is going to be detached,\n> > the return value of the function will decide if device will continue\n> > be detached or not, this support application to do condition check\n> > at runtime.\n> \n> I assume that you've added device locking to avoid having to do\n> handshake before detach, correct? \nYes.\n>Is this a shared lock of some kind, or\n> is it a private lock? If it's shared lock, what happens if the process\n> holding that lock crashes?\n\nIt’s a kind of process's private lock, but a shared device be locked any process will prevent it be detached.\n\n> \n> >\n> > PMD Impact:\n> > ===========\n> >\n> > Currently device removing is not handled well in secondary process on\n> most\n> > pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up\n> > primary process since it reset all shared data. So we introduced new API\n> > rte_eth_dev_release_port_local which only reset ethdev's state to unsued\n> but\n> > not touch shared data so other process will not be impacted.\n> > Since not all device driver is target to support primary-secondary\n> > process model, so the patch set only fix this on all Intel devices and\n> > vdev, it can be refereneced by other driver when equevalent fix is required\n> \n> Nitpick - why the naming mismatch between *_private() and *_local()?\nAgree.\n\"private\" make the API more identical.\n\n> \n> >\n> > Limitation:\n> > ===========\n> >\n> > The solution does not cover the case that primary process exit while\n> > secondary processes still be active. Though this is not a typial use\n> > case, but if this happens:\n> > 1. secondary process can't attach / detach any shared device since no\n> > primary exist.\n> > 2. secondary process still can attach / detach private device.\n> > 3. secondary process still can detach a share device privately but may\n> > not attach it back, that ethdev slot will become zombie slot.\n> \n> I think this should be explicit and by design. Shared devices can only\n> be communicated to all secondaries through a primary process. No primary\n> - no shared devices. I don't think we can do anything about it unless we\n> implement some kind of peer-to-peer IPC (which isn't happening as far as\n> i'm aware).\n\nAgree.\n\n> \n> \n> \n> Thanks for your work on this patchset!\n\nThanks for the design review and all the helpful inputs.\n\nRegards\nQi\n\n> \n> --\n> Thanks,\n> Anatoly", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id D04F12BDF;\n\tTue, 19 Jun 2018 04:43:43 +0200 (CEST)", "from mga02.intel.com (mga02.intel.com [134.134.136.20])\n\tby dpdk.org (Postfix) with ESMTP id 8870A2BDB\n\tfor <dev@dpdk.org>; Tue, 19 Jun 2018 04:43:42 +0200 (CEST)", "from fmsmga005.fm.intel.com ([10.253.24.32])\n\tby orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t18 Jun 2018 19:43:41 -0700", "from fmsmsx108.amr.corp.intel.com ([10.18.124.206])\n\tby fmsmga005.fm.intel.com with ESMTP; 18 Jun 2018 19:43:37 -0700", "from shsmsx151.ccr.corp.intel.com (10.239.6.50) by\n\tFMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP\n\tServer (TLS) id 14.3.319.2; Mon, 18 Jun 2018 19:43:36 -0700", "from shsmsx103.ccr.corp.intel.com ([169.254.4.51]) by\n\tSHSMSX151.ccr.corp.intel.com ([169.254.3.116]) with mapi id\n\t14.03.0319.002; Tue, 19 Jun 2018 10:43:34 +0800" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.51,241,1526367600\"; d=\"scan'208\";a=\"238283826\"", "From": "\"Zhang, Qi Z\" <qi.z.zhang@intel.com>", "To": "\"Burakov, Anatoly\" <anatoly.burakov@intel.com>, \"thomas@monjalon.net\"\n\t<thomas@monjalon.net>", "CC": "\"Ananyev, Konstantin\" <konstantin.ananyev@intel.com>, \"dev@dpdk.org\"\n\t<dev@dpdk.org>, \"Richardson, Bruce\" <bruce.richardson@intel.com>,\n\t\"Yigit, Ferruh\" <ferruh.yigit@intel.com>, \"Shelton, Benjamin H\"\n\t<benjamin.h.shelton@intel.com>, \"Vangati, Narender\"\n\t<narender.vangati@intel.com>", "Thread-Topic": "[PATCH 00/22] enable hotplug on multi-process", "Thread-Index": "AQHUBLvL/y4TFuLVsUaBNF3VaDnsQ6Rm1Psw", "Date": "Tue, 19 Jun 2018 02:43:34 +0000", "Message-ID": "<039ED4275CED7440929022BC67E7061153239F80@SHSMSX103.ccr.corp.intel.com>", "References": "<20180607123849.14439-1-qi.z.zhang@intel.com>\n\t<afff840b-f76f-a008-5298-51f3303763b2@intel.com>", "In-Reply-To": "<afff840b-f76f-a008-5298-51f3303763b2@intel.com>", "Accept-Language": "en-US", "Content-Language": "en-US", "X-MS-Has-Attach": "", "X-MS-TNEF-Correlator": "", "x-titus-metadata-40": "eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiOGViMDVjNjQtYzRmYS00MTk0LWEwODctOGU2MWFmOTE1MGQ3IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoibkQ4R2Zlb2VmZUYyN3JMMnhyR3FEa2ZoMml3V1lqT3lCa29JWFZEa2s3dnB6amt2VFNTQlVRRFdlM2tUeUczYyJ9", "x-ctpclassification": "CTP_NT", "dlp-product": "dlpe-windows", "dlp-version": "11.0.200.100", "dlp-reaction": "no-action", "x-originating-ip": "[10.239.127.40]", "Content-Type": "text/plain; charset=\"utf-8\"", "Content-Transfer-Encoding": "base64", "MIME-Version": "1.0", "Subject": "Re: [dpdk-dev] [PATCH 00/22] enable hotplug on multi-process", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 82821, "web_url": "http://patches.dpdk.org/comment/82821/", "msgid": "<20180628015016.42144-1-qi.z.zhang@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20180628015016.42144-1-qi.z.zhang@intel.com", "date": "2018-06-28T01:49:57", "subject": "[dpdk-dev] [PATCH v5 00/24] enable hotplug on multi-process", "submitter": { "id": 504, "url": "http://patches.dpdk.org/api/people/504/?format=api", "name": "Qi Zhang", "email": "qi.z.zhang@intel.com" }, "content": "v6:\n- remove bus->scan_one, since ABI break is not necessary.\n- remove patch for failsafe PMD since it will not support secondary.\n- fix wrong implemenation on ixgbe.\n- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for\n secondary process, so we don't need to patch on PMD if PMD use the\n default remove function.\n- add release notes update.\n\nv5:\n- since we will keep mp thread separate from interrupt thread,\n it is not necessary to use temporary thread, we use rte_eal_alarm_set.\n- remove the change in rte_eth_dev_release_port, since there is a better\n way to prevent rte_eth_dev_release_port be called after\n rte_eth_dev_release_port_private.\n- fix the issue that lock does not take effect on secondary due to\n previous re-work\n- fix the issue when the first attached device is a private device from\n secondary. (patch 8/24)\n- work around for reply a sync request in separate thread, this is still\n an open and in discussion as below.\n https://mails.dpdk.org/archives/dev/2018-June/105359.html\n\nv4:\n- since mp thread will be merged to interrupt thread, the fix on v3\n for sync IPC deadlock will not work. the new version enable the\n machanism to invoke a mp action callback in a temporary thread to\n avoid the IPC deadlock, with this, secondary to primary request\n impelemtation also be simplified, since we can use sync request\n directly in a separate thread.\n\nv3:\n- enable mp init callback register to help non-eal module to initialize\n mp channel during rte_eal_init\n- fix when attach share device from secondary.\n 1) dead lock due to sync IPC be invoked in rte_malloc in primary\n process when handle secondary request to attach device, the\n solution is primary process to issue share device attach/detach\n in interrupt thread.\n 2) return port_id not correct.\n- check nb_sent and nb_received in sync IPC.\n- fix memory leak duirng error handling at attach_on_secondary.\n- improve clean_lock_callback to only lock/unlock spinlock once\n- improve error code return in check-reply during async IPC.\n- remove rte_ prefix of internal function in ethdev_mp.c\n- sample code improvement.\n 1) rename sample to \"hotplug_mp\", and move to example/multi-process.\n 2) cleanup header include.\n 3) call rte_eal_cleanup before exit.\n\nv2:\n- rename rte_ethdev_mp.* to ethdev_mp.*\n- rename rte_ethdev_lock.* to ethdev_lock.*\n- move internal funciton to ethdev_private.h\n- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and\n rte_eth_dev_[un]lock_with_callback\n- lock callbacks will be removed automatically after device is detached.\n- add experimental tag for all new APIs.\n- fix coding style issue.\n- fix wrong lisence header in sample code.\n- fix spelling \n- fix meson.build.\n- improve comments. \n\nBackground:\n===========\n\nCurrently secondary process will only sync ethdev from primary\nprocess at init stage, but it will not be aware if device\nis attached/detached on primary process at runtime.\n\nWhile there is the requirement from application that take\nprimary-secondary process model. The primary process work as a\nresource management process, it will create/destroy virtual device\nat runtime, while the secondary process deal with the network stuff\nwith these devices.\n\nSolution:\n=========\n\nSo the orignial intention is to fix this gap, but beyond that\nthe patch set provide a more comprehesive solution to handle\ndifferent hotplug cases in multi-process situation, it cover below\nscenario:\n\n1. Attach a share device from primary\n2. Detach a share device from primary\n3. Attach a share device from secondary\n4. Detach a share device from secondary\n5. Attach a private device from secondary\n6. Detach a private device from secondary\n7. Detach a share device from secondary privately\n8. Attach a share device from secondary privately\n\nIn primary-secondary process model, we assume ethernet devices are\nshared by default. that means attach or detach a device on any process\nwill broadcast to all other processes through mp channel then device\ninformation will be synchronized on all processes.\n\nAny failure during attaching process will cause inconsistent status\nbetween processes, so proper rollback action should be considered.\nAlso, it is not safe to detach a share device when other process still\nuse it, so a handshake mechanism is introduced.\n\nScenario for Case 1, 2:\n\nattach device from primary\na) primary attach the new device if failed goto h).\nb) primary send attach sync request to all secondary.\nc) secondary receive request and attach device and send reply.\nd) primary check the reply if all success go to i).\ne) primary send attach rollback sync request to all secondary.\nf) secondary receive the request and detach device and send reply.\ng) primary receive the reply and detach device as rollback action.\nh) attach fail\ni) attach success\n\ndetach device from primary\na) primary perform pre-detach check, if device is locked, goto i).\nb) primary send pre-detach sync request to all secondary.\nc) secondary perform pre-detach check and send reply.\nd) primary check the reply if any fail goto i).\ne) primary send detach sync request to all secondary\nf) secondary detach the device and send reply (assume no fail)\ng) primary detach the device.\nh) detach success\ni) detach failed\n\nScenario for case 3, 4:\n\nattach device from secondary:\na) seconary send asycn request to primary and wait on a condition\n which will be released by matched response from primary.\nb) primary receive the request and attach the new device if failed\n goto i).\nc) primary forward attach request to all secondary as async request\n (because this in mp thread context, use sync request will deadlock,\n same reason for all following async request.)\nd) secondary receive request and attach device and send reply.\ne) primary check the reply if all success go to j).\nf) primary send attach rollback async request to all secondary.\ng) secondary receive the request and detach device and send reply.\nh) primary receive the reply and detach device as rollback action.\ni) send fail response to secondary, goto k).\nj) send success response to secondary.\nk) secondary process receive response and return.\n \ndetach device from secondary:\na) secondary send async request to primary and wait on a condition\n which will be released by matched response from primary.\nb) primary receive the request and perform pre-detach check, if device\n is locked, goto j).\nc) primary send pre-detach async request to all secondary.\nd) secondary perform pre-detach check and send reply.\ne) primary check the reply if any fail goto j).\nf) primary send detach async request to all secondary\ng) secondary detach the device and send reply\nh) primary detach the device.\ni) send success response to secondary, goto k).\nj) send fail response to secondary.\nk) secondary process receive response and return.\n\nCase 5, 6:\nSecondary process can attach private device which only visible to\nitself, in this case no IPC is involved, primary process is not allowed\nto have private device so far.\n\nCase 7, 8:\nSecondary process can also temporally to detach a share device\n\"privately\" then attach it back later, this action also not impact other\nprocesses.\n\nAPIs chenages:\n==============\n\nrte_eth_dev_attach and rte_eth_dev_attach are extended to support\nshare device attach/detach in primary-secondary process model, it will\nbe called in case 1,2,3,4.\n\nNew API rte_eth_dev_attach_private and rte_eth_dev_detach_private are\nintroduced to cover case 5,6,7,8, this API can only be invoked in\nsecondary process.\n\nNew API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let\napplication lock or unlock on specific ethdev, a locked device\ncan't be detached. This help applicaiton to prevent unexpected\ndevice detaching, especially in multi-process envrionment.\nAslo the new API let application to register a callback function\nwhich will be invoked before a device is going to be detached,\nthe return value of the function will decide if device will continue\nbe detached or not, this support application to do condition check\nat runtime.\n\nPMD Impact:\n===========\n\nCurrently device removing is not handled well in secondary process on\nmost pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up\nprimary process since it reset all shared data. So we introduced new API\nrte_eth_dev_release_port_local which only reset ethdev's state to unsued\nbut not touch shared data so other process will not be impacted.\nSince not all device driver is target to support primary-secondary\nprocess model, so the patch set only fix this on all Intel devices and\nvdev, it can be refereneced by other driver when equevalent fix is\nrequired\n\nLimitation:\n===========\n\n1. The solution does not cover the case that primary process exit while\n secondary processes still be active. Though this is not a typial use\n case, but if this happens:\n a. secondary process can't attach / detach any shared device since no\n primary exist.\n b. secondary process still can attach / detach private device.\n c. secondary process still can detach a share device privately but may\n not attach it back, that ethdev slot will become zombie slot.\n\n2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe\n mechanism can be improved to support attach private device on secondary\n process, but this is not the scope of this patchset.\n\nExample:\n========\n\nThe patchset also contains a example to demonstrate device hotplug\nin multi-process model, below are detail instructions.\n\n/* start sample code as primary then secondary */\n./hotplug_mp --proc-type=auto\n\nCommand Line Example:\n\n>help\n>list\n\n/* attach a af_packet vdev */\n>attach net_af_packet,iface=eth0\n\n/* detach port 0 */\n>detach 0\n\n/* attach a private af_packet vdev (secondary process only)*/\n>attachp net_af_packet,iface=eth0\n\n/* detach a private device (secondary process only) */\n>detachp 0\n\n/* lock port 0 */\n>lock 0\n\n/* unlock port 0 */\n>unlock 0\n\nQi Zhang (19):\n ethdev: add function to release port in local process\n eal: enable multi process init callback\n ethdev: enable hotplug on multi-process\n ethdev: introduce device lock\n ethdev: support attach or detach share device from secondary\n ethdev: support attach private device as first\n net/i40e: enable port detach on secondary process\n net/ixgbe: enable port detach on secondary process\n net/af_packet: enable port detach on secondary process\n net/bonding: enable port detach on secondary process\n net/kni: enable port detach on secondary process\n net/null: enable port detach on secondary process\n net/octeontx: enable port detach on secondary process\n net/pcap: enable port detach on secondary process\n net/softnic: enable port detach on secondary process\n net/tap: enable port detach on secondary process\n net/vhost: enable port detach on secondary process\n examples/multi_process: add hotplug sample\n doc: update release notes for multi process hotplug\n\n doc/guides/rel_notes/release_18_08.rst | 21 ++\n drivers/net/af_packet/rte_eth_af_packet.c | 11 +\n drivers/net/bonding/rte_eth_bond_pmd.c | 11 +\n drivers/net/i40e/i40e_ethdev.c | 2 +\n drivers/net/ixgbe/ixgbe_ethdev.c | 3 +\n drivers/net/kni/rte_eth_kni.c | 11 +\n drivers/net/null/rte_eth_null.c | 16 +-\n drivers/net/octeontx/octeontx_ethdev.c | 16 ++\n drivers/net/pcap/rte_eth_pcap.c | 15 +-\n drivers/net/softnic/rte_eth_softnic.c | 19 +-\n drivers/net/tap/rte_eth_tap.c | 17 +-\n drivers/net/vhost/rte_eth_vhost.c | 11 +\n examples/multi_process/Makefile | 1 +\n examples/multi_process/hotplug_mp/Makefile | 23 ++\n examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++\n examples/multi_process/hotplug_mp/commands.h | 10 +\n examples/multi_process/hotplug_mp/main.c | 41 +++\n lib/librte_eal/common/eal_common_proc.c | 51 +++-\n lib/librte_eal/common/eal_private.h | 5 +\n lib/librte_eal/common/include/rte_eal.h | 34 +++\n lib/librte_eal/linuxapp/eal/eal.c | 2 +\n lib/librte_ethdev/Makefile | 2 +\n lib/librte_ethdev/ethdev_lock.c | 140 +++++++++\n lib/librte_ethdev/ethdev_lock.h | 31 ++\n lib/librte_ethdev/ethdev_mp.c | 415 +++++++++++++++++++++++++++\n lib/librte_ethdev/ethdev_mp.h | 42 +++\n lib/librte_ethdev/ethdev_private.h | 42 +++\n lib/librte_ethdev/meson.build | 2 +\n lib/librte_ethdev/rte_ethdev.c | 312 ++++++++++++++++++--\n lib/librte_ethdev/rte_ethdev.h | 169 +++++++++++\n lib/librte_ethdev/rte_ethdev_core.h | 5 +\n lib/librte_ethdev/rte_ethdev_driver.h | 13 +\n lib/librte_ethdev/rte_ethdev_pci.h | 3 +\n 33 files changed, 1815 insertions(+), 37 deletions(-)\n create mode 100644 examples/multi_process/hotplug_mp/Makefile\n create mode 100644 examples/multi_process/hotplug_mp/commands.c\n create mode 100644 examples/multi_process/hotplug_mp/commands.h\n create mode 100644 examples/multi_process/hotplug_mp/main.c\n create mode 100644 lib/librte_ethdev/ethdev_lock.c\n create mode 100644 lib/librte_ethdev/ethdev_lock.h\n create mode 100644 lib/librte_ethdev/ethdev_mp.c\n create mode 100644 lib/librte_ethdev/ethdev_mp.h\n create mode 100644 lib/librte_ethdev/ethdev_private.h", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 9053F4C74;\n\tThu, 28 Jun 2018 03:50:04 +0200 (CEST)", "from mga05.intel.com (mga05.intel.com [192.55.52.43])\n\tby dpdk.org (Postfix) with ESMTP id 63E5E1F28\n\tfor <dev@dpdk.org>; Thu, 28 Jun 2018 03:50:02 +0200 (CEST)", "from orsmga003.jf.intel.com ([10.7.209.27])\n\tby fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t27 Jun 2018 18:50:01 -0700", "from dpdk51.sh.intel.com ([10.67.110.190])\n\tby orsmga003.jf.intel.com with ESMTP; 27 Jun 2018 18:49:42 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.51,281,1526367600\"; d=\"scan'208\";a=\"62771784\"", "From": "Qi Zhang <qi.z.zhang@intel.com>", "To": "thomas@monjalon.net,\n\tanatoly.burakov@intel.com", "Cc": "konstantin.ananyev@intel.com, dev@dpdk.org, bruce.richardson@intel.com, \n\tferruh.yigit@intel.com, benjamin.h.shelton@intel.com,\n\tnarender.vangati@intel.com, Qi Zhang <qi.z.zhang@intel.com>", "Date": "Thu, 28 Jun 2018 09:49:57 +0800", "Message-Id": "<20180628015016.42144-1-qi.z.zhang@intel.com>", "X-Mailer": "git-send-email 2.13.6", "In-Reply-To": "<20180607123849.14439-1-qi.z.zhang@intel.com>", "References": "<20180607123849.14439-1-qi.z.zhang@intel.com>", "Subject": "[dpdk-dev] [PATCH v5 00/24] enable hotplug on multi-process", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 83652, "web_url": "http://patches.dpdk.org/comment/83652/", "msgid": "<20180712011514.45006-1-qi.z.zhang@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20180712011514.45006-1-qi.z.zhang@intel.com", "date": "2018-07-12T01:14:55", "subject": "[dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "submitter": { "id": 504, "url": "http://patches.dpdk.org/api/people/504/?format=api", "name": "Qi Zhang", "email": "qi.z.zhang@intel.com" }, "content": "v13:\n- Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,\n so, modify the sample code to use rte_eal_hotplug_add and\n rte_eal_hotplug_remove to attach/detach device.\n\nv12:\n- fix return value in eal_dev_hotplug_request_to_primary.\n- add more error log in rte_eal_hotplug_add.\n- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove\n any failure due to IPC error will return -ENOMSG, but not -1.\n- remove unnecessary changes from previous rework.\n\nv11:\n- move out common code from pci_vfio_unmap_secondary and\n pci_vfio_unmap_primary.\n- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h\n- fix reply check in eal_dev_hotplug_request_to_primary.\n- move skeleton code for attaching device from secondary from patch 6/19\n to patch 5/19 to improve code readability.\n\nv10:\n- Since hotplug add/remove a vdev on a secondary process will sync on\n all processes now, it is not necessary to support private vdev for\n a secondary process which is identified by a not-NULL devargs in\n \"--vdev\". So re-work on all vdev driver changes to simpified device\n probe scenario on a secondary process, devargs will be ignored on\n secondary process now.\n- fix lisence header in example/multi-process/hotplug_mp/Makefile.\n\nv9:\n- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to\n eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of\n devices will be synced in multi-process.\n- Fix couple issue when a device is bound to vfio.\n 1) The device can't be detached clearly in a secondary process, which\n also cause it can't be attached again, due to the error that\n /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)\n 2) repeat detach/attach device will cause \"cannot find TAILQ entry\n for PCI device\" due to incorrect PCI address compare.\n (see patch 2/19).\n- Removed device lock.\n- Removed private device support.\n- Fix commit log grammar issue\n\nv8:\n- update rte_eal_version.map due to new API added.\n- minor reword on release note.\n- minor fix on commit log and code style.\n\nNOTE:\n Some issues which is not related with this patchset is expected when\n play with hotplug_mp sample as belows.\n\n- Attach a PCI device twice may cause device can't be detached\n below fix is required:\n https://patches.dpdk.org/patch/42030/\n\n- ixgbe device can't detached, below fix is required\n https://patches.dpdk.org/patch/42031/\n\nv7:\n- update rte_ethdev_version.map for new APIs.\n- improve code readability in __handle_secondary_request by use goto.\n- add comments to explain why need to call rte_eal_alarm_set.\n- add error log when process_mp_init_callbacks failed.\n- reword release notes base on Anatoly's suggestion.\n- add back previous \"Acked-by\" and \"Reviewed-by\" in commit log.\n\n NOTE: current patchset depends on below IPC fix, or it may not be able\n to attach a shared vdev.\n https://patches.dpdk.org/patch/41647/\n\nv6:\n- remove bus->scan_one, since ABI break is not necessary.\n- remove patch for failsafe PMD since it will not support secondary.\n- fix wrong implemenation on ixgbe.\n- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for\n secondary process, so we don't need to patch on PMD if PMD use the\n default remove function.\n- add release notes update.\n- agreed to use strdup(peer) as workaround for repling a sync request in seperate\n thread.\n\nv5:\n- since we will keep mp thread separate from interrupt thread,\n it is not necessary to use temporary thread, we use rte_eal_alarm_set.\n- remove the change in rte_eth_dev_release_port, since there is a better\n way to prevent rte_eth_dev_release_port be called after\n rte_eth_dev_release_port_private.\n- fix the issue that lock does not take effect on secondary due to\n previous re-work\n- fix the issue when the first attached device is a private device from\n secondary. (patch 8/24)\n- work around for reply a sync request in separate thread, this is still\n an open and in discussion as below.\n https://mails.dpdk.org/archives/dev/2018-June/105359.html\n\nv4:\n- since mp thread will be merged to interrupt thread, the fix on v3\n for sync IPC deadlock will not work. the new version enable the\n machanism to invoke a mp action callback in a temporary thread to\n avoid the IPC deadlock, with this, secondary to primary request\n impelemtation also be simplified, since we can use sync request\n directly in a separate thread.\n\nv3:\n- enable mp init callback register to help non-eal module to initialize\n mp channel during rte_eal_init\n- fix when attach share device from secondary.\n 1) dead lock due to sync IPC be invoked in rte_malloc in primary\n process when handle secondary request to attach device, the\n solution is primary process to issue share device attach/detach\n in interrupt thread.\n 2) return port_id not correct.\n- check nb_sent and nb_received in sync IPC.\n- fix memory leak duirng error handling at attach_on_secondary.\n- improve clean_lock_callback to only lock/unlock spinlock once\n- improve error code return in check-reply during async IPC.\n- remove rte_ prefix of internal function in ethdev_mp.c\n- sample code improvement.\n 1) rename sample to \"hotplug_mp\", and move to example/multi-process.\n 2) cleanup header include.\n 3) call rte_eal_cleanup before exit.\n\nv2:\n- rename rte_ethdev_mp.* to ethdev_mp.*\n- rename rte_ethdev_lock.* to ethdev_lock.*\n- move internal funciton to ethdev_private.h\n- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and\n rte_eth_dev_[un]lock_with_callback\n- lock callbacks will be removed automatically after device is detached.\n- add experimental tag for all new APIs.\n- fix coding style issue.\n- fix wrong lisence header in sample code.\n- fix spelling \n- fix meson.build.\n- improve comments. \n\nBackground:\n===========\n\nCurrently secondary process will only sync ethdev from primary\nprocess at init stage, but it will not be aware if device\nis attached/detached on primary process at runtime.\n\nWhile there is the requirement from application that take\nprimary-secondary process model. The primary process work as a\nresource management process, it will create/destroy virtual device\nat runtime, while the secondary process deal with the network stuff\nwith these devices.\n\nSolution:\n=========\n\nSo the orignial intention is to fix this gap, but beyond that\nthe patch set provide a more comprehesive solution to handle\ndifferent hotplug cases in multi-process situation, it cover below\nscenario:\n\n1. Attach a device from the primary\n2. Detach a device from the primary\n3. Attach a device from a secondary\n4. Detach a device from a secondary\n\nIn primary-secondary process model, we assume ethernet devices are\nshared by default. that means attach or detach a device on any process\nwill broadcast to all other processes through mp channel then device\ninformation will be synchronized on all processes.\n\nAny failure during attaching or detaching process will cause inconsistent\nstatus between processes, so proper rollback action should be considered.\n\nScenario for Case 1, 2:\n\nattach device from primary\na) primary attach the new device if failed goto h).\nb) primary send attach sync request to all secondary.\nc) secondary receive request and attach device and send reply.\nd) primary check the reply if all success go to i).\ne) primary send attach rollback sync request to all secondary.\nf) secondary receive the request and detach device and send reply.\ng) primary receive the reply and detach device as rollback action.\nh) attach fail\ni) attach success\n\ndetach device from primary\na) primary perform pre-detach check, if device is locked, goto i).\nb) primary send pre-detach sync request to all secondary.\nc) secondary perform pre-detach check and send reply.\nd) primary check the reply if any fail goto i).\ne) primary send detach sync request to all secondary\nf) secondary detach the device and send reply (assume no fail)\ng) primary detach the device.\nh) detach success\ni) detach failed\n\nScenario for case 3, 4:\n\nattach device from secondary:\na) seconary send asycn request to primary and wait on a condition\n which will be released by matched response from primary.\nb) primary receive the request and attach the new device if failed\n goto i).\nc) primary forward attach request to all secondary as async request\n (because this in mp thread context, use sync request will deadlock,\n same reason for all following async request.)\nd) secondary receive request and attach device and send reply.\ne) primary check the reply if all success go to j).\nf) primary send attach rollback async request to all secondary.\ng) secondary receive the request and detach device and send reply.\nh) primary receive the reply and detach device as rollback action.\ni) send fail response to secondary, goto k).\nj) send success response to secondary.\nk) secondary process receive response and return.\n \ndetach device from secondary:\na) secondary send async request to primary and wait on a condition\n which will be released by matched response from primary.\nb) primary receive the request and perform pre-detach check, if device\n is locked, goto j).\nc) primary send pre-detach async request to all secondary.\nd) secondary perform pre-detach check and send reply.\ne) primary check the reply if any fail goto j).\nf) primary send detach async request to all secondary\ng) secondary detach the device and send reply\nh) primary detach the device.\ni) send success response to secondary, goto k).\nj) send fail response to secondary.\nk) secondary process receive response and return.\n\nAPIs chenages:\n==============\n\nscope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.\nIn primary-secondary process model, rte_eal_hotplug_add will guarantee\nthat device be attached on all processes, while rte_eal_hotplug_remove will\nguarantee device be detached on all processes.\n\n\nPMD Impact:\n===========\n\nCurrently device removing is not handled well in secondary process on\nmost pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up\nprimary process since it reset all shared data. So we introduced new API\nrte_eth_dev_release_port_local which only reset ethdev's state to unsued\nbut not touch shared data so other process will not be impacted.\nSince not all device driver is target to support primary-secondary\nprocess model, so the patch set only fix this on all Intel devices and\nvdev, it can be refereneced by other driver when equevalent fix is\nrequired\n\nExample:\n========\n\nThe patchset also contains a example to demonstrate device hotplug\nin multi-process model, below are detail instructions.\n\n/* start sample code as primary then secondary */\n./hotplug_mp --proc-type=auto\n\nCommand Line Example:\n\n>help\n>list\n\n/* attach a pci device */\n> attach 0000:81:00.0\n\n/* detach the pci device */\n> detach 0000:81:00.0\n\n/* attach a vdev af_packet device */\n> attach net_af_packet,iface=eth0\n\n/* detach the vdev af_packet device */\n> detach net_af_packet\n\nQi Zhang (19):\n ethdev: add function to release port in local process\n bus/pci: fix PCI address compare\n bus/pci: enable vfio unmap resource for secondary\n vfio: remove uneccessary IPC for group fd clear\n eal: enable hotplug on multi-process\n eal: support attach or detach share device from secondary\n net/i40e: enable hotplug on secondary process\n net/ixgbe: enable hotplug on secondary process\n net/af_packet: enable hotplug on secondary process\n net/bonding: enable hotplug on secondary process\n net/kni: enable hotplug on secondary process\n net/null: enable hotplug on secondary process\n net/octeontx: enable hotplug on secondary process\n net/pcap: enable hotplug on secondary process\n net/softnic: enable hotplug on secondary process\n net/tap: enable hotplug on secondary process\n net/vhost: enable hotplug on secondary process\n examples/multi_process: add hotplug sample\n doc: update release notes for multi process hotplug\n\n doc/guides/rel_notes/release_18_08.rst | 11 +\n drivers/bus/pci/linux/pci_vfio.c | 129 +++++++--\n drivers/net/af_packet/rte_eth_af_packet.c | 7 +-\n drivers/net/bonding/rte_eth_bond_pmd.c | 7 +-\n drivers/net/i40e/i40e_ethdev.c | 2 +\n drivers/net/ixgbe/ixgbe_ethdev.c | 3 +\n drivers/net/kni/rte_eth_kni.c | 7 +-\n drivers/net/null/rte_eth_null.c | 12 +-\n drivers/net/octeontx/octeontx_ethdev.c | 9 +\n drivers/net/pcap/rte_eth_pcap.c | 11 +-\n drivers/net/softnic/rte_eth_softnic.c | 15 +-\n drivers/net/tap/rte_eth_tap.c | 13 +-\n drivers/net/vhost/rte_eth_vhost.c | 7 +-\n examples/multi_process/Makefile | 1 +\n examples/multi_process/hotplug_mp/Makefile | 23 ++\n examples/multi_process/hotplug_mp/commands.c | 214 +++++++++++++++\n examples/multi_process/hotplug_mp/commands.h | 10 +\n examples/multi_process/hotplug_mp/main.c | 41 +++\n lib/librte_eal/bsdapp/eal/Makefile | 1 +\n lib/librte_eal/common/eal_common_dev.c | 177 ++++++++++++-\n lib/librte_eal/common/eal_private.h | 37 +++\n lib/librte_eal/common/hotplug_mp.c | 348 +++++++++++++++++++++++++\n lib/librte_eal/common/hotplug_mp.h | 48 ++++\n lib/librte_eal/common/include/rte_dev.h | 6 +\n lib/librte_eal/common/meson.build | 1 +\n lib/librte_eal/linuxapp/eal/Makefile | 1 +\n lib/librte_eal/linuxapp/eal/eal.c | 6 +\n lib/librte_eal/linuxapp/eal/eal_vfio.c | 45 +---\n lib/librte_eal/linuxapp/eal/eal_vfio.h | 1 -\n lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c | 8 -\n lib/librte_ethdev/rte_ethdev.c | 12 +\n lib/librte_ethdev/rte_ethdev_driver.h | 16 +-\n lib/librte_ethdev/rte_ethdev_pci.h | 8 +\n 33 files changed, 1134 insertions(+), 103 deletions(-)\n create mode 100644 examples/multi_process/hotplug_mp/Makefile\n create mode 100644 examples/multi_process/hotplug_mp/commands.c\n create mode 100644 examples/multi_process/hotplug_mp/commands.h\n create mode 100644 examples/multi_process/hotplug_mp/main.c\n create mode 100644 lib/librte_eal/common/hotplug_mp.c\n create mode 100644 lib/librte_eal/common/hotplug_mp.h", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id E9CB51B56E;\n\tThu, 12 Jul 2018 03:15:01 +0200 (CEST)", "from mga03.intel.com (mga03.intel.com [134.134.136.65])\n\tby dpdk.org (Postfix) with ESMTP id 09A9E1B509\n\tfor <dev@dpdk.org>; Thu, 12 Jul 2018 03:14:56 +0200 (CEST)", "from fmsmga006.fm.intel.com ([10.253.24.20])\n\tby orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t11 Jul 2018 18:14:54 -0700", "from dpdk51.sh.intel.com ([10.67.110.190])\n\tby fmsmga006.fm.intel.com with ESMTP; 11 Jul 2018 18:14:35 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.51,339,1526367600\"; d=\"scan'208\";a=\"244990441\"", "From": "Qi Zhang <qi.z.zhang@intel.com>", "To": "thomas@monjalon.net,\n\tanatoly.burakov@intel.com", "Cc": "konstantin.ananyev@intel.com, dev@dpdk.org, bruce.richardson@intel.com, \n\tferruh.yigit@intel.com, benjamin.h.shelton@intel.com,\n\tnarender.vangati@intel.com, Qi Zhang <qi.z.zhang@intel.com>", "Date": "Thu, 12 Jul 2018 09:14:55 +0800", "Message-Id": "<20180712011514.45006-1-qi.z.zhang@intel.com>", "X-Mailer": "git-send-email 2.13.6", "In-Reply-To": "<20180607123849.14439-1-qi.z.zhang@intel.com>", "References": "<20180607123849.14439-1-qi.z.zhang@intel.com>", "Subject": "[dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 83653, "web_url": "http://patches.dpdk.org/comment/83653/", "msgid": "<20180712011810.45130-1-qi.z.zhang@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20180712011810.45130-1-qi.z.zhang@intel.com", "date": "2018-07-12T01:18:10", "subject": "[dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "submitter": { "id": 504, "url": "http://patches.dpdk.org/api/people/504/?format=api", "name": "Qi Zhang", "email": "qi.z.zhang@intel.com" }, "content": "v13:\n- Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,\n so, modify the sample code to use rte_eal_hotplug_add and\n rte_eal_hotplug_remove to attach/detach device.\n\nv12:\n- fix return value in eal_dev_hotplug_request_to_primary.\n- add more error log in rte_eal_hotplug_add.\n- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove\n any failure due to IPC error will return -ENOMSG, but not -1.\n- remove unnecessary changes from previous rework.\n\nv11:\n- move out common code from pci_vfio_unmap_secondary and\n pci_vfio_unmap_primary.\n- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h\n- fix reply check in eal_dev_hotplug_request_to_primary.\n- move skeleton code for attaching device from secondary from patch 6/19\n to patch 5/19 to improve code readability.\n\nv10:\n- Since hotplug add/remove a vdev on a secondary process will sync on\n all processes now, it is not necessary to support private vdev for\n a secondary process which is identified by a not-NULL devargs in\n \"--vdev\". So re-work on all vdev driver changes to simpified device\n probe scenario on a secondary process, devargs will be ignored on\n secondary process now.\n- fix lisence header in example/multi-process/hotplug_mp/Makefile.\n\nv9:\n- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to\n eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of\n devices will be synced in multi-process.\n- Fix couple issue when a device is bound to vfio.\n 1) The device can't be detached clearly in a secondary process, which\n also cause it can't be attached again, due to the error that\n /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)\n 2) repeat detach/attach device will cause \"cannot find TAILQ entry\n for PCI device\" due to incorrect PCI address compare.\n (see patch 2/19).\n- Removed device lock.\n- Removed private device support.\n- Fix commit log grammar issue\n\nv8:\n- update rte_eal_version.map due to new API added.\n- minor reword on release note.\n- minor fix on commit log and code style.\n\nNOTE:\n Some issues which is not related with this patchset is expected when\n play with hotplug_mp sample as belows.\n\n- Attach a PCI device twice may cause device can't be detached\n below fix is required:\n https://patches.dpdk.org/patch/42030/\n\n- ixgbe device can't detached, below fix is required\n https://patches.dpdk.org/patch/42031/\n\nv7:\n- update rte_ethdev_version.map for new APIs.\n- improve code readability in __handle_secondary_request by use goto.\n- add comments to explain why need to call rte_eal_alarm_set.\n- add error log when process_mp_init_callbacks failed.\n- reword release notes base on Anatoly's suggestion.\n- add back previous \"Acked-by\" and \"Reviewed-by\" in commit log.\n\n NOTE: current patchset depends on below IPC fix, or it may not be able\n to attach a shared vdev.\n https://patches.dpdk.org/patch/41647/\n\nv6:\n- remove bus->scan_one, since ABI break is not necessary.\n- remove patch for failsafe PMD since it will not support secondary.\n- fix wrong implemenation on ixgbe.\n- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for\n secondary process, so we don't need to patch on PMD if PMD use the\n default remove function.\n- add release notes update.\n- agreed to use strdup(peer) as workaround for repling a sync request in seperate\n thread.\n\nv5:\n- since we will keep mp thread separate from interrupt thread,\n it is not necessary to use temporary thread, we use rte_eal_alarm_set.\n- remove the change in rte_eth_dev_release_port, since there is a better\n way to prevent rte_eth_dev_release_port be called after\n rte_eth_dev_release_port_private.\n- fix the issue that lock does not take effect on secondary due to\n previous re-work\n- fix the issue when the first attached device is a private device from\n secondary. (patch 8/24)\n- work around for reply a sync request in separate thread, this is still\n an open and in discussion as below.\n https://mails.dpdk.org/archives/dev/2018-June/105359.html\n\nv4:\n- since mp thread will be merged to interrupt thread, the fix on v3\n for sync IPC deadlock will not work. the new version enable the\n machanism to invoke a mp action callback in a temporary thread to\n avoid the IPC deadlock, with this, secondary to primary request\n impelemtation also be simplified, since we can use sync request\n directly in a separate thread.\n\nv3:\n- enable mp init callback register to help non-eal module to initialize\n mp channel during rte_eal_init\n- fix when attach share device from secondary.\n 1) dead lock due to sync IPC be invoked in rte_malloc in primary\n process when handle secondary request to attach device, the\n solution is primary process to issue share device attach/detach\n in interrupt thread.\n 2) return port_id not correct.\n- check nb_sent and nb_received in sync IPC.\n- fix memory leak duirng error handling at attach_on_secondary.\n- improve clean_lock_callback to only lock/unlock spinlock once\n- improve error code return in check-reply during async IPC.\n- remove rte_ prefix of internal function in ethdev_mp.c\n- sample code improvement.\n 1) rename sample to \"hotplug_mp\", and move to example/multi-process.\n 2) cleanup header include.\n 3) call rte_eal_cleanup before exit.\n\nv2:\n- rename rte_ethdev_mp.* to ethdev_mp.*\n- rename rte_ethdev_lock.* to ethdev_lock.*\n- move internal funciton to ethdev_private.h\n- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and\n rte_eth_dev_[un]lock_with_callback\n- lock callbacks will be removed automatically after device is detached.\n- add experimental tag for all new APIs.\n- fix coding style issue.\n- fix wrong lisence header in sample code.\n- fix spelling \n- fix meson.build.\n- improve comments. \n\nBackground:\n===========\n\nCurrently secondary process will only sync ethdev from primary\nprocess at init stage, but it will not be aware if device\nis attached/detached on primary process at runtime.\n\nWhile there is the requirement from application that take\nprimary-secondary process model. The primary process work as a\nresource management process, it will create/destroy virtual device\nat runtime, while the secondary process deal with the network stuff\nwith these devices.\n\nSolution:\n=========\n\nSo the orignial intention is to fix this gap, but beyond that\nthe patch set provide a more comprehesive solution to handle\ndifferent hotplug cases in multi-process situation, it cover below\nscenario:\n\n1. Attach a device from the primary\n2. Detach a device from the primary\n3. Attach a device from a secondary\n4. Detach a device from a secondary\n\nIn primary-secondary process model, we assume ethernet devices are\nshared by default. that means attach or detach a device on any process\nwill broadcast to all other processes through mp channel then device\ninformation will be synchronized on all processes.\n\nAny failure during attaching or detaching process will cause inconsistent\nstatus between processes, so proper rollback action should be considered.\n\nScenario for Case 1, 2:\n\nattach device from primary\na) primary attach the new device if failed goto h).\nb) primary send attach sync request to all secondary.\nc) secondary receive request and attach device and send reply.\nd) primary check the reply if all success go to i).\ne) primary send attach rollback sync request to all secondary.\nf) secondary receive the request and detach device and send reply.\ng) primary receive the reply and detach device as rollback action.\nh) attach fail\ni) attach success\n\ndetach device from primary\na) primary perform pre-detach check, if device is locked, goto i).\nb) primary send pre-detach sync request to all secondary.\nc) secondary perform pre-detach check and send reply.\nd) primary check the reply if any fail goto i).\ne) primary send detach sync request to all secondary\nf) secondary detach the device and send reply (assume no fail)\ng) primary detach the device.\nh) detach success\ni) detach failed\n\nScenario for case 3, 4:\n\nattach device from secondary:\na) seconary send asycn request to primary and wait on a condition\n which will be released by matched response from primary.\nb) primary receive the request and attach the new device if failed\n goto i).\nc) primary forward attach request to all secondary as async request\n (because this in mp thread context, use sync request will deadlock,\n same reason for all following async request.)\nd) secondary receive request and attach device and send reply.\ne) primary check the reply if all success go to j).\nf) primary send attach rollback async request to all secondary.\ng) secondary receive the request and detach device and send reply.\nh) primary receive the reply and detach device as rollback action.\ni) send fail response to secondary, goto k).\nj) send success response to secondary.\nk) secondary process receive response and return.\n \ndetach device from secondary:\na) secondary send async request to primary and wait on a condition\n which will be released by matched response from primary.\nb) primary receive the request and perform pre-detach check, if device\n is locked, goto j).\nc) primary send pre-detach async request to all secondary.\nd) secondary perform pre-detach check and send reply.\ne) primary check the reply if any fail goto j).\nf) primary send detach async request to all secondary\ng) secondary detach the device and send reply\nh) primary detach the device.\ni) send success response to secondary, goto k).\nj) send fail response to secondary.\nk) secondary process receive response and return.\n\nAPIs chenages:\n==============\n\nscope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.\nIn primary-secondary process model, rte_eal_hotplug_add will guarantee\nthat device be attached on all processes, while rte_eal_hotplug_remove will\nguarantee device be detached on all processes.\n\n\nPMD Impact:\n===========\n\nCurrently device removing is not handled well in secondary process on\nmost pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up\nprimary process since it reset all shared data. So we introduced new API\nrte_eth_dev_release_port_local which only reset ethdev's state to unsued\nbut not touch shared data so other process will not be impacted.\nSince not all device driver is target to support primary-secondary\nprocess model, so the patch set only fix this on all Intel devices and\nvdev, it can be refereneced by other driver when equevalent fix is\nrequired\n\nExample:\n========\n\nThe patchset also contains a example to demonstrate device hotplug\nin multi-process model, below are detail instructions.\n\n/* start sample code as primary then secondary */\n./hotplug_mp --proc-type=auto\n\nCommand Line Example:\n\n>help\n>list\n\n/* attach a pci device */\n> attach 0000:81:00.0\n\n/* detach the pci device */\n> detach 0000:81:00.0\n\n/* attach a vdev af_packet device */\n> attach net_af_packet,iface=eth0\n\n/* detach the vdev af_packet device */\n> detach net_af_packet\n\nQi Zhang (19):\n ethdev: add function to release port in local process\n bus/pci: fix PCI address compare\n bus/pci: enable vfio unmap resource for secondary\n vfio: remove uneccessary IPC for group fd clear\n eal: enable hotplug on multi-process\n eal: support attach or detach share device from secondary\n net/i40e: enable hotplug on secondary process\n net/ixgbe: enable hotplug on secondary process\n net/af_packet: enable hotplug on secondary process\n net/bonding: enable hotplug on secondary process\n net/kni: enable hotplug on secondary process\n net/null: enable hotplug on secondary process\n net/octeontx: enable hotplug on secondary process\n net/pcap: enable hotplug on secondary process\n net/softnic: enable hotplug on secondary process\n net/tap: enable hotplug on secondary process\n net/vhost: enable hotplug on secondary process\n examples/multi_process: add hotplug sample\n doc: update release notes for multi process hotplug\n\n doc/guides/rel_notes/release_18_08.rst | 11 +\n drivers/bus/pci/linux/pci_vfio.c | 129 +++++++--\n drivers/net/af_packet/rte_eth_af_packet.c | 7 +-\n drivers/net/bonding/rte_eth_bond_pmd.c | 7 +-\n drivers/net/i40e/i40e_ethdev.c | 2 +\n drivers/net/ixgbe/ixgbe_ethdev.c | 3 +\n drivers/net/kni/rte_eth_kni.c | 7 +-\n drivers/net/null/rte_eth_null.c | 12 +-\n drivers/net/octeontx/octeontx_ethdev.c | 9 +\n drivers/net/pcap/rte_eth_pcap.c | 11 +-\n drivers/net/softnic/rte_eth_softnic.c | 15 +-\n drivers/net/tap/rte_eth_tap.c | 13 +-\n drivers/net/vhost/rte_eth_vhost.c | 7 +-\n examples/multi_process/Makefile | 1 +\n examples/multi_process/hotplug_mp/Makefile | 23 ++\n examples/multi_process/hotplug_mp/commands.c | 214 +++++++++++++++\n examples/multi_process/hotplug_mp/commands.h | 10 +\n examples/multi_process/hotplug_mp/main.c | 41 +++\n lib/librte_eal/bsdapp/eal/Makefile | 1 +\n lib/librte_eal/common/eal_common_dev.c | 177 ++++++++++++-\n lib/librte_eal/common/eal_private.h | 37 +++\n lib/librte_eal/common/hotplug_mp.c | 348 +++++++++++++++++++++++++\n lib/librte_eal/common/hotplug_mp.h | 48 ++++\n lib/librte_eal/common/include/rte_dev.h | 6 +\n lib/librte_eal/common/meson.build | 1 +\n lib/librte_eal/linuxapp/eal/Makefile | 1 +\n lib/librte_eal/linuxapp/eal/eal.c | 6 +\n lib/librte_eal/linuxapp/eal/eal_vfio.c | 45 +---\n lib/librte_eal/linuxapp/eal/eal_vfio.h | 1 -\n lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c | 8 -\n lib/librte_ethdev/rte_ethdev.c | 12 +\n lib/librte_ethdev/rte_ethdev_driver.h | 16 +-\n lib/librte_ethdev/rte_ethdev_pci.h | 8 +\n 33 files changed, 1134 insertions(+), 103 deletions(-)\n create mode 100644 examples/multi_process/hotplug_mp/Makefile\n create mode 100644 examples/multi_process/hotplug_mp/commands.c\n create mode 100644 examples/multi_process/hotplug_mp/commands.h\n create mode 100644 examples/multi_process/hotplug_mp/main.c\n create mode 100644 lib/librte_eal/common/hotplug_mp.c\n create mode 100644 lib/librte_eal/common/hotplug_mp.h", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 95AB81B5DE;\n\tThu, 12 Jul 2018 03:17:36 +0200 (CEST)", "from mga03.intel.com (mga03.intel.com [134.134.136.65])\n\tby dpdk.org (Postfix) with ESMTP id E68101B5C9\n\tfor <dev@dpdk.org>; Thu, 12 Jul 2018 03:17:34 +0200 (CEST)", "from orsmga005.jf.intel.com ([10.7.209.41])\n\tby orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t11 Jul 2018 18:17:33 -0700", "from dpdk51.sh.intel.com ([10.67.110.190])\n\tby orsmga005.jf.intel.com with ESMTP; 11 Jul 2018 18:17:31 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.51,339,1526367600\"; d=\"scan'208\";a=\"239656172\"", "From": "Qi Zhang <qi.z.zhang@intel.com>", "To": "thomas@monjalon.net,\n\tanatoly.burakov@intel.com", "Cc": "konstantin.ananyev@intel.com, dev@dpdk.org, bruce.richardson@intel.com, \n\tferruh.yigit@intel.com, benjamin.h.shelton@intel.com,\n\tnarender.vangati@intel.com, Qi Zhang <qi.z.zhang@intel.com>", "Date": "Thu, 12 Jul 2018 09:18:10 +0800", "Message-Id": "<20180712011810.45130-1-qi.z.zhang@intel.com>", "X-Mailer": "git-send-email 2.13.6", "In-Reply-To": "<20180607123849.14439-1-qi.z.zhang@intel.com>", "References": "<20180607123849.14439-1-qi.z.zhang@intel.com>", "Subject": "[dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 83679, "web_url": "http://patches.dpdk.org/comment/83679/", "msgid": "<2770006.tjtdisTQ3j@xps>", "list_archive_url": "https://inbox.dpdk.org/dev/2770006.tjtdisTQ3j@xps", "date": "2018-07-12T08:30:27", "subject": "Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "submitter": { "id": 685, "url": "http://patches.dpdk.org/api/people/685/?format=api", "name": "Thomas Monjalon", "email": "thomas@monjalon.net" }, "content": "12/07/2018 03:14, Qi Zhang:\n> v13:\n> - Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,\n> so, modify the sample code to use rte_eal_hotplug_add and\n> rte_eal_hotplug_remove to attach/detach device.\n\nYes, this is what I tried to explain you.\n\nI think it is now too late for 18.08.\nWe see that this patchset deserves more reviews.", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 1DC011B3A9;\n\tThu, 12 Jul 2018 10:30:32 +0200 (CEST)", "from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com\n\t[66.111.4.28]) by dpdk.org (Postfix) with ESMTP id ADECF4C88\n\tfor <dev@dpdk.org>; Thu, 12 Jul 2018 10:30:30 +0200 (CEST)", "from compute1.internal (compute1.nyi.internal [10.202.2.41])\n\tby mailout.nyi.internal (Postfix) with ESMTP id 348A721B35;\n\tThu, 12 Jul 2018 04:30:30 -0400 (EDT)", "from mailfrontend2 ([10.202.2.163])\n\tby compute1.internal (MEProxy); Thu, 12 Jul 2018 04:30:30 -0400", "from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184])\n\tby mail.messagingengine.com (Postfix) with ESMTPA id 7E14210289;\n\tThu, 12 Jul 2018 04:30:28 -0400 (EDT)" ], "DKIM-Signature": [ "v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h=\n\tcc:content-transfer-encoding:content-type:date:from:in-reply-to\n\t:message-id:mime-version:references:subject:to:x-me-sender\n\t:x-me-sender:x-sasl-enc; s=mesmtp; bh=5YxOyIzR9bw8duz3Dvj3koZ5w8\n\tGYKxeFvMUqCAJfnxo=; b=SZpHTQzHistj84JzS53tzpNfWaPVzKjkxqBCpqEtO4\n\tRMtN/Uvhd824UlEV7HkAxxfK86U1cJMyDrj9YrowwjV/xN3YtZsN2mqp7Su1TWck\n\tODAZe/iyTXovs6jviajTR1kBahjs41PP9WwGg72o51f7Zul3NserX3CLs9Ugd3Hj\n\tg=", "v=1; a=rsa-sha256; c=relaxed/relaxed; d=\n\tmessagingengine.com; h=cc:content-transfer-encoding:content-type\n\t:date:from:in-reply-to:message-id:mime-version:references\n\t:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=5YxOyI\n\tzR9bw8duz3Dvj3koZ5w8GYKxeFvMUqCAJfnxo=; b=gLZWeJwffhIt7uVfUte4Ds\n\tV9EDcrM28t/9OPjTujoJN6GWx8TL5XpO2rc/R3Y7ky9t3+9kk4hqZPmLsouET7t3\n\twLHEl91b/z/chHQCP7EnJELNZ7etoG7s0geM5Mf1QzwLWuHfGsPXCeiq1nuEC3+h\n\tsNv2CI/uhtuKGK09OvUV+cdn3aZ8jN6nsiO6uOaN8+7gflSubwkeiD291SGh4G+u\n\thSck1BqUiH/sQ09z/vCSIyD3HCQq/djRKm8hrii4Jl7q7ORItv5Z5ovJVOZnU5ZT\n\t4/NPEsUNrHcbPcfiYt0VOSXe8ii12ytWyBg060RA4CpRuqrdOI2eVCC3uaZmMTlg\n\t==" ], "X-ME-Proxy": "<xmx:pRFHWxKGDTHviNaxIwTuoszO9NY-Id0VRMQxWwGrotAzcJCLBJCrpg>\n\t<xmx:pRFHWynXZg5XbKQ0Ku0fb3ogho1SYlqEStkzJwOSNqWtCyQTH2WVoQ>\n\t<xmx:pRFHW-eR60E637pvWZI0v1ig_IOFwQs398hk0NSVQYT5ChixVTxNYQ>\n\t<xmx:pRFHWzUSys3fxt2cwmztCjnR1wdUwZub_xhTKlffb2hkIpKz1hb4Tg>\n\t<xmx:pRFHW14F73xFP2b_ALB871WDC2s1uPaCY56CnyVxENvpWdI71Se-Uw>\n\t<xmx:phFHWyF9_Rn1MZTdfAJMD7j8dLE5Ptf7u-Ay5OfosKgK5eyaRyhceQ>", "X-ME-Sender": "<xms:pRFHW-MscZiha5DtfyF9ZSbQRlYfW-nddShE88q_7EQptspZLTpHvA>", "From": "Thomas Monjalon <thomas@monjalon.net>", "To": "Qi Zhang <qi.z.zhang@intel.com>", "Cc": "dev@dpdk.org, anatoly.burakov@intel.com, konstantin.ananyev@intel.com,\n\tbruce.richardson@intel.com, ferruh.yigit@intel.com,\n\tbenjamin.h.shelton@intel.com, narender.vangati@intel.com", "Date": "Thu, 12 Jul 2018 10:30:27 +0200", "Message-ID": "<2770006.tjtdisTQ3j@xps>", "In-Reply-To": "<20180712011514.45006-1-qi.z.zhang@intel.com>", "References": "<20180607123849.14439-1-qi.z.zhang@intel.com>\n\t<20180712011514.45006-1-qi.z.zhang@intel.com>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "7Bit", "Content-Type": "text/plain; charset=\"us-ascii\"", "Subject": "Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 83681, "web_url": "http://patches.dpdk.org/comment/83681/", "msgid": "<039ED4275CED7440929022BC67E706115325966D@SHSMSX103.ccr.corp.intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/039ED4275CED7440929022BC67E706115325966D@SHSMSX103.ccr.corp.intel.com", "date": "2018-07-12T09:11:00", "subject": "Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "submitter": { "id": 504, "url": "http://patches.dpdk.org/api/people/504/?format=api", "name": "Qi Zhang", "email": "qi.z.zhang@intel.com" }, "content": "> -----Original Message-----\n> From: Thomas Monjalon [mailto:thomas@monjalon.net]\n> Sent: Thursday, July 12, 2018 4:30 PM\n> To: Zhang, Qi Z <qi.z.zhang@intel.com>\n> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,\n> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce\n> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,\n> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender\n> <narender.vangati@intel.com>\n> Subject: Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process\n> \n> 12/07/2018 03:14, Qi Zhang:\n> > v13:\n> > - Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,\n> > so, modify the sample code to use rte_eal_hotplug_add and\n> > rte_eal_hotplug_remove to attach/detach device.\n> \n> Yes, this is what I tried to explain you.\n> \n> I think it is now too late for 18.08.\n\nUnderstand, but probably patch 2,3,4 could be considered in 18.08, since they fix general issue but not just for hotplug mp.\nWhat do you think?\n\n\n> We see that this patchset deserves more reviews.\n> \n>", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 706161B4F5;\n\tThu, 12 Jul 2018 11:11:12 +0200 (CEST)", "from mga12.intel.com (mga12.intel.com [192.55.52.136])\n\tby dpdk.org (Postfix) with ESMTP id EF1C31B295\n\tfor <dev@dpdk.org>; Thu, 12 Jul 2018 11:11:09 +0200 (CEST)", "from fmsmga002.fm.intel.com ([10.253.24.26])\n\tby fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t12 Jul 2018 02:11:08 -0700", "from fmsmsx108.amr.corp.intel.com ([10.18.124.206])\n\tby fmsmga002.fm.intel.com with ESMTP; 12 Jul 2018 02:11:04 -0700", "from FMSMSX110.amr.corp.intel.com (10.18.116.10) by\n\tFMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP\n\tServer (TLS) id 14.3.319.2; Thu, 12 Jul 2018 02:11:03 -0700", "from shsmsx151.ccr.corp.intel.com (10.239.6.50) by\n\tfmsmsx110.amr.corp.intel.com (10.18.116.10) with Microsoft SMTP\n\tServer (TLS) id 14.3.319.2; Thu, 12 Jul 2018 02:11:03 -0700", "from shsmsx103.ccr.corp.intel.com ([169.254.4.100]) by\n\tSHSMSX151.ccr.corp.intel.com ([169.254.3.17]) with mapi id\n\t14.03.0319.002; Thu, 12 Jul 2018 17:11:00 +0800" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.51,342,1526367600\"; d=\"scan'208\";a=\"66316793\"", "From": "\"Zhang, Qi Z\" <qi.z.zhang@intel.com>", "To": "Thomas Monjalon <thomas@monjalon.net>", "CC": "\"dev@dpdk.org\" <dev@dpdk.org>, \"Burakov, Anatoly\"\n\t<anatoly.burakov@intel.com>, \"Ananyev, Konstantin\"\n\t<konstantin.ananyev@intel.com>, \"Richardson, Bruce\"\n\t<bruce.richardson@intel.com>, \"Yigit, Ferruh\" <ferruh.yigit@intel.com>,\n\t\"Shelton, Benjamin H\" <benjamin.h.shelton@intel.com>, \"Vangati, Narender\"\n\t<narender.vangati@intel.com>", "Thread-Topic": "[dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "Thread-Index": "AQHUGbqpbdYTRfAMHk69RJK70PpZq6SLSy0Q", "Date": "Thu, 12 Jul 2018 09:11:00 +0000", "Message-ID": "<039ED4275CED7440929022BC67E706115325966D@SHSMSX103.ccr.corp.intel.com>", "References": "<20180607123849.14439-1-qi.z.zhang@intel.com>\n\t<20180712011514.45006-1-qi.z.zhang@intel.com>\n\t<2770006.tjtdisTQ3j@xps>", "In-Reply-To": "<2770006.tjtdisTQ3j@xps>", "Accept-Language": "en-US", "Content-Language": "en-US", "X-MS-Has-Attach": "", "X-MS-TNEF-Correlator": "", "x-titus-metadata-40": "eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNjBkMjgzYzItZDgyZS00OGFhLTgzMjgtMWM4MmU0MzI5MDZmIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiTzZuaFg5ZXlmM1wvYVZwbmNyb1M5bG9OdDRQaG1OMWxLTnF1OVg4U0NDZHZ3VXptWlNmd3QyaUZsQTlvYzZGck4ifQ==", "x-ctpclassification": "CTP_NT", "dlp-product": "dlpe-windows", "dlp-version": "11.0.200.100", "dlp-reaction": "no-action", "x-originating-ip": "[10.239.127.40]", "Content-Type": "text/plain; charset=\"us-ascii\"", "Content-Transfer-Encoding": "quoted-printable", "MIME-Version": "1.0", "Subject": "Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 83682, "web_url": "http://patches.dpdk.org/comment/83682/", "msgid": "<6746975.Jtb5olJ73J@xps>", "list_archive_url": "https://inbox.dpdk.org/dev/6746975.Jtb5olJ73J@xps", "date": "2018-07-12T09:21:07", "subject": "Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "submitter": { "id": 685, "url": "http://patches.dpdk.org/api/people/685/?format=api", "name": "Thomas Monjalon", "email": "thomas@monjalon.net" }, "content": "12/07/2018 11:11, Zhang, Qi Z:\n> From: Thomas Monjalon [mailto:thomas@monjalon.net]\n> > 12/07/2018 03:14, Qi Zhang:\n> > > v13:\n> > > - Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,\n> > > so, modify the sample code to use rte_eal_hotplug_add and\n> > > rte_eal_hotplug_remove to attach/detach device.\n> > \n> > Yes, this is what I tried to explain you.\n> > \n> > I think it is now too late for 18.08.\n> \n> Understand, but probably patch 2,3,4 could be considered in 18.08, since they fix general issue but not just for hotplug mp.\n> What do you think?\n\nYes, you are right.\n\nPlease send a separate patchset and try to get reviews.\nGaetan, Anatoly, please review patches 2 and 3.", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 53DE21B46E;\n\tThu, 12 Jul 2018 11:21:12 +0200 (CEST)", "from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com\n\t[66.111.4.28]) by dpdk.org (Postfix) with ESMTP id EFFE81B46E\n\tfor <dev@dpdk.org>; Thu, 12 Jul 2018 11:21:10 +0200 (CEST)", "from compute1.internal (compute1.nyi.internal [10.202.2.41])\n\tby mailout.nyi.internal (Postfix) with ESMTP id 8228620F12;\n\tThu, 12 Jul 2018 05:21:10 -0400 (EDT)", "from mailfrontend2 ([10.202.2.163])\n\tby compute1.internal (MEProxy); Thu, 12 Jul 2018 05:21:10 -0400", "from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184])\n\tby mail.messagingengine.com (Postfix) with ESMTPA id 08E491028D;\n\tThu, 12 Jul 2018 05:21:08 -0400 (EDT)" ], "DKIM-Signature": [ "v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h=\n\tcc:content-transfer-encoding:content-type:date:from:in-reply-to\n\t:message-id:mime-version:references:subject:to:x-me-sender\n\t:x-me-sender:x-sasl-enc; s=mesmtp; bh=HGQsGUTqZPSR9CegV/KIm9bLOk\n\t63AAxQNNeTwJOnU90=; b=I+QDQ4kOw/fLPh1EeS96cd9cRPj/xcy979PKHts/rR\n\tc2HggTHJk/I+BdXphp/3tJciDtvadkiXjXTdtWy1kB08fXAs+H3o4fel6lQsWwwS\n\t0zPbdru8kkKxD+r+pdhQrS5ZV5vKeVBq6TF18buhmW07qZdN4JFPTH00L2zkjEpM\n\tU=", "v=1; a=rsa-sha256; c=relaxed/relaxed; d=\n\tmessagingengine.com; h=cc:content-transfer-encoding:content-type\n\t:date:from:in-reply-to:message-id:mime-version:references\n\t:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=HGQsGU\n\tTqZPSR9CegV/KIm9bLOk63AAxQNNeTwJOnU90=; b=J20MPdzR2ep+z4bmP+x6L/\n\tJ9L4dyaqKRzmSpqIb8EdO5SpQHzqYDIx42Q3Y02C1F1kRFMVySt116WqvgJ4bmja\n\t9xna9yd/B6S39c0F+ZIv22maqhjQZ3xdvtda4S2Gjbrkn1yODfW1xB/UeD/IJ7w6\n\tu2exM1OmS24IlfqGRXjCSE0IvpQexVseuc+hqU/cd5F7Jr8knnoe5UQT0un/pNo0\n\t45nQTuouzyrx3/wDNyIQZLtzMkoq5gvE0cmCMhDey/50s+3f+7tmKEAdq0pJI0kK\n\t9oxAxSOC072zNSmfEs8S1IemyU0cSTFsnPUQE1sgGL0kRp4YaEMUGD3qijBkd2CA\n\t==" ], "X-ME-Proxy": "<xmx:hh1HW33ewMrWO8kT4Oq-YPOmlrUhWijnfixZvB4Y33gvC0N1i0soQg>\n\t<xmx:hh1HW9w4BEv15MWZKQagW0344Ln1IJBwbfEk7lV2ZZxqvd1O1vy3Gg>\n\t<xmx:hh1HW9tzUDTyawFm0qbVkMec_LtNNdVHM0wPOflKcSiJNO2nzu4_5A>\n\t<xmx:hh1HW-NYDiET3MtH0FARt4UMLi2KG-a_I5X5Gy5F8b_QSoNOEH5GNQ>\n\t<xmx:hh1HW1w6TxlH9CNbEtePhs49xEU3lYnVP7bgVcrt6qXxlM1UTpZCBw>\n\t<xmx:hh1HW7_G2phUH3uhLE9hOCZe43g6RqHL7j9uyPNtOSNYxgicW0dzKg>", "X-ME-Sender": "<xms:hh1HW9PfCBP6JmYsID5acvvQ6s6GF4AjcNZhOFdZZY0R2dPwEvhWkg>", "From": "Thomas Monjalon <thomas@monjalon.net>", "To": "\"Zhang, Qi Z\" <qi.z.zhang@intel.com>", "Cc": "dev@dpdk.org, \"Burakov, Anatoly\" <anatoly.burakov@intel.com>, \"Ananyev, \n\tKonstantin\" <konstantin.ananyev@intel.com>, \"Richardson,\n\tBruce\" <bruce.richardson@intel.com>, \"Yigit,\n\tFerruh\" <ferruh.yigit@intel.com>, \n\t\"Shelton, Benjamin H\" <benjamin.h.shelton@intel.com>, \"Vangati,\n\tNarender\" <narender.vangati@intel.com>, gaetan.rivet@6wind.com", "Date": "Thu, 12 Jul 2018 11:21:07 +0200", "Message-ID": "<6746975.Jtb5olJ73J@xps>", "In-Reply-To": "<039ED4275CED7440929022BC67E706115325966D@SHSMSX103.ccr.corp.intel.com>", "References": "<20180607123849.14439-1-qi.z.zhang@intel.com>\n\t<2770006.tjtdisTQ3j@xps>\n\t<039ED4275CED7440929022BC67E706115325966D@SHSMSX103.ccr.corp.intel.com>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "7Bit", "Content-Type": "text/plain; charset=\"us-ascii\"", "Subject": "Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null } ]