Cover Detail
Show a cover letter.
GET /api/covers/52577/?format=api
{ "id": 52577, "url": "http://patches.dpdk.org/api/covers/52577/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/cover/20190410112006.21644-1-honnappa.nagarahalli@arm.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20190410112006.21644-1-honnappa.nagarahalli@arm.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20190410112006.21644-1-honnappa.nagarahalli@arm.com", "date": "2019-04-10T11:20:03", "name": "[v4,0/3] lib/rcu: add RCU library supporting QSBR mechanism", "submitter": { "id": 1045, "url": "http://patches.dpdk.org/api/people/1045/?format=api", "name": "Honnappa Nagarahalli", "email": "honnappa.nagarahalli@arm.com" }, "mbox": "http://patches.dpdk.org/project/dpdk/cover/20190410112006.21644-1-honnappa.nagarahalli@arm.com/mbox/", "series": [ { "id": 4235, "url": "http://patches.dpdk.org/api/series/4235/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=4235", "date": "2019-04-10T11:20:03", "name": "lib/rcu: add RCU library supporting QSBR mechanism", "version": 4, "mbox": "http://patches.dpdk.org/series/4235/mbox/" } ], "comments": "http://patches.dpdk.org/api/covers/52577/comments/", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 9003A5589;\n\tWed, 10 Apr 2019 13:20:23 +0200 (CEST)", "from foss.arm.com (foss.arm.com [217.140.101.70])\n\tby dpdk.org (Postfix) with ESMTP id D6A665398\n\tfor <dev@dpdk.org>; Wed, 10 Apr 2019 13:20:22 +0200 (CEST)", "from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249])\n\tby usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2831015AB;\n\tWed, 10 Apr 2019 04:20:22 -0700 (PDT)", "from qc2400f-1.austin.arm.com (qc2400f-1.austin.arm.com\n\t[10.118.13.209])\n\tby usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id\n\tAA6263F68F; Wed, 10 Apr 2019 04:20:21 -0700 (PDT)" ], "From": "Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>", "To": "konstantin.ananyev@intel.com, stephen@networkplumber.org,\n\tpaulmck@linux.ibm.com, marko.kovacevic@intel.com, dev@dpdk.org", "Cc": "honnappa.nagarahalli@arm.com, gavin.hu@arm.com, dharmik.thakkar@arm.com, \n\tmalvika.gupta@arm.com", "Date": "Wed, 10 Apr 2019 06:20:03 -0500", "Message-Id": "<20190410112006.21644-1-honnappa.nagarahalli@arm.com>", "X-Mailer": "git-send-email 2.17.1", "In-Reply-To": "<20181122033055.3431-1-honnappa.nagarahalli@arm.com>", "References": "<20181122033055.3431-1-honnappa.nagarahalli@arm.com>", "Subject": "[dpdk-dev] [PATCH v4 0/3] lib/rcu: add RCU library supporting QSBR\n\tmechanism", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "Lock-less data structures provide scalability and determinism.\nThey enable use cases where locking may not be allowed\n(for ex: real-time applications).\n\nIn the following paras, the term 'memory' refers to memory allocated\nby typical APIs like malloc or anything that is representative of\nmemory, for ex: an index of a free element array.\n\nSince these data structures are lock less, the writers and readers\nare accessing the data structures concurrently. Hence, while removing\nan element from a data structure, the writers cannot return the memory\nto the allocator, without knowing that the readers are not\nreferencing that element/memory anymore. Hence, it is required to\nseparate the operation of removing an element into 2 steps:\n\nDelete: in this step, the writer removes the reference to the element from\nthe data structure but does not return the associated memory to the\nallocator. This will ensure that new readers will not get a reference to\nthe removed element. Removing the reference is an atomic operation.\n\nFree(Reclaim): in this step, the writer returns the memory to the\nmemory allocator, only after knowing that all the readers have stopped\nreferencing the deleted element.\n\nThis library helps the writer determine when it is safe to free the\nmemory.\n\nThis library makes use of thread Quiescent State (QS). QS can be\ndefined as 'any point in the thread execution where the thread does\nnot hold a reference to shared memory'. It is upto the application to\ndetermine its quiescent state. Let us consider the following diagram:\n\n Time -------------------------------------------------->\n\n | |\n RT1 $++++****D1****+++***D2*|**+++|+++**D3*****++++$\n | |\n RT2 $++++****D1****++|+**D2|***++++++**D3*****++++$\n | |\n RT3 $++++****D1****+++***|D2***|++++++**D2*****++++$\n | |\n |<--->|\n Del | Free\n |\n Cannot free memory\n during this period\n (Grace Period)\n\nRTx - Reader thread\n< and > - Start and end of while(1) loop\n***Dx*** - Reader thread is accessing the shared data structure Dx.\n i.e. critical section.\n+++ - Reader thread is not accessing any shared data structure.\n i.e. non critical section or quiescent state.\nDel - Point in time when the reference to the entry is removed using\n atomic operation.\nFree - Point in time when the writer can free the entry.\nGrace Period - Time duration between Del and Free, during which memory cannot\n be freed.\n\nAs shown, thread RT1 accesses data structures D1, D2 and D3. When it is\naccessing D2, if the writer has to remove an element from D2, the\nwriter cannot free the memory associated with that element immediately.\nThe writer can return the memory to the allocator only after the reader\nstops referencing D2. In other words, reader thread RT1 has to enter\na quiescent state.\n\nSimilarly, since thread RT3 is also accessing D2, writer has to wait till\nRT3 enters quiescent state as well.\n\nHowever, the writer does not need to wait for RT2 to enter quiescent state.\nThread RT2 was not accessing D2 when the delete operation happened.\nSo, RT2 will not get a reference to the deleted entry.\n\nIt can be noted that, the critical sections for D2 and D3 are quiescent states\nfor D1. i.e. for a given data structure Dx, any point in the thread execution\nthat does not reference Dx is a quiescent state.\n\nSince memory is not freed immediately, there might be a need for\nprovisioning of additional memory, depending on the application requirements.\n\nIt is important to make sure that this library keeps the overhead of\nidentifying the end of grace period and subsequent freeing of memory,\nto a minimum. The following paras explain how grace period and critical\nsection affect this overhead.\n\nThe writer has to poll the readers to identify the end of grace period.\nPolling introduces memory accesses and wastes CPU cycles. The memory\nis not available for reuse during grace period. Longer grace periods\nexasperate these conditions.\n\nThe length of the critical section and the number of reader threads\nis proportional to the duration of the grace period. Keeping the critical\nsections smaller will keep the grace period smaller. However, keeping the\ncritical sections smaller requires additional CPU cycles(due to additional\nreporting) in the readers.\n\nHence, we need the characteristics of small grace period and large critical\nsection. This library addresses this by allowing the writer to do\nother work without having to block till the readers report their quiescent\nstate.\n\nFor DPDK applications, the start and end of while(1) loop (where no\nreferences to shared data structures are kept) act as perfect quiescent\nstates. This will combine all the shared data structure accesses into a\nsingle, large critical section which helps keep the overhead on the\nreader side to a minimum.\n\nDPDK supports pipeline model of packet processing and service cores.\nIn these use cases, a given data structure may not be used by all the\nworkers in the application. The writer does not have to wait for all\nthe workers to report their quiescent state. To provide the required\nflexibility, this library has a concept of QS variable. The application\ncan create one QS variable per data structure to help it track the\nend of grace period for each data structure. This helps keep the grace\nperiod to a minimum.\n\nThe application has to allocate memory and initialize a QS variable.\n\nApplication can call rte_rcu_qsbr_get_memsize to calculate the size\nof memory to allocate. This API takes maximum number of reader threads,\nusing this variable, as a parameter. Currently, a maximum of 1024 threads\nare supported.\n\nFurther, the application can initialize a QS variable using the API\nrte_rcu_qsbr_init.\n\nEach reader thread is assumed to have a unique thread ID. Currently, the\nmanagement of the thread ID (for ex: allocation/free) is left to the\napplication. The thread ID should be in the range of 0 to\nmaximum number of threads provided while creating the QS variable.\nThe application could also use lcore_id as the thread ID where applicable.\n\nrte_rcu_qsbr_thread_register API will register a reader thread\nto report its quiescent state. This can be called from a reader thread.\nA control plane thread can also call this on behalf of a reader thread.\nThe reader thread must call rte_rcu_qsbr_thread_online API to start reporting\nits quiescent state.\n\nSome of the use cases might require the reader threads to make\nblocking API calls (for ex: while using eventdev APIs). The writer thread\nshould not wait for such reader threads to enter quiescent state.\nThe reader thread must call rte_rcu_qsbr_thread_offline API, before calling\nblocking APIs. It can call rte_rcu_qsbr_thread_online API once the blocking\nAPI call returns.\n\nThe writer thread can trigger the reader threads to report their quiescent\nstate by calling the API rte_rcu_qsbr_start. It is possible for multiple\nwriter threads to query the quiescent state status simultaneously. Hence,\nrte_rcu_qsbr_start returns a token to each caller.\n\nThe writer thread has to call rte_rcu_qsbr_check API with the token to get the\ncurrent quiescent state status. Option to block till all the reader threads\nenter the quiescent state is provided. If this API indicates that all the\nreader threads have entered the quiescent state, the application can free the\ndeleted entry.\n\nThe APIs rte_rcu_qsbr_start and rte_rcu_qsbr_check are lock free. Hence, they\ncan be called concurrently from multiple writers even while running\nas worker threads.\n\nThe separation of triggering the reporting from querying the status provides\nthe writer threads flexibility to do useful work instead of blocking for the\nreader threads to enter the quiescent state or go offline. This reduces the\nmemory accesses due to continuous polling for the status.\n\nrte_rcu_qsbr_synchronize API combines the functionality of rte_rcu_qsbr_start\nand blocking rte_rcu_qsbr_check into a single API. This API triggers the reader\nthreads to report their quiescent state and polls till all the readers enter\nthe quiescent state or go offline. This API does not allow the writer to\ndo useful work while waiting and also introduces additional memory accesses\ndue to continuous polling.\n\nThe reader thread must call rte_rcu_qsbr_thread_offline and\nrte_rcu_qsbr_thread_unregister APIs to remove itself from reporting its\nquiescent state. The rte_rcu_qsbr_check API will not wait for this reader\nthread to report the quiescent state status anymore.\n\nThe reader threads should call rte_rcu_qsbr_update API to indicate that they\nentered a quiescent state. This API checks if a writer has triggered a\nquiescent state query and update the state accordingly.\n\nPatch v4:\n1) Library changes\n a) Fixed the compilation issue on x86 (Konstantin)\n b) Rebased with latest master\n\nPatch v3:\n1) Library changes\n a) Moved the registered thread ID array to the end of the\n structure (Konstantin)\n b) Removed the compile time constant RTE_RCU_MAX_THREADS\n c) Added code to keep track of registered number of threads\n\nPatch v2:\n1) Library changes\n a) Corrected the RTE_ASSERT checks (Konstantin)\n b) Replaced RTE_ASSERT with 'if' checks for non-datapath APIs (Konstantin)\n c) Made rte_rcu_qsbr_thread_register/unregister non-datapath critical APIs\n d) Renamed rte_rcu_qsbr_update to rte_rcu_qsbr_quiescent (Ola)\n e) Used rte_smp_mb() in rte_rcu_qsbr_thread_online API for x86 (Konstantin)\n f) Removed the macro to access the thread QS counters (Konstantin)\n2) Test cases\n a) Added additional test cases for removing RTE_ASSERT\n3) Documentation\n a) Changed the figure to make it bigger (Marko)\n b) Spelling and format corrections (Marko)\n\nPatch v1:\n1) Library changes\n a) Changed the maximum number of reader threads to 1024\n b) Renamed rte_rcu_qsbr_register/unregister_thread to\n rte_rcu_qsbr_thread_register/unregister\n c) Added rte_rcu_qsbr_thread_online/offline API. These are optimized\n version of rte_rcu_qsbr_thread_register/unregister API. These\n also provide the flexibility for performance when the requested\n maximum number of threads is higher than the current number of\n threads.\n d) Corrected memory orderings in rte_rcu_qsbr_update\n e) Changed the signature of rte_rcu_qsbr_start API to return the token\n f) Changed the signature of rte_rcu_qsbr_start API to not take the\n expected number of QS states to wait.\n g) Added debug logs\n h) Added API and programmer guide documentation.\n\nRFC v3:\n1) Library changes\n a) Rebased to latest master\n b) Added new API rte_rcu_qsbr_get_memsize\n c) Add support for memory allocation for QSBR variable (Konstantin)\n d) Fixed a bug in rte_rcu_qsbr_check (Konstantin)\n2) Testcase changes\n a) Separated stress tests into a performance test case file\n b) Added performance statistics\n\nRFC v2:\n1) Cover letter changes\n a) Explian the parameters that affect the overhead of using RCU\n and their effect\n b) Explain how this library addresses these effects to keep\n the overhead to minimum\n2) Library changes\n a) Rename the library to avoid confusion (Matias, Bruce, Konstantin)\n b) Simplify the code/remove APIs to keep this library inline with\n other synchronisation mechanisms like locks (Konstantin)\n c) Change the design to support more than 64 threads (Konstantin)\n d) Fixed version map to remove static inline functions\n3) Testcase changes\n a) Add boundary and additional functional test cases\n b) Add stress test cases (Paul E. McKenney)\n\nDharmik Thakkar (1):\n test/rcu_qsbr: add API and functional tests\n\nHonnappa Nagarahalli (2):\n rcu: add RCU library supporting QSBR mechanism\n doc/rcu: add lib_rcu documentation\n\n MAINTAINERS | 5 +\n app/test/Makefile | 2 +\n app/test/autotest_data.py | 12 +\n app/test/meson.build | 5 +\n app/test/test_rcu_qsbr.c | 1004 +++++++++++++++++\n app/test/test_rcu_qsbr_perf.c | 615 ++++++++++\n config/common_base | 6 +\n doc/api/doxy-api-index.md | 3 +-\n doc/api/doxy-api.conf.in | 1 +\n .../prog_guide/img/rcu_general_info.svg | 509 +++++++++\n doc/guides/prog_guide/index.rst | 1 +\n doc/guides/prog_guide/rcu_lib.rst | 179 +++\n lib/Makefile | 2 +\n lib/librte_rcu/Makefile | 23 +\n lib/librte_rcu/meson.build | 5 +\n lib/librte_rcu/rte_rcu_qsbr.c | 237 ++++\n lib/librte_rcu/rte_rcu_qsbr.h | 554 +++++++++\n lib/librte_rcu/rte_rcu_version.map | 11 +\n lib/meson.build | 2 +-\n mk/rte.app.mk | 1 +\n 20 files changed, 3175 insertions(+), 2 deletions(-)\n create mode 100644 app/test/test_rcu_qsbr.c\n create mode 100644 app/test/test_rcu_qsbr_perf.c\n create mode 100644 doc/guides/prog_guide/img/rcu_general_info.svg\n create mode 100644 doc/guides/prog_guide/rcu_lib.rst\n create mode 100644 lib/librte_rcu/Makefile\n create mode 100644 lib/librte_rcu/meson.build\n create mode 100644 lib/librte_rcu/rte_rcu_qsbr.c\n create mode 100644 lib/librte_rcu/rte_rcu_qsbr.h\n create mode 100644 lib/librte_rcu/rte_rcu_version.map" }