From patchwork Tue Aug 1 16:54:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 27354 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 2BCD6A2BD; Tue, 1 Aug 2017 18:56:15 +0200 (CEST) Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) by dpdk.org (Postfix) with ESMTP id 2BB99A1B2 for ; Tue, 1 Aug 2017 18:55:47 +0200 (CEST) Received: by mail-wm0-f54.google.com with SMTP id m85so20353360wma.0 for ; Tue, 01 Aug 2017 09:55:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=1BUV1hDQeHMlgbHdzXNyiSIVHGzJzgQfxWqtpbUCJbU=; b=lMYY456QQqGfhcQjlfdwKK9TH1NE6D6MCvQCQKHGB8VCBt1z0YLiRA2oyqOaEzOmXu Djc5B1tD/CpMwiAirMVZ/8QCzzjlxPjPVQZP/rwwgkOjSlSWuzqOOlz4sBc4ah4Htj74 L19Yt7BUtlirMYqUs9jPA0cNB2l5Y8ZQKbwOlrA6UGR9SWhmQ06FD8HMWXgie1SnOkpa Sj4BRrFvMTq8ip0wP98zoUzI994tA+DZu80Ldg8t4MrK9awef4ziJ6vrIa/PfCiX/oLr sHHJUJpUK8PxFQRRXrekRjlzvT+ugzIGN+jpMY2E7dPyLWOot+FJmDeZkq6jJfouYcxZ mlCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=1BUV1hDQeHMlgbHdzXNyiSIVHGzJzgQfxWqtpbUCJbU=; b=p9z4jK4R6jzZmYsSndHM6IDbY2Q4m8drY7zzzeQCnR0HncJL0/fdDZlp6UzDIkHJYi V8M8zJWru2Npn/8u4xaftKsLa8SAR9fO2eRyX/a/oLLVTxPjO6VlA7MvwqK2bhVLPAlg AFdBpA2j0JkeEhClweyAi6vI122lCZuDLDbFou3fWdmAiBENHftW4Ez81XKm+a+NwAeP yrr0D1yf9yi09D+xelh0gcVd2QlqZQTj2+DVvLhHdvIsw8tlZiCmDWX91IG0IPSIXr4X AbQUVvanDFfDs5JPnq5zR1jAj1xOJZFFVeuc/YLzn6mD1tNdXFo29YXT4Tgfrqg9Epox P+1Q== X-Gm-Message-State: AIVw113ugKx68Lq5LdPnu9VKOP7rnp1adQiZD1NJw/Surss6kI04nIzn xB/varfqWzMD5kiW6Hs= X-Received: by 10.28.6.212 with SMTP id 203mr1990977wmg.126.1501606546178; Tue, 01 Aug 2017 09:55:46 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id g16sm1710462wmd.37.2017.08.01.09.55.45 for (version=TLS1_2 cipher=AES128-SHA bits=128/128); Tue, 01 Aug 2017 09:55:45 -0700 (PDT) From: Adrien Mazarguil To: dev@dpdk.org Date: Tue, 1 Aug 2017 18:54:26 +0200 Message-Id: <8c327520c513fbcddcafbdfce05d3edebba257f3.1501598384.git.adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: References: Subject: [dpdk-dev] [PATCH v1 39/48] net/mlx4: separate interrupt handling X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Private functions are now prefixed with "mlx4_" to prevent them from conflicting with their mlx5 PMD counterparts at link time. No impact on functionality. Signed-off-by: Adrien Mazarguil --- drivers/net/mlx4/Makefile | 1 + drivers/net/mlx4/mlx4.c | 340 +--------------------------------- drivers/net/mlx4/mlx4.h | 11 ++ drivers/net/mlx4/mlx4_intr.c | 376 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 392 insertions(+), 336 deletions(-) diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile index 77aaad2..37dcdf7 100644 --- a/drivers/net/mlx4/Makefile +++ b/drivers/net/mlx4/Makefile @@ -37,6 +37,7 @@ LIB = librte_pmd_mlx4.a # Sources. SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4.c SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_flow.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_intr.c SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_utils.c # Basic CFLAGS. diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c index c99f040..284575c 100644 --- a/drivers/net/mlx4/mlx4.c +++ b/drivers/net/mlx4/mlx4.c @@ -58,7 +58,6 @@ #include #include #include -#include #include #include #include @@ -99,18 +98,6 @@ const char *pmd_mlx4_init_params[] = { NULL, }; -static int -mlx4_rx_intr_enable(struct rte_eth_dev *dev, uint16_t idx); - -static int -mlx4_rx_intr_disable(struct rte_eth_dev *dev, uint16_t idx); - -static int -priv_rx_intr_vec_enable(struct priv *priv); - -static void -priv_rx_intr_vec_disable(struct priv *priv); - /* Allocate a buffer on the stack and fill it with a printf format string. */ #define MKSTR(name, ...) \ char name[snprintf(NULL, 0, __VA_ARGS__) + 1]; \ @@ -2048,9 +2035,6 @@ mlx4_rx_queue_release(void *dpdk_rxq) rte_free(rxq); } -static int priv_intr_uninstall(struct priv *priv); -static int priv_intr_install(struct priv *priv); - /** * DPDK callback to start the device. * @@ -2075,7 +2059,7 @@ mlx4_dev_start(struct rte_eth_dev *dev) ret = priv_mac_addr_add(priv); if (ret) goto err; - ret = priv_intr_install(priv); + ret = mlx4_intr_install(priv); if (ret) { ERROR("%p: interrupt handler installation failed", (void *)dev); @@ -2113,7 +2097,7 @@ mlx4_dev_stop(struct rte_eth_dev *dev) DEBUG("%p: detaching flows from all RX queues", (void *)dev); priv->started = 0; mlx4_priv_flow_stop(priv); - priv_intr_uninstall(priv); + mlx4_intr_uninstall(priv); priv_mac_addr_del(priv); } @@ -2229,7 +2213,7 @@ mlx4_dev_close(struct rte_eth_dev *dev) claim_zero(ibv_close_device(priv->ctx)); } else assert(priv->ctx == NULL); - priv_intr_uninstall(priv); + mlx4_intr_uninstall(priv); memset(priv, 0, sizeof(*priv)); } @@ -2441,7 +2425,7 @@ mlx4_stats_reset(struct rte_eth_dev *dev) * @return * 0 on success, negative errno value otherwise and rte_errno is set. */ -static int +int mlx4_link_update(struct rte_eth_dev *dev, int wait_to_complete) { const struct priv *priv = dev->data->dev_private; @@ -2716,322 +2700,6 @@ priv_get_mac(struct priv *priv, uint8_t (*mac)[ETHER_ADDR_LEN]) return 0; } -static void mlx4_link_status_alarm(struct priv *priv); - -/** - * Collect interrupt events. - * - * @param priv - * Pointer to private structure. - * @param events - * Pointer to event flags holder. - * - * @return - * Number of events. - */ -static int -priv_collect_interrupt_events(struct priv *priv, uint32_t *events) -{ - struct ibv_async_event event; - int port_change = 0; - struct rte_eth_link *link = &priv->dev->data->dev_link; - const struct rte_intr_conf *const intr_conf = - &priv->dev->data->dev_conf.intr_conf; - int ret = 0; - - *events = 0; - /* Read all message and acknowledge them. */ - for (;;) { - if (ibv_get_async_event(priv->ctx, &event)) - break; - if ((event.event_type == IBV_EVENT_PORT_ACTIVE || - event.event_type == IBV_EVENT_PORT_ERR) && - intr_conf->lsc) { - port_change = 1; - ret++; - } else if (event.event_type == IBV_EVENT_DEVICE_FATAL && - intr_conf->rmv) { - *events |= (1 << RTE_ETH_EVENT_INTR_RMV); - ret++; - } else - DEBUG("event type %d on port %d not handled", - event.event_type, event.element.port_num); - ibv_ack_async_event(&event); - } - if (!port_change) - return ret; - mlx4_link_update(priv->dev, 0); - if (((link->link_speed == 0) && link->link_status) || - ((link->link_speed != 0) && !link->link_status)) { - if (!priv->intr_alarm) { - /* Inconsistent status, check again later. */ - priv->intr_alarm = 1; - rte_eal_alarm_set(MLX4_INTR_ALARM_TIMEOUT, - (void (*)(void *)) - mlx4_link_status_alarm, - priv); - } - } else { - *events |= (1 << RTE_ETH_EVENT_INTR_LSC); - } - return ret; -} - -/** - * Process scheduled link status check. - * - * @param priv - * Pointer to private structure. - */ -static void -mlx4_link_status_alarm(struct priv *priv) -{ - uint32_t events; - int ret; - - assert(priv->intr_alarm == 1); - priv->intr_alarm = 0; - ret = priv_collect_interrupt_events(priv, &events); - if (ret > 0 && events & (1 << RTE_ETH_EVENT_INTR_LSC)) - _rte_eth_dev_callback_process(priv->dev, - RTE_ETH_EVENT_INTR_LSC, - NULL, NULL); -} - -/** - * Handle interrupts from the NIC. - * - * @param priv - * Pointer to private structure. - */ -static void -mlx4_interrupt_handler(struct priv *priv) -{ - int ret; - uint32_t ev; - int i; - - ret = priv_collect_interrupt_events(priv, &ev); - if (ret > 0) { - for (i = RTE_ETH_EVENT_UNKNOWN; - i < RTE_ETH_EVENT_MAX; - i++) { - if (ev & (1 << i)) { - ev &= ~(1 << i); - _rte_eth_dev_callback_process(priv->dev, i, - NULL, NULL); - ret--; - } - } - if (ret) - WARN("%d event%s not processed", ret, - (ret > 1 ? "s were" : " was")); - } -} - -/** - * Uninstall interrupt handler. - * - * @param priv - * Pointer to private structure. - * - * @return - * 0 on success, negative errno value otherwise and rte_errno is set. - */ -static int -priv_intr_uninstall(struct priv *priv) -{ - int err = rte_errno; /* Make sure rte_errno remains unchanged. */ - - if (priv->intr_handle.fd != -1) { - rte_intr_callback_unregister(&priv->intr_handle, - (void (*)(void *)) - mlx4_interrupt_handler, - priv); - priv->intr_handle.fd = -1; - } - rte_eal_alarm_cancel((void (*)(void *))mlx4_link_status_alarm, priv); - priv->intr_alarm = 0; - priv_rx_intr_vec_disable(priv); - rte_errno = err; - return 0; -} - -/** - * Install interrupt handler. - * - * @param priv - * Pointer to private structure. - * - * @return - * 0 on success, negative errno value otherwise and rte_errno is set. - */ -static int -priv_intr_install(struct priv *priv) -{ - const struct rte_intr_conf *const intr_conf = - &priv->dev->data->dev_conf.intr_conf; - int rc; - - priv_intr_uninstall(priv); - if (intr_conf->rxq && priv_rx_intr_vec_enable(priv) < 0) - goto error; - if (intr_conf->lsc | intr_conf->rmv) { - priv->intr_handle.fd = priv->ctx->async_fd; - rc = rte_intr_callback_register(&priv->intr_handle, - (void (*)(void *)) - mlx4_interrupt_handler, - priv); - if (rc < 0) { - rte_errno = -rc; - goto error; - } - } - return 0; -error: - priv_intr_uninstall(priv); - return -rte_errno; -} - -/** - * Allocate queue vector and fill epoll fd list for Rx interrupts. - * - * @param priv - * Pointer to private structure. - * - * @return - * 0 on success, negative errno value otherwise and rte_errno is set. - */ -static int -priv_rx_intr_vec_enable(struct priv *priv) -{ - unsigned int i; - unsigned int rxqs_n = priv->rxqs_n; - unsigned int n = RTE_MIN(rxqs_n, (uint32_t)RTE_MAX_RXTX_INTR_VEC_ID); - unsigned int count = 0; - struct rte_intr_handle *intr_handle = &priv->intr_handle; - - priv_rx_intr_vec_disable(priv); - intr_handle->intr_vec = malloc(sizeof(intr_handle->intr_vec[rxqs_n])); - if (intr_handle->intr_vec == NULL) { - rte_errno = ENOMEM; - ERROR("failed to allocate memory for interrupt vector," - " Rx interrupts will not be supported"); - return -rte_errno; - } - for (i = 0; i != n; ++i) { - struct rxq *rxq = (*priv->rxqs)[i]; - - /* Skip queues that cannot request interrupts. */ - if (!rxq || !rxq->channel) { - /* Use invalid intr_vec[] index to disable entry. */ - intr_handle->intr_vec[i] = - RTE_INTR_VEC_RXTX_OFFSET + - RTE_MAX_RXTX_INTR_VEC_ID; - continue; - } - if (count >= RTE_MAX_RXTX_INTR_VEC_ID) { - rte_errno = E2BIG; - ERROR("too many Rx queues for interrupt vector size" - " (%d), Rx interrupts cannot be enabled", - RTE_MAX_RXTX_INTR_VEC_ID); - priv_rx_intr_vec_disable(priv); - return -rte_errno; - } - intr_handle->intr_vec[i] = RTE_INTR_VEC_RXTX_OFFSET + count; - intr_handle->efds[count] = rxq->channel->fd; - count++; - } - if (!count) - priv_rx_intr_vec_disable(priv); - else - intr_handle->nb_efd = count; - return 0; -} - -/** - * Clean up Rx interrupts handler. - * - * @param priv - * Pointer to private structure. - */ -static void -priv_rx_intr_vec_disable(struct priv *priv) -{ - struct rte_intr_handle *intr_handle = &priv->intr_handle; - - rte_intr_free_epoll_fd(intr_handle); - free(intr_handle->intr_vec); - intr_handle->nb_efd = 0; - intr_handle->intr_vec = NULL; -} - -/** - * DPDK callback for Rx queue interrupt enable. - * - * @param dev - * Pointer to Ethernet device structure. - * @param idx - * Rx queue index. - * - * @return - * 0 on success, negative errno value otherwise and rte_errno is set. - */ -static int -mlx4_rx_intr_enable(struct rte_eth_dev *dev, uint16_t idx) -{ - struct priv *priv = dev->data->dev_private; - struct rxq *rxq = (*priv->rxqs)[idx]; - int ret; - - if (!rxq || !rxq->channel) - ret = EINVAL; - else - ret = ibv_req_notify_cq(rxq->cq, 0); - if (ret) { - rte_errno = ret; - WARN("unable to arm interrupt on rx queue %d", idx); - } - return -ret; -} - -/** - * DPDK callback for Rx queue interrupt disable. - * - * @param dev - * Pointer to Ethernet device structure. - * @param idx - * Rx queue index. - * - * @return - * 0 on success, negative errno value otherwise and rte_errno is set. - */ -static int -mlx4_rx_intr_disable(struct rte_eth_dev *dev, uint16_t idx) -{ - struct priv *priv = dev->data->dev_private; - struct rxq *rxq = (*priv->rxqs)[idx]; - struct ibv_cq *ev_cq; - void *ev_ctx; - int ret; - - if (!rxq || !rxq->channel) { - ret = EINVAL; - } else { - ret = ibv_get_cq_event(rxq->cq->channel, &ev_cq, &ev_ctx); - if (ret || ev_cq != rxq->cq) - ret = EINVAL; - } - if (ret) { - rte_errno = ret; - WARN("unable to disable interrupt on rx queue %d", - idx); - } else { - ibv_ack_cq_events(rxq->cq, 1); - } - return -ret; -} - /** * Verify and store value for device argument. * diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h index 528607c..f815dd8 100644 --- a/drivers/net/mlx4/mlx4.h +++ b/drivers/net/mlx4/mlx4.h @@ -178,4 +178,15 @@ struct priv { LIST_HEAD(mlx4_flows, rte_flow) flows; }; +/* mlx4.c */ + +int mlx4_link_update(struct rte_eth_dev *dev, int wait_to_complete); + +/* mlx4_intr.c */ + +int mlx4_intr_uninstall(struct priv *priv); +int mlx4_intr_install(struct priv *priv); +int mlx4_rx_intr_disable(struct rte_eth_dev *dev, uint16_t idx); +int mlx4_rx_intr_enable(struct rte_eth_dev *dev, uint16_t idx); + #endif /* RTE_PMD_MLX4_H_ */ diff --git a/drivers/net/mlx4/mlx4_intr.c b/drivers/net/mlx4/mlx4_intr.c new file mode 100644 index 0000000..bcf4d59 --- /dev/null +++ b/drivers/net/mlx4/mlx4_intr.c @@ -0,0 +1,376 @@ +/*- + * BSD LICENSE + * + * Copyright 2017 6WIND S.A. + * Copyright 2017 Mellanox + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/** + * @file + * Interrupts handling for mlx4 driver. + */ + +#include +#include +#include +#include + +/* Verbs headers do not support -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-Wpedantic" +#endif +#include +#ifdef PEDANTIC +#pragma GCC diagnostic error "-Wpedantic" +#endif + +#include +#include +#include +#include + +#include "mlx4.h" +#include "mlx4_utils.h" + +static void mlx4_link_status_alarm(struct priv *priv); + +/** + * Clean up Rx interrupts handler. + * + * @param priv + * Pointer to private structure. + */ +static void +mlx4_rx_intr_vec_disable(struct priv *priv) +{ + struct rte_intr_handle *intr_handle = &priv->intr_handle; + + rte_intr_free_epoll_fd(intr_handle); + free(intr_handle->intr_vec); + intr_handle->nb_efd = 0; + intr_handle->intr_vec = NULL; +} + +/** + * Allocate queue vector and fill epoll fd list for Rx interrupts. + * + * @param priv + * Pointer to private structure. + * + * @return + * 0 on success, negative errno value otherwise and rte_errno is set. + */ +static int +mlx4_rx_intr_vec_enable(struct priv *priv) +{ + unsigned int i; + unsigned int rxqs_n = priv->rxqs_n; + unsigned int n = RTE_MIN(rxqs_n, (uint32_t)RTE_MAX_RXTX_INTR_VEC_ID); + unsigned int count = 0; + struct rte_intr_handle *intr_handle = &priv->intr_handle; + + mlx4_rx_intr_vec_disable(priv); + intr_handle->intr_vec = malloc(sizeof(intr_handle->intr_vec[rxqs_n])); + if (intr_handle->intr_vec == NULL) { + rte_errno = ENOMEM; + ERROR("failed to allocate memory for interrupt vector," + " Rx interrupts will not be supported"); + return -rte_errno; + } + for (i = 0; i != n; ++i) { + struct rxq *rxq = (*priv->rxqs)[i]; + + /* Skip queues that cannot request interrupts. */ + if (!rxq || !rxq->channel) { + /* Use invalid intr_vec[] index to disable entry. */ + intr_handle->intr_vec[i] = + RTE_INTR_VEC_RXTX_OFFSET + + RTE_MAX_RXTX_INTR_VEC_ID; + continue; + } + if (count >= RTE_MAX_RXTX_INTR_VEC_ID) { + rte_errno = E2BIG; + ERROR("too many Rx queues for interrupt vector size" + " (%d), Rx interrupts cannot be enabled", + RTE_MAX_RXTX_INTR_VEC_ID); + mlx4_rx_intr_vec_disable(priv); + return -rte_errno; + } + intr_handle->intr_vec[i] = RTE_INTR_VEC_RXTX_OFFSET + count; + intr_handle->efds[count] = rxq->channel->fd; + count++; + } + if (!count) + mlx4_rx_intr_vec_disable(priv); + else + intr_handle->nb_efd = count; + return 0; +} + +/** + * Collect interrupt events. + * + * @param priv + * Pointer to private structure. + * @param events + * Pointer to event flags holder. + * + * @return + * Number of events. + */ +static int +mlx4_collect_interrupt_events(struct priv *priv, uint32_t *events) +{ + struct ibv_async_event event; + int port_change = 0; + struct rte_eth_link *link = &priv->dev->data->dev_link; + const struct rte_intr_conf *const intr_conf = + &priv->dev->data->dev_conf.intr_conf; + int ret = 0; + + *events = 0; + /* Read all message and acknowledge them. */ + for (;;) { + if (ibv_get_async_event(priv->ctx, &event)) + break; + if ((event.event_type == IBV_EVENT_PORT_ACTIVE || + event.event_type == IBV_EVENT_PORT_ERR) && + intr_conf->lsc) { + port_change = 1; + ret++; + } else if (event.event_type == IBV_EVENT_DEVICE_FATAL && + intr_conf->rmv) { + *events |= (1 << RTE_ETH_EVENT_INTR_RMV); + ret++; + } else { + DEBUG("event type %d on port %d not handled", + event.event_type, event.element.port_num); + } + ibv_ack_async_event(&event); + } + if (!port_change) + return ret; + mlx4_link_update(priv->dev, 0); + if (((link->link_speed == 0) && link->link_status) || + ((link->link_speed != 0) && !link->link_status)) { + if (!priv->intr_alarm) { + /* Inconsistent status, check again later. */ + priv->intr_alarm = 1; + rte_eal_alarm_set(MLX4_INTR_ALARM_TIMEOUT, + (void (*)(void *)) + mlx4_link_status_alarm, + priv); + } + } else { + *events |= (1 << RTE_ETH_EVENT_INTR_LSC); + } + return ret; +} + +/** + * Process scheduled link status check. + * + * @param priv + * Pointer to private structure. + */ +static void +mlx4_link_status_alarm(struct priv *priv) +{ + uint32_t events; + int ret; + + assert(priv->intr_alarm == 1); + priv->intr_alarm = 0; + ret = mlx4_collect_interrupt_events(priv, &events); + if (ret > 0 && events & (1 << RTE_ETH_EVENT_INTR_LSC)) + _rte_eth_dev_callback_process(priv->dev, + RTE_ETH_EVENT_INTR_LSC, + NULL, NULL); +} + +/** + * Handle interrupts from the NIC. + * + * @param priv + * Pointer to private structure. + */ +static void +mlx4_interrupt_handler(struct priv *priv) +{ + int ret; + uint32_t ev; + int i; + + ret = mlx4_collect_interrupt_events(priv, &ev); + if (ret > 0) { + for (i = RTE_ETH_EVENT_UNKNOWN; + i < RTE_ETH_EVENT_MAX; + i++) { + if (ev & (1 << i)) { + ev &= ~(1 << i); + _rte_eth_dev_callback_process(priv->dev, i, + NULL, NULL); + ret--; + } + } + if (ret) + WARN("%d event%s not processed", ret, + (ret > 1 ? "s were" : " was")); + } +} + +/** + * Uninstall interrupt handler. + * + * @param priv + * Pointer to private structure. + * + * @return + * 0 on success, negative errno value otherwise and rte_errno is set. + */ +int +mlx4_intr_uninstall(struct priv *priv) +{ + int err = rte_errno; /* Make sure rte_errno remains unchanged. */ + + if (priv->intr_handle.fd != -1) { + rte_intr_callback_unregister(&priv->intr_handle, + (void (*)(void *)) + mlx4_interrupt_handler, + priv); + priv->intr_handle.fd = -1; + } + rte_eal_alarm_cancel((void (*)(void *))mlx4_link_status_alarm, priv); + priv->intr_alarm = 0; + mlx4_rx_intr_vec_disable(priv); + rte_errno = err; + return 0; +} + +/** + * Install interrupt handler. + * + * @param priv + * Pointer to private structure. + * + * @return + * 0 on success, negative errno value otherwise and rte_errno is set. + */ +int +mlx4_intr_install(struct priv *priv) +{ + const struct rte_intr_conf *const intr_conf = + &priv->dev->data->dev_conf.intr_conf; + int rc; + + mlx4_intr_uninstall(priv); + if (intr_conf->rxq && mlx4_rx_intr_vec_enable(priv) < 0) + goto error; + if (intr_conf->lsc | intr_conf->rmv) { + priv->intr_handle.fd = priv->ctx->async_fd; + rc = rte_intr_callback_register(&priv->intr_handle, + (void (*)(void *)) + mlx4_interrupt_handler, + priv); + if (rc < 0) { + rte_errno = -rc; + goto error; + } + } + return 0; +error: + mlx4_intr_uninstall(priv); + return -rte_errno; +} + +/** + * DPDK callback for Rx queue interrupt disable. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * Rx queue index. + * + * @return + * 0 on success, negative errno value otherwise and rte_errno is set. + */ +int +mlx4_rx_intr_disable(struct rte_eth_dev *dev, uint16_t idx) +{ + struct priv *priv = dev->data->dev_private; + struct rxq *rxq = (*priv->rxqs)[idx]; + struct ibv_cq *ev_cq; + void *ev_ctx; + int ret; + + if (!rxq || !rxq->channel) { + ret = EINVAL; + } else { + ret = ibv_get_cq_event(rxq->cq->channel, &ev_cq, &ev_ctx); + if (ret || ev_cq != rxq->cq) + ret = EINVAL; + } + if (ret) { + rte_errno = ret; + WARN("unable to disable interrupt on rx queue %d", + idx); + } else { + ibv_ack_cq_events(rxq->cq, 1); + } + return -ret; +} + +/** + * DPDK callback for Rx queue interrupt enable. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * Rx queue index. + * + * @return + * 0 on success, negative errno value otherwise and rte_errno is set. + */ +int +mlx4_rx_intr_enable(struct rte_eth_dev *dev, uint16_t idx) +{ + struct priv *priv = dev->data->dev_private; + struct rxq *rxq = (*priv->rxqs)[idx]; + int ret; + + if (!rxq || !rxq->channel) + ret = EINVAL; + else + ret = ibv_req_notify_cq(rxq->cq, 0); + if (ret) { + rte_errno = ret; + WARN("unable to arm interrupt on rx queue %d", idx); + } + return -ret; +}