[dpdk-dev,1/6] net/vmxnet3: retain counters on restart

Message ID 1495216560-12920-1-git-send-email-ciwillia@brocade.com (mailing list archive)
State Superseded, archived
Delegated to: Ferruh Yigit
Headers

Checks

Context Check Description
ci/checkpatch warning coding style issues
ci/Intel-compilation success Compilation OK

Commit Message

Chas Williams May 19, 2017, 5:55 p.m. UTC
  From: Nachiketa Prachanda <nprachan@brocade.com>

Most nics like virtio, igb/ixgbe etc. don't reset counters on
dev_start and arguably this helps in monitoring the counters
across a longer time span with multiple device start/stops.
vmxnet3 behavior is opposite to that and counters are reset by
the host side implementation each time the device is restarted.

Change the driver to save the counters in its private context
before it is reset by writing CMD_ACTIVATE to REG_CMD.

Signed-off-by: Nachiketa Prachanda <nprachan@brocade.com>
---
 drivers/net/vmxnet3/vmxnet3_ethdev.c | 102 ++++++++++++++++++++++++++++-------
 drivers/net/vmxnet3/vmxnet3_ethdev.h |   2 +
 2 files changed, 86 insertions(+), 18 deletions(-)
  

Comments

Shrikrishna Khare May 24, 2017, 9:09 p.m. UTC | #1
On Fri, 19 May 2017, Charles (Chas) Williams wrote:

> From: Nachiketa Prachanda <nprachan@brocade.com>
> 
> Most nics like virtio, igb/ixgbe etc. don't reset counters on
> dev_start and arguably this helps in monitoring the counters
> across a longer time span with multiple device start/stops.
> vmxnet3 behavior is opposite to that and counters are reset by
> the host side implementation each time the device is restarted.
> 
> Change the driver to save the counters in its private context
> before it is reset by writing CMD_ACTIVATE to REG_CMD.
> 
> Signed-off-by: Nachiketa Prachanda <nprachan@brocade.com>

This won't be able to deal with vMotion or suspend/resume?
  
Nachi Prachanda May 25, 2017, 6:31 p.m. UTC | #2
> From: Shrikrishna Khare [mailto:skhare@shri-linux.eng.vmware.com]
> Sent: Wednesday, May 24, 2017 2:10 PM
> 
> On Fri, 19 May 2017, Charles (Chas) Williams wrote:
> 
> > From: Nachiketa Prachanda <nprachan@brocade.com>
> >
> > Most nics like virtio, igb/ixgbe etc. don't reset counters on
> > dev_start and arguably this helps in monitoring the counters across a
> > longer time span with multiple device start/stops.
> > vmxnet3 behavior is opposite to that and counters are reset by the
> > host side implementation each time the device is restarted.
> >
> > Change the driver to save the counters in its private context before
> > it is reset by writing CMD_ACTIVATE to REG_CMD.
> >
> > Signed-off-by: Nachiketa Prachanda <nprachan@brocade.com>
> 
> This won't be able to deal with vMotion or suspend/resume?

Correct - this can't deal with the VM suspend/resume unless hypervisor maintains the counter. But this patch doesn't make that behavior any worse than what it was before.
  
Shrikrishna Khare May 25, 2017, 8:27 p.m. UTC | #3
On Thu, 25 May 2017, Nachi Prachanda wrote:

> > From: Shrikrishna Khare [mailto:skhare@shri-linux.eng.vmware.com]
> > Sent: Wednesday, May 24, 2017 2:10 PM
> > 
> > On Fri, 19 May 2017, Charles (Chas) Williams wrote:
> > 
> > > From: Nachiketa Prachanda <nprachan@brocade.com>
> > >
> > > Most nics like virtio, igb/ixgbe etc. don't reset counters on
> > > dev_start and arguably this helps in monitoring the counters across a
> > > longer time span with multiple device start/stops.
> > > vmxnet3 behavior is opposite to that and counters are reset by the
> > > host side implementation each time the device is restarted.
> > >
> > > Change the driver to save the counters in its private context before
> > > it is reset by writing CMD_ACTIVATE to REG_CMD.
> > >
> > > Signed-off-by: Nachiketa Prachanda <nprachan@brocade.com>
> > 
> > This won't be able to deal with vMotion or suspend/resume?
> 
> Correct - this can't deal with the VM suspend/resume unless hypervisor maintains the counter. But this patch doesn't make that behavior any worse than what it was before.

The current code always resets stats, but am concerned that this patch 
will make the behavior inconsistent for cases like suspend/resume.

Wondering if this will be better handled by the device emulation instead 
of the driver (for igb/ixgbe, is this handled by the hardware?).

If we were to handle this in the device emulation, what would be the 
goals/requirements:
 - device start/stop should not reset stats?
 - any other operations where we would like to maintain/reset stats?
 - what might be the expectation around how accurate the stats need to be?
 - any other requirement on the device?

Also, note that if we proceed with this patch, and later extend device 
support to not reset stats, driver with this patch running on the extended 
device will report incorrect stats.

Thanks,
Shri
  
Nachi Prachanda May 25, 2017, 10:08 p.m. UTC | #4
> From: Shrikrishna Khare [mailto:skhare@shri-linux.eng.vmware.com]
> Sent: Thursday, May 25, 2017 1:27 PM
> 
> On Thu, 25 May 2017, Nachi Prachanda wrote:
> 
> > > From: Shrikrishna Khare [mailto:skhare@shri-linux.eng.vmware.com]
> > > Sent: Wednesday, May 24, 2017 2:10 PM
> > >
> > > On Fri, 19 May 2017, Charles (Chas) Williams wrote:
> > >
> > > > From: Nachiketa Prachanda <nprachan@brocade.com>
> > > >
> > > > Most nics like virtio, igb/ixgbe etc. don't reset counters on
> > > > dev_start and arguably this helps in monitoring the counters
> > > > across a longer time span with multiple device start/stops.
> > > > vmxnet3 behavior is opposite to that and counters are reset by the
> > > > host side implementation each time the device is restarted.
> > > >
> > > > Change the driver to save the counters in its private context
> > > > before it is reset by writing CMD_ACTIVATE to REG_CMD.
> > > >
> > > > Signed-off-by: Nachiketa Prachanda <nprachan@brocade.com>
> > >
> > > This won't be able to deal with vMotion or suspend/resume?
> >
> > Correct - this can't deal with the VM suspend/resume unless hypervisor
> maintains the counter. But this patch doesn't make that behavior any worse
> than what it was before.
> 
> The current code always resets stats, but am concerned that this patch will
> make the behavior inconsistent for cases like suspend/resume.
> 
> Wondering if this will be better handled by the device emulation instead of the
> driver (for igb/ixgbe, is this handled by the hardware?).

A little more nuanced.. - see below.

> If we were to handle this in the device emulation, what would be the
> goals/requirements:
>  - device start/stop should not reset stats?
>  - any other operations where we would like to maintain/reset stats?
>  - what might be the expectation around how accurate the stats need to be?
>  - any other requirement on the device?

I haven't thought about dealing it at the emulation layer - but the expectation
would be not clear counters not cleared for the lifetime of the device - and have
a way to clear them from the driver when needed.

don't know if there is a standard behavior about resetting counters. But
for igb/ixgb the counters are read/clear registers and they are maintained
at driver. May not be always accurate if the hardware is reset without updating
the driver's counter - but at least ensures that it is monotonically increasing since
on each read driver only gets the delta.

virtio emulation doesn't provide the counters - mostly the receive/send functions
updates the counters. 

> 
> Also, note that if we proceed with this patch, and later extend device support
> to not reset stats, driver with this patch running on the extended device will
> report incorrect stats.

Agree - it will need some work if the emulation changes.

Regards,
Nachi

> Thanks,
> Shri
  
Shrikrishna Khare May 26, 2017, 5:29 p.m. UTC | #5
On Thu, 25 May 2017, Nachi Prachanda wrote:

> > From: Shrikrishna Khare [mailto:skhare@shri-linux.eng.vmware.com]
> > Sent: Thursday, May 25, 2017 1:27 PM
> > 
> > On Thu, 25 May 2017, Nachi Prachanda wrote:
> > 
> > > > From: Shrikrishna Khare [mailto:skhare@shri-linux.eng.vmware.com]
> > > > Sent: Wednesday, May 24, 2017 2:10 PM
> > > >
> > > > On Fri, 19 May 2017, Charles (Chas) Williams wrote:
> > > >
> > > > > From: Nachiketa Prachanda <nprachan@brocade.com>
> > > > >
> > > > > Most nics like virtio, igb/ixgbe etc. don't reset counters on
> > > > > dev_start and arguably this helps in monitoring the counters
> > > > > across a longer time span with multiple device start/stops.
> > > > > vmxnet3 behavior is opposite to that and counters are reset by the
> > > > > host side implementation each time the device is restarted.
> > > > >
> > > > > Change the driver to save the counters in its private context
> > > > > before it is reset by writing CMD_ACTIVATE to REG_CMD.
> > > > >
> > > > > Signed-off-by: Nachiketa Prachanda <nprachan@brocade.com>
> > > >
> > > > This won't be able to deal with vMotion or suspend/resume?
> > >
> > > Correct - this can't deal with the VM suspend/resume unless hypervisor
> > maintains the counter. But this patch doesn't make that behavior any worse
> > than what it was before.
> > 
> > The current code always resets stats, but am concerned that this patch will
> > make the behavior inconsistent for cases like suspend/resume.
> > 
> > Wondering if this will be better handled by the device emulation instead of the
> > driver (for igb/ixgbe, is this handled by the hardware?).
> 
> A little more nuanced.. - see below.
> 
> > If we were to handle this in the device emulation, what would be the
> > goals/requirements:
> >  - device start/stop should not reset stats?
> >  - any other operations where we would like to maintain/reset stats?
> >  - what might be the expectation around how accurate the stats need to be?
> >  - any other requirement on the device?
> 
> I haven't thought about dealing it at the emulation layer - but the expectation
> would be not clear counters not cleared for the lifetime of the device - and have
> a way to clear them from the driver when needed.
> 
> don't know if there is a standard behavior about resetting counters. But
> for igb/ixgb the counters are read/clear registers and they are maintained
> at driver. May not be always accurate if the hardware is reset without updating
> the driver's counter - but at least ensures that it is monotonically increasing since
> on each read driver only gets the delta.
> 
> virtio emulation doesn't provide the counters - mostly the receive/send functions
> updates the counters. 
> 
> > 
> > Also, note that if we proceed with this patch, and later extend device support
> > to not reset stats, driver with this patch running on the extended device will
> > report incorrect stats.
> 
> Agree - it will need some work if the emulation changes.

I spent some time looking at the emulation code. It turns out that these 
stats are already retained across vMotion, suspend/resume. And we have no 
immediate plan to change the device behavior to retain these stats across 
device start/stop. So am fine with this patch. Thank you for being patient 
with this.

Thanks,
Shri


> 
> Regards,
> Nachi
> 
> > Thanks,
> > Shri
>
  
Shrikrishna Khare May 26, 2017, 5:31 p.m. UTC | #6
On Fri, 19 May 2017, Charles (Chas) Williams wrote:

> From: Nachiketa Prachanda <nprachan@brocade.com>
> 
> Most nics like virtio, igb/ixgbe etc. don't reset counters on
> dev_start and arguably this helps in monitoring the counters
> across a longer time span with multiple device start/stops.
> vmxnet3 behavior is opposite to that and counters are reset by
> the host side implementation each time the device is restarted.
> 
> Change the driver to save the counters in its private context
> before it is reset by writing CMD_ACTIVATE to REG_CMD.
> 
> Signed-off-by: Nachiketa Prachanda <nprachan@brocade.com>

Acked-by: Shrikrishna Khare <skhare@vmware.com>
  
Nachi Prachanda May 26, 2017, 7:01 p.m. UTC | #7
> -----Original Message-----
> From: Shrikrishna Khare [mailto:skhare@shri-linux.eng.vmware.com]
> Sent: Friday, May 26, 2017 10:29 AM
> To: Nachi Prachanda
> Cc: skhare@vmware.com; Chas Williams III; dev@dpdk.org
> Subject: RE: [PATCH 1/6] net/vmxnet3: retain counters on restart
> 
> 
> 
> On Thu, 25 May 2017, Nachi Prachanda wrote:
> 
> > > From: Shrikrishna Khare [mailto:skhare@shri-linux.eng.vmware.com]
> > > Sent: Thursday, May 25, 2017 1:27 PM
> > >
> > > On Thu, 25 May 2017, Nachi Prachanda wrote:
> > >
> > > > > From: Shrikrishna Khare
> > > > > [mailto:skhare@shri-linux.eng.vmware.com]
> > > > > Sent: Wednesday, May 24, 2017 2:10 PM
> > > > >
> > > > > On Fri, 19 May 2017, Charles (Chas) Williams wrote:
> > > > >
> > > > > > From: Nachiketa Prachanda <nprachan@brocade.com>
> > > > > >
> > > > > > Most nics like virtio, igb/ixgbe etc. don't reset counters on
> > > > > > dev_start and arguably this helps in monitoring the counters
> > > > > > across a longer time span with multiple device start/stops.
> > > > > > vmxnet3 behavior is opposite to that and counters are reset by
> > > > > > the host side implementation each time the device is restarted.
> > > > > >
> > > > > > Change the driver to save the counters in its private context
> > > > > > before it is reset by writing CMD_ACTIVATE to REG_CMD.
> > > > > >
> > > > > > Signed-off-by: Nachiketa Prachanda <nprachan@brocade.com>
> > > > >
> > > > > This won't be able to deal with vMotion or suspend/resume?
> > > >
> > > > Correct - this can't deal with the VM suspend/resume unless
> > > > hypervisor
> > > maintains the counter. But this patch doesn't make that behavior any
> > > worse than what it was before.
> > >
> > > The current code always resets stats, but am concerned that this
> > > patch will make the behavior inconsistent for cases like suspend/resume.
> > >
> > > Wondering if this will be better handled by the device emulation
> > > instead of the driver (for igb/ixgbe, is this handled by the hardware?).
> >
> > A little more nuanced.. - see below.
> >
> > > If we were to handle this in the device emulation, what would be the
> > > goals/requirements:
> > >  - device start/stop should not reset stats?
> > >  - any other operations where we would like to maintain/reset stats?
> > >  - what might be the expectation around how accurate the stats need to
> be?
> > >  - any other requirement on the device?
> >
> > I haven't thought about dealing it at the emulation layer - but the
> > expectation would be not clear counters not cleared for the lifetime
> > of the device - and have a way to clear them from the driver when needed.
> >
> > don't know if there is a standard behavior about resetting counters.
> > But for igb/ixgb the counters are read/clear registers and they are
> > maintained at driver. May not be always accurate if the hardware is
> > reset without updating the driver's counter - but at least ensures
> > that it is monotonically increasing since on each read driver only gets the
> delta.
> >
> > virtio emulation doesn't provide the counters - mostly the
> > receive/send functions updates the counters.
> >
> > >
> > > Also, note that if we proceed with this patch, and later extend
> > > device support to not reset stats, driver with this patch running on
> > > the extended device will report incorrect stats.
> >
> > Agree - it will need some work if the emulation changes.
> 
> I spent some time looking at the emulation code. It turns out that these stats
> are already retained across vMotion, suspend/resume. And we have no
> immediate plan to change the device behavior to retain these stats across
> device start/stop. So am fine with this patch. Thank you for being patient with
> this.

Thanks for checking this out.
Nachi

> 
> Thanks,
> Shri
> 
> 
> >
> > Regards,
> > Nachi
> >
> > > Thanks,
> > > Shri
> >
  
Chas Williams June 15, 2017, 12:16 p.m. UTC | #8
This series addresses some local issues with the vmxnet3 driver that
others might find of interest.

Changes in v2:

- net/vmxnet3: Implement retrieval of extended stats

    .id field wasn't being filled in xstats.

- net/vmxnet3: receive queue memory leak

    The leak and buffer reallcation were split.  The buffer reallocation
    patch may be superceded by another commit so isn't included in this
    series while under discussion.
  
Ferruh Yigit June 28, 2017, 11:30 a.m. UTC | #9
On 6/15/2017 1:16 PM, Charles (Chas) Williams wrote:
> This series addresses some local issues with the vmxnet3 driver that
> others might find of interest.
> 
> Changes in v2:
> 
> - net/vmxnet3: Implement retrieval of extended stats
> 
>     .id field wasn't being filled in xstats.
> 
> - net/vmxnet3: receive queue memory leak
> 
>     The leak and buffer reallcation were split.  The buffer reallocation
>     patch may be superceded by another commit so isn't included in this
>     series while under discussion.
> 

Series applied to dpdk-next-net/master, thanks.
  
Ferruh Yigit June 28, 2017, 12:52 p.m. UTC | #10
On 6/28/2017 12:30 PM, Ferruh Yigit wrote:
> On 6/15/2017 1:16 PM, Charles (Chas) Williams wrote:
>> This series addresses some local issues with the vmxnet3 driver that
>> others might find of interest.
>>
>> Changes in v2:
>>
>> - net/vmxnet3: Implement retrieval of extended stats
>>
>>     .id field wasn't being filled in xstats.
>>
>> - net/vmxnet3: receive queue memory leak
>>
>>     The leak and buffer reallcation were split.  The buffer reallocation
>>     patch may be superceded by another commit so isn't included in this
>>     series while under discussion.
>>
> 
> Series applied to dpdk-next-net/master, thanks.

Hi Charles,

Patches require vmxnet3.ini update, "Extended stats" and "Link status
event" ones I guess. I missed them to before getting patchset.

Would you mind sending another patch to update features document?

Thanks,
ferruh
  
Chas Williams June 28, 2017, 1:09 p.m. UTC | #11
On 06/28/2017 08:52 AM, Ferruh Yigit wrote:
> On 6/28/2017 12:30 PM, Ferruh Yigit wrote:
>> On 6/15/2017 1:16 PM, Charles (Chas) Williams wrote:
>>> This series addresses some local issues with the vmxnet3 driver that
>>> others might find of interest.
>>>
>>> Changes in v2:
>>>
>>> - net/vmxnet3: Implement retrieval of extended stats
>>>
>>>     .id field wasn't being filled in xstats.
>>>
>>> - net/vmxnet3: receive queue memory leak
>>>
>>>     The leak and buffer reallcation were split.  The buffer reallocation
>>>     patch may be superceded by another commit so isn't included in this
>>>     series while under discussion.
>>>
>>
>> Series applied to dpdk-next-net/master, thanks.
>
> Hi Charles,
>
> Patches require vmxnet3.ini update, "Extended stats" and "Link status
> event" ones I guess. I missed them to before getting patchset.
>
> Would you mind sending another patch to update features document?
>
> Thanks,
> ferruh
>

I will get to it today.
  
Chas Williams June 28, 2017, 5:15 p.m. UTC | #12
On 06/28/2017 08:52 AM, Ferruh Yigit wrote:
> On 6/28/2017 12:30 PM, Ferruh Yigit wrote:
>> On 6/15/2017 1:16 PM, Charles (Chas) Williams wrote:
>>> This series addresses some local issues with the vmxnet3 driver that
>>> others might find of interest.
>>>
>>> Changes in v2:
>>>
>>> - net/vmxnet3: Implement retrieval of extended stats
>>>
>>>     .id field wasn't being filled in xstats.
>>>
>>> - net/vmxnet3: receive queue memory leak
>>>
>>>     The leak and buffer reallcation were split.  The buffer reallocation
>>>     patch may be superceded by another commit so isn't included in this
>>>     series while under discussion.
>>>
>>
>> Series applied to dpdk-next-net/master, thanks.
>
> Hi Charles,
>
> Patches require vmxnet3.ini update, "Extended stats" and "Link status
> event" ones I guess. I missed them to before getting patchset.
>
> Would you mind sending another patch to update features document?
>
> Thanks,
> ferruh
>

Curiously, the vmxnet3 already indicates that it is handling link
status event interrupts.


; Supported features of the 'vmxnet3' network poll mode driver.
;
; Refer to default.ini for the full list of available PMD features.
;
[Features]
Speed capabilities   = P
Link status          = Y
Link status event    = Y
  
Ferruh Yigit June 28, 2017, 5:54 p.m. UTC | #13
On 6/28/2017 6:15 PM, Charles (Chas) Williams wrote:
> 
> 
> On 06/28/2017 08:52 AM, Ferruh Yigit wrote:
>> On 6/28/2017 12:30 PM, Ferruh Yigit wrote:
>>> On 6/15/2017 1:16 PM, Charles (Chas) Williams wrote:
>>>> This series addresses some local issues with the vmxnet3 driver that
>>>> others might find of interest.
>>>>
>>>> Changes in v2:
>>>>
>>>> - net/vmxnet3: Implement retrieval of extended stats
>>>>
>>>>     .id field wasn't being filled in xstats.
>>>>
>>>> - net/vmxnet3: receive queue memory leak
>>>>
>>>>     The leak and buffer reallcation were split.  The buffer reallocation
>>>>     patch may be superceded by another commit so isn't included in this
>>>>     series while under discussion.
>>>>
>>>
>>> Series applied to dpdk-next-net/master, thanks.
>>
>> Hi Charles,
>>
>> Patches require vmxnet3.ini update, "Extended stats" and "Link status
>> event" ones I guess. I missed them to before getting patchset.
>>
>> Would you mind sending another patch to update features document?
>>
>> Thanks,
>> ferruh
>>
> 
> Curiously, the vmxnet3 already indicates that it is handling link
> status event interrupts.

You are right, this can be by mistake.

> 
> 
> ; Supported features of the 'vmxnet3' network poll mode driver.
> ;
> ; Refer to default.ini for the full list of available PMD features.
> ;
> [Features]
> Speed capabilities   = P
> Link status          = Y
> Link status event    = Y
>
  

Patch

diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c b/drivers/net/vmxnet3/vmxnet3_ethdev.c
index 17b471f..4d2c024 100644
--- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
+++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
@@ -85,6 +85,7 @@  static void vmxnet3_dev_allmulticast_enable(struct rte_eth_dev *dev);
 static void vmxnet3_dev_allmulticast_disable(struct rte_eth_dev *dev);
 static int vmxnet3_dev_link_update(struct rte_eth_dev *dev,
 				   int wait_to_complete);
+static void vmxnet3_hw_stats_save(struct vmxnet3_hw *hw);
 static void vmxnet3_dev_stats_get(struct rte_eth_dev *dev,
 				  struct rte_eth_stats *stats);
 static void vmxnet3_dev_info_get(struct rte_eth_dev *dev,
@@ -351,6 +352,10 @@  eth_vmxnet3_dev_init(struct rte_eth_dev *eth_dev)
 	RTE_ASSERT((hw->rxdata_desc_size & ~VMXNET3_RXDATA_DESC_SIZE_MASK) ==
 		   hw->rxdata_desc_size);
 
+	/* clear shadow stats */
+	memset(hw->saved_tx_stats, 0, sizeof(hw->saved_tx_stats));
+	memset(hw->saved_rx_stats, 0, sizeof(hw->saved_rx_stats));
+
 	return 0;
 }
 
@@ -707,6 +712,9 @@  vmxnet3_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
+	/* Save stats before it is reset by CMD_ACTIVATE */
+	vmxnet3_hw_stats_save(hw);
+
 	ret = vmxnet3_setup_driver_shared(dev);
 	if (ret != VMXNET3_SUCCESS)
 		return ret;
@@ -820,47 +828,105 @@  vmxnet3_dev_close(struct rte_eth_dev *dev)
 }
 
 static void
+vmxnet3_hw_tx_stats_get(struct vmxnet3_hw *hw, unsigned int q,
+			struct UPT1_TxStats *res)
+{
+#define VMXNET3_UPDATE_TX_STAT(h, i, f, r)		\
+		((r)->f = (h)->tqd_start[(i)].stats.f +	\
+			(h)->saved_tx_stats[(i)].f)
+
+	VMXNET3_UPDATE_TX_STAT(hw, q, ucastPktsTxOK, res);
+	VMXNET3_UPDATE_TX_STAT(hw, q, mcastPktsTxOK, res);
+	VMXNET3_UPDATE_TX_STAT(hw, q, bcastPktsTxOK, res);
+	VMXNET3_UPDATE_TX_STAT(hw, q, ucastBytesTxOK, res);
+	VMXNET3_UPDATE_TX_STAT(hw, q, mcastBytesTxOK, res);
+	VMXNET3_UPDATE_TX_STAT(hw, q, bcastBytesTxOK, res);
+	VMXNET3_UPDATE_TX_STAT(hw, q, pktsTxError, res);
+	VMXNET3_UPDATE_TX_STAT(hw, q, pktsTxDiscard, res);
+
+#undef VMXNET3_UPDATE_TX_STAT
+}
+
+static void
+vmxnet3_hw_rx_stats_get(struct vmxnet3_hw *hw, unsigned int q,
+			struct UPT1_RxStats *res)
+{
+#define VMXNET3_UPDATE_RX_STAT(h, i, f, r)		\
+		((r)->f = (h)->rqd_start[(i)].stats.f +	\
+			(h)->saved_rx_stats[(i)].f)
+
+	VMXNET3_UPDATE_RX_STAT(hw, q, ucastPktsRxOK, res);
+	VMXNET3_UPDATE_RX_STAT(hw, q, mcastPktsRxOK, res);
+	VMXNET3_UPDATE_RX_STAT(hw, q, bcastPktsRxOK, res);
+	VMXNET3_UPDATE_RX_STAT(hw, q, ucastBytesRxOK, res);
+	VMXNET3_UPDATE_RX_STAT(hw, q, mcastBytesRxOK, res);
+	VMXNET3_UPDATE_RX_STAT(hw, q, bcastBytesRxOK, res);
+	VMXNET3_UPDATE_RX_STAT(hw, q, pktsRxError, res);
+	VMXNET3_UPDATE_RX_STAT(hw, q, pktsRxOutOfBuf, res);
+
+#undef VMXNET3_UPDATE_RX_STATS
+}
+
+static void
+vmxnet3_hw_stats_save(struct vmxnet3_hw *hw)
+{
+	unsigned int i;
+
+	VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD, VMXNET3_CMD_GET_STATS);
+
+	RTE_BUILD_BUG_ON(RTE_ETHDEV_QUEUE_STAT_CNTRS < VMXNET3_MAX_TX_QUEUES);
+
+	for (i = 0; i < hw->num_tx_queues; i++)
+		vmxnet3_hw_tx_stats_get(hw, i, &hw->saved_tx_stats[i]);
+	for (i = 0; i < hw->num_rx_queues; i++)
+		vmxnet3_hw_rx_stats_get(hw, i, &hw->saved_rx_stats[i]);
+}
+
+static void
 vmxnet3_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 {
 	unsigned int i;
 	struct vmxnet3_hw *hw = dev->data->dev_private;
+	struct UPT1_TxStats txStats;
+	struct UPT1_RxStats rxStats;
 
 	VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD, VMXNET3_CMD_GET_STATS);
 
 	RTE_BUILD_BUG_ON(RTE_ETHDEV_QUEUE_STAT_CNTRS < VMXNET3_MAX_TX_QUEUES);
 	for (i = 0; i < hw->num_tx_queues; i++) {
-		struct UPT1_TxStats *txStats = &hw->tqd_start[i].stats;
+		vmxnet3_hw_tx_stats_get(hw, i, &txStats);
+
+		stats->q_opackets[i] = txStats.ucastPktsTxOK +
+			txStats.mcastPktsTxOK +
+			txStats.bcastPktsTxOK;
 
-		stats->q_opackets[i] = txStats->ucastPktsTxOK +
-					txStats->mcastPktsTxOK +
-					txStats->bcastPktsTxOK;
-		stats->q_obytes[i] = txStats->ucastBytesTxOK +
-					txStats->mcastBytesTxOK +
-					txStats->bcastBytesTxOK;
+		stats->q_obytes[i] = txStats.ucastBytesTxOK +
+			txStats.mcastBytesTxOK +
+			txStats.bcastBytesTxOK;
 
 		stats->opackets += stats->q_opackets[i];
 		stats->obytes += stats->q_obytes[i];
-		stats->oerrors += txStats->pktsTxError + txStats->pktsTxDiscard;
+		stats->oerrors += txStats.pktsTxError + txStats.pktsTxDiscard;
 	}
 
 	RTE_BUILD_BUG_ON(RTE_ETHDEV_QUEUE_STAT_CNTRS < VMXNET3_MAX_RX_QUEUES);
 	for (i = 0; i < hw->num_rx_queues; i++) {
-		struct UPT1_RxStats *rxStats = &hw->rqd_start[i].stats;
+		vmxnet3_hw_rx_stats_get(hw, i, &rxStats);
 
-		stats->q_ipackets[i] = rxStats->ucastPktsRxOK +
-					rxStats->mcastPktsRxOK +
-					rxStats->bcastPktsRxOK;
+		stats->q_ipackets[i] = rxStats.ucastPktsRxOK +
+			rxStats.mcastPktsRxOK +
+			rxStats.bcastPktsRxOK;
 
-		stats->q_ibytes[i] = rxStats->ucastBytesRxOK +
-					rxStats->mcastBytesRxOK +
-					rxStats->bcastBytesRxOK;
+		stats->q_ibytes[i] = rxStats.ucastBytesRxOK +
+			rxStats.mcastBytesRxOK +
+			rxStats.bcastBytesRxOK;
 
 		stats->ipackets += stats->q_ipackets[i];
 		stats->ibytes += stats->q_ibytes[i];
 
-		stats->q_errors[i] = rxStats->pktsRxError;
-		stats->ierrors += rxStats->pktsRxError;
-		stats->rx_nombuf += rxStats->pktsRxOutOfBuf;
+		stats->q_errors[i] = rxStats.pktsRxError;
+		stats->ierrors += rxStats.pktsRxError;
+		stats->rx_nombuf += rxStats.pktsRxOutOfBuf;
 	}
 }
 
diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.h b/drivers/net/vmxnet3/vmxnet3_ethdev.h
index 7a03262..e93e4a7 100644
--- a/drivers/net/vmxnet3/vmxnet3_ethdev.h
+++ b/drivers/net/vmxnet3/vmxnet3_ethdev.h
@@ -122,6 +122,8 @@  struct vmxnet3_hw {
 	Vmxnet3_MemRegs	      *memRegs;
 	uint64_t	      memRegsPA;
 #define VMXNET3_VFT_TABLE_SIZE     (VMXNET3_VFT_SIZE * sizeof(uint32_t))
+	UPT1_TxStats	      saved_tx_stats[VMXNET3_MAX_TX_QUEUES];
+	UPT1_RxStats	      saved_rx_stats[VMXNET3_MAX_RX_QUEUES];
 };
 
 #define VMXNET3_REV_3		2		/* Vmxnet3 Rev. 3 */