[v3,15/15] crypto/mlx5: set feature flags and capabilities

Message ID 20210504210857.3398397-16-matan@nvidia.com (mailing list archive)
State Changes Requested, archived
Delegated to: akhil goyal
Headers
Series drivers: introduce mlx5 crypto PMD |

Checks

Context Check Description
ci/checkpatch warning coding style issues
ci/github-robot success github build: passed
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS

Commit Message

Matan Azrad May 4, 2021, 9:08 p.m. UTC
  From: Shiri Kuzin <shirik@nvidia.com>

Add the supported capabilities to the crypto driver.

Add supported feature flags.

Add crypto driver documentation.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 doc/guides/cryptodevs/features/mlx5.ini |  37 ++++++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          | 152 ++++++++++++++++++++++++
 doc/guides/rel_notes/release_21_05.rst  |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       |  39 +++++-
 5 files changed, 231 insertions(+), 3 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
  

Comments

Akhil Goyal May 8, 2021, 12:13 p.m. UTC | #1
> +
> +Overview
> +--------
> +
> +The device can provide disk encryption services, allowing data encryption
> +and decryption towards a disk. Having all encryption/decryption
> +operations done in a single device can reduce cost and overheads of the
> related
> +FIPS certification, as ConnectX-6 is FIPS 140-2 level-2 ready.
> +The encryption cipher is AES-XTS of 256/512 key size.

256/512 bit key size

> +
> +MKEY is a memory region object in the hardware, that holds address
> translation information and
> +attributes per memory area. Its ID must be tied to addresses provided to
> the hardware.
> +The encryption operations are performed with MKEY read\write

read/write

> transactions, when
> +the MKEY is configured to perform crypto operations.
> +
> +The encryption does not require text to be aligned to the AES block size
> (128b).
> +
> +In order to move the device to crypto operational mode, credential and KEK
> +(Key Encrypting Key) should be set as the first step.
> +The credential will be used by the software in order to perform crypto login,
> and the KEK is
> +the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive
> data
> +wrapping.
> +The credential and the AES-XTS keys should be provided to the hardware, as
> ciphertext
> +encrypted by the KEK.
> +
> +A keytag (64 bits) should be appended to the AES-XTS keys (before
> wrapping),
> +and will be validated when the hardware attempts to access it.
> +
> +For security reasons and to increase robustness, this driver only deals with
> virtual
> +memory addresses. The way resources allocations are handled by the
> kernel,
> +combined with hardware specifications that allow handling virtual memory
> +addresses directly, ensure that DPDK applications cannot access random
> +physical memory (or memory that does not belong to the current process).
> +
> +The PMD uses libibverbs and libmlx5 to access the device firmware or to
> +access the hardware components directly.
> +There are different levels of objects and bypassing abilities.
> +To get the best performances:
> +
> +- Verbs is a complete high-level generic API.
> +- Direct Verbs is a device-specific API.
> +- DevX allows to access firmware objects.
> +
> +Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
> +libibverbs.
> +
> +Mellanox mlx5 PCI device can be probed by a number of different PCI
> devices, such as
> +net / vDPA / RegEx. To select the crypto PMD, ``class=crypto``
> +should be specified as a device parameter. The crypto device can be probed
> and
> +used with other Mellanox classes by adding more options in the class.
> +For example: ``class=net:crypto`` will probe both the net PMD and the
> crypto
> +PMD.
> +
> +When crypto engines are defined to work in wrapped import method, they
> come out
> +of the factory in Commissioning mode, and thus, cannot be used for crypto
> operations
> +yet. A dedicated tool is used for changing the mode from Commissioning to
> +Operational, while setting the first import_KEK and credential in plaintext.
> +The mlxreg dedicated tool should be used as follows:
> +
> +- Set CRYPTO_OPERATIONAL register to set the device in crypto operational
> mode.
> +
> +  The input to this tool is:
> +    The first credential in plaintext, 40B.
> +    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
> +
> +  Example:
> +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL -
> -get
> +
> +  The "wrapped_crypto_operational" value will be "0x00000000".
> +  The command to set the register should be executed only once, and all the
> +  values mentioned above should be specified in the same command.
> +
> +  Example:
> +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
> +  --set "credential[0]=0x10000000, credential[1]=0x10000000,
> kek[0]=0x00000000"
> +
> +  All values not specified will remain 0.
> +  "wrapped_crypto_going_to_commissioning" and
> "wrapped_crypto_operational"
> +  should not be specified.
> +
> +  All the device ports should set it in order to move to operational mode.
> +
> +- Query CRYPTO_OPERATIONAL register to make sure the device is in
> Operational
> +  mode.
> +
> +  Example:
> +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL -
> -get
> +  The "wrapped_crypto_operational" value will be "0x00000001" if the
> mode was
> +  successfully changed to operational mode.
> +
> +
> +Driver options
> +--------------
> +
> +- ``class`` parameter [string]
> +
> +  Select the class of the driver that should probe the device.
> +  `crypto` for the mlx5 crypto driver.
> +
> +- ``wcs_file`` parameter [string] - mandatory
> +
> +  File path including only the wrapped credential in string format of
> hexadecimal
> +  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap
> algorithm).
> +
> +- ``import_kek_id`` parameter [int]
> +
> +  The identifier of the KEK, default value is 0 represents the operational
> +  register import_kek..
> +
> +- ``credential_id`` parameter [int]
> +
> +  The identifier of the credential, default value is 0 represents the
> operational
> +  register credential.
> +
> +- ``max_segs_num`` parameter [int]
> +
> +  Maximum number of mbuf chain segments(src or dest), default value is 8.
> +
> +- ``keytag`` parameter [int]
> +
> +  The plaintext of the keytag appanded to the AES-XTS keys, default value is
> 0.
> +
> +
> +Limitations
> +-----------
> +
> +- AES-XTS keys provided in Xform must include keytag and should be
> wrappend.
> +- The supported data-unit lengths are: 512B, 1KB, 1MB. In case the
> `dataunit_len`
> +  is not provided in the cipher Xform, the OP length is limitted to the above

Xform should be xform

Spell check  - limited

> values.
> +
> +
> +Supported NICs
> +--------------
> +
> +* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
> +
> +Prerequisites
> +-------------
> +
> +- Mellanox OFED version: **5.3**
> +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.

Since the driver is by default compiled off due to the dependency on external
Libraries, I would recommend to add few lines here as well for compilation.
Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

And I do not see any updates to the test application for testing this driver.
Is this driver really tested or is it work in progress? If it is work in progress,
We should defer this PMD to next release.

> diff --git a/doc/guides/rel_notes/release_21_05.rst
> b/doc/guides/rel_notes/release_21_05.rst
> index 0970a2331a..9030fd8b98 100644
> --- a/doc/guides/rel_notes/release_21_05.rst
> +++ b/doc/guides/rel_notes/release_21_05.rst
> @@ -275,6 +275,11 @@ New Features
>    * Added support for crypto adapter forward mode in octeontx2 event and
> crypto
>      device driver.
> 
> +* **Added support for Nvidia crypto device driver.**
> +
> +  * Added mlx5 crypto driver to support AES-XTS cipher operations.
> +    the first device to support it is ConnectX-6.

'The' 

> +
> 
>  Removed Items
>  -------------
> diff --git a/drivers/crypto/mlx5/mlx5_crypto.c
> b/drivers/crypto/mlx5/mlx5_crypto.c
> index 896ca60866..f318ff4682 100644
> --- a/drivers/crypto/mlx5/mlx5_crypto.c
> +++ b/drivers/crypto/mlx5/mlx5_crypto.c
> @@ -22,6 +22,14 @@
>  #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
>  #define MLX5_CRYPTO_MAX_QPS 1024
>  #define MLX5_CRYPTO_MAX_SEGS 56
> +#define MLX5_CRYPTO_FEATURE_FLAGS \
> +	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
> RTE_CRYPTODEV_FF_HW_ACCELERATED | \
> +	RTE_CRYPTODEV_FF_IN_PLACE_SGL |
> RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
> +	RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
> +	RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
> +	RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
> +	RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
> +	RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
> 
>  TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
> 
> 	TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
> @@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
> 
>  uint8_t mlx5_crypto_driver_id;
> 
> -const struct rte_cryptodev_capabilities
> -		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
> +const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
> +	{		/* AES XTS */
> +		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
> +		{.sym = {
> +			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
> +			{.cipher = {
> +				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
> +				.block_size = 16,
> +				.key_size = {
> +					.min = 32,
> +					.max = 64,
> +					.increment = 32
> +				},
> +				.iv_size = {
> +					.min = 16,
> +					.max = 16,
> +					.increment = 0
> +				},
> +				.dataunit_set =
> +
> 	RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
> +
> 	RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
> +			}, }
> +		}, }
> +	},
> +};
> +
> 
>  static const char mlx5_crypto_drv_name[] =
> RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
> 
> @@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
>  	RTE_SET_USED(dev);
>  	if (dev_info != NULL) {
>  		dev_info->driver_id = mlx5_crypto_driver_id;
> -		dev_info->feature_flags = 0;
> +		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
>  		dev_info->capabilities = mlx5_crypto_caps;
>  		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
>  		dev_info->min_mbuf_headroom_req = 0;
> @@ -954,6 +986,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver
> *pci_drv,
>  	crypto_dev->dev_ops = &mlx5_crypto_ops;
>  	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
>  	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
> +	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
>  	crypto_dev->driver_id = mlx5_crypto_driver_id;
>  	priv = crypto_dev->data->dev_private;
>  	priv->ctx = ctx;
> --
> 2.25.1
  
Matan Azrad May 9, 2021, 8:47 a.m. UTC | #2
From: Akhil Goyal
> > +
> > +Overview
> > +--------
> > +
> > +The device can provide disk encryption services, allowing data
> > +encryption and decryption towards a disk. Having all
> > +encryption/decryption operations done in a single device can reduce
> > +cost and overheads of the
> > related
> > +FIPS certification, as ConnectX-6 is FIPS 140-2 level-2 ready.
> > +The encryption cipher is AES-XTS of 256/512 key size.
> 
> 256/512 bit key size
> 
> > +
> > +MKEY is a memory region object in the hardware, that holds address
> > translation information and
> > +attributes per memory area. Its ID must be tied to addresses provided
> > +to
> > the hardware.
> > +The encryption operations are performed with MKEY read\write
> 
> read/write
> 
> > transactions, when
> > +the MKEY is configured to perform crypto operations.
> > +
> > +The encryption does not require text to be aligned to the AES block
> > +size
> > (128b).
> > +
> > +In order to move the device to crypto operational mode, credential
> > +and KEK (Key Encrypting Key) should be set as the first step.
> > +The credential will be used by the software in order to perform
> > +crypto login,
> > and the KEK is
> > +the AES Key Wrap Algorithm (rfc3394) key that will be used for
> > +sensitive
> > data
> > +wrapping.
> > +The credential and the AES-XTS keys should be provided to the
> > +hardware, as
> > ciphertext
> > +encrypted by the KEK.
> > +
> > +A keytag (64 bits) should be appended to the AES-XTS keys (before
> > wrapping),
> > +and will be validated when the hardware attempts to access it.
> > +
> > +For security reasons and to increase robustness, this driver only
> > +deals with
> > virtual
> > +memory addresses. The way resources allocations are handled by the
> > kernel,
> > +combined with hardware specifications that allow handling virtual
> > +memory addresses directly, ensure that DPDK applications cannot
> > +access random physical memory (or memory that does not belong to the
> current process).
> > +
> > +The PMD uses libibverbs and libmlx5 to access the device firmware or
> > +to access the hardware components directly.
> > +There are different levels of objects and bypassing abilities.
> > +To get the best performances:
> > +
> > +- Verbs is a complete high-level generic API.
> > +- Direct Verbs is a device-specific API.
> > +- DevX allows to access firmware objects.
> > +
> > +Enabling librte_crypto_mlx5 causes DPDK applications to be linked
> > +against libibverbs.
> > +
> > +Mellanox mlx5 PCI device can be probed by a number of different PCI
> > devices, such as
> > +net / vDPA / RegEx. To select the crypto PMD, ``class=crypto`` should
> > +be specified as a device parameter. The crypto device can be probed
> > and
> > +used with other Mellanox classes by adding more options in the class.
> > +For example: ``class=net:crypto`` will probe both the net PMD and the
> > crypto
> > +PMD.
> > +
> > +When crypto engines are defined to work in wrapped import method,
> > +they
> > come out
> > +of the factory in Commissioning mode, and thus, cannot be used for
> > +crypto
> > operations
> > +yet. A dedicated tool is used for changing the mode from
> > +Commissioning to Operational, while setting the first import_KEK and
> credential in plaintext.
> > +The mlxreg dedicated tool should be used as follows:
> > +
> > +- Set CRYPTO_OPERATIONAL register to set the device in crypto
> > +operational
> > mode.
> > +
> > +  The input to this tool is:
> > +    The first credential in plaintext, 40B.
> > +    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek
> data.
> > +
> > +  Example:
> > +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name
> CRYPTO_OPERATIONAL -
> > -get
> > +
> > +  The "wrapped_crypto_operational" value will be "0x00000000".
> > +  The command to set the register should be executed only once, and
> > + all the  values mentioned above should be specified in the same
> command.
> > +
> > +  Example:
> > +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name
> CRYPTO_OPERATIONAL
> > + --set "credential[0]=0x10000000, credential[1]=0x10000000,
> > kek[0]=0x00000000"
> > +
> > +  All values not specified will remain 0.
> > +  "wrapped_crypto_going_to_commissioning" and
> > "wrapped_crypto_operational"
> > +  should not be specified.
> > +
> > +  All the device ports should set it in order to move to operational mode.
> > +
> > +- Query CRYPTO_OPERATIONAL register to make sure the device is in
> > Operational
> > +  mode.
> > +
> > +  Example:
> > +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name
> CRYPTO_OPERATIONAL -
> > -get
> > +  The "wrapped_crypto_operational" value will be "0x00000001" if the
> > mode was
> > +  successfully changed to operational mode.
> > +
> > +
> > +Driver options
> > +--------------
> > +
> > +- ``class`` parameter [string]
> > +
> > +  Select the class of the driver that should probe the device.
> > +  `crypto` for the mlx5 crypto driver.
> > +
> > +- ``wcs_file`` parameter [string] - mandatory
> > +
> > +  File path including only the wrapped credential in string format of
> > hexadecimal
> > +  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap
> > algorithm).
> > +
> > +- ``import_kek_id`` parameter [int]
> > +
> > +  The identifier of the KEK, default value is 0 represents the
> > + operational  register import_kek..
> > +
> > +- ``credential_id`` parameter [int]
> > +
> > +  The identifier of the credential, default value is 0 represents the
> > operational
> > +  register credential.
> > +
> > +- ``max_segs_num`` parameter [int]
> > +
> > +  Maximum number of mbuf chain segments(src or dest), default value is
> 8.
> > +
> > +- ``keytag`` parameter [int]
> > +
> > +  The plaintext of the keytag appanded to the AES-XTS keys, default
> > + value is
> > 0.
> > +
> > +
> > +Limitations
> > +-----------
> > +
> > +- AES-XTS keys provided in Xform must include keytag and should be
> > wrappend.
> > +- The supported data-unit lengths are: 512B, 1KB, 1MB. In case the
> > `dataunit_len`
> > +  is not provided in the cipher Xform, the OP length is limitted to
> > + the above
> 
> Xform should be xform
> 
> Spell check  - limited


Ok, for all the above comments. Will adjust in v4.
 
> > values.
> > +
> > +
> > +Supported NICs
> > +--------------
> > +
> > +* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
> > +
> > +Prerequisites
> > +-------------
> > +
> > +- Mellanox OFED version: **5.3**
> > +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
> 
> Since the driver is by default compiled off due to the dependency on
> external Libraries, I would recommend to add few lines here as well for
> compilation.
> Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

Why? all Mellanox drivers has the same external dependencies.
I added here link for the doc explains it well.

> And I do not see any updates to the test application for testing this driver.

You can see update to l2fwd_crypto, we tested with this example for the first stage.
Everything looks ok there.

> Is this driver really tested or is it work in progress? If it is work in progress, We
> should defer this PMD to next release.

We can continue test this driver for a long time.
The basics were tested in the above crypto example what proofs that driver works well.
If we will find issues in other tests, we will send fixes.

I'll send v4 with your comments adjustment soon.

> 
> > diff --git a/doc/guides/rel_notes/release_21_05.rst
> > b/doc/guides/rel_notes/release_21_05.rst
> > index 0970a2331a..9030fd8b98 100644
> > --- a/doc/guides/rel_notes/release_21_05.rst
> > +++ b/doc/guides/rel_notes/release_21_05.rst
> > @@ -275,6 +275,11 @@ New Features
> >    * Added support for crypto adapter forward mode in octeontx2 event
> > and crypto
> >      device driver.
> >
> > +* **Added support for Nvidia crypto device driver.**
> > +
> > +  * Added mlx5 crypto driver to support AES-XTS cipher operations.
> > +    the first device to support it is ConnectX-6.
> 
> 'The'
> 
> > +
> >
> >  Removed Items
> >  -------------
> > diff --git a/drivers/crypto/mlx5/mlx5_crypto.c
> > b/drivers/crypto/mlx5/mlx5_crypto.c
> > index 896ca60866..f318ff4682 100644
> > --- a/drivers/crypto/mlx5/mlx5_crypto.c
> > +++ b/drivers/crypto/mlx5/mlx5_crypto.c
> > @@ -22,6 +22,14 @@
> >  #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5  #define
> > MLX5_CRYPTO_MAX_QPS 1024  #define MLX5_CRYPTO_MAX_SEGS 56
> > +#define MLX5_CRYPTO_FEATURE_FLAGS \
> > +     (RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
> > RTE_CRYPTODEV_FF_HW_ACCELERATED | \
> > +     RTE_CRYPTODEV_FF_IN_PLACE_SGL |
> > RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
> > +     RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
> > +     RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
> > +     RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
> > +     RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
> > +     RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
> >
> >  TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list
> > =
> >
> >       TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
> > @@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
> >
> >  uint8_t mlx5_crypto_driver_id;
> >
> > -const struct rte_cryptodev_capabilities
> > -             mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
> > +const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
> > +     {               /* AES XTS */
> > +             .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
> > +             {.sym = {
> > +                     .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
> > +                     {.cipher = {
> > +                             .algo = RTE_CRYPTO_CIPHER_AES_XTS,
> > +                             .block_size = 16,
> > +                             .key_size = {
> > +                                     .min = 32,
> > +                                     .max = 64,
> > +                                     .increment = 32
> > +                             },
> > +                             .iv_size = {
> > +                                     .min = 16,
> > +                                     .max = 16,
> > +                                     .increment = 0
> > +                             },
> > +                             .dataunit_set =
> > +
> >       RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
> > +
> >       RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
> > +                     }, }
> > +             }, }
> > +     },
> > +};
> > +
> >
> >  static const char mlx5_crypto_drv_name[] =
> > RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
> >
> > @@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev
> *dev,
> >       RTE_SET_USED(dev);
> >       if (dev_info != NULL) {
> >               dev_info->driver_id = mlx5_crypto_driver_id;
> > -             dev_info->feature_flags = 0;
> > +             dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
> >               dev_info->capabilities = mlx5_crypto_caps;
> >               dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
> >               dev_info->min_mbuf_headroom_req = 0; @@ -954,6 +986,7 @@
> > mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
> >       crypto_dev->dev_ops = &mlx5_crypto_ops;
> >       crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
> >       crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
> > +     crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
> >       crypto_dev->driver_id = mlx5_crypto_driver_id;
> >       priv = crypto_dev->data->dev_private;
> >       priv->ctx = ctx;
> > --
> > 2.25.1
  
Akhil Goyal May 9, 2021, 9:14 a.m. UTC | #3
> > > +Supported NICs
> > > +--------------
> > > +
> > > +* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
> > > +
> > > +Prerequisites
> > > +-------------
> > > +
> > > +- Mellanox OFED version: **5.3**
> > > +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
> >
> > Since the driver is by default compiled off due to the dependency on
> > external Libraries, I would recommend to add few lines here as well for
> > compilation.
> > Like to compile rdma-core and set PKG_CONFIG_LIBDIR.
> 
> Why? all Mellanox drivers has the same external dependencies.
> I added here link for the doc explains it well.

This is a crypto PMD, not a NIC PMD. Somebody working on crypto PMDs,
do not really care about the NIC PMDs.
Hence it would be convenient to have compilation information here as well.
You can refer to other document for details, but basic info should be added
here as well.

> 
> > And I do not see any updates to the test application for testing this driver.
> 
> You can see update to l2fwd_crypto, we tested with this example for the first
> stage.
> Everything looks ok there.

L2fwd-crypto is an app which only test data path with no packet validation.
It does not tell if your encryption is correctly done as per standards or not.
Did you test interoperability with l2fwd-crypto?
All basic configuration tests are also not done, like cleanup etc of the PMD.
I haven't seen a driver getting merge without the unit test application run.
Test app helps you comply with the way dpdk drivers are meant to be written.

> 
> > Is this driver really tested or is it work in progress? If it is work in progress,
> We
> > should defer this PMD to next release.
> 
> We can continue test this driver for a long time.
> The basics were tested in the above crypto example what proofs that driver
> works well.
> If we will find issues in other tests, we will send fixes.
> 
> I'll send v4 with your comments adjustment soon.
>
  
Matan Azrad May 9, 2021, 2:19 p.m. UTC | #4
From: Akhil Goyal
> > > > +Supported NICs
> > > > +--------------
> > > > +
> > > > +* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT
> (2x200G)
> > > > +
> > > > +Prerequisites
> > > > +-------------
> > > > +
> > > > +- Mellanox OFED version: **5.3**
> > > > +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
> > >
> > > Since the driver is by default compiled off due to the dependency on
> > > external Libraries, I would recommend to add few lines here as well
> > > for compilation.
> > > Like to compile rdma-core and set PKG_CONFIG_LIBDIR.
> >
> > Why? all Mellanox drivers has the same external dependencies.
> > I added here link for the doc explains it well.
> 
> This is a crypto PMD, not a NIC PMD. Somebody working on crypto PMDs, do
> not really care about the NIC PMDs.
> Hence it would be convenient to have compilation information here as well.
> You can refer to other document for details, but basic info should be added
> here as well.

The link explains how to install OFED, this is only what the user need to take from the link.
The basic is to install OFED.
I don't see a reason to duplicate doc section which are exactly the same. 

> >
> > > And I do not see any updates to the test application for testing this driver.
> >
> > You can see update to l2fwd_crypto, we tested with this example for
> > the first stage.
> > Everything looks ok there.
> 
> L2fwd-crypto is an app which only test data path with no packet validation.
> It does not tell if your encryption is correctly done as per standards or not.
> Did you test interoperability with l2fwd-crypto?
> All basic configuration tests are also not done, like cleanup etc of the PMD.
> I haven't seen a driver getting merge without the unit test application run.
> Test app helps you comply with the way dpdk drivers are meant to be
> written.

We adjusted the l2fwd-crypto to the dataunit feature and wrapped keys.
We validated data integrity from the packet returns back from the crypto net port.
As I said, encryption\decryption with AES-XTS is working well.

Now, is too late to update the test application to the above features, the driver code is here for a long time, no one ask about the test adjustment until now.

We can add the adjustment to increase validity for the next release to all the remaining crypto apps (test\test-crypto-perf).

For now, we have one validation with l2fwd-crypto And any user can run it and see how to use mlx5 driver.


> > > Is this driver really tested or is it work in progress? If it is
> > > work in progress,
> > We
> > > should defer this PMD to next release.
> >
> > We can continue test this driver for a long time.
> > The basics were tested in the above crypto example what proofs that
> > driver works well.
> > If we will find issues in other tests, we will send fixes.
> >
> > I'll send v4 with your comments adjustment soon.
> >
  
Akhil Goyal May 11, 2021, 6:04 p.m. UTC | #5
Hi Matan,
> > > > > +Prerequisites
> > > > > +-------------
> > > > > +
> > > > > +- Mellanox OFED version: **5.3**
> > > > > +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
> > > >
> > > > Since the driver is by default compiled off due to the dependency on
> > > > external Libraries, I would recommend to add few lines here as well
> > > > for compilation.
> > > > Like to compile rdma-core and set PKG_CONFIG_LIBDIR.
> > >
> > > Why? all Mellanox drivers has the same external dependencies.
> > > I added here link for the doc explains it well.
> >
> > This is a crypto PMD, not a NIC PMD. Somebody working on crypto PMDs,
> do
> > not really care about the NIC PMDs.
> > Hence it would be convenient to have compilation information here as
> well.
> > You can refer to other document for details, but basic info should be added
> > here as well.
> 
> The link explains how to install OFED, this is only what the user need to take
> from the link.
> The basic is to install OFED.
> I don't see a reason to duplicate doc section which are exactly the same.

As I compiled the PMD, it was not convenient to read the whole document.
And it is not needed to compile linux and everything.
I just needed rdma-core and set it in PKG_CONFIG_LIBDIR.

The reason I am insisting here is, when somebody do small changes in
Crypto library, he may need to do subsequent changes in all PMDs.
For which compilation steps should be easily accessible in the PMD doc
So that the patch can be compiled properly.

Hence I just recommend to have 3-4 lines to enable the compilation
In the PMD doc.

> > >
> > > > And I do not see any updates to the test application for testing this
> driver.
> > >
> > > You can see update to l2fwd_crypto, we tested with this example for
> > > the first stage.
> > > Everything looks ok there.
> >
> > L2fwd-crypto is an app which only test data path with no packet validation.
> > It does not tell if your encryption is correctly done as per standards or not.
> > Did you test interoperability with l2fwd-crypto?
> > All basic configuration tests are also not done, like cleanup etc of the PMD.
> > I haven't seen a driver getting merge without the unit test application run.
> > Test app helps you comply with the way dpdk drivers are meant to be
> > written.
> 
> We adjusted the l2fwd-crypto to the dataunit feature and wrapped keys.
> We validated data integrity from the packet returns back from the crypto net
> port.
> As I said, encryption\decryption with AES-XTS is working well.

Do you test interoperability here? Encryption by MLX5 and decryption
By another PMD/stack and vice-versa.
Test app is supposed to have test vectors which will work on any platform.
Hence data validation is done properly.

> 
> Now, is too late to update the test application to the above features, the
> driver code is here for a long time, no one ask about the test adjustment until
> now.

Can we defer to next release? I apologize for not asking it earlier. But this is
kind of obvious for somebody working in DPDK.
Please check that none of the PMD is merged without test app in the past 3-4yrs.

> 
> We can add the adjustment to increase validity for the next release to all the
> remaining crypto apps (test\test-crypto-perf).
> 
> For now, we have one validation with l2fwd-crypto And any user can run it
> and see how to use mlx5 driver.

The user cannot be sure of the basic things of a crypto PMD are in order or not.
As l2fwd-crypto does not test every basic thing.
For eg. Session deletion, PMD stop, PMD close, PMD restart,
setting up multiple sessions(l2fwd support single session).

Running datapath of a single use case is not sufficient for a PMD.
This is a POC and it need to comply with the environment.

I hope the doubts are clear now and we are OK to defer to next release.

Regards,
Akhil
  
Matan Azrad May 12, 2021, 6:07 a.m. UTC | #6
From: Akhil Goyal
> Hi Matan,
> > > > > > +Prerequisites
> > > > > > +-------------
> > > > > > +
> > > > > > +- Mellanox OFED version: **5.3**
> > > > > > +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
> > > > >
> > > > > Since the driver is by default compiled off due to the
> > > > > dependency on external Libraries, I would recommend to add few
> > > > > lines here as well for compilation.
> > > > > Like to compile rdma-core and set PKG_CONFIG_LIBDIR.
> > > >
> > > > Why? all Mellanox drivers has the same external dependencies.
> > > > I added here link for the doc explains it well.
> > >
> > > This is a crypto PMD, not a NIC PMD. Somebody working on crypto
> > > PMDs,
> > do
> > > not really care about the NIC PMDs.
> > > Hence it would be convenient to have compilation information here as
> > well.
> > > You can refer to other document for details, but basic info should
> > > be added here as well.
> >
> > The link explains how to install OFED, this is only what the user need
> > to take from the link.
> > The basic is to install OFED.
> > I don't see a reason to duplicate doc section which are exactly the same.
> 
> As I compiled the PMD, it was not convenient to read the whole document.
> And it is not needed to compile linux and everything.
> I just needed rdma-core and set it in PKG_CONFIG_LIBDIR.

But compilation is not enough to run, you still cannot test if you break thigs by compilation.
You need to install also the kernel modules.
That's what we explain in all our drivers.

 
> The reason I am insisting here is, when somebody do small changes in Crypto
> library, he may need to do subsequent changes in all PMDs.
> For which compilation steps should be easily accessible in the PMD doc So
> that the patch can be compiled properly.

Not enough.

> Hence I just recommend to have 3-4 lines to enable the compilation In the
> PMD doc.

We are not doing it in others mlx5 drivers.
If you insist, we will do.

> > > > > And I do not see any updates to the test application for testing
> > > > > this
> > driver.
> > > >
> > > > You can see update to l2fwd_crypto, we tested with this example
> > > > for the first stage.
> > > > Everything looks ok there.
> > >
> > > L2fwd-crypto is an app which only test data path with no packet
> validation.
> > > It does not tell if your encryption is correctly done as per standards or
> not.
> > > Did you test interoperability with l2fwd-crypto?
> > > All basic configuration tests are also not done, like cleanup etc of the
> PMD.
> > > I haven't seen a driver getting merge without the unit test application
> run.
> > > Test app helps you comply with the way dpdk drivers are meant to be
> > > written.
> >
> > We adjusted the l2fwd-crypto to the dataunit feature and wrapped keys.
> > We validated data integrity from the packet returns back from the
> > crypto net port.
> > As I said, encryption\decryption with AES-XTS is working well.
> 
> Do you test interoperability here? Encryption by MLX5 and decryption By
> another PMD/stack and vice-versa.

Compared to open-ssl results.

> Test app is supposed to have test vectors which will work on any platform.
> Hence data validation is done properly.

I think open-ssl is good standard to check too.

> >
> > Now, is too late to update the test application to the above features,
> > the driver code is here for a long time, no one ask about the test
> > adjustment until now.

I acked as maintainer of this PMD.

> Can we defer to next release? I apologize for not asking it earlier. But this is
> kind of obvious for somebody working in DPDK.
> Please check that none of the PMD is merged without test app in the past 3-
> 4yrs.

This is not true.

> > We can add the adjustment to increase validity for the next release to
> > all the remaining crypto apps (test\test-crypto-perf).
> >
> > For now, we have one validation with l2fwd-crypto And any user can run
> > it and see how to use mlx5 driver.
> 
> The user cannot be sure of the basic things of a crypto PMD are in order or
> not.
> As l2fwd-crypto does not test every basic thing.
> For eg. Session deletion, PMD stop, PMD close, PMD restart, setting up
> multiple sessions(l2fwd support single session).
> 
> Running datapath of a single use case is not sufficient for a PMD.
> This is a POC and it need to comply with the environment.

There are only one\ two cases we need in this version and we tested them.

> I hope the doubts are clear now and we are OK to defer to next release.

As I said, we don't agree, it is yours. 


> Regards,
> Akhil
>
  
Akhil Goyal May 12, 2021, 6:55 a.m. UTC | #7
> From: Akhil Goyal
> > Hi Matan,
> > > > > > > +Prerequisites
> > > > > > > +-------------
> > > > > > > +
> > > > > > > +- Mellanox OFED version: **5.3**
> > > > > > > +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
> > > > > >
> > > > > > Since the driver is by default compiled off due to the
> > > > > > dependency on external Libraries, I would recommend to add few
> > > > > > lines here as well for compilation.
> > > > > > Like to compile rdma-core and set PKG_CONFIG_LIBDIR.
> > > > >
> > > > > Why? all Mellanox drivers has the same external dependencies.
> > > > > I added here link for the doc explains it well.
> > > >
> > > > This is a crypto PMD, not a NIC PMD. Somebody working on crypto
> > > > PMDs,
> > > do
> > > > not really care about the NIC PMDs.
> > > > Hence it would be convenient to have compilation information here as
> > > well.
> > > > You can refer to other document for details, but basic info should
> > > > be added here as well.
> > >
> > > The link explains how to install OFED, this is only what the user need
> > > to take from the link.
> > > The basic is to install OFED.
> > > I don't see a reason to duplicate doc section which are exactly the same.
> >
> > As I compiled the PMD, it was not convenient to read the whole document.
> > And it is not needed to compile linux and everything.
> > I just needed rdma-core and set it in PKG_CONFIG_LIBDIR.
> 
> But compilation is not enough to run, you still cannot test if you break thigs
> by compilation.
> You need to install also the kernel modules.
> That's what we explain in all our drivers.
> 

For a person doing minor changes in the Lib, he cannot test each hardware.
He can only do compilation and that is what he is expected to do.

> 
> > The reason I am insisting here is, when somebody do small changes in
> Crypto
> > library, he may need to do subsequent changes in all PMDs.
> > For which compilation steps should be easily accessible in the PMD doc So
> > that the patch can be compiled properly.
> 
> Not enough.
> 
> > Hence I just recommend to have 3-4 lines to enable the compilation In the
> > PMD doc.
> 
> We are not doing it in others mlx5 drivers.
> If you insist, we will do.
> 
> > > > > > And I do not see any updates to the test application for testing
> > > > > > this
> > > driver.
> > > > >
> > > > > You can see update to l2fwd_crypto, we tested with this example
> > > > > for the first stage.
> > > > > Everything looks ok there.
> > > >
> > > > L2fwd-crypto is an app which only test data path with no packet
> > validation.
> > > > It does not tell if your encryption is correctly done as per standards or
> > not.
> > > > Did you test interoperability with l2fwd-crypto?
> > > > All basic configuration tests are also not done, like cleanup etc of the
> > PMD.
> > > > I haven't seen a driver getting merge without the unit test application
> > run.
> > > > Test app helps you comply with the way dpdk drivers are meant to be
> > > > written.
> > >
> > > We adjusted the l2fwd-crypto to the dataunit feature and wrapped keys.
> > > We validated data integrity from the packet returns back from the
> > > crypto net port.
> > > As I said, encryption\decryption with AES-XTS is working well.
> >
> > Do you test interoperability here? Encryption by MLX5 and decryption By
> > another PMD/stack and vice-versa.
> 
> Compared to open-ssl results.
> 
> > Test app is supposed to have test vectors which will work on any platform.
> > Hence data validation is done properly.
> 
> I think open-ssl is good standard to check too.
Yes it is good standard.
OK but what about the other basic cases that I listed below.

Did you mention any limitation that all these cases are not supported as of now.

> 
> > >
> > > Now, is too late to update the test application to the above features,
> > > the driver code is here for a long time, no one ask about the test
> > > adjustment until now.
> 
> I acked as maintainer of this PMD.
Is it really enough for the new PMD?

> 
> > Can we defer to next release? I apologize for not asking it earlier. But this is
> > kind of obvious for somebody working in DPDK.
> > Please check that none of the PMD is merged without test app in the past 3-
> > 4yrs.
> 
> This is not true.
> 
> > > We can add the adjustment to increase validity for the next release to
> > > all the remaining crypto apps (test\test-crypto-perf).
> > >
> > > For now, we have one validation with l2fwd-crypto And any user can run
> > > it and see how to use mlx5 driver.
> >
> > The user cannot be sure of the basic things of a crypto PMD are in order or
> > not.
> > As l2fwd-crypto does not test every basic thing.
> > For eg. Session deletion, PMD stop, PMD close, PMD restart, setting up
> > multiple sessions(l2fwd support single session).
> >
> > Running datapath of a single use case is not sufficient for a PMD.
> > This is a POC and it need to comply with the environment.
> 
> There are only one\ two cases we need in this version and we tested them.
> 
> > I hope the doubts are clear now and we are OK to defer to next release.
> 
> As I said, we don't agree, it is yours.
> 
As I said, fix the issues, I can still merge today but not after that.

Regards,
Akhil
  

Patch

diff --git a/doc/guides/cryptodevs/features/mlx5.ini b/doc/guides/cryptodevs/features/mlx5.ini
new file mode 100644
index 0000000000..a89526add0
--- /dev/null
+++ b/doc/guides/cryptodevs/features/mlx5.ini
@@ -0,0 +1,37 @@ 
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Symmetric crypto       = Y
+HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
diff --git a/doc/guides/cryptodevs/index.rst b/doc/guides/cryptodevs/index.rst
index 279f56a002..747409c441 100644
--- a/doc/guides/cryptodevs/index.rst
+++ b/doc/guides/cryptodevs/index.rst
@@ -22,6 +22,7 @@  Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst
new file mode 100644
index 0000000000..4ccec78be8
--- /dev/null
+++ b/doc/guides/cryptodevs/mlx5.rst
@@ -0,0 +1,152 @@ 
+.. SPDX-License-Identifier: BSD-3-Clause
+   Copyright 2021 Mellanox Technologies, Ltd
+
+.. include:: <isonum.txt>
+
+MLX5 Crypto Driver
+==================
+
+The MLX5 crypto driver library
+(**librte_crypto_mlx5**) provides support for **Mellanox ConnectX-6**
+family adapters.
+
+Overview
+--------
+
+The device can provide disk encryption services, allowing data encryption
+and decryption towards a disk. Having all encryption/decryption
+operations done in a single device can reduce cost and overheads of the related
+FIPS certification, as ConnectX-6 is FIPS 140-2 level-2 ready.
+The encryption cipher is AES-XTS of 256/512 key size.
+
+MKEY is a memory region object in the hardware, that holds address translation information and
+attributes per memory area. Its ID must be tied to addresses provided to the hardware.
+The encryption operations are performed with MKEY read\write transactions, when
+the MKEY is configured to perform crypto operations.
+
+The encryption does not require text to be aligned to the AES block size (128b).
+
+In order to move the device to crypto operational mode, credential and KEK
+(Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform crypto login, and the KEK is
+the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive data
+wrapping.
+The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
+encrypted by the KEK.
+
+A keytag (64 bits) should be appended to the AES-XTS keys (before wrapping),
+and will be validated when the hardware attempts to access it.
+
+For security reasons and to increase robustness, this driver only deals with virtual
+memory addresses. The way resources allocations are handled by the kernel,
+combined with hardware specifications that allow handling virtual memory
+addresses directly, ensure that DPDK applications cannot access random
+physical memory (or memory that does not belong to the current process).
+
+The PMD uses libibverbs and libmlx5 to access the device firmware or to
+access the hardware components directly.
+There are different levels of objects and bypassing abilities.
+To get the best performances:
+
+- Verbs is a complete high-level generic API.
+- Direct Verbs is a device-specific API.
+- DevX allows to access firmware objects.
+
+Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
+libibverbs.
+
+Mellanox mlx5 PCI device can be probed by a number of different PCI devices, such as
+net / vDPA / RegEx. To select the crypto PMD, ``class=crypto``
+should be specified as a device parameter. The crypto device can be probed and
+used with other Mellanox classes by adding more options in the class.
+For example: ``class=net:crypto`` will probe both the net PMD and the crypto
+PMD.
+
+When crypto engines are defined to work in wrapped import method, they come out
+of the factory in Commissioning mode, and thus, cannot be used for crypto operations
+yet. A dedicated tool is used for changing the mode from Commissioning to
+Operational, while setting the first import_KEK and credential in plaintext.
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto operational mode.
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and all the
+  values mentioned above should be specified in the same command.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
+  --set "credential[0]=0x10000000, credential[1]=0x10000000, kek[0]=0x00000000"
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and  "wrapped_crypto_operational"
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
+  successfully changed to operational mode.
+
+
+Driver options
+--------------
+
+- ``class`` parameter [string]
+
+  Select the class of the driver that should probe the device.
+  `crypto` for the mlx5 crypto driver.
+
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of hexadecimal
+  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the operational
+  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the operational
+  register credential.
+
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is 8.
+
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
+
+
+Limitations
+-----------
+
+- AES-XTS keys provided in Xform must include keytag and should be wrappend.
+- The supported data-unit lengths are: 512B, 1KB, 1MB. In case the `dataunit_len`
+  is not provided in the cipher Xform, the OP length is limitted to the above values.
+
+
+Supported NICs
+--------------
+
+* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
+
+Prerequisites
+-------------
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index 0970a2331a..9030fd8b98 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -275,6 +275,11 @@  New Features
   * Added support for crypto adapter forward mode in octeontx2 event and crypto
     device driver.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    the first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 896ca60866..f318ff4682 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -22,6 +22,14 @@ 
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
 #define MLX5_CRYPTO_MAX_SEGS 56
+#define MLX5_CRYPTO_FEATURE_FLAGS \
+	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
+	RTE_CRYPTODEV_FF_IN_PLACE_SGL | RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
+	RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -31,8 +39,32 @@  int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
-const struct rte_cryptodev_capabilities
-		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
+	{		/* AES XTS */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+			{.cipher = {
+				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
+				.block_size = 16,
+				.key_size = {
+					.min = 32,
+					.max = 64,
+					.increment = 32
+				},
+				.iv_size = {
+					.min = 16,
+					.max = 16,
+					.increment = 0
+				},
+				.dataunit_set =
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
+			}, }
+		}, }
+	},
+};
+
 
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
@@ -67,7 +99,7 @@  mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	RTE_SET_USED(dev);
 	if (dev_info != NULL) {
 		dev_info->driver_id = mlx5_crypto_driver_id;
-		dev_info->feature_flags = 0;
+		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
 		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
@@ -954,6 +986,7 @@  mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
 	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
 	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
+	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;