Azure Key Vault is a secrets storage solution designed to contain keys and certificates for secure authentication in public cloud infrastructure. It exposes an API that allows us to request different types of credentials based on an access policy. Premium SKUs secure tokens in a dedicated HSM, which is often a contractual obligation.
Access policies allow you to lock down the operations different principals (users, groups and system identities) can perform on different objects within the vault.
Many Azure services (e.g. application gateways for SSL certificates) support fetching certificates from a vault.
Keys are used to encrypt and decrypt content. You can use a key (which makes it a KEK) to "wrap" another key to guard against MITM attacks while the key is in transit.
Secrets are unstructured bits of text that might be used a passphrase or password for a resource.