Pengenalan kepada Vault

1. Gambaran keseluruhan

Dalam tutorial ini, kita akan meneroka Hashicorp's Vault - alat popular yang digunakan untuk menguruskan maklumat sensitif dalam seni bina aplikasi moden dengan selamat .

Topik utama yang akan kita bahas, merangkumi:

  • Masalah apa yang cuba diselesaikan oleh Vault
  • Seni bina dan konsep utama Vault
  • Penyediaan persekitaran ujian sederhana
  • Berinteraksi dengan Vault menggunakan alat baris perintahnya

2. Masalah dengan Maklumat Sensitif

Sebelum menggali Vault, mari cuba memahami masalah yang cuba diselesaikannya: pengurusan maklumat sensitif.

Sebilangan besar aplikasi memerlukan akses ke data sensitif agar dapat berfungsi dengan baik . Sebagai contoh, aplikasi e-commerce mungkin mempunyai nama pengguna / kata laluan yang dikonfigurasi di suatu tempat agar dapat menyambung ke pangkalan data. Mungkin juga memerlukan kunci API untuk disatukan dengan penyedia perkhidmatan lain, seperti pintu masuk pembayaran, logistik, dan rakan perniagaan yang lain.

Kelayakan pangkalan data dan Kunci API adalah beberapa contoh maklumat sensitif yang perlu kita simpan dan sediakan untuk aplikasi kita dengan cara yang selamat.

Penyelesaian yang mudah adalah menyimpan bukti kelayakan tersebut dalam fail konfigurasi dan membacanya pada waktu permulaan. Masalah dengan pendekatan ini jelas. Sesiapa yang mempunyai akses ke fail ini berkongsi keistimewaan pangkalan data yang sama dengan aplikasi kami - biasanya memberinya akses penuh ke semua data yang disimpan.

Kita boleh membuat masalah menjadi lebih sukar dengan menyulitkan fail tersebut. Pendekatan ini, bagaimanapun, tidak akan menambah banyak aspek keselamatan keseluruhan. Terutama, kerana aplikasi kita mesti mempunyai akses ke kunci induk. Penyulitan, apabila digunakan dengan cara ini, hanya akan mencapai rasa aman yang "palsu".

Aplikasi moden dan persekitaran awan cenderung menambah kerumitan tambahan: perkhidmatan yang diedarkan, pelbagai pangkalan data, sistem pesanan dan sebagainya, semuanya mempunyai maklumat sensitif yang tersebar sedikit di mana-mana, sehingga meningkatkan risiko pelanggaran keselamatan.

Jadi, apa yang boleh kita lakukan? Ayuh Lindungi!

3. Apa itu Vault?

Hashicorp Vault menangani masalah menguruskan maklumat sensitif - rahsia dalam bahasa Vault. "Menguruskan" dalam konteks ini bermaksud bahawa Vault mengendalikan semua aspek maklumat yang sensitif : penjanaan, penyimpanan, penggunaannya dan, terakhir tetapi paling tidak, pembatalannya.

Hashicorp menawarkan dua versi Vault. Versi sumber terbuka, yang digunakan dalam artikel ini, gratis untuk digunakan, bahkan di lingkungan komersial. Versi berbayar juga tersedia, yang merangkumi sokongan teknikal pada SLA yang berbeza dan ciri tambahan, seperti sokongan HSM (Hardware Security Module).

3.1. Seni Bina & Ciri Utama

Seni bina Vault sangat menipu. Komponen utamanya adalah:

  • Latar belakang kegigihan - penyimpanan untuk semua rahsia
  • Pelayan API yang menangani permintaan pelanggan dan melakukan operasi secara rahsia
  • Sejumlah enjin rahsia, satu untuk setiap jenis jenis rahsia yang disokong

Dengan mendelegasikan semua pengendalian rahsia ke Vault, kami dapat mengurangkan beberapa masalah keselamatan:

  • Aplikasi kami tidak perlu menyimpannya lagi - minta Vault bila diperlukan dan buang
  • Kita dapat menggunakan rahsia jangka pendek, sehingga membatasi "jendela peluang" di mana penyerang dapat menggunakan rahsia yang dicuri

Vault menyulitkan semua data dengan kunci penyulitan sebelum menulisnya ke kedai. Kunci penyulitan ini disulitkan oleh kunci lain - kunci induk, digunakan hanya pada permulaan.

Titik utama dalam pelaksanaan Vault adalah ia tidak menyimpan kunci induk di pelayan. Ini bermaksud bahawa Vault bahkan tidak dapat mengakses data yang disimpan setelah dimulakan. Pada titik ini, contoh Vault dikatakan dalam keadaan "tertutup".

Kemudian, kami akan melalui langkah-langkah yang diperlukan untuk menghasilkan kunci induk dan membongkar contoh Vault.

Setelah ditutup, Vault akan siap menerima permintaan API. Permintaan itu, tentu saja, memerlukan pengesahan, yang membawa kita bagaimana Vault mengesahkan pelanggan dan memutuskan apa yang boleh atau tidak dapat mereka lakukan.

3.2. Pengesahan

Untuk mengakses rahsia di Vault, pelanggan perlu mengesahkan dirinya menggunakan salah satu kaedah yang disokong . Kaedah paling mudah menggunakan Token, yang hanya rentetan yang dihantar pada setiap permintaan API menggunakan tajuk HTTP khas.

Apabila mula-mula dipasang, Vault secara automatik menghasilkan "token root". Token ini setara dengan root superuser dalam sistem Linux, jadi penggunaannya harus dibatasi minimum. Sebagai amalan terbaik, kita harus menggunakan token root ini hanya untuk membuat token lain dengan lebih sedikit hak istimewa dan kemudian membatalkannya. Ini bukan masalah, kerana kita kemudian dapat menghasilkan token root yang lain menggunakan kunci unseal.

Vault juga menyokong mekanisme pengesahan lain seperti LDAP, JWT, TLS Certificate, antara lain. Semua mekanisme tersebut dibina berdasarkan mekanisme token asas: setelah Vault mengesahkan klien kami, ia akan memberikan token yang kemudian dapat kami gunakan untuk mengakses API lain.

Token mempunyai beberapa sifat yang berkaitan dengannya. Sifat utama adalah:

  • Satu set Dasar yang berkaitan (lihat bahagian seterusnya)
  • Masa untuk hidup
  • Adakah ia boleh diperbaharui
  • Kiraan penggunaan maksimum

Kecuali diberitahu sebaliknya, token yang dibuat oleh Vault akan membentuk hubungan ibu bapa-anak. Token kanak-kanak boleh mempunyai tahap keistimewaan yang sama dengan yang dimiliki oleh ibu bapa.

Sebaliknya tidak benar: kita boleh - dan biasanya melakukannya - membuat token anak dengan polisi yang ketat Perkara penting lain mengenai hubungan ini: Apabila kita membatalkan token, semua token anak, dan keturunannya juga tidak sah .

3.3. Dasar

Polisi menentukan dengan tepat rahsia mana yang dapat diakses oleh pelanggan dan operasi mana yang dapat dilakukan dengan mereka . Mari kita lihat bagaimana dasar mudah:

path "secret/accounting" { capabilities = [ "read" ] }

Di sini kami telah menggunakan sintaks HCL (Hashicorp's Configuration Language) untuk menentukan dasar kami. Vault juga menyokong JSON untuk tujuan ini, tetapi kami akan mematuhi HCL dalam contoh kami kerana lebih mudah dibaca.

Policies in Vault are “deny by default”. A token attached to this sample policy will get access to secrets stored under secret/accounting and nothing else. At creation time a token can be attached to multiple policies. This is very useful because it allows us to create and test smaller policies and then apply them as required.

Another important aspect of policies is that they leverage lazy-evaluation. This means that we can update a given policy and all tokens will be affected immediately.

The policies described so far are also called Access Control List Policies, or ACL Policies. Vault also supports two additional policy types: EGP and RGP policies. Those are only available in the paid versions and extend the basic policy syntax with Sentinel support.

When available, this allows us to take into account in our policies additional attributes such as time of the day, multiple authentication factors, client network origin, and so on. For instance, we can define a policy that allows access to a given secret only on business hours.

We can find more details on the policy syntax in Vault's documentation.

4. Secret Types

Vault support a range of different secret types which address different use cases:

  • Key-Value: simple static key-values pairs
  • Dynamically generated credentials: generated by Vault upon request by a client
  • Cryptographic keys: Used to perform cryptographic functions with client data

Each secret type is defined by the following attributes:

  • A mountpoint, which defines its REST API prefix
  • A set of operations exposed through the corresponding API
  • A set of configuration parameters

A given secret instance is accessible via a path, much like a directory tree in a file system. The first component of this path corresponds to the mount point where all secrets of this type are located.

For instance, the string secret/my-application corresponds to the path under which we can find key-value pairs for my-application.

4.1. Key-Value Secrets

Key-Value secrets are, as the name implies, simple pairs in the available under a given path. For instance, we can store the pair foo=bar under the path /secret/my-application.

Later on, we use the same path to retrieve the same pair or pairs – multiple pairs can be stored under the same path.

Vault support three kinds of Key-Value secrets:

  • Non-versioned Key-Pairs, where updates replace existing values
  • Versioned Key-Pairs, which keep up to a configurable number of old versions
  • Cubbyhole, a special type of non-versioned key-pairs whose values are scoped to a given access token (more on those later).

Key-Value secrets are static by nature, so there is no concept of an associated expiration associated with them. The main use case for this kind of secret is to store credentials to access external systems, such as API keys.

In such scenarios credential updates are a semi-manual process, usually requiring someone to acquire new credentials and using Vault's command line or its UI to enter the new values.

4.2. Dynamically Generated Secrets

Dynamic secrets are generated on the fly by Vault when requested by an application. Vault support several types of dynamic secrets, including the following ones:

  • Database credentials
  • SSH Key Pairs
  • X.509 Certificates
  • AWS Credentials
  • Google Cloud service accounts
  • Active Directory accounts

All these follow the same usage pattern. First, we configure the secret engine with the details required to connect to the associated service. Then, we define one or more roles, which describe the actual secret creation.

Let's take the Database secret engine as an example. First, we must configure Vault with all user database connections details, including credentials from a preexisting user with admin privileges to create new users.

Then we create one or more roles (Vault roles, not Database roles) containing the actual SQL statements used to create a new user. Those usually include not only the user creation statement but also all the required grant statements required to access schema objects (tables, views and so on).

When a client accesses the corresponding API, Vault will create a new temporary user in the database using the provided statements and return its credentials. The client can then use those credentials to access the database during the period defined by the time-to-live attribute of the requested role.

Once a credential reaches its expiration time, Vault will automatically revoke any privilege associated with this user. A client can also request Vault to renew those credentials. The renewal process will happen only if supported by the specific database driver and allowed by the associated policy.

4.3. Cryptographic Keys

Secret engines of type handle cryptographic functions such as encryption, decryption, signing and so on. All those operations use cryptographic keys generated and stored internally by Vault. Unless explicitly told to do so, Vault will never expose a given cryptographic key.

The associated API allows clients to send Vault plain-text data and receive an encrypted version of it. The opposite is also possible: We can send encrypted data and get back the original text.

Currently, there is only one engine of this type: the Transit engine. This engine supports popular keys types, such as RSA and ECDSA, and also supports Convergent Encryption. When using this mode, a given plaintext value always result in the same cyphertext result, a property that is very useful in some applications.

For instance, we can use this mode to encrypt credit card numbers in a transaction log table. With convergent encryption, every time we insert a new transaction, the encrypted credit card value would be the same, thus allowing the use of regular SQL queries for reporting, searching and so on.

5. Vault Setup

In this section, we will create a local test environment so we test the Vault's capabilities.

Vault's deployment is simple: just download the package that corresponds to our operating system and extracts its executable (vault or vault.exe on Windows) to some directory on our PATH. This executable contains the server and is also the standard client. There is also an official Docker image available, but we will not cover it here.

Vault support a development mode, which is fine for some quick testing and getting used to its command line tool, but it is way too simplistic for real use cases: all data is lost on restart and API access uses plain HTTP.

Instead, we'll use file-based persistent storage and setup HTTPS so we can explore some of the real-life configuration details that can be a source of problems.

5.1. Starting Vault Server

Vault uses a configuration file using HCL or JSON format. The following file defines all the configuration needed to start our server using a file storage and a self-signed certificate:

storage "file" { path = "./vault-data" } listener "tcp" { address = "127.0.0.1:8200" tls_cert_file = "./src/test/vault-config/localhost.cert" tls_key_file = "./src/test/vault-config/localhost.key" }

Now, let's run Vault. Open a command shell, go to the directory containing our configuration file and run this command:

$ vault server -config ./vault-test.hcl

Vault will start and show a few initialization messages. They'll include its version, some configuration details and the address where the API is available. That's it – our Vault server is up and running.

5.2. Vault Initialization

Our Vault server now is running, but since this is its first run, we need to initialize it.

Let's open a new shell and execute the following commands to achieve this:

$ export VAULT_ADDR=//localhost:8200 $ export VAULT_CACERT=./src/test/vault-config/localhost.cert $ vault operator init

Here we have defined a few environment variables, so we don't have to pass them to Vault every time as parameters:

  • VAULT_ADDR: base URI where our API server will serve requests
  • VAULT_CACERT: Path to our server's certificate public key

In our case, we use the VAULT_CACERT so we can use HTTPS to access Vault's API. We need this because we're using self-signed certificates. This would not be necessary for productions environments, where we usually have access to CA-signed certificates.

After issuing the above command, we should see a message like this:

Unseal Key 1:  Unseal Key 2:  Unseal Key 3:  Unseal Key 4:  Unseal Key 5:  Initial Root Token:  ... more messages omitted

The five first lines are the master key shares that we will later use to unseal Vault's storage. Please note that Vault only displays the master key shares will during initialization – and never more.Take note and store them safely or we'll lose access to our secrets upon server restart!

Also, please take note of the root token, as we will need it later. Unlike unseal keys, root tokens can easily be generated at a later time, so it is safe to destroy it once all configuration tasks are complete. Since we will be issuing commands later that require an authentication token, let's save the root token for now in an environment variable:

$ export VAULT_TOKEN= (Unix/Linux)

Let's see our server status now that we have initialized it, with the following command:

$ vault status Key Value --- ----- Seal Type shamir Sealed true Total Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 0.10.4 HA Enabled false

We can see that Vault is still sealed. We can also follow the unseal progress: “0/3” means that Vault needs three shares, but got none so far. Let's move ahead and provide it with our shares.

5.3. Vault Unseal

We now unseal Vault so we can start using its secret services. We need to provide any three of the five key shares in order to complete the unseal process:

$ vault operator unseal  $ vault operator unseal  $ vault operator unseal 

After issuing each command vault will print the unseal progress, including how many shares it needs. Upon sending the last key share, we'll see a message like this:

Key Value --- ----- Seal Type shamir Sealed false ... other properties omitted

The “Sealed” property is “false” in this case, which means that Vault is ready to accept commands.

6. Testing Vault

In this section, we will test our Vault setup using two of its supported secret types: Key/Value and Database. We will also show how to create new tokens with specific policies attached to them.

6.1. Using Key/Value Secrets

First, let's store secret Key-Value pairs and read them back. Assuming the command shell used to initialize Vault is still open, we use the following command to store those pairs under the secret/fakebank path:

$ vault kv put secret/fakebank api_key=abc1234 api_secret=1a2b3c4d

We can now recover those pairs at any time with the following command:

$ vault kv get secret/fakebank ======= Data ======= Key Value --- ----- api_key abc1234 api_secret 1a2b3c4d 

This simple test shows us that Vault is working as it should. We can now test some additional functionalities.

6.2. Creating New Tokens

So far we have used the root token in order to authenticate our requests. Since a root token is way too powerful, it is considered a best practice to use tokens with fewer privileges and shorter time-to-live.

Let's create a new token that we can use just like the root token, but expires after just a minute:

$ vault token create -ttl 1m Key Value --- ----- token  token_accessor  token_duration 1m token_renewable true token_policies ["root"] identity_policies [] policies ["root"]

Let's test this token, using it to read the key/value pairs that we've created before:

$ export VAULT_TOKEN= $ vault kv get secret/fakebank ======= Data ======= Key Value --- ----- api_key abc1234 api_secret 1a2b3c4d

If we wait a minute and try to reissue this command, we get an error message:

$ vault kv get secret/fakebank Error making API request. URL: GET //localhost:8200/v1/sys/internal/ui/mounts/secret/fakebank Code: 403. Errors: * permission denied

The message indicates that our token is no longer valid, which is what we've expected.

6.3. Testing Policies

The sample token we've created in the previous section was shorted lived, but still very powerful. Let's now use policies to create more restricted tokens.

For instance, let's define a policy that allows only read access to the secret/fakebank path we used before:

$ cat > sample-policy.hcl <
    

Now we create a token with this policy with the following command:

$ export VAULT_TOKEN= $ vault token create -policy=fakebank-ro Key Value --- ----- token token_accessor token_duration 768h token_renewable true token_policies ["default" "fakebank-ro"] identity_policies [] policies ["default" "fakebank-ro"]

As we've done before, let's read our secret values using this token:

$ export VAULT_TOKEN= $ vault kv get secret/fakebank ======= Data ======= Key Value --- ----- api_key abc1234 api_secret 1a2b3c4d

So far, so good. We can read data, as expected. Let's see what happens when we try to update this secret:

$ vault kv put secret/fakebank api_key=foo api_secret=bar Error writing data to secret/fakebank: Error making API request. URL: PUT //127.0.0.1:8200/v1/secret/fakebank Code: 403. Errors: * permission denied

Since our policy does not explicitly allows writes, Vault returns a 403 – Access Denied status code.

6.4. Using Dynamic Database Credentials

As our final example in this article, let's use Vault's Database secret engine in order to create dynamic credentials. We assume here that we have a MySQL server available locally and that we can access it with “root” privileges. We will also use a very simple schema consisting of a single table – account .

The SQL script used to create this schema and the privileged user is available here.

Now, let's configure Vault to use this database. The database secret engine is not enabled by default, so we must fix this before we can proceed:

$ vault secrets enable database Success! Enabled the database secrets engine at: database/

We now create a database configuration resource :

$ vault write database/config/mysql-fakebank \ plugin_name=mysql-legacy-database-plugin \ connection_url="{{username}}:{{password}}@tcp(127.0.0.1:3306)/fakebank" \ allowed_roles="*" \ username="fakebank-admin" \ password="Sup&rSecre7!"

The path prefix database/config is where all database configurations must be stored. We choose the name mysql-fakebank so we can easily figure out to which database this configuration refers to. As for the configuration keys:

  • plugin_name: Defines which database plugin will be used. The available plugin names are described in Vault's docs
  • connection_url: This is a template used by the plugin when connecting to the database. Notice the {{username}} and {{password}} template placeholders. When connecting to the database, Vault will replace those placeholders by actual values
  • allowed_roles: Define which Vault roles (discussed next) can use this configuration. In our case we use “*”, so its available to all roles
  • username & password: This is the account that Vault will use to perform database operations, such as creating a new user and revoking its privileges

Vault Database Role Setup

The final configuration task is to create a Vault database role resource that contains the SQL commands required to create a user. We can create as many roles as needed, according to our security requirements.

Here, we create a role that grants read-only access to all tables of the fakebank schema:

$ vault write database/roles/fakebank-accounts-ro \ db_name=mysql-fakebank \ creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT SELECT ON fakebank.* TO '{{name}}'@'%';" 

The database engine defines the path prefix database/roles as the location to store roles. fakebank-accounts-ro is the role name that we'll later use when creating dynamic credentials. We also supply the following keys:

  • db_name: Name of an existing database configuration. Corresponds to the last part of the path we used when creating the configuration resource
  • creation_statements: A list of SQL statement templates that Vault will use to create a new user

Creating Dynamic Credentials

Once we have a database role and its corresponding configuration ready, we generate new dynamic credentials with the following command:

$ vault read database/creds/fakebank-accounts-ro Key Value --- ----- lease_id database/creds/fakebank-accounts-ro/0c0a8bef-761a-2ef2-2fed-4ee4a4a076e4 lease_duration 1h lease_renewable true password username 

The database/creds prefix is used to generate credentials for the available roles. Since we have used the fakebank-accounts-ro role, the returned username/password will be restricted to select operations.

We can verify this by connecting to the database using the supplied credentials and then performing some SQL commands:

$ mysql -h 127.0.0.1 -u -p fakebank Enter password: MySQL [fakebank]> select * from account; ... omitted for brevity 2 rows in set (0.00 sec) MySQL [fakebank]> delete from account; ERROR 1142 (42000): DELETE command denied to user 'v-fake-9xoSKPkj1'@'localhost' for table 'account' 

We can see that the first select completed successfully, but we could not perform the delete statement. Finally, if we wait for one hour and try to connect using those same credentials, we will not be able to connect anymore to the database. Vault has automatically revoked all privileges from this user

7. Conclusion

In this article have explored the basics of Hashicorp's Vault, including some background on the problem it tries to address, its architecture and basic use.

Along the way, we have created a simple but functional test environment that we´ll use in follow-up articles.

The next article will cover a very specific use case for Vault: Using it in the context of Spring Boot application. Stay tuned!