Well, maybe it's not your Vault

the pun in the title is borrowed from my colleague and I modified a bit

These days I'm learning how to use Hashicorp Vault (properly) for two reasons.

1. I need to use it (and possibly support others to use it in the future) at current work
2. I am growing to like the design of the software as I understand them better and better

I don't know about you, but for me, Vault wasn't a terribly easy software to integrate into my software stack despite its simple design. (I appreciate Hashicorp's approaches to keep things simple for most of their tools, though) In fact, I couldn't help but avoiding this software as much as possible to keep my tech stack simpler for a while and now it's time to embrace it.

so personally it was like several attempts to watch 🏰GOT that took me two years to go through for just first 6 episodes because I didn't quite get it and no real motivation was there (other than discovering more people that are enjoying the show) until episode 7 and I started understanding things much better after that and I could appreciate the whole story later on)

Alright, now I'm kind of motivated

I told you I need to use it for my project now. 👨🏻‍💻😅

But I still struggled to understand what exactly I needed to do right to make this software not painful but pleasant to work with. So I tried this and that for hours and I'm feeling a little more comfortable now.

Although my experience with Vault is quite limited and I don't fully understand how Vault works in every detail since I still learned a few things that might be worth to share at least to future myself.

And these below are personal impressions about Vault as a relatively new user that will help me to approach doing better. So let me start.

🚰 Vault is more necessary than you might think.

Since I deploy software mostly to k8s aka Kubernetes, I will talk in that sense.

k8s secret is not surprisingly it's not a perfect place to store your important data.

I'm not saying you shouldn't use k8s secret at all, of course. I like it as an intermediate medium to deliver secrets sometimes. I'm simply saying k8s secret is most likely shouldn't be the single source of truth for your sensitive data.

here are a few whys:

• you might need that secret data somewhere other than the same k8s cluster
• you might need that secret data somewhere outside of k8s.
• no straight forward recommended practices to back up and restore your secret data.
• it's not encrypted.
• secret data are essentially stateful (you might need to update/delete them anytime) and statefulness is generally not well suited (or at least need extra care) in k8s environments.

  for example, do you run your production database in k8s? as of today, I'm pretty sure many will say, "No, I run my database in GCP or AWS as managed service" or "I run it on k8s but different cluster than my app cluster"

So simply put your secrets need to be kept somewhere more persistent and that place should provide better access control and that system should be isolated away (to be safe from outages and stuff) from your app cluster. Yes, I'm claiming your secrets should be kept in separate database software. Otherwise, you will encounter many issues.

Then why not using, for example, MySQL for that purpose?

Because sensitive data has different characteristics.

• Sensitive data should require more security layers than the general-purpose database of course
• Sensitive data doesn't necessarily require such flexibility of having columns and relations and so on like SQL based database provide.
• the combined size of sensitive data is quite smaller if you compare it to other general types of data.

these are also probably why you don't naturally come up with an idea about keeping secrets in a (general) database.

🗄️ Vault is essentially a database specialized for credential-like data.

So for starters, maybe if you can conceptualize adding Vault like you are just adding one more database into your stack instead of a-scary-unfamiliar-security-tool-slowing-down-your-development-process, then you might feel less overwhelmed.

Vault also tries to treat everything as data instead of hiding that details away from you:

• do you want to give a user a role to access which data? write that as data just like you would write another secret in Vault.
• do you want your user to authenticate via Github? write that as data
• do you want to encrypt some data? write that as data, although in this case Vault will not keep the data for you

🕵️ Vault can be quite sensitive.

Unlike the usual database that most of the time you can simply log in with username and password, Vault almost always wants more than that even to be able to talk to the server. And that, of course, is the first thing that you will feel either frightened or frustrated.

And I think there are at least two good reasons for that. First, it requires a somewhat necessary minimal security barrier to be secure of course. That's why mTLS is kind of very normal in Vault which means that you need a bunch more(ca certs, client cert, and client private key, etc) than just username and password.

🧗🏻‍♂️Even after you have everything you need, things still could be quite manual (on purpose I think)

What I'm saying is when you log in to Vault you don't get access right away, you just get/create (access) token(s) that now you can carry or bear with you to get the access.

And I know that's how websites or other software authenticate accounts too, but because of those heavy liftings are usually taken care of by frameworks or libraries, in usual development experiences they are abstracted away and handled automatically.

But with Vault, you might have to work little more than usual to make things work.

🖨 To let Consul Template work, you need to provide Vault token to Consul Template and even specify how to handle them.

👩🏻‍💼🐾 When you configure Vault Agent Template for the first time, you might be confused about why there even is an option to where to sink the token.

These things are pretty much still required for you to do work even if you are already logged in or you've set up automatic authentication.

If you ignore your responsibility of understanding how Vault and other tools handle these tokens and fail to feed it to Vault properly, then you might struggle to use Vault or other related tools I mentioned above which are ironically suppose to help you not care too much about vault specifics.

So my takeaway is:

• I like Vault but we (or at least me) may need to think slightly differently about what Vault is and how it actually cooperates with your apps and development than just simply captured on technical aspects. So that we can embrace Vault smoother.

Thanks for reading and your feedback will be helpful to improve this post!