A Safer Shape for API Keys

  • api-keys
  • security

An API key is something that is very easy to think of as just another random string until something happens.

A database backup gets leaked. Headers get logged with requests by accident. The support person needs to determine which key was associated with the failed request. Three API keys were generated for the same account but no one remembers after six months which of them was the production key.

These problems are not uncommon. These are common product issues once there are users of an API.

The error here is assuming that API key generation is only concerned with creating a sufficiently long string. Of course, entropy is important. But so is the format of the key, because this will impact how it gets stored, logged, used and how easy it will be to manage.

This is the structure I use:

  • keep the public prefix
  • do not store the whole key
  • store only the hash of the whole key.
{public_prefix}.{secret_suffix}

This little bit of structure goes a long way.

What a key has to survive

For API keys issued by our system, I normally want them to have these characteristics:

  • A database leak should not allow extracting usable API keys
  • log entries should tell which key was used, but shouldn’t contain the secret itself.
  • support should be able to retrieve key information without requiring user to copy-paste the whole key.
  • user is supposed to see only the whole key at the moment of creation
  • Multiple keys per account is expected behavior
  • Key revocation, expiration, scopes and last use date/time should be attributes of the key entry.

The main point here is that API keys are bearer tokens. Anyone who knows the value of such token can use it. There is no two-factor authentication involved; there is no password recovery mechanism in the middle of the request. If the API key appears in the log entry, Slack message, analytics report, or in the database dump, we can assume that it can be used until it gets revoked.

That means the storage strategy for us should start with one rigid rule: the whole API key should not be restorable from our database.

Everything else builds on top of that.

When the database stores the secret

The worst version is also the easiest version:

CREATE TABLE api_keys (
  id UUID PRIMARY KEY,
  account_id UUID NOT NULL,
  key_value TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

When a request comes in, just look up the key_value. If support needs a particular key, search for it in the database. If a user has forgotten a key, perhaps display it. It’s all very easy because the database holds the secret.

That simplicity is exactly the problem.

A leak in the database means a leak in all the keys. The attacker does not have to guess anything, nor decrypt anything; the keys can simply be replayed directly to the API. A data breach at this point becomes an auth breach for all API clients.

I do not believe that such a tradeoff can be justified for issued API keys, which are credentials. The database must know enough to validate a key but not enough to reveal it.

Hashing fixes storage but not operations

The natural fix is to hash the key:

generate random key
store sha256(key)
on request, hash the received key and compare

This is much better already. No more bearer credentials, a databse dump now contain just hashes.

But the hash-only version is awkward to operate.

But this approach creates different problems. Suppose a request fails on production. You need to know which key it was by looking at the log line. You cannot log the whole key. Logging the hash looks better but now your logs contain a lot of data that becomes usable only if you understand that it is a hash and you know what you are searching for.

Same issue for the support. Our customer sends us a ticket saying “my production key does not work”. It would be nicer if they do not need to paste the whole key into their ticket. But if we have only the key itself or its hash as an identifier, then every support process will be annoying.

Having multiple keys per account makes the problem even worse. For example, a user may have a key for production, a key for staging and an internal service. Having a field called name is helpful, but people tend to forget to update it. The key itself should have a small, non-secret handle.

That is precisely where the prefix helps.

Giving the key a public handle

The full key has two parts:

ak_live_7hG9pQ2mLx4r.BZJqf4YoT8zYbWyvld9SgGk4p2XnUQ1Wk5mFvEo3cRA

The first part is the public prefix:

ak_live_7hG9pQ2mLx4r

The second part is the secret suffix:

BZJqf4YoT8zYbWyvld9SgGk4p2XnUQ1Wk5mFvEo3cRA

Prefix is only an identifier. It is neither any sort of permission, nor it is any kind of proof of anything.

I prefer to have a small human-readable identifier in the prefix, like ak_live or ak_test, followed by some random identifier. This will prevent users from making some very common errors, for example, using their testing API key in production. Random part of the key should be sufficiently unique for usage in logs and database lookup.

Suffix is the actual secret. You should generate it with a cryptographically strong pseudo-random number generator. I would expect the minimum length of random data is 32 bytes encoded in base64url or any other safe-for-URL alphabet. Length of your string doesn’t define randomness of your secret, but entropy does.

Delimiter is something trivial. A dot will do. Parsing and validation is all that matters.

What lands in the table

The database stores the prefix and a hash of the full key:

CREATE TABLE api_keys (
  id UUID PRIMARY KEY,
  account_id UUID NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,

  name TEXT NOT NULL,
  key_prefix TEXT NOT NULL UNIQUE,
  key_hash BYTEA NOT NULL UNIQUE,
  hash_version INTEGER NOT NULL DEFAULT 1,

  scopes TEXT[] NOT NULL DEFAULT '{}',
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  last_used_at TIMESTAMPTZ,
  expires_at TIMESTAMPTZ,
  revoked_at TIMESTAMPTZ
);

CREATE INDEX idx_api_keys_account
  ON api_keys (account_id);

CREATE INDEX idx_api_keys_active_account
  ON api_keys (account_id)
  WHERE revoked_at IS NULL;

There are a few choices hidden in this table.

  • key_prefix is stored in plain text because it is not a secret. It is the thing we are allowed to put in logs, dashboards and support tools.
  • key_hash is the verification value. I prefer an HMAC, for example HMAC-SHA-256 over the full key using a server-side secret:
key_hash = hmac_sha256(api_key_hash_secret, full_key)

SHA-256 alone on a high entropy server-side generated key is a whole different world compared to user password hashing. In case of API key the entropy needs to be sufficient for the offline guessing not to be feasible. However, the main benefit of the keyed hash approach is that a database breach doesn’t provide the attacker even with raw hash values to try them out along with the application secret.

The reason why hash_version column is needed is that we need to be able to migrate our secrets and algorithms without doing strange things. For example, we might want to change HMAC secret or even the encoding itself in the future.

The additional columns are here for good reasons::

  • name allows us to name our keys: prod backend, staging worker.
  • scopes allow us to define scopes for our keys.
  • last_used_at allows users or support to detect old unused keys.
  • expires_at allows us to use temporary credentials.
  • revoked_at allows us to revoke keys.

I would not make account_id unique. Multiple keys per account should be normal.

Show it once

When creating a key, the flow is:

generate random prefix id
generate random secret suffix
full_key = prefix + "." + suffix
key_hash = hmac_sha256(api_key_hash_secret, full_key)

store key_prefix and key_hash
show full_key to the user once

That last line is a very important part of a secure design.

Once created, the system cannot display the whole key since it doesn’t possess it anymore. This is not a lack of feature. This is the core of it. And this fact needs to be stated directly at the key creation time: please copy and save the key, as it will never be displayed again.

If the user loses their key, they can create a new one and revoke the old one.

Also, I don’t use UUIDs as my primary secret. A UUID can be handy for the purposes of identifying objects in the database, but not for creating credentials.

Authenticating without revealing

The request should send the key in a header:

Authorization: Bearer ak_live_7hG9pQ2mLx4r.BZJqf4YoT8zYbWyvld9SgGk4p2XnUQ1Wk5mFvEo3cRA

I avoid query parameters for API keys. They leak too easily into access logs, browser history, caches, analytics systems and error reports. Headers do not magically solve this problem, but they are the right default.

The authentication flow is:

read Authorization header
parse the key into prefix and suffix
validate the format
compute HMAC over the full received key
load the key record by prefix
compare the stored hash with the computed hash
check revoked_at, expires_at and scopes
attach account_id and key_id to the request context

The lookup can be boring:

SELECT *
FROM api_keys
WHERE key_prefix = $1;

Then compare the stored hash with the computed hash using a constant-time comparison.

You can also query by both prefix and hash:

SELECT *
FROM api_keys
WHERE key_prefix = $1
  AND key_hash = $2;

I prefer to keep the comparison logic explicit in the application code because that is credential validation, and the code should reflect that.

Whichever form of query you select, an invalid key response must be bland. Avoid informing the client as to whether or not the prefix was correct, whether the suffix was incorrect, whether it was expired, or whether it had been revoked. The 401 shape must always be consistent.

Logging

And that’s where the prefix pays off.

When a request comes in, the logs might contain something like:

api_key_prefix=ak_live_7hG9pQ2mLx4r

Once authentication is successful, logs may contain more detailed information:

account_id=acc_123
api_key_id=key_456
api_key_prefix=ak_live_7hG9pQ2mLx4r

However, the suffix must be absolutely off-limits.

It looks obvious, but let’s stress this again. Redact Authorization in the logging middleware. Redact Authorization in exceptions. Redact Authorization in HTTP client debugging. Redact Authorization from structured logs as soon as they are out of process.

The prefix will give you all you need to debug your app without leaving credentials in logs.

When our support team sees a failing request with api_key_prefix=ak_live_7hG9pQ2mLx4r, they can easily locate the key record. They will check which account it is associated with, its expiration date, revocation status, last usage date and scopes.

No need to ask the user for a secret anymore.

What this does not solve

This technique guards against certain types of errors. This does not mean that you should now assume that your API keys are perfectly secure.

If the user publishes the full key on GitHub, the key is compromised. The correct course of action remains the same - revoke and rotate.

If the attacker obtains application access, and can see the HMAC secret and the database, the hash doesn’t help you much anymore. You have a bigger problem now.

If a key has wide scopes and no expiration date, then the blast radius is wide. Adding prefixes and hashes to the keys will not improve your authorization mechanism.

If you use API keys via query parameters, sooner or later they will end up somewhere unpleasant.

So I still want the surrounding controls:

  • scoped keys as the default option
  • easy revocation
  • clear creation and last used time stamps
  • expiration date (optional)
  • rate limiting per key/account
  • secret scanning using key prefixes
  • creation/revocation/scope change audit events

The key format is only one layer. It just happens to be a layer that makes the rest easier.