Credential Scanning Tool: detect-secrets
Background
The detect-secrets tool is an open source project that uses heuristics and rules to scan for a wide range of secrets. We can extend the tool with custom rules and heuristics via a simple Python plugin API.
Unlike other credential scanning tools, detect-secrets does not attempt to check a project's entire git history when invoked, but instead scans the project's current state. This means that the tool runs quickly which makes it ideal for use in continuous integration pipelines.
detect-secrets employs the concept of a "baseline file", i.e. a list of known secrets already present in the repository, and we can configure it to ignore any of these pre-existing secrets when running. This makes it easy to gradually introduce the tool into a pre-existing project.
The baseline file also provides a simple and convenient way of handling false positives. We can white-list the false positive in the baseline file to ignore it on future invocations of the tool.
Setup
# install system dependencies: diff, jq, python3 (if on Linux-based OS)
apt-get install -y diffutils jq python3 python3-pip
# install system dependencies: diff, jq, python3 (if on Windows)
winget install Python.Python.3
choco install diffutils jq -y
# install the detect-secrets tool
python3 -m pip install detect-secrets
# run the tool to establish a list of known secrets
# review this file thoroughly and check it into the repository
detect-secrets scan > .secrets.baseline
Pre-commit hook
It is recommended to use detect-secrets in your development environment as a Git pre-commit hook.
First, follow the pre-commit installation instructions to install the tool in your development environment.
Then, add the following to your .pre-commit-config.yaml:
repos:
- repo: https://github.com/Yelp/detect-secrets
rev: v1.4.0
hooks:
- id: detect-secrets
args: ['--baseline', '.secrets.baseline']
Usage in CI pipelines
# backup the list of known secrets
cp .secrets.baseline .secrets.new
# find all the secrets in the repository
detect-secrets scan --baseline .secrets.new $(find . -type f ! -name '.secrets.*' ! -path '*/.git*')
# if there is any difference between the known and newly detected secrets, break the build
list_secrets() { jq -r '.results | keys[] as $key | "\($key),\(.[$key] | .[] | .hashed_secret)"' "$1" | sort; }
if ! diff <(list_secrets .secrets.baseline) <(list_secrets .secrets.new) >&2; then
echo "Detected new secrets in the repo" >&2
exit 1
fi