What "tests pass" actually covers

def add(a, b):
    return a + b

API_KEY = "sk_live_51H8xJ2K9mN3pQ7rT5vW"

def test_add():
    assert add(2, 3) == 5

1 passed in 0.08s

The test suite passes. The function under test works correctly. Nothing about a green test run says anything at all about the hardcoded credential sitting three lines above it, real or fake, committed directly into a source file. Tests check that the code does what it's supposed to; they don't check for things that shouldn't be there in the first place, and a real credential accidentally committed is exactly that kind of problem; a human reviewer skimming a diff can miss it just as easily as an automated test suite that was never asked to look.

A second, separate check, chained to the first

SECRET_PATTERN = re.compile(r"(sk_live_|AKIA|-----BEGIN)[A-Za-z0-9/+=_-]*")

def scan_for_secrets():
    found = False
    for path in glob.glob("*.py"):
        with open(path) as f:
            for i, line in enumerate(f, 1):
                if SECRET_PATTERN.search(line):
                    print(f"POTENTIAL SECRET: {path}:{i}")
                    found = True
    return not found

A real, deliberately simple pattern: common real credential prefixes, sk_live_ (a live API key), AKIA (an AWS access key), -----BEGIN (a private key block). Nothing sophisticated, and that's fine, most accidental credential leaks are exactly this recognizable once something is actually looking.

Running both checks as one real gate

tests_ok = run_tests()
secrets_ok = scan_for_secrets()

if tests_ok and secrets_ok:
    sys.exit(0)
else:
    sys.exit(1)

1 passed in 0.08s
POTENTIAL SECRET: calculator.py:5

tests passed: True
no secrets found: False
GATE: FAIL

Real exit code 1. In a real CI system, this is the signal that blocks a merge, tests were fine, and the gate still failed, because passing tests was never the only bar.

A real false positive, worth naming rather than hiding

The first version of this exact scanner also flagged itself:

POTENTIAL SECRET: calculator.py:5
POTENTIAL SECRET: ci_gate.py:14

Line 14 of ci_gate.py is the regex pattern definition itself, containing the literal text sk_live_ as part of the pattern, not a real secret. A real, common category of false positive for this kind of tool: a scanner's own source code frequently contains the exact strings it's searching for. The fix is a one-line exclusion, skip the scanner's own file, not a smarter pattern:

for path in glob.glob("*.py"):
    if path == "ci_gate.py":
        continue

Worth including rather than editing out of the story: a real secret scanner in production code runs into this exact category of self-matching false positive regularly, test fixtures containing intentionally fake-looking keys, documentation examples, the scanner's own pattern definitions, and handling it is a normal, expected part of running one, not a sign the tool is broken.

The takeaway

A CI gate that only runs tests answers "does the logic work." A CI gate that also runs a real secret scan answers a completely different, equally important question: "did anything that shouldn't be committed get committed anyway." Chaining both into one script with one combined exit code, confirmed above with a real credential caught and a real false positive resolved, is what makes that second question something a pipeline checks automatically, on every single commit, instead of something that depends on a reviewer happening to notice.