clerk: c508b13

2 files changed, 12 insertions(+), 3 deletions(-)

ci: stabilize benchmark regression check

The -benchtime=200x gate produced false positives: at 200 iterations,
microsecond-scale benchmarks yield samples with 15-35% CV, and benchstat
reports the noise as significant (a +49% "regression" appeared between
two commits whose only diff was this workflow). Lengthen each sample to
100ms and drop count to 6 so noise averages out within the same job
budget.

Author: Oleksandr Smirnov olexsmir@gmail.com

Committed at: 2026-08-24 00:12:32 +0300

Authored at: 2026-08-23 23:53:47 +0300

Change ID: uulplpywmrlwnnryqktooxmkkxmmruzs

Parent: e00aad8

jump to

M	.github/scripts/check-bench-regression.py
M	.github/workflows/benchmarks.yml

M .github/scripts/check-bench-regression.py

···
        4
        4
         Reads benchstat CSV comparison (old vs new) from stdin. benchstat only

      
        5
        5
         reports a numeric delta for statistically significant changes (p < 0.05);

      
        6
        6
         noisy comparisons are marked "~" and ignored.

      
        
        7
        +Only sec/op rows are judged; B/op and allocs/op deltas are informational

      
        
        8
        +(deliberate tradeoffs like buffering input shift memory without slowing

      
        
        9
        +down).

      
        7
        10
         """

      
        8
        11
         import csv

      
        9
        12
         import sys

      ···
        30
        33
         

      
        31
        34
         def main():

      
        32
        35
             regressions = []

      
        
        36
        +    unit = ""

      
        33
        37
             for row in csv.reader(sys.stdin):

      
        34
        
        -        if len(row) < 7 or row[0] in ("", "geomean"):

      
        
        38
        +        if row and row[0] == "":

      
        
        39
        +            unit = row[1] if len(row) > 1 else ""

      
        
        40
        +            continue

      
        
        41
        +        if len(row) < 7 or row[0] in ("geomean",):

      
        
        42
        +            continue

      
        
        43
        +        if unit != "sec/op":

      
        35
        44
                     continue

      
        36
        45
                 d = delta_pct(row[5])  # vs base column

      
        37
        46
                 if d is not None and d > THRESHOLD:

M .github/workflows/benchmarks.yml

···
        38
        38
               - name: Run base benchmarks

      
        39
        39
                 if: github.event_name == 'pull_request'

      
        40
        40
                 working-directory: __base

      
        41
        
        -        run: go test -bench . -benchmem -run '^$' -count=10 -benchtime=200x ./... > ../bench-base.txt 2>&1

      
        
        41
        +        run: go test -bench . -benchmem -run '^$' -count=6 -benchtime=100ms ./... > ../bench-base.txt 2>&1

      
        42
        42
         

      
        43
        43
               - name: Run benchmarks

      
        44
        
        -        run: go test -bench . -benchmem -run '^$' -count=10 -benchtime=200x ./... > bench.txt 2>&1

      
        
        44
        +        run: go test -bench . -benchmem -run '^$' -count=6 -benchtime=100ms ./... > bench.txt 2>&1

      
        45
        45
         

      
        46
        46
               - name: Print benchmark numbers

      
        47
        47
                 run: benchstat bench.txt