Skip to content

Conversation

@YangKeao
Copy link
Member

@YangKeao YangKeao commented Dec 4, 2025

What problem does this PR solve?

Issue Number: close #1014

Problem Summary:

What is changed and how it works:

  1. Provide a new binary to analyze the audit log.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

I used the following script to fetch analyze result for upstream and downstream cluster:

#!/bin/bash

# Exit immediately if a command exits with a non-zero status
set -e

# 1. Capture Arguments
BASE_TIME="$1"   # e.g., 2023-10-27T10:00:00Z
DURATION="$2"    # e.g., 1h, 30m, 10s
OFFSET="$3"      # e.g., 1h (The shift for the second command)

# Check if arguments are provided
if [[ -z "$BASE_TIME" || -z "$DURATION" || -z "$OFFSET" ]]; then
    echo "Usage: $0 <RFC3339-Time> <Duration> <Offset>"
    echo "Example: $0 2023-10-27T10:00:00Z 1h 20m"
    exit 1
fi

# 2. Calculate Timestamps
# Note: --iso-8601=seconds ensures strict RFC3339 output

# Command 1: Start = Base, End = Base + Duration
CMD1_START="$BASE_TIME"
CMD1_END=$(date -d "$BASE_TIME + $DURATION" --iso-8601=seconds)

# Command 2: Start = Base + Offset, End = Base + Duration + Offset
# We utilize the fact that date -d can accept cumulative string additions
CMD2_START=$(date -d "$BASE_TIME + $OFFSET" --iso-8601=seconds)
CMD2_END=$(date -d "$BASE_TIME + $DURATION + $OFFSET" --iso-8601=seconds)

# 3. Print Info (Optional, for debugging)
echo "--- Configuration ---"
echo "Upstream Window:   $CMD1_START -> $CMD1_END"
echo "Downstream Window: $CMD2_START -> $CMD2_END"
echo "---------------------"

# 4. Execute Commands

# Command 1 (Upstream)
./auditanalyzer \
  --input=xxxxxxxxx \
  --start-time "$CMD1_START" \
  --end-time "$CMD1_END" \
  --output ./upstream.csv

# Command 2 (Downstream / Replay)
./auditanalyzer \
  --input=xxxxxxxxx \
  --start-time "$CMD2_START" \
  --end-time "$CMD2_END" \
  --output ./downstream.csv

Run it with

./analyze.sh '2025-12-01T12:00:00.000Z' 10min '100day'

The 100day is the offset between upstream and downstream. The 10min is the duration.

Notable changes

  • Has configuration change
  • Has HTTP API interfaces change
  • Has tiproxyctl change
  • Other user behavior changes

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot bot requested a review from djshow832 December 4, 2025 12:10
@ti-chi-bot
Copy link

ti-chi-bot bot commented Dec 4, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign xhebox for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot requested a review from xhebox December 4, 2025 12:10
@ti-chi-bot ti-chi-bot bot added the size/XL label Dec 4, 2025
@codecov-commenter
Copy link

Codecov Report

❌ Patch coverage is 0% with 191 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@d9ef65a). Learn more about missing BASE report.

Files with missing lines Patch % Lines
pkg/sqlreplay/replay/audit_log_analyzer.go 0.00% 74 Missing ⚠️
pkg/sqlreplay/cmd/audit_log_analyzer.go 0.00% 63 Missing ⚠️
cmd/auditloganalyzer/main.go 0.00% 54 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1016   +/-   ##
=======================================
  Coverage        ?   66.93%           
=======================================
  Files           ?      143           
  Lines           ?    14503           
  Branches        ?        0           
=======================================
  Hits            ?     9708           
  Misses          ?     4114           
  Partials        ?      681           
Flag Coverage Δ
unit 66.93% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@YangKeao YangKeao marked this pull request as draft December 5, 2025 11:22
@YangKeao YangKeao changed the title auditloganalyzer: implement a basic audit log analyzer WIP, auditloganalyzer: implement a basic audit log analyzer Dec 5, 2025
@YangKeao
Copy link
Member Author

YangKeao commented Dec 5, 2025

/hold

We might add more functions to it.

@YangKeao YangKeao force-pushed the impl-1014 branch 7 times, most recently from 6e27865 to 49f6092 Compare December 8, 2025 09:11
record := []string{
sql,
fmt.Sprintf("%d", group.ExecutionCount),
group.TotalCostTime.String(),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not output a number? It's easier to order and calculate avg.

w := csv.NewWriter(f)
for sql, group := range result {
record := []string{
sql,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also output the digests so that it's easier to filter a specific SQL.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And stmt type

@YangKeao YangKeao force-pushed the impl-1014 branch 5 times, most recently from 4a19a43 to 2083882 Compare December 9, 2025 15:21
@ti-chi-bot ti-chi-bot bot added size/XXL and removed size/XL labels Dec 9, 2025
Signed-off-by: Yang Keao <yangkeao@chunibyo.icu>
@ti-chi-bot ti-chi-bot bot added the size/XL label Dec 9, 2025
ExecutionCount int
TotalCostTime time.Duration
TotalAffectedRows int64
StmtTypes stmtTypesSet
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why multiple statement types?

if err != nil {
return result, errors.Errorf("parsing cost time failed: %s", costTimeStr)
}
costTime = time.Duration(millis) * (time.Millisecond)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to convert int to Duration and then convert back

connInfo.lastCmd.PreparedStmt = sql
}
connInfo.lastCmd.StmtType = kvs[auditPluginKeyStmtType]
connInfo.lastCmd.kvs = kvs
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kvs is cleared before each loop, so saving kvs in it is wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support comparing execution time and affected rows from audit logs and group by SQL digest

3 participants