flowchart LR
subgraph 准备阶段
direction TB
A[UBT 改造] --> B[环境变量配置]
B --> C[指定高版本Clang]
end
subgraph 构建分析阶段
direction TB
D[CodeChecker log] --> E[生成 compile_commands.json]
E --> F[FixClangDatabase.py 后处理]
F --> G[修复后的编译数据库]
end
subgraph 分析执行阶段
direction TB
H[CodeChecker analyze] --> I[生成 plist 报告]
I --> J[CodeChecker parse]
J --> K[HTML 报告]
end
subgraph 报告处理阶段
direction TB
L[ExtractNotification.py] --> M[p4 annotate / git blame]
M --> N[Markdown 问题报告]
end
准备阶段 --> 构建分析阶段
构建分析阶段 --> 分析执行阶段
I --> 报告处理阶段
阶段
主要工具
输出
准备阶段
修改后的 UBT
支持系统编译器的构建环境
构建分析阶段
CodeChecker log + FixClangDatabase.py
修复后的 compile_commands.json
分析执行阶段
CodeChecker analyze/parse
plist 诊断文件 + HTML 报告
报告处理阶段
ExtractNotification.py
Markdown 格式的问题清单
UBT 改造
为了让 UE 项目能够使用系统安装的高版本 Clang 进行静态分析，我们需要对 UBT 进行一些改造。主要修改集中在 Linux 平台相关的工具链配置。
if(ShouldUseLibcxx()){// libc++ and its abi libLinkCommandString+=" -nodefaultlibs";if(LinuxCommon.GetDefaultBoolValueFromEnvVar("UE_LINUX_USE_SYSTEM_LIBCXX")){LinkCommandString+=" -lc++";LinkCommandString+=" -lc++abi";}else{LinkCommandString+=" -L"+"ThirdParty/Unix/LibCxx/lib/Unix/"+LinkEnvironment.Architecture.LinuxName+"/";LinkCommandString+=" "+"ThirdParty/Unix/LibCxx/lib/Unix/"+LinkEnvironment.Architecture.LinuxName+"/libc++.a";LinkCommandString+=" "+"ThirdParty/Unix/LibCxx/lib/Unix/"+LinkEnvironment.Architecture.LinuxName+"/libc++abi.a";}// ...其他链接库...}
flowchart TB
A["ClangSA Plugin<br />(默认配置)"] --> B["Projects/<br />(项目级配置)"]
B --> C["Projects/Plugins/XXX/<br />(插件级配置)"]
B --> D["Projects/Source/XXX/<br />(模块级配置)"]
style A fill:#f9f,stroke:#333
style B fill:#bbf,stroke:#333
style C fill:#bfb,stroke:#333
style D fill:#bfb,stroke:#333
子目录的配置会继承父目录的配置，并可以追加启用或禁用特定的 checker。
ClangSA.Build.cs
以下是 ClangSA.Build.cs 的核心实现。该模块提供了三个主要功能：
规则配置管理：读取并合并目录层级的 checker 配置
UE 静态分析集成：通过 SetupClangUESA 配置 UE 内置的静态分析
clang-tidy 配置分发：通过 SetupClangTidy 将配置文件复制到中间目录
点击展开 ClangSA.Build.cs 完整代码
```csharp
// Copyright Epic Games, Inc. All Rights Reserved.
using EpicGames.Core;
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using UnrealBuildTool;
public class ClangSA : ModuleRules
{
public class CheckerRules
{
public bool Configured;
public ArrayList EnableCheckers;
public ArrayList DisableCheckers;
public HashSet FinalCheckers;
public CheckerRules() {
Configured = false;
EnableCheckers = new ArrayList();
DisableCheckers = new ArrayList();
FinalCheckers = null;
}
}
static private Dictionary<string, CheckerRules> RulesInDirectory = new Dictionary<string, CheckerRules>();
public ClangSA(ReadOnlyTargetRules Target) : base(Target)
{
Type = ModuleType.External;
}
static public string GetClangSAEnableFilePath(String DirectoryPath)
{
return Path.Combine(DirectoryPath, ".clang-sa.enable");
}
static public string GetClangSADisableFilePath(String DirectoryPath)
{
return Path.Combine(DirectoryPath, ".clang-sa.disable");
}
static public string GetClangSAEnableFilePath(ReadOnlyTargetRules Target) {
return GetClangSAEnableFilePath(Path.Combine(Target.ProjectFile.Directory.FullName, "Plugins", "ClangSA", "Source", "ClangSA"));
}
static public CheckerRules GetCheckRules(String DirectoryPath) {
DirectoryPath = Path.GetFullPath(DirectoryPath);
CheckerRules rules;
lock (RulesInDirectory) {
if (RulesInDirectory.TryGetValue(DirectoryPath, out rules)) {
return rules;
}
}
rules = new CheckerRules();
string enableFilePath = GetClangSAEnableFilePath(DirectoryPath);
if (File.Exists(enableFilePath))
{
rules.Configured = true;
string[] checkers = File.ReadAllLines(enableFilePath);
foreach (string checker in checkers)
{
int comment = checker.IndexOf('#');
string trimmedChecker;
if (comment >= 0)
{
trimmedChecker = checker.Substring(0, comment).Trim();
}
else
{
trimmedChecker = checker.Trim();
}
if (!string.IsNullOrWhiteSpace(trimmedChecker))
{
rules.EnableCheckers.Add(trimmedChecker);
}
}
}
string disableFilePath = GetClangSADisableFilePath(DirectoryPath);
if (File.Exists(disableFilePath))
{
rules.Configured = true;
string[] checkers = File.ReadAllLines(disableFilePath);
foreach (string checker in checkers)
{
int comment = checker.IndexOf('#');
string trimmedChecker;
if (comment >= 0)
{
trimmedChecker = checker.Substring(0, comment).Trim();
}
else
{
trimmedChecker = checker.Trim();
}
if (!string.IsNullOrWhiteSpace(trimmedChecker))
{
rules.DisableCheckers.Add(trimmedChecker);
}
}
}
lock (RulesInDirectory)
{
RulesInDirectory.Add(DirectoryPath, rules);
}
return rules;
}
static public CheckerRules GetClangSACheckers(ReadOnlyTargetRules Target, String DirectoryFullName)
{
String ProjectFullName = Target.ProjectFile.Directory.FullName;
String PluginFullName = Path.Combine(Target.ProjectFile.Directory.FullName, "Plugins", "ClangSA", "Source", "ClangSA");
String ParentFullName = Path.GetDirectoryName(DirectoryFullName);
CheckerRules CurrentRules = GetCheckRules(DirectoryFullName);
if (CurrentRules.FinalCheckers != null) {
return CurrentRules;
}
CurrentRules.FinalCheckers = new HashSet { };
if (!DirectoryFullName.Equals(PluginFullName) && !string.IsNullOrEmpty(ParentFullName) && !ParentFullName.Equals(DirectoryFullName))
{
CheckerRules ParentRules;
if (DirectoryFullName.Equals(ProjectFullName))
{
ParentRules = GetClangSACheckers(Target, PluginFullName);
}
else
{
ParentRules = GetClangSACheckers(Target, ParentFullName);
if (ParentRules.Configured)
{
CurrentRules.Configured = true;
}
}
foreach (var item in ParentRules.FinalCheckers)
{
CurrentRules.FinalCheckers.Add(item);
}
}
foreach (var item in CurrentRules.DisableCheckers)
{
CurrentRules.FinalCheckers.Remove(item.ToString());
}
foreach (var item in CurrentRules.EnableCheckers)
{
CurrentRules.FinalCheckers.Add(item.ToString());
}
return CurrentRules;
}
static public CheckerRules GetClangSACheckers(ReadOnlyTargetRules Target, ModuleRules Module)
{
return GetClangSACheckers(Target, Module.ModuleDirectory);
}
static public void SetupClangUESA(ReadOnlyTargetRules Target, ModuleRules Module)
{
if (Target.StaticAnalyzer != StaticAnalyzer.Clang && Target.StaticAnalyzer != StaticAnalyzer.Default)
{
System.Console.WriteLine($"-------------- Disable UE Static Analysis For {Module.Name} because whole disabled --------------");
return;
}
CheckerRules Rules = GetClangSACheckers(Target, Module);
if (!Rules.Configured)
{
System.Console.WriteLine($"-------------- Disable UE Static Analysis For {Module.Name} because no rules --------------");
return;
}
if (Rules.FinalCheckers == null)
{
System.Console.WriteLine($"-------------- Disable UE Static Analysis For {Module.Name} because no rules --------------");
return;
}
if (Rules.FinalCheckers.Count == 0)
{
System.Console.WriteLine($"-------------- Disable UE Static Analysis For {Module.Name} because no rules --------------");
return;
}
Module.bDisableStaticAnalysis = false;
Module.StaticAnalyzerCheckers = Rules.FinalCheckers;
System.Console.WriteLine($"-------------- Enable UE Static Analysis For {Module.Name} --------------");
System.Console.WriteLine($"-- Checkers: {string.Join(",", Rules.FinalCheckers)}");
}
static private string GetClangTidyConfigureFile(ReadOnlyTargetRules Target, string DirectoryFullName, string FileName)
{
string CheckModuleFullName = Path.Combine(DirectoryFullName, FileName);
if (File.Exists(CheckModuleFullName))
{
return CheckModuleFullName;
}
string ProjectFullName = Target.ProjectFile.Directory.FullName;
string PluginFullName = Path.Combine(Target.ProjectFile.Directory.FullName, "Plugins", "ClangSA", "Source", "ClangSA");
string FallbackFullName = Path.Combine(PluginFullName, FileName);
string ParentFullName = Path.GetDirectoryName(DirectoryFullName);
if (!DirectoryFullName.Equals(PluginFullName) && !DirectoryFullName.Equals(ProjectFullName)
&& !string.IsNullOrEmpty(ParentFullName) && !ParentFullName.Equals(DirectoryFullName))
{
return GetClangTidyConfigureFile(Target, ParentFullName, FileName);
}
return FallbackFullName;
}
static public void SetupClangTidy(ReadOnlyTargetRules Target, ModuleRules Module)
{
if (Module.Type == ModuleType.External)
{
System.Console.WriteLine($"-------------- Disable ClangTidy For {Module.Name} because it's external module --------------");
return;
}
// Target.IntermediateEnvironment
string PlatformIntermediateFolder = GetPlatformIntermediateFolder(Target.Platform, Target.Architectures, false);
string PlatformIntermediateFolderNoArch = GetPlatformIntermediateFolder(Target.Platform, null, false);
string GeneratedCodeDirectory = GetGeneratedCodeDirectory(Target, Module, PlatformIntermediateFolderNoArch).FullName;
string ModuleCodeDirectory = DirectoryReference.Combine(Target.ProjectFile.Directory, PlatformIntermediateFolder,
GetTargetIntermediateFolderName(Target.Name, Target.IntermediateEnvironment),
Target.Configuration.ToString(), Module.ShortName ?? Module.Name).FullName;
// string ModuleDirectory = Module.ModuleDirectory;
string[] CopyFiles = new string[] {
".codechecker.clang-tidy.args",
".codechecker.skipfile",
".codechecker.yaml",
".clang-tidy",
};
string[] CopyToDirs = new string[] {
GeneratedCodeDirectory,
ModuleCodeDirectory,
};
System.Console.WriteLine($"-------------- Setup ClangTidy For {Module.Name} --------------");
foreach (string CopyFile in CopyFiles)
{
string ConfigureFilePath = GetClangTidyConfigureFile(Target, Module.ModuleDirectory, CopyFile);
foreach (string CopyToDir in CopyToDirs)
{
string TargetFilePath = Path.Combine(CopyToDir, CopyFile);
System.Console.WriteLine($"Setup ClangTidy: Copy {ConfigureFilePath} to {TargetFilePath}");
if (!Directory.Exists(CopyToDir))
{
Directory.CreateDirectory(CopyToDir);
}
File.Copy(ConfigureFilePath, TargetFilePath, true);
}
}
}
static public void SetupClangSA(ReadOnlyTargetRules Target, ModuleRules Module)
{
SetupClangUESA(Target, Module);
SetupClangTidy(Target, Module);
}
// UnrealBuildTool.UEBuildTarget.GetTargetIntermediateFolderName 是引擎内部接口，类不是Public的
// 这里复制了一份出来，保持代码逻辑一致
public static string GetTargetIntermediateFolderName(string TargetName, UnrealIntermediateEnvironment IntermediateEnvironment)
{
string TargetFolderName = TargetName;
switch (IntermediateEnvironment)
{
case UnrealIntermediateEnvironment.IWYU:
TargetFolderName += "IWYU";
break;
case UnrealIntermediateEnvironment.NonUnity:
TargetFolderName += "NU";
break;
case UnrealIntermediateEnvironment.Analyze:
TargetFolderName += "SA";
break;
}
return TargetFolderName;
}
// UnrealBuildTool.UEBuildTarget.GetPlatformIntermediateFolder 是引擎内部接口，类不是Public的
// 这里复制了一份出来，保持代码逻辑一致
public static string GetPlatformIntermediateFolder(UnrealTargetPlatform Platform, UnrealArchitectures Architectures, bool External)
{
// now that we have the platform, we can set the intermediate path to include the platform/architecture name
string FolderPath = Path.Combine("Intermediate", External ? "External" : String.Empty, "Build", Platform.ToString());
if (Architectures != null)
{
FolderPath = Path.Combine(FolderPath, UnrealArchitectureConfig.ForPlatform(Platform).GetFolderNameForArchitectures(Architectures));
}
return FolderPath;
}
static public DirectoryReference GetGeneratedCodeDirectory(ReadOnlyTargetRules Target, ModuleRules Module, string PlatformIntermediateFolderNoArch)
{
DirectoryReference GeneratedCodeDirectory = null;
// Get the base directory
// if (Module.Context)
// {
// GeneratedCodeDirectory = Module.Context.DefaultOutputBaseDir;
// }
// else
// {
GeneratedCodeDirectory = Target.ProjectFile.Directory;
// }
// Get the subfolder containing generated code - we don't need architeceture information since these are shared between all arches for a platform
GeneratedCodeDirectory = DirectoryReference.Combine(GeneratedCodeDirectory, PlatformIntermediateFolderNoArch, GetTargetIntermediateFolderName(Target.Name, Target.IntermediateEnvironment), "Inc");
// Append the binaries subfolder, if present. We rely on this to ensure that build products can be filtered correctly.
if (Module.BinariesSubFolder != null)
{
GeneratedCodeDirectory = DirectoryReference.Combine(GeneratedCodeDirectory, Module.BinariesSubFolder);
}
// Finally, append the module name (using the ShortName if it has been set)
GeneratedCodeDirectory = DirectoryReference.Combine(GeneratedCodeDirectory, Module.ShortName ?? Module.Name);
return GeneratedCodeDirectory;
}
}
```
</details>
#### 在模块中集成静态分析
在你的模块的 `.Build.cs` 文件中，调用 `ClangSA.SetupClangSA` 即可启用静态分析：
```csharp
public class LyraGame : ModuleRules
{
public LyraGame(ReadOnlyTargetRules Target) : base(Target)
{
// ... 其他配置 ...
// 启用静态分析
ClangSA.SetupClangSA(Target, this);
}
}
```
### FixClangDatabase.py：编译数据库后处理
UBT 生成的 `compile_commands.json` 存在以下问题，需要后处理：
1. **响应文件（@file）未展开**：UBT 使用 `.rsp` 文件存储编译参数，需要展开为完整参数
2. **相对路径问题**：部分路径是相对路径，需要转换为绝对路径
3. **缺少 PCH 包含**：需要手动添加共享 PCH 的 include
4. **文件过滤**：只保留需要分析的源文件
点击展开 FixClangDatabase.py 完整代码
```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import codecs
import re
import argparse
import os
import sys
import json
from typing import List, Tuple, Set, Optional, Dict
__VERSION__ = "1.0.0"
CLANG_DATABASE_IGNORE_OPTIONS = set(["-c"])
def try_read_file(file_path):
_err = None
for try_encoding in ["utf-8", "utf-8-sig", "GB18030"]:
try:
ret = codecs.open(file_path, "r", encoding=try_encoding)
return ret
except Exception as e:
if _err is None:
_err = e
if not os.path.exists(file_path):
break
raise _err
class ClangDatabaseProcessor:
def __init__(
self,
engine_source: str,
input_file: str,
output_file: str,
include_path: List[str],
exclude_path: List[str],
pch: Optional[str],
verbose: bool,
):
self.engine_source = engine_source
self.input_file = input_file
self.output_file = output_file
self.include_path: List[re.Pattern] = [
re.compile(p, re.IGNORECASE) for p in include_path
]
self.exclude_path: List[re.Pattern] = [
re.compile(p, re.IGNORECASE) for p in exclude_path
]
self.expanded_cache: Dict[str, List[str]] = dict()
self.pch = pch
self.verbose = verbose
def process(self):
# Implement the processing logic here
result = []
json_data = self._parse_json()
for item in json_data:
if not self._should_include(item):
continue
if self.verbose:
print(f"Processing command for file: {item['file']}")
result.append(self._resolve_item(item))
json.dump(result, self.output_file, indent=2)
def _parse_flag_with_file(self, token, previous_token, flag):
if previous_token == flag:
file_path = token
prefix = ""
suffix = ""
if (file_path.startswith('"') and file_path.endswith('"')) or (
file_path.startswith("'") and file_path.endswith("'")
):
file_path = file_path[1:-1]
prefix = token[:1]
suffix = token[:1]
return (file_path, prefix, suffix)
if not token.startswith(flag):
return (None, None, None)
flag_len = len(flag)
prefix = token[:flag_len]
suffix = ""
file_path = token[flag_len:]
if (file_path.startswith('"') and file_path.endswith('"')) or (
file_path.startswith("'") and file_path.endswith("'")
):
file_path = file_path[1:-1]
prefix = token[: (flag_len + 1)]
suffix = token[flag_len : (flag_len + 1)]
return (file_path, prefix, suffix)
def _resolve_token(self, token, previous_token):
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "@"
)
if file_path:
return self._resolve_response(file_path)
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "-I"
)
if file_path:
return self._resolve_flag_with_file_path(file_path, prefix, suffix)
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "-isystem"
)
if file_path:
return self._resolve_flag_with_file_path(file_path, prefix, suffix)
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "-MF"
)
if file_path:
return self._resolve_flag_with_file_path(file_path, prefix, suffix)
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "-o"
)
if file_path:
return self._resolve_flag_with_file_path(file_path, prefix, suffix)
return [token]
def _resolve_response(self, file_path):
if not os.path.isabs(file_path):
file_path = os.path.join(self.engine_source, file_path)
if file_path in self.expanded_cache:
return self.expanded_cache[file_path]
expanded_paths = []
try:
with try_read_file(file_path) as rsp_file:
previous_token = ""
for line in rsp_file:
line = line.strip()
if not line:
continue
for token in self._split_tokens(line):
if token in CLANG_DATABASE_IGNORE_OPTIONS:
continue
expanded_paths.extend(
self._resolve_token(token, previous_token)
)
previous_token = token
self.expanded_cache[file_path] = expanded_paths
if self.verbose:
print(
f" - Expand response file '{file_path}' to {len(expanded_paths)} args"
)
except Exception as e:
print(
f" - Error reading response file '{file_path}': {e}", file=sys.stderr
)
return [f'@"{file_path}"']
return expanded_paths
def _resolve_flag_with_file_path(self, file_path, prefix, suffix):
if not os.path.isabs(file_path):
file_path = os.path.realpath(os.path.join(self.engine_source, file_path))
return [f"{prefix}{file_path}{suffix}"]
def _next_token(self, input, start_idx):
sz = len(input)
while start_idx < sz and input[start_idx].isspace():
start_idx += 1
if start_idx >= sz:
return (None, sz)
string_quote = None
end_idx = start_idx
while end_idx < sz and not input[end_idx].isspace():
if input[end_idx] == '"' or input[end_idx] == "'":
string_quote = input[end_idx]
end_idx += 1
ignore_next = False
while end_idx < sz and (ignore_next or input[end_idx] != string_quote):
ignore_next = input[end_idx] == "\\"
end_idx += 1
if end_idx < sz:
end_idx += 1
else:
end_idx += 1
return (input[start_idx:end_idx], end_idx)
def _split_tokens(self, input):
# Split the input into tokens based on whitespace
idx = 0
sz = len(input)
ret = []
while idx < sz:
token, end_idx = self._next_token(input, idx)
if token is not None:
ret.append(token)
idx = end_idx
return ret
def _resolve_item(self, item):
command_args = []
checked_pch = False
# Resolve response files
tokens = self._split_tokens(item["command"])
previous_token = ""
for token in tokens:
if token in CLANG_DATABASE_IGNORE_OPTIONS:
continue
command_args.extend(self._resolve_token(token, previous_token))
previous_token = token
if not checked_pch:
checked_pch = True
if self.pch and self.pch.strip():
command_args.extend(["-include", f'"{self.pch}"'])
if "-Wno-unused-command-line-argument" not in command_args:
command_args.append("-Wno-unused-command-line-argument")
if self.verbose:
print(
f" * Resolve command for file: {item['file']} from {len(tokens)} args to {len(command_args)} args"
)
item["command"] = " ".join(command_args)
# Patch file
file_path = item["file"]
if file_path.startswith("@"):
file_path = file_path[1:]
if file_path.endswith(".d.rsp"):
file_path = file_path[: -len(".d.rsp")]
item["file"] = file_path
return item
def _should_include(self, item):
if "file" not in item or "command" not in item:
return False
file_param = item["file"]
if self.include_path:
if not any(p.search(file_param) for p in self.include_path):
return False
if self.exclude_path and any(p.search(file_param) for p in self.exclude_path):
return False
return True
def _parse_json(self):
json_data = json.load(self.input_file)
# Process the JSON data as needed
return json_data
def main():
global __VERSION__
parser = argparse.ArgumentParser(usage="%(prog)s [options...]")
parser.add_argument("REMAINDER", nargs=argparse.REMAINDER, help="task names")
parser.add_argument(
"-v",
"--version",
action="store_true",
help="show version and exit",
dest="version",
default=False,
)
parser.add_argument(
"-V",
"--verbose",
action="store_true",
help="show verbose",
dest="verbose",
default=False,
)
parser.add_argument(
"-i",
"--input",
action="store",
help="set input clang database build file (default: compile_commands.json)",
dest="input",
default="compile_commands.json",
)
parser.add_argument(
"-o",
"--output",
action="store",
help="set output clang database build file (default: compile_commands.json)",
dest="output",
default=None,
)
parser.add_argument(
"--include-path",
action="append",
help="only keep files match include path(regex)",
dest="include_path",
default=[],
)
parser.add_argument(
"--exclude-path",
action="append",
help="only keep files match exclude path(regex)",
dest="exclude_path",
default=[],
)
parser.add_argument(
"--pch",
action="store",
help="set the pch file to include",
dest="pch",
default=None,
)
if os.path.exists(
os.path.join(os.getcwd(), "Engine", "Source", "UnrealGame.Target.cs")
):
find_default_engine_source = os.path.join(os.getcwd(), "Engine", "Source")
elif os.path.exists(
os.path.join(os.getcwd(), "..", "Engine", "Source", "UnrealGame.Target.cs")
):
find_default_engine_source = os.path.realpath(
os.path.join(os.getcwd(), "..", "Engine", "Source")
)
elif os.path.exists(
os.path.join(
os.getcwd(), "..", "..", "Engine", "Source", "UnrealGame.Target.cs"
)
):
find_default_engine_source = os.path.realpath(
os.path.join(os.getcwd(), "..", "..", "Engine", "Source")
)
else:
find_default_engine_source = os.getcwd()
parser.add_argument(
"-e",
"--engine-source",
action="store",
help="set path of Engine/Source, it will be used to convert relative paths",
dest="engine_source",
default=find_default_engine_source,
)
options = parser.parse_args()
if options.version:
print(__VERSION__)
return 0
if options.input.strip() == "-":
input_file = sys.stdin
else:
input_file = try_read_file(options.input)
output_file_path = options.output
if output_file_path is None:
(input_base, input_ext) = os.path.splitext(options.input)
if input_ext:
output_file_path = input_base + ".fixed" + input_ext
else:
output_file_path = input_base + ".fixed"
processor = ClangDatabaseProcessor(
engine_source=options.engine_source,
input_file=input_file,
output_file=codecs.open(output_file_path, "w", encoding="utf-8"),
include_path=options.include_path,
exclude_path=options.exclude_path,
pch=options.pch,
verbose=options.verbose,
)
print(f"[FixClangDatabase]: Using engine source at {options.engine_source}")
print(f"[FixClangDatabase]: Process {options.input} -> {output_file_path}")
processor.process()
return 0
if __name__ == "__main__":
exit(main())
```
#### 命令行参数说明
| 参数 | 说明 | 默认值 |
|------|------|--------|
| `-i, --input` | 输入的编译数据库文件 | `compile_commands.json` |
| `-o, --output` | 输出的编译数据库文件 | `.fixed.json` |
| `--include-path` | 只保留匹配的文件路径（正则） | 无 |
| `--exclude-path` | 排除匹配的文件路径（正则） | 无 |
| `--pch` | 要包含的 PCH 文件路径 | 无 |
| `-e, --engine-source` | 引擎源码路径，用于转换相对路径 | 自动检测 |
### ExtractNotification.py：问题报告生成
静态分析会生成大量诊断信息，但并非所有问题都需要立即处理。`ExtractNotification.py` 用于提取需要重点关注的严重问题，并通过 VCS（p4/git）查找每行代码的最后提交人。
#### 工作流程
```mermaid
flowchart LR
A[plist 诊断文件] --> B[解析 XML/plist]
B --> C[过滤重要 checker]
C --> D[定位问题文件和行号]
D --> E[查询 VCS blame]
E --> F[生成 Markdown 报告]
```
#### 支持的重要 Checker
以下 checker 的问题会被提取到通知报告中：
| Checker | 说明 |
|---------|------|
| `clang-analyzer-core.CallAndMessage` | 空指针调用（过滤 `object pointer is null`） |
| `clang-analyzer-core.NonNullParamChecker` | 非空参数传入空值 |
| `clang-analyzer-core.NullDereference` | 空指针解引用 |
| `clang-diagnostic-shorten-64-to-32` | 64 位到 32 位截断 |
| `bugprone-use-after-move` | move 后使用 |
| `bugprone-swapped-arguments` | 参数顺序错误 |
| `bugprone-suspicious-enum-usage` | 可疑的枚举使用 |
| `clang-diagnostic-error` | 编译错误 |
点击展开 ExtractNotification.py 完整代码
```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import codecs
import re
import argparse
import os
import sys
import glob
import xml.dom.minidom
import subprocess
from typing import List, Any, Optional
__VERSION__ = "1.0.0"
class ClangSANotificationFilter:
def __init__(self, checker_name: str, message_filter: List[Any] = []):
self.checker_name = checker_name
self.checker_name_lowercase = checker_name.lower()
self.message_filter = message_filter
CLANG_SA_NOTIFICATION_CHECKERS = dict()
CLANG_SA_NOTIFICATION_CHECKERS_LOWERCASE = dict()
for checker in [
ClangSANotificationFilter(
"clang-analyzer-core.CallAndMessage",
[re.compile(r"object\s+pointer\s+is\s+null", re.IGNORECASE)],
),
ClangSANotificationFilter("clang-analyzer-core.NonNullParamChecker"),
ClangSANotificationFilter("clang-analyzer-core.NullDereference"),
ClangSANotificationFilter("clang-diagnostic-shorten-64-to-32"),
ClangSANotificationFilter("bugprone-use-after-move"),
ClangSANotificationFilter("bugprone-swapped-arguments"),
ClangSANotificationFilter("bugprone-suspicious-enum-usage"),
ClangSANotificationFilter("clang-diagnostic-error"),
]:
CLANG_SA_NOTIFICATION_CHECKERS[checker.checker_name] = checker
CLANG_SA_NOTIFICATION_CHECKERS_LOWERCASE[checker.checker_name_lowercase] = checker
CLANG_SA_DEFAULT_IGNORE_RULES = [
# 忽略引擎核心模块的问题（这些通常不是项目代码）
re.compile(
r"Engine/Source/Runtime/(Core|CoreUObject|Engine|SlateCore)", re.IGNORECASE
),
# 忽略第三方库
re.compile(r"Projects/Plugins/UnLua/Source/ThirdParty", re.IGNORECASE),
re.compile(r"deps/third_party/install", re.IGNORECASE),
]
# 匹配 C/C++ 源文件的正则表达式
CLANG_SA_SOURCE_FILE_PATTERN = re.compile(r"^.*\.(cc|cxx|cpp|c)$", re.IGNORECASE)
class ClangSANotificationFile:
"""表示一个需要通知的问题文件位置"""
def __init__(self, file_path: str, column: int, line: int):
self.file_path = file_path.replace("\\", "/") # 统一使用 Unix 风格路径
self.column = column
self.line = line
self.notifications = []
def try_read_file(file_path):
"""尝试以多种编码读取文件，自动检测编码"""
_err = None
if not os.path.isabs(file_path):
file_path = os.path.join(file_path)
for try_encoding in ["utf-8", "utf-8-sig", "GB18030"]:
try:
ret = codecs.open(file_path, "r", encoding=try_encoding)
return ret
except Exception as e:
if _err is None:
_err = e
if not os.path.exists(file_path):
break
raise _err
def try_decode_buffer(buffer):
"""尝试以多种编码解码字节buffer"""
for try_encoding in ["utf-8", "utf-8-sig", "GB18030"]:
try:
return buffer.decode(try_encoding)
except Exception:
continue
return buffer.decode(sys.getfilesystemencoding(), errors="ignore")
def _check_notification_file(file_path: str, ignore_path: List[Any]):
"""检查文件路径是否应该被忽略"""
for rule in ignore_path:
if re.search(rule, file_path):
return False
return True
def _find_notification_file(
data: dict, ignore_path: List[Any]
) -> Optional[ClangSANotificationFile]:
"""
从诊断数据中查找问题所在的源文件位置
优先返回 .cpp/.c 源文件，如果没有则返回头文件
"""
fallback_header_file = None
# 首先检查直接的 location 字段
if "location" in data:
location = data["location"]
if (
isinstance(location, dict)
and "file" in location
and "line" in location
and "col" in location
):
file_path = location["file"]
line = location["line"]
column = location["col"]
if _check_notification_file(file_path, ignore_path):
if CLANG_SA_SOURCE_FILE_PATTERN.match(file_path):
return ClangSANotificationFile(file_path, column, line)
else:
fallback_header_file = ClangSANotificationFile(
file_path, column, line
)
if "path" in data:
path_nodes = data["path"]
# 倒序找文件，优先使用源文件
for path_node in reversed(path_nodes):
if "location" in path_node:
location = path_node["location"]
if (
isinstance(location, dict)
and "file" in location
and "line" in location
and "col" in location
):
file_path = location["file"]
line = location["line"]
column = location["col"]
if _check_notification_file(file_path, ignore_path):
if CLANG_SA_SOURCE_FILE_PATTERN.match(file_path):
return ClangSANotificationFile(file_path, column, line)
elif fallback_header_file is None:
fallback_header_file = ClangSANotificationFile(
file_path, column, line
)
return fallback_header_file
class ClangSANotificationInfo:
"""封装单个诊断通知的完整信息"""
def __init__(self, data: dict, ignore_path: List[Any]):
self.data = data
self.check_name = data.get("check_name", "")
self.description = data.get("description", "")
# 对编译错误添加额外说明
if self.check_name == "clang-diagnostic-error":
self.description += "（编译失败，通常是由于头文件依赖不严谨）"
self.category = data.get("category", "")
# issue_hash 用于去重和生成报告链接
self.issue_hash = data.get("issue_hash_content_of_line_in_context", "")
self.plist_file_path = data.get("plist_file_path", "")
# 查找问题所在的源文件
self.issue_file = _find_notification_file(data, ignore_path)
self.last_committer = None # 稍后通过 VCS 查询填充
self.location = None
if "location" in data:
location = data["location"]
if (
isinstance(location, dict)
and "file" in location
and "line" in location
and "col" in location
):
self.location = ClangSANotificationFile(
location["file"], location["col"], location["line"]
)
def is_valid(self):
"""检查通知信息是否有效（必须有文件位置信息）"""
return self.issue_file is not None and self.location is not None
class VCS_MODE:
"""版本控制系统模式枚举"""
NONE = None
GIT = "git"
P4 = "p4"
def build_array_element(node: xml.dom.minidom.Element):
"""解析 plist 的 array 元素，返回 Python 列表"""
ret = []
for child in node.childNodes:
if child.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
child_data = build_object_element(child)
if child_data is not None:
ret.append(child_data)
return ret
def build_dict_element(node: xml.dom.minidom.Element):
"""解析 plist 的 dict 元素，返回 Python 字典"""
ret = {}
current_key = None
for child in node.childNodes:
if child.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
if child.nodeName == "key":
# 提取 key 的文本内容
current_key = ""
for text_node in child.childNodes:
if text_node.nodeType == xml.dom.minidom.Node.TEXT_NODE:
current_key += text_node.nodeValue
else:
# 当前节点是 key 对应的值
if current_key is not None:
value = build_object_element(child)
if value is not None:
ret[current_key] = value
current_key = None
return ret
def get_text_content(node: xml.dom.minidom.Element):
"""提取 XML 元素的文本内容"""
text = ""
for child in node.childNodes:
if child.nodeType == xml.dom.minidom.Node.TEXT_NODE:
text += child.nodeValue
return text
def build_object_element(node: xml.dom.minidom.Element):
"""
根据 plist DTD 规范解析各种类型的元素
@see http://www.apple.com/DTDs/PropertyList-1.0.dtd
"""
tag_name = node.nodeName
if tag_name == "array":
return build_array_element(node)
elif tag_name == "dict":
return build_dict_element(node)
elif tag_name == "string":
return get_text_content(node)
elif tag_name == "integer":
text = get_text_content(node).strip()
try:
return int(text)
except ValueError:
return text
elif tag_name == "real":
text = get_text_content(node).strip()
try:
return float(text)
except ValueError:
return text
elif tag_name == "true":
return True
elif tag_name == "false":
return False
elif tag_name == "data":
# Base64 encoded data
return get_text_content(node).strip()
elif tag_name == "date":
# ISO 8601 date string
return get_text_content(node).strip()
return None
def rebuild_diagnostic(diagnostic: dict | list, file_indexes: List[str]) -> dict | list:
"""
重建诊断数据结构，将文件索引替换为实际的文件路径
plist 中的 file 字段存储的是索引号，需要通过 file_indexes 映射为路径
"""
if isinstance(diagnostic, list):
return [rebuild_diagnostic(item, file_indexes) for item in diagnostic]
ret = {}
for k in diagnostic:
v = diagnostic[k]
if k == "file" and isinstance(v, int):
# 将文件索引映射为实际的文件路径
ret[k] = file_indexes[v]
elif isinstance(v, dict):
# 递归处理嵌套的字典
ret[k] = rebuild_diagnostic(v, file_indexes)
elif isinstance(v, list):
# 递归处理嵌套的列表
ret[k] = [rebuild_diagnostic(item, file_indexes) for item in v]
else:
ret[k] = v
return ret
def build_plist_element(
node: xml.dom.minidom.Element, shared_index: dict
) -> List[dict]:
"""
解析 plist 根元素，提取所有诊断信息
shared_index 用于跨文件去重（相同 checker + issue_hash 的问题只保留一个）
"""
ret = []
for child in node.childNodes:
if child.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
plist_data = build_object_element(child)
# plist 必须包含 files（文件路径列表）和 diagnostics（诊断列表）
if (
not plist_data
or "files" not in plist_data
or "diagnostics" not in plist_data
):
continue
file_indexes = plist_data["files"] # 文件路径索引表
diagnostics = plist_data["diagnostics"]
for diagnostic in diagnostics:
# 必须有 checker 名称和 issue hash 才能处理
if (
"check_name" not in diagnostic
or "issue_hash_content_of_line_in_context" not in diagnostic
):
continue
check_name = diagnostic["check_name"]
issue_hash = diagnostic["issue_hash_content_of_line_in_context"]
# 使用 checker 名称 + issue hash 作为去重 key
if check_name not in shared_index:
shared_index[check_name] = dict()
check_name_map = shared_index[check_name]
if issue_hash in check_name_map:
continue # 跳过重复的问题
# 重建诊断数据，将文件索引替换为路径
rebuild_dict = rebuild_diagnostic(diagnostic, file_indexes)
if rebuild_dict:
check_name_map[issue_hash] = rebuild_dict
ret.append(rebuild_dict)
return ret
def build_plist_data(file_path: str, shared_index: dict) -> List[dict]:
"""解析单个 plist 文件，返回诊断信息列表"""
doc = xml.dom.minidom.parseString(try_read_file(file_path).read())
ret = []
for node in doc.childNodes:
if node.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
if node.nodeName == "plist":
res = build_plist_element(node, shared_index)
for item in res:
item["plist_file_path"] = file_path
ret.extend(res)
else:
child = build_object_element(node)
if child:
child["plist_file_path"] = file_path
ret.append(child)
return ret
def filter_notification_diagnostics(diagnostics: List[dict]) -> List[dict]:
"""
过滤诊断信息，只保留需要通知的重要问题
根据 CLANG_SA_NOTIFICATION_CHECKERS 定义的 checker 列表过滤
"""
ret = []
for diagnostic in diagnostics:
if "check_name" not in diagnostic:
continue
check_name = diagnostic["check_name"].lower()
# 检查是否是需要通知的 checker
if check_name not in CLANG_SA_NOTIFICATION_CHECKERS_LOWERCASE:
continue
selected = False
filter = CLANG_SA_NOTIFICATION_CHECKERS_LOWERCASE[check_name]
# 如果 checker 有消息过滤规则，还需要匹配消息内容
if filter.message_filter:
selected = False
description = diagnostic.get("description", "")
for pattern in filter.message_filter:
if re.search(pattern, description):
selected = True
break
else:
selected = True
if selected:
ret.append(diagnostic)
return ret
def build_notification_list(
notification_diagnostics: List[dict], ignore_path: List[Any]
) -> List[ClangSANotificationInfo]:
"""将过滤后的诊断信息转换为通知对象列表"""
ret = []
for diagnostic in notification_diagnostics:
notification_info = ClangSANotificationInfo(diagnostic, ignore_path)
if notification_info.is_valid():
ret.append(notification_info)
return ret
def patch_statistics_body_dom(
node: xml.dom.minidom.Node, url_prefix: str
) -> xml.dom.minidom.Node:
"""修补统计 HTML 中的链接，添加 URL 前缀"""
if node.nodeName == "a":
href = node.getAttribute("href")
# 跳过绝对 URL
if href.startswith("http://") or href.startswith("https://"):
return node
if not href.startswith("/"):
href = "/" + href
# 添加 URL 前缀
if url_prefix:
if not url_prefix.endswith("/"):
node.setAttribute("href", url_prefix + href)
else:
node.setAttribute("href", url_prefix + href[1:])
# 递归处理子节点
for child_node in node.childNodes:
if child_node.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
patch_statistics_body_dom(child_node, url_prefix)
return node
def scan_statistics_body_dom(
node: xml.dom.minidom.Node, url_prefix: str
) -> Optional[str]:
"""扫描并修补统计页面的 body 内容"""
res = []
for child_node in node.childNodes:
if child_node.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
res.append(child_node.toxml())
continue
res.append(patch_statistics_body_dom(child_node, url_prefix).toxml())
return "".join(res) if res else None
def pick_statistics_html_file(
statistics: Optional[str], url_prefix: str
) -> Optional[str]:
"""
从 CodeChecker 生成的统计 HTML 文件中提取 body 内容
用于嵌入到 Markdown 报告中
"""
if not statistics or not os.path.exists(statistics):
return None
with try_read_file(statistics) as f:
html_content = f.read()
# 提取 ... 之间的内容
body_begin = html_content.find("")
body_end = html_content.find("")
body_content = (
html_content[body_begin + 6 : body_end]
if body_begin != -1 and body_end != -1
else ""
)
if not body_content:
return None
# 去除换行，便于解析
body_content = "".join([x.strip() for x in body_content.splitlines()])
doc = xml.dom.minidom.parseString(f"{body_content}")
for node in doc.childNodes:
if node.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
res = scan_statistics_body_dom(node, url_prefix)
if res:
return res
return None
# ============ VCS 相关的缓存和正则表达式 ============
CLANG_SA_LAST_COMMITTER_CACHE = dict() # 最终结果缓存: file:line -> committer
CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE = dict() # P4 文件级缓存
CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE = dict() # Git 行级缓存
CLANG_SA_LAST_COMMITTER_GIT_FILE_NOT_FOUND = set() # 找不到的文件集合
CLANG_SA_GIT_IGNORE_COMMITTERS = set(["europa_server"]) # 忽略的提交者（如自动化账号）
# Git blame 输出解析用的正则表达式
CLANG_SA_GIT_SHA_RULE = re.compile("^[0-9a-fA-F]+")
CLANG_SA_GIT_AUTHOR_RULE = re.compile("^author\s+(?P.+)$")
CLANG_SA_GIT_COMMITTER_RULE = re.compile("^committer\s+(?P.+)$")
CLANG_SA_GIT_SUMMARY_RULE = re.compile("^summary\s+(?P.+)$")
def parse_p4_annotate_line(line: str) -> Optional[str]:
"""
解析 p4 annotate 输出的单行
格式: changelist: username content
"""
pattern = r"^(\d+):\s+(\S+)\s+(.*)$"
match = re.match(pattern, line)
if match:
return {
"changelist": int(match.group(1)),
"username": match.group(2),
"content": match.group(3),
}
return None
def parse_git_blame_lines(lines: List[str]) -> Optional[dict]:
"""
解析 git blame --line-porcelain 的输出
提取 commit hash、author 和 summary 信息
"""
if not lines:
return None
first_line = lines[0]
sha_hash = CLANG_SA_GIT_SHA_RULE.match(first_line)
if not sha_hash:
return None
author = None
content = None
committer = None
# 遍历各行，提取需要的字段
for line in lines:
if author is not None and content is not None and committer is not None:
break
if author is None:
author_match = CLANG_SA_GIT_AUTHOR_RULE.match(line)
if author_match:
author = author_match.group("AUTHOR")
continue
if content is None:
content_match = CLANG_SA_GIT_SUMMARY_RULE.match(line)
if content_match:
content = content_match.group("SUMMARY")
continue
if committer is None:
committer_match = CLANG_SA_GIT_COMMITTER_RULE.match(line)
if committer_match:
committer = committer_match.group("COMMITTER")
continue
# 如果没有 author，使用 committer
if author is None:
author = committer
if sha_hash and author and content:
return {
"commit": sha_hash,
"username": author,
"content": content,
}
return None
def find_last_committer_p4_file_cache(
file: ClangSANotificationFile, rel_file_path: str, rewrite_vcs_root: Optional[str]
) -> List[str]:
"""
通过 p4 annotate 获取文件每行的最后提交信息
结果会缓存以避免重复查询
"""
global CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE
if file.file_path in CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE:
return CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE[file.file_path]
try:
# 执行 p4 annotate 命令，使用 -c 和 -u 参数
if rel_file_path and rewrite_vcs_root:
cmd = ["p4", "annotate", "-c", "-u", f"{rewrite_vcs_root}/{rel_file_path}"]
else:
cmd = ["p4", "annotate", "-c", "-u", file.file_path]
cmd_str = '"' + '" "'.join(cmd) + '"'
print(f"Running command: {cmd_str}")
result = subprocess.run(
cmd, capture_output=True, text=True, timeout=30, encoding="utf-8"
)
if result.returncode != 0:
print(f"P4 annotate failed: {result.stderr}", file=sys.stderr)
else:
# 解析每行的 annotate 信息
ret = [parse_p4_annotate_line(x) for x in result.stdout.splitlines()]
CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE[file.file_path] = ret
return ret
except subprocess.TimeoutExpired:
print("P4 annotate command timeout", file=sys.stderr)
except Exception as e:
print(f"Error executing p4 annotate: {e}", file=sys.stderr)
# 失败时缓存空列表，避免重复查询
CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE[file.file_path] = []
return []
def find_last_committer_git_file_cache(
file: ClangSANotificationFile, rel_file_path: str, rewrite_vcs_root: Optional[str]
) -> List[str]:
"""
通过 git blame 获取指定行的最后提交信息
支持重试机制：如果当前提交者在忽略列表中，会尝试查询更早的提交
"""
global CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE
global CLANG_SA_LAST_COMMITTER_GIT_FILE_NOT_FOUND
key = f"{file.file_path}:{file.line}"
# 检查缓存
if key in CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE:
return CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE[key]
if file.file_path in CLANG_SA_LAST_COMMITTER_GIT_FILE_NOT_FOUND:
return None
try:
# 最多回溯 10 个版本，跳过自动化账号的提交
for retry_revision_depth in range(10):
if retry_revision_depth == 0:
REVISION = "HEAD"
else:
REVISION = f"HEAD~{retry_revision_depth}"
# 构建 git blame 命令
if rel_file_path and rewrite_vcs_root:
cmd = [
"git",
"blame",
"-L",
f"{file.line},{file.line}", # 只查询指定行
"--line-porcelain", # 详细输出格式
REVISION,
"--",
f"{rewrite_vcs_root}/{rel_file_path}",
]
else:
cmd = [
"git",
"blame",
"-L",
f"{file.line},{file.line}",
"-u",
"--line-porcelain",
REVISION,
"--",
file.file_path,
]
cmd_str = '"' + '" "'.join(cmd) + '"'
print(f"Running command: {cmd_str}")
result = subprocess.run(
cmd, capture_output=True, text=True, timeout=30, encoding="utf-8"
)
if result.returncode != 0:
# 文件不存在，记录下来避免重复查询
if result.stderr.find("no such path") >= 0:
CLANG_SA_LAST_COMMITTER_GIT_FILE_NOT_FOUND.add(file.file_path)
return None
print(f"git blame failed: {result.stderr}", file=sys.stderr)
else:
ret = parse_git_blame_lines(result.stdout.splitlines())
# 如果提交者在忽略列表中，继续查找更早的提交
if ret and ret["username"] in CLANG_SA_GIT_IGNORE_COMMITTERS:
continue
CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE[key] = ret
return ret
except subprocess.TimeoutExpired:
print("git blame command timeout", file=sys.stderr)
except Exception as e:
print(f"Error executing git blame: {e}", file=sys.stderr)
CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE[key] = None
return None
def find_last_committer(
file: ClangSANotificationFile,
rel_file_path: str,
vcs_mode: VCS_MODE,
rewrite_vcs_root: Optional[str],
) -> Optional[str]:
"""
查找指定文件行的最后提交人
根据 vcs_mode 选择使用 p4 或 git
"""
global CLANG_SA_LAST_COMMITTER_CACHE
key = f"{file.file_path}:{file.line}"
# 检查结果缓存
if key in CLANG_SA_LAST_COMMITTER_CACHE:
return CLANG_SA_LAST_COMMITTER_CACHE[key]
if vcs_mode == VCS_MODE.P4:
p4_cache = find_last_committer_p4_file_cache(
file, rel_file_path, rewrite_vcs_root
)
if file.line > len(p4_cache):
CLANG_SA_LAST_COMMITTER_CACHE[key] = ""
value = ""
else:
p4_annotate = p4_cache[file.line - 1]
if p4_annotate:
value = p4_annotate["username"]
else:
value = ""
elif vcs_mode == VCS_MODE.GIT:
# 使用 git blame 查询
git_cache = find_last_committer_git_file_cache(
file, rel_file_path, rewrite_vcs_root
)
if git_cache:
value = git_cache["username"]
else:
value = ""
CLANG_SA_LAST_COMMITTER_CACHE[key] = value
return value
def build_notification_markdown(
notification_list: List[ClangSANotificationInfo],
url_prefix: Optional[str],
depot_root: Optional[str],
statistics: Optional[str],
vcs_mode: VCS_MODE,
rewrite_vcs_root: Optional[str],
) -> str:
"""
生成 Markdown 格式的问题报告
包含问题列表表格、详情链接和统计信息
"""
md_lines = ["# 静态分析报告", ""]
if notification_list:
# 生成问题表格
md_lines.extend(["## 严重问题列表", ""])
md_lines.append("| 文件 | Checker分类 | Checker Name | 详情 | 最后提交人 |")
md_lines.append("|--------|-------------|----------------|------|------------|")
else:
md_lines.extend(["## 🎉🎉无严重问题🎉🎉", ""])
# 处理 URL 前缀
if url_prefix:
if url_prefix.endswith("/"):
_url_prefix = url_prefix
else:
_url_prefix = url_prefix + "/"
else:
_url_prefix = None
# 遍历问题列表，生成表格行
for notification in notification_list:
file_path = notification.issue_file.file_path
# 从完整路径中提取相对路径
if depot_root:
dpidx = file_path.find(depot_root)
if dpidx >= 0:
file_path = file_path[dpidx + len(depot_root) + 1 :]
# 生成带链接的文件路径（指向 HTML 报告）
if _url_prefix and notification.plist_file_path:
file_url = f"[{file_path}:{notification.issue_file.line}]({_url_prefix}{os.path.basename(notification.plist_file_path)}.html#reportHash={notification.issue_hash})"
else:
file_url = f"{file_path}:{notification.issue_file.line}"
# 查询最后提交人
if notification.last_committer is None:
notification.last_committer = find_last_committer(
notification.issue_file, file_path, vcs_mode, rewrite_vcs_root
)
md_lines.append(
f"| {file_url} | {notification.category} | {notification.check_name} | {notification.description} | {notification.last_committer} |"
)
# 添加详情链接
if _url_prefix:
md_lines.extend(
[
"",
"## 详情链接",
"",
f"- 问题列表: <{_url_prefix}index.html>",
f"- 问题统计: <{_url_prefix}statistics.html>",
]
)
# 添加统计信息（从 HTML 中提取）
statistics_html = pick_statistics_html_file(statistics, url_prefix)
if statistics_html:
md_lines.extend(["", "## 统计信息", ""])
md_lines.append(statistics_html)
md_lines.append("")
return "\n".join(md_lines)
def main():
global __VERSION__
parser = argparse.ArgumentParser(usage="%(prog)s [options...]")
parser.add_argument("REMAINDER", nargs=argparse.REMAINDER, help="task names")
parser.add_argument(
"-v",
"--version",
action="store_true",
help="show version and exit",
dest="version",
default=False,
)
parser.add_argument(
"-V",
"--verbose",
action="store_true",
help="show verbose",
dest="verbose",
default=False,
)
parser.add_argument(
"-p",
"--plist",
action="store",
help="set directory of plist files",
dest="plist",
default=None,
)
parser.add_argument(
"-u",
"--url-prefix",
action="store",
help="set URL prefix for html files",
dest="url_prefix",
default=None,
)
parser.add_argument(
"-d",
"--depot-root",
action="store",
help="search string to find depot root for files",
dest="depot_root",
default="ProjectDepot/Stream_Depot",
)
parser.add_argument(
"-i",
"--ignore-path",
action="append",
help="file path rules to ignore",
dest="ignore_path",
default=[],
)
parser.add_argument(
"--statistics",
action="store",
help="set statistics file",
dest="statistics",
default=None,
)
parser.add_argument(
"-o",
"--output",
action="store",
help="set output file path",
dest="output",
default="diagnostic_notification.md",
)
parser.add_argument(
"--with-p4",
action="store_true",
help="enable P4 support",
dest="with_p4",
default=False,
)
parser.add_argument(
"--with-git",
action="store_true",
help="enable git support",
dest="with_git",
default=False,
)
parser.add_argument(
"--rewrite-vcs-root",
action="store",
help="rewrite VCS root path",
dest="rewrite_vcs_root",
default=None,
)
options = parser.parse_args()
if options.version:
print(__VERSION__)
return 0
if not options.plist:
parser.print_help()
return 1
# metadata_file = os.path.join(options.plist, "metadata.json")
# if not os.path.exists(metadata_file):
# print(f"Can not find {metadata_file}")
# return 1
plist_files = glob.glob(os.path.join(options.plist, "*.plist"))
if not plist_files:
print(f"Can not find any plist files in {options.plist}")
return 0
# Process each plist file
plist_data = []
notification_diagnostics = []
plist_file_idx = 0
shared_index = dict()
for plist_file in plist_files:
plist_file_idx += 1
if options.verbose:
print(f"[{plist_file_idx}/{len(plist_files)}] Processing {plist_file}")
try:
current_plist_data = build_plist_data(plist_file, shared_index)
current_notification_diagnostics = filter_notification_diagnostics(
current_plist_data
)
print(
f"[{plist_file_idx}/{len(plist_files)}] Processed {plist_file}: {len(current_plist_data)} diagnostics, {len(current_notification_diagnostics)} need notifications"
)
plist_data.extend(current_plist_data)
notification_diagnostics.extend(current_notification_diagnostics)
except Exception as e:
print(f"Error processing {plist_file}: {e}")
continue
print(
f"Parsed {len(plist_data)} diagnostics from {len(plist_files)} files, {len(notification_diagnostics)} need notifications"
)
if options.verbose:
if notification_diagnostics:
print(f"Filtered notifications:")
for diagnostic in notification_diagnostics:
print(f" - {diagnostic}")
if options.ignore_path:
ignore_path = [re.compile(p, re.IGNORECASE) for p in options.ignore_path]
else:
ignore_path = CLANG_SA_DEFAULT_IGNORE_RULES
notification_list = build_notification_list(notification_diagnostics, ignore_path)
vcs_mode = VCS_MODE.NONE
if options.with_p4:
vcs_mode = VCS_MODE.P4
elif options.with_git:
vcs_mode = VCS_MODE.GIT
depot_root = options.depot_root.replace("\\", "/") if options.depot_root else None
if options.output:
with codecs.open(options.output, "w", encoding="utf-8") as f:
f.write(
build_notification_markdown(
notification_list,
options.url_prefix,
depot_root,
options.statistics,
vcs_mode,
options.rewrite_vcs_root,
)
)
print(f"Notification markdown written to {options.output}")
return 0
if __name__ == "__main__":
exit(main())
```
</details>
#### 命令行参数说明
| 参数 | 说明 | 默认值 |
|------|------|--------|
| `-p, --plist` | plist 诊断文件目录 | 必填 |
| `-u, --url-prefix` | HTML 报告的 URL 前缀 | 无 |
| `-d, --depot-root` | 仓库根目录标识字符串 | `ProjectDepot/Stream_Depot` |
| `-i, --ignore-path` | 忽略的文件路径规则（正则） | 默认忽略引擎核心模块 |
| `--statistics` | 统计信息文件路径 | 无 |
| `-o, --output` | 输出 Markdown 文件路径 | `diagnostic_notification.md` |
| `--with-p4` | 启用 Perforce 支持 | `false` |
| `--with-git` | 启用 Git 支持 | `false` |
| `--rewrite-vcs-root` | 重写 VCS 根路径 | 无 |
#### 使用示例
```bash
# 使用 Git blame 查找提交人
python3 ExtractNotification.py \
-p ./codechecker-result \
-u "https://ci.example.com/static-analysis/" \
--with-git \
-o notification.md
# 使用 P4 annotate 查找提交人
python3 ExtractNotification.py \
-p ./codechecker-result \
-u "https://ci.example.com/static-analysis/" \
--with-p4 \
--rewrite-vcs-root "//depot/main" \
-o notification.md
```
#### 输出报告示例
生成的 Markdown 报告格式如下：
```markdown
# 静态分析报告
## 严重问题列表
| 文件 | Checker分类 | Checker Name | 详情 | 最后提交人 |
|--------|-------------|----------------|------|------------|
| [GameMode.cpp:123](url#hash) | Logic error | clang-analyzer-core.NullDereference | Dereference of null pointer | zhangsan |
| [PlayerController.cpp:456](url#hash) | Logic error | bugprone-use-after-move | 'ptr' used after it was moved | lisi |
## 详情链接
- 问题列表: <https://ci.example.com/static-analysis/index.html>
- 问题统计: <https://ci.example.com/static-analysis/statistics.html>
```
## 最佳实践
### CI/CD 集成
建议将静态分析集成到 CI/CD 流程中，实现自动化检测：
```mermaid
flowchart LR
subgraph CI Pipeline
A[代码提交] --> B[构建触发]
B --> C[静态分析]
C --> D{是否有严重问题?}
D -->|是| E[发送通知]
D -->|否| F[归档报告]
E --> G[阻止合入]
F --> H[允许合入]
end
```
### 渐进式推进策略
对于大型存量项目，建议采用渐进式推进策略：
1. **第一阶段**：仅启用编译错误检测（`clang-diagnostic-error`），确保头文件依赖完整
2. **第二阶段**：启用空指针类 checker，修复高危问题
3. **第三阶段**：启用更多 bugprone 类 checker，提升代码质量
4. **第四阶段**：启用代码风格类 checker，统一编码规范
### 配置建议
#### .codechecker.yaml 示例
```yaml
analyzer:
- --ctu # 启用跨翻译单元分析
- --analyzers clangsa clang-tidy # 使用 clang-sa 和 clang-tidy
checker_config:
- clang-tidy:WarningsAsErrors=
- clang-analyzer-core.NullDereference:Enabled=true
- clang-analyzer-core.CallAndMessage:Enabled=true
```
#### .codechecker.skipfile 示例
```text
# 跳过第三方库
+*/ThirdParty/*
+*/Intermediate/*
+*/Generated/*
# 跳过测试代码（可选）
+*/Tests/*
```
## 总结
本文介绍了在 UE 项目中集成 CodeChecker 和 clang-tidy 进行静态分析的完整方案：
1. **UBT 改造**：通过环境变量控制编译器版本和工具链配置，支持使用系统高版本 Clang
2. **编译数据库处理**：使用 `FixClangDatabase.py` 修复 UBT 生成的编译数据库
3. **模块化配置**：通过 `ClangSA` 模块实现按目录层级的 checker 配置继承
4. **问题报告生成**：使用 `ExtractNotification.py` 提取重要问题并查找代码提交人
这套方案已在我们的项目中稳定运行，帮助团队在 CI 阶段发现了大量潜在问题，显著提升了代码质量。
## 参考资料
- [CodeChecker 官方文档](https://codechecker.readthedocs.io/)
- [clang-tidy Checks 列表](https://clang.llvm.org/extra/clang-tidy/checks/list.html)
- [Clang Static Analyzer](https://clang-analyzer.llvm.org/)
- [UE 官方文档 - Static Analysis](https://docs.unrealengine.com/5.0/en-US/static-code-analysis-in-unreal-engine/)
欢迎有相关需求的同学一起交流讨论。