前言
现代化的C++项目可以借助多种静态分析工具来检查和发现潜在问题,包括空指针访问风险、未定义行为(UB)、内存错误等。
在我们的项目中,服务器工程基于 CMake 构建系统,可以方便地利用社区支持来集成这些静态分析工具,如 clang-tidy、clang-analyzer、CodeChecker 等。然而,另一部分 C++ 代码运行在 UE(Unreal Engine)引擎中。由于 UE 在编译流程上做了一些特殊扩展,再加上 UBT(UnrealBuildTool)工具在某些能力上的限制,直接开启静态分析会遇到一些问题:
- UE 自带的 Clang 版本较低,部分新版 checker 无法使用
- UBT 生成的编译数据库(compile_commands.json)需要后处理才能被 CodeChecker/clang-tidy 正确解析
- 静态分析的检查规则需要按模块粒度灵活配置
本文主要分享我们对 UBT 的改造方案,以及提供的工具链来提取和处理构建参数,以便让 CodeChecker/clang-tidy 执行分析。同时,我们还实现了通过 Perforce(p4)或 Git 提取问题代码对应行的最后提交人信息(类似 git blame),以便生成问题报告供后续审查。
希望本文可以给有类似需求的同学提供参考。
整体架构
整个静态分析流程可以分为以下几个阶段:
flowchart LR
subgraph 准备阶段
direction TB
A[UBT 改造] --> B[环境变量配置]
B --> C[指定高版本Clang]
end
subgraph 构建分析阶段
direction TB
D[CodeChecker log] --> E[生成 compile_commands.json]
E --> F[FixClangDatabase.py 后处理]
F --> G[修复后的编译数据库]
end
subgraph 分析执行阶段
direction TB
H[CodeChecker analyze] --> I[生成 plist 报告]
I --> J[CodeChecker parse]
J --> K[HTML 报告]
end
subgraph 报告处理阶段
direction TB
L[ExtractNotification.py] --> M[p4 annotate / git blame]
M --> N[Markdown 问题报告]
end
准备阶段 --> 构建分析阶段
构建分析阶段 --> 分析执行阶段
I --> 报告处理阶段
| 阶段 | 主要工具 | 输出 |
|---|---|---|
| 准备阶段 | 修改后的 UBT | 支持系统编译器的构建环境 |
| 构建分析阶段 | CodeChecker log + FixClangDatabase.py | 修复后的 compile_commands.json |
| 分析执行阶段 | CodeChecker analyze/parse | plist 诊断文件 + HTML 报告 |
| 报告处理阶段 | ExtractNotification.py | Markdown 格式的问题清单 |
UBT 改造
为了让 UE 项目能够使用系统安装的高版本 Clang 进行静态分析,我们需要对 UBT 进行一些改造。主要修改集中在 Linux 平台相关的工具链配置。
环境变量配置
我们通过环境变量来控制 UBT 的行为,这样可以在不修改项目配置的情况下灵活切换:
| 环境变量 | 作用 | 默认值 |
|---|---|---|
UE_FORCE_USE_SYSTEM_COMPILER | 强制使用系统编译器而非 UE 自带的 Clang | false |
UE_LINUX_USE_FIX_DEPS | 启用依赖修复(解决循环依赖问题) | false |
UE_LINUX_USE_SYSTEM_LIBCXX | 使用系统的 libc++ 而非 UE 自带版本 | false |
LinuxCommon.cs 修改
文件路径: Engine/Source/Programs/UnrealBuildTool/Platform/Linux/LinuxCommon.cs
增加环境变量读取支持,UE_FORCE_USE_SYSTEM_COMPILER 用于总控使用系统的编译器(提升 Clang 版本):
static public bool bForceUseSystemCompiler = GetDefaultForceUseSystemCompiler();
public static bool GetDefaultBoolValueFromEnvVar(string VarName)
{
string? env = Environment.GetEnvironmentVariable(VarName);
if (!string.IsNullOrWhiteSpace(env))
{
return !string.Equals(env.Trim(), "0", StringComparison.OrdinalIgnoreCase);
}
return false;
}
public static bool GetDefaultForceUseSystemCompiler()
{
return GetDefaultBoolValueFromEnvVar("UE_FORCE_USE_SYSTEM_COMPILER");
}
LinuxToolChain.cs 修改
文件路径: Engine/Source/Programs/UnrealBuildTool/Platform/Linux/LinuxToolChain.cs
注意:以下修改仅适用于 UE 5.6 以下版本。引擎组某些定制化的组件存在循环依赖,在实际解决之前可以先用这种方式适配。
主要改动有几处:
1. 启用依赖修复
修改 bUseFixdeps 的赋值,增加环境变量控制:
// 方式一:仅通过环境变量控制
bUseFixdeps = LinuxCommon.GetDefaultBoolValueFromEnvVar("UE_LINUX_USE_FIX_DEPS");
// 方式二:Windows 交叉编译时默认启用,其他情况通过环境变量控制
bUseFixdeps = BuildHostPlatform.Current.Platform == UnrealTargetPlatform.Win64
|| LinuxCommon.GetDefaultBoolValueFromEnvVar("UE_LINUX_USE_FIX_DEPS");
2. 系统 libc++ 支持
在 ShouldUseLibcxx() 函数处按需加上环境变量判断。
⚠️ ABI 兼容性警告
这部分要注意 ABI 兼容性问题。UE 自带的第三方库都是用自带的 STL 版本编译的,所以最好使用 UE 的 libc++ 版本。
如果你自编译了 LLVM 套件,且 STL 的 ABI 版本不同(如改成了 v2),会导致符号对不上。推荐方案是:高版本编译器 + 低版本 libc++。
如果需要修改,参考以下改动:
GetCompileArguments_Global 函数:
if (ShouldUseLibcxx() && !LinuxCommon.GetDefaultBoolValueFromEnvVar("UE_LINUX_USE_SYSTEM_LIBCXX"))
{
// 使用 UE 自带的 libc++ 头文件
// ...原有逻辑...
}
链接库位置修改:
if (ShouldUseLibcxx())
{
// libc++ and its abi lib
LinkCommandString += " -nodefaultlibs";
if (LinuxCommon.GetDefaultBoolValueFromEnvVar("UE_LINUX_USE_SYSTEM_LIBCXX"))
{
LinkCommandString += " -lc++";
LinkCommandString += " -lc++abi";
}
else
{
LinkCommandString += " -L" + "ThirdParty/Unix/LibCxx/lib/Unix/" + LinkEnvironment.Architecture.LinuxName + "/";
LinkCommandString += " " + "ThirdParty/Unix/LibCxx/lib/Unix/" + LinkEnvironment.Architecture.LinuxName + "/libc++.a";
LinkCommandString += " " + "ThirdParty/Unix/LibCxx/lib/Unix/" + LinkEnvironment.Architecture.LinuxName + "/libc++abi.a";
}
// ...其他链接库...
}
ClangWarnings.cs 修改
文件路径: Engine/Source/Programs/UnrealBuildTool/ToolChain/ClangWarnings.cs
高版本 Clang 会引入新的 warning 或将某些 warning 提升为 error。为了让 UE 项目能够正常编译,需要在 GetDisabledWarnings 函数中关闭这些诊断:
// 通用 warning 关闭
Arguments.Add("-Wno-misleading-indentation");
Arguments.Add("-Wno-vexing-parse");
Arguments.Add("-Wno-error=macro-redefined");
Arguments.Add("-Wno-error=shorten-64-to-32");
Arguments.Add("-Wno-error=shadow");
Arguments.Add("-Wno-error=logical-op-parentheses");
Arguments.Add("-Wno-error=unused-value");
Arguments.Add("-Wno-error=extra-qualification");
Arguments.Add("-Wno-error=range-loop-construct");
Arguments.Add("-Wno-error=comment");
Arguments.Add("-Wno-error=reorder-ctor");
Arguments.Add("-Wno-error=deprecated-comma-subscript");
Arguments.Add("-Wno-error=c++20-extensions");
Arguments.Add("-Wno-error=single-bit-bitfield-constant-conversion");
Arguments.Add("-Wno-error=null-conversion");
Arguments.Add("-Wno-error=dangling");
// Clang 18+ 特有的 warning
if (ClangVersion >= new VersionNumber(18))
{
Arguments.Add("-Wno-deprecated-this-capture"); // https://clang.llvm.org/docs/DiagnosticsReference.html#wdeprecated-this-capture
if (ClangVersion <= new VersionNumber(19))
{
Arguments.Add("-Wno-enum-constexpr-conversion"); // https://clang.llvm.org/docs/DiagnosticsReference.html#wenum-constexpr-conversion
}
Arguments.Add("-Wno-deprecated-literal-operator");
Arguments.Add("-Wno-vla-cxx-extension");
Arguments.Add("-Wno-invalid-unevaluated-string");
Arguments.Add("-Wno-error=nontrivial-memcall");
}
// Clang 19+ 特有的 warning
if (ClangVersion >= new VersionNumber(19))
{
Arguments.Add("-Wno-error=missing-template-arg-list-after-template-kw");
}
// 如果有遗漏,按实际编译错误添加即可
构建与分析流程
完成 UBT 改造后,就可以开始执行静态分析了。整个流程分为以下几个步骤:
第一步:生成编译数据库
使用 CodeChecker 的 log 命令包装构建过程,生成 compile_commands.json:
env UE_FORCE_USE_SYSTEM_COMPILER=1 UE_LINUX_USE_FIX_DEPS=1 \
CodeChecker log -o compile_commands.json \
-b 'make LyraServer ARGS="-ForceUseSystemCompiler -StaticAnalyzer=Clang -StaticAnalyzerMode=deep -StaticAnalyzerOutputType=html"' \
2>&1 | tee build.game.log
参数说明:
-ForceUseSystemCompiler:强制使用系统编译器-StaticAnalyzer=Clang:启用 Clang 静态分析器-StaticAnalyzerMode=deep:深度分析模式-StaticAnalyzerOutputType=html:输出 HTML 格式报告
第二步:查找 PCH 文件
UE 使用共享 PCH(Precompiled Header)来加速编译。静态分析时需要正确包含 PCH 文件:
# 按优先级查找 SharedPCH 文件
FIND_PCH_FILE=$(find $PWD/Projects/Intermediate/Build/Linux/x64/LyraServer/Development/Engine -name "SharedPCH.*Exceptions.*.h" | head -n 1)
if [[ -z "$FIND_PCH_FILE" ]]; then
FIND_PCH_FILE=$(find $PWD/Projects/Intermediate/Build/Linux/x64/LyraServer/Development/Core -name "SharedPCH.*Exceptions.*.h" | head -n 1)
fi
if [[ -z "$FIND_PCH_FILE" ]]; then
FIND_PCH_FILE=$(find $PWD/Projects/Intermediate/Build/Linux/x64/LyraServer/Development -name "SharedPCH.*Exceptions.*.h" | head -n 1)
fi
# 降级查找不带 Exceptions 的版本
if [[ -z "$FIND_PCH_FILE" ]]; then
FIND_PCH_FILE=$(find $PWD/Projects/Intermediate/Build/Linux/x64/LyraServer/Development/Engine -name "SharedPCH.*.h" | head -n 1)
fi
if [[ -z "$FIND_PCH_FILE" ]]; then
FIND_PCH_FILE=$(find $PWD/Projects/Intermediate/Build/Linux/x64/LyraServer/Development/Core -name "SharedPCH.*.h" | head -n 1)
fi
if [[ -z "$FIND_PCH_FILE" ]]; then
FIND_PCH_FILE=$(find $PWD/Projects/Intermediate/Build/Linux/x64/LyraServer/Development -name "SharedPCH.*.h" | head -n 1)
fi
echo "Found PCH file: $FIND_PCH_FILE"
第三步:修复编译数据库
UBT 生成的 compile_commands.json 存在一些问题,需要通过 FixClangDatabase.py 进行后处理:
python3 Projects/Script/FixClangDatabase.py \
-i compile_commands.json \
--pch "$FIND_PCH_FILE" \
--include-path "Projects/Source/LyraGame" \
--include-path "LyraGame/Module.LyraGame"
# 备份原文件并使用修复后的版本
mv -f compile_commands.json compile_commands.json.bak
mv -f compile_commands.fixed.json compile_commands.json
第四步:执行静态分析
使用 CodeChecker 的 analyze 命令执行分析:
CodeChecker analyze \
--config Projects/Plugins/ClangSA/Source/ClangSA/.codechecker.yaml \
-i Projects/Plugins/ClangSA/Source/ClangSA/.codechecker.skipfile \
-j 30 \
-o codechecker-result \
compile_commands.json 2>&1 | tee build.game.log
参数说明:
--config:分析配置文件,定义启用的 checker 和选项-i:跳过文件列表,排除不需要分析的文件-j 30:并行度,根据机器配置调整-o:输出目录
第五步:生成报告
将分析结果转换为 HTML 格式的可视化报告:
CodeChecker parse -e html ./codechecker-result -o ./codechecker-html || true
工具脚本详解
本节详细介绍各个辅助工具的实现原理和使用方法。
ClangSA 模块规则配置
为了让静态分析的检查规则能够按模块粒度灵活配置,我们创建了一个 UE 模块 ClangSA。它通过读取目录下的配置文件来决定每个模块启用哪些 checker。
配置文件
| 文件名 | 作用 |
|---|---|
.clang-sa.enable | 启用的 checker 列表,每行一个 |
.clang-sa.disable | 禁用的 checker 列表,每行一个 |
.clang-tidy | clang-tidy 配置文件 |
.codechecker.yaml | CodeChecker 配置文件 |
.codechecker.skipfile | 跳过分析的文件列表 |
配置继承机制
配置文件支持目录层级继承:
flowchart TB
A["ClangSA Plugin<br />(默认配置)"] --> B["Projects/<br />(项目级配置)"]
B --> C["Projects/Plugins/XXX/<br />(插件级配置)"]
B --> D["Projects/Source/XXX/<br />(模块级配置)"]
style A fill:#f9f,stroke:#333
style B fill:#bbf,stroke:#333
style C fill:#bfb,stroke:#333
style D fill:#bfb,stroke:#333
子目录的配置会继承父目录的配置,并可以追加启用或禁用特定的 checker。
ClangSA.Build.cs
以下是 ClangSA.Build.cs 的核心实现。该模块提供了三个主要功能:
- 规则配置管理:读取并合并目录层级的 checker 配置
- UE 静态分析集成:通过
SetupClangUESA配置 UE 内置的静态分析 - clang-tidy 配置分发:通过
SetupClangTidy将配置文件复制到中间目录
点击展开 ClangSA.Build.cs 完整代码
// Copyright Epic Games, Inc. All Rights Reserved.
using EpicGames.Core;
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using UnrealBuildTool;
public class ClangSA : ModuleRules
{
public class CheckerRules
{
public bool Configured;
public ArrayList EnableCheckers;
public ArrayList DisableCheckers;
public HashSet<String> FinalCheckers;
public CheckerRules() {
Configured = false;
EnableCheckers = new ArrayList();
DisableCheckers = new ArrayList();
FinalCheckers = null;
}
}
static private Dictionary<string, CheckerRules> RulesInDirectory = new Dictionary<string, CheckerRules>();
public ClangSA(ReadOnlyTargetRules Target) : base(Target)
{
Type = ModuleType.External;
}
static public string GetClangSAEnableFilePath(String DirectoryPath)
{
return Path.Combine(DirectoryPath, ".clang-sa.enable");
}
static public string GetClangSADisableFilePath(String DirectoryPath)
{
return Path.Combine(DirectoryPath, ".clang-sa.disable");
}
static public string GetClangSAEnableFilePath(ReadOnlyTargetRules Target) {
return GetClangSAEnableFilePath(Path.Combine(Target.ProjectFile.Directory.FullName, "Plugins", "ClangSA", "Source", "ClangSA"));
}
static public CheckerRules GetCheckRules(String DirectoryPath) {
DirectoryPath = Path.GetFullPath(DirectoryPath);
CheckerRules rules;
lock (RulesInDirectory) {
if (RulesInDirectory.TryGetValue(DirectoryPath, out rules)) {
return rules;
}
}
rules = new CheckerRules();
string enableFilePath = GetClangSAEnableFilePath(DirectoryPath);
if (File.Exists(enableFilePath))
{
rules.Configured = true;
string[] checkers = File.ReadAllLines(enableFilePath);
foreach (string checker in checkers)
{
int comment = checker.IndexOf('#');
string trimmedChecker;
if (comment >= 0)
{
trimmedChecker = checker.Substring(0, comment).Trim();
}
else
{
trimmedChecker = checker.Trim();
}
if (!string.IsNullOrWhiteSpace(trimmedChecker))
{
rules.EnableCheckers.Add(trimmedChecker);
}
}
}
string disableFilePath = GetClangSADisableFilePath(DirectoryPath);
if (File.Exists(disableFilePath))
{
rules.Configured = true;
string[] checkers = File.ReadAllLines(disableFilePath);
foreach (string checker in checkers)
{
int comment = checker.IndexOf('#');
string trimmedChecker;
if (comment >= 0)
{
trimmedChecker = checker.Substring(0, comment).Trim();
}
else
{
trimmedChecker = checker.Trim();
}
if (!string.IsNullOrWhiteSpace(trimmedChecker))
{
rules.DisableCheckers.Add(trimmedChecker);
}
}
}
lock (RulesInDirectory)
{
RulesInDirectory.Add(DirectoryPath, rules);
}
return rules;
}
static public CheckerRules GetClangSACheckers(ReadOnlyTargetRules Target, String DirectoryFullName)
{
String ProjectFullName = Target.ProjectFile.Directory.FullName;
String PluginFullName = Path.Combine(Target.ProjectFile.Directory.FullName, "Plugins", "ClangSA", "Source", "ClangSA");
String ParentFullName = Path.GetDirectoryName(DirectoryFullName);
CheckerRules CurrentRules = GetCheckRules(DirectoryFullName);
if (CurrentRules.FinalCheckers != null) {
return CurrentRules;
}
CurrentRules.FinalCheckers = new HashSet<string> { };
if (!DirectoryFullName.Equals(PluginFullName) && !string.IsNullOrEmpty(ParentFullName) && !ParentFullName.Equals(DirectoryFullName))
{
CheckerRules ParentRules;
if (DirectoryFullName.Equals(ProjectFullName))
{
ParentRules = GetClangSACheckers(Target, PluginFullName);
}
else
{
ParentRules = GetClangSACheckers(Target, ParentFullName);
if (ParentRules.Configured)
{
CurrentRules.Configured = true;
}
}
foreach (var item in ParentRules.FinalCheckers)
{
CurrentRules.FinalCheckers.Add(item);
}
}
foreach (var item in CurrentRules.DisableCheckers)
{
CurrentRules.FinalCheckers.Remove(item.ToString());
}
foreach (var item in CurrentRules.EnableCheckers)
{
CurrentRules.FinalCheckers.Add(item.ToString());
}
return CurrentRules;
}
static public CheckerRules GetClangSACheckers(ReadOnlyTargetRules Target, ModuleRules Module)
{
return GetClangSACheckers(Target, Module.ModuleDirectory);
}
static public void SetupClangUESA(ReadOnlyTargetRules Target, ModuleRules Module)
{
if (Target.StaticAnalyzer != StaticAnalyzer.Clang && Target.StaticAnalyzer != StaticAnalyzer.Default)
{
System.Console.WriteLine($"-------------- Disable UE Static Analysis For {Module.Name} because whole disabled --------------");
return;
}
CheckerRules Rules = GetClangSACheckers(Target, Module);
if (!Rules.Configured)
{
System.Console.WriteLine($"-------------- Disable UE Static Analysis For {Module.Name} because no rules --------------");
return;
}
if (Rules.FinalCheckers == null)
{
System.Console.WriteLine($"-------------- Disable UE Static Analysis For {Module.Name} because no rules --------------");
return;
}
if (Rules.FinalCheckers.Count == 0)
{
System.Console.WriteLine($"-------------- Disable UE Static Analysis For {Module.Name} because no rules --------------");
return;
}
Module.bDisableStaticAnalysis = false;
Module.StaticAnalyzerCheckers = Rules.FinalCheckers;
System.Console.WriteLine($"-------------- Enable UE Static Analysis For {Module.Name} --------------");
System.Console.WriteLine($"-- Checkers: {string.Join(",", Rules.FinalCheckers)}");
}
static private string GetClangTidyConfigureFile(ReadOnlyTargetRules Target, string DirectoryFullName, string FileName)
{
string CheckModuleFullName = Path.Combine(DirectoryFullName, FileName);
if (File.Exists(CheckModuleFullName))
{
return CheckModuleFullName;
}
string ProjectFullName = Target.ProjectFile.Directory.FullName;
string PluginFullName = Path.Combine(Target.ProjectFile.Directory.FullName, "Plugins", "ClangSA", "Source", "ClangSA");
string FallbackFullName = Path.Combine(PluginFullName, FileName);
string ParentFullName = Path.GetDirectoryName(DirectoryFullName);
if (!DirectoryFullName.Equals(PluginFullName) && !DirectoryFullName.Equals(ProjectFullName)
&& !string.IsNullOrEmpty(ParentFullName) && !ParentFullName.Equals(DirectoryFullName))
{
return GetClangTidyConfigureFile(Target, ParentFullName, FileName);
}
return FallbackFullName;
}
static public void SetupClangTidy(ReadOnlyTargetRules Target, ModuleRules Module)
{
if (Module.Type == ModuleType.External)
{
System.Console.WriteLine($"-------------- Disable ClangTidy For {Module.Name} because it's external module --------------");
return;
}
// Target.IntermediateEnvironment
string PlatformIntermediateFolder = GetPlatformIntermediateFolder(Target.Platform, Target.Architectures, false);
string PlatformIntermediateFolderNoArch = GetPlatformIntermediateFolder(Target.Platform, null, false);
string GeneratedCodeDirectory = GetGeneratedCodeDirectory(Target, Module, PlatformIntermediateFolderNoArch).FullName;
string ModuleCodeDirectory = DirectoryReference.Combine(Target.ProjectFile.Directory, PlatformIntermediateFolder,
GetTargetIntermediateFolderName(Target.Name, Target.IntermediateEnvironment),
Target.Configuration.ToString(), Module.ShortName ?? Module.Name).FullName;
// string ModuleDirectory = Module.ModuleDirectory;
string[] CopyFiles = new string[] {
".codechecker.clang-tidy.args",
".codechecker.skipfile",
".codechecker.yaml",
".clang-tidy",
};
string[] CopyToDirs = new string[] {
GeneratedCodeDirectory,
ModuleCodeDirectory,
};
System.Console.WriteLine($"-------------- Setup ClangTidy For {Module.Name} --------------");
foreach (string CopyFile in CopyFiles)
{
string ConfigureFilePath = GetClangTidyConfigureFile(Target, Module.ModuleDirectory, CopyFile);
foreach (string CopyToDir in CopyToDirs)
{
string TargetFilePath = Path.Combine(CopyToDir, CopyFile);
System.Console.WriteLine($"Setup ClangTidy: Copy {ConfigureFilePath} to {TargetFilePath}");
if (!Directory.Exists(CopyToDir))
{
Directory.CreateDirectory(CopyToDir);
}
File.Copy(ConfigureFilePath, TargetFilePath, true);
}
}
}
static public void SetupClangSA(ReadOnlyTargetRules Target, ModuleRules Module)
{
SetupClangUESA(Target, Module);
SetupClangTidy(Target, Module);
}
// UnrealBuildTool.UEBuildTarget.GetTargetIntermediateFolderName 是引擎内部接口,类不是Public的
// 这里复制了一份出来,保持代码逻辑一致
public static string GetTargetIntermediateFolderName(string TargetName, UnrealIntermediateEnvironment IntermediateEnvironment)
{
string TargetFolderName = TargetName;
switch (IntermediateEnvironment)
{
case UnrealIntermediateEnvironment.IWYU:
TargetFolderName += "IWYU";
break;
case UnrealIntermediateEnvironment.NonUnity:
TargetFolderName += "NU";
break;
case UnrealIntermediateEnvironment.Analyze:
TargetFolderName += "SA";
break;
}
return TargetFolderName;
}
// UnrealBuildTool.UEBuildTarget.GetPlatformIntermediateFolder 是引擎内部接口,类不是Public的
// 这里复制了一份出来,保持代码逻辑一致
public static string GetPlatformIntermediateFolder(UnrealTargetPlatform Platform, UnrealArchitectures Architectures, bool External)
{
// now that we have the platform, we can set the intermediate path to include the platform/architecture name
string FolderPath = Path.Combine("Intermediate", External ? "External" : String.Empty, "Build", Platform.ToString());
if (Architectures != null)
{
FolderPath = Path.Combine(FolderPath, UnrealArchitectureConfig.ForPlatform(Platform).GetFolderNameForArchitectures(Architectures));
}
return FolderPath;
}
static public DirectoryReference GetGeneratedCodeDirectory(ReadOnlyTargetRules Target, ModuleRules Module, string PlatformIntermediateFolderNoArch)
{
DirectoryReference GeneratedCodeDirectory = null;
// Get the base directory
// if (Module.Context)
// {
// GeneratedCodeDirectory = Module.Context.DefaultOutputBaseDir;
// }
// else
// {
GeneratedCodeDirectory = Target.ProjectFile.Directory;
// }
// Get the subfolder containing generated code - we don't need architeceture information since these are shared between all arches for a platform
GeneratedCodeDirectory = DirectoryReference.Combine(GeneratedCodeDirectory, PlatformIntermediateFolderNoArch, GetTargetIntermediateFolderName(Target.Name, Target.IntermediateEnvironment), "Inc");
// Append the binaries subfolder, if present. We rely on this to ensure that build products can be filtered correctly.
if (Module.BinariesSubFolder != null)
{
GeneratedCodeDirectory = DirectoryReference.Combine(GeneratedCodeDirectory, Module.BinariesSubFolder);
}
// Finally, append the module name (using the ShortName if it has been set)
GeneratedCodeDirectory = DirectoryReference.Combine(GeneratedCodeDirectory, Module.ShortName ?? Module.Name);
return GeneratedCodeDirectory;
}
}
在模块中集成静态分析
在你的模块的 .Build.cs 文件中,调用 ClangSA.SetupClangSA 即可启用静态分析:
public class LyraGame : ModuleRules
{
public LyraGame(ReadOnlyTargetRules Target) : base(Target)
{
// ... 其他配置 ...
// 启用静态分析
ClangSA.SetupClangSA(Target, this);
}
}
FixClangDatabase.py:编译数据库后处理
UBT 生成的 compile_commands.json 存在以下问题,需要后处理:
- 响应文件(@file)未展开:UBT 使用
.rsp文件存储编译参数,需要展开为完整参数 - 相对路径问题:部分路径是相对路径,需要转换为绝对路径
- 缺少 PCH 包含:需要手动添加共享 PCH 的 include
- 文件过滤:只保留需要分析的源文件
点击展开 FixClangDatabase.py 完整代码
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import codecs
import re
import argparse
import os
import sys
import json
from typing import List, Tuple, Set, Optional, Dict
__VERSION__ = "1.0.0"
CLANG_DATABASE_IGNORE_OPTIONS = set(["-c"])
def try_read_file(file_path):
_err = None
for try_encoding in ["utf-8", "utf-8-sig", "GB18030"]:
try:
ret = codecs.open(file_path, "r", encoding=try_encoding)
return ret
except Exception as e:
if _err is None:
_err = e
if not os.path.exists(file_path):
break
raise _err
class ClangDatabaseProcessor:
def __init__(
self,
engine_source: str,
input_file: str,
output_file: str,
include_path: List[str],
exclude_path: List[str],
pch: Optional[str],
verbose: bool,
):
self.engine_source = engine_source
self.input_file = input_file
self.output_file = output_file
self.include_path: List[re.Pattern] = [
re.compile(p, re.IGNORECASE) for p in include_path
]
self.exclude_path: List[re.Pattern] = [
re.compile(p, re.IGNORECASE) for p in exclude_path
]
self.expanded_cache: Dict[str, List[str]] = dict()
self.pch = pch
self.verbose = verbose
def process(self):
# Implement the processing logic here
result = []
json_data = self._parse_json()
for item in json_data:
if not self._should_include(item):
continue
if self.verbose:
print(f"Processing command for file: {item['file']}")
result.append(self._resolve_item(item))
json.dump(result, self.output_file, indent=2)
def _parse_flag_with_file(self, token, previous_token, flag):
if previous_token == flag:
file_path = token
prefix = ""
suffix = ""
if (file_path.startswith('"') and file_path.endswith('"')) or (
file_path.startswith("'") and file_path.endswith("'")
):
file_path = file_path[1:-1]
prefix = token[:1]
suffix = token[:1]
return (file_path, prefix, suffix)
if not token.startswith(flag):
return (None, None, None)
flag_len = len(flag)
prefix = token[:flag_len]
suffix = ""
file_path = token[flag_len:]
if (file_path.startswith('"') and file_path.endswith('"')) or (
file_path.startswith("'") and file_path.endswith("'")
):
file_path = file_path[1:-1]
prefix = token[: (flag_len + 1)]
suffix = token[flag_len : (flag_len + 1)]
return (file_path, prefix, suffix)
def _resolve_token(self, token, previous_token):
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "@"
)
if file_path:
return self._resolve_response(file_path)
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "-I"
)
if file_path:
return self._resolve_flag_with_file_path(file_path, prefix, suffix)
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "-isystem"
)
if file_path:
return self._resolve_flag_with_file_path(file_path, prefix, suffix)
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "-MF"
)
if file_path:
return self._resolve_flag_with_file_path(file_path, prefix, suffix)
(file_path, prefix, suffix) = self._parse_flag_with_file(
token, previous_token, "-o"
)
if file_path:
return self._resolve_flag_with_file_path(file_path, prefix, suffix)
return [token]
def _resolve_response(self, file_path):
if not os.path.isabs(file_path):
file_path = os.path.join(self.engine_source, file_path)
if file_path in self.expanded_cache:
return self.expanded_cache[file_path]
expanded_paths = []
try:
with try_read_file(file_path) as rsp_file:
previous_token = ""
for line in rsp_file:
line = line.strip()
if not line:
continue
for token in self._split_tokens(line):
if token in CLANG_DATABASE_IGNORE_OPTIONS:
continue
expanded_paths.extend(
self._resolve_token(token, previous_token)
)
previous_token = token
self.expanded_cache[file_path] = expanded_paths
if self.verbose:
print(
f" - Expand response file '{file_path}' to {len(expanded_paths)} args"
)
except Exception as e:
print(
f" - Error reading response file '{file_path}': {e}", file=sys.stderr
)
return [f'@"{file_path}"']
return expanded_paths
def _resolve_flag_with_file_path(self, file_path, prefix, suffix):
if not os.path.isabs(file_path):
file_path = os.path.realpath(os.path.join(self.engine_source, file_path))
return [f"{prefix}{file_path}{suffix}"]
def _next_token(self, input, start_idx):
sz = len(input)
while start_idx < sz and input[start_idx].isspace():
start_idx += 1
if start_idx >= sz:
return (None, sz)
string_quote = None
end_idx = start_idx
while end_idx < sz and not input[end_idx].isspace():
if input[end_idx] == '"' or input[end_idx] == "'":
string_quote = input[end_idx]
end_idx += 1
ignore_next = False
while end_idx < sz and (ignore_next or input[end_idx] != string_quote):
ignore_next = input[end_idx] == "\\"
end_idx += 1
if end_idx < sz:
end_idx += 1
else:
end_idx += 1
return (input[start_idx:end_idx], end_idx)
def _split_tokens(self, input):
# Split the input into tokens based on whitespace
idx = 0
sz = len(input)
ret = []
while idx < sz:
token, end_idx = self._next_token(input, idx)
if token is not None:
ret.append(token)
idx = end_idx
return ret
def _resolve_item(self, item):
command_args = []
checked_pch = False
# Resolve response files
tokens = self._split_tokens(item["command"])
previous_token = ""
for token in tokens:
if token in CLANG_DATABASE_IGNORE_OPTIONS:
continue
command_args.extend(self._resolve_token(token, previous_token))
previous_token = token
if not checked_pch:
checked_pch = True
if self.pch and self.pch.strip():
command_args.extend(["-include", f'"{self.pch}"'])
if "-Wno-unused-command-line-argument" not in command_args:
command_args.append("-Wno-unused-command-line-argument")
if self.verbose:
print(
f" * Resolve command for file: {item['file']} from {len(tokens)} args to {len(command_args)} args"
)
item["command"] = " ".join(command_args)
# Patch file
file_path = item["file"]
if file_path.startswith("@"):
file_path = file_path[1:]
if file_path.endswith(".d.rsp"):
file_path = file_path[: -len(".d.rsp")]
item["file"] = file_path
return item
def _should_include(self, item):
if "file" not in item or "command" not in item:
return False
file_param = item["file"]
if self.include_path:
if not any(p.search(file_param) for p in self.include_path):
return False
if self.exclude_path and any(p.search(file_param) for p in self.exclude_path):
return False
return True
def _parse_json(self):
json_data = json.load(self.input_file)
# Process the JSON data as needed
return json_data
def main():
global __VERSION__
parser = argparse.ArgumentParser(usage="%(prog)s [options...]")
parser.add_argument("REMAINDER", nargs=argparse.REMAINDER, help="task names")
parser.add_argument(
"-v",
"--version",
action="store_true",
help="show version and exit",
dest="version",
default=False,
)
parser.add_argument(
"-V",
"--verbose",
action="store_true",
help="show verbose",
dest="verbose",
default=False,
)
parser.add_argument(
"-i",
"--input",
action="store",
help="set input clang database build file (default: compile_commands.json)",
dest="input",
default="compile_commands.json",
)
parser.add_argument(
"-o",
"--output",
action="store",
help="set output clang database build file (default: compile_commands.json)",
dest="output",
default=None,
)
parser.add_argument(
"--include-path",
action="append",
help="only keep files match include path(regex)",
dest="include_path",
default=[],
)
parser.add_argument(
"--exclude-path",
action="append",
help="only keep files match exclude path(regex)",
dest="exclude_path",
default=[],
)
parser.add_argument(
"--pch",
action="store",
help="set the pch file to include",
dest="pch",
default=None,
)
if os.path.exists(
os.path.join(os.getcwd(), "Engine", "Source", "UnrealGame.Target.cs")
):
find_default_engine_source = os.path.join(os.getcwd(), "Engine", "Source")
elif os.path.exists(
os.path.join(os.getcwd(), "..", "Engine", "Source", "UnrealGame.Target.cs")
):
find_default_engine_source = os.path.realpath(
os.path.join(os.getcwd(), "..", "Engine", "Source")
)
elif os.path.exists(
os.path.join(
os.getcwd(), "..", "..", "Engine", "Source", "UnrealGame.Target.cs"
)
):
find_default_engine_source = os.path.realpath(
os.path.join(os.getcwd(), "..", "..", "Engine", "Source")
)
else:
find_default_engine_source = os.getcwd()
parser.add_argument(
"-e",
"--engine-source",
action="store",
help="set path of Engine/Source, it will be used to convert relative paths",
dest="engine_source",
default=find_default_engine_source,
)
options = parser.parse_args()
if options.version:
print(__VERSION__)
return 0
if options.input.strip() == "-":
input_file = sys.stdin
else:
input_file = try_read_file(options.input)
output_file_path = options.output
if output_file_path is None:
(input_base, input_ext) = os.path.splitext(options.input)
if input_ext:
output_file_path = input_base + ".fixed" + input_ext
else:
output_file_path = input_base + ".fixed"
processor = ClangDatabaseProcessor(
engine_source=options.engine_source,
input_file=input_file,
output_file=codecs.open(output_file_path, "w", encoding="utf-8"),
include_path=options.include_path,
exclude_path=options.exclude_path,
pch=options.pch,
verbose=options.verbose,
)
print(f"[FixClangDatabase]: Using engine source at {options.engine_source}")
print(f"[FixClangDatabase]: Process {options.input} -> {output_file_path}")
processor.process()
return 0
if __name__ == "__main__":
exit(main())
命令行参数说明
| 参数 | 说明 | 默认值 |
|---|---|---|
-i, --input | 输入的编译数据库文件 | compile_commands.json |
-o, --output | 输出的编译数据库文件 | <input>.fixed.json |
--include-path | 只保留匹配的文件路径(正则) | 无 |
--exclude-path | 排除匹配的文件路径(正则) | 无 |
--pch | 要包含的 PCH 文件路径 | 无 |
-e, --engine-source | 引擎源码路径,用于转换相对路径 | 自动检测 |
ExtractNotification.py:问题报告生成
静态分析会生成大量诊断信息,但并非所有问题都需要立即处理。ExtractNotification.py 用于提取需要重点关注的严重问题,并通过 VCS(p4/git)查找每行代码的最后提交人。
工作流程
flowchart LR
A[plist 诊断文件] --> B[解析 XML/plist]
B --> C[过滤重要 checker]
C --> D[定位问题文件和行号]
D --> E[查询 VCS blame]
E --> F[生成 Markdown 报告]
支持的重要 Checker
以下 checker 的问题会被提取到通知报告中:
| Checker | 说明 |
|---|---|
clang-analyzer-core.CallAndMessage | 空指针调用(过滤 object pointer is null) |
clang-analyzer-core.NonNullParamChecker | 非空参数传入空值 |
clang-analyzer-core.NullDereference | 空指针解引用 |
clang-diagnostic-shorten-64-to-32 | 64 位到 32 位截断 |
bugprone-use-after-move | move 后使用 |
bugprone-swapped-arguments | 参数顺序错误 |
bugprone-suspicious-enum-usage | 可疑的枚举使用 |
clang-diagnostic-error | 编译错误 |
点击展开 ExtractNotification.py 完整代码
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import codecs
import re
import argparse
import os
import sys
import glob
import xml.dom.minidom
import subprocess
from typing import List, Any, Optional
__VERSION__ = "1.0.0"
class ClangSANotificationFilter:
def __init__(self, checker_name: str, message_filter: List[Any] = []):
self.checker_name = checker_name
self.checker_name_lowercase = checker_name.lower()
self.message_filter = message_filter
CLANG_SA_NOTIFICATION_CHECKERS = dict()
CLANG_SA_NOTIFICATION_CHECKERS_LOWERCASE = dict()
for checker in [
ClangSANotificationFilter(
"clang-analyzer-core.CallAndMessage",
[re.compile(r"object\s+pointer\s+is\s+null", re.IGNORECASE)],
),
ClangSANotificationFilter("clang-analyzer-core.NonNullParamChecker"),
ClangSANotificationFilter("clang-analyzer-core.NullDereference"),
ClangSANotificationFilter("clang-diagnostic-shorten-64-to-32"),
ClangSANotificationFilter("bugprone-use-after-move"),
ClangSANotificationFilter("bugprone-swapped-arguments"),
ClangSANotificationFilter("bugprone-suspicious-enum-usage"),
ClangSANotificationFilter("clang-diagnostic-error"),
]:
CLANG_SA_NOTIFICATION_CHECKERS[checker.checker_name] = checker
CLANG_SA_NOTIFICATION_CHECKERS_LOWERCASE[checker.checker_name_lowercase] = checker
CLANG_SA_DEFAULT_IGNORE_RULES = [
# 忽略引擎核心模块的问题(这些通常不是项目代码)
re.compile(
r"Engine/Source/Runtime/(Core|CoreUObject|Engine|SlateCore)", re.IGNORECASE
),
# 忽略第三方库
re.compile(r"Projects/Plugins/UnLua/Source/ThirdParty", re.IGNORECASE),
re.compile(r"deps/third_party/install", re.IGNORECASE),
]
# 匹配 C/C++ 源文件的正则表达式
CLANG_SA_SOURCE_FILE_PATTERN = re.compile(r"^.*\.(cc|cxx|cpp|c)$", re.IGNORECASE)
class ClangSANotificationFile:
"""表示一个需要通知的问题文件位置"""
def __init__(self, file_path: str, column: int, line: int):
self.file_path = file_path.replace("\\", "/") # 统一使用 Unix 风格路径
self.column = column
self.line = line
self.notifications = []
def try_read_file(file_path):
"""尝试以多种编码读取文件,自动检测编码"""
_err = None
if not os.path.isabs(file_path):
file_path = os.path.join(file_path)
for try_encoding in ["utf-8", "utf-8-sig", "GB18030"]:
try:
ret = codecs.open(file_path, "r", encoding=try_encoding)
return ret
except Exception as e:
if _err is None:
_err = e
if not os.path.exists(file_path):
break
raise _err
def try_decode_buffer(buffer):
"""尝试以多种编码解码字节buffer"""
for try_encoding in ["utf-8", "utf-8-sig", "GB18030"]:
try:
return buffer.decode(try_encoding)
except Exception:
continue
return buffer.decode(sys.getfilesystemencoding(), errors="ignore")
def _check_notification_file(file_path: str, ignore_path: List[Any]):
"""检查文件路径是否应该被忽略"""
for rule in ignore_path:
if re.search(rule, file_path):
return False
return True
def _find_notification_file(
data: dict, ignore_path: List[Any]
) -> Optional[ClangSANotificationFile]:
"""
从诊断数据中查找问题所在的源文件位置
优先返回 .cpp/.c 源文件,如果没有则返回头文件
"""
fallback_header_file = None
# 首先检查直接的 location 字段
if "location" in data:
location = data["location"]
if (
isinstance(location, dict)
and "file" in location
and "line" in location
and "col" in location
):
file_path = location["file"]
line = location["line"]
column = location["col"]
if _check_notification_file(file_path, ignore_path):
if CLANG_SA_SOURCE_FILE_PATTERN.match(file_path):
return ClangSANotificationFile(file_path, column, line)
else:
fallback_header_file = ClangSANotificationFile(
file_path, column, line
)
if "path" in data:
path_nodes = data["path"]
# 倒序找文件,优先使用源文件
for path_node in reversed(path_nodes):
if "location" in path_node:
location = path_node["location"]
if (
isinstance(location, dict)
and "file" in location
and "line" in location
and "col" in location
):
file_path = location["file"]
line = location["line"]
column = location["col"]
if _check_notification_file(file_path, ignore_path):
if CLANG_SA_SOURCE_FILE_PATTERN.match(file_path):
return ClangSANotificationFile(file_path, column, line)
elif fallback_header_file is None:
fallback_header_file = ClangSANotificationFile(
file_path, column, line
)
return fallback_header_file
class ClangSANotificationInfo:
"""封装单个诊断通知的完整信息"""
def __init__(self, data: dict, ignore_path: List[Any]):
self.data = data
self.check_name = data.get("check_name", "")
self.description = data.get("description", "")
# 对编译错误添加额外说明
if self.check_name == "clang-diagnostic-error":
self.description += "(编译失败,通常是由于头文件依赖不严谨)"
self.category = data.get("category", "")
# issue_hash 用于去重和生成报告链接
self.issue_hash = data.get("issue_hash_content_of_line_in_context", "")
self.plist_file_path = data.get("plist_file_path", "")
# 查找问题所在的源文件
self.issue_file = _find_notification_file(data, ignore_path)
self.last_committer = None # 稍后通过 VCS 查询填充
self.location = None
if "location" in data:
location = data["location"]
if (
isinstance(location, dict)
and "file" in location
and "line" in location
and "col" in location
):
self.location = ClangSANotificationFile(
location["file"], location["col"], location["line"]
)
def is_valid(self):
"""检查通知信息是否有效(必须有文件位置信息)"""
return self.issue_file is not None and self.location is not None
class VCS_MODE:
"""版本控制系统模式枚举"""
NONE = None
GIT = "git"
P4 = "p4"
def build_array_element(node: xml.dom.minidom.Element):
"""解析 plist 的 array 元素,返回 Python 列表"""
ret = []
for child in node.childNodes:
if child.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
child_data = build_object_element(child)
if child_data is not None:
ret.append(child_data)
return ret
def build_dict_element(node: xml.dom.minidom.Element):
"""解析 plist 的 dict 元素,返回 Python 字典"""
ret = {}
current_key = None
for child in node.childNodes:
if child.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
if child.nodeName == "key":
# 提取 key 的文本内容
current_key = ""
for text_node in child.childNodes:
if text_node.nodeType == xml.dom.minidom.Node.TEXT_NODE:
current_key += text_node.nodeValue
else:
# 当前节点是 key 对应的值
if current_key is not None:
value = build_object_element(child)
if value is not None:
ret[current_key] = value
current_key = None
return ret
def get_text_content(node: xml.dom.minidom.Element):
"""提取 XML 元素的文本内容"""
text = ""
for child in node.childNodes:
if child.nodeType == xml.dom.minidom.Node.TEXT_NODE:
text += child.nodeValue
return text
def build_object_element(node: xml.dom.minidom.Element):
"""
根据 plist DTD 规范解析各种类型的元素
@see http://www.apple.com/DTDs/PropertyList-1.0.dtd
"""
tag_name = node.nodeName
if tag_name == "array":
return build_array_element(node)
elif tag_name == "dict":
return build_dict_element(node)
elif tag_name == "string":
return get_text_content(node)
elif tag_name == "integer":
text = get_text_content(node).strip()
try:
return int(text)
except ValueError:
return text
elif tag_name == "real":
text = get_text_content(node).strip()
try:
return float(text)
except ValueError:
return text
elif tag_name == "true":
return True
elif tag_name == "false":
return False
elif tag_name == "data":
# Base64 encoded data
return get_text_content(node).strip()
elif tag_name == "date":
# ISO 8601 date string
return get_text_content(node).strip()
return None
def rebuild_diagnostic(diagnostic: dict | list, file_indexes: List[str]) -> dict | list:
"""
重建诊断数据结构,将文件索引替换为实际的文件路径
plist 中的 file 字段存储的是索引号,需要通过 file_indexes 映射为路径
"""
if isinstance(diagnostic, list):
return [rebuild_diagnostic(item, file_indexes) for item in diagnostic]
ret = {}
for k in diagnostic:
v = diagnostic[k]
if k == "file" and isinstance(v, int):
# 将文件索引映射为实际的文件路径
ret[k] = file_indexes[v]
elif isinstance(v, dict):
# 递归处理嵌套的字典
ret[k] = rebuild_diagnostic(v, file_indexes)
elif isinstance(v, list):
# 递归处理嵌套的列表
ret[k] = [rebuild_diagnostic(item, file_indexes) for item in v]
else:
ret[k] = v
return ret
def build_plist_element(
node: xml.dom.minidom.Element, shared_index: dict
) -> List[dict]:
"""
解析 plist 根元素,提取所有诊断信息
shared_index 用于跨文件去重(相同 checker + issue_hash 的问题只保留一个)
"""
ret = []
for child in node.childNodes:
if child.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
plist_data = build_object_element(child)
# plist 必须包含 files(文件路径列表)和 diagnostics(诊断列表)
if (
not plist_data
or "files" not in plist_data
or "diagnostics" not in plist_data
):
continue
file_indexes = plist_data["files"] # 文件路径索引表
diagnostics = plist_data["diagnostics"]
for diagnostic in diagnostics:
# 必须有 checker 名称和 issue hash 才能处理
if (
"check_name" not in diagnostic
or "issue_hash_content_of_line_in_context" not in diagnostic
):
continue
check_name = diagnostic["check_name"]
issue_hash = diagnostic["issue_hash_content_of_line_in_context"]
# 使用 checker 名称 + issue hash 作为去重 key
if check_name not in shared_index:
shared_index[check_name] = dict()
check_name_map = shared_index[check_name]
if issue_hash in check_name_map:
continue # 跳过重复的问题
# 重建诊断数据,将文件索引替换为路径
rebuild_dict = rebuild_diagnostic(diagnostic, file_indexes)
if rebuild_dict:
check_name_map[issue_hash] = rebuild_dict
ret.append(rebuild_dict)
return ret
def build_plist_data(file_path: str, shared_index: dict) -> List[dict]:
"""解析单个 plist 文件,返回诊断信息列表"""
doc = xml.dom.minidom.parseString(try_read_file(file_path).read())
ret = []
for node in doc.childNodes:
if node.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
if node.nodeName == "plist":
res = build_plist_element(node, shared_index)
for item in res:
item["plist_file_path"] = file_path
ret.extend(res)
else:
child = build_object_element(node)
if child:
child["plist_file_path"] = file_path
ret.append(child)
return ret
def filter_notification_diagnostics(diagnostics: List[dict]) -> List[dict]:
"""
过滤诊断信息,只保留需要通知的重要问题
根据 CLANG_SA_NOTIFICATION_CHECKERS 定义的 checker 列表过滤
"""
ret = []
for diagnostic in diagnostics:
if "check_name" not in diagnostic:
continue
check_name = diagnostic["check_name"].lower()
# 检查是否是需要通知的 checker
if check_name not in CLANG_SA_NOTIFICATION_CHECKERS_LOWERCASE:
continue
selected = False
filter = CLANG_SA_NOTIFICATION_CHECKERS_LOWERCASE[check_name]
# 如果 checker 有消息过滤规则,还需要匹配消息内容
if filter.message_filter:
selected = False
description = diagnostic.get("description", "")
for pattern in filter.message_filter:
if re.search(pattern, description):
selected = True
break
else:
selected = True
if selected:
ret.append(diagnostic)
return ret
def build_notification_list(
notification_diagnostics: List[dict], ignore_path: List[Any]
) -> List[ClangSANotificationInfo]:
"""将过滤后的诊断信息转换为通知对象列表"""
ret = []
for diagnostic in notification_diagnostics:
notification_info = ClangSANotificationInfo(diagnostic, ignore_path)
if notification_info.is_valid():
ret.append(notification_info)
return ret
def patch_statistics_body_dom(
node: xml.dom.minidom.Node, url_prefix: str
) -> xml.dom.minidom.Node:
"""修补统计 HTML 中的链接,添加 URL 前缀"""
if node.nodeName == "a":
href = node.getAttribute("href")
# 跳过绝对 URL
if href.startswith("http://") or href.startswith("https://"):
return node
if not href.startswith("/"):
href = "/" + href
# 添加 URL 前缀
if url_prefix:
if not url_prefix.endswith("/"):
node.setAttribute("href", url_prefix + href)
else:
node.setAttribute("href", url_prefix + href[1:])
# 递归处理子节点
for child_node in node.childNodes:
if child_node.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
patch_statistics_body_dom(child_node, url_prefix)
return node
def scan_statistics_body_dom(
node: xml.dom.minidom.Node, url_prefix: str
) -> Optional[str]:
"""扫描并修补统计页面的 body 内容"""
res = []
for child_node in node.childNodes:
if child_node.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
res.append(child_node.toxml())
continue
res.append(patch_statistics_body_dom(child_node, url_prefix).toxml())
return "".join(res) if res else None
def pick_statistics_html_file(
statistics: Optional[str], url_prefix: str
) -> Optional[str]:
"""
从 CodeChecker 生成的统计 HTML 文件中提取 body 内容
用于嵌入到 Markdown 报告中
"""
if not statistics or not os.path.exists(statistics):
return None
with try_read_file(statistics) as f:
html_content = f.read()
# 提取 <body>...</body> 之间的内容
body_begin = html_content.find("<body>")
body_end = html_content.find("</body>")
body_content = (
html_content[body_begin + 6 : body_end]
if body_begin != -1 and body_end != -1
else ""
)
if not body_content:
return None
# 去除换行,便于解析
body_content = "".join([x.strip() for x in body_content.splitlines()])
doc = xml.dom.minidom.parseString(f"<body>{body_content}</body>")
for node in doc.childNodes:
if node.nodeType != xml.dom.minidom.Node.ELEMENT_NODE:
continue
res = scan_statistics_body_dom(node, url_prefix)
if res:
return res
return None
# ============ VCS 相关的缓存和正则表达式 ============
CLANG_SA_LAST_COMMITTER_CACHE = dict() # 最终结果缓存: file:line -> committer
CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE = dict() # P4 文件级缓存
CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE = dict() # Git 行级缓存
CLANG_SA_LAST_COMMITTER_GIT_FILE_NOT_FOUND = set() # 找不到的文件集合
CLANG_SA_GIT_IGNORE_COMMITTERS = set(["europa_server"]) # 忽略的提交者(如自动化账号)
# Git blame 输出解析用的正则表达式
CLANG_SA_GIT_SHA_RULE = re.compile("^[0-9a-fA-F]+")
CLANG_SA_GIT_AUTHOR_RULE = re.compile("^author\s+(?P<AUTHOR>.+)$")
CLANG_SA_GIT_COMMITTER_RULE = re.compile("^committer\s+(?P<COMMITTER>.+)$")
CLANG_SA_GIT_SUMMARY_RULE = re.compile("^summary\s+(?P<SUMMARY>.+)$")
def parse_p4_annotate_line(line: str) -> Optional[str]:
"""
解析 p4 annotate 输出的单行
格式: changelist: username content
"""
pattern = r"^(\d+):\s+(\S+)\s+(.*)$"
match = re.match(pattern, line)
if match:
return {
"changelist": int(match.group(1)),
"username": match.group(2),
"content": match.group(3),
}
return None
def parse_git_blame_lines(lines: List[str]) -> Optional[dict]:
"""
解析 git blame --line-porcelain 的输出
提取 commit hash、author 和 summary 信息
"""
if not lines:
return None
first_line = lines[0]
sha_hash = CLANG_SA_GIT_SHA_RULE.match(first_line)
if not sha_hash:
return None
author = None
content = None
committer = None
# 遍历各行,提取需要的字段
for line in lines:
if author is not None and content is not None and committer is not None:
break
if author is None:
author_match = CLANG_SA_GIT_AUTHOR_RULE.match(line)
if author_match:
author = author_match.group("AUTHOR")
continue
if content is None:
content_match = CLANG_SA_GIT_SUMMARY_RULE.match(line)
if content_match:
content = content_match.group("SUMMARY")
continue
if committer is None:
committer_match = CLANG_SA_GIT_COMMITTER_RULE.match(line)
if committer_match:
committer = committer_match.group("COMMITTER")
continue
# 如果没有 author,使用 committer
if author is None:
author = committer
if sha_hash and author and content:
return {
"commit": sha_hash,
"username": author,
"content": content,
}
return None
def find_last_committer_p4_file_cache(
file: ClangSANotificationFile, rel_file_path: str, rewrite_vcs_root: Optional[str]
) -> List[str]:
"""
通过 p4 annotate 获取文件每行的最后提交信息
结果会缓存以避免重复查询
"""
global CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE
if file.file_path in CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE:
return CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE[file.file_path]
try:
# 执行 p4 annotate 命令,使用 -c 和 -u 参数
if rel_file_path and rewrite_vcs_root:
cmd = ["p4", "annotate", "-c", "-u", f"{rewrite_vcs_root}/{rel_file_path}"]
else:
cmd = ["p4", "annotate", "-c", "-u", file.file_path]
cmd_str = '"' + '" "'.join(cmd) + '"'
print(f"Running command: {cmd_str}")
result = subprocess.run(
cmd, capture_output=True, text=True, timeout=30, encoding="utf-8"
)
if result.returncode != 0:
print(f"P4 annotate failed: {result.stderr}", file=sys.stderr)
else:
# 解析每行的 annotate 信息
ret = [parse_p4_annotate_line(x) for x in result.stdout.splitlines()]
CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE[file.file_path] = ret
return ret
except subprocess.TimeoutExpired:
print("P4 annotate command timeout", file=sys.stderr)
except Exception as e:
print(f"Error executing p4 annotate: {e}", file=sys.stderr)
# 失败时缓存空列表,避免重复查询
CLANG_SA_LAST_COMMITTER_P4_FILE_CACHE[file.file_path] = []
return []
def find_last_committer_git_file_cache(
file: ClangSANotificationFile, rel_file_path: str, rewrite_vcs_root: Optional[str]
) -> List[str]:
"""
通过 git blame 获取指定行的最后提交信息
支持重试机制:如果当前提交者在忽略列表中,会尝试查询更早的提交
"""
global CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE
global CLANG_SA_LAST_COMMITTER_GIT_FILE_NOT_FOUND
key = f"{file.file_path}:{file.line}"
# 检查缓存
if key in CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE:
return CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE[key]
if file.file_path in CLANG_SA_LAST_COMMITTER_GIT_FILE_NOT_FOUND:
return None
try:
# 最多回溯 10 个版本,跳过自动化账号的提交
for retry_revision_depth in range(10):
if retry_revision_depth == 0:
REVISION = "HEAD"
else:
REVISION = f"HEAD~{retry_revision_depth}"
# 构建 git blame 命令
if rel_file_path and rewrite_vcs_root:
cmd = [
"git",
"blame",
"-L",
f"{file.line},{file.line}", # 只查询指定行
"--line-porcelain", # 详细输出格式
REVISION,
"--",
f"{rewrite_vcs_root}/{rel_file_path}",
]
else:
cmd = [
"git",
"blame",
"-L",
f"{file.line},{file.line}",
"-u",
"--line-porcelain",
REVISION,
"--",
file.file_path,
]
cmd_str = '"' + '" "'.join(cmd) + '"'
print(f"Running command: {cmd_str}")
result = subprocess.run(
cmd, capture_output=True, text=True, timeout=30, encoding="utf-8"
)
if result.returncode != 0:
# 文件不存在,记录下来避免重复查询
if result.stderr.find("no such path") >= 0:
CLANG_SA_LAST_COMMITTER_GIT_FILE_NOT_FOUND.add(file.file_path)
return None
print(f"git blame failed: {result.stderr}", file=sys.stderr)
else:
ret = parse_git_blame_lines(result.stdout.splitlines())
# 如果提交者在忽略列表中,继续查找更早的提交
if ret and ret["username"] in CLANG_SA_GIT_IGNORE_COMMITTERS:
continue
CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE[key] = ret
return ret
except subprocess.TimeoutExpired:
print("git blame command timeout", file=sys.stderr)
except Exception as e:
print(f"Error executing git blame: {e}", file=sys.stderr)
CLANG_SA_LAST_COMMITTER_GIT_FILE_CACHE[key] = None
return None
def find_last_committer(
file: ClangSANotificationFile,
rel_file_path: str,
vcs_mode: VCS_MODE,
rewrite_vcs_root: Optional[str],
) -> Optional[str]:
"""
查找指定文件行的最后提交人
根据 vcs_mode 选择使用 p4 或 git
"""
global CLANG_SA_LAST_COMMITTER_CACHE
key = f"{file.file_path}:{file.line}"
# 检查结果缓存
if key in CLANG_SA_LAST_COMMITTER_CACHE:
return CLANG_SA_LAST_COMMITTER_CACHE[key]
if vcs_mode == VCS_MODE.P4:
p4_cache = find_last_committer_p4_file_cache(
file, rel_file_path, rewrite_vcs_root
)
if file.line > len(p4_cache):
CLANG_SA_LAST_COMMITTER_CACHE[key] = ""
value = ""
else:
p4_annotate = p4_cache[file.line - 1]
if p4_annotate:
value = p4_annotate["username"]
else:
value = ""
elif vcs_mode == VCS_MODE.GIT:
# 使用 git blame 查询
git_cache = find_last_committer_git_file_cache(
file, rel_file_path, rewrite_vcs_root
)
if git_cache:
value = git_cache["username"]
else:
value = ""
CLANG_SA_LAST_COMMITTER_CACHE[key] = value
return value
def build_notification_markdown(
notification_list: List[ClangSANotificationInfo],
url_prefix: Optional[str],
depot_root: Optional[str],
statistics: Optional[str],
vcs_mode: VCS_MODE,
rewrite_vcs_root: Optional[str],
) -> str:
"""
生成 Markdown 格式的问题报告
包含问题列表表格、详情链接和统计信息
"""
md_lines = ["# 静态分析报告", ""]
if notification_list:
# 生成问题表格
md_lines.extend(["## 严重问题列表", ""])
md_lines.append("| 文件 | Checker分类 | Checker Name | 详情 | 最后提交人 |")
md_lines.append("|--------|-------------|----------------|------|------------|")
else:
md_lines.extend(["## 🎉🎉无严重问题🎉🎉", ""])
# 处理 URL 前缀
if url_prefix:
if url_prefix.endswith("/"):
_url_prefix = url_prefix
else:
_url_prefix = url_prefix + "/"
else:
_url_prefix = None
# 遍历问题列表,生成表格行
for notification in notification_list:
file_path = notification.issue_file.file_path
# 从完整路径中提取相对路径
if depot_root:
dpidx = file_path.find(depot_root)
if dpidx >= 0:
file_path = file_path[dpidx + len(depot_root) + 1 :]
# 生成带链接的文件路径(指向 HTML 报告)
if _url_prefix and notification.plist_file_path:
file_url = f"[{file_path}:{notification.issue_file.line}]({_url_prefix}{os.path.basename(notification.plist_file_path)}.html#reportHash={notification.issue_hash})"
else:
file_url = f"{file_path}:{notification.issue_file.line}"
# 查询最后提交人
if notification.last_committer is None:
notification.last_committer = find_last_committer(
notification.issue_file, file_path, vcs_mode, rewrite_vcs_root
)
md_lines.append(
f"| {file_url} | {notification.category} | {notification.check_name} | {notification.description} | {notification.last_committer} |"
)
# 添加详情链接
if _url_prefix:
md_lines.extend(
[
"",
"## 详情链接",
"",
f"- 问题列表: <{_url_prefix}index.html>",
f"- 问题统计: <{_url_prefix}statistics.html>",
]
)
# 添加统计信息(从 HTML 中提取)
statistics_html = pick_statistics_html_file(statistics, url_prefix)
if statistics_html:
md_lines.extend(["", "## 统计信息", ""])
md_lines.append(statistics_html)
md_lines.append("")
return "\n".join(md_lines)
def main():
global __VERSION__
parser = argparse.ArgumentParser(usage="%(prog)s [options...]")
parser.add_argument("REMAINDER", nargs=argparse.REMAINDER, help="task names")
parser.add_argument(
"-v",
"--version",
action="store_true",
help="show version and exit",
dest="version",
default=False,
)
parser.add_argument(
"-V",
"--verbose",
action="store_true",
help="show verbose",
dest="verbose",
default=False,
)
parser.add_argument(
"-p",
"--plist",
action="store",
help="set directory of plist files",
dest="plist",
default=None,
)
parser.add_argument(
"-u",
"--url-prefix",
action="store",
help="set URL prefix for html files",
dest="url_prefix",
default=None,
)
parser.add_argument(
"-d",
"--depot-root",
action="store",
help="search string to find depot root for files",
dest="depot_root",
default="ProjectDepot/Stream_Depot",
)
parser.add_argument(
"-i",
"--ignore-path",
action="append",
help="file path rules to ignore",
dest="ignore_path",
default=[],
)
parser.add_argument(
"--statistics",
action="store",
help="set statistics file",
dest="statistics",
default=None,
)
parser.add_argument(
"-o",
"--output",
action="store",
help="set output file path",
dest="output",
default="diagnostic_notification.md",
)
parser.add_argument(
"--with-p4",
action="store_true",
help="enable P4 support",
dest="with_p4",
default=False,
)
parser.add_argument(
"--with-git",
action="store_true",
help="enable git support",
dest="with_git",
default=False,
)
parser.add_argument(
"--rewrite-vcs-root",
action="store",
help="rewrite VCS root path",
dest="rewrite_vcs_root",
default=None,
)
options = parser.parse_args()
if options.version:
print(__VERSION__)
return 0
if not options.plist:
parser.print_help()
return 1
# metadata_file = os.path.join(options.plist, "metadata.json")
# if not os.path.exists(metadata_file):
# print(f"Can not find {metadata_file}")
# return 1
plist_files = glob.glob(os.path.join(options.plist, "*.plist"))
if not plist_files:
print(f"Can not find any plist files in {options.plist}")
return 0
# Process each plist file
plist_data = []
notification_diagnostics = []
plist_file_idx = 0
shared_index = dict()
for plist_file in plist_files:
plist_file_idx += 1
if options.verbose:
print(f"[{plist_file_idx}/{len(plist_files)}] Processing {plist_file}")
try:
current_plist_data = build_plist_data(plist_file, shared_index)
current_notification_diagnostics = filter_notification_diagnostics(
current_plist_data
)
print(
f"[{plist_file_idx}/{len(plist_files)}] Processed {plist_file}: {len(current_plist_data)} diagnostics, {len(current_notification_diagnostics)} need notifications"
)
plist_data.extend(current_plist_data)
notification_diagnostics.extend(current_notification_diagnostics)
except Exception as e:
print(f"Error processing {plist_file}: {e}")
continue
print(
f"Parsed {len(plist_data)} diagnostics from {len(plist_files)} files, {len(notification_diagnostics)} need notifications"
)
if options.verbose:
if notification_diagnostics:
print(f"Filtered notifications:")
for diagnostic in notification_diagnostics:
print(f" - {diagnostic}")
if options.ignore_path:
ignore_path = [re.compile(p, re.IGNORECASE) for p in options.ignore_path]
else:
ignore_path = CLANG_SA_DEFAULT_IGNORE_RULES
notification_list = build_notification_list(notification_diagnostics, ignore_path)
vcs_mode = VCS_MODE.NONE
if options.with_p4:
vcs_mode = VCS_MODE.P4
elif options.with_git:
vcs_mode = VCS_MODE.GIT
depot_root = options.depot_root.replace("\\", "/") if options.depot_root else None
if options.output:
with codecs.open(options.output, "w", encoding="utf-8") as f:
f.write(
build_notification_markdown(
notification_list,
options.url_prefix,
depot_root,
options.statistics,
vcs_mode,
options.rewrite_vcs_root,
)
)
print(f"Notification markdown written to {options.output}")
return 0
if __name__ == "__main__":
exit(main())
命令行参数说明
| 参数 | 说明 | 默认值 |
|---|---|---|
-p, --plist | plist 诊断文件目录 | 必填 |
-u, --url-prefix | HTML 报告的 URL 前缀 | 无 |
-d, --depot-root | 仓库根目录标识字符串 | ProjectDepot/Stream_Depot |
-i, --ignore-path | 忽略的文件路径规则(正则) | 默认忽略引擎核心模块 |
--statistics | 统计信息文件路径 | 无 |
-o, --output | 输出 Markdown 文件路径 | diagnostic_notification.md |
--with-p4 | 启用 Perforce 支持 | false |
--with-git | 启用 Git 支持 | false |
--rewrite-vcs-root | 重写 VCS 根路径 | 无 |
使用示例
# 使用 Git blame 查找提交人
python3 ExtractNotification.py \
-p ./codechecker-result \
-u "https://ci.example.com/static-analysis/" \
--with-git \
-o notification.md
# 使用 P4 annotate 查找提交人
python3 ExtractNotification.py \
-p ./codechecker-result \
-u "https://ci.example.com/static-analysis/" \
--with-p4 \
--rewrite-vcs-root "//depot/main" \
-o notification.md
输出报告示例
生成的 Markdown 报告格式如下:
# 静态分析报告
## 严重问题列表
| 文件 | Checker分类 | Checker Name | 详情 | 最后提交人 |
|--------|-------------|----------------|------|------------|
| [GameMode.cpp:123](url#hash) | Logic error | clang-analyzer-core.NullDereference | Dereference of null pointer | zhangsan |
| [PlayerController.cpp:456](url#hash) | Logic error | bugprone-use-after-move | 'ptr' used after it was moved | lisi |
## 详情链接
- 问题列表: <https://ci.example.com/static-analysis/index.html>
- 问题统计: <https://ci.example.com/static-analysis/statistics.html>
最佳实践
CI/CD 集成
建议将静态分析集成到 CI/CD 流程中,实现自动化检测:
flowchart LR
subgraph CI Pipeline
A[代码提交] --> B[构建触发]
B --> C[静态分析]
C --> D{是否有严重问题?}
D -->|是| E[发送通知]
D -->|否| F[归档报告]
E --> G[阻止合入]
F --> H[允许合入]
end
渐进式推进策略
对于大型存量项目,建议采用渐进式推进策略:
- 第一阶段:仅启用编译错误检测(
clang-diagnostic-error),确保头文件依赖完整 - 第二阶段:启用空指针类 checker,修复高危问题
- 第三阶段:启用更多 bugprone 类 checker,提升代码质量
- 第四阶段:启用代码风格类 checker,统一编码规范
配置建议
.codechecker.yaml 示例
analyzer:
- --ctu # 启用跨翻译单元分析
- --analyzers clangsa clang-tidy # 使用 clang-sa 和 clang-tidy
checker_config:
- clang-tidy:WarningsAsErrors=
- clang-analyzer-core.NullDereference:Enabled=true
- clang-analyzer-core.CallAndMessage:Enabled=true
.codechecker.skipfile 示例
# 跳过第三方库
+*/ThirdParty/*
+*/Intermediate/*
+*/Generated/*
# 跳过测试代码(可选)
+*/Tests/*
总结
本文介绍了在 UE 项目中集成 CodeChecker 和 clang-tidy 进行静态分析的完整方案:
- UBT 改造:通过环境变量控制编译器版本和工具链配置,支持使用系统高版本 Clang
- 编译数据库处理:使用
FixClangDatabase.py修复 UBT 生成的编译数据库 - 模块化配置:通过
ClangSA模块实现按目录层级的 checker 配置继承 - 问题报告生成:使用
ExtractNotification.py提取重要问题并查找代码提交人
这套方案已在我们的项目中稳定运行,帮助团队在 CI 阶段发现了大量潜在问题,显著提升了代码质量。
参考资料
欢迎有相关需求的同学一起交流讨论。