File Tracker Log file format

Question

In Visual Studio 2010 incremental builds are done using the File Tracker (Microsoft.Build.Utilities.FileTracker). It seems that it is responsible to the creation of these *.1.tlog files on the intermediate directory.

I couldn't find any reference to the syntax of these .tlog files.
They contain a list of paths to files that are read/written while the tracker tracks the execution of some tool, in order to check which files should be compiled in an incremental build. However, these files also contain some special characters such as "^" and "|".
Another thing I noticed was that these files are sometimes edited from Visual Studio targets files. For example, in Microsoft.CppCommon.targets on CustomBuildStep target I found the following line:

 <!-- Appended tlog to track custom build events -->
    <WriteLinesToFile File="$(IntDir)$(ProjectName).write.1.tlog" Lines="@(CustomBuildStep->'^%(Identity)');@(CustomBuildStep->MetaData('Outputs')->FullPath()->Distinct())"/>

So this probably means that the project file is dependent on the custom build step outputs.

My questions are:

• Does anyone know of a reference for the .tlog file syntax?
• In which cases is the tracker-log used on Visual studio? I know of the CL and maybe Link tasks that use it, but it seems that Visual Studio IDE itself uses it in order to decide whether to run msbuild at all for a certain project.

Thanks

---

EDIT

Another hint:

CanonicalTrackedInputFiles Class is document as "the filetracking log interpreter for .read. tracking logs in canonical form or those that have been rooted (^) to make them canonical"
When I have time I'll dig into it a bit more. Perhaps this class and others under Microsoft.Build.Utilities could be used to help us work with tlog files instead of working with the raw text tlog files directly.

See also: CanonicalTrackedOutputFiles Class, FlatTrackingData Class and of course FileTracker Class.

Orly

Answer 1

The tlog file format is now documented here: https://learn.microsoft.com/en-us/visualstudio/extensibility/visual-cpp-project-extensibility?view=vs-2017#tlog-files

Read .tlog format

Read .tlog files (.read..tlog) contain information about source files and their dependencies.

A caret (^) at the beginning of a line indicates one or more sources. Sources that share the same dependencies are separated by a vertical bar (|).

Dependency files are listed after the sources, each on its own line. All file names are full paths.

Sport

Thank you for this link. It's about time they added some documentation for the tracker and tlog files! – Orly

Answer 2

This stuff doesn't seem to be documented anywhere, so I've had to figure this out based on trial and error and by staring at some example targets/xml/props files:

The reason the custom build step writes to the tlog file by hand is that Build|Clean - and probably its command line counterpart - scrapes the tlog files to find which files to delete. It seems to look for all files matching *.write.tlog, or maybe *.1.write.tlog, in the intermediate folder, read the list of file names in each, and delete the files named. So, if the custom build steps knows what its outputs are, it can simply record them in a tlog file, and interact with Build|Clean that way.

(You can try this out yourself - build your project, create some temp files, add your own tlog file to your project's intermediate folder containing the paths to the temp files, then do a Build|Clean. Your temp files will be deleted along with the usual build artefacts.)

In a tlog file, a file with no prefix is the name of an output file. These files are deleted when you do a Build|Clean.

A file with a ^ before it a comment, I think - perhaps obviously, Build|Clean doesn't touch any of these.

As for |, I've only seen it in comment lines, used to separate different file names. I suspect this is just a convention, and not some special syntax, since if you put multiple output files on line, separated by |, Build|Clean doesn't delete them.

Expel

Thanks for the answer Tom. The tlog files are a result of running tracker.exe behind the scenes, so you can do some more experiments with it if you want. The tracker seems like a really handy tool that records all files read/written when you run some executable in its context. Anyway I really hoped for some formal definition of the tlog format, more than guesses, so if you ever find such reference please let me know. – Orly

I thought the '^' was to indicate a file (say file 'A') and any files after it that don't have the '^' are files that file 'A' is dependent on. Files that have the same dependencies are separated by '|' so that dependencies don't have to be listed multiple times. – Cochard

Answer 3

The tlog file is generated by the file tracking system in MSBuild. It isn't specific to individual compilers, it can capture any file reference by anything done in the MSBuild process, depending on how it is configured.

Pennyworth

I know that. You did not answer my question: Do you know a reference to the .tlog file syntax? – Orly