Improve Native Compile Speed

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Improve Native Compile Speed

Daniel Lacasse
If you are depending on a big project, let say Boost, you get a big performance hit. Please correct me if I'm wrong.

Story: Better input selection for AbstractNativeCompileTask
The AbstractNativeCompileTask specified @InputFiles on FileCollection includes. The up-to-date check would then hash every files in all include path. In practice, this doesn't work. You may only be using a couple header files directly and because of the header dependencies, this may only gets to a handful of files. Boost has about 10k of header files which is the perfect example for showing this problem.

Implementation
The proposed solution is to have a list of include directory as the Input for this task. Probably a List<File> and annotate with @Input. We would keep the power of having File object but remove the hashing of every single files within the include directory.

Story: Improve incremental compilation
Since we will only rely on header parsing for header dependencies, it would be a good idea to improve the current processing. I don't take credit this next improvement. For the full explanation, see http://martine.github.io/ninja/manual.html#ref_headers. Basically, instead of parsing all files in order to find the #include which is the current Gradle way, we can use the initial compiler run with specific compiler option to output the dependency graph as the compiler sees it. In gcc and clang it's the -MMD and -MF flags and msvc it's the /showIncludes flag.

Implementation:
For msvc, a new option is needed on the toolchain for telling Gradle what is the prefix String for each header dependency output line from cl.exe. This option is needed for non-english Visual Studio installation. This can be auto-detected by Gradle but just in case Microsoft decide to port Visual Studio in English pirate. For gcc and clang, -MF would point to a file in the build/tmp/<taskName> directory.

For improved speed - and I assume that Gradle cache is extremely fast and efficient - Gradle would save the header dependency information in it's cache. This means Gradle would parse the gcc and clang dependency file as soon as it's generated.

The rest of the incremental compile stays the same.

Story: Even better incremental compilation
I don't think Gradle does any distinction between prebuilt libraries and project libraries. A really good optimisation is to automatically mark prebuilt libraries includes as up-to-date.

Implementation:
This can be quickly win by removing the header dependencies of prebuilt library directly when the cache is generated. This way Gradle has no knowledge of the prebuilt library which prevent unnecessary parsing.

Open Issues:
If a project is up-to-date and you decide to upgrade only the prebuilt libraries to a newer version, the compile task will stay up-to-date. This may not be the expected behaviour.

--
Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Improve Native Compile Speed

Adam Murdoch

On 10 Jul 2014, at 9:19 pm, Daniel Lacasse <[hidden email]> wrote:

If you are depending on a big project, let say Boost, you get a big performance hit. Please correct me if I'm wrong.

Story: Better input selection for AbstractNativeCompileTask
The AbstractNativeCompileTask specified @InputFiles on FileCollection includes. The up-to-date check would then hash every files in all include path.

This isn’t quite what happens. It doesn’t hash every header file on every build invocation - it hashes each header file once and caches the result, and then uses (last modified timestamp, length) to invalidate this cached hash.

There are really two problems here:

1. Gradle considers files that aren’t actually being used as inputs (and on the flip side, is ignoring other files that are).
2. The implementation of the up-to-date checks are inefficient.

More on this below.

In practice, this doesn't work. You may only be using a couple header files directly and because of the header dependencies, this may only gets to a handful of files. Boost has about 10k of header files which is the perfect example for showing this problem.

Implementation
The proposed solution is to have a list of include directory as the Input for this task. Probably a List<File> and annotate with @Input. We would keep the power of having File object but remove the hashing of every single files within the include directory.

I think a better option would be to have the task declare which header files each source file depends on, and have Gradle do the up-to-date checks based on this. The task calculates this stuff anyway.


Story: Improve incremental compilation
Since we will only rely on header parsing for header dependencies, it would be a good idea to improve the current processing. I don't take credit this next improvement. For the full explanation, see http://martine.github.io/ninja/manual.html#ref_headers. Basically, instead of parsing all files in order to find the #include which is the current Gradle way, we can use the initial compiler run with specific compiler option to output the dependency graph as the compiler sees it. In gcc and clang it's the -MMD and -MF flags and msvc it's the /showIncludes flag.

Implementation:
For msvc, a new option is needed on the toolchain for telling Gradle what is the prefix String for each header dependency output line from cl.exe. This option is needed for non-english Visual Studio installation. This can be auto-detected by Gradle but just in case Microsoft decide to port Visual Studio in English pirate. For gcc and clang, -MF would point to a file in the build/tmp/<taskName> directory.

For improved speed - and I assume that Gradle cache is extremely fast and efficient - Gradle would save the header dependency information in it's cache. This means Gradle would parse the gcc and clang dependency file as soon as it's generated.

The rest of the incremental compile stays the same.

I don’t want to make this change. Firstly, I don’t think it’s inherently faster, and secondly, it prevents us doing interesting performance improvements later.

Instead, we should make the existing implementation faster, in the following ways:

1. Use a single cache of parsed header files that is shared by (at least) all compilation tasks in a build, or (better) all compilation tasks on a given machine.
2. Calculate whether a the graph for a given (header file, include roots) is up-to-date or not once per build, rather than once per (file, task).
3. Change the daemon to watch for file system changes, and invalidate the above stuff as things change.
4. Make the up-to-date check faster for those files we know will not change (or are very unlikely to change), such as files in the Gradle artefact cache, or those managed by the system package manager. There would have to be some way to tell Gradle that you want or don’t want this optimisation.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
CTO Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com