Incremental builds and ASLR

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Incremental builds and ASLR

Rene Groeschke
Hey guys,

while working on the gradle native support I've noticed some issues with
incremental builds.

I've added an implementation of the
AbstractLanguageIncrementalBuildIntegrationTest For ObjectiveC and
Objective-CPP, which works without problems with gcc and with clang on
osx, but I noticed randomly failing tests when using clang on ubuntu.
After some digging I noticed that the generated object files can differ,
even when created from exactly the same sources/flags.

When comparing the according assembler code, I see that the address
space is chosen randomly (see. diff here: http://pastebin.com/uY1LERGX).
This behavior comes from a security feature called ASLR (Address space
layout randomization). Toget our tests to pass on every invocation, I
needed to disable ASLR by globally configuring it through
/proc/sys/kernel/randomize_va_space  (0 for disabling, 2 for enabling, 1
for "conservative randomization) or by disabling it locally via ">
setarch `uname -m` -R /bin/bash".

So, ASLR breaks our incremental builds because the generated object
files can differ.

I havn't yet figured out why ASLR currently does not affect our c / c++
integration tests with clang. There are also a bunch of raised bugs that
prevents/prevented llvm to be complete deterministic. Some of them were
already solved in newer llvm versions. To get the incremental build
running we would need to be able to disable ASLR in our native
toolchains. Of course this comes with the cost of less secure output
that ASLR introduced.

I see 3 options at the moment:

- don't deal with ASLR at all with the cost of possibly loosing
incremental build support.
- always disable ASLR if possible to have deterministic builds (with the
cost of loosing ASLR introduced security).
- model this in our toolchains to allow enabling/disabling ASLR support.

Thoughts?




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Incremental builds and ASLR

Daz DeBoer-2
For now, I'd suggest that we don't deal with ASLR, and we lose some incremental build support for compilers that are non-deterministic.

As far as I can tell, this will only have an effect in this scenario: 
- A source file or any of it's included headers (loaded transitively) has changed since the last time we compiled
- The change to these source inputs has no real effect on the output (for example removing a comment from the end of a line, or adding an irrelavent compiler flag)
- The compiler we are using is non-deterministic

Later I think that modelling this might be the way to go. For GCC the implementation of 'determinism' might be to use '-frandom-seed', whereas on clang/linux we might need to resort to something more drastic (like you have on CI). We could also fail if the toolchain is not able to produce deterministic output (when required).

We might be able to detect determinism in the compiler by compiling a sample source file multiple times: this source contain elements known to invoke randomisation.

Daz



On Tue, Feb 25, 2014 at 12:32 AM, Rene Groeschke <[hidden email]> wrote:
Hey guys,

while working on the gradle native support I've noticed some issues with
incremental builds.

I've added an implementation of the
AbstractLanguageIncrementalBuildIntegrationTest For ObjectiveC and
Objective-CPP, which works without problems with gcc and with clang on
osx, but I noticed randomly failing tests when using clang on ubuntu.
After some digging I noticed that the generated object files can differ,
even when created from exactly the same sources/flags.

When comparing the according assembler code, I see that the address
space is chosen randomly (see. diff here: http://pastebin.com/uY1LERGX).
This behavior comes from a security feature called ASLR (Address space
layout randomization). Toget our tests to pass on every invocation, I
needed to disable ASLR by globally configuring it through
/proc/sys/kernel/randomize_va_space  (0 for disabling, 2 for enabling, 1
for "conservative randomization) or by disabling it locally via ">
setarch `uname -m` -R /bin/bash".

So, ASLR breaks our incremental builds because the generated object
files can differ.

I havn't yet figured out why ASLR currently does not affect our c / c++
integration tests with clang.

Maybe the code that we're using doesn't invoke address-randomisation: if we had more sophisticated test code it might expose this.
 
There are also a bunch of raised bugs that
prevents/prevented llvm to be complete deterministic. Some of them were
already solved in newer llvm versions. To get the incremental build
running we would need to be able to disable ASLR in our native
toolchains. Of course this comes with the cost of less secure output
that ASLR introduced.

I see 3 options at the moment:

- don't deal with ASLR at all with the cost of possibly loosing
incremental build support.
- always disable ASLR if possible to have deterministic builds (with the
cost of loosing ASLR introduced security).
- model this in our toolchains to allow enabling/disabling ASLR support.

Thoughts?




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: Incremental builds and ASLR

Adam Murdoch
In reply to this post by Rene Groeschke

On 25 Feb 2014, at 6:32 pm, Rene Groeschke <[hidden email]> wrote:

Hey guys,

while working on the gradle native support I've noticed some issues with
incremental builds.

I've added an implementation of the
AbstractLanguageIncrementalBuildIntegrationTest For ObjectiveC and
Objective-CPP, which works without problems with gcc and with clang on
osx, but I noticed randomly failing tests when using clang on ubuntu.
After some digging I noticed that the generated object files can differ,
even when created from exactly the same sources/flags.

When comparing the according assembler code, I see that the address
space is chosen randomly (see. diff here: http://pastebin.com/uY1LERGX).
This behavior comes from a security feature called ASLR (Address space
layout randomisation).

You sure that’s the case? ASLR is applied at execution time, not compile time (it’s completely pointless to do this at compile time).

There are apparently some bugs in older versions of Clang that prevent it producing deterministic output, which looks more likely the problem we’re hitting based on that diff you posted above. The clang version that comes with Ubuntu 12.04 was affected, which means any test running on agent1 - agent3 is probably going to fail some times.

Have you seen this test fail on the build vms?


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com



Reply | Threaded
Open this post in threaded view
|

Re: Incremental builds and ASLR

Rene Groeschke


26 Feb 2014 05:51



You sure that’s the case? ASLR is applied at execution time, not compile time (it’s completely pointless to do this at compile time).

There are apparently some bugs in older versions of Clang that prevent it producing deterministic output, which looks more likely the problem we’re hitting based on that diff you posted above. The clang version that comes with Ubuntu 12.04 was affected, which means any test running on agent1 - agent3 is probably going to fail some times.

Have you seen this test fail on the build vms?

I know there where bugs fixed in clang related to deterministic builds, but none of them seem to be related to the issue I have seen. I've tested this with also on the build vms with the same behaviour. I ran tests compiling the same source 10 (20) times and got different kinds of object files generated. When disabling ASLR by calling "setarch `uname -m` -R /bin/bash upfront, this test passes and always generates exactly the same object file (on ubuntu 12.04 and 13.10). The test I used for this can be found here:

https://github.com/gradle/gradle/blob/master/subprojects/cpp/src/integTest/groovy/org/gradle/nativebinaries/language/objectivec/ObjectiveCLanguageIncrementalBuildIntegrationTest.groovy#L51

Best regards,
Rene

--
Rene Groeschke
Principal Engineer,
Gradleware Inc. - Gradle Training, Support, Consulting
[hidden email]
http://gradleware.com


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com



25 Feb 2014 08:32
Hey guys,

while working on the gradle native support I've noticed some issues with
incremental builds.

I've added an implementation of the
AbstractLanguageIncrementalBuildIntegrationTest For ObjectiveC and
Objective-CPP, which works without problems with gcc and with clang on
osx, but I noticed randomly failing tests when using clang on ubuntu.
After some digging I noticed that the generated object files can differ,
even when created from exactly the same sources/flags.

When comparing the according assembler code, I see that the address
space is chosen randomly (see. diff here: http://pastebin.com/uY1LERGX).
This behavior comes from a security feature called ASLR (Address space
layout randomization). Toget our tests to pass on every invocation, I
needed to disable ASLR by globally configuring it through
/proc/sys/kernel/randomize_va_space (0 for disabling, 2 for enabling, 1
for "conservative randomization) or by disabling it locally via ">
setarch `uname -m` -R /bin/bash".

So, ASLR breaks our incremental builds because the generated object
files can differ.

I havn't yet figured out why ASLR currently does not affect our c / c++
integration tests with clang. There are also a bunch of raised bugs that
prevents/prevented llvm to be complete deterministic. Some of them were
already solved in newer llvm versions. To get the incremental build
running we would need to be able to disable ASLR in our native
toolchains. Of course this comes with the cost of less secure output
that ASLR introduced.

I see 3 options at the moment:

- don't deal with ASLR at all with the cost of possibly loosing
incremental build support.
- always disable ASLR if possible to have deterministic builds (with the
cost of loosing ASLR introduced security).
- model this in our toolchains to allow enabling/disabling ASLR support.

Thoughts?




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Incremental builds and ASLR

Adam Murdoch

On 27 Feb 2014, at 12:02 am, Rene Groeschke <[hidden email]> wrote:



26 Feb 2014 05:51



You sure that’s the case? ASLR is applied at execution time, not compile time (it’s completely pointless to do this at compile time).

There are apparently some bugs in older versions of Clang that prevent it producing deterministic output, which looks more likely the problem we’re hitting based on that diff you posted above. The clang version that comes with Ubuntu 12.04 was affected, which means any test running on agent1 - agent3 is probably going to fail some times.

Have you seen this test fail on the build vms?

I know there where bugs fixed in clang related to deterministic builds, but none of them seem to be related to the issue I have seen. I've tested this with also on the build vms with the same behaviour. I ran tests compiling the same source 10 (20) times and got different kinds of object files generated. When disabling ASLR by calling "setarch `uname -m` -R /bin/bash upfront, this test passes and always generates exactly the same object file (on ubuntu 12.04 and 13.10). The test I used for this can be found here:

https://github.com/gradle/gradle/blob/master/subprojects/cpp/src/integTest/groovy/org/gradle/nativebinaries/language/objectivec/ObjectiveCLanguageIncrementalBuildIntegrationTest.groovy#L51

Ok, let’s just treat this the same way we treat the timestamps that visual c++ puts into the object files, rather than mess with the machine settings.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com