integration tests

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

integration tests

Adam Murdoch-2
Hi,

I've just checked in some test cases into src/test/groovy which I've
termed 'integration' tests. I think they might need a better home, but
I've added them there so that they are being run in the meantime.

The intended scope of these tests is somewhere between the unit tests
(test a single class in isolation) and the existing integration tests (a
broad pass through gradle as a whole). They are intended to 1. check
that a bunch of classes work together, 2. be more focused on a
particular behaviour than the other integration tests, and 3. be faster
to run. In particular, I think these sorts of tests are handy for
testing error handling.

So, where should they live? One option is to leave them where they are,
as they are pretty fast. Another option is to run them as part of the
integration test suite, possibly porting the existing int tests to junit.


Adam

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: integration tests

hans_d
Administrator

On Aug 17, 2008, at 12:46 AM, Adam Murdoch wrote:

> Hi,
>
> I've just checked in some test cases into src/test/groovy which  
> I've termed 'integration' tests. I think they might need a better  
> home, but I've added them there so that they are being run in the  
> meantime.
>
> The intended scope of these tests is somewhere between the unit  
> tests (test a single class in isolation) and the existing  
> integration tests (a broad pass through gradle as a whole). They  
> are intended to 1. check that a bunch of classes work together, 2.  
> be more focused on a particular behaviour than the other  
> integration tests, and 3. be faster to run. In particular, I think  
> these sorts of tests are handy for testing error handling.
>
> So, where should they live? One option is to leave them where they  
> are, as they are pretty fast.

I think it is a good place where they are now as they are fast. It  
might make sense to organize this kind of integration tests into its  
own package structure.

When executing them a test fails:

java.lang.AssertionError:
Expected: not null
      got: null

        at org.junit.Assert.assertThat(Assert.java:502)
        at org.junit.Assert.assertThat(Assert.java:492)
        at org.gradle.AbstractIntegrationTest.getTestBuildFile
(AbstractIntegrationTest.java:43)
        at  
org.gradle.ProjectLoadingIntegrationTest.handlesSimilarlyNamedBuildFiles
InSameDirectory(ProjectLoadingIntegrationTest.java:26)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
        at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
        at org.junit.internal.runners.MethodRoadie.runTestMethod
(MethodRoadie.java:98)
        at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
        at  
org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters
(MethodRoadie.java:87)
        at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:
77)
        at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
        at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod
(JUnit4ClassRunner.java:88)
        at org.junit.internal.runners.JUnit4ClassRunner.runMethods
(JUnit4ClassRunner.java:51)
        at org.junit.internal.runners.JUnit4ClassRunner$1.run
(JUnit4ClassRunner.java:44)
        at org.junit.internal.runners.ClassRoadie.runUnprotected
(ClassRoadie.java:27)
        at org.junit.internal.runners.ClassRoadie.runProtected
(ClassRoadie.java:37)
        at org.junit.internal.runners.JUnit4ClassRunner.run
(JUnit4ClassRunner.java:42)
        at com.intellij.rt.junit4.Junit4ClassSuite.run(Junit4ClassSuite.java:
99)
        at com.intellij.rt.execution.junit.JUnitStarter.main
(JUnitStarter.java:40)

> Another option is to run them as part of the integration test  
> suite, possibly porting the existing int tests to junit.

Porting those tests to a test framework would probably makes sense  
anyway. I don't know why I haven't done this from the very beginning.  
I anyway would like to refactor the integration test execution. Right  
now all the integration tests are executed after the distribution  
archives are generated. But only the wrapper integration test needs  
the archives. The other tests only need the exploded distribution.  
When doing this I will port them to JUnit.

- Hans

--
Hans Dockter
Gradle Project lead
http://www.gradle.org





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: integration tests

hans_d
Administrator

On Aug 18, 2008, at 10:06 AM, Hans Dockter wrote:

>
> On Aug 17, 2008, at 12:46 AM, Adam Murdoch wrote:
>
>> Hi,
>>
>> I've just checked in some test cases into src/test/groovy which  
>> I've termed 'integration' tests. I think they might need a better  
>> home, but I've added them there so that they are being run in the  
>> meantime.
>>
>> The intended scope of these tests is somewhere between the unit  
>> tests (test a single class in isolation) and the existing  
>> integration tests (a broad pass through gradle as a whole). They  
>> are intended to 1. check that a bunch of classes work together, 2.  
>> be more focused on a particular behaviour than the other  
>> integration tests, and 3. be faster to run. In particular, I think  
>> these sorts of tests are handy for testing error handling.
>>
>> So, where should they live? One option is to leave them where they  
>> are, as they are pretty fast.
>
> I think it is a good place where they are now as they are fast. It  
> might make sense to organize this kind of integration tests into  
> its own package structure.
>
> When executing them a test fails:
>
> java.lang.AssertionError:
> Expected: not null
>      got: null
>
> at org.junit.Assert.assertThat(Assert.java:502)
> at org.junit.Assert.assertThat(Assert.java:492)
> at org.gradle.AbstractIntegrationTest.getTestBuildFile
> (AbstractIntegrationTest.java:43)
> at  
> org.gradle.ProjectLoadingIntegrationTest.handlesSimilarlyNamedBuildFil
> esInSameDirectory(ProjectLoadingIntegrationTest.java:26)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:25)
> at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
> at org.junit.internal.runners.MethodRoadie.runTestMethod
> (MethodRoadie.java:98)
> at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:
> 79)
> at  
> org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters
> (MethodRoadie.java:87)
> at org.junit.internal.runners.MethodRoadie.runTest
> (MethodRoadie.java:77)
> at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
> at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod
> (JUnit4ClassRunner.java:88)
> at org.junit.internal.runners.JUnit4ClassRunner.runMethods
> (JUnit4ClassRunner.java:51)
> at org.junit.internal.runners.JUnit4ClassRunner$1.run
> (JUnit4ClassRunner.java:44)
> at org.junit.internal.runners.ClassRoadie.runUnprotected
> (ClassRoadie.java:27)
> at org.junit.internal.runners.ClassRoadie.runProtected
> (ClassRoadie.java:37)
> at org.junit.internal.runners.JUnit4ClassRunner.run
> (JUnit4ClassRunner.java:42)
> at com.intellij.rt.junit4.Junit4ClassSuite.run
> (Junit4ClassSuite.java:99)
> at com.intellij.rt.execution.junit.JUnitStarter.main
> (JUnitStarter.java:40)

They fail only when I execute them from intelliJ. So it looks like a  
problem on my side.

- Hans

--
Hans Dockter
Gradle Project lead
http://www.gradle.org





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: integration tests

Adam Murdoch-2
In reply to this post by hans_d


Hans Dockter wrote:

>
> On Aug 17, 2008, at 12:46 AM, Adam Murdoch wrote:
>
>> Hi,
>>
>> I've just checked in some test cases into src/test/groovy which I've
>> termed 'integration' tests. I think they might need a better home,
>> but I've added them there so that they are being run in the meantime.
>>
>> The intended scope of these tests is somewhere between the unit tests
>> (test a single class in isolation) and the existing integration tests
>> (a broad pass through gradle as a whole). They are intended to 1.
>> check that a bunch of classes work together, 2. be more focused on a
>> particular behaviour than the other integration tests, and 3. be
>> faster to run. In particular, I think these sorts of tests are handy
>> for testing error handling.
>>
>> So, where should they live? One option is to leave them where they
>> are, as they are pretty fast.
>
> I think it is a good place where they are now as they are fast. It
> might make sense to organize this kind of integration tests into its
> own package structure.
>
I agree. I'll move them out of the top level package.

> When executing them a test fails:
>
> java.lang.AssertionError:
> Expected: not null
>      got: null
>
>     at org.junit.Assert.assertThat(Assert.java:502)
>     at org.junit.Assert.assertThat(Assert.java:492)
>     at
> org.gradle.AbstractIntegrationTest.getTestBuildFile(AbstractIntegrationTest.java:43)
>
>     at
> org.gradle.ProjectLoadingIntegrationTest.handlesSimilarlyNamedBuildFilesInSameDirectory(ProjectLoadingIntegrationTest.java:26)
>
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>     at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
>     at
> org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
>
>     at
> org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
>     at
> org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
>
>     at
> org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
>     at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
>     at
> org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
>
>     at
> org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
>
>     at
> org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
>
>     at
> org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
>
>     at
> org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
>     at
> org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
>
>     at
> com.intellij.rt.junit4.Junit4ClassSuite.run(Junit4ClassSuite.java:99)
>     at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:40)
>
You'll need to add src/test/resources to your classpath when you run
them, and add *.gradle to the set of resources that intellij copies into
the test class dir.  I'll update the wiki to mention this.


Adam

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: integration tests

hans_d
Administrator

On Aug 18, 2008, at 10:53 AM, Adam Murdoch wrote:

>
>
> Hans Dockter wrote:
>>
>> On Aug 17, 2008, at 12:46 AM, Adam Murdoch wrote:
>>
>>> Hi,
>>>
>>> I've just checked in some test cases into src/test/groovy which  
>>> I've termed 'integration' tests. I think they might need a better  
>>> home, but I've added them there so that they are being run in the  
>>> meantime.
>>>
>>> The intended scope of these tests is somewhere between the unit  
>>> tests (test a single class in isolation) and the existing  
>>> integration tests (a broad pass through gradle as a whole). They  
>>> are intended to 1. check that a bunch of classes work together,  
>>> 2. be more focused on a particular behaviour than the other  
>>> integration tests, and 3. be faster to run. In particular, I  
>>> think these sorts of tests are handy for testing error handling.
>>>
>>> So, where should they live? One option is to leave them where  
>>> they are, as they are pretty fast.
>>
>> I think it is a good place where they are now as they are fast. It  
>> might make sense to organize this kind of integration tests into  
>> its own package structure.
>>
> I agree. I'll move them out of the top level package.
>
>> When executing them a test fails:
>>
>> java.lang.AssertionError:
>> Expected: not null
>>      got: null
>>
>>     at org.junit.Assert.assertThat(Assert.java:502)
>>     at org.junit.Assert.assertThat(Assert.java:492)
>>     at org.gradle.AbstractIntegrationTest.getTestBuildFile
>> (AbstractIntegrationTest.java:43)
>>     at  
>> org.gradle.ProjectLoadingIntegrationTest.handlesSimilarlyNamedBuildFi
>> lesInSameDirectory(ProjectLoadingIntegrationTest.java:26)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke
>> (NativeMethodAccessorImpl.java:39)
>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>>     at org.junit.internal.runners.TestMethod.invoke
>> (TestMethod.java:59)
>>     at org.junit.internal.runners.MethodRoadie.runTestMethod
>> (MethodRoadie.java:98)
>>     at org.junit.internal.runners.MethodRoadie$2.run
>> (MethodRoadie.java:79)
>>     at  
>> org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(
>> MethodRoadie.java:87)
>>     at org.junit.internal.runners.MethodRoadie.runTest
>> (MethodRoadie.java:77)
>>     at org.junit.internal.runners.MethodRoadie.run
>> (MethodRoadie.java:42)
>>     at  
>> org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod
>> (JUnit4ClassRunner.java:88)
>>     at org.junit.internal.runners.JUnit4ClassRunner.runMethods
>> (JUnit4ClassRunner.java:51)
>>     at org.junit.internal.runners.JUnit4ClassRunner$1.run
>> (JUnit4ClassRunner.java:44)
>>     at org.junit.internal.runners.ClassRoadie.runUnprotected
>> (ClassRoadie.java:27)
>>     at org.junit.internal.runners.ClassRoadie.runProtected
>> (ClassRoadie.java:37)
>>     at org.junit.internal.runners.JUnit4ClassRunner.run
>> (JUnit4ClassRunner.java:42)
>>     at com.intellij.rt.junit4.Junit4ClassSuite.run
>> (Junit4ClassSuite.java:99)
>>     at com.intellij.rt.execution.junit.JUnitStarter.main
>> (JUnitStarter.java:40)
>>
> You'll need to add src/test/resources to your classpath when you  
> run them, and add *.gradle to the set of resources that intellij  
> copies into the test class dir.  I'll update the wiki to mention this.

That was the reason why the tests have failed. They pass now. When  
you execute ProjectLoadingIntegrationTest as a single test it takes 2  
seconds on my machine which is mostly the Groovy startup and init  
time. You pay this only once when we do a build, as we execute all  
our JUnit tests in one JVM. This makes things slightly less stable  
but improves the performance dramatically.

- Hans

--
Hans Dockter
Gradle Project lead
http://www.gradle.org





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email