src/test or extra test project ?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

src/test or extra test project ?

Dierk König
Here is a small design question:
why do we need a special handling for testing
in the source sets, tasks, configurations, etc?

Having played a little with multi-project setup,
it seems to me that a separate test project
that depends on the main project would
just as well do the job without introducing
any special handling.

IDEs would map this as dependent modules
and would thus provide the same support as
today.

Some tasks may become easier like excluding
test code from coverage, checkstyle,
and documentation.

Any opinions on the benefits/downsides of
this? Has someone ever tried this setup?

cheers
Dierk

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: src/test or extra test project ?

Etienne Studer
This is exactly what we do in our multi-project setup, too. And for the same reasons you listed. It works well for me.

I'd be interested to hear, too, what other developers think and experience.

Regards, Etienne



On 17.10.2010, at 13:50, Dierk König wrote:

> Here is a small design question:
> why do we need a special handling for testing
> in the source sets, tasks, configurations, etc?
>
> Having played a little with multi-project setup,
> it seems to me that a separate test project
> that depends on the main project would
> just as well do the job without introducing
> any special handling.
>
> IDEs would map this as dependent modules
> and would thus provide the same support as
> today.
>
> Some tasks may become easier like excluding
> test code from coverage, checkstyle,
> and documentation.
>
> Any opinions on the benefits/downsides of
> this? Has someone ever tried this setup?
>
> cheers
> Dierk
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

Etienne Studer
Senior Software Developer

Canoo Engineering AG
Kirschgartenstrasse 5
CH-4051 Basel

T +41 61 228 94 44
F +41 61 228 94 49

[hidden email]
www.canoo.com



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: src/test or extra test project ?

rstudner
In reply to this post by Dierk König
Dierk -

would be great if you could post up an example build.gradle (just the stubs) and the lightest directory structure.

I can envision what you are doing here.. and i'm about to embark on a "Gradle Journey" with a 4 module enterprise app.. and i'm trying my best to get as many gotchas taken care of before I jump 2 feet in :)

Roger

On Oct 17, 2010, at 7:50 AM, Dierk König wrote:

> Here is a small design question:
> why do we need a special handling for testing
> in the source sets, tasks, configurations, etc?
>
> Having played a little with multi-project setup,
> it seems to me that a separate test project
> that depends on the main project would
> just as well do the job without introducing
> any special handling.
>
> IDEs would map this as dependent modules
> and would thus provide the same support as
> today.
>
> Some tasks may become easier like excluding
> test code from coverage, checkstyle,
> and documentation.
>
> Any opinions on the benefits/downsides of
> this? Has someone ever tried this setup?
>
> cheers
> Dierk
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: src/test or extra test project ?

Robert Fischer
I go for a hybrid route: I have the most basic unit test code (and
ScalaCheck) living alongside my code, and any kind of integrative or
functional tests living elsewhere.  This allows me to easily run some
key sanity checking code, and to only have to recompile that smaller
subset when I want to check those changes quickly.

~~ Robert.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: src/test or extra test project ?

Dierk König
In reply to this post by rstudner
with the current special testing support, one can do it along these lines:
(all files are one-liners)

settings.gradle // include "main","test"
build.gradle      // allprojects { apply plugin:"java" }
|
+-- main
|     build.gradle // empty placeholder
+-- test
      build.gradle // test.dependsOn ":main:build"

You can start the tests via
 gradle test
.

But my point is a bit different.

The current design makes testing a special concept
and I'm not sure whether this makes things simpler
or more complex...

> I can envision what you are doing here.. and i'm about to embark on a "Gradle Journey" with a 4 module enterprise app.. and i'm trying my best to get as many gotchas taken care of before I jump 2 feet in :)

In this case I would forget all the above and
simply put my tests under src/test/java.

cheers
Dierk
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: src/test or extra test project ?

Steve Ebersole
In reply to this post by Etienne Studer
In my opinion it really all comes down to how confident are you that your
changes did not cause regressions when you build that module/project (with its
tests executed).  Relying on the execution of an external module/project for
that confidence is a recipe for problems in my experience.

We've struggled with this back and forth in Hibernate because most of our
tests are more the functional variety; we have very few unit tests.  For
numerous reasons we tried moving the test suite into a separate module as you
suggest.  In my experience that caused a lot of problems where developers
would not even run the test suite, so be careful.

Logically speaking I think a separation of testing concerns makes sense, but
that does not necessarily mean separate modules.  In Gradle you might have a
unit-test source set and a functional-test source set.  Or you might decide to
keep unit tests with the module they test and move functional-tests to a
separate module.

As long as after you build the module in question you are confident you have
introduced no regressions I think that whatever path you choose here will be
fine.

By the way, I've never experienced the types of difficulties you outline with
regards to IDE set up nor your points about checkstyle, documentation, etc so
hard for me to speak to them specifically


On Sunday, October 17, 2010, at 07:02 am, Etienne Studer wrote:

> This is exactly what we do in our multi-project setup, too. And for the
> same reasons you listed. It works well for me.
>
> I'd be interested to hear, too, what other developers think and experience.
>
> Regards, Etienne
>
> On 17.10.2010, at 13:50, Dierk König wrote:
> > Here is a small design question:
> > why do we need a special handling for testing
> > in the source sets, tasks, configurations, etc?
> >
> > Having played a little with multi-project setup,
> > it seems to me that a separate test project
> > that depends on the main project would
> > just as well do the job without introducing
> > any special handling.
> >
> > IDEs would map this as dependent modules
> > and would thus provide the same support as
> > today.
> >
> > Some tasks may become easier like excluding
> > test code from coverage, checkstyle,
> > and documentation.
> >
> > Any opinions on the benefits/downsides of
> > this? Has someone ever tried this setup?
> >
> > cheers
> > Dierk
> >
> > ---------------------------------------------------------------------
> >
> > To unsubscribe from this list, please visit:
> >    http://xircles.codehaus.org/manage_email
>
> Etienne Studer
> Senior Software Developer
>
> Canoo Engineering AG
> Kirschgartenstrasse 5
> CH-4051 Basel
>
> T +41 61 228 94 44
> F +41 61 228 94 49
>
> [hidden email]
> www.canoo.com
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email

---
Steve Ebersole <[hidden email]>
http://hibernate.org

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: src/test or extra test project ?

Dierk König
Hi Steve,

thanks a lot for sharing your experiences.

Steve Ebersole:

> In my opinion it really all comes down to how confident are you that your
> changes did not cause regressions when you build that module/project (with its
> tests executed).  Relying on the execution of an external module/project for
> that confidence is a recipe for problems in my experience.

The idea would be that executing the tests would not be any different.
Same gradle call, same dependencies.

> We've struggled with this back and forth in Hibernate because most of our
> tests are more the functional variety; we have very few unit tests.  For
> numerous reasons we tried moving the test suite into a separate module as you
> suggest.  In my experience that caused a lot of problems where developers
> would not even run the test suite, so be careful.

Well, as a line of last defense, the tests are executed on the build server
with every commit anyway.
Whoever forgets to run the tests locally and breaks the build will
feel the peer pressure :-)

As a side note: in the early days of XP when we had this discussion, I was
known as the one who proposed an extreme version of _your_ position :-)
with Tests in same source dir as production code and if possible even
in the same file!
This was all to make sure that devs are always aware of the tests.

However, today this has become obsolete since any decent IDE has
a package view that shows production code and their tests
side-by-side not matter what the physical locations are.

Anyway, thanks for your insights and keep gradling Hibernate!
Dierk
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: src/test or extra test project ?

Steve Ebersole
On Sunday, October 17, 2010, at 01:48 pm, Dierk König wrote:
> The idea would be that executing the tests would not be any different.
> Same gradle call, same dependencies.

I am not sure what you mean by "same gradle call, same dependencies" here.  
However, this did open my eyes to another possibility (or maybe this is what
you meant)...  In Gradle you could set it up so that building the main module
forced an execution of the testsuite module as well.  That opens up some
possibilities not available in Maven.  

One main difficulty with Hibernate in regards to its testsuite is the fact
that we need to run that numerous times (against different databases on
varying jvms) in the CI case.  You'd really prefer the module itself to get
built, installed and uploaded just once during this process.  Moving testsuite
to a separate module made that easier.  I started investigating writing a
plugin for Gradle to allow iteration through a set of tests with different
"set ups".  I think I'll need to dust that code off and finish it up now.

---
Steve Ebersole <[hidden email]>
http://hibernate.org

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: src/test or extra test project ?

Adam Murdoch-3
In reply to this post by Dierk König

On 17/10/2010, at 10:50 PM, Dierk König wrote:

Here is a small design question:
why do we need a special handling for testing
in the source sets, tasks, configurations, etc?

A very interesting question.

Beyond the Test task class itself, there's actually very little special handling for testing in the implementation of Gradle. Almost all the support comes from the generic source set concept. The only real special handling is in the java plugin, which adds a 'test' source set, which has a dependency on the 'main' source set.



Having played a little with multi-project setup,
it seems to me that a separate test project
that depends on the main project would
just as well do the job without introducing
any special handling.

IDEs would map this as dependent modules
and would thus provide the same support as
today.

Some tasks may become easier like excluding
test code from coverage, checkstyle,
and documentation.

Introducing the concept of test code also makes this stuff easier for the majority of projects, because we can make use of certain conventions about what test code means. That is, you generally don't want to report on the coverage of your test code, or to include it in the API documentation, or publish a jar for it, and so on.

But on the other hand, using a separate project for test code might help highlight that all code is equally important for delivery, whether it is test or production code.

Personally, I think that test code is an important concept for a build tool to model. People do distinguish between the test and production code in how they talk about the code, what they expect the code will do, how it will participate in the build. It would be a shame not to model that reality.

However, there are some awkward aspects to how we model things now, which suggest things might be done better.

One example is that there are two distinct task namespaces. The first part of a task's fully qualified path is a hierarchical project path. The second part is a munging of the source set name and task name. So you have task paths like :core:compileTestJava. Whereas, it might be better to have a task path like :core:test:java:compile.

Another example is that it is difficult to apply behaviour to certain source sets and not others. One solution to this which we've discussed is to be able to apply plugins to source sets. So you could do this kind of thing:

sourceSets {
    main {
        apply plugin: 'antlr' // use Antlr only for production code
    }
    test {
        apply plugin: 'groovy' // use Groovy only for testing
    }
    functionalTest {
        apply plugin: 'functional-test'
    }
}

if, instead, the main, test and functionalTest source sets where separate projects, this would work more naturally.

Another awkwardness is when you want to model things such as a project which has only tests. You apply the 'java-base' plugin and then add in a 'test' source set and a 'test' task. If, instead the tests were in a separate project, we could have a 'test' plugin which you could apply, and there's be no need to use the *-base plugins.

Perhaps an improvement to our model would be to treat source sets as a kind of project. Or flip it around, so that what is currently a project is simply a composite source set. Or both.

The result would be that each logical project would contain a 'main' subproject and a 'test' subproject. Each of the 'main' and 'test' projects would have their own 'classes', 'compileJava' and 'clean' tasks and so on. And their own 'compile' and 'runtime' configurations. And their own build directories. And their own jar publications. And their own build files.

The logical project would simply be an aggregate project. It would inject configuration into the child projects. Or not - they might have their build files.

One thing to note is that this structure would be by convention only. You'd be able to combine the various types of projects into whatever structure you'd like.

It's a very interesting question, indeed.


--
Adam Murdoch
Gradle Developer
http://www.gradle.org
CTO, Gradle Inc. - Gradle Training, Support, Consulting
http://www.gradle.biz

Reply | Threaded
Open this post in threaded view
|

Re: src/test or extra test project ?

cmtopinka
This post has NOT been accepted by the mailing list yet.
This post was updated on .
In reply to this post by Dierk König
I'm trying to convert a multi-module legacy project that's setup a bit differently mostly because of the way we use subversion. Project modules are checked out flat into the Eclipse workspace.  Tests are all contained in one module. So far the project seems to build fine but I'm having problems using the test task. Can you suggest a different route? Thanks

I've started down this road.

main (web app)
  settings.gradle // include "module","test"
  build.gradle      // allprojects { apply plugin:"java" }
module (services, utils, etc)
  modulename.gradle //dependencies {compile project(':dependentModule')}
test
  test.gradle //sourceSets {test {java { srcDir 'src' } resources { srcDir 'resources' }}}
                  // not sure what else goes here.


My settings.gradle looks like this
####################################
include 'modulea', 'moduleb'

rootProject.name = 'webapp'
rootProject.children.each {project ->
    String fileBaseName = project.name.toLowerCase()
    project.buildFileName = "${fileBaseName}.gradle"
        project.projectDir = new File(settingsDir.parent, project.name)
    assert project.projectDir.isDirectory()
        assert project.buildFile.isFile()
}
########################################

build.gradle is headed this direction

#########################################
apply plugin: 'war'

allprojects {
        apply plugin: 'java'

        repositories {
                mavenCentral()
                maven {
                        url "http://repository.codehaus.org/"
                }
        }

        dependencies {
                compile 'org.springframework:spring-webmvc:3.2.3.RELEASE'
                    .
                    .
                    .

                testCompile 'junit:junit:4.+'
                testCompile 'org.testng:testng:6.8.5'
                testCompile 'org.springframework:spring-test:3.2.3.RELEASE'
               
        }
       
        test {
                useTestNG()
        }
}

subprojects {
        dependencies {
                compile 'log4j:log4j:1.2.17'
                    .
                    .
                    .
        }
}

dependencies {
        compile project(':modulea')
        compile project(':moduleb')

        providedCompile 'log4j:log4j:1.2.17'
        providedCompile 'org.apache.tomcat:servlet-api:6.0.37'
       
}


sourceSets {
        main {
                java { srcDir 'JavaSource' }
                resources { srcDir 'resources' }
        }
}

#########################################