Standardising the JUnit XML test report format
Locked
 
|
Hi All,
I have noticed that the JUnit XML reports generated by Gradle are not being correctly parsed by Jenkins and other applications when a test is skipped. This is in relation to http://issues.gradle.org/browse/GRADLE-2731. From what I can find out there is no 'standard' format that these reports produce, however Ant and Maven(at least) have standardised on the format described in the issue and the linked StackOverflow post.
So when a test is skipped in Gradle it produces(for example):
<testsuite name="com.foo.FooTest" tests="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
<ignored-testcase name="some skipped test" classname="com.foo.FooTest" time="0.01"/>
<system-out><![CDATA[]]></system-out>
<system-err><![CDATA[]]></system-err> </testsuite> However when Ant runs the same test(changes highlighted):
<testsuite name="com.foo.FooTest" tests="1"skipped="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
<testcase name="some skipped test" classname="com.foo.FooTest" time="0.01"> <skipped/>
</testcase> <system-out><![CDATA[]]></system-out>
<system-err><![CDATA[]]></system-err> </testsuite> Ant generates a normal testcase tag and adds a child element; and Ant also generates a skipped attribute. This is compared to the ignored-testcase.
The only concern that I have with changing this output would be any tool that has changed to only support the Gradle format.
I am writing a fix for this so is there anything that I have missed or overlooked in the above.
I hope that I can help out,
Lee
|
Re: Standardising the JUnit XML test report format
|
|
We've been pretty conservative about making changes to this in the past.
We did make a change a little while ago to add the ability to track output per test case, and we added an opt in switch for it (http://www.gradle.org/docs/current/javadoc/org/gradle/api/tasks/testing/JUnitXmlReport.html#setOutputPerTestCase(boolean)). In this case though, I think we should just make the change after testing for compatibility with different tools. The chances that something has specific handling for Gradle's format is low. Can anyone recall why we deviate from Ant and Maven here? On 9 Feb 2014, at 5:32 pm, Lee Symes <[hidden email]> wrote: > Hi All, > > I have noticed that the JUnit XML reports generated by Gradle are not being correctly parsed by Jenkins and other applications when a test is skipped. This is in relation to http://issues.gradle.org/browse/GRADLE-2731. From what I can find out there is no 'standard' format that these reports produce, however Ant and Maven(at least) have standardised on the format described in the issue and the linked StackOverflow post. > > So when a test is skipped in Gradle it produces(for example): > <testsuite name="com.foo.FooTest" tests="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045"> > <ignored-testcase name="some skipped test" classname="com.foo.FooTest" time="0.01"/> > <system-out><![CDATA[]]></system-out> > <system-err><![CDATA[]]></system-err> > </testsuite> > > However when Ant runs the same test(changes highlighted): > <testsuite name="com.foo.FooTest" tests="1"skipped="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045"> > <testcase name="some skipped test" classname="com.foo.FooTest" time="0.01"> > <skipped/> > </testcase> > <system-out><![CDATA[]]></system-out> > <system-err><![CDATA[]]></system-err> > </testsuite> > > Ant generates a normal testcase tag and adds a child element; and Ant also generates a skipped attribute. This is compared to the ignored-testcase. > > The only concern that I have with changing this output would be any tool that has changed to only support the Gradle format. > > I am writing a fix for this so is there anything that I have missed or overlooked in the above. > > I hope that I can help out, > > Lee -- Luke Daley Principal Engineer, Gradleware http://gradleware.com --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
|
|
Hi again, I've committed my code into my fork of Gradle(https://github.com/leesdolphin/gradle/compare). In my searches though I have noticed that there was a pull request(https://github.com/gradle/gradle/pull/183) to make effectively the same changes. This pull request was shelved for backwards compatibility reasons.
I have also changed the test specs & helper objects to include the changes to annotations and element naming. I'll submit a pull request tomorrow once I have submitted my CLA. Yours, Lee On 10 February 2014 07:34, Luke Daley <[hidden email]> wrote: We've been pretty conservative about making changes to this in the past. |
Re: Standardising the JUnit XML test report format
|
|
Hi Lee,
Did you submit this pull request? We'd like to get this into 1.12. > Lee Symes <mailto:[hidden email]> > 12 February 2014 9:29 pm > Hi again, > > I've committed my code into my fork of > Gradle(https://github.com/leesdolphin/gradle/compare). In my searches > though I have noticed that there was a pull > request(https://github.com/gradle/gradle/pull/183) to make effectively > the same changes. This pull request was shelved for backwards > compatibility reasons. > > I have added a skipped counter to the JUnit TestClassResult. Also > changed the generation code to: > > * Add the skipped annotation. > * Output the ignored test case element name as testcase and added an > empty sub element <skipped/> > > I have also changed the test specs & helper objects to include the > changes to annotations and element naming. > > I'll submit a pull request tomorrow once I have submitted my CLA. > > Yours, > > Lee > > > > > Luke Daley <mailto:[hidden email]> > 10 February 2014 7:34 am > We've been pretty conservative about making changes to this in the past. > > We did make a change a little while ago to add the ability to track > output per test case, and we added an opt in switch for it > (http://www.gradle.org/docs/current/javadoc/org/gradle/api/tasks/testing/JUnitXmlReport.html#setOutputPerTestCase(boolean)). > In this case though, I think we should just make the change after > testing for compatibility with different tools. The chances that > something has specific handling for Gradle's format is low. > > Can anyone recall why we deviate from Ant and Maven here? > > > Lee Symes <mailto:[hidden email]> > 9 February 2014 5:32 pm > Hi All, > > I have noticed that the JUnit XML reports generated by Gradle are not > being correctly parsed by Jenkins and other applications when a test > is skipped. This is in relation to > http://issues.gradle.org/browse/GRADLE-2731. From what I can find out > there is no 'standard' format that these reports produce, however Ant > and Maven(at least) have standardised on the format described in the > issue and the linked StackOverflow post > <http://stackoverflow.com/questions/4922867/junit-xml-format-specification-that-hudson-supports>. > > > So when a test is skipped in Gradle it produces(for example): > <testsuite name="com.foo.FooTest" tests="1" failures="0" errors="0" > timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045"> > <ignored-testcase name="some skipped test" classname="com.foo.FooTest" > time="0.01"/> > <system-out><![CDATA[]]></system-out> > <system-err><![CDATA[]]></system-err> > </testsuite> > > However when Ant runs the same test(changes highlighted): > <testsuite name="com.foo.FooTest" tests="1"*skipped="1"* failures="0" > errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" > time="0.045"> > <*testcase* name="some skipped test" classname="com.foo.FooTest" > time="0.01"> > *<skipped/>* > </testcase> > <system-out><![CDATA[]]></system-out> > <system-err><![CDATA[]]></system-err> > </testsuite> > > Ant generates a normal testcase tag and adds a child element; and Ant > also generates a skipped attribute. This is compared to the > ignored-testcase. > > The only concern that I have with changing this output would be any > tool that has changed to only support the Gradle format. > > I am writing a fix for this so is there anything that I have missed or > overlooked in the above. > > I hope that I can help out, > > Lee --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
|
|
Hi Luke, My apologies; It had slipped my mind! I have sent the CLA and added a pull request(https://github.com/gradle/gradle/pull/247).
Yours, Lee On 17 February 2014 11:34, Luke Daley <[hidden email]> wrote: Hi Lee, |
«
Return to gradle-dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |