Locale dependent test cases

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Locale dependent test cases

Harald Schmitt
Hello,

the integration test case
org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze
bad code"() does not pass with de Locale (and some others) because it
checks for an exception message, that is localized.
    failure.error.contains("Name 'class1' must match pattern")

When I build gradle the :codeQuality:integTest fails because of that.
What is the preffered solution?
1) Test for the German message, too
  || failure.error.contains("'class1' entspricht nicht dem Muster")
2) Add an annotation that this test is only run with "en" locale
3) Load the localized message from checkstyle code, which adds a
dependency to that jar
4) Other?

Best regards,
Harald

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Harald Schmitt
Am 17.07.2014 10:16, schrieb Harald Schmitt:

> Hello,
>
> the integration test case
> org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze
> bad code"() does not pass with de Locale (and some others) because it
> checks for an exception message, that is localized.
>     failure.error.contains("Name 'class1' must match pattern")
>
> When I build gradle the :codeQuality:integTest fails because of that.
> What is the preffered solution?
> 1) Test for the German message, too
>   || failure.error.contains("'class1' entspricht nicht dem Muster")
> 2) Add an annotation that this test is only run with "en" locale
I went ahead and implemented and tested this as
    @Requires(TestPrecondition.LANGUAGE_EN)
If you like this solution I can submit it, right away.

> 3) Load the localized message from checkstyle code, which adds a
> dependency to that jar
> 4) Other?
>
> Best regards,
> Harald
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Luke Daley-2
It would be better to explicitly set the locale for the test builds so we don’t battle these issues one by one.

This would probably require some changes in Gradle to forward the locale setting for all forked processes. We would also need to add locale handling to the daemon matching.

@Devs: do you think this is worth doing? 

I can see a case for a multi national development team wanting to enforce that the build runs with a consistent locale.

On 18 July 2014 at 1:10:33 am, Harald Schmitt ([hidden email]) wrote:

Am 17.07.2014 10:16, schrieb Harald Schmitt:

> Hello,
>
> the integration test case
> org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze
> bad code"() does not pass with de Locale (and some others) because it
> checks for an exception message, that is localized.
> failure.error.contains("Name 'class1' must match pattern")
>
> When I build gradle the :codeQuality:integTest fails because of that.
> What is the preffered solution?
> 1) Test for the German message, too
> || failure.error.contains("'class1' entspricht nicht dem Muster")
> 2) Add an annotation that this test is only run with "en" locale
I went ahead and implemented and tested this as
@Requires(TestPrecondition.LANGUAGE_EN)
If you like this solution I can submit it, right away.

> 3) Load the localized message from checkstyle code, which adds a
> dependency to that jar
> 4) Other?
>
> Best regards,
> Harald
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
> http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Radim Kubacki
On Fri, Jul 18, 2014 at 1:23 AM, Luke Daley <[hidden email]> wrote:
It would be better to explicitly set the locale for the test builds so we don’t battle these issues one by one.

This would probably require some changes in Gradle to forward the locale setting for all forked processes. We would also need to add locale handling to the daemon matching.

@Devs: do you think this is worth doing? 

I can see a case for a multi national development team wanting to enforce that the build runs with a consistent locale.

Setting locale explicitly for test builds sounds like good idea to me. The consistency is really key thing.  

On 18 July 2014 at 1:10:33 am, Harald Schmitt ([hidden email]) wrote:

Am 17.07.2014 10:16, schrieb Harald Schmitt:

> Hello,
>
> the integration test case
> org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze
> bad code"() does not pass with de Locale (and some others) because it
> checks for an exception message, that is localized.
> failure.error.contains("Name 'class1' must match pattern")
>
> When I build gradle the :codeQuality:integTest fails because of that.
> What is the preffered solution?
> 1) Test for the German message, too
> || failure.error.contains("'class1' entspricht nicht dem Muster")
> 2) Add an annotation that this test is only run with "en" locale
I went ahead and implemented and tested this as
@Requires(TestPrecondition.LANGUAGE_EN)
If you like this solution I can submit it, right away.

> 3) Load the localized message from checkstyle code, which adds a
> dependency to that jar
> 4) Other?
>
> Best regards,
> Harald
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
> http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Harald Schmitt
Am 18.07.2014 08:36, schrieb Radim Kubacki:
> On Fri, Jul 18, 2014 at 1:23 AM, Luke Daley <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     It would be better to explicitly set the locale for the test builds
>     so we don’t battle these issues one by one.
But your test's should verify that gradle runs under different locales.
If you force your tests to a single locale you loose that.
I think it is a bad idea to tweak the test environment to pass the tests.

>
>     This would probably require some changes in Gradle to forward the
>     locale setting for all forked processes. We would also need to add
>     locale handling to the daemon matching.
>
>     @Devs: do you think this is worth doing?
>
>     I can see a case for a multi national development team wanting to
>     enforce that the build runs with a consistent locale.
>
> Setting locale explicitly for test builds sounds like good idea to me.
> The consistency is really key thing.  
>
>     On 18 July 2014 at 1:10:33 am, Harald Schmitt ([hidden email]
>     <mailto:[hidden email]>) wrote:
>
>>     Am 17.07.2014 10:16, schrieb Harald Schmitt:
>>     > Hello,
>>     >  
>>     > the integration test case
>>     > org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze
>>     > bad code"() does not pass with de Locale (and some others) because it
>>     > checks for an exception message, that is localized.
>>     >     failure.error.contains("Name 'class1' must match pattern")
>>     >  
>>     > When I build gradle the :codeQuality:integTest fails because of that.
>>     > What is the preffered solution?
>>     > 1) Test for the German message, too
>>     >   || failure.error.contains("'class1' entspricht nicht dem Muster")
>>     > 2) Add an annotation that this test is only run with "en" locale
>>     I went ahead and implemented and tested this as
>>     @Requires(TestPrecondition.LANGUAGE_EN)
>>     If you like this solution I can submit it, right away.
>>     > 3) Load the localized message from checkstyle code, which adds a
>>     > dependency to that jar
>>     > 4) Other?
>>     >  
>>     > Best regards,
>>     > Harald
>>     >  
>>     > ---------------------------------------------------------------------
>>     > To unsubscribe from this list, please visit:
>>     >  
>>     >     http://xircles.codehaus.org/manage_email
>>     >  
>>     >  
>>     >  
>>
>>
>>     ---------------------------------------------------------------------
>>     To unsubscribe from this list, please visit:
>>
>>     http://xircles.codehaus.org/manage_email
>>
>>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Luke Daley-2


On 18 July 2014 at 6:10:52 pm, Harald Schmitt ([hidden email]) wrote:

Am 18.07.2014 08:36, schrieb Radim Kubacki: 
> On Fri, Jul 18, 2014 at 1:23 AM, Luke Daley <[hidden email] 
> <mailto:[hidden email]>> wrote: 
> 
> It would be better to explicitly set the locale for the test builds 
> so we don’t battle these issues one by one. 
But your test's should verify that gradle runs under different locales. 
If you force your tests to a single locale you loose that. 
I think it is a bad idea to tweak the test environment to pass the tests. 

Then we should explicitly test locale support where it matters.

Forcing the locale is better than using @Requires(TestPrecondition.LANGUAGE_EN) to just ignore the test. 



> 
> This would probably require some changes in Gradle to forward the 
> locale setting for all forked processes. We would also need to add 
> locale handling to the daemon matching. 
> 
> @Devs: do you think this is worth doing? 
> 
> I can see a case for a multi national development team wanting to 
> enforce that the build runs with a consistent locale. 
> 
> Setting locale explicitly for test builds sounds like good idea to me. 
> The consistency is really key thing. 
> 
> On 18 July 2014 at 1:10:33 am, Harald Schmitt ([hidden email] 
> <mailto:[hidden email]>) wrote: 
> 
>> Am 17.07.2014 10:16, schrieb Harald Schmitt: 
>> > Hello, 
>> > 
>> > the integration test case 
>> > org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze 
>> > bad code"() does not pass with de Locale (and some others) because it 
>> > checks for an exception message, that is localized. 
>> > failure.error.contains("Name 'class1' must match pattern") 
>> > 
>> > When I build gradle the :codeQuality:integTest fails because of that. 
>> > What is the preffered solution? 
>> > 1) Test for the German message, too 
>> > || failure.error.contains("'class1' entspricht nicht dem Muster") 
>> > 2) Add an annotation that this test is only run with "en" locale 
>> I went ahead and implemented and tested this as 
>> @Requires(TestPrecondition.LANGUAGE_EN) 
>> If you like this solution I can submit it, right away. 
>> > 3) Load the localized message from checkstyle code, which adds a 
>> > dependency to that jar 
>> > 4) Other? 
>> > 
>> > Best regards, 
>> > Harald 
>> > 
>> > --------------------------------------------------------------------- 
>> > To unsubscribe from this list, please visit: 
>> > 
>> > http://xircles.codehaus.org/manage_email 
>> > 
>> > 
>> > 
>> 
>> 
>> --------------------------------------------------------------------- 
>> To unsubscribe from this list, please visit: 
>> 
>> http://xircles.codehaus.org/manage_email 
>> 
>> 
> 


--------------------------------------------------------------------- 
To unsubscribe from this list, please visit: 

http://xircles.codehaus.org/manage_email 



Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Radim Kubacki
In reply to this post by Harald Schmitt
On Fri, Jul 18, 2014 at 10:10 AM, Harald Schmitt <[hidden email]> wrote:
Am 18.07.2014 08:36, schrieb Radim Kubacki:
> On Fri, Jul 18, 2014 at 1:23 AM, Luke Daley <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     It would be better to explicitly set the locale for the test builds
>     so we don’t battle these issues one by one.
But your test's should verify that gradle runs under different locales.
If you force your tests to a single locale you loose that.
I think it is a bad idea to tweak the test environment to pass the tests.

There is a big problem with the default locale for the VM. It affects a lot of things (l10n messages, sorting, formatting, ...). If you decide to tweak it during the test run it can cause interference between your tests. Of course it would help if your code does not depend on default locale but this is not always the case (think about all 3rd party libraries for example). In such case it can be OK to run one test task with locale X and another task to run them with locale Y. I mean it should be possible to set locale for the (unit/other-)test or do it on task level to set it for the test runner. In both cases we want to achieve consistent/reproducible build. User's default locale settings is external dependency that needs to be factored out.
>
>     This would probably require some changes in Gradle to forward the
>     locale setting for all forked processes. We would also need to add
>     locale handling to the daemon matching.
>
>     @Devs: do you think this is worth doing?
>
>     I can see a case for a multi national development team wanting to
>     enforce that the build runs with a consistent locale.
>
> Setting locale explicitly for test builds sounds like good idea to me.
> The consistency is really key thing.
>
>     On 18 July 2014 at 1:10:33 am, Harald Schmitt ([hidden email]
>     <mailto:[hidden email]>) wrote:
>
>>     Am 17.07.2014 10:16, schrieb Harald Schmitt:
>>     > Hello,
>>     >
>>     > the integration test case
>>     > org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze
>>     > bad code"() does not pass with de Locale (and some others) because it
>>     > checks for an exception message, that is localized.
>>     >     failure.error.contains("Name 'class1' must match pattern")
>>     >
>>     > When I build gradle the :codeQuality:integTest fails because of that.
>>     > What is the preffered solution?
>>     > 1) Test for the German message, too
>>     >   || failure.error.contains("'class1' entspricht nicht dem Muster")
>>     > 2) Add an annotation that this test is only run with "en" locale
>>     I went ahead and implemented and tested this as
>>     @Requires(TestPrecondition.LANGUAGE_EN)
>>     If you like this solution I can submit it, right away.
>>     > 3) Load the localized message from checkstyle code, which adds a
>>     > dependency to that jar
>>     > 4) Other?
>>     >
>>     > Best regards,
>>     > Harald
>>     >
>>     > ---------------------------------------------------------------------
>>     > To unsubscribe from this list, please visit:
>>     >
>>     >     http://xircles.codehaus.org/manage_email
>>     >
>>     >
>>     >
>>
>>
>>     ---------------------------------------------------------------------
>>     To unsubscribe from this list, please visit:
>>
>>     http://xircles.codehaus.org/manage_email
>>
>>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Harald Schmitt
Am 18.07.2014 10:29, schrieb Radim Kubacki:

> On Fri, Jul 18, 2014 at 10:10 AM, Harald Schmitt <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Am 18.07.2014 08:36, schrieb Radim Kubacki:
>     > On Fri, Jul 18, 2014 at 1:23 AM, Luke Daley
>     <[hidden email] <mailto:[hidden email]>
>     > <mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     >
>     >     It would be better to explicitly set the locale for the test
>     builds
>     >     so we don’t battle these issues one by one.
>     But your test's should verify that gradle runs under different locales.
>     If you force your tests to a single locale you loose that.
>     I think it is a bad idea to tweak the test environment to pass the
>     tests.
>
>
> There is a big problem with the default locale for the VM. It affects a
> lot of things (l10n messages, sorting, formatting, ...). If you decide
> to tweak it during the test run it can cause interference between your
> tests. Of course it would help if your code does not depend on default
> locale but this is not always the case (think about all 3rd party
> libraries for example).
IMHO that is a big argument to do CI builds with different environments
(default locales).
> In such case it can be OK to run one test task
> with locale X and another task to run them with locale Y. I mean it
> should be possible to set locale for the (unit/other-)test or do it on
> task level to set it for the test runner. In both cases we want to
> achieve consistent/reproducible build. User's default locale settings is
> external dependency that needs to be factored out.
Or is something that has to be tested. You don't want to force a gradle
user to change his default locale, right? Then you should run test with
your user's settings.

>
>     >
>     >     This would probably require some changes in Gradle to forward the
>     >     locale setting for all forked processes. We would also need to add
>     >     locale handling to the daemon matching.
>     >
>     >     @Devs: do you think this is worth doing?
>     >
>     >     I can see a case for a multi national development team wanting to
>     >     enforce that the build runs with a consistent locale.
>     >
>     > Setting locale explicitly for test builds sounds like good idea to me.
>     > The consistency is really key thing.
>     >
>     >     On 18 July 2014 at 1:10:33 am, Harald Schmitt
>     ([hidden email] <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>) wrote:
>     >
>     >>     Am 17.07.2014 10:16, schrieb Harald Schmitt:
>     >>     > Hello,
>     >>     >
>     >>     > the integration test case
>     >>     >
>     org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze
>     >>     > bad code"() does not pass with de Locale (and some others)
>     because it
>     >>     > checks for an exception message, that is localized.
>     >>     >     failure.error.contains("Name 'class1' must match pattern")
>     >>     >
>     >>     > When I build gradle the :codeQuality:integTest fails
>     because of that.
>     >>     > What is the preffered solution?
>     >>     > 1) Test for the German message, too
>     >>     >   || failure.error.contains("'class1' entspricht nicht dem
>     Muster")
>     >>     > 2) Add an annotation that this test is only run with "en"
>     locale
>     >>     I went ahead and implemented and tested this as
>     >>     @Requires(TestPrecondition.LANGUAGE_EN)
>     >>     If you like this solution I can submit it, right away.
>     >>     > 3) Load the localized message from checkstyle code, which
>     adds a
>     >>     > dependency to that jar
>     >>     > 4) Other?
>     >>     >
>     >>     > Best regards,
>     >>     > Harald
>     >>     >
>     >>     >
>     ---------------------------------------------------------------------
>     >>     > To unsubscribe from this list, please visit:
>     >>     >
>     >>     >     http://xircles.codehaus.org/manage_email
>     >>     >
>     >>     >
>     >>     >
>     >>
>     >>
>     >>    
>     ---------------------------------------------------------------------
>     >>     To unsubscribe from this list, please visit:
>     >>
>     >>     http://xircles.codehaus.org/manage_email
>     >>
>     >>
>     >
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe from this list, please visit:
>
>         http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Adam Murdoch
In reply to this post by Radim Kubacki

On 18 Jul 2014, at 6:29 pm, Radim Kubacki <[hidden email]> wrote:

On Fri, Jul 18, 2014 at 10:10 AM, Harald Schmitt <[hidden email]> wrote:
Am 18.07.2014 08:36, schrieb Radim Kubacki:
> On Fri, Jul 18, 2014 at 1:23 AM, Luke Daley <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     It would be better to explicitly set the locale for the test builds
>     so we don’t battle these issues one by one.
But your test's should verify that gradle runs under different locales.
If you force your tests to a single locale you loose that.
I think it is a bad idea to tweak the test environment to pass the tests.

There is a big problem with the default locale for the VM. It affects a lot of things (l10n messages, sorting, formatting, ...). If you decide to tweak it during the test run it can cause interference between your tests.

This is exactly the kind of thing GradleExecuter takes care of. The test would declare that it expects the build to be run with ‘en’ as the language, and the executer would either run in-process, if the current JVM’s language is ‘en’ or would run the build in a forked process with the appropriate -D options.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
CTO Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com



Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Adam Murdoch
In reply to this post by Harald Schmitt

On 18 Jul 2014, at 1:10 am, Harald Schmitt <[hidden email]> wrote:

Am 17.07.2014 10:16, schrieb Harald Schmitt:
Hello,

the integration test case
org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze
bad code"() does not pass with de Locale (and some others) because it
checks for an exception message, that is localized.
   failure.error.contains("Name 'class1' must match pattern")

When I build gradle the :codeQuality:integTest fails because of that.
What is the preffered solution?
1) Test for the German message, too
 || failure.error.contains("'class1' entspricht nicht dem Muster")
2) Add an annotation that this test is only run with "en" locale
I went ahead and implemented and tested this as
   @Requires(TestPrecondition.LANGUAGE_EN)
If you like this solution I can submit it, right away.

I’d rather you added something like GradleExecuter.withDefaultLanguage() that the test can use, similar to withDefaultCharacterEncoding().

The InProcessGradleExecuter.assertCanExecute() method can assert that the requested language == the actual language.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
CTO Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com



Reply | Threaded
Open this post in threaded view
|

Re: Locale dependent test cases

Harald Schmitt
Am 22.07.2014 um 02:17 schrieb Adam Murdoch:

>
> On 18 Jul 2014, at 1:10 am, Harald Schmitt <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>> Am 17.07.2014 10:16, schrieb Harald Schmitt:
>>> Hello,
>>>
>>> the integration test case
>>> org.gradle.api.plugins.quality.CheckstylePluginIntegrationTest."analyze
>>> bad code"() does not pass with de Locale (and some others) because it
>>> checks for an exception message, that is localized.
>>>    failure.error.contains("Name 'class1' must match pattern")
>>>
>>> When I build gradle the :codeQuality:integTest fails because of that.
>>> What is the preffered solution?
>>> 1) Test for the German message, too
>>>  || failure.error.contains("'class1' entspricht nicht dem Muster")
>>> 2) Add an annotation that this test is only run with "en" locale
>> I went ahead and implemented and tested this as
>>    @Requires(TestPrecondition.LANGUAGE_EN)
>> If you like this solution I can submit it, right away.
>
> I’d rather you added something like
> GradleExecuter.withDefaultLanguage() that the test can use, similar
> to withDefaultCharacterEncoding().
>
> The InProcessGradleExecuter.assertCanExecute() method can assert that
> the requested language == the actual language.
That makes sense, I implemented it and made a pull request:
https://github.com/gradle/gradle/pull/303

>
>
> --
> Adam Murdoch
> Gradle Co-founder
> http://www.gradle.org
> CTO Gradleware Inc. - Gradle Training, Support, Consulting
> http://www.gradleware.com
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email