Failures with non-ascii filenames on OSX

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Failures with non-ascii filenames on OSX

Daz DeBoer-2
G'day
I'm getting failures running the new non-ascii character test in MavenHttpRepoResolveIntegrationTest on my Mac. When debugging, it seems like File.getCanonicalPath() is converting all non-ascii characters to '?'.

I've got 'file.encoding' set to 'UTF-8' for the test process, and I'm running with JDK 1.7.0_25.

Is anyone else seeing this problem? Any hints?

Thanks
Daz

Reply | Threaded
Open this post in threaded view
|

Re: Failures with non-ascii filenames on OSX

Adam Murdoch

On 28 Apr 2014, at 2:56 am, Daz DeBoer <[hidden email]> wrote:

G'day
I'm getting failures running the new non-ascii character test in MavenHttpRepoResolveIntegrationTest on my Mac. When debugging, it seems like File.getCanonicalPath() is converting all non-ascii characters to '?'.

I've got 'file.encoding' set to 'UTF-8' for the test process, and I'm running with JDK 1.7.0_25.

Is anyone else seeing this problem? Any hints?

I’ll take a look.

Do the non-ascii tests in the ‘ivy’ and ‘maven’ projects work for you?


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com

Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA: http://www.gradlesummit.com

Reply | Threaded
Open this post in threaded view
|

Re: Failures with non-ascii filenames on OSX

Adam Murdoch
In reply to this post by Daz DeBoer-2

Should be fixed now. You’ll need to run `./gradlew ideaWorkspace` to configure things correctly to make them work in IDEA.

On 28 Apr 2014, at 2:56 am, Daz DeBoer <[hidden email]> wrote:

G'day
I'm getting failures running the new non-ascii character test in MavenHttpRepoResolveIntegrationTest on my Mac. When debugging, it seems like File.getCanonicalPath() is converting all non-ascii characters to '?'.

I've got 'file.encoding' set to 'UTF-8' for the test process, and I'm running with JDK 1.7.0_25.

Is anyone else seeing this problem? Any hints?

Thanks
Daz



--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com

Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA: http://www.gradlesummit.com

Reply | Threaded
Open this post in threaded view
|

Re: Failures with non-ascii filenames on OSX

Daz DeBoer-2
In reply to this post by Adam Murdoch

Yep, that's the only test that fails. I think that gradle is handling things fine, but our test fixtures use the file name, which is reported incorrectly.
The fixtures don't actually need to read the file name, since they create the file in the first place. So that's a potential avenue for a fix. I did something like that to get the ivy fixtures handling layouts better.
But I'm hopeful there's a simple JVM/OS fix.
Daz

On 27 Apr 2014 15:51, "Adam Murdoch" <[hidden email]> wrote:

On 28 Apr 2014, at 2:56 am, Daz DeBoer <[hidden email]> wrote:

G'day
I'm getting failures running the new non-ascii character test in MavenHttpRepoResolveIntegrationTest on my Mac. When debugging, it seems like File.getCanonicalPath() is converting all non-ascii characters to '?'.

I've got 'file.encoding' set to 'UTF-8' for the test process, and I'm running with JDK 1.7.0_25.

Is anyone else seeing this problem? Any hints?

I’ll take a look.

Do the non-ascii tests in the ‘ivy’ and ‘maven’ projects work for you?


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com

Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA: http://www.gradlesummit.com

Reply | Threaded
Open this post in threaded view
|

Re: Failures with non-ascii filenames on OSX

Daz DeBoer-2
In reply to this post by Adam Murdoch

On Mon, Apr 28, 2014 at 1:18 AM, Adam Murdoch <[hidden email]> wrote:

Should be fixed now. You’ll need to run `./gradlew ideaWorkspace` to configure things correctly to make them work in IDEA.

Yep, all working for me now. Thanks for sorting that out: I think I could have struggled with that for a while!
 

On 28 Apr 2014, at 2:56 am, Daz DeBoer <[hidden email]> wrote:

G'day
I'm getting failures running the new non-ascii character test in MavenHttpRepoResolveIntegrationTest on my Mac. When debugging, it seems like File.getCanonicalPath() is converting all non-ascii characters to '?'.

I've got 'file.encoding' set to 'UTF-8' for the test process, and I'm running with JDK 1.7.0_25.

Is anyone else seeing this problem? Any hints?

Thanks
Daz



--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com

Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA: http://www.gradlesummit.com




--
Reply | Threaded
Open this post in threaded view
|

Re: Failures with non-ascii filenames on OSX

Adam Murdoch
In reply to this post by Daz DeBoer-2

On 29 Apr 2014, at 12:59 am, Daz DeBoer <[hidden email]> wrote:

Yep, that's the only test that fails. I think that gradle is handling things fine, but our test fixtures use the file name, which is reported incorrectly.


It does expose some potential problems in what Gradle does. The base problem here is that the names used as input in the build script were composed differently to the expected outputs in the test fixture, because the hfs+ file system normalises file names (we happened to use an accented character as test input, and these can be composed in a few different ways in unicode).

So, you could see a case where the strings of a particular (group, module, version) are composed differently in various places - in a build script, in different pom or ivy.xml files, in the cache on the file system, etc. This means, the strings all look the same to a human in each place but are represented using different unicode characters, so are not equal as far as String.equals() and String.hashcode() are concerned.

This would cause some subtle and difficult to track down problems. For example, we might always download a module because we never find it in the cache, or conflict resolution might include a module multiple times because it sees a bunch of different identifiers. The problem is made tricky because the names all look the same when displayed to a human.

Net result is that we should be normalising all the strings we receive as input.


The fixtures don't actually need to read the file name, since they create the file in the first place. So that's a potential avenue for a fix. I did something like that to get the ivy fixtures handling layouts better.
But I'm hopeful there's a simple JVM/OS fix.
Daz

On 27 Apr 2014 15:51, "Adam Murdoch" <[hidden email]> wrote:

On 28 Apr 2014, at 2:56 am, Daz DeBoer <[hidden email]> wrote:

G'day
I'm getting failures running the new non-ascii character test in MavenHttpRepoResolveIntegrationTest on my Mac. When debugging, it seems like File.getCanonicalPath() is converting all non-ascii characters to '?'.

I've got 'file.encoding' set to 'UTF-8' for the test process, and I'm running with JDK 1.7.0_25.

Is anyone else seeing this problem? Any hints?

I’ll take a look.

Do the non-ascii tests in the ‘ivy’ and ‘maven’ projects work for you?


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com

Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA: http://www.gradlesummit.com



--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com

Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA: http://www.gradlesummit.com