Changes to dependency resolution mechanism

[Daz DeBoer-2]

G'day

Just wanted to announce some fairly significant updates to the way we are performing dependency resolution for dynamic versions.

Motivation

Using component metadata rules, it's possible to update the 'status' of an ivy module during resolution. This status value is then used to determine which module version to choose when using a resolver like 'latest.release' (choose the newest version with a status of 'release').

Until now, there has been an issue with caching of these values. Since we cache the mapping of 'latest.release' to a specific version, any changes to the rules were not reflected until cache expiry. This is a problem because we think that component metadata rules should be 'live', and not have the result of these rules cached.

Implementation

This issue is now fixed, but it involved some significant changes. The biggest change is that the resolver is no longer responsible for mapping a dynamic version to a single version: the resolver simply returns a list of available versions for a module, and the high-level UserResolverChain chooses the best matching version from that list. This means that each ModuleVersionRepository now has 3 functions: list versions for a module, get metadata for a module version, and get artifacts.

• A ModuleVersionRepository (resolver) does not map a dynamic version to a static version: instead it lists the available versions for a module
• ModuleVersionRepository now has 3 functions: 
• list versions for a module
• get metadata for a module version
• get artifacts.
• This module version listing is cached per ModuleVersionRepository
• UserResolverChain chooses the best matching version from the list.
• If the module version metadata is required to choose the best matching version (ie to determine 'latest.release'), then each module is resolved and the component metadata rules are applied. This means that changes to the rules take effect immediately

There are a few implications of this change:

• Since the version listing is cached, if you have previously resolved "group:module:1.+" and then resolve "group:module:2.+:, the second resolve will use the cached version listing. So this means that the resolution of '2.+' could contain stale data until cache expiry, even though that particular selector has not been previously resolved.
• For custom ivy resolvers (deprecated), an extra HTTP GET request is made for ivy.xml when resolving an uncached dynamic version.

This change greatly simplified ExternalResourceResolver, and should make it easier for us to move forward with planned improvements to make dependency resolution variant aware.

Thoughts and feedback welcome.

cheers

Daz

[Adam Murdoch]

On 16 Feb 2014, at 2:59 pm, Daz DeBoer <[hidden email]> wrote:

G'day

Just wanted to announce some fairly significant updates to the way we are performing dependency resolution for dynamic versions.

Motivation

Using component metadata rules, it's possible to update the 'status' of an ivy module during resolution. This status value is then used to determine which module version to choose when using a resolver like 'latest.release' (choose the newest version with a status of 'release').

Until now, there has been an issue with caching of these values. Since we cache the mapping of 'latest.release' to a specific version, any changes to the rules were not reflected until cache expiry. This is a problem because we think that component metadata rules should be 'live', and not have the result of these rules cached.

Implementation

This issue is now fixed, but it involved some significant changes. The biggest change is that the resolver is no longer responsible for mapping a dynamic version to a single version: the resolver simply returns a list of available versions for a module, and the high-level UserResolverChain chooses the best matching version from that list. This means that each ModuleVersionRepository now has 3 functions: list versions for a module, get metadata for a module version, and get artifacts.

• A ModuleVersionRepository (resolver) does not map a dynamic version to a static version: instead it lists the available versions for a module
• ModuleVersionRepository now has 3 functions: 
• list versions for a module
• get metadata for a module version
• get artifacts.
• This module version listing is cached per ModuleVersionRepository
• UserResolverChain chooses the best matching version from the list.
• If the module version metadata is required to choose the best matching version (ie to determine 'latest.release'), then each module is resolved and the component metadata rules are applied. This means that changes to the rules take effect immediately

This is good stuff. We can also later push this up higher again, to take into account all incoming selectors when selecting a version (eg given `1.+` and `[1.2,1.5]` we should choose 1.5 even when 1.6 is available - that kind of thing).

There are a few implications of this change:

• Since the version listing is cached, if you have previously resolved "group:module:1.+" and then resolve "group:module:2.+:, the second resolve will use the cached version listing. So this means that the resolution of '2.+' could contain stale data until cache expiry, even though that particular selector has not been previously resolved.

This also means we make fewer HTTP requests when there are multiple dynamic selectors for a given module resolved during a build, and also only get the meta-data file once when resolving a dynamic selector against multiple repositories.

It also means that new versions don’t appear at arbitrary points during the build - so, for example, if I resolve `latest.integration` and determine that versions 1.1 and 1.2 are present (selecting 1.2), and then later resolve `latest.release`, I don’t suddenly have 1.3 appear.

• For custom ivy resolvers (deprecated), an extra HTTP GET request is made for ivy.xml when resolving an uncached dynamic version.

I think we can live with this. We could possibly have the cache manager that we pass to the resolver deal with this and short-circuit downloading any meta-data file that we’ve already downloaded in the same resolve.

--

Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com