Dependency resolution metadata. How is it gathered?

[stefanoborini]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Issue

I tried to navigate the code but I could not find it. Maybe you can give me a better answer.

I suspect that the dependency resolver in poetry needs to know the dependency tree, and possibly backtrack if some constraints are not satisfied, but since the dependency tree depends on the version of the package downloaded (a package v1.2 might have different dependencies from v1.3), the tree must be rebuilt at every backtrack, at least for those parts that do not satisfy a found constraint.

How is the dependency list and constraints gathered? do you download the packages, unpack them every time, and get the subdependency information from the setup.py/pyproject.toml? Are they cached, or is an index built somewhere? As far as I know, pypi does not provide metadata files, hence the question.

[abn]

1. If package is on PyPI, the package information available at https://pypi.org/pypi/{package}/json.

   poetry/poetry/repositories/pypi_repository.py

   Lines 199 to 218 in 3a789a7

   	def get_package_info(self, name: str) -> dict:
   	"""
   	Return the package information given its name.
   	The information is returned from the cache if it exists
   	or retrieved from the remote server.
   	"""
   	if self._disable_cache:
   	return self._get_package_info(name)
   	return self._cache.store("packages").remember_forever(
   	name, lambda: self._get_package_info(name)
   	)
   	def _get_package_info(self, name: str) -> dict:
   	data = self._get(f"pypi/{name}/json")
   	if data is None:
   	raise PackageNotFound(f"Package [{name}] not found.")
   	return data

2. If package is in a private repository, the simple endpoint is used. (eg: https://pypi.org/simple/poetry-core/).

   poetry/poetry/repositories/legacy_repository.py

   Lines 347 to 420 in 3a789a7

   	def _get_release_info(self, name: str, version: str) -> dict:
   	page = self._get("/{}/".format(canonicalize_name(name).replace(".", "-")))
   	if page is None:
   	raise PackageNotFound(f'No package named "{name}"')
   	data = PackageInfo(
   	name=name,
   	version=version,
   	summary="",
   	platform=None,
   	requires_dist=[],
   	requires_python=None,
   	files=[],
   	cache_version=str(self.CACHE_VERSION),
   	)
   	links = list(page.links_for_version(Version.parse(version)))
   	if not links:
   	raise PackageNotFound(
   	'No valid distribution links found for package: "{}" version: "{}"'.format(
   	name, version
   	)
   	)
   	urls = defaultdict(list)
   	files = []
   	for link in links:
   	if link.is_wheel:
   	urls["bdist_wheel"].append(link.url)
   	elif link.filename.endswith(
   	(".tar.gz", ".zip", ".bz2", ".xz", ".Z", ".tar")
   	):
   	urls["sdist"].append(link.url)
   	file_hash = f"{link.hash_name}:{link.hash}" if link.hash else None
   	if not link.hash or (
   	link.hash_name not in ("sha256", "sha384", "sha512")
   	and hasattr(hashlib, link.hash_name)
   	):
   	with temporary_directory() as temp_dir:
   	filepath = Path(temp_dir) / link.filename
   	self._download(link.url, str(filepath))
   	known_hash = (
   	getattr(hashlib, link.hash_name)() if link.hash_name else None
   	)
   	required_hash = hashlib.sha256()
   	chunksize = 4096
   	with filepath.open("rb") as f:
   	while True:
   	chunk = f.read(chunksize)
   	if not chunk:
   	break
   	if known_hash:
   	known_hash.update(chunk)
   	required_hash.update(chunk)
   	if not known_hash or known_hash.hexdigest() == link.hash:
   	file_hash = "{}:{}".format(
   	required_hash.name, required_hash.hexdigest()
   	)
   	files.append({"file": link.filename, "hash": file_hash})
   	data.files = files
   	info = self._get_info_from_urls(urls)
   	data.summary = info.summary
   	data.requires_dist = info.requires_dist
   	data.requires_python = info.requires_python
   	return data.asdict()

3. If metadata is not availably, filename is used to determine python constraints if any. Then the file is downloaded, and inspected.

   poetry/poetry/inspection/info.py

   Lines 580 to 590 in 3a789a7

   	@classmethod
   	def from_path(cls, path: Path) -> "PackageInfo":
   	"""
   	Gather package information from a given path (bdist, sdist, directory).
   	:param path: Path to inspect.
   	"""
   	try:
   	return cls.from_bdist(path=path)
   	except PackageInfoError:
   	return cls.from_sdist(path=path)

4. If it is a wheel, static metadata is extracted.
5. If it is an sdist, it is unpacked and any static metadata is collected (either via poetry pyproject.toml if applicable or sdist metadata if not). If static metadata is unavailable or incomplete, the sdist is metadata is constrcuted via PEP517 metadata build.
6. If the above fails, and project has a setup.py file, we parse it.

   poetry/poetry/inspection/info.py

   Lines 297 to 352 in 3a789a7

   	@classmethod
   	def from_setup_files(cls, path: Path) -> "PackageInfo":
   	"""
   	Mechanism to parse package information from a `setup.[py|cfg]` file. This uses the implementation
   	at `poetry.utils.setup_reader.SetupReader` in order to parse the file. This is not reliable for
   	complex setup files and should only attempted as a fallback.
   	:param path: Path to `setup.py` file
   	"""
   	if not cls.has_setup_files(path):
   	raise PackageInfoError(
   	path, "No setup files (setup.py, setup.cfg) was found."
   	)
   	try:
   	result = SetupReader.read_from_directory(path)
   	except Exception as e:
   	raise PackageInfoError(path, e)
   	python_requires = result["python_requires"]
   	if python_requires is None:
   	python_requires = "*"
   	requires = ""
   	for dep in result["install_requires"]:
   	requires += dep + "\n"
   	if result["extras_require"]:
   	requires += "\n"
   	for extra_name, deps in result["extras_require"].items():
   	requires += f"[{extra_name}]\n"
   	for dep in deps:
   	requires += dep + "\n"
   	requires += "\n"
   	requirements = parse_requires(requires)
   	info = cls(
   	name=result.get("name"),
   	version=result.get("version"),
   	summary=result.get("description", ""),
   	requires_dist=requirements or None,
   	requires_python=python_requires,
   	)
   	if not (info.name and info.version) and not info.requires_dist:
   	# there is nothing useful here
   	raise PackageInfoError(
   	path,
   	"No core metadata (name, version, requires-dist) could be retrieved.",
   	)
   	return info

[stefanoborini]

wow. Thanks.

[abn]

Hope that answers your question. :)