Merge master into develop (#2206)

* export: fix exporting extras sub-dependencies (#1294)

* Support POETRY_HOME for install (#794)

Allow the `POETRY_HOME` environment variable to be passed during installation to change the default installation directory of `~/.poetry`:

```
POETRY_HOME=/etc/poetry python get-poetry.py
```

* * check if relative filename is in excluded file list (#1459)

* * check if relative filename is in excluded file list
* removed find_excluded_files() method from wheel.py

* added test for excluding files in wheels

* creating an own test data folder, for testing excluding files by pyproject.toml

* use as_posix() to respect windows file path delimiters

* Exclude nested items (#784) (#1464)

* This PR impliments the feature request #784.

When a folder is explicit defined in `pyproject.toml` as excluded, all nested data, including subfolder, are excluded. It is no longer neccessary to use the glob `folder/**/*`

* use `Path` instead of `os.path.join` to create string for globbing

* try to fix linting error

* create glob pattern string by concatenating and not using Path

* using `os.path.isdir()`` for checking of explicit excluded name is a folder, because pathlib's `is_dir()` raises in exception under windows of name contains globing characters

* Remove nested data when wildcards where used.

Steps to do this are:
1. expand any wildcard used
2. if expanded path is a folder append  **/* and expand again

* fix linting

* only glob a second time if path is dir

* implement @sdispater 's suggestion for better readability

* fix glob for windows?

* On Windows, testing if a path with a glob is a directory will raise an OSError

* pathlibs  glob function doesn't return the correct case (https://bugs.python.org/issue26655). So switching back to  glob.glob()

* removing obsolete imports

* Update dependencies

* Deprecate allows-prereleases in favor of allow-prereleases for consistency

* Fix tests for Python 2.7

* Fix linting

* Fix linting

* Fix linting

* Fix typing import

* Correct a couple typos in get-poetry.py (#573)

* Docs: `self:update` changed to `self update` (#1588)

* Fix GitHub actions cache issues on develop (#1918)

* Fix Github actions cache issues

* Fix Github Actions cache issues (#1928)

* Add --source option to "poetry add" (#1912)

* Add --source option to 'poetry add'

* Add tests for 'poetry add --source'

* Merge master into develop (#2070)

* Fix Github actions cache issues (#1908)

* Fix case of `-f` flag

* Make it clearer what options to pass to `--format`

* fix (masonry.api): `get_requires_for_build_wheel` must return additional list of requirements for building a package, not listed in `pyproject.toml` and not dependencies for the package itself (#1875)

fix (tests): adopted tests

* Lazy Keyring intialization for PasswordManager (#1892)

* Fix Github Actions cache issues (#1928)

* Avoid nested quantifiers with overlapping character space on git url parsing (#1902 (#1913)

* fix (git): match for `\w` instead of `.` for getting user

* change (vcs.git): hold pattern of the regex parts in a dictionary to be consistent over all regexs

* new (vcs.git): test for `parse_url` and some fixes for the regex pattern

* new (vcs.git): test for `parse_url` with string that should fail

* fix (test.vcs.git): make flake8 happy

* fix: correct parsing of wheel version with regex. (#1932)

The previous regexp was only taking the first integer of the version number,
this presented problems when the major version number reached double digits.

Poetry would determine that the version of the dependency is '1', rather than,
ie: '14'. This caused failures to solve versions.

* Fix errors when using the --help option (#1910)

* Fix how repository credentials are retrieved from env vars (#1909)

# Conflicts:
#	poetry/utils/password_manager.py

* Fix downloading packages from Simplepypi (#1851)

* fix downloading packages from simplepypi

* unused code removed

* remove unused imports

* Upgrade dependencies for the 1.0.3 release (#1965)

* Bump version to 1.0.3 (#1966)

* Fix non-compliant Git URL matching

RFC 3986 § 2.3 permits more characters in a URL than were matched. This
corrects that, though there may be other deficiencies. This was a
regression from v1.0.2, where at least “.” was matched without error.

* Update README.md "Updating Poetry"

Currently the note in "Updating Poetry" is different from the one below in "Enable tab completion for Bash, Fish, or Zsh". This MR is to make them more consistent.

* init: change dev dependency prompt

* Fix CI issues (#2069)

Co-authored-by: brandonaut <brandon@hubermx.com>
Co-authored-by: finswimmer <finswimmer77@gmail.com>
Co-authored-by: Yannick PÉROUX <yannick.peroux@gmail.com>
Co-authored-by: Edward George <edwardgeorge@gmail.com>
Co-authored-by: Jan Škoda <skoda@jskoda.cz>
Co-authored-by: Andrew Marshall <andrew@johnandrewmarshall.com>
Co-authored-by: Andrew Selzer <andrewfselzer@gmail.com>
Co-authored-by: Andrii Maletskyi <andrii.maletskyi@gmail.com>

* pre-commit: replace isort mirror with isort upstream (#2118)

The isort pre-commit mirror has been deprecated. This change updates
configuration to use the upstream package repository instead of the
mirror.

* Add cache list command (#1187)

* Add poetry.locations.REPOSITORY_CACHE_DIR

The repository cache directory is used in multiple places in the
codebase. This change ensures that the value is reused.

* Add cache list command

This introduces a new cache sub-command that lists all available
caches.

Relates-to: #1162

Co-authored-by: Tom Milligan <tommilligan@users.noreply.github.com>
Co-authored-by: David Cramer <dcramer@users.noreply.github.com>
Co-authored-by: finswimmer <finswimmer77@gmail.com>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: Justin Mayer <entroP@gmail.com>
Co-authored-by: Yannick PÉROUX <yannick.peroux@gmail.com>
Co-authored-by: brandonaut <brandon@hubermx.com>
Co-authored-by: Edward George <edwardgeorge@gmail.com>
Co-authored-by: Jan Škoda <skoda@jskoda.cz>
Co-authored-by: Andrew Marshall <andrew@johnandrewmarshall.com>
Co-authored-by: Andrew Selzer <andrewfselzer@gmail.com>
Co-authored-by: Andrii Maletskyi <andrii.maletskyi@gmail.com>
Co-authored-by: Arun Babu Neelicattu <arun.neelicattu@gmail.com>

.pre-commit-config.yaml

@@ -9,8 +9,8 @@ repos:
     hooks:
       - id: flake8
 
-  - repo: https://github.com/pre-commit/mirrors-isort
-    rev: v4.3.21
+  - repo: https://github.com/timothycrosley/isort
+    rev: 4.3.21
     hooks:
       - id: isort
         additional_dependencies: [toml]

CHANGELOG.md

@@ -27,7 +27,6 @@
 - Fixed an error when parsing some git URLs ([#2018](https://github.com/python-poetry/poetry/pull/2018)).
 
 
-
 ## [1.0.3] - 2020-01-31
 
 ### Fixed

docs/docs/cli.md

@@ -452,3 +452,15 @@ The `env` command regroups sub commands to interact with the virtualenvs
 associated with a specific project.
 
 See [Managing environments](./managing-environments.md) for more information about these commands.
+
+## cache
+
+The `cache` command regroups sub commands to interact with Poetry's cache.
+
+### cache list
+
+The `cache list` command lists Poetry's available caches.
+
+```bash
+poetry cache list
+```

poetry/console/commands/add.py

@@ -33,6 +33,12 @@ class AddCommand(EnvCommand, InitCommand):
             "Platforms for which the dependency must be installed.",
             flag=False,
         ),
+        option(
+            "source",
+            None,
+            "Name of the source to use to install the package.",
+            flag=False,
+        ),
         option("allow-prereleases", None, "Accept prereleases."),
         option(
             "dry-run",
@@ -86,7 +92,9 @@ If you do not specify a version constraint, poetry will choose a suitable one ba
                     raise ValueError("Package {} is already present".format(name))
 
         requirements = self._determine_requirements(
-            packages, allow_prereleases=self.option("allow-prereleases")
+            packages,
+            allow_prereleases=self.option("allow-prereleases"),
+            source=self.option("source"),
         )
 
         for _constraint in requirements:
@@ -123,6 +131,9 @@ If you do not specify a version constraint, poetry will choose a suitable one ba
             if self.option("platform"):
                 constraint["platform"] = self.option("platform")
 
+            if self.option("source"):
+                constraint["source"] = self.option("source")
+
             if len(constraint) == 1 and "version" in constraint:
                 constraint = constraint["version"]
 

poetry/console/commands/cache/cache.py

+from poetry.console.commands.cache.list import CacheListCommand
+
 from ..command import Command
 from .clear import CacheClearCommand
 
@@ -7,7 +9,7 @@ class CacheCommand(Command):
     name = "cache"
     description = "Interact with Poetry's cache"
 
-    commands = [CacheClearCommand()]
+    commands = [CacheClearCommand(), CacheListCommand()]
 
     def handle(self):
         return self.call("help", self._config.name)

poetry/console/commands/cache/clear.py

@@ -16,19 +16,17 @@ class CacheClearCommand(Command):
 
     def handle(self):
         from cachy import CacheManager
-        from poetry.locations import CACHE_DIR
-        from poetry.utils._compat import Path
+        from poetry.locations import REPOSITORY_CACHE_DIR
 
         cache = self.argument("cache")
 
         parts = cache.split(":")
         root = parts[0]
 
-        base_cache = Path(CACHE_DIR) / "cache" / "repositories"
-        cache_dir = base_cache / root
+        cache_dir = REPOSITORY_CACHE_DIR / root
 
         try:
-            cache_dir.relative_to(base_cache)
+            cache_dir.relative_to(REPOSITORY_CACHE_DIR)
         except ValueError:
             raise ValueError("{} is not a valid repository cache".format(root))
 

poetry/console/commands/cache/list.py

+import os
+
+from ..command import Command
+
+
+class CacheListCommand(Command):
+
+    name = "list"
+    description = "List Poetry's caches."
+
+    def handle(self):
+        from poetry.locations import REPOSITORY_CACHE_DIR
+
+        if os.path.exists(str(REPOSITORY_CACHE_DIR)):
+            caches = list(sorted(REPOSITORY_CACHE_DIR.iterdir()))
+            if caches:
+                for cache in caches:
+                    self.line("<info>{}</>".format(cache.name))
+                return 0
+
+        self.line("<warning>No caches found</>")

poetry/console/commands/init.py

@@ -203,7 +203,7 @@ The <c1>init</c1> command creates a basic <comment>pyproject.toml</> file in the
             f.write(content)
 
     def _determine_requirements(
-        self, requires, allow_prereleases=False
+        self, requires, allow_prereleases=False, source=None
     ):  # type: (List[str], bool) -> List[Dict[str, str]]
         if not requires:
             requires = []
@@ -299,7 +299,9 @@ The <c1>init</c1> command creates a basic <comment>pyproject.toml</> file in the
             elif "version" not in requirement:
                 # determine the best version automatically
                 name, version = self._find_best_version_for_package(
-                    requirement["name"], allow_prereleases=allow_prereleases
+                    requirement["name"],
+                    allow_prereleases=allow_prereleases,
+                    source=source,
                 )
                 requirement["version"] = version
                 requirement["name"] = name
@@ -314,6 +316,7 @@ The <c1>init</c1> command creates a basic <comment>pyproject.toml</> file in the
                     requirement["name"],
                     requirement["version"],
                     allow_prereleases=allow_prereleases,
+                    source=source,
                 )
 
                 requirement["name"] = name
@@ -323,13 +326,13 @@ The <c1>init</c1> command creates a basic <comment>pyproject.toml</> file in the
         return result
 
     def _find_best_version_for_package(
-        self, name, required_version=None, allow_prereleases=False
+        self, name, required_version=None, allow_prereleases=False, source=None
     ):  # type: (...) -> Tuple[str, str]
         from poetry.version.version_selector import VersionSelector
 
         selector = VersionSelector(self._get_pool())
         package = selector.find_best_candidate(
-            name, required_version, allow_prereleases=allow_prereleases
+            name, required_version, allow_prereleases=allow_prereleases, source=source
         )
 
         if not package:

poetry/locations.py

+from .utils._compat import Path
 from .utils.appdirs import user_cache_dir
 from .utils.appdirs import user_config_dir
 
 
 CACHE_DIR = user_cache_dir("pypoetry")
 CONFIG_DIR = user_config_dir("pypoetry")
+
+REPOSITORY_CACHE_DIR = Path(CACHE_DIR) / "cache" / "repositories"

poetry/repositories/legacy_repository.py

@@ -15,7 +15,7 @@ from cachy import CacheManager
 
 import poetry.packages
 
-from poetry.locations import CACHE_DIR
+from poetry.locations import REPOSITORY_CACHE_DIR
 from poetry.packages import Package
 from poetry.packages import dependency_from_pep_508
 from poetry.packages.utils.link import Link
@@ -174,7 +174,7 @@ class LegacyRepository(PyPiRepository):
         self._client_cert = client_cert
         self._cert = cert
         self._inspector = Inspector()
-        self._cache_dir = Path(CACHE_DIR) / "cache" / "repositories" / name
+        self._cache_dir = REPOSITORY_CACHE_DIR / name
         self._cache = CacheManager(
             {
                 "default": "releases",

poetry/repositories/pypi_repository.py

@@ -15,7 +15,7 @@ from requests import get
 from requests import session
 from requests.exceptions import TooManyRedirects
 
-from poetry.locations import CACHE_DIR
+from poetry.locations import REPOSITORY_CACHE_DIR
 from poetry.packages import Package
 from poetry.packages import dependency_from_pep_508
 from poetry.packages.utils.link import Link
@@ -57,7 +57,7 @@ class PyPiRepository(RemoteRepository):
         self._disable_cache = disable_cache
         self._fallback = fallback
 
-        release_cache_dir = Path(CACHE_DIR) / "cache" / "repositories" / "pypi"
+        release_cache_dir = REPOSITORY_CACHE_DIR / "pypi"
         self._cache = CacheManager(
             {
                 "default": "releases",

poetry/version/version_selector.py

@@ -15,6 +15,7 @@ class VersionSelector(object):
         package_name,  # type: str
         target_package_version=None,  # type:  Union[str, None]
         allow_prereleases=False,  # type: bool
+        source=None,  # type: str
     ):  # type: (...) -> Union[Package, bool]
         """
         Given a package name and optional version,
@@ -26,7 +27,7 @@ class VersionSelector(object):
             constraint = parse_constraint("*")
 
         candidates = self._pool.find_packages(
-            package_name, constraint, allow_prereleases=True
+            package_name, constraint, allow_prereleases=True, repository=source
         )
         only_prereleases = all([c.version.is_prerelease() for c in candidates])
 

tests/console/commands/test_add.py

@@ -4,6 +4,8 @@ import pytest
 
 from cleo.testers import CommandTester
 
+from poetry.repositories.legacy_repository import LegacyRepository
+from poetry.semver import Version
 from poetry.utils._compat import Path
 from tests.helpers import get_dependency
 from tests.helpers import get_package
@@ -634,6 +636,72 @@ Package operations: 1 install, 0 updates, 0 removals
     }
 
 
+def test_add_constraint_with_source(app, poetry, installer):
+    repo = LegacyRepository(name="my-index", url="https://my-index.fake")
+    repo.add_package(get_package("cachy", "0.2.0"))
+    repo._cache.store("matches").put("cachy:0.2.0", [Version.parse("0.2.0")], 5)
+
+    poetry.pool.add_repository(repo)
+
+    command = app.find("add")
+    tester = CommandTester(command)
+
+    tester.execute("cachy=0.2.0 --source my-index")
+
+    expected = """\
+
+Updating dependencies
+Resolving dependencies...
+
+Writing lock file
+
+
+Package operations: 1 install, 0 updates, 0 removals
+
+  - Installing cachy (0.2.0)
+"""
+
+    assert expected == tester.io.fetch_output()
+
+    assert len(installer.installs) == 1
+
+    content = app.poetry.file.read()["tool"]["poetry"]
+
+    assert "cachy" in content["dependencies"]
+    assert content["dependencies"]["cachy"] == {
+        "version": "0.2.0",
+        "source": "my-index",
+    }
+
+
+def test_add_constraint_with_source_that_does_not_exist(app):
+    command = app.find("add")
+    tester = CommandTester(command)
+
+    with pytest.raises(ValueError) as e:
+        tester.execute("foo --source i-dont-exist")
+
+    assert 'Repository "i-dont-exist" does not exist.' == str(e.value)
+
+
+def test_add_constraint_not_found_with_source(app, poetry, mocker):
+    repo = LegacyRepository(name="my-index", url="https://my-index.fake")
+    mocker.patch.object(repo, "find_packages", return_value=[])
+
+    poetry.pool.add_repository(repo)
+
+    pypi = poetry.pool.repositories[0]
+    pypi.add_package(get_package("cachy", "0.2.0"))
+
+    command = app.find("add")
+    tester = CommandTester(command)
+
+    with pytest.raises(ValueError) as e:
+        tester.execute("cachy --source my-index")
+
+    assert "Could not find a matching version of package cachy" == str(e.value)
+
+
 def test_add_to_section_that_does_no_exist_yet(app, repo, installer):
     command = app.find("add")
     tester = CommandTester(command)

tests/console/commands/test_cache.py

+import uuid
+
+import pytest
+
+from cleo.testers import CommandTester
+
+
+@pytest.fixture
+def repository_cache_dir(monkeypatch, tmpdir):
+    import poetry.locations
+    from poetry.utils._compat import Path
+
+    path = Path(str(tmpdir))
+    monkeypatch.setattr(poetry.locations, "REPOSITORY_CACHE_DIR", path)
+    return path
+
+
+@pytest.fixture
+def repository_one():
+    return "01_{}".format(uuid.uuid4())
+
+
+@pytest.fixture
+def repository_two():
+    return "02_{}".format(uuid.uuid4())
+
+
+@pytest.fixture
+def mock_caches(repository_cache_dir, repository_one, repository_two):
+    (repository_cache_dir / repository_one).mkdir()
+    (repository_cache_dir / repository_two).mkdir()
+
+
+def test_cache_list(app, mock_caches, repository_one, repository_two):
+    command = app.find("cache list")
+    tester = CommandTester(command)
+
+    tester.execute()
+
+    expected = """\
+{}
+{}
+""".format(
+        repository_one, repository_two
+    )
+
+    assert expected == tester.io.fetch_output()
+
+
+def test_cache_list_empty(app, repository_cache_dir):
+    command = app.find("cache list")
+    tester = CommandTester(command)
+
+    tester.execute()
+
+    expected = """\
+No caches found
+"""
+
+    assert expected == tester.io.fetch_output()