doc: improve repository documentation

Co-authored-by: Bjorn Neergaard <bjorn@neersighted.com>

docs/repositories.md

@@ -11,7 +11,8 @@ menu:
 
 # Repositories
 
-## Using the PyPI repository
+Poetry supports the use of [PyPI](https://pypi.org) and private repositories for discovery of
+packages as well as for publishing your projects.
 
 By default, Poetry is configured to use the [PyPI](https://pypi.org) repository,
 for package installation and publishing.
@@ -21,24 +22,283 @@ on PyPI.
 
 This represents most cases and will likely be enough for most users.
 
+### Private Repository Example
 
-## Using a private repository
+#### Installing from private package sources
+By default, Poetry discovers and installs packages from [PyPI](https://pypi.org). But, you want to
+install a dependency to your project for a [simple API repository](#simple-api-repository)? Let's
+do it.
 
-However, at times, you may need to keep your package private while still being
-able to share it with your teammates. In this case, you will need to use a private
-repository.
+First, [configure](#project-configuration) the [package source](#package-source) repository to your
+project.
 
-### Adding a repository
+```bash
+poetry source add foo https://pypi.example.org/simple/
+```
+
+Then, assuming the repository requires authentication, configure credentials for it.
+
+```bash
+poetry config http-basic.foo username password
+```
+
+Once this is done, you can add dependencies to your project from this source.
+
+```bash
+poetry add --source foo private-package
+```
+
+#### Publishing to a private repository
+
+Great, now all that is left is to publish your package. Assuming you'd want to share it privately
+with your team, you can configure the
+[Upload API](https://warehouse.pypa.io/api-reference/legacy.html#upload-api) endpoint for your
+[publishable repository](#publishable-repository).
+
+```bash
+poetry config repositories.foo https://pypi.example.org/legacy/
+```
+
+{{% note %}}
+
+If you need to use a different credential for your [package source](#package-sources), then it is
+recommended to use a different name for your publishing repository.
+
+```bash
+poetry config repositories.foo-pub https://pypi.example.org/legacy/
+poetry config http-basic.foo-pub username password
+```
+
+{{% /note %}}
+
+Now, all the is left is to build and publish your project using the
+[`publish`]({{< relref "cli#publish" >}}).
+
+```bash
+poetry publish --build --repository foo-pub
+```
+
+## Package Sources
+
+By default, Poetry is configured to use the Python ecosystem's canonical package index
+[PyPI](https://pypi.org).
+
+{{% note %}}
+
+With the exception of the implicitly configured source for [PyPI](https://pypi.org) named `pypi`,
+package sources are local to a project and must be configured within the project's
+[`pyproject.toml`]({{< relref "pyproject" >}}) file. This is **not** the same configuration used
+when publishing a package.
+
+{{% /note %}}
+
+### Project Configuration
+
+These package sources maybe managed using the [`source`]({{< relref "cli#source" >}}) command for
+your project.
+
+```bash
+poetry source add foo https://foo.bar/simple/
+```
+
+This will generate the following configuration snippet in your
+[`pyproject.toml`]({{< relref "pyproject" >}}) file.
+
+```toml
+[[tool.poetry.source]]
+name = "foo"
+url = "https://foo.bar/simple/"
+default = false
+secondary = false
+```
+
+{{% warning %}}
+
+If package sources are defined for a project, these will take precedence over
+[PyPI](https://pypi.org). If you do not want this to be the case, you should declare **all** package
+sources to be [secondary](#secondary-package-sources).
+
+{{% /warning %}}
+
+See [Supported Package Sources](#supported-package-sources) for source type specific information.
+
+{{% note %}}
+
+If your package source requires [credentials](#configuring-credentials) or
+[certificates](#certificates), please refer to the relevant sections below.
+
+{{% /note %}}
+
+#### Default Package Source
+
+By default, Poetry configures [PyPI](https://pypi.org) as the default package source for your
+project. You can alter this behaviour and exclusively look up packages only from the configured
+package sources by adding a **single** source with `default = true`.
+
+```bash
+poetry source add --default foo https://foo.bar/simple/
+```
+
+{{% warning %}}
+
+Configuring a custom package source as default, will effectively disable [PyPI](https://pypi.org)
+as a package source for your project.
 
-Adding a new repository is easy with the `config` command.
+{{% /warning %}}
+
+#### Secondary Package Sources
+
+If package sources are configured as secondary, all it means is that these will be given a lower
+priority when selecting compatible package distribution that also exists in your default package
+source.
+
+You can configure a package source as a secondary source with `secondary = true` in your package
+source configuration.
 
 ```bash
-poetry config repositories.foo https://foo.bar/simple/
+poetry source add --secondary foo https://foo.bar/simple/
 ```
 
-This will set the url for repository `foo` to `https://foo.bar/simple/`.
+There can be more than one secondary package source.
+
+{{% note %}}
+
+All package sources (including secondary sources) will be searched during the package lookup
+process. These network requests will occur for all sources, regardless of if the package is
+found at one or more sources.
 
-### Configuring credentials
+If you wish to avoid this, you may explicitly specify which source to search in for a particular
+package.
+
+```bash
+poetry add --source pypi httpx
+```
+
+{{% /note %}}
+
+### Supported Package Sources
+
+#### Python Package Index (PyPI)
+
+Poetry interacts with [PyPI](https://pypi.org) via its
+[JSON API](https://warehouse.pypa.io/api-reference/json.html). This is used to retrieve a requested
+package's versions, metadata, files, etc.
+
+{{% note %}}
+
+If the the package's published metadata is invalid, Poetry will download the available bdist/sdist to
+inspect it locally to identify the relevant metadata.
+
+{{% /note %}}
+
+If you want to explicitly select a package from [PyPI](https://pypi.org) you can use the `--source`
+option with the [`add`]({{< relref "cli#add" >}}) command, like shown below.
+
+```bash
+poetry add --source pypi httpx@^0.22.0
+```
+
+This will generate the following configuration snippet in your `pyproject.toml` file.
+
+```toml
+httpx = {version = "^0.22.0", source = "pypi"}
+```
+
+{{% warning %}}
+
+If any source within a project is configured with `default = true`, The implicit `pypi` source will
+be disabled and not used for any packages.
+
+{{% /warning %}}
+
+#### Simple API Repository
+
+Poetry can fetch and install package dependencies from public or private custom repositories that
+implement the simple repository API as described in [PEP 503](https://peps.python.org/pep-0503/).
+
+{{% warning %}}
+
+When using sources that distributes large wheels without providing file checksum in file URLs,
+Poetry will download each candidate wheel at least once in order to generate the checksum. This can
+manifest as long dependency resolution times when adding packages from this source.
+
+{{% /warning %}}
+
+These package sources maybe configured via the following command in your project.
+
+```bash
+poetry source add testpypi https://test.pypi.org/simple/
+```
+
+{{% note %}}
+
+Note the trailing `/simple/`. This is important when configuring
+[PEP 503](https://peps.python.org/pep-0503/) compliant package sources.
+
+{{% /note %}}
+
+In addition to [PEP 503](https://peps.python.org/pep-0503/), Poetry can also handle simple API
+repositories that implement [PEP 658](https://peps.python.org/pep-0658/) (*Introduced in 1.2.0*).
+This is helpful in reducing dependency resolution time for packages from these sources as Poetry can
+avoid having to download each candidate distribution, in order to determine associated metadata.
+
+{{% note %}}
+
+*Why does Poetry insist on downloading all candidate distributions for all platforms when metadata
+is not available?*
+
+The need for this stems from the fact that Poetry's lock file is platform-agnostic. This means, in
+order to resolve dependencies for a project, Poetry needs metadata for all platform specific
+distributions. And when this metadata is not readily available, downloading the distribution and
+inspecting it locally is the only remaining option.
+
+{{% /note %}}
+
+#### Single Page Link Source
+
+*Introduced in 1.2.0*
+
+Some projects choose to release their binary distributions via a single page link source that
+partially follows the structure of a package page in [PEP 503](https://peps.python.org/pep-0503/).
+
+These package sources maybe configured via the following command in your project.
+
+```bash
+poetry source add jax https://storage.googleapis.com/jax-releases/jax_releases.html
+```
+
+{{% note %}}
+
+All caveats regarding slower resolution times described for simple API repositories do apply here as
+well.
+
+{{% /note %}}
+
+
+## Publishable Repositories
+
+Poetry treats repositories to which you publish packages as user specific and not project specific
+configuration unlike [package sources](#package-sources]. Poetry, today, only supports the
+[Legacy Upload API](https://warehouse.pypa.io/api-reference/legacy.html#upload-api) when publishing
+your project.
+
+These are configured using the [`config`]({{< relref "cli#config" >}}) command, under the
+`repository` key.
+
+```bash
+poetry config repository.testpypi https://upload.test.pypi.org/legacy/
+```
+
+{{% note %}}
+
+[Legacy Upload API](https://warehouse.pypa.io/api-reference/legacy.html#upload-api) URLs are
+typically different to the same one provided by the repository for the simple API. You'll note that
+in the example of [Test PyPI](https://test.pypi.org/), both the host (`upload.test.pypi.org`) as
+well as the path (`/legacy`) are different to it's simple API (`https://test.pypi.org/simple`).
+
+{{% /note %}}
+
+## Configuring Credentials
 
 If you want to store your credentials for a specific repository, you can do so easily:
 
@@ -49,6 +309,7 @@ poetry config http-basic.foo username password
 If you do not specify the password you will be prompted to write it.
 
 {{% note %}}
+
 To publish to PyPI, you can set your credentials for the repository named `pypi`.
 
 Note that it is recommended to use [API tokens](https://pypi.org/help/#apitoken)
@@ -65,8 +326,8 @@ call to `config`.
 ```bash
 poetry config http-basic.pypi username password
 ```
-{{% /note %}}
 
+{{% /note %}}
 
 You can also specify the username and password when using the `publish` command
 with the `--username` and `--password` options.
@@ -109,7 +370,10 @@ You can prevent this by adding double dashes to prevent any following argument f
 poetry config -- http-basic.pypi myUsername -myPasswordStartingWithDash
 ```
 
-#### Custom certificate authority and mutual TLS authentication
+## Certificates
+
+### Custom certificate authority and mutual TLS authentication
+
 Poetry supports repositories that are secured by a custom certificate authority as well as those that require
 certificate-based client authentication.  The following will configure the "foo" repository to validate the repository's
 certificate using a custom certificate authority and use a client certificate (note that these config variables do not
@@ -120,53 +384,27 @@ poetry config certificates.foo.cert /path/to/ca.pem
 poetry config certificates.foo.client-cert /path/to/client.pem
 ```
 
-### Install dependencies from a private repository
+## Caches
 
-Now that you can publish to your private repository, you need to be able to
-install dependencies from it.
+Poetry employs multiple caches for package sources in order to improve user experience and avoid duplicate network
+requests.
 
-For that, you have to edit your `pyproject.toml` file, like so
+The first level cache is a [Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control)
+header based cache for almost all HTTP requests.
 
-```toml
-[[tool.poetry.source]]
-name = "foo"
-url = "https://foo.bar/simple/"
-```
-
-From now on, Poetry will also look for packages in your private repository.
-
-{{% note %}}
-Any custom repository will have precedence over PyPI.
+Further, every HTTP backed package source caches metadata associated with a package once it is fetched or generated.
+Additionally, downloaded files (package distributions) are also cached.
 
-If you still want PyPI to be your primary source for your packages
-you can declare custom repositories as secondary.
+## Debugging Issues
+If you encounter issues with package sources, one of the simplest steps you might take to debug an issue is rerunning
+your command with the `--no-cache` flag.
 
-```toml
-[[tool.poetry.source]]
-name = "foo"
-url = "https://foo.bar/simple/"
-secondary = true
+```bash
+poetry --no-cache add pycowsay
 ```
-{{% /note %}}
-
-If your private repository requires HTTP Basic Auth be sure to add the username and
-password to your `http-basic` configuration using the example above (be sure to use the
-same name that is in the `tool.poetry.source` section). If your repository requires either
-a custom certificate authority or client certificates, similarly refer to the example above to configure the
-`certificates` section. Poetry will use these values to authenticate to your private repository when downloading or
-looking for packages.
-
-
-### Disabling the PyPI repository
-
-If you want your packages to be exclusively looked up from a private
-repository, you can set it as the default one by using the `default` keyword
 
-```toml
-[[tool.poetry.source]]
-name = "foo"
-url = "https://foo.bar/simple/"
-default = true
-```
+If this solves your issue, you can consider clearing your cache using the [`cache`]({{< relref "cli#cache-clear" >}})
+command.
 
-A default source will also be the fallback source if you add other sources.
+Alternatively, you could also consider enabling very verbose loging `-vvv` along with the `--no-cache` to see network
+requests being made in the logs.