在 Python 包中导入 vendored 依赖项而不修改 sys.path 或 3rd 方包

使用自定义导入器的选项 #4 的问题是您仍然需要重写导入.换句话说，setuptools 使用的自定义导入器钩子根本没有解决供应商化的命名空间问题，相反，如果供应商化的包丢失，它可以动态导入顶级包(一个问题是pip 使用手册解决 分拆过程).setuptools 实际上使用了选项 #1，他们重写了供应商包的源代码.参见例如 thesetuptools 供应商子包中的 >packaging 项目；setuptools.extern 命名空间由自定义导入钩子处理，如果从供应商包导入失败，则重定向到 setuptools._vendor 或顶级名称.

用于更新供应商软件包的 pip 自动化执行以下步骤:

所以本质上，pip 方法最重要的部分，重写供应商包导入非常简单；解释为简化逻辑并删除pip特定部分，它只是以下过程:

导入shutil导入子流程进口重新从 functools 导入部分从 itertools 导入链从 pathlib 导入路径WHITELIST = {'README.txt', '__init__.py', 'vendor.txt'}def delete_all(*paths, whitelist=frozenset()):对于路径中的项目:如果 item.is_dir():Shutil.rmtree(项目，ignore_errors=True)elif item.is_file() 和 item.name 不在白名单中:item.unlink()def iter_subtree(路径):"""递归生成子树中的所有文件，深度优先"""如果不是 path.is_dir():如果 path.is_file():屈服路径返回对于 path.iterdir() 中的项目:如果 item.is_dir():从 iter_subtree(item) 产生elif item.is_file():产量项目def patch_vendor_imports(文件，替换):text = file.read_text('utf8')替换替换:文本 = 替换(文本)file.write_text(文本，'utf8')def find_vendored_libs(vendor_dir, whitelist):vendored_libs = []路径 = []对于 vendor_dir.iterdir() 中的项目:如果 item.is_dir():vendored_libs.append(item.name)elif item.is_file() 和 item.name 不在白名单中:vendored_libs.append(item.stem) # 不带扩展名else: # 不是目录或不在 whilelist 中的文件继续路径.附加(项目)返回 vendored_libs，路径定义供应商(vendor_dir):# 目标包是<parent>.<vendor_dir>;foo/_vendor ->foo._vendorpkgname = f'{vendor_dir.parent.name}.{vendor_dir.name}'# 删除所有内容delete_all(*vendor_dir.iterdir(), whitelist=WHITELIST)# 使用pip安装子进程.运行(['pip', '安装', '-t', str(vendor_dir),'-r', str(vendor_dir/'vendor.txt'),'--no-compile', '--no-deps'])# 删除不需要的东西删除所有(*vendor_dir.glob('*.dist-info'),*vendor_dir.glob('*.egg-info'),vendor_dir/'bin')vendored_libs，路径= find_vendored_libs(vendor_dir，WHITELIST)替换 = []对于 vendored_libs 中的 lib:替换 += (部分(# 导入栏 -> 导入 foo._vendor.barrepile(r'(^s*)import {}
'.format(lib), flags=re.M).sub,r'1from {} import {}
'.format(pkgname, lib)),部分(# 来自酒吧 -> 来自 foo._vendor.barrepile(r'(^s*)from {}(.|s+)'.format(lib), flags=re.M).sub,r'1from {}.{}2'.format(pkgname, lib)),)对于 chain.from_iterable(map(iter_subtree, paths)) 中的文件:patch_vendor_imports(文件，替换)如果 __name__ == '__main__':# 这假设这是 foo/_vendor 旁边的 foo 脚本here = Path('__file__').resolve().parentvendor_dir = 此处/'foo'/'_vendor'断言 (vendor_dir/'vendor.txt').exists(), '_vendor/vendor.txt 文件未找到'assert (vendor_dir/'__init__.py').exists(), '_vendor/__init__.py 文件未找到'供应商(vendor_dir)

Summary

I am working on a series of add-ons for Anki, an open-source flashcard program. Anki add-ons are shipped as Python packages, with the basic folder structure looking as follows:

anki_addons/
    addon_name_1/
        __init__.py
    addon_name_2/
        __init__.py

anki_addons is appended to sys.path by the base app, which then imports each add_on with import <addon_name>.

The problem I have been trying to solve is to find a reliable way to ship packages and their dependencies with my add-ons while not polluting global state or falling back to manual edits of the vendored packages.

Specifics

Specifically, given an add-on structure like this...

addon_name_1/
    __init__.py
    _vendor/
        __init__.py
        library1
        library2
        dependency_of_library2
        ...

...I would like to be able to import any arbitrary package that is included in the _vendor directory, e.g.:

from ._vendor import library1

The main difficulty with relative imports like this is that they do not work for packages that also depend on other packages imported through absolute references (e.g. import dependency_of_library2 in the source code of library2)

Solution attempts

So far I have explored the following options:

I've been stuck on this for quite a few hours now and I'm beginning to think that I'm either completely missing an easy way to do this, or that there is something fundamentally wrong with my entire approach.

Is there no way I can ship a dependency tree of third-party packages with my code, without having to resort to sys.path hacks or modifying the packages in question?

Edit:

Just to clarify: I don't have any control over how add-ons are imported from the anki_addons folder. anki_addons is just the directory provided by the base app where all add-ons are installed into. It is added to the sys path, so the add-on packages therein pretty much just behave like any other python package located in Python's module look-up paths.

First of all, I'd advice against vendoring; a few major packages did use vendoring before but have switched away to avoid the pain of having to handle vendoring. One such example is the requests library. If you are relying on people using pip install to install your package, then just use dependencies and tell people about virtual environments. Don't assume you need to shoulder the burden of keeping dependencies untangled or need to stop people from installing dependencies in the global Python site-packages location.

At the same time, I appreciate that a plug-in environment of a third-party tool is something different, and if adding dependencies to the Python installation used by that tool is cumbersome or impossible vendorizing may be a viable option. I see that Anki distributes extensions as .zip files without setuptools support, so that's certainly such an environment.

So if you choose to vendor dependencies, then use a script to manage your dependencies and update their imports. This is your option #1, but automated.

This is the path that the pip project has chosen, see their tasks subdirectory for their automation, which builds on the invoke library. See the pip project vendoring README for their policy and rationale (chief among those is that pip needs to bootstrap itself, e.g. have their dependencies available to be able to install anything).

You should not use any of the other options; you already enumerated the issues with #2 and #3.

The issue with option #4, using a custom importer, is that you still need to rewrite imports. Put differently, the custom importer hook used by setuptools doesn't solve the vendorized namespace problem at all, it instead makes it possible to dynamically import top-level packages if the vendorized packages are missing (a problem that pip solves with a manual debundling process). setuptools actually uses option #1, where they rewrite the source code for vendorized packages. See for example these lines in the packaging project in the setuptools vendored subpackage; the setuptools.extern namespace is handled by the custom import hook, which then redirects either to setuptools._vendor or the top-level name if importing from the vendorized package fails.

The pip automation to update vendored packages takes the following steps:

So in essence, the most important part of the pip approach, the rewriting of vendored package imports is quite simple; paraphrased to simplify the logic and removing the pip specific parts, it is simply the following process:

import shutil
import subprocess
import re

from functools import partial
from itertools import chain
from pathlib import Path

WHITELIST = {'README.txt', '__init__.py', 'vendor.txt'}

def delete_all(*paths, whitelist=frozenset()):
    for item in paths:
        if item.is_dir():
            shutil.rmtree(item, ignore_errors=True)
        elif item.is_file() and item.name not in whitelist:
            item.unlink()

def iter_subtree(path):
    """Recursively yield all files in a subtree, depth-first"""
    if not path.is_dir():
        if path.is_file():
            yield path
        return
    for item in path.iterdir():
        if item.is_dir():
            yield from iter_subtree(item)
        elif item.is_file():
            yield item

def patch_vendor_imports(file, replacements):
    text = file.read_text('utf8')
    for replacement in replacements:
        text = replacement(text)
    file.write_text(text, 'utf8')

def find_vendored_libs(vendor_dir, whitelist):
    vendored_libs = []
    paths = []
    for item in vendor_dir.iterdir():
        if item.is_dir():
            vendored_libs.append(item.name)
        elif item.is_file() and item.name not in whitelist:
            vendored_libs.append(item.stem)  # without extension
        else:  # not a dir or a file not in the whilelist
            continue
        paths.append(item)
    return vendored_libs, paths

def vendor(vendor_dir):
    # target package is <parent>.<vendor_dir>; foo/_vendor -> foo._vendor
    pkgname = f'{vendor_dir.parent.name}.{vendor_dir.name}'

    # remove everything
    delete_all(*vendor_dir.iterdir(), whitelist=WHITELIST)

    # install with pip
    subprocess.run([
        'pip', 'install', '-t', str(vendor_dir),
        '-r', str(vendor_dir / 'vendor.txt'),
        '--no-compile', '--no-deps'
    ])

    # delete stuff that's not needed
    delete_all(
        *vendor_dir.glob('*.dist-info'),
        *vendor_dir.glob('*.egg-info'),
        vendor_dir / 'bin')

    vendored_libs, paths = find_vendored_libs(vendor_dir, WHITELIST)

    replacements = []
    for lib in vendored_libs:
        replacements += (
            partial(  # import bar -> import foo._vendor.bar
                repile(r'(^s*)import {}
'.format(lib), flags=re.M).sub,
                r'1from {} import {}
'.format(pkgname, lib)
            ),
            partial(  # from bar -> from foo._vendor.bar
                repile(r'(^s*)from {}(.|s+)'.format(lib), flags=re.M).sub,
                r'1from {}.{}2'.format(pkgname, lib)
            ),
        )

    for file in chain.from_iterable(map(iter_subtree, paths)):
        patch_vendor_imports(file, replacements)

if __name__ == '__main__':
    # this assumes this is a script in foo next to foo/_vendor
    here = Path('__file__').resolve().parent
    vendor_dir = here / 'foo' / '_vendor'
    assert (vendor_dir / 'vendor.txt').exists(), '_vendor/vendor.txt file not found'
    assert (vendor_dir / '__init__.py').exists(), '_vendor/__init__.py file not found'
    vendor(vendor_dir)

这篇关于在 Python 包中导入 vendored 依赖项而不修改 sys.path 或 3rd 方包的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！