搭建私有pypi源

2019-02-14

最近公司的单元测试任务经常失败，基本都是因为网络原因下载pip包失败，为此，花了点时间搭建了私有的pypi源来解决此类问题。

准备工作

需要准备一台服务器，分配足够的磁盘空间（不少于500G）。

安装bandersnatch

我试着安装了几个不同版本的bandersnatch，只有1.11版本安装成功。

wget https://bitbucket.org/pypa/bandersnatch/get/1.11.tar.gz
tar -xzvf 1.11.tar.gz

安装

cd pypa-bandersnatch-76b72f3ebd6c
python setup.py install
pip install -r pypa-bandersnatch-76b72f3ebd6c/requirements.txt

初始化及配置

首次运行时创建配置文件

bandersnatch mirror

这一步,会创建默认的配置文件/etc/bandersnatch.conf

修改配置文件

vim /etc/bandersnatch.conf

[mirror]
; The directory where the mirror data will be stored.
directory = /opt/pip

; The PyPI server which will be mirrored.
; master = https://testpypi.python.org
; scheme for PyPI server MUST be https
master = https://pypi.python.org

; The network socket timeout to use for all connections. This is set to a
; somewhat aggressively low value: rather fail quickly temporarily and re-run
; the client soon instead of having a process hang infinitely and have TCP not
; catching up for ages.
timeout = 10

; Number of worker threads to use for parallel downloads.
; Recommendations for worker thread setting:
; - leave the default of 3 to avoid overloading the pypi master
; - official servers located in data centers could run 10 workers
; - anything beyond 10 is probably unreasonable and avoided by bandersnatch
workers = 3

; Whether to hash package indexes
; Note that package index directory hashing is incompatible with pip, and so
; this should only be used in an environment where it is behind an application
; that can translate URIs to filesystem locations.  For example, with the
; following Apache RewriteRule:
;     RewriteRule ^([^/])([^/]*)/$ /mirror/pypi/web/simple/$1/$1$2/
;     RewriteRule ^([^/])([^/]*)/([^/]+)$/ /mirror/pypi/web/simple/$1/$1$2/$3
; Setting this to true would put the package 'abc' index in simple/a/abc.
; Recommended setting: the default of false for full pip/pypi compatability.
hash-index = false

; Whether to stop a sync quickly after an error is found or whether to continue
; syncing but not marking the sync as successful. Value should be "true" or
; "false".
stop-on-error = false

; Whether or not files that have been deleted on the master should be deleted
; on the mirror, too.
; IMPORTANT: if you are running an official mirror than you *need* to leave
; this on.
delete-packages = true

; Advanced logging configuration. Uncomment and set to the location of a
; python logging format logging config file.
; log-config = /etc/bandersnatch-log.conf

[statistics]
; A glob pattern matching all access log files that should be processed to
; generate daily access statistics that will be aggregated on the master PyPI.
access-log-pattern = /var/log/nginx/*.pypi.python.org*access*

; vim: set ft=cfg:

开始同步

bandersnatch  mirror

添加到定时任务

添加到cron的定时任务，例如每天同步。

[root@Ctrust ~]# cat /etc/cron.daily/sync-pip.cron
#!/bin/bash
/usr/bin/bandersnatch mirror