apline无法向gitlab上传git lfs问题
2020-12-29 18:32
标签：end ica coding 导致 int view 异常 finish space 在k8s中基于alpine做底层系统的容器进行git lfs push操作时，发现报错无法上传成功 而git clone、commit等操作均正常，并且git lfs push的操作在另一个机器A上能正常执行。 先根据报错信息搜索，基本回答是此报错是git server(gitlab)侧发生错误，不过可以使用如下命令来打印trace跟踪信息(via: https://github.com/git-lfs/git-lfs/issues/3613#issuecomment-483852691) 确实打印了更多信息，包括与Server的请求过程，如下： 其中关键请求如下 同时到gitlab server执行tail -f /var/log/gitlab//.log，日志如下 可见与client对应，500的报错信息如下，但是搜索后发现没有相关线索 线索中断，想到机器A可以正常push，所以进行对比观察，执行相同操作，client日志如下 其中关键请求如下，其中500请求已变成200 而gitalb server对应日志如下 变化如下 gitlab server相同、操作的仓库分支相同的情况下，这样的差异推测是client客户端导致的了，先来看版本 其中正常的机器A是物理机centos用yum安装的git和git-lfs；异常的是跑在k8s集群中基于alpine的pod，是通过apk安装的，遂尝试apk降级git和git-lfs到机器A正常的版本。apk降级需要到其市场查找目标版本https://pkgs.alpinelinux.org/packages?name=git-lfs&branch=v3.9 。 再次执行原push等操作，终于正常了！以防万一，再到原本正常的机器A把git-lfs升级一下，看是否也异常。 果不其然，也重现了相同异常。 首先异常与git版本没有关系，只和git-lfs版本有关。之后又尝试了更低的2.3.x、更高的2.6.x~2.11.x。确定是从2.5.x开始的版本有异常。另外还需要关注gitlab的版本,目前为9.4.1 目前只能推测该gitlab/lfs server版本与git lfs client push不兼容，所以采取的方案是lfs降为2.4.x，alpine使用v3.8 repository重新安装git-lfs。 “架构人生，迭代生命” ——深邃老夏，搜索summer_deep微信公众号可获取更多帮助 apline无法向gitlab上传git lfs问题 标签：end ica coding 导致 int view 异常 finish space 原文地址：https://www.cnblogs.com/deepsummer/p/13791583.html1 背景
Fatal error: Server error: http://git.ops.xxxxx.com/xxxx/yyyy.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3
error: failed to push some refs to ‘http://git.ops.xxxx.com/xxxx/yyyy.git‘
2 探索思路
2.1 从报错入手
GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push origin xxx/xxx
07:18:13.442791 trace git-lfs: HTTP: POST http://xxxx:xxxx@git.ops.yunnex.com/suncard/packages.git/info/lfs/objects/batch
> POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1
> Host: git.ops.yunnex.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 158
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)
>
{"operation":"upload","objects":[{"oid":"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e","size":3}],"ref":{"name":"refs/heads/test-branch"}}07:18:13.618655 trace git-lfs: HTTP: 200
PUT /suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3 HTTP/1.1
> Host: git.ops.yunnex.com
> Authorization: Basic * * * * *
> Content-Length: 3
> Content-Type: text/plain; charset=utf-8
> User-Agent: git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)
>
07:18:13.621492 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
dd
07:18:13.621527 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
07:18:13.621536 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
07:18:13.621563 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
07:18:13.621571 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
07:18:13.621579 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
07:18:13.621522 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
07:18:13.786463 trace git-lfs: HTTP: 500
200 POST /suncard/packages.git/info/lfs/objects/batch
500 PUT /suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3
==> /var/log/gitlab/gitlab-rails/production.log "upload", "objects"=>[{"oid"=>"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e", "size"=>3}], "ref"=>{"name"=>"refs/heads/test-branch"}, "namespace_id"=>"suncard", "project_id"=>"packages.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e", "size"=>3}], "ref"=>{"name"=>"refs/heads/test-branch"}}}
Completed 200 OK in 138ms (Views: 0.3ms | ActiveRecord: 4.7ms)
Started PUT "/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3" for 10.10.51.195 at 2020-09-24 15:23:17 +0800
Processing by Projects::LfsStorageController#upload_finalize as HTML
Parameters: {"namespace_id"=>"suncard", "project_id"=>"packages.git", "oid"=>"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e", "size"=>"3"}
Completed 500 Internal Server Error in 104ms (ActiveRecord: 2.6ms)
NoMethodError (undefined method `include?‘ for nil:NilClass):
app/controllers/projects/lfs_storage_controller.rb:55:in `tmp_filename‘
app/controllers/projects/lfs_storage_controller.rb:23:in `upload_finalize‘
lib/gitlab/i18n.rb:45:in `with_locale‘
lib/gitlab/i18n.rb:51:in `with_user_locale‘
app/controllers/application_controller.rb:294:in `set_locale‘
lib/gitlab/performance_bar/peek_performance_bar_with_rack_body.rb:16:in `call‘
lib/gitlab/middleware/multipart.rb:93:in `call‘
lib/gitlab/request_profiler/middleware.rb:14:in `call‘
lib/gitlab/middleware/go.rb:16:in `call‘
lib/gitlab/etag_caching/middleware.rb:11:in `call‘
lib/gitlab/request_context.rb:18:in `call‘
lib/gitlab/metrics/connection_rack_middleware.rb:31:in `call‘
...不断重试重复相同日志
==> /var/log/gitlab/nginx/gitlab_access.log NoMethodError (undefined method `include?‘ for nil:NilClass):
app/controllers/projects/lfs_storage_controller.rb:55:in `tmp_filename‘
app/controllers/projects/lfs_storage_controller.rb:23:in `upload_finalize‘
lib/gitlab/i18n.rb:45:in `with_locale‘
2.2 对比测试
23:15:15.348869 trace git-lfs: HTTP: POST http://production:Use_1990@git.ops.yunnex.com/suncard/packages.git/info/lfs/objects/batch
> POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1
> Host: git.ops.yunnex.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 158
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)
>
{"operation":"upload","objects":[{"oid":"9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25","size":3}],"ref":{"name":"refs/heads/test-branch"}}23:15:15.525308 trace git-lfs: HTTP: 200
PUT /suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3 HTTP/1.1
> Host: git.ops.yunnex.com
> Authorization: Basic * * * * *
> Content-Length: 3
> Content-Type: application/octet-stream
> User-Agent: git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)
>
23:15:15.525963 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
23:15:15.525968 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
23:15:15.525969 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
23:15:15.525970 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
23:15:15.525972 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
23:15:15.526006 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
23:15:15.526027 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
23:15:15.931456 trace git-lfs: HTTP: 200
200 POST /suncard/packages.git/info/lfs/objects/batch
200 PUT /suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3
==> /var/log/gitlab/gitlab-rails/production.log "upload", "objects"=>[{"oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>4}], "ref"=>{"name"=>"refs/heads/test-branch"}, "namespace_id"=>"suncard", "project_id"=>"packages.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>4}], "ref"=>{"name"=>"refs/heads/test-branch"}}}
Completed 200 OK in 195ms (Views: 0.3ms | ActiveRecord: 6.1ms)
Started PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4/authorize" for 192.168.1.25 at 2020-09-23 18:36:28 +0800
Processing by Projects::LfsStorageController#upload_authorize as HTML
Parameters: {"namespace_id"=>"suncard", "project_id"=>"packages.git", "oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>"4"}
Completed 200 OK in 114ms (Views: 0.2ms | ActiveRecord: 3.0ms)
Started PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4" for 192.168.1.25 at 2020-09-23 18:36:28 +0800
Processing by Projects::LfsStorageController#upload_finalize as HTML
Parameters: {"namespace_id"=>"suncard", "project_id"=>"packages.git", "oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>"4"}
Completed 200 OK in 169ms (ActiveRecord: 51.1ms)
Updating statistics for project 32
==> /var/log/gitlab/nginx/gitlab_access.log # 多了1个200 请求
PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4/authorize"
# 下列请求从500变为200
200 PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4"
git --version 1.8.3.1(正常) 2.20.4(异常)
git lfs version 2.4.2(正常) 2.5.1(异常)
再通过降通过降repository版本来安装低版本的软件包，原alpine是v3.9，现在尝试改为v3.8apk add --no-cache --repository http://mirrors.aliyun.com/alpine/v3.8/main --repository http://mirrors.aliyun.com/alpine/v3.8/community ‘git-lfs=2.4.2-r1‘
安装过程：
mkdir /tmp/test
wget https://github.com/git-lfs/git-lfs/releases/download/v2.5.1/git-lfs-linux-amd64-v2.5.1.tar.gz
tar -xvf git-lfs-linux-amd64-v2.5.1.tar.gz
# 这样就可以在不影响原git-lfs安装的情况下进行测试，后续用/tmp/test/git-lfs进行push等操作
/tmp/test/git-lfs version
3 结论
sudo gitlab-rake gitlab:env:info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Git Version: 2.13.0
Sidekiq Version:5.0.0
Go Version: unknown
GitLab information
Version: 9.4.1
Revision: 6678b4e
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: http://git.ops.yunnex.com
HTTP Clone URL: http://git.ops.yunnex.com/some-group/some-project.git
SSH Clone URL: git@git.ops.yunnex.com:some-group/some-project.git
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 5.3.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
文章标题：apline无法向gitlab上传git lfs问题
文章链接：http://soscw.com/index.php/essay/39109.html