stale/gomodproxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
Serge Zaitsev 2018-09-19 11:20:09 +02:00
parent 95a937ef2c
commit ce767e10ee
11 changed files with 918 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.gitignore vendored
View File

@ -4,9 +4,14 @@
*.dll
*.so
*.dylib
/gomodproxy
# Test binary, build with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Misc
_gopath
_gocache

104
README.md
View File

@ -1 +1,103 @@
# gomodproxy
# gomodproxy
gomodproxy is a caching proxy for [Go modules].
Go 1.11 has introduced optional proxy support via GOPROXY environment variable. It is essential for use cases where you want to have better control over your dependencies and handle scenarios when GitHub is down or some open-source dependency has been removed.
## Getting started
gomodproxy requires Go 1.11 or newer. There are no plans to support `vgo` or Go 1.11 beta versions.
```
# Download and install from sources
git clone https://github.com/sixt/gomodproxy.git
cd gomodproxy
go build ./cmd/gomodproxy
# Run it
./gomodproxy -addr :8000
# Build some project using the proxy, dependencies will be downloaded to $HOME/.gomodproxy
...
GOPROXY=http://127.0.0.1:8000 go build
```
To let gomodproxy access the private Git repositories you may provide SSH keys or username/password for HTTPS access:
```
./gomodproxy \
-git bitbucket.org/mycompany:/path/to/id_rsa \
-git example.com/:/path/to/id_rsa_for_example_com \
-git github.com/mycompany:username:password
```
## Features
* Small, pragmatic and easy to use.
* Self-contained, does not require Go, Git or any other software to be installed. Just run the binary.
* Supports Git out of the box. Other VCS are going to be supported via the plugins.
* Caches dependencies in memory or to the local disk. S3 support is planned in the nearest future. Other store types are likely to be supported via plugins.
## How it works
The entry point is cmd/gomodproxy/main.go. According to the given command-line flags it initializes:
* the `api` package implementing for Go proxy API
* the `vcs` package implementing the Git client
* the `store` package implementing in-memory and disk-based stores
### API
API package implements the HTTP proxy API as described in the [Go documentation].
**GET /:module/@v/list**
Queries the VCS to retrieve either a list of version tags, or the latest commit hash if the package does not use semantic versioning. This is the only request that is not cached and always contains the recent VCS hosting information.
**GET /:module/@v/:version.info**
Returns a JSON specifying the module version and the timestamps of the corresponding commit.
**GET /:module/@v/:version.mod**
If a `go.mod` file is present in the sources of the requested module - it is returned unmodified. Otherwise a minimal synthetic `go.mod` with no required module dependencies is generated.
**GET /:module/@v/:version.zip**
Returns ZIP archive contents with the snapshot of the requested module version. To keep the checksums unchanged, we follow the same (sometimes weird) refinements as does the Go tool - stripping off vendor directories, setting file timestamps back to 1980 etc.
On every request API tries to look for a module in the caches, and if it's not there - it fetches the requested revision using the `vcs` package and fulfils the caches.
### VCS
VCS package defines an interface for a typical VCS client and implements a Git client using `go-git` library:
It closely follows the logic of how `go get` fetches the modules, and implements all the quirks, such as go-imports meta tag resolution, or removing vendor directories from the repos.
The plugins are planned to be implemented as external command-line utilities written in any programming language. The protocol is to be defined yet.
### Store
Store package defines an interface for a caching store and provides the following store implementations:
* In-memory LRU cache of given capacity
* Disk-based directory cache
* S3 store
Other store implementations are planned to be supported similarly to VCS plugins, as external utilities following a defined command-line protocol.
## Contributing
The code in this project is licensed under Apache 2.0 license.
Please, use the search tool before opening a new issue. If you have found a bug please provide as many details as you can. Also, if you are interested in helping this project you may review the open issues and leave some feedback or react to them.
If you make a pull request, please consider the following:
* Open your pull request against `master` branch.
* Your pull request should have no more than two commits, otherwise please squash them.
* All tests should pass.
* You should add or modify tests to cover your proposed code changes.
* If your pull request contains a new feature, please document it in the README.
[Go modules]: https://github.com/golang/go/wiki/Modules
[Go documentation]: https://golang.org/cmd/go/#hdr-Module_proxy_protocol

103
cmd/gomodproxy/main.go Normal file
View File

@ -0,0 +1,103 @@
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"log"
"net"
"net/http"
"os"
"os/signal"
"path/filepath"
"strings"
"time"
"github.com/sixt/gomodproxy/pkg/api"
)
func prettyLog(v ...interface{}) {
s := ""
msg := ""
if len(v)%2 != 0 {
msg = fmt.Sprintf("%s", v[0])
v = v[1:]
}
s = fmt.Sprintf("%20s ", msg)
for i := 0; i < len(v); i = i + 2 {
s = s + fmt.Sprintf("%v=%v ", v[i], v[i+1])
}
log.Println(s)
}
func jsonLog(v ...interface{}) {
entry := map[string]interface{}{}
if len(v)%2 != 0 {
entry["msg"] = v[0]
v = v[1:]
}
for i := 0; i < len(v); i = i + 2 {
entry[fmt.Sprintf("%v", v[i])] = v[i+1]
}
json.NewEncoder(os.Stdout).Encode(entry)
}
type listFlag []string
func (f *listFlag) String() string { return strings.Join(*f, " ") }
func (f *listFlag) Set(s string) error { *f = append(*f, s); return nil }
func main() {
gitPaths := listFlag{}
addr := flag.String("addr", ":0", "http server address")
verbose := flag.Bool("v", false, "verbose logging")
json := flag.Bool("json", false, "json structured logging")
dir := flag.String("dir", filepath.Join(os.Getenv("HOME"), ".gomodproxy"), "cache directory")
flag.Var(&gitPaths, "git", "list of git settings")
flag.Parse()
ln, err := net.Listen("tcp", *addr)
if err != nil {
log.Fatal("net.Listen:", err)
}
defer ln.Close()
fmt.Println("Listening on", ln.Addr())
options := []api.Option{}
if *verbose || *json {
if *json {
options = append(options, api.Log(jsonLog))
} else {
options = append(options, api.Log(prettyLog))
}
}
for _, path := range gitPaths {
kv := strings.SplitN(path, "=", 2)
if len(kv) != 2 {
log.Fatal("bad git path:", path)
}
options = append(options, api.Git(kv[0], kv[1]))
}
options = append(options, api.Memory(), api.CacheDir(*dir))
sigc := make(chan os.Signal, 1)
signal.Notify(sigc, os.Interrupt)
srv := &http.Server{Handler: api.New(options...)}
go func() {
if err := srv.Serve(ln); err != nil {
log.Fatal(err)
}
}()
<-sigc
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
srv.Shutdown(ctx)
}

18
go.mod Normal file
View File

@ -0,0 +1,18 @@
module github.com/sixt/gomodproxy
require (
github.com/emirpasic/gods v1.9.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/kevinburke/ssh_config v0.0.0-20180830205328-81db2a75821e // indirect
github.com/mitchellh/go-homedir v1.0.0 // indirect
github.com/pelletier/go-buffruneio v0.2.0 // indirect
github.com/sergi/go-diff v1.0.0 // indirect
github.com/src-d/gcfg v1.3.0 // indirect
github.com/xanzy/ssh-agent v0.2.0 // indirect
golang.org/x/crypto v0.0.0-20180910181607-0e37d006457b // indirect
golang.org/x/net v0.0.0-20180911220305-26e67e76b6c3 // indirect
golang.org/x/text v0.3.0 // indirect
gopkg.in/src-d/go-billy.v4 v4.2.1 // indirect
gopkg.in/src-d/go-git.v4 v4.7.0
gopkg.in/warnings.v0 v0.1.2 // indirect
)

33
go.sum Normal file
View File

@ -0,0 +1,33 @@
github.com/emirpasic/gods v1.9.0 h1:rUF4PuzEjMChMiNsVjdI+SyLu7rEqpQ5reNFnhC7oFo=
github.com/emirpasic/gods v1.9.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
github.com/kevinburke/ssh_config v0.0.0-20180830205328-81db2a75821e h1:RgQk53JHp/Cjunrr1WlsXSZpqXn+uREuHvUVcK82CV8=
github.com/kevinburke/ssh_config v0.0.0-20180830205328-81db2a75821e/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/mitchellh/go-homedir v1.0.0 h1:vKb8ShqSby24Yrqr/yDYkuFz8d0WUjys40rvnGC8aR0=
github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/pelletier/go-buffruneio v0.2.0 h1:U4t4R6YkofJ5xHm3dJzuRpPZ0mr5MMCoAWooScCR7aA=
github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtbSNq5BcowyYOWdKo=
github.com/sergi/go-diff v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/src-d/gcfg v1.3.0 h1:2BEDr8r0I0b8h/fOqwtxCEiq2HJu8n2JGZJQFGXWLjg=
github.com/src-d/gcfg v1.3.0/go.mod h1:p/UMsR43ujA89BJY9duynAwIpvqEujIH/jFlfL7jWoI=
github.com/xanzy/ssh-agent v0.2.0 h1:Adglfbi5p9Z0BmK2oKU9nTG+zKfniSfnaMYB+ULd+Ro=
github.com/xanzy/ssh-agent v0.2.0/go.mod h1:0NyE30eGUDliuLEHJgYte/zncp2zdTStcOnWhgSqHD8=
golang.org/x/crypto v0.0.0-20180910181607-0e37d006457b h1:2b9XGzhjiYsYPnKXoEfL7klWZQIt8IfyRCz62gCqqlQ=
golang.org/x/crypto v0.0.0-20180910181607-0e37d006457b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/net v0.0.0-20180911220305-26e67e76b6c3 h1:czFLhve3vsQetD6JOJ8NZZvGQIXlnN3/yXxbT6/awxI=
golang.org/x/net v0.0.0-20180911220305-26e67e76b6c3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/sys v0.0.0-20180903190138-2b024373dcd9/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/src-d/go-billy.v4 v4.2.1 h1:omN5CrMrMcQ+4I8bJ0wEhOBPanIRWzFC953IiXKdYzo=
gopkg.in/src-d/go-billy.v4 v4.2.1/go.mod h1:tm33zBoOwxjYHZIE+OV8bxTWFMJLrconzFMd38aARFk=
gopkg.in/src-d/go-git.v4 v4.7.0 h1:WXB+2gCoRhQiAr//IMHpIpoDsTrDgvjDORxt57e8XTA=
gopkg.in/src-d/go-git.v4 v4.7.0/go.mod h1:CzbUWqMn4pvmvndg3gnh5iZFmSsbhyhUWdI0IQ60AQo=
gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME=
gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=

230
pkg/api/api.go Normal file
View File

@ -0,0 +1,230 @@
package api
import (
"archive/zip"
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"path/filepath"
"regexp"
"strings"
"time"
"unicode"
"github.com/sixt/gomodproxy/pkg/store"
"github.com/sixt/gomodproxy/pkg/vcs"
)
type logger = func(v ...interface{})
type api struct {
log logger
vcsPaths []vcsPath
stores []store.Store
}
type vcsPath struct {
prefix string
vcs func(module string) vcs.VCS
}
type Option func(*api)
var (
apiList = regexp.MustCompile(`^/(?P<module>.*)/@v/list$`)
apiInfo = regexp.MustCompile(`^/(?P<module>.*)/@v/(?P<version>.*).info$`)
apiMod = regexp.MustCompile(`^/(?P<module>.*)/@v/(?P<version>.*).mod$`)
apiZip = regexp.MustCompile(`^/(?P<module>.*)/@v/(?P<version>.*).zip$`)
)
func New(options ...Option) http.Handler {
api := &api{log: func(...interface{}) {}}
for _, opt := range options {
opt(api)
}
return api
}
func Log(log logger) Option { return func(api *api) { api.log = log } }
func Git(prefix string, auth string) Option {
a := vcs.Key(auth)
if creds := strings.SplitN(auth, ":", 2); len(creds) == 2 {
a = vcs.Password(creds[0], creds[1])
}
return func(api *api) {
api.vcsPaths = append(api.vcsPaths, vcsPath{
prefix: prefix,
vcs: func(module string) vcs.VCS {
return vcs.NewGit(api.log, module, a)
},
})
}
}
func Memory() Option {
return func(api *api) {
api.stores = append(api.stores, store.Memory())
}
}
func CacheDir(dir string) Option {
return func(api *api) {
api.stores = append(api.stores, store.Disk(dir))
}
}
func decodeBangs(s string) string {
buf := []rune{}
bang := false
for _, r := range []rune(s) {
if bang {
bang = false
buf = append(buf, unicode.ToUpper(r))
continue
}
if r == '!' {
bang = true
continue
}
buf = append(buf, r)
}
return string(buf)
}
func (api *api) ServeHTTP(w http.ResponseWriter, r *http.Request) {
now := time.Now()
defer func() { api.log("api.ServeHTTP", "method", r.Method, "url", r.URL, "time", time.Since(now)) }()
for _, route := range []struct {
regexp *regexp.Regexp
handler func(w http.ResponseWriter, r *http.Request, module, version string)
}{
{apiList, api.list},
{apiInfo, api.info},
{apiMod, api.mod},
{apiZip, api.zip},
} {
if m := route.regexp.FindStringSubmatch(r.URL.Path); m != nil {
module, version := m[1], ""
if len(m) > 2 {
version = m[2]
}
module = decodeBangs(module)
route.handler(w, r, module, version)
return
}
}
http.NotFound(w, r)
}
func (api *api) vcs(ctx context.Context, module string) vcs.VCS {
for _, path := range api.vcsPaths {
if strings.HasPrefix(module, path.prefix) {
return path.vcs(module)
}
}
return vcs.NewGit(api.log, module, vcs.NoAuth())
}
func (api *api) module(ctx context.Context, module string, version vcs.Version) ([]byte, time.Time, error) {
for _, store := range api.stores {
if snapshot, err := store.Get(ctx, module, version); err == nil {
return snapshot.Data, snapshot.Timestamp, nil
}
}
timestamp, err := api.vcs(ctx, module).Timestamp(ctx, version)
if err != nil {
return nil, time.Time{}, err
}
b := &bytes.Buffer{}
zr, err := api.vcs(ctx, module).Zip(ctx, version)
if err != nil {
return nil, time.Time{}, err
}
defer zr.Close()
if _, err := io.Copy(b, zr); err != nil {
return nil, time.Time{}, err
}
for i := len(api.stores) - 1; i >= 0; i-- {
err := api.stores[i].Put(ctx, store.Snapshot{
Module: module,
Version: version,
Timestamp: timestamp,
Data: b.Bytes(),
})
if err != nil {
api.log("api.module.Put", "module", module, "version", version, "error", err)
}
}
return b.Bytes(), timestamp, nil
}
func (api *api) list(w http.ResponseWriter, r *http.Request, module, version string) {
api.log("api.list", "module", module)
list, err := api.vcs(r.Context(), module).List(r.Context())
if err != nil {
api.log("api.list", "module", module, "error", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
for _, v := range list {
fmt.Fprintln(w, string(v))
}
}
func (api *api) info(w http.ResponseWriter, r *http.Request, module, version string) {
api.log("api.info", "module", module, "version", version)
_, t, err := api.module(r.Context(), module, vcs.Version(version))
if err != nil {
api.log("api.info", "module", module, "version", version, "error", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
json.NewEncoder(w).Encode(struct {
Version string
Time time.Time
}{version, t})
}
func (api *api) mod(w http.ResponseWriter, r *http.Request, module, version string) {
api.log("api.mod", "module", module, "version", version)
b, _, err := api.module(r.Context(), module, vcs.Version(version))
if err == nil {
if zr, err := zip.NewReader(bytes.NewReader(b), int64(len(b))); err == nil {
for _, f := range zr.File {
if f.Name == filepath.Join(module+"@"+string(version), "go.mod") {
if r, err := f.Open(); err == nil {
defer r.Close()
io.Copy(w, r)
return
}
}
}
}
}
w.Write([]byte(fmt.Sprintf("module %s\n", module)))
}
func (api *api) zip(w http.ResponseWriter, r *http.Request, module, version string) {
api.log("api.zip", "module", module, "version", version)
b, _, err := api.module(r.Context(), module, vcs.Version(version))
if err != nil {
api.log("api.zip", "module", module, "version", version, "error", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
io.Copy(w, bytes.NewReader(b))
}

49
pkg/store/disk.go Normal file
View File

@ -0,0 +1,49 @@
package store
import (
"context"
"io/ioutil"
"os"
"path/filepath"
"github.com/sixt/gomodproxy/pkg/vcs"
)
type disk string
func Disk(dir string) Store { return disk(dir) }
func (d disk) Put(ctx context.Context, snapshot Snapshot) error {
dir := string(d)
timeFile := filepath.Join(dir, snapshot.Key()+".time")
zipFile := filepath.Join(dir, snapshot.Key()+".zip")
if err := os.MkdirAll(filepath.Dir(timeFile), 0755); err != nil {
return err
}
t, err := snapshot.Timestamp.MarshalText()
if err != nil {
return err
}
if err := ioutil.WriteFile(timeFile, t, 0644); err != nil {
return err
}
return ioutil.WriteFile(zipFile, snapshot.Data, 0644)
}
func (d disk) Get(ctx context.Context, module string, version vcs.Version) (Snapshot, error) {
dir := string(d)
s := Snapshot{Module: module, Version: version}
t, err := ioutil.ReadFile(filepath.Join(dir, s.Key()+".time"))
if err != nil {
return Snapshot{}, err
}
if err := s.Timestamp.UnmarshalText(t); err != nil {
return Snapshot{}, err
}
s.Data, err = ioutil.ReadFile(filepath.Join(dir, s.Key()+".zip"))
return s, err
}
func (d disk) Close() error { return nil }

41
pkg/store/mem.go Normal file
View File

@ -0,0 +1,41 @@
package store
import (
"context"
"errors"
"sync"
"github.com/sixt/gomodproxy/pkg/vcs"
)
type memory struct {
sync.Mutex
cache []Snapshot
}
func Memory() Store { return &memory{} }
func (m *memory) Put(ctx context.Context, snapshot Snapshot) error {
m.Lock()
defer m.Unlock()
for _, item := range m.cache {
if item.Module == snapshot.Module && item.Version == snapshot.Version {
return nil
}
}
m.cache = append(m.cache, snapshot)
return nil
}
func (m *memory) Get(ctx context.Context, module string, version vcs.Version) (Snapshot, error) {
m.Lock()
defer m.Unlock()
for _, snapshot := range m.cache {
if snapshot.Module == module && snapshot.Version == version {
return snapshot, nil
}
}
return Snapshot{}, errors.New("not found")
}
func (m *memory) Close() error { return nil }

25
pkg/store/store.go Normal file
View File

@ -0,0 +1,25 @@
package store
import (
"context"
"time"
"github.com/sixt/gomodproxy/pkg/vcs"
)
type Store interface {
Put(ctx context.Context, snapshot Snapshot) error
Get(ctx context.Context, module string, version vcs.Version) (Snapshot, error)
Close() error
}
type Snapshot struct {
Module string
Version vcs.Version
Timestamp time.Time
Data []byte
}
func (s Snapshot) Key() string {
return s.Module + "@" + string(s.Version)
}

221
pkg/vcs/git.go Normal file
View File

@ -0,0 +1,221 @@
package vcs
import (
"archive/zip"
"bytes"
"context"
"errors"
"fmt"
"io"
"io/ioutil"
"path/filepath"
"strings"
"time"
"gopkg.in/src-d/go-git.v4"
"gopkg.in/src-d/go-git.v4/config"
"gopkg.in/src-d/go-git.v4/plumbing"
"gopkg.in/src-d/go-git.v4/plumbing/object"
"gopkg.in/src-d/go-git.v4/plumbing/transport"
"gopkg.in/src-d/go-git.v4/plumbing/transport/http"
"gopkg.in/src-d/go-git.v4/plumbing/transport/ssh"
"gopkg.in/src-d/go-git.v4/storage/memory"
)
const remoteName = "origin"
type gitVCS struct {
log logger
module string
auth Auth
}
func NewGit(l logger, module string, auth Auth) VCS {
return &gitVCS{log: l, module: module, auth: auth}
}
func (g *gitVCS) List(ctx context.Context) ([]Version, error) {
g.log("gitVCS.List", "module", g.module)
repo, err := g.repo(ctx)
if err != nil {
return nil, err
}
remote, err := repo.Remote(remoteName)
if err != nil {
return nil, err
}
auth, err := g.authMethod()
if err != nil {
return nil, err
}
refs, err := remote.List(&git.ListOptions{Auth: auth})
if err != nil {
return nil, err
}
list := []Version{}
masterHash := ""
for _, ref := range refs {
name := ref.Name()
if name == plumbing.Master {
masterHash = ref.Hash().String()
} else if name.IsTag() && strings.HasPrefix(name.String(), "refs/tags/v") {
list = append(list, Version(strings.TrimPrefix(name.String(), "refs/tags/")))
}
}
if len(list) == 0 {
if masterHash == "" {
return nil, errors.New("no tags and no master branch found")
}
short := masterHash[:12]
t, err := g.Timestamp(ctx, Version("v0.0.0-20060102150405-"+short))
if err != nil {
return nil, err
}
list = []Version{Version(fmt.Sprintf("v0.0.0-%s-%s", t.Format("20060102150405"), short))}
}
g.log("gitVCS.List", "module", g.module, "list", list)
return list, nil
}
func (g *gitVCS) Timestamp(ctx context.Context, version Version) (time.Time, error) {
g.log("gitVCS.Timestamp", "module", g.module, "version", version)
ci, err := g.commit(ctx, version)
if err != nil {
return time.Time{}, err
}
g.log("gitVCS.Timestamp", "module", g.module, "version", version, "timestamp", ci.Committer.When)
return ci.Committer.When, nil
}
func isVendoredPackage(name string) bool {
var i int
if strings.HasPrefix(name, "vendor/") {
i += len("vendor/")
} else if j := strings.Index(name, "/vendor/"); j >= 0 {
i += len("/vendor/")
} else {
return false
}
return strings.Contains(name[i:], "/")
}
func (g *gitVCS) Zip(ctx context.Context, version Version) (io.ReadCloser, error) {
g.log("gitVCS.Zip", "module", g.module, "version", version)
ci, err := g.commit(ctx, version)
if err != nil {
return nil, err
}
tree, err := ci.Tree()
if err != nil {
return nil, err
}
b := &bytes.Buffer{}
zw := zip.NewWriter(b)
tree.Files().ForEach(func(f *object.File) error {
// go mod strips vendored directories from the zip, and we do the same
// to match the checksums in the go.sum
if isVendoredPackage(f.Name) {
return nil
}
w, err := zw.Create(filepath.Join(g.module+"@"+string(version), f.Name))
if err != nil {
return err
}
r, err := f.Reader()
if err != nil {
return err
}
defer r.Close()
io.Copy(w, r)
return nil
})
zw.Close()
return ioutil.NopCloser(bytes.NewBuffer(b.Bytes())), nil
}
func (g *gitVCS) repo(ctx context.Context) (*git.Repository, error) {
repo, err := git.Init(memory.NewStorage(), nil)
if err != nil {
return nil, err
}
schema := "https://"
if g.auth.Key != "" {
schema = "ssh://"
}
repoRoot := g.module
if meta, err := MetaImports(ctx, g.module); err == nil {
repoRoot = meta
}
_, err = repo.CreateRemote(&config.RemoteConfig{
Name: remoteName,
URLs: []string{schema + repoRoot + ".git"},
})
return repo, err
}
func (g *gitVCS) commit(ctx context.Context, version Version) (*object.Commit, error) {
repo, err := g.repo(ctx)
if err != nil {
return nil, err
}
auth, err := g.authMethod()
if err != nil {
return nil, err
}
err = repo.FetchContext(ctx, &git.FetchOptions{
RemoteName: remoteName,
Auth: auth,
})
if err != nil {
return nil, err
}
version = Version(strings.TrimSuffix(string(version), "+incompatible"))
hash := version.Hash()
if version.IsSemVer() {
tags, err := repo.Tags()
if err != nil {
return nil, err
}
tags.ForEach(func(t *plumbing.Reference) error {
if t.Name().String() == "refs/tags/"+string(version) {
hash = t.Hash().String()
annotated, err := repo.TagObject(t.Hash())
if err == nil {
hash = annotated.Target.String()
}
}
return nil
})
} else {
commits, err := repo.CommitObjects()
if err != nil {
return nil, err
}
commits.ForEach(func(ci *object.Commit) error {
if strings.HasPrefix(ci.Hash.String(), version.Hash()) {
hash = ci.Hash.String()
}
return nil
})
}
g.log("gitVCS.commit", "module", g.module, "version", version, "hash", hash)
return repo.CommitObject(plumbing.NewHash(hash))
}
func (g *gitVCS) authMethod() (transport.AuthMethod, error) {
if g.auth.Key != "" {
return ssh.NewPublicKeysFromFile("git", g.auth.Key, "")
} else if g.auth.Username != "" {
return &http.BasicAuth{Username: g.auth.Username, Password: g.auth.Password}, nil
}
return nil, nil
}

90
pkg/vcs/vcs.go Normal file
View File

@ -0,0 +1,90 @@
package vcs
import (
"context"
"encoding/xml"
"errors"
"io"
"net/http"
"regexp"
"strings"
"time"
)
type logger = func(v ...interface{})
type Version string
var reSemVer = regexp.MustCompile(`^v\d+\.\d+\.\d+$`)
func (v Version) IsSemVer() bool { return reSemVer.MatchString(string(v)) }
func (v Version) Hash() string {
fields := strings.Split(string(v), "-")
if len(fields) != 3 {
return ""
}
return fields[2]
}
type Module interface {
Timestamp(ctx context.Context, version Version) (time.Time, error)
Zip(ctx context.Context, version Version) (io.ReadCloser, error)
}
type VCS interface {
List(ctx context.Context) ([]Version, error)
Module
}
type Auth struct {
Username string
Password string
Key string
}
func NoAuth() Auth { return Auth{} }
func Password(username, password string) Auth { return Auth{Username: username, Password: password} }
func Key(key string) Auth { return Auth{Key: key} }
func MetaImports(ctx context.Context, module string) (string, error) {
if strings.HasPrefix(module, "github.com/") || strings.HasPrefix(module, "bitbucket.org/") {
return module, nil
}
// TODO: use context
res, err := http.Get("https://" + module + "?go-get=1")
if err != nil {
return "", err
}
defer res.Body.Close()
html := struct {
HTML string `xml:"html"`
Head struct {
Meta []struct {
Name string `xml:"name,attr"`
Content string `xml:"content,attr"`
} `xml:"meta"`
} `xml:"head"`
}{}
dec := xml.NewDecoder(res.Body)
dec.Strict = false
dec.AutoClose = xml.HTMLAutoClose
dec.Entity = xml.HTMLEntity
if err := dec.Decode(&html); err != nil {
return "", err
}
for _, meta := range html.Head.Meta {
if meta.Name == "go-import" {
if f := strings.Fields(meta.Content); len(f) == 3 {
if f[0] != module {
return "", errors.New("prefix does not match the module")
}
url := f[2]
if i := strings.Index(url, "://"); i >= 0 {
url = url[i+3:]
}
return url, nil
}
}
}
return "", errors.New("go-import meta tag not found")
}