Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] Platform refactor #93

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
FROM mcr.microsoft.com/devcontainers/typescript-node:0-16-bullseye
ARG GOLANG_VERSION=1.18.6
ARG GOLANG_PACKAGE=https://go.dev/dl/go${GOLANG_VERSION}.linux-amd64.tar.gz
ARG GRPC_VERSION=v1.48.2
ARG PROTOC_GEN_GO_VERSION=v1.28.1
ARG PROTOC_GEN_GO_GRPC_VERSION=v1.1.0

# Install python and build tools from apt
RUN apt-get update && apt-get install -y cmake git build-essential python3-venv python3-pip python-is-python3 wget && rm -rf /var/lib/apt/lists/*

# Install golang from binary package
RUN rm -rf /go && wget ${GOLANG_PACKAGE} -qO- | tar -C / -xz
ENV GOPATH=/go
ENV PATH $GOPATH/bin:$PATH
RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
RUN mkdir -p /build

# Build GRPC; note this issue before updating: https://github.com/protocolbuffers/protobuf-javascript/issues/127
WORKDIR /build
RUN git clone --recurse-submodules -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc
WORKDIR /build/grpc
RUN mkdir -p cmake/build; cd cmake/build; cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=/usr/local ../.. && make -j 8 && make install

# Build api-interfaces
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@${PROTOC_GEN_GO_VERSION}
RUN go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@${PROTOC_GEN_GO_GRPC_VERSION}
24 changes: 24 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/typescript-node
{
"name": "Node.js & TypeScript",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"build": {
"dockerfile": "Dockerfile",
}

// Features to add to the dev container. More info: https://containers.dev/features.
// "features": {},

// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],

// Use 'postCreateCommand' to run commands after the container is created.
// "postCreateCommand": "yarn install",

// Configure tool-specific properties.
// "customizations": {},

// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
// "remoteUser": "root"
}
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,5 @@ node_modules
.idea

pybuild/
.history/
.history/
dist/
3 changes: 0 additions & 3 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +0,0 @@
[submodule "src/tensorizer"]
path = src/tensorizer
url = https://github.com/coreweave/tensorizer.git
108 changes: 5 additions & 103 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,108 +1,10 @@
## 📝 Table of Contents
# TESTING - DO NOT USE
This branch is a work in progress; it is not expected to work with our production APIs at this time.

- [About](#about)
- [Getting Started](#getting_started)
- [Usage](#usage)
## Generating protobuf stubs

## About <a name = "about"></a>

Api-Interfaces is the gRPC protocol for communication between the
[api-bridge](https://github.com/Stability-AI/api-bridge), the [api-web](https://github.com/Stability-AI/api-web),
and the [generator_server](https://github.com/Stability-AI/generator_server). Additionally, any
other client application communicating directly with `api-web` also use the interfaces generated
from this repository.

See [here](https://www.notion.so/stabilityai/Protocol-Buffer-Development-Guidelines-94c6a5b9082d4aa6a2d6bf50b86fe0ac) for some development guides.

## Getting Started <a name = "getting_started"></a>

These instructions will get you an environment setup to build the interface files from the proto source files.

### Prerequisites

The following items are needed to develop api-interfaces:
- [golang](https://go.dev/) >= 1.18
- [nodejs](https://nodejs.org/en/) >= 16.16.0
- [cmake](https://cmake.org/) >= 3.14
- [protoc](https://github.com/protocolbuffers/protobuf#protocol-compiler-installation)
- [grpc](https://grpc.io/)

It is recommended to use ssh cloning with this project for `git` and for `go get`, although `https`
does appear to work. To force ssh (for github) put the following in your `.gitconfig`:

```ini
[url "ssh://[email protected]/"]
insteadOf = https://github.com/
```

### Setup and building

After all the prerequisites are installed and available, this project can be setup by the following:

```shell
git clone --recurse-submodules [email protected]:Stability-AI/api-interfaces.git
cd api-interfaces
cmake .
cmake --build .
```

This will produce files for the various languages in [gooseai](./gooseai) to support the proto
files in [src](./src). *When rebuilding the files it is recommended to do a clean before as there
have been instances of not all files being regenerated without it.*

## 🎈 Usage <a name="usage"></a>

The generated files are all output in [gooseai](./gooseai). How to use these files depends on the
programming language being used. The following sections provide details for each of the supported
languages.

The files have different usages and not all are required depending on the situation:
| Suffix | Client | Server |
|-------------|--------|--------|
| _grpc_pb | ✔️1 | ✔️ |
| _pb_service | ✔️2 | |
| _pb | ✔️ | ✔️ |


1. Not needed for typescript/javascript clients.
2. Only needed for typscript/javascripts clients.


### Golang

For Golang the interfaces can be added to the project as a normal module require. To add them run:

```shell
go get github.com/Stability-AI/api-interfaces@latest
npm i --dev
npm run generate
```

Similarly to update them just run the same command with the short sha of the version to update to.
Use them as you would a normal module.

### Python

With the current output, the best way to consume these is to add them as a git submodule to your
project. It is recommended to use ssh clone when adding the submodule. To update them just
checkout the newer version from within the submodule (and remember to commit the submodule change
to your project).

To use them make sure the files are on the python path.


### Typescript / Javascript

With the current output, the best way to consume these is to add them as a git submodule to your
project. It is recommended to use ssh clone when adding the submodule. To update them just
checkout the newer version from within the submodule (and remember to commit the submodule change
to your project).

To use them make sure they are in a location that can be found by your typescript/javascript files.

*NOTE: Typescript requires both the typescript and javascript files to be available.*

### Other Languages / Custom Build

If not using the CMake defined builds to generate make sure when building from the source proto
files that the following proto include paths are set:
- `src/proto`
- `src/tensorizer/proto`
23 changes: 23 additions & 0 deletions buf.gen.go.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
version: v1
managed:
enabled: true
go_package_prefix:
# <module_name> : name in go.mod
# <relative_path> : where generated code should be output
default: github.com/stability-ai/api-interfaces/src
# Remove `except` field if googleapis is not used
except:
- buf.build/googleapis/googleapis
plugins:
- plugin: buf.build/connectrpc/go:v1.11.0
out: '.'
opt:
- paths=source_relative
- plugin: buf.build/grpc/go:v1.3.0
out: '.'
opt:
- paths=source_relative
- plugin: buf.build/protocolbuffers/go
out: '.'
opt:
- paths=source_relative
8 changes: 8 additions & 0 deletions buf.gen.js.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
version: v1
managed:
enabled: true
plugins:
- plugin: buf.build/bufbuild/connect-es:v0.12.0
out: '.'
- plugin: buf.build/bufbuild/es
out: '.'
8 changes: 8 additions & 0 deletions buf.gen.kt.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
version: v1
managed:
enabled: true
plugins:
- plugin: buf.build/bufbuild/connect-kotlin:v0.1.7
out: '.'
- plugin: buf.build/protocolbuffers/kotlin
out: '.'
10 changes: 10 additions & 0 deletions buf.gen.py.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
version: v1
managed:
enabled: true
plugins:
- plugin: buf.build/grpc/python:v1.56.2
out: '.'
- plugin: buf.build/protocolbuffers/python
out: '.'
- plugin: buf.build/protocolbuffers/pyi:v23.4
out: '.'
8 changes: 8 additions & 0 deletions buf.gen.swift.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
version: v1
managed:
enabled: true
plugins:
- plugin: buf.build/bufbuild/connect-swift:v0.5.0
out: '.'
- plugin: buf.build/apple/swift
out: '.'
1 change: 0 additions & 1 deletion buf.work.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
version: v1
directories:
- src/proto
- src/tensorizer/proto
4 changes: 4 additions & 0 deletions buf.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
version: v1
name: buf.build/stability-ai/api-interfaces
breaking:
use:
- FILE
build:
excludes:
- node_modules
- build
lint:
use:
- DEFAULT
3 changes: 3 additions & 0 deletions gooseai/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading