upgrade supabase docker, improve docker setup and docs (#809)

.gitignore

@@ -50,5 +50,7 @@ next-env.d.ts
 
 # supabase
 v2.code-workspace
+# supabase setup upgrade temp folder
+libs/.docker/ui/supabase/supabase-setup
 
 google_cloud_service_key.json

fern/docs.yml

@@ -48,8 +48,8 @@ navigation:
             path: ./mdx/sdk/saml.mdx
       - section: Deployments
         contents:
-          - page: Docker
-            path: ./mdx/deploy/docker.mdx     
+          - page: Docker Compose
+            path: ./mdx/deploy/docker-compose.mdx     
           - page: Render
             path: ./mdx/deploy/render.mdx
           - page: Replit
@@ -59,7 +59,7 @@ navigation:
       - section: Logging
         contents:
           - page: Langfuse
-            path: ./mdx/dependencies/langfuse.mdx
+            path: ./mdx/logging/langfuse.mdx
       - section: Vector Stores
         contents:
           - page: Weaviate
@@ -75,7 +75,7 @@ navigation:
       - section: Memory 
         contents: 
           - page: Motorhead
-            path: ./mdx/dependencies/motorhead.mdx
+            path: ./mdx/memory/motorhead.mdx
       - section: Dependencies
         contents:
           - page: Supabase

fern/mdx/dependencies/supabase.mdx

@@ -1,6 +1,6 @@
 Supabase is used for the Superagent UI, it provides the database, Authentication with well know auth providers and a Bucket for file storage.
 
-**Website** - [supabase.com/](https://supabase.com/)
+**Website** - [supabase.com](https://supabase.com)
 <br/>
 **Github** - [supabase/supabase](https://github.com/supabase/supabase)
 
@@ -57,7 +57,7 @@ For a production setup refer to the [official documentation](https://supabase.co
    ./run.sh
    ```
 
-   - The script will ensure the superagent docker network exists, stops any previously running containers, then runs supabase with docker-compose.
+   - The script will ensure the superagent docker network exists, stops any previously running containers, then runs supabase with docker compose.
 
    ``` bash
     # Check if the network exists
@@ -73,7 +73,7 @@ For a production setup refer to the [official documentation](https://supabase.co
 
 6. You can access the supabase ui in the browser to confirm the service is up.
 
-    [http://localhost:8082](http://localhost:8082) or [http://supabase-kong:8082](http://supabase:8082) if you have added the following to your `hosts` file.
+    [http://localhost:8000](http://localhost:8000) or [http://supabase-kong:8000](http://supabase-kong:8000) if you have added the following to your `hosts` file.
 
     ``` bash
     # hosts file
@@ -84,11 +84,16 @@ For a production setup refer to the [official documentation](https://supabase.co
 
 ### Supabase Storage
 
-File storage is already configured in the provided supabase docker setup. This is created when you deploy using the script from `libs\.docker\ui\supabase\volumes\db\init\data.sql`
+File storage is already configured in the provided supabase docker setup. This is created when you deploy using the script from `libs\.docker\ui\supabase\volumes\db\init\data.sql`. Though there is one manual step needed to enable superagent to access it:
+
+#### Required setup to enable storage access
 
 You can access the storage and manage files at the following url
 
 - [http://localhost:8000/project/default/storage/buckets/superagent](http://localhost:8000/project/default/storage/buckets/superagent)
+- Click the drop-down next to the `superagent` stoage
+- Edit
+- Set the toggle to `public`
 
 ### Supabase container logs
 

fern/mdx/deploy/docker.mdx → fern/mdx/deploy/docker-compose.mdx

@@ -1,8 +1,8 @@
-# Run Superagent in Docker
+# Run Superagent with Docker Compose
 
-You can run some or all of Superagent with Docker, either locally or as part of your production deployment. Examples are provided to enable you to quickly get started running Superagent and its dependeincies with Docker, these can be used to quickly spin up and try out Superagent, or be used as the basis for a production deployment.
+You can run some or all of Superagent with Docker Compose, either locally or as part of your production deployment. Examples are provided to enable you to quickly get started running Superagent and its dependencies, these can be used to quickly spin up and try out Superagent, or be used as the basis for a production deployment.
 
-While we provide examples on how to run the various dependencies with Docker, you can also choose to run any of the dependencies with cloud or hosted versions where available, pick and choose the services you wish to run with docker or swap out the enviroment configuration to point at cloud hosted versions if you prefer.
+While we provide examples on how to run the various dependencies and integrations with Docker, you can also choose to run any of them with cloud or hosted versions where available, pick and choose the services you wish to run with docker or swap out the enviroment configuration to point at cloud hosted versions if you prefer.
 
 ## Deployment options
 
@@ -54,11 +54,11 @@ Here is an example hosts file with most of the services listed. Add or remove se
 
 ### 1. Deploy Supabase
 
-First setup supabase. Refer to our guide [Supabase setup](https://docs.superagent.sh/overview/deployments/render). The guide explains how to setup and run Supabase with docker compose using our provided example.
+First setup supabase. Refer to our guide [Supabase setup](https://docs.superagent.sh/overview/dependencies/supabase). The guide explains how to setup and run Supabase with docker compose using our provided example.
 
 ### 2. Deploy Langfuse
 
-Refer to our guide [Langfuse setup](https://docs.superagent.sh/overview/dependencies/langfuse). The guide explains how to run a local version of Langfuse with docker compose using our provided example, or how to setup Langfuse cloud if preferred.
+Refer to our guide [Langfuse setup](https://docs.superagent.sh/overview/logging/langfuse). The guide explains how to run a local version of Langfuse with docker compose using our provided example, or how to setup Langfuse cloud if preferred.
 
 ### 3. Deploy a vector store
 
@@ -71,7 +71,7 @@ Refer to our guides on setting up one of the supported vector stores in docker o
 
 The main docker compose file is located in `libs\.docker\docker-compose.yml` and contains only the Superagent API. Other core services are provided as additional compose files and can be appended to the `docker compose up` command to deploy them all at once.
 
-In order to deploy all the core services an example script is provided in `libs\.docker\run.sh`.  You can modify this script to ommit any services by commenting out or deleting lines.
+In order to deploy all the core services an example script is provided in `libs\.docker\run.sh`.  You can modify this script to ommit any services you wish to deploy elsewhere see [Swapping out core dependencies](#swapping-out-core-dependencies).
 
 To run all of these services in docker, cd into the `libs/.docker` folder, run the script `run.sh`:
 
@@ -103,7 +103,7 @@ Langfuse: [http://localhost:3100](http://localhost:3100)
 
 ## Swapping out core dependencies
 
-The `libs/.docker/run.sh` script runs all the core dependencies. You can omit any part that you may wish to deploy elsewhere. For example, if you wanted to use a cloud hosted database for the superagent api, then you can comment/delete the lines for `-f superagent/db/docker-compose.pgdb.yml \` and `-f superagent/db/docker-compose.pgadmin.yml \` then modify the `libs/.docker/.env` file with the configuration for your own POSTGRES db.
+The `libs/.docker/run.sh` script runs all the core dependencies. You can omit any part that you may wish to deploy elsewhere. For example, if you wanted to use a cloud hosted database for the superagent api, then you can delete the lines for `-f superagent/db/docker-compose.pgdb.yml \` and `-f superagent/db/docker-compose.pgadmin.yml \` then modify the `libs/.docker/.env` file with the configuration for your own POSTGRES db.
 
 ``` bash
 # Remove any running services
@@ -117,7 +117,7 @@ fi
 
 # Run the core services
 docker-compose -f docker-compose.yml \
-        ### Comment out or remove any parts you have deployed elsewhere.
+        ### Delete any services you have deployed elsewhere.
         -f superagent/db/docker-compose.pgdb.yml \
         -f superagent/db/docker-compose.pgadmin.yml \
         -f superagent/motorhead/docker-compose.motorhead.yml \

fern/mdx/dependencies/langfuse.mdx → fern/mdx/logging/langfuse.mdx

@@ -14,6 +14,8 @@ For a production setup refer to the [official documentation](https://langfuse.co
 
 ### Run langfuse with docker compose
 
+The following setup is provided based on the [docker compose](https://langfuse.com/docs/deployment/local) documentation from langfuse.
+
 1. `cd` into the langfuse example folder
 
     ``` bash
@@ -82,7 +84,7 @@ Use the provided `uninstall.sh` script to remove running services for lanfguse
 
 ## Setting up Langfuse Cloud
 
-Alternatively you can use the cloud hosted version of Langfuse.
+Alternatively you can use the [cloud hosted](https://langfuse.com/docs/deployment/cloud) version of Langfuse.
 
 1. Sign up for an account here [https://cloud.langfuse.com/](https://cloud.langfuse.com/)
 2. Setup a project named `superagent`

fern/mdx/vector-stores/weaviate.mdx

@@ -1,6 +1,6 @@
 Use weaviate as the vector store for RAG applications, with by self hosting with Docker or by using the weaviate cloud version.
 
-**Website** - [weaviate..io](https://weaviate.io/)
+**Website** - [weaviate.io](https://weaviate.io/)
 <br/>
 **Github** - [weaviate/weaviate](https://github.com/weaviate/weaviate)
 
@@ -82,7 +82,7 @@ Use the provided `uninstall.sh` script to remove the running services for weavia
 
 Alternatively you can use the cloud hosted version of weaviate.
 
-1. Sign up for an account here [https://cloud.weaviate.com/](https://cloud.weaviate.com/)
+1. Sign up for an account here [https://console.weaviate.cloud/](https://console.weaviate.cloud/)
 2. Create a new cluster named `superagent`
 3. Once the cluster has deployed, expand the cluster `Details` and under `Authentication` click `API Keys` - copy your key.
 4. Enter the values in the superagent docker `.env` file, paste the key in the variable `WEAVIATE_API_KEY`.

libs/.docker/README.md

@@ -2,161 +2,4 @@
 
 ## Getting Started
 
-In the `.docker` folder there are several docker-compose files available.
-
-After you have carefully set all the values in both the root `superagent/.env`, `ui/.env` and `.docker/docker.env` files, you can spin the backend up in docker:, see [Setup environment variables](#setup-environment-variables)
-
-- `docker-compose.yml` - Main file, starts the Superagent API, a PostgresDB and PGAdmin to administer the DB, fine for most use cases (combine with other dockerfiles using the `-f` flag)
-
-  ``` bash
-  docker-compose --env-file docker.env up --build -d
-  ```
-
-- `docker-compose.api.yml` - Can be used to independently start Superagent API
-
-  ``` bash
-  docker-compose --env-file docker.env -f docker-compose.api.yml up --build -d
-  ```
-
-- `docker-compose.pgdb.yml` - Can be used to independently start the PostgresDB with PGAdmin
-
-  ``` bash
-  docker-compose --env-file docker.env -f docker-compose.pgdb.yml up --build -d
-  ```
-
-- `docker-compose.mh.yml` - Can be used to start a [motorhead](https://github.com/getmetal/motorhead) server. Motorhead is a local memory and information retrieval server for LLMs.
-
-Stop everything with:
-
-``` bash
-docker-compose --env-file docker.env down
-
-# Additionally remove orphaned containers if there are any
-docker-compose --env-file docker.env down --remove-orphans
-
-# Delete eveything including volumes images, this will result in total data loss
-docker-compose --env-file docker.env down --rmi all -v
-
-# Remove all unused networks
-docker network prune
-
-# To remove all stopped containers, unused networks, dangling images, and build cache, you can use:
-docker system prune
-```
-
-## Setup environment variables
-
-1. Create and fill in the API and UI environment variables. Check the `README` for both the API and the UI for details.
-
-   ``` bash
-   # create the api env file
-   cp superagent/.env.example superagent/.env
-
-   # create the ui env file
-   cp ui/.env.example ui/.env
-   ```
-
-2. Copy the `.docker/docker.env.example` into a `docker.env` file.
-
-   ``` bash
-   # the example values are placeholders but they will work just fine locally.
-   cp .docker/docker.env.example .docker/docker.env
-   ```
-
-3. Set the NETWORK_MODE mode suitable for your setup:
-
-   You can create a custom network to use with `docker network create superagent_default` this will allow all the services to communicate with each other even if you start them separately. If you run `a supabase database for the frontend set the network as supabase_network_ui``.
-
-   ``` bash
-   # Create and use a docker network.
-   # Create the network first with `docker network create superagent_default`
-   NETWORK_MODE=superagent_default
-
-   # Or Simulate running the docker resources on localhost
-   NETWORK_MODE=host
-
-   # Or run the docker resources on localhost the supabase network
-   NETWORK_MODE=supabase_network_ui 
-   ```
-
-   If running `superagent_default` mode, you may need to add the docker host names for the services to your `hosts` file on Windows, then use these values in place of `localhost` or `127.0.0.1` in your `.env` files. This is especially needed when running the UI in docker as the browser will not be able to access the docker host, while the UI server code can.
-
-   - Open a text editor as admin (right click icon, 'Run as administrator')
-   - Open the file `C:\Windows\System32\drivers\etc\hosts`
-   - Add the following lines at the bottom
-
-   ``` bash
-   127.0.0.1  superagent-api
-   127.0.0.1  pgdb
-   127.0.0.1  pgadmin
-   127.0.0.1  motorhead
-   ```
-
-   You can then access any of these resources, running in docker, using these host names in your `superagent/.env` files and in the browser. e.g
-
-   Superagent API: `http://superagent-api:8080`
-
-   PGAdmin: `http://pgadmin:5050`
-
-   Motorhead API: `http://motorhead:8081`
-
-4. In order to connect to the db running in docker, with the example values from `docker.env`, set the following in the root `superagent/.env`. Set the `DATABASE_URL` and `DATABASE_MIGRATION_URL` to the same value and comment out, or remove the `DATABASE_SHADOW_URL`
-
-   - `superagent/.env`
-
-    ``` bash
-    # .env
-    # 'superbase_default' network mode or supabase network
-    DATABASE_URL=postgresql://admin:password@pgdb:5432/superagent
-    DATABASE_MIGRATION_URL=postgresql://admin:password@pgdb:5432/superagent
-    # DATABASE_SHADOW_URL= # Mandatory for Neon DB
-
-    # 'host' network mode
-    DATABASE_URL=postgresql://admin:password@localhost:5432/superagent
-    DATABASE_MIGRATION_URL=postgresql://admin:password@localhost:5432/superagent
-    # DATABASE_SHADOW_URL= # Mandatory for Neon DB
-    ```
-
-    ``` bash
-    # superagent/.env
-    # to use the selfhosted motorhead set
-    # Memory (mandatory)
-    MEMORY_API_URL=http://motorhead:8081
-    ```
-
-
-    - `ui/.env`
-
-    ``` bash
-    # 'superbase_default' network mode or supabase
-    NEXT_PUBLIC_SUPERAGENT_API_URL="http://superagent-api:8080/api/v1"
-
-    # 'host' network mode
-    NEXT_PUBLIC_SUPERAGENT_API_URL="http://127.0.0.1:8080/api/v1"
-    ```
-
-    ``` bash
-    # online dashboard or local CLI dashboard
-    NEXT_PUBLIC_SUPABASE_URL=
-    NEXT_PUBLIC_SUPABASE_ANON_KEY=
-    ```
-
-## Viewing logs in the Docker containers
-
-Each container is named so you can easily view the logs using docker on the CMD line
-
-``` bash
-# API logs
-docker logs superagent-api -f
-
-# DB logs
-docker logs pgdb -f
-
-## Adminer logs
-docker logs pgadmin -f
-
-## Motorhead logs
-docker logs motorhead -f
-
-```
-
+Read the getting started guide [https://docs.superagent.sh/overview/deployments/docker-compose](https://docs.superagent.sh/overview/deployments/docker-compose)

libs/.docker/run.sh

@@ -8,13 +8,14 @@ if ! docker network ls | grep -q superagent_network; then
 fi
 
 # Run the core services
-docker-compose -f docker-compose.yml \
+docker compose -f docker-compose.yml \
         -f superagent/db/docker-compose.pgdb.yml \
         -f superagent/db/docker-compose.pgadmin.yml \
         -f superagent/motorhead/docker-compose.motorhead.yml \
         -f ui/docker-compose.ui.yml \
         up \
         --build \
         -d
-
-docker logs superagent-ui
+
+docker logs superagent-ui
+docker logs superagent-api

libs/.docker/ui/docker-compose.ui.yml

@@ -22,8 +22,6 @@ services:
     command: /bin/sh -c "npm run supabase:docker:push --debug && npm start"
     environment:
       - PORT=3000
-      - SUPABASE_HOST=${SUPABASE_HOST}
-      - SUPABASE_PORT=${SUPABASE_PORT}
       - SUPABASE_DB_URL=${SUPABASE_DB_URL}
       - SUPABASE_URL=${NEXT_PUBLIC_SUPABASE_URL}
       - SUPABASE_KEY=${NEXT_PUBLIC_SUPABASE_ANON_KEY}

libs/.docker/ui/supabase/README.md

@@ -1,4 +1,4 @@
-# Supabase docker-compose setup for Superagent
+# Supabase docker compose setup for Superagent
 
 A basic configuration is provided here for using Supabase with Superagent. Based on the official docker-compose configuration from Supabase.
 
@@ -8,7 +8,7 @@ A basic configuration is provided here for using Supabase with Superagent. Based
 
 ## Official documentation
 
-[https://supabase.com/docs/guides/self-hosting/docker](https://supabase.com/docs/guides/self-hosting/docker)
+[https://supabase.com/docs/guides/self-hosting/docker-compose](https://supabase.com/docs/guides/self-hosting/docker-compose)
 
 ## Supabase docker-compose official github repo