Skip to content

Latest commit

 

History

History
405 lines (258 loc) · 12.4 KB

development.md

File metadata and controls

405 lines (258 loc) · 12.4 KB
Raw

Development

Home

Table of Contents

Getting Started

To set up your local environment and begin developing for this project, complete the following steps.

System Requirements

Note: This document is geared toward MacOS but can be easily applied to Linux. Contribution for Windows users is welcome.

For reference, this document was written while testing on a 2018 MacBook Pro running MacOS Big Sur and Docker Desktop 3.2.2.

Make sure the Xcode Command Line Tools are installed.

host:~$ sudo xcodebuild -license
host:~$ xcode-select --install

Install Homebrew.

host:~$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install Docker Desktop.

host:~$ brew install --cask docker

Due to resource utilization in the application, it is recommended to increase memory available to containers.

In Docker Desktop, visit Settings > Resources > Advanced. Set "Memory" to at least 4GB.

Return to Top

Fork and Clone the Project

Visit GitHub and navigate to the rice-crc/voyages repository. In the top-right corner of the page, click the "Fork" button.

For more information, check GitHub's documentation on forking repositories.

Clone the fork to your local machine.

host:~/Projects$ git clone https://github.com/<username>/voyages.git

Add the parent repository as an upstream remote.

host:~/Projects$ cd voyages

host:~/Projects/voyages$ git remote add upstream https://github.com/rice-crc/voyages.git

Note: if you are cloning for development, it is likely that you will be working on different branches at different times. It is very important to create separate DB and Solr Cores for each branch since the schema/data could be different. In the localsettings-local.py.default example file, there is support for automatically using a branch-dependent DB and Solr Core name in the configuration files. The convention for both names is: voyage_{branch name, removing the prefix 'feature/' if present}

Return to Top

Installation

Copy and rename the default localsettings file for local environments.

host:~/Projects/voyages$ cp voyages/localsettings-local.py.default voyages/localsettings.py

Build and run the containers necessary to work on the project.

For details, see docker-compose.yml.

host:~/Projects/voyages$ docker-compose up -d --build

Create the database. Note: append a suffix for the branch (e.g. voyages_develop).

host:~/Projects/voyages$ docker exec -i voyages-mysql mysql -uroot -pvoyages -e "create database voyages"

Create the database user and grant privileges. Note: append a suffix for the branch (e.g. voyages_develop).

host:~/Projects/voyages$ docker exec -i voyages-mysql mysql -uroot -pvoyages -e "create user 'voyages'@'%' identified by 'voyages'"
host:~/Projects/voyages$ docker exec -i voyages-mysql mysql -uroot -pvoyages -e "grant all on voyages.* to 'voyages'@'%'"

Download the latest emory-voyages_prod-*.sql.tgz MySQL dump from the Google Drive share and expand to the data/voyages_prod.sql file.

Import the database dump to MySQL.

host:~/Projects/voyages$ docker exec -i voyages-mysql mysql -uroot -pvoyages voyages < data/voyages_prod.sql

Verify the data import.

host:~/Projects/voyages$ docker exec -i voyages-mysql mysql -uvoyages -pvoyages -e "show databases"
host:~/Projects/voyages$ docker exec -i voyages-mysql mysql -uvoyages -pvoyages -e "show tables from voyages"
host:~/Projects/voyages$ docker exec -i voyages-mysql mysql -uvoyages -pvoyages -e "select * from voyages.voyage_voyage limit 1"

Create the Solr index. Note: append a suffix for the branch (e.g. voyages_develop).

host:~/Projects/voyages$ docker exec -i voyages-solr solr create_core -c voyages -d /srv/voyages/solr

Run the process to build the index. This will take a very long time (around 60-90 minutes).

If some of the reindex submissions fail due to timeouts, you may want to:

  • Check your Docker memory allocation. Again, set a minimum of 4GB (see System Requirements)
  • Check your Solr memory allocation in the docker-compose.yml file and consider allocating 2gb rather than the 1gb default we have set it to
  • Use the pysolr "--batch-size" flag in the below command to make smaller requests, e.g., --batch-size=200
host:~/Projects/voyages$ docker exec -i voyages-django bash -c 'python3 manage.py rebuild_index --noinput'

Run media asset tasks. These may also take a very long time.

host:~/Projects/voyages$ docker exec -i voyages-django bash -c 'python3 manage.py compilemessages'
host:~/Projects/voyages$ docker exec -i voyages-django bash -c 'python3 manage.py compilescss'
host:~/Projects/voyages$ docker exec -i voyages-django bash -c 'python3 manage.py collectstatic --noinput'
host:~/Projects/voyages$ docker exec -i voyages-django bash -c 'python3 manage.py compress --force'
host:~/Projects/voyages$ docker exec -i voyages-django bash -c 'python3 manage.py thumbnail cleanup'

Note the following project resources:

Return to Top

TLDR Workflow

The following is meant to be a concise reference for the development process.

For more details, check the full Development Workflow section below.

# Setup.

git clone https://github.com/<username>/voyages.git
cd voyages

# Any time you start new work, make sure your develop branch is up-to-date
# with the remote on GitHub.

git checkout develop
git fetch upstream && git rebase upstream/develop

# Create a working branch to isolate your changes and begin your work.

git checkout -b short-desc

# Before any commit, always make sure your branch is up-to-date with the remote.

# When your work is complete, pull the latest changes, resolve any merge
# conflicts, and then commit your work.

git fetch upstream && git rebase upstream/develop
git add . && git commit

# When ready, push your branch to your fork. Visit the repository on GitHub
# and make a Pull Request.

git push origin HEAD

# Once the Pull Request is accepted and merged, clean up your work.

git checkout develop && git branch -D short-desc
git fetch upstream && git rebase upstream/develop
git push

Visit the repository on GitHub to make a Pull Request.

Return to Top

Development Workflow

There are two protected branches in the repository: main and develop (which serves as an integration branch).

Update Your Fork

Ensure your fork is current and that you have the latest changes from the upstream develop branch.

host:~/Projects/voyages (develop)$ git fetch upstream

Rebase the upstream repository's develop branch.

host:~/Projects/voyages (develop)$ git rebase upstream/develop

Your local develop branch is now up-to-date with upstream.

To update your fork on GitHub, push your changes.

host:~/Projects/voyages (develop)$ git push

Return to Top

Do Your Work

Continue your development work.

Create a Working Branch

Create a new local branch using the naming convention "short-desc".

host:~/Projects/voyages (develop)$  git checkout -b short-desc

Stage & Commit Your Work

Do the work required.

Stage your changes and commit as needed. Each commit should be logically atomic.

host:~/Projects/voyages (short-desc *)$ git add .
host:~/Projects/voyages (short-desc +)$ git commit

Formatting Commit Messages

Use the following convention for commit messages.

  • The first line should follow the pattern: "A short, grammatically correct sentence ending with punctuation."
  • Enter a line break for the second line.
  • Beginning with the third line, describe in detail what has changed and why. Markdown lists can be used and will be displayed in GitHub.

Pull the Latest Upstream Changes

Ensure no additional changes have been made to the upstream repository and rebase if necessary.

host:~/Projects/voyages (short-desc)$ git fetch upstream
host:~/Projects/voyages (short-desc)$ git rebase upstream/develop

Push Your Working Branch

Push your local branch to your forked repository (origin) so that a Pull Request may be created.

host:~/Projects/voyages (short-desc)$ git push origin HEAD

Return to Top

Make a Pull Request

On GitHub, navigate to your forked repository and create a new pull request against upstream/develop which describes the proposed changes.

Return to Top

Clean Up Your Work

Once the pull request has been reviewed, approved, and merged into the upstream repository, clean up your work.

Delete the Working Branch

host:~/Projects/voyages (short-desc)$ git checkout develop
host:~/Projects/voyages (develop)$ git branch -D short-desc

Update Your Fork

host:~/Projects/voyages (develop)$ git fetch upstream
host:~/Projects/voyages (develop)$ git rebase upstream/develop

host:~/Projects/voyages (develop)$ git push

Return to Top

Deployment Workflow

The OCI build instance pulls updates from the main branch in rice-crc/voyages every five minutes.

When changes are received, an offline Django container is automatically built on each of the remote app instances.

Once complete, Nginx containers on the app instances are updated to point at the new Django containers.

The new containers are live and the now-offline containers are removed.

Return to Top

Frequently Asked Questions

Is data persisted between restarts of the containers?

Yes, using Docker named volumes.

Be careful if you remove the MySQL and Solr volumes as you'll need to rerun the lengthy reindex.

first, why would I want to remove these volumes? Because docker named volumes persist between container restarts, that means that if you switch between branches that, say, have different SQL or Solr schemas, you'll find conflicts arising.

what are my alternatives to wiping out the volumes? If you're going to be switching frequently between different branches that have breaking changes to components that use persistent data, then consider, for instance, changing the names of the database and solr core you use between these in your localsettings file.

How can I blow away my local copy of the project and restart?

Remove the containers.

host:~/Projects/voyages$ docker-compose down
host:~/Projects/voyages$ docker container prune
host:~/Projects/voyages$ docker image prune
host:~/Projects/voyages$ docker network prune
host:~/Projects/voyages$ docker volume prune

Remove the local directory.

host:~/Projects/voyages$ cd ..
host:~/Projects$ rm -rf voyages

Begin from “Getting Started” > “Fork & Clone the Project”

Return to Top

Miscellanious Tasks

# List running containers

docker ps

# View container logs

docker logs --tail 50 --follow --timestamps <id>

# View container resource statistics

docker ps -q | xargs docker stats --no-stream
docker ps -q | xargs docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}" --no-stream

# View container size statistics

docker system df

# Connect to a container

docker exec -it <name> bash

# Rebuild a single container

docker-compose up -d --build <name>

# Run a command on a container

docker exec -i bash -c 'echo hello'

Return to Top