Ocean Assist is developing a new micro-economic model that directs globalized resources into effective localized action in order to achieve measurable conservation results. The goal of the Ocean Assist platform is to develop a carbon offset ecosystem that uses machine learning to validate conservation actions and rewards XRP micro-payments using authentication with the Xumm App. Ocean Assist will implement a novel real-world use case of video action detection and classification to identify and reward sustainability actions using a decentralized geo-location XRPL network authorization. This value can be translated into authenticated carbon offset credits with Non-Fungible Token receipts registered on the XRPL, which will be sold to businesses in order to produce sustainable revenue for an action rewards XRP liquidity pool.
A Comprehensive Study of Deep Video Action Recognition is an excellent analysis of previous neural model implementations and research regarding action detection, upon which we have based our own implementation strategy. In particular, our aim is to utilize self-supervised learning to leverage a large amount of unlabeled input data by designing a pretext task to obtain free supervisory signals from data itself.
- Identify waterfront scene or site landmarks
- Identify clean up action
- Object detection within clean up action sequence
- Compare beginning and end or before and after of a scene to validate removal of object.
- Collect sample video of before and after site cleaning as well as actions of cleaning that area (1000+ scenes of sample media recommended).
- Create a standard to normalize user data collection including orientation of camera, resolution, distance from scene, area size, etc.
Dataset Preparation
Notes on Video Data format for MMAction from OpenMMLab
MMAction supports two types of data format: raw frames and video. The former is widely used in previous projects such as TSN. This is fast (especially when SSD is available) but fails to scale to the fast-growing datasets. (For example, the newest edition of Kinetics has 650K videos and the total frames will take up several TBs.) The latter saves much space but is slower due to video decoding at execution time. To alleviate this issue, we use decord for efficient video loading.
For action recognition, both formats are supported. For temporal action detection and spatial-temporal action detection, we still recommend the format of raw frames.
Supported datasets
The supported datasets are listed below. We provide shell scripts for data preparation under the path $MMACTION/data_tools/. To ease usage, we provide tutorials of data deployment for each dataset.
- HMDB51: See PREPARING_HMDB51.md
- UCF101: See PREPARING_UCF101.md
- Kinetics400: See PREPARING_KINETICS400.md
- THUMOS14: See PREPARING_TH14.md
- AVA: See PREPARING_AVA.md
You can switch to GETTING_STARTED.md to train and test the model.
Prepare annotations
Prepare videos
Please refer to the official website and/or the official script to prepare the videos. Note that the videos should be arranged in either (1) a two-level directory organized by ${CLASS_NAME}/${VIDEO_ID} or (2) a single-level directory. It is recommended using (1) for action recognition datasets (such as UCF101 and Kinetics) and using (2) for action detection datasets or those with multiple annotations per video (such as THUMOS14 and AVA).
To extract frames (optical flow, to be specific), dense_flow is needed.
Ocean Assist will experiment with a couple of different pre-trained action detection models. We will be evaluating their accuracy of desired action prediction alongside latency and throughput speeds. Training models is expensive, so we will evaluate the cost effectiveness for project feasibility before running sample data on these models.
Pre-trained models include:
The Ocean Assist platform model will run on either a GPU cluster cloud infrastructure or we will purchase our own NVIDIA GPU hardware outright, whichever is more economically viable for prototyping and production. Options for infrastructure deployment include:
- NVIDIA H100CNX - For real time action verification (production use)
- Lambda Labs Cloud GPU Services & hardware
- Google Computing Services
- Microsoft Azure
In order to prevent spamming or exploitation of the rewards platform, users will need to authenticate their account using Xumm wallet sign in and pin the geo-location of an action event. The user will not be able to submit another action event in that same location (or within a certain distance range) until a minimum amount of time has passed. Here is example code for a geo-location based XRPL NFT minter by BlockLagoon.
A simple graphic user interface to track environmental footprint, clean-up actions, estimated carbon offset, rewards points, and XRP payouts.
A curated user experience made using React and Three.js, designed to incentivize healthy gamified competition amongst different areas, groups, individuals, or countries! Challenge a friend to a clean-up competition from different parts of the world.
Feeling generous? Reward others for their efforts directly with a simple XRP donation function. Example embeddable button code by Evan Schwartz here: https://github.com/emschwartz/ripple-donate-widget
The Rewards Calculator will be an automated system (potentially using XRPL web hooks after the amendment is passed) that rewards XRP for accumulated action points. Action points will be awarded on a scale based on the action identified by the model. For instance, picking up trash on the beach or other waterway will earn more points than placing a cup in a recycling receptacle, but both would receive less points than planting a tree. The inclusion of different action parameters and relative weights in the point system will be determined by their estimated CO2 emission reduction. The action points can then be converted into XRP and transferred in micro-payments when they have accumulated to a minimum XRP value (to be determined).
The action rewards XRP liquidity pool will implement the expanded multi-signature capabilities recently introduced to the XRPL. This will increase the security of rewards assets and create a decentralized authorization structure for future changes to the rewards system after it has been initiated.
A calculator to estimate consumption and environmental footprint for individuals and businesses. Sample carbon calculator repositories include:
- https://github.com/sarah926/carbonFootprint
- https://github.com/lizroper0/fe-earthbff
- https://github.com/absambam/Carbon-Footprint-Calculator (travel footprint calculator)
In order to provide sustainable rewards to the active conservationists in the ecosystem, Ocean Assist plans to offer verifiable carbon offset packages utilizing the issuance of Non-Fungible Tokens on the XRPL.
More than actual emissions units can be traded and sold under the Kyoto Protocols emissions trading scheme.
The other units which may be transferred under the scheme, each equal to one tonne of CO2, may be in the form of:
- A removal unit (RMU) on the basis of land use, land-use change and forestry (LULUCF) activities such as reforestation
- An emission reduction unit (ERU) generated by a joint implementation project
- A certified emission reduction (CER) generated from a clean development mechanism project activity
The Climate Action Reserve is an offsets program working to ensure integrity, transparency, and financial value in the North American carbon market. It does this by establishing regulatory quality standards for the development, quantification, and verification of GHG emission reduction projects in North America; issuing carbon offset credits known as Climate Reserve Tonnes (CRTs) generated from such projects; and tracking the transaction of credits over time in a transparent, publicly-accessible system. Adherence to the Reserve’s high standards ensures that emission reductions associated with projects are real, permanent, and additional, thereby instilling confidence in the environmental benefit, credibility, and efficiency of the U.S. carbon market. Ocean Assist will register with the Climate Action Reserve and form the rewards system around the platforms ability to prove compliance with different reduction standards evaluated by the organisation.