Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code cleanup, expand README, fix some minor bugs #20

Merged
merged 31 commits into from
May 9, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
8c4e5d8
various fixes (read changelog)
3vorp Apr 14, 2024
ae8d871
update dependencies
3vorp Apr 14, 2024
2f5896d
more changes, read changelog
3vorp Apr 16, 2024
0193afa
fix lockfile problems
3vorp Apr 16, 2024
9b745d2
grammar
3vorp Apr 16, 2024
bd7af19
fix public/private class conflict
3vorp Apr 16, 2024
0f6565b
remove trailing whitespace
3vorp Apr 17, 2024
7b21d15
fix array-splice conflicts, typos, style
3vorp Apr 17, 2024
17c4776
quick type fix
3vorp Apr 18, 2024
3966df3
fix platform-specific nested key issue
3vorp Apr 19, 2024
7f32430
optional JSONDatabase constructor, shuffle readme
3vorp Apr 20, 2024
3944c63
quick fix to delete $props
3vorp Apr 20, 2024
1fd6b92
quick readme fix
3vorp Apr 20, 2024
cefdf0a
"advanced" readme section
3vorp Apr 21, 2024
945865a
clean up jsdoc
3vorp Apr 22, 2024
298f2b3
move around a few sections
3vorp Apr 22, 2024
b702ed2
refactor php to be cleaner and more readable
3vorp Apr 22, 2024
1fa5907
add more tests to collection.values()
3vorp Apr 22, 2024
e07e526
fix posix end of file stuff
3vorp Apr 22, 2024
38af96b
rename concernedField to field
3vorp Apr 23, 2024
9474723
don't start a method with double underscores
3vorp Apr 24, 2024
31a2fb8
run php formatter
3vorp Apr 28, 2024
e1ab419
add some utility methods, fix remove() error
3vorp Apr 28, 2024
c0d41e4
revert sec() and token changes
3vorp Apr 30, 2024
f21ef49
Merge branch 'TheRolfFR:main' into main
3vorp Apr 30, 2024
d2c06aa
array-contains-none
3vorp Apr 30, 2024
2cde398
better browser check, last-minute readme changes
3vorp May 1, 2024
dd17eda
readRaw(original = false)
3vorp May 3, 2024
a39b98f
fix many editField problems
3vorp May 4, 2024
6dd4aa2
editField operations return confirmations
3vorp May 7, 2024
ab42fe8
fix unknown operation message, jsdoc
3vorp May 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Exposed `Collection.collectionName` as a readonly property for TypeScript usage.
- TypeScript overview to README.md.
- Optional replacement argument for `array-splice` edit fields.
- Optional constructor for the `JSONDatabase` PHP class to reduce repetitive code.

### Changed

Expand Down
112 changes: 62 additions & 50 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,28 +14,29 @@ _Self hosted Firestore-like database with API endpoints based on micro bulk oper

## Installation

Installing the JavaScript wrapper is as simple as running:
```sh
npm install firestorm-db
```
npm install --save firestorm-db
```
To install the PHP backend used to actually query data, copy the sample [php](./php/) folder to your hosting platform and edit the configuration files.
More information about configuring Firestorm server-side is given in the [PHP](#php-part) section.

# JavaScript Part

The JavaScript [index.js](./src/index.js) file is just an [Axios](https://www.npmjs.com/package/axios) wrapper of the library.
The JavaScript [index.js](./src/index.js) file is simply an [Axios](https://www.npmjs.com/package/axios) wrapper of the PHP API.

## How to use it
## JavaScript setup

First, you need to configure your API address, and your token if needed:
First, set your API address (and your writing token if needed) using the `address()` and `token()` functions:

```js
require("dotenv").config(); // add some env variables
const firestorm = require("firestorm-db");

// ex: 'http://example.com/path/to/firestorm/root/'
firestorm.address(process.env.FIRESTORM_URL);
firestorm.address("http://example.com/path/to/firestorm/root/");

// only necessary if you want to write or access private collections
// must match token stored in tokens.php file
firestorm.token(process.env.FIRESTORM_TOKEN);
firestorm.token("my_secret_token_probably_from_an_env_file");
```

Now you can use Firestorm to its full potential:
Expand All @@ -53,23 +54,24 @@ userCollection
.catch((err) => console.error(err));
```

### Collection constructor
## Create your first Collection

A collection takes one required argument and one optional argument:
A Firestorm collection takes one required argument and one optional argument:

- The name of the collection as a `String`.
- The name of the collection as a `string`.
- The method adder, which lets you inject methods to query results. It's implemented similarly to [`Array.prototype.map`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map), taking an outputted element as an argument, modifying the element with methods and data inside a callback, and returning the modified element at the end.

```js
const firestorm = require("firestorm-db");

const userCollection = firestorm.collection("users", (el) => {
// inject a method into the element
el.hello = () => `${el.name} says hello!`;
// return the modified element back with the injected method
return el;
});

// if you have a 'users' table with a printable field named name
// assumes you have a 'users' table with a printable field called 'name'
const johnDoe = await userCollection.get(123456789);
// gives { name: "John Doe", hello: Function }

Expand Down Expand Up @@ -148,18 +150,18 @@ Edit objects have an `id` of the element, a `field` to edit, an `operation` with
| `array-delete` | Yes | `number` | Removes an element at a certain index in an array field. Check the PHP [array_splice](https://www.php.net/manual/function.array-splice) offset for more info. |
| `array-splice` | Yes | `[number, number]` | Removes certain elements. Check the PHP [array_splice](https://www.php.net/manual/function.array-splice) offset and length for more info. |

<br>
Various other methods and constants exist in the JavaScript wrapper, which will make more sense once you learn what's actually happening behind the scenes.

# PHP Part

The PHP files handle files, read, and writes, through `GET` and `POST` requests sent by the JavaScript wrapper. All Firestorm methods correspond to an equivalent Axios request to the relevant PHP file.
The PHP files handle files, read, and writes, through `GET` and `POST` requests sent by the JavaScript wrapper. All JavaScript methods correspond to an equivalent Axios request to the relevant PHP file.

## PHP setup

The basic files to handle requests can be found and copied [here](./php/). The two files that need editing are `tokens.php` and `config.php`.
The server-side files to handle requests can be found and copied [here](./php/). The two files that need editing are `tokens.php` and `config.php`.

- `tokens.php` contains the tokens using a `$db_tokens` array. You will use these tokens to write data or read private tables.
- `config.php` stores all of your collections config. This exports a `$database_list` variable with an array of `JSONDatabase` instances.
- `tokens.php` contains writing tokens declared in a `$db_tokens` array. These correspond to the tokens used with `firestorm.token()` in the JavaScript wrapper.
- `config.php` stores all of your collections. This file needs to declare a `$database_list` array of `JSONDatabase` instances.

```php
<?php
Expand All @@ -168,33 +170,33 @@ require_once('./classes/JSONDatabase.php');

$database_list = array();

// without constructor
$tmp = new JSONDatabase;
$tmp->folderPath = './files/';
$tmp->fileName = 'users';
$tmp->autoKey = false;
$tmp->fileName = 'paths';
$tmp->autoKey = true;
$tmp->autoIncrement = false;

$database_list[$tmp->fileName] = $tmp;

$tmp = new JSONDatabase;
// with constructor ($fileName, $autoKey = true, $autoIncrement = true)
$tmp = new JSONDatabase('users', false);
$tmp->folderPath = './files/';
$tmp->fileName = 'paths';
$tmp->autoKey = true;
$tmp->autoIncrement = false;

$database_list[$tmp->fileName] = $tmp;
?>
```

- The database will be stored in `<folderPath>/<filename>.json` (default folder is `./files/`).
- `autoKey` controls whether to automatically generate the key name or to have explicit key names (default `true`).
- `autoIncrement` controls whether to simply start generating key names from zero or to use a random ID each time (defualt `true`).
- The key in the `$database_list` array is what the collection will be called in JavaScript (this can be different from the JSON filename if needed).
- The database will be stored in `<folderPath>/<filename>.json` (default folder: `./files/`).
- `autoKey` controls whether to automatically generate the key name or to have explicit key names (default: `true`).
- `autoIncrement` controls whether to simply start generating key names from zero or to use a [random ID](https://www.php.net/manual/en/function.uniqid.php) each time (default: `true`).
- The key in the `$database_list` array is what the collection will be called in JavaScript in the Collection constructor (this can be different from the JSON filename if needed).

# Firestorm Files

File API functions are detailed in the `files.php` PHP script. If you do not want to include this functionality, then just delete this file.
Firestorm's file APIs are implemented in `files.php`. If you don't need file-related features, then simply delete this file.

You have to add 2 new configuration variables to your `config.php` file:
To work with files server-side, you need two new configuration variables in `config.php`:

```php
// Extension whitelist
Expand All @@ -205,23 +207,21 @@ $authorized_file_extension = array('.txt', '.png', '.jpg', '.jpeg');
$STORAGE_LOCATION = dirname($_SERVER['SCRIPT_FILENAME']) . '/uploads/';
```

You can use the wrapper functions in order to upload, get and delete a file.
If the folder is accessible from server url, you can directly type its address.

## File rights

The PHP scripts create folders and files, so the script will fail if the PHP user doesn't have write permissions.
You can give rights to a folder with the following command:
Additionally, since the PHP scripts create folders and files, the script will fail if the PHP user doesn't have write permissions.
You can give Firestorm rights to a folder with the following command:

```sh
sudo chown -R www-data "/path/to/uploads/"
```

From there, you can use the functions in `firestorm.files` (detailed below) from the JavaScript wrapper.
If your upload folder is accessible from a server URL, you can directly use its address to retrieve the file.

## Upload a file

In order to upload a file, you have to give the function a `FormData` object. This class is generated from forms and is [native in modern browsers](https://developer.mozilla.org/en-US/docs/Web/API/FormData/FormData) but in Node.js can be imported with the [form-data package](https://www.npmjs.com/package/form-data).
`firestorm.files.upload` uses a `FormData` object to represent an uploaded file. This class is generated from forms and is [native in modern browsers](https://developer.mozilla.org/en-US/docs/Web/API/FormData/FormData), and with Node.js can be installed with the [form-data](https://www.npmjs.com/package/form-data) package.

The uploaded file content can be a [String](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String), a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob), a [Buffer](https://nodejs.org/api/buffer.html) or an [ArrayBuffer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer).
The uploaded file content can be a [String](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String), a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob), a [Buffer](https://nodejs.org/api/buffer.html), or an [ArrayBuffer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer).

There is additionally an overwrite option in order to avoid mistakes.

Expand All @@ -246,7 +246,7 @@ uploadPromise

## Get a file

You can get a file via its direct file URL location or its content with a request.
`firestorm.files.get` takes a file's direct URL location or its content as its parameter.

```js
const firestorm = require("firestorm-db");
Expand All @@ -261,7 +261,7 @@ getPromise

## Delete a file

Because I am a nice guy, I thought about deletion too. So I figured I would put a method to delete the files too.
`firestorm.files.delete` has the same interface as `firestorm.files.get`, but as the name suggests, it deletes the file rather than gets it.

```js
const firestorm = require("firestorm-db");
Expand All @@ -275,7 +275,7 @@ deletePromise
.catch((err) => console.error(err));
```

# TypeScript
# TypeScript Support

Firestorm ships with TypeScript support out of the box.

Expand All @@ -300,7 +300,7 @@ const johnDoe = await userCollection.get(123456789);
// type: { name: string, password: string, pets: string[] }
```

Methods should also be stored in this interface:
Methods should also be stored in this interface, and will be filtered out from write operations:

```ts
import firestorm from "firestorm-db";
Expand All @@ -312,9 +312,21 @@ interface User {
hello(): string;
}

const johnDoe = await userCollection.get(123456789);
const hello = johnDoe.hello();
// type: string
const johnDoe = await userCollection.get(123456789, (el) => {
// interface types should agree with injected methods
el.hello = () => `${el.name} says hello!`;
return el;
});
const hello = johnDoe.hello(); // type: string

await userCollection.add({
id: "123456788",
name: "Mary Doe",
// error: Object literal may only specify known properties, and 'hello' does not exist in type 'Addable<User>'.
hello() {
return "Mary Doe says hello!"
}
})
```

## Additional types
Expand All @@ -329,7 +341,7 @@ const deleteConfirmation = await firestorm.files.delete("/quote.txt");
// type: firestorm.WriteConfirmation
```

# Memory warning
# Memory Warning

Handling big collections can cause memory allocation issues like:

Expand All @@ -344,11 +356,11 @@ If you encounter a memory allocation issue, you have to allow more memory throug
memory_limit = 256M
```

# API endpoints
# API Endpoints

If you want to manually send requests without using the JavaScript wrapper, simply make an HTTP request to the relevant PHP file with your content. Read requests are `GET` requests sent to `get.php`, write requests are `POST` requests sent to `post.php` with provided JSON data, and file requests are requests sent to `files.php` with provided form data.
If you want to manually send requests without using the JavaScript wrapper, simply make an HTTP request to the relevant PHP file with your content. Read requests are `GET` requests sent to `<your_address_here>/get.php`, write requests are `POST` requests sent to `<your_address_here>/post.php` with provided JSON data, and file requests are requests sent to `<your_address_here>/files.php` with provided form data.

The first keys in the request will always be the same, and further keys will depend on the specific method:
The first keys in a Firestorm request will always be the same regardless of its type, and further keys will depend on the specific method:

```json
{
Expand Down
7 changes: 7 additions & 0 deletions php/classes/JSONDatabase.php
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,13 @@ class JSONDatabase {
public $autoKey = true;
public $autoIncrement = true;

public function __construct(string $fileName = 'db', bool $autoKey = true, bool $autoIncrement = true) {
// if no/some args provided they just fall back to their defaults
$this->fileName = $fileName;
$this->autoKey = $autoKey;
$this->autoIncrement = $autoIncrement;
}

public function fullPath() {
return $this->folderPath . $this->fileName . $this->fileExt;
}
Expand Down
5 changes: 5 additions & 0 deletions php/config.php
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@
// The database_list key is what the collection will be called in JavaScript
$database_list["my_collection_name"] = $db;

// This can be simplified into the following constructor:
// - Note: all of these arguments are optional and will fall back to their defaults if not provided
// - Order: (fileName, autoKey, autoIncrement)
$database_list["my_collection_name"] = new JSONDatabase("my_json_name", true, true);

/**
* File handling:
* If you don't need this functionality, delete this section and files.php.
Expand Down
24 changes: 12 additions & 12 deletions tests/config.php
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,18 @@
$STORAGE_LOCATION = dirname($_SERVER['SCRIPT_FILENAME']) . '/uploads/';

$database_list = array();
foreach($props as $prop) {
$dbName = $prop[0];
$autoKey = $prop[1];
$autoKeyIncrement = count($prop) > 2 and $prop[2] == true;

$tmp_db = new JSONDatabase;
$tmp_db->fileName = $dbName;
$tmp_db->autoKey = $autoKey;
$tmp_db->autoIncrement = $autoKeyIncrement;

$database_list[$dbName] = $tmp_db;
}

// test with constructor/optional args
$database_list['house'] = new JSONDatabase('house', false);

// test without constructor
$tmp = new JSONDatabase;
$tmp->fileName = 'base';
$tmp->autoKey = true;
$tmp->autoIncrement = true;

$database_list[$tmp->fileName] = $tmp;
3vorp marked this conversation as resolved.
Show resolved Hide resolved


$log_path = "firestorm.log";

Expand Down
Loading