Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add Avro Schema input processor #1753

Merged
merged 20 commits into from
Apr 26, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,10 @@ The following table provides a short summary of available features for supported
<td><a href="./docs/usage.md#generate-models-from-asyncapi-documents">AsyncAPI</a></td>
<td>We support the following AsyncAPI versions: <em>2.0.0 -> 2.6.0</em>, which generates models for all the defined message payloads. It supports the following schemaFormats AsyncAPI Schema object, JSON Schema draft 7, <a href="./examples/asyncapi-avro-schema">AVRO 1.9</a>, <a href="./examples/asyncapi-raml-schema">RAML 1.0 data type</a>, and <a href="./examples/asyncapi-openapi-schema">OpenAPI 3.0 Schema</a>.</td>
</tr>
<tr>
<td><a href="./docs/usage.md#generate-models-from-avro-schema-documents">Avro Schema<a></td>
<td>we support the Avro Schema version <em>1.11.1</em> to generate models</td>
akkshitgupta marked this conversation as resolved.
Show resolved Hide resolved
</tr>
<tr>
<td><a href="./docs/usage.md#generate-models-from-json-schema-documents">JSON Schema</a></td>
<td>We support the following JSON Schema versions: <em>Draft-4, Draft-6 and Draft-7</em></td>
Expand Down
11 changes: 11 additions & 0 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ For more specific integration options, please check out the [integration documen
- [Generate models from AsyncAPI documents](#generate-models-from-asyncapi-documents)
* [Limitations and Compatibility](#limitations-and-compatibility)
+ [Polymorphism](#polymorphism)
- [Generate models from Avro Schema documents](#generate-models-from-avro-schema-documents)
- [Generate models from JSON Schema documents](#generate-models-from-json-schema-documents)
- [Generate models from Swagger 2.0 documents](#generate-models-from-swagger-20-documents)
* [Limitations and Compatibility](#limitations-and-compatibility-1)
Expand Down Expand Up @@ -107,6 +108,16 @@ There are three ways to generate models for a JSON Schema document.

The library expects the `$schema` property for the document to be set in order to understand the input format. By default, if no other inputs are detected, it defaults to `JSON Schema draft 7`. The process of interpreting a JSON Schema to a model can be read [here](./inputs/JSON_Schema.md).

## Generate models from Avro Schema documents

See the below example to get started with Avro Schema for generating models.

- [Generate from an Avro Schema JS Object](../examples/avro-schema-from-object)

The Avro input processor expects the `type` property, as per [Avro Schema specs](https://avro.apache.org/docs/1.11.1/specification/#schema-declaration), in the input object in order to proceed successfully.

> Note: Currently, we support `record` datatype for generating the `Object Model`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorrect, we support enum as well, is there any other types that you can define with Avro that we dont support after merging this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonaslagoni As of now, we do not support map, fixed, byte datatypes. I don't get your point about enum, it is being handled as an Enum Model separately.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My question is, why are you making this Note, and which cases of Avro are we NOT supporting since you highlight record here? 🤨

Copy link
Collaborator Author

@akkshitgupta akkshitgupta Apr 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Avro Schema, a JSON Object is defined using one of the two datatypes: map and record

We do not handle the map datatype as I am not able to think of any use case. What would we been seeking in input object with single datatype. This is the reason I mentioned this explicitly in note.

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maps can easily be converted to our DictionaryMetaModel, with key being StringMetaModel and the value what ever is defined 🙂

Copy link
Collaborator Author

@akkshitgupta akkshitgupta Apr 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't know this could be converted to DictionaryMetaModel.
Below is the example I am considering. Please suggest any changes if required 😄

{
   "name": "additionalInfo",
   "type": "map",
   "value": "string"
}

Also, Just wanna check if there is any use case of fixed and byte datatype defined in Avro Schema or not.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the Avro specification so bad at explaining what is possible... Can you have the value be an array or record? 🤨 Or is only primitive types allowed?

fixed and byte I would convert to string models. We don't have official byte representation in the core models.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to check for this non primitive value type. However in my opinion I don't think non primitive is supposed to be there else it would have been explicitly mentioned there as done for the fields attribute in record type

https://avro.apache.org/docs/1.11.1/specification/#schema-record


## Generate models from Swagger 2.0 documents

There are one way to generate models from a Swagger 2.0 document.
Expand Down
1 change: 1 addition & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ These examples show a specific input and how they can be used:
- [asyncapi-raml-schema](./asyncapi-raml-schema) - A basic example of how to use Modelina with an AsyncAPI document using RAML 1.0 data types as payload format.
- [asyncapi-from-parser](./asyncapi-from-parser) - A basic example where an AsyncAPI JS object from the [parser-js](https://github.com/asyncapi/parser-js) is used to generate models.
- [asyncapi-from-v1-parser](./asyncapi-from-v1-parser) - A basic example where an AsyncAPI JS object from the old v1 [parser-js](https://github.com/asyncapi/parser-js) is used to generate models.
- [avro-schema-from-object](./avro-schema-from-object) - A basic example where an Avro Schema JS Object is used to generate models.
- [json-schema-draft7-from-object](./json-schema-draft7-from-object) - A basic example where a JSON Schema draft 7 JS object is used to generate models.
- [json-schema-draft6-from-object](./json-schema-draft6-from-object) - A basic example where a JSON Schema draft 6 JS object is used to generate models.
- [json-schema-draft4-from-object](./json-schema-draft4-from-object) - A basic example where a JSON Schema draft 4 JS object is used to generate models.
Expand Down
17 changes: 17 additions & 0 deletions examples/avro-schema-from-object/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Avro Schema Input

A basic example of how to use Modelina with an Avro Schema JS object to generate models.

## How to run this example

Run this example using:

```sh
npm i && npm run start
```

If you are on Windows, use the `start:windows` script instead:

```sh
npm i && npm run start:windows
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`Should be able to render model using Avro Schema and should log expected output to console 1`] = `
Array [
"class Person {
private _reservedName: string;
private _serialNo: string;
private _email: string | null;
private _age: number | null;
private _favoriteProgrammingLanguage: ProgrammingLanguage;
private _certifications: string[];
private _address: Address;
private _weight: number;
private _height: number;
private _someid: string;

constructor(input: {
reservedName: string,
serialNo: string,
email: string | null,
age: number | null,
favoriteProgrammingLanguage: ProgrammingLanguage,
certifications: string[],
address: Address,
weight: number,
height: number,
someid: string,
}) {
this._reservedName = input.reservedName;
this._serialNo = input.serialNo;
this._email = input.email;
this._age = input.age;
this._favoriteProgrammingLanguage = input.favoriteProgrammingLanguage;
this._certifications = input.certifications;
this._address = input.address;
this._weight = input.weight;
this._height = input.height;
this._someid = input.someid;
}

get reservedName(): string { return this._reservedName; }
set reservedName(reservedName: string) { this._reservedName = reservedName; }

get serialNo(): string { return this._serialNo; }
set serialNo(serialNo: string) { this._serialNo = serialNo; }

get email(): string | null { return this._email; }
set email(email: string | null) { this._email = email; }

get age(): number | null { return this._age; }
set age(age: number | null) { this._age = age; }

get favoriteProgrammingLanguage(): ProgrammingLanguage { return this._favoriteProgrammingLanguage; }
set favoriteProgrammingLanguage(favoriteProgrammingLanguage: ProgrammingLanguage) { this._favoriteProgrammingLanguage = favoriteProgrammingLanguage; }

get certifications(): string[] { return this._certifications; }
set certifications(certifications: string[]) { this._certifications = certifications; }

get address(): Address { return this._address; }
set address(address: Address) { this._address = address; }

get weight(): number { return this._weight; }
set weight(weight: number) { this._weight = weight; }

get height(): number { return this._height; }
set height(height: number) { this._height = height; }

get someid(): string { return this._someid; }
set someid(someid: string) { this._someid = someid; }
}",
]
`;
15 changes: 15 additions & 0 deletions examples/avro-schema-from-object/index.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
const spy = jest.spyOn(global.console, 'log').mockImplementation(() => {
return;
});
import { generate } from './index';

describe('Should be able to render model using Avro Schema', () => {
afterAll(() => {
jest.restoreAllMocks();
});
test('and should log expected output to console', async () => {
await generate();
expect(spy.mock.calls.length).toEqual(3);
akkshitgupta marked this conversation as resolved.
Show resolved Hide resolved
expect(spy.mock.calls[0]).toMatchSnapshot();
});
});
81 changes: 81 additions & 0 deletions examples/avro-schema-from-object/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
import { TypeScriptGenerator } from '../../src';

const generator = new TypeScriptGenerator();
const AvroSchemaDoc = {
name: 'Person',
namespace: 'com.company',
type: 'record',
fields: [
{ name: 'name', type: 'string', example: 'Donkey', minLength: 0 },
{ name: 'serialNo', type: 'string', minLength: 0, maxLength: 50 },
{
name: 'email',
type: ['null', 'string'],
example: '[email protected]',
pattern: '^[\\w-\\.]+@([\\w-]+\\.)+[\\w-]{2,4}$'
},
{
name: 'age',
type: ['null', 'int'],
default: null,
example: 123,
exclusiveMinimum: 0,
exclusiveMaximum: 200
},
{
name: 'favoriteProgrammingLanguage',
type: {
name: 'ProgrammingLanguage',
type: 'enum',
symbols: ['JS', 'Java', 'Go', 'Rust', 'C'],
default: 'JS'
}
},
{
name: 'certifications',
type: {
type: 'array',
items: 'string',
minItems: 1,
maxItems: 500,
uniqueItems: true
}
},
{
name: 'address',
type: {
name: 'Address',
type: 'record',
fields: [
{ name: 'zipcode', type: 'int', example: 53003 },
{ name: 'country', type: ['null', 'string'] }
]
}
},
{
name: 'weight',
type: 'float',
example: 65.1,
minimum: 0,
maximum: 500
},
{
name: 'height',
type: 'double',
example: 1.85,
minimum: 0,
maximum: 3.0
},
{ name: 'someid', type: 'string', logicalType: 'uuid' }
]
};
akkshitgupta marked this conversation as resolved.
Show resolved Hide resolved

export async function generate(): Promise<void> {
const models = await generator.generate(AvroSchemaDoc);
for (const model of models) {
console.log(model.result);
}
}
if (require.main === module) {
generate();
}
10 changes: 10 additions & 0 deletions examples/avro-schema-from-object/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions examples/avro-schema-from-object/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"config": {
"example_name": "avro-schema-from-object"
},
"scripts": {
"install": "cd ../.. && npm i",
"start": "../../node_modules/.bin/ts-node --cwd ../../ ./examples/$npm_package_config_example_name/index.ts",
"start:windows": "..\\..\\node_modules\\.bin\\ts-node --cwd ..\\..\\ .\\examples\\%npm_package_config_example_name%\\index.ts",
"test": "../../node_modules/.bin/jest --config=../../jest.config.js ./examples/$npm_package_config_example_name/index.spec.ts",
"test:windows": "..\\..\\node_modules\\.bin\\jest --config=..\\..\\jest.config.js examples/%npm_package_config_example_name%/index.spec.ts"
}
}
Loading
Loading