Data driven parser

.github/workflows/buildcheck.yml

@@ -11,7 +11,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v1
         with:
-          python-version: 3.8
+          python-version: 3.10.5
 
       - name: Install dependencies
         run: |

Pipfile

@@ -12,4 +12,4 @@ pytest = ">=2.7.2"
 [dev-packages]
 
 [requires]
-python_version = "3.8"
+python_version = "3.10"

README.md

@@ -161,32 +161,37 @@ generation.
 
 ## The gen() function
 
-The gen() function has a short name to be convenient in templates.
+The `gen()` function has a short name to be convenient in templates.
 
-You can call the gen() function from within a template to delegate the
-required work for generating a certain node type within a larger generation.
-(See Template example).  Thus the templates can effectively "call" each other,
-without the need for template-include or template-inheritance features, but
-those features in jinja2 are still available if they are needed.
+You can call the `gen()` function from within a template to delegate the
+required work for generating a certain node type to a separate template file.
+(See Template example later).  Thus the templates can effectively "call" each
+other, without the need for template-include or template-inheritance features
+that jinja2 provides (but those features are of course still possible to use).
 
-gen() can be called with just the node (evaluating the passed parameter type at run-time)
-or by explicitly stating a template:
+`gen()` can be called with (1) just the node (evaluating the passed parameter type at run-time)
+or (2) by explicitly stating a template:
 
-1. Providing the node reference only.
+1. Providing the node reference only:
 ```
 def gen(node : AST)
+```
 example use:
+```
  gen(node)
-
 ```
+
 This variant will dynamically determine the node type (a subclass of AST)
-and generate using the predetermined template for that node type.
+and generate using the predetermined template for that node type.  (See 
+`default_templates` variable).
 
-2. Providing the node and a specific template.  The default template for the
-   node type is not used. The specified template is used instead.
+2. Providing the node and a specific template.  The specified template is
+   used regardless of the node type:
 ```
 def gen(node : AST, templatename : str)
+```
 example use:
+```
 gen(node, 'My-alternative-method-template.tpl')
 ```
 
@@ -209,7 +214,8 @@ template with useful information here.
 * **suffix** should try to mimic the normal suffix for the file format that is
 being generated.  For example, if the templates aims to generate a HTML
 document, name the template with .html.  If it is generating a programming
-language, use an appropriate suffix for the file, etc.
+language, use an appropriate suffix for the file, etc.  If there is no obvious
+type, we use ".tpl".
 
 Examples:
 
@@ -225,25 +231,30 @@ Service-rust.rs
 ### Variable use in templates
 
 * The standard functions in vsc_generator.py will pass in only a single node
-of AST type to the template generation framework.  The node will be of a
-particular type, depending on what type of template is being rendered
-(or more correctly, what was specified when the generation function was called
- -- either a top level function like render_ast_with_template_file()
- or a call to gen() with the node in question as parameter.)
+of AST type to the template generation framework.  (If you define your own
+custom generator, it could choose to do something different).
+
+The passed node and will have a particular node type (subclass to AST).  The
+type depends on what type of template is being rendered (or more correctly,
+what was specified when the generation function was called -- either a top
+level function like render_ast_with_template_file() or a call to gen() with
+the node in question as parameter.)
 
 * When using the provided generator convenience functions, the node that is
 "passed to the template" (roughly speaking) will always be named **item**
 
-* Since the class members are public, they can be walked directly through
-dot-notation in code that is embedded in the template.  For example, if the
-passed item is a Service the template can get to the namespaces list by
-just referencing it:  `item.namespaces`
+* The class member variables (referring to the children of a node) in
+all AST nodes are public.  This means they can be referenced directly through
+dot-notation inside code that is embedded in the jinja template.  For example,
+if the passed `item` is a Service the template can get to the _list_ of
+namespaces in the service by just referencing it: `item.namespaces`
+
+This of course means that dot-notation could also be chained:
+`item.namespaces[1].methods[0].description` - the description of th efirst
+method name in the second namespace.
 
-Dot notation can be chained as needed:
-`item.namespaces[1].methods[0].description` - the first method name in the
-second namespace.
-...but of course it is more likely to use loop constructs to iterate over
-lists than to address specific indexes like that:
+It is however more likely to use loop constructs to iterate over lists than to
+address specific indexes like that:
 
 ```
 {% for i in item.namespaces %}
@@ -253,15 +264,15 @@ lists than to address specific indexes like that:
 
 # Advanced features
 
-jinja2 is a very capable templating language.  Advanced generators can of
-course make use of any features in python or jinja2 to create an advanced
-generator.  Any features that might be applicable in more than one place would
-however be best generalized and introduced into the vsc_generator.py helper
-modules, for better reuse.
+jinja2 is a very capable templating language.  Generators can make use of any
+features in python or jinja2 to create an advanced generator.  Any features
+that might be applicable in more than one place would however be best
+generalized and introduced into the vsc_generator.py helper module, for
+better reuse between custom generator implementations.
 
 ### Template example
 
-This templates expects that the "item" passed in is the Service object.
+This template expects that the "item" passed in is a Service object.
 It calls the gen() function from within the template to delegate work
 to a separate template for Methods.
 
@@ -282,6 +293,17 @@ to a separate template for Methods.
 {% endfor %}
 ```
 
+The gen() function in the vsc_generator implementation will determine the node
+type of `x` at runtime, yielding `Service`.  It will will then look into the
+`default_templates` variable to see which is the template file to use for a
+Service node, and generate the node using that template.
+
+The global `default_templates` variable is defined by vsc_generator to point
+to some templates used for test/demonstration. A custom generator
+implementation would modify this variable, or simply overwrite the value after
+including the vsc_generator as a module (or later on, this might be passed in
+at run-time in a different way).
+
 # Future plans, new proposals and enhancements
 
 Please refer to [GitHub tickets](https://github.com/GENIVI/vsc-tools/issues)

model/vsc_generator.py

@@ -18,7 +18,8 @@
 #    according to given templates.
 
 # It's useful to have these classes in our namespace directly
-from vsc_parser import AST, Argument, Method, Event, Member, Option, Namespace, Service
+from vsc_parser import AST, Argument, Enum, Error, Event, Include, Member, Method, Namespace, Option, Property, Service, Struct, Typedef
+
 import vsc_parser # For other features from parser module
 import anytree
 import getopt
@@ -37,21 +38,19 @@
         autoescape = jinja2.select_autoescape([])
         )
 
-# This is important. We want the control blocks in the template to NOT
-# result in any extra white space when rendering templates.
-# However, this might be a choice made by each generator, so then we need
-# to export the ability to keep these public for the using code to modify
-# them.
+# We want the control blocks in the template to NOT result in any extra
+# white space when rendering templates. However, this might be a choice
+# made by each generator, so we need to export the ability to keep these
+# settings public for other code to modify them.
 jinja_env.trim_blocks = True
 jinja_env.lstrip_blocks = True
 
 default_templates = {}
 
 # Exception:
 class GeneratorError(BaseException):
-   def __init__(self, m):
-       self.msg = m
-
+    def __init__(self, m):
+        self.msg = m
 
 # ---------- GENERATION FUNCTIONS ------------
 
@@ -62,35 +61,59 @@ def get_template(filename):
 # Frontend to overloaded gen() function:
 
 def gen(node : AST, second = None):
+    # Processing of lists of objects?
+    if type(node) == list or type(node) == tuple:
+        # Generate each node and return a list of results. A list is not
+        # intended to be printed directly as output, but to be processed by
+        # a jinja filter, such as |join(', ')
+
+        if len(node) == 0:
+            return []
+        else:
+            # Recurse over each item in list, and return a list of strings
+            # that is generated for each one.
+            return [gen(x, second) for x in node]
 
-   # Processing of lists of objects
-   if type(node) == list or type(node) == tuple:
-       # Generate each node and return a list of results.
-       # A list is not intended to be printed directly as output, but to be
-       # processed by a jinja filter, such as |join(', ')
-       return [gen(x, second) for x in node]
-
-   # OK, now dispatch gen() depending on the input type
-   if second is None:          # No explicit template -> use default for the node type
-       return _gen_type(node)
-   elif type(second) == str:   # Explicit template -> use it
-       return _gen_tmpl(node, second)
-   else:
-      print(f'node is of type {type(node)}, second arg is of type {type(second)}  ({type(second).__class__}, {type(second).__class__.__name__})')
-      raise GeneratorError(f'Wrong use of gen() function! Usage: pass the node as first argument (you passed a {type(node)}), and optionally template name (str) as second argument. (You passed a {second.__name__})')
+    else:
+        # Processing single item!
+        # second argument is either an explicit template, or None. If it is
+        # None, then the node type will be used to determine the template.
+        if second is None:          # No explicit template -> use default for the node type
+            return _gen_type(node)
+        elif type(second) == str:   # Explicit template -> use it
+            return _gen_tmpl(node, second)
+        else:
+            print(f'node is of type {type(node)}, second arg is of type {type(second)}  ({type(second).__class__}, {type(second).__class__.__name__})')
+            raise GeneratorError(f'Wrong use of gen() function! Usage: pass the node as first argument (you passed a {type(node)}), and optionally template name (str) as second argument. (You passed a {second.__name__})')
 
 # Implementation of typed variants of gen():
 
 # If no template is specified, use the default template for the node type.
 # A default template must be defined for this node type to use the function
 # this way.
 def _gen_type(node : AST):
+    # It is currently unexpected to receive None.  But minor changes might
+    # change this later.  An alternative is to generate an empty string,
+    # but for now let's make sure this case is noticed with an exception,
+    # so it can be investigated.
+    if node is None:
+        raise GeneratorError(f"Unexpected 'None' node received")
+        return ""
+
     nodetype=type(node).__name__
+
+    # If the output-template refers to a member variable that was _not
+    # defined_ in the node, then it shows up as Undefined type. This
+    # happens when optional things do not appear within the service
+    # definition. We just generate empty strings for those items.
+    if nodetype == "Undefined":
+        return ""
+
     tpl = default_templates.get(nodetype)
     if tpl is None:
-       raise GeneratorError(f'gen() function called with node of type {nodetype} but no default template is defined for the type {nodetype}')
+        raise GeneratorError(f'gen() function called with node of type {nodetype} but no default template is defined for the type {nodetype}')
     else:
-       return get_template(tpl).render({ 'item' : node})
+        return get_template(tpl).render({ 'item' : node})
 
 # If template name directly specified, just use it.
 def _gen_tmpl(node : AST, templatefile: str):
@@ -101,16 +124,16 @@ def _gen_tmpl(node : AST, templatefile: str):
 # Instead of providing a template file, provide the template text itself
 # (for unit tests mostly).  See gen() for more comments/explanation.
 def _gen_with_text_template(node: AST, second: str):
-   # Processing of lists of objects, see gen() for explanation
+    # Processing of lists of objects, see gen() for explanation
    if type(node) == list or type(node) == tuple:
-      return [_gen_with_text_template(x, second) for x in node]
+       return [_gen_with_text_template(x, second) for x in node]
    if second is None:
        return _gen_type(node)
    elif type(second) == str:  # "second" is here the template text, not a filename
        return jinja2.Template(second).render({'item' : node})
    else:
-      print(f'node is of type {type(node)}, second arg is of type {type(second)}  ({type(second).__class__}, {type(second).__class__.__name__})')
-      raise GeneratorError(f'Wrong use of gen() function! Usage: pass the node as first argument (you passed a {type(node)}), and optionally template name (str) as second argument. (You passed a {second.__name__})')
+       print(f'node is of type {type(node)}, second arg is of type {type(second)}  ({type(second).__class__}, {type(second).__class__.__name__})')
+       raise GeneratorError(f'Wrong use of gen() function! Usage: pass the node as first argument (you passed a {type(node)}), and optionally template name (str) as second argument. (You passed a {second.__name__})')
 
 
 # Entry point for passing a dictionary of variables instead:
@@ -120,38 +143,53 @@ def gen_dict_with_template_file(variables : dict, templatefile):
 # Export the gen() function and classes into jinja template land
 # so that they can be referred to inside templates.
 
+# The AST class definitions are not required to reference the member
+# variables on objects, so they will be rarely used.  But possibly some
+# template will have logic to create an AST node on-the-fly and then call
+# the generator on it, and this will require knowledge of the AST
+# (sub)class.
+
 jinja_env.globals.update(
- gen=gen,
- AST=AST,
- Argument=Argument,
- Method=Method,
- Event=Event,
- Member=Member,
- Option=Option,
- Namespace=Namespace,
- Service=Service)
-
-# ---------- TEST / SIMPLE USAGE ------------
+        gen=gen,  # Most common function to reference from template
+        AST=AST,  # AST classes and subclasses....
+        Argument=Argument,
+        Enum=Enum,
+        Error=Error,
+        Event=Event,
+        Include=Include,
+        Member=Member,
+        Method=Method,
+        Namespace=Namespace,
+        Option=Option,
+        Property=Property,
+        Service=Service,
+        Struct=Struct,
+        Typedef=Typedef)
+
+# This code file can be used in two ways.  Either calling this file as a
+# program using the main entry points here, and specifying input parameter.
+# Alternatively, for more advanced usage, the file might be included as a
+# module in a custom generator implementation.  That implementation is
+# likely to call some of the funcctions that were defined above.
+
+# For the first case, here follows the main entry points and configuration:
 
 def usage():
     print("usage: generator.py <input-yaml-file (path)> <output-template-file (name only, not path)>")
     sys.exit(1)
 
-def test(argv):
-    if not len(argv) == 3:
-        usage()
-    ast = vsc_parser.get_ast_from_file(argv[1])
-    print(gen(ast, argv[2]))
-
-# TEMP TEST TO BE MOVED
+# This is a default definition for our current generation tests.
+# It may need to be changed, or defined differently in a custom generator
 default_templates = {
         'Service' : 'Service-simple_doc.html',
         'Argument' : 'Argument-simple_doc.html'
         }
 
-# If run as a script, generate a single YAML file and single template
-# (for testing)
+# If run as a script, generate a single YAML file using a single root template
 if __name__ == "__main__":
-    # execute only if run as a script
-    test(sys.argv)
+    if not len(sys.argv) == 3:
+        usage()
+    ast = vsc_parser.get_ast_from_file(sys.argv[1])
+    templatename = sys.argv[2]
+    print(gen(ast, templatename))