(Python) Validate code components extraction

[betogaona7]

For python documents, we can validate (or replace) GPT's code components extraction by using the ast library: Ex.

def extract_classes_and_functions(source_code):

    parsed_tree = ast.parse(source_code)

    classes = []
    functions = []

    for node in ast.walk(parsed_tree):
        if isinstance(node, ast.ClassDef):
            classes.append(node)
        elif isinstance(node, ast.FunctionDef):
            functions.append(node)

    return classes, functions

[betogaona7]

This is language-dependent, so maybe could work as part of a set of functions for validation depending in the user's input, and not as an optimization in the pipeline. Right now, the extraction is language-agnostic limited only by Langchain splitter's supported programming languages: https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/code_splitter