Skip to content

Commit

Permalink
Transform for Syntactic Construct Extractor
Browse files Browse the repository at this point in the history
This tranform extracts the base syntactic concepts from the multi-language source codes and represent these concepts in an unified langauge-agnostic representation that can be further used for multi-lnaguage data profiling. While programming languages expose similar syntactic building blocks to represent programming intent, such as importing packages/libraries, functions, classes, loops, conditionals, comments and others, these concepts are expressed through language-specific grammar, defined by distinct keywords and syntactic form.

Signed-off-by: Pankaj Thorat <[email protected]>
  • Loading branch information
pankajskku committed Sep 30, 2024
1 parent 7a8a949 commit 085a28c
Show file tree
Hide file tree
Showing 569 changed files with 11,721 additions and 0 deletions.
50 changes: 50 additions & 0 deletions transforms/code/syntactic_concept_extractor/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
REPOROOT=../../..
# Use make help, to see the available rules
include $(REPOROOT)/.make.defaults

setup::
@# Help: Recursively make $@ all subdirs
$(MAKE) RULE=$@ .recurse

clean::
@# Help: Recursively make $@ all subdirs
$(MAKE) RULE=$@ .recurse

build::
@# Help: Recursively make $@ in subdirs
$(MAKE) RULE=$@ .recurse
venv::
@# Help: Recursively make $@ in subdirs
$(MAKE) RULE=$@ .recurse

publish::
@# Help: Recursively make $@ in all subdirs
@$(MAKE) RULE=$@ .recurse

test::
@# Help: Recursively make $@ in all subdirs
@$(MAKE) RULE=$@ .recurse

test-src::
@# Help: Recursively make $@ in all subdirs
$(MAKE) RULE=$@ .recurse

set-versions:
@# Help: Recursively $@ in all subdirs
$(MAKE) RULE=$@ .recurse

.PHONY: workflow-venv
workflow-venv:
$(MAKE) -C kfp_ray workflow-venv

.PHONY: workflow-test
workflow-test:
$(MAKE) -C kfp_ray workflow-test

.PHONY: workflow-upload
workflow-upload:
$(MAKE) -C kfp_ray workflow-upload

.PHONY: workflow-build
workflow-build:
$(MAKE) -C kfp_ray workflow-build
56 changes: 56 additions & 0 deletions transforms/code/syntactic_concept_extractor/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Syntactic Construct Extractor

This module extracts the base syntactic concepts from the multi-language source codes and represent these concepts in an unified langauge-agnostic representation that can be further used for multi-lnaguage data profiling. While programming languages expose similar syntactic building blocks to represent programming intent, such as importing packages/libraries, functions, classes, loops, conditionals, comments and others, these concepts are expressed through language-specific grammar, defined by distinct keywords and syntactic form. Our framework abstracts language-specific concepts by transforming them into a unified, language-agnostic representation called universal base syntactic representation (UBSR), referred to as a concept, which is consistently encoded within the proposed schema structure. The current version support the base syntactic concept for importing/including package/libraries, comments, functions.

Table 1 outlines the fields of the UBSR, which maps AST nodes to a structured schema. This schema captures syntactic nodes (based on AST node types) and the relationships between those nodes (derived from AST edges). The UBSR framework currently supports 21 languages, grouped according to their syntactic paradigms.

**Table 1: UBSR Schema Representation**


| **Key** | **Possible Values** | **Description** |
|-----------------------|----------------------------------------------------|----------------------------------------------------------------|
| **"nodes":** | | |
| `"id"` | Integer (e.g., `0`, `1`) | Unique identifier of the node. |
| `"code_snippet"` | String (e.g., `"ubsr_package math"`) | A snippet of code or a description of the node. |
| `"node_type"` | String (e.g., `"ubsr_root"`, `"ubsr_package"`, etc.)| Type of node representing various syntactic concepts. |
| `"parents"` | Array of Integers (e.g., `[1, 2]`) | List of parent node IDs. |
| `"children"` | Array of Integers (e.g., `[1, 2]`) | List of child node IDs. |
| **"metadata" (within nodes):** | | |
| `"info"` | String | General information about the node. |
| `"language"` | String (`"cpp"`, `"python"`, etc.) | Programming language of the node. |
| `"original_code"` | String (e.g., `"int main() {...}"`) | Original code snippet corresponding to the node. |
| `"loc_original_code"` | Integer | Line of code of the concept. |
| **"edges":** | | |
| `"directed_relation"` | String (`"parent_node"`) | Type of relationship between nodes e.g. parent-child. |
| `"metadata"` | Object | Additional metadata for the edge, which can be empty. |


As shown in Table 2, the framework standardizes code representation by categorizing languages within these paradigms for 21 languages. In cases where certain concepts are absent in a language, they are marked as NA in the table. The base syntactic concepts extracted from the UBSR derived from code can be used to derive syntactic and semantic insights of the code data.

**Table 2: Base Syntactic Concepts Supported by the UBSR across Different Syntactical Paradigms**

| **Syntactical Paradigms** | **Languages Supported (Known\*)** | **Package** | **Function** | **Comment** |
|----------------------------------------------------|---------------------------------------------------------------------------------------------------|-------------|--------------|-------------|
| **C-like Syntax** | **C\***, **Java\***, **C#**, **CPP**, **Objective C**, **Rust**, **Golang**, Kotlin | Yes | Yes | Yes |
| **Scripting and Dynamic Syntax** | **Python\***, **JavaScript\***, **Dart**, **Typescript** | Yes | Yes | Yes |
| | QML | Yes | NA | Yes |
| | **Perl** | Yes | Yes | NA |
| **Functional and Expression-Oriented Syntax** | **Haskell\***, Elm\*, Agda, **D**, **Nim**, **Scala** | Yes | Yes | Yes |
| | **Ocaml** | Yes | NA | Yes |


* [python](python/README.md) - provides the base python-based syntactic concept extractor
implementation.
* [ray](ray/README.md) - provides the base ray-based syntactic concept extractor
implementation.



**Offline Path for Syntactic Rule Generation**

The offline path is critical for expanding and refining the syntactic rule database, enabling the USR framework to adapt to new languages and syntactic constructs. This process leverages LLMs to generate syntactic rules for languages that are not yet included in the rule database. To achieve this, we utilize a Few-shot Chain of Thought prompting technique, guiding the LLM through a step-by-step rule generation process. By providing carefully curated training exemplars and detailed instructions, this method ensures the LLM can accurately generalize from these examples to produce effective syntactic rules for a wide range of languages. This structured approach enhances the flexibility of the UBSR framework, allowing it to seamlessly handle evolving language constructs.

The implementation for UI-based offline customization tool is present [here](python/src/offline-customizations). To run the tool, use the following command.

`streamlit run LLM_runner_app.py`

Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"input": "multi-package.parquet",
"contents": "Contents",
"language": "Language"
}
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
venv/
37 changes: 37 additions & 0 deletions transforms/code/syntactic_concept_extractor/python/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
test-data/output
output/*
/output/
data-processing-lib/

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class


# Distribution / packaging
bin/
build/
develop-eggs/
dist/
eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
.tox/
htmlcov
.coverage
.cache
nosetests.xml
coverage.xml
44 changes: 44 additions & 0 deletions transforms/code/syntactic_concept_extractor/python/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
FROM docker.io/python:3.10.14-slim-bullseye

RUN pip install --upgrade --no-cache-dir pip

# install pytest
RUN pip install --no-cache-dir pytest

# Create a user and use it to run the transform
RUN useradd -ms /bin/bash dpk
USER dpk
WORKDIR /home/dpk

# Copy and install data processing libraries
# These are expected to be placed in the docker context before this is run (see the make image).
COPY --chown=dpk:root data-processing-lib-python/ data-processing-lib-python/
RUN cd data-processing-lib-python && pip install --no-cache-dir -e .

COPY --chown=dpk:root src/ src/
COPY --chown=dpk:root pyproject.toml pyproject.toml
COPY --chown=dpk:root README.md README.md

RUN pip install --no-cache-dir -e .

# copy the main() entry point to the image
COPY ./src/syntactic_concept_extractor_transform_python.py .

# copy some of the samples in
COPY ./src/syntactic_concept_extractor_local.py local/

# Copy the tree-sitter bindings (this is the important part)
COPY --chown=ray:users ../../input/tree-sitter-bindings/ /home/dpk/input/tree-sitter-bindings/

# copy test
# COPY test/ test/
# COPY test-data/ test-data/

# Set environment
ENV PYTHONPATH /home/dpk

# Put these at the end since they seem to upset the docker cache.
ARG BUILD_DATE
ARG GIT_COMMIT
LABEL build-date=$BUILD_DATE
LABEL git-commit=$GIT_COMMIT
52 changes: 52 additions & 0 deletions transforms/code/syntactic_concept_extractor/python/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Define the root of the local git clone for the common rules to be able
# know where they are running from.
REPOROOT=../../../..
# Include a library of common .transform.* targets which most
# transforms should be able to reuse. However, feel free
# to override/redefine the rules below.

# $(REPOROOT)/.make.versions file contains the versions

TRANSFORM_NAME=syntactic_concept_extractor

include $(REPOROOT)/transforms/.make.transforms

venv:: .transforms.python-venv

test:: .transforms.python-test

clean:: .transforms.clean

image:: .transforms.python-image

test-src:: .transforms.test-src

setup:: .transforms.setup

build:: build-dist image

publish: publish-image

publish-image:: .transforms.publish-image-python

setup:: .transforms.setup

# distribution versions is the same as image version.
set-versions:
$(MAKE) TRANSFORM_PYTHON_VERSION=$(SYNTACTIC_CONCEPT_EXTRACTOR_PYTHON_VERSION) TOML_VERSION=$(SYNTACTIC_CONCEPT_EXTRACTOR_PYTHON_VERSION) .transforms.set-versions

build-dist:: .defaults.build-dist

publish-dist:: .defaults.publish-dist

# Ensure RUN_ARGS has a default value
RUN_ARGS ?= ""

# run-cli-sample: .transforms.run-cli-python-sample

run-local-sample: .transforms.run-local-sample

run-local-python-sample:
$(MAKE) RUN_FILE=syntactic_concept_extractor_local_python.py \
RUN_ARGS="--content 'Contents' --language 'Language'" \
.transforms.run-local-python-sample
43 changes: 43 additions & 0 deletions transforms/code/syntactic_concept_extractor/python/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Base Syntactic Concept Extractor Transform


## Configuration and command line Options

The set of dictionary keys holding [SyntacticConceptExtractorTransform](src/syntactic_concept_extractor_transform.py)
configuration for values are as follows:

* content - specifies the column name in the dataframe that has the code snippet
* language - specifies the programming languages of the code snippet

## Running

### Launched Command Line Options
The following command line arguments are available in addition to
the options provided by
the [python launcher](../../../../data-processing-lib/doc/python-launcher-options.md).

### Running the samples
To run the samples, use the following `make` targets

* `run-local-sample` - runs src/syntactic_concept_extractor_local.py
* `run-local-python-sample` - runs src/syntactic_concept_extractor_local_python.py

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
```shell
make run-local-sample
...
```
Then
```shell
ls output
```
To see results of the transform.

### Transforming data using the transform image

To use the transform image to transform your data, please refer to the
[running images quickstart](../../../../doc/quick-start/run-transform-image.md),
substituting the name of this transform image and runtime as appropriate.
Loading

0 comments on commit 085a28c

Please sign in to comment.