Merge pull request #225 from xnuinside/v0.32.0_release

V0.32.0 release

CHANGELOG.txt

@@ -1,6 +1,16 @@
-**v0.31.4**
+**v0.32.0**
+
+### Improvements
+1. Added support for several ALTER statements (ADD, DROP, RENAME, etc) - https://github.com/xnuinside/simple-ddl-parser/issues/215
+In 'alter' output added several keys:
+    1. 'dropped_columns' - to store information about columns that was in table, but after dropped by alter
+    2. 'renamed_columns' - to store information about columns that was renamed
+    3. 'modified_columns' - to track alter column changes for defaults, datetype, etc. Argument stores previous columns states.
+
 ### Fixes
 1. Include source column names in FOREIGN KEY references. Fix for: https://github.com/xnuinside/simple-ddl-parser/issues/196
+2. ALTER statement now will be parsed correctly if names & schemas written differently in `create table` statement and alter. 
+For example, if in create table you use quotes like "schema_name"."table_name", but in alter was schema_name.table_name - previously it didn't work, but now parser understand that it is the same table.
 
 **v0.31.3**
 ### Improvements

README.md

@@ -335,7 +335,7 @@ In output you will have names like 'dbo' and 'TO_Requests', not '[dbo]' and '[TO
 
 - STATEMENTS: PRIMARY KEY, CHECK, FOREIGN KEY in table definitions (in create table();)
 
-- ALTER TABLE STATEMENTS: ADD CHECK (with CONSTRAINT), ADD FOREIGN KEY (with CONSTRAINT), ADD UNIQUE, ADD DEFAULT FOR, ALTER TABLE ONLY, ALTER TABLE IF EXISTS; ALTER .. PRIMARY KEY; ALTER .. USING INDEX TABLESPACE
+- ALTER TABLE STATEMENTS: ADD CHECK (with CONSTRAINT), ADD FOREIGN KEY (with CONSTRAINT), ADD UNIQUE, ADD DEFAULT FOR, ALTER TABLE ONLY, ALTER TABLE IF EXISTS; ALTER .. PRIMARY KEY; ALTER .. USING INDEX TABLESPACE; ALTER .. ADD; ALTER .. MODIFY; ALTER .. ALTER COLUMN; etc
 
 - PARTITION BY statement
 
@@ -461,24 +461,15 @@ To set logging level you should provide argument 'log_level'
 
 ```
 
-### TODO in next Releases (if you don't see feature that you need - open the issue)
-
--1. Update command line to parse all arguments, that supported by Parser
-0. Add support for ALTER TABLE ... ADD COLUMN
-1. Add more support for CREATE type IS TABLE (example: CREATE OR REPLACE TYPE budget_tbl_typ IS TABLE OF NUMBER(8,2);
-2. Add support (ignore correctly) ALTER TABLE ... DROP CONSTRAINT ..., ALTER TABLE ... DROP INDEX ...
-3. Change output for CHECKS -> 'checks': [{"column_name": str, "operator": =
->=|<|>|<=..., "value": value}]
-4. Add support for ALTER TABLE ... ADD INDEX 
-
-## non-feature todo
-
-0. Provide API to get result as Python Object
-1. Add online demo (UI) to parse ddl
-
-
 ### Thanks for involving & contributions
 
+Most biggest 'Thanks' ever goes for contributions in parser:
+https://github.com/dmaresma
+https://github.com/cfhowes
+https://github.com/swiatek25
+https://github.com/slurpyb
+https://github.com/PBalsdon
+
 Big thanks for the involving & contribution with test cases with DDL samples & opening issues goes to:
 
 * https://github.com/kukigai , 
@@ -489,14 +480,22 @@ for help with debugging & testing support for BigQuery dialect DDLs:
 * https://github.com/ankitdata ,
 * https://github.com/kalyan939
 
-And most biggest 'Thanks' ever goes for contributions in parser:
-https://github.com/swiatek25
-https://github.com/slurpyb
-https://github.com/dmaresma
-https://github.com/PBalsdon
-
 
 ## Changelog
+**v0.32.0**
+
+### Improvements
+1. Added support for several ALTER statements (ADD, DROP, RENAME, etc) - https://github.com/xnuinside/simple-ddl-parser/issues/215
+In 'alter' output added several keys:
+    1. 'dropped_columns' - to store information about columns that was in table, but after dropped by alter
+    2. 'renamed_columns' - to store information about columns that was renamed
+    3. 'modified_columns' - to track alter column changes for defaults, datetype, etc. Argument stores previous columns states.
+
+### Fixes
+1. Include source column names in FOREIGN KEY references. Fix for: https://github.com/xnuinside/simple-ddl-parser/issues/196
+2. ALTER statement now will be parsed correctly if names & schemas written differently in `create table` statement and alter. 
+For example, if in create table you use quotes like "schema_name"."table_name", but in alter was schema_name.table_name - previously it didn't work, but now parser understand that it is the same table.
+
 **v0.31.3**
 ### Improvements
 #### Snowflake update:

docs/README.rst

@@ -361,7 +361,7 @@ Supported Statements
   STATEMENTS: PRIMARY KEY, CHECK, FOREIGN KEY in table definitions (in create table();)
 
 * 
-  ALTER TABLE STATEMENTS: ADD CHECK (with CONSTRAINT), ADD FOREIGN KEY (with CONSTRAINT), ADD UNIQUE, ADD DEFAULT FOR, ALTER TABLE ONLY, ALTER TABLE IF EXISTS; ALTER .. PRIMARY KEY; ALTER .. USING INDEX TABLESPACE
+  ALTER TABLE STATEMENTS: ADD CHECK (with CONSTRAINT), ADD FOREIGN KEY (with CONSTRAINT), ADD UNIQUE, ADD DEFAULT FOR, ALTER TABLE ONLY, ALTER TABLE IF EXISTS; ALTER .. PRIMARY KEY; ALTER .. USING INDEX TABLESPACE; ALTER .. ADD; ALTER .. MODIFY; ALTER .. ALTER COLUMN; etc
 
 * 
   PARTITION BY statement
@@ -518,33 +518,16 @@ To set logging level you should provide argument 'log_level'
 
        DDLParser(ddl, log_level=logging.INFO).run(group_by_type=True)
 
-TODO in next Releases (if you don't see feature that you need - open the issue)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-
-
-
-#. Add support for ALTER TABLE ... ADD COLUMN
-#. Add more support for CREATE type IS TABLE (example: CREATE OR REPLACE TYPE budget_tbl_typ IS TABLE OF NUMBER(8,2);
-#. Add support (ignore correctly) ALTER TABLE ... DROP CONSTRAINT ..., ALTER TABLE ... DROP INDEX ...
-#. Change output for CHECKS -> 'checks': [{"column_name": str, "operator": =
-   ..
-
-      =|<|>|<=..., "value": value}]
-
-
-#. Add support for ALTER TABLE ... ADD INDEX 
-
-non-feature todo
-----------------
-
-
-#. Provide API to get result as Python Object
-#. Add online demo (UI) to parse ddl
-
 Thanks for involving & contributions
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+Most biggest 'Thanks' ever goes for contributions in parser:
+https://github.com/dmaresma
+https://github.com/cfhowes
+https://github.com/swiatek25
+https://github.com/slurpyb
+https://github.com/PBalsdon
+
 Big thanks for the involving & contribution with test cases with DDL samples & opening issues goes to:
 
 
@@ -558,15 +541,30 @@ for help with debugging & testing support for BigQuery dialect DDLs:
 * https://github.com/ankitdata ,
 * https://github.com/kalyan939
 
-And most biggest 'Thanks' ever goes for contributions in parser:
-https://github.com/swiatek25
-https://github.com/slurpyb
-https://github.com/dmaresma
-https://github.com/PBalsdon
-
 Changelog
 ---------
 
+**v0.32.0**
+
+Improvements
+^^^^^^^^^^^^
+
+
+#. Added support for several ALTER statements (ADD, DROP, RENAME, etc) - https://github.com/xnuinside/simple-ddl-parser/issues/215
+   In 'alter' output added several keys:
+
+   #. 'dropped_columns' - to store information about columns that was in table, but after dropped by alter
+   #. 'renamed_columns' - to store information about columns that was renamed
+   #. 'modified_columns' - to track alter column changes for defaults, datetype, etc. Argument stores previous columns states.
+
+Fixes
+^^^^^
+
+
+#. Include source column names in FOREIGN KEY references. Fix for: https://github.com/xnuinside/simple-ddl-parser/issues/196
+#. ALTER statement now will be parsed correctly if names & schemas written differently in ``create table`` statement and alter. 
+   For example, if in create table you use quotes like "schema_name"."table_name", but in alter was schema_name.table_name - previously it didn't work, but now parser understand that it is the same table.
+
 **v0.31.3**
 
 Improvements

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "simple-ddl-parser"
-version = "0.31.3"
+version = "0.32.0"
 description = "Simple DDL Parser to parse SQL & dialects like HQL, TSQL (MSSQL), Oracle, AWS Redshift, Snowflake, MySQL, PostgreSQL, etc ddl files to json/python dict with full information about columns: types, defaults, primary keys, etc.; sequences, alters, custom types & other entities from ddl."
 authors = ["Iuliia Volkova <[email protected]>"]
 license = "MIT"

simple_ddl_parser/ddl_parser.py

@@ -189,6 +189,11 @@ def t_ID(self, t: LexToken):
         else:
             t = self.tokens_not_columns_names(t)
 
+        if self.lexer.is_alter:
+            _type = tok.alter_tokens.get(t.value)
+            if _type:
+                t.type = _type
+
         self.capitalize_tokens(t)
         self.commat_type(t)
 

simple_ddl_parser/dialects/hql.py

@@ -1,16 +1,18 @@
+from typing import List
+
 from simple_ddl_parser.utils import check_spec, remove_par
 
 
 class HQL:
-    def p_expression_location(self, p):
+    def p_expression_location(self, p: List) -> None:
         """expr : expr LOCATION STRING
         | expr LOCATION DQ_STRING
         | expr LOCATION table_property_equals"""
         p[0] = p[1]
         p_list = list(p)
         p[0]["location"] = p_list[-1]
 
-    def p_expression_clustered(self, p):
+    def p_expression_clustered(self, p: List) -> None:
         """expr : expr ID ON LP pid RP
         | expr ID by_smthg"""
         p[0] = p[1]
@@ -21,20 +23,20 @@ def p_expression_clustered(self, p):
         else:
             p[0][f"{p_list[2].lower()}_{p_list[3].lower()}"] = p_list[-2]
 
-    def p_expression_into_buckets(self, p):
+    def p_expression_into_buckets(self, p: List) -> None:
         """expr : expr INTO ID ID"""
         p[0] = p[1]
         p_list = list(p)
         p[0][f"{p_list[2].lower()}_{p_list[-1].lower()}"] = p_list[-2]
 
-    def p_row_format(self, p):
+    def p_row_format(self, p: List) -> None:
         """row_format : ROW FORMAT SERDE
         | ROW FORMAT
         """
         p_list = list(p)
         p[0] = {"serde": p_list[-1] == "SERDE"}
 
-    def p_expression_row_format(self, p):
+    def p_expression_row_format(self, p: List) -> None:
         """expr : expr row_format id
         | expr row_format STRING
         """
@@ -47,7 +49,7 @@ def p_expression_row_format(self, p):
 
         p[0]["row_format"] = format
 
-    def p_expression_with_serde(self, p):
+    def p_expression_with_serde(self, p: List) -> None:
         """expr : expr WITH SERDEPROPERTIES multi_assignments"""
         p[0] = p[1]
         p_list = list(p)
@@ -56,20 +58,20 @@ def p_expression_with_serde(self, p):
         row_format["properties"] = p_list[-1]
         p[0]["row_format"] = row_format
 
-    def p_expression_tblproperties(self, p):
+    def p_expression_tblproperties(self, p: List) -> None:
         """expr : expr TBLPROPERTIES multi_assignments"""
         p[0] = p[1]
         p[0]["tblproperties"] = list(p)[-1]
 
-    def p_multi_assignments(self, p):
+    def p_multi_assignments(self, p: List) -> None:
         """multi_assignments : LP assignment
         | multi_assignments RP
         | multi_assignments COMMA assignment"""
         p_list = remove_par(list(p))
         p[0] = p_list[1]
         p[0].update(p_list[-1])
 
-    def p_assignment(self, p):
+    def p_assignment(self, p: List) -> None:
         """assignment : id id id
         |  STRING id STRING
         |  id id STRING
@@ -83,43 +85,43 @@ def p_assignment(self, p):
                 p_list[-1] = p_list[-1].split("=")[-1]
             p[0] = {p_list[1]: p_list[-1]}
 
-    def p_expression_comment(self, p):
+    def p_expression_comment(self, p: List) -> None:
         """expr : expr COMMENT STRING"""
         p[0] = p[1]
         p_list = list(p)
         p[0]["comment"] = check_spec(p_list[-1])
 
-    def p_expression_terminated_by(self, p):
+    def p_expression_terminated_by(self, p: List) -> None:
         """expr : expr id TERMINATED BY id
         | expr id TERMINATED BY STRING
         """
         p[0] = p[1]
         p_list = list(p)
         p[0][f"{p[2].lower()}_terminated_by"] = check_spec(p_list[-1])
 
-    def p_expression_map_keys_terminated_by(self, p):
+    def p_expression_map_keys_terminated_by(self, p: List) -> None:
         """expr : expr MAP KEYS TERMINATED BY id
         | expr MAP KEYS TERMINATED BY STRING
         """
         p[0] = p[1]
         p_list = list(p)
         p[0]["map_keys_terminated_by"] = check_spec(p_list[-1])
 
-    def p_expression_skewed_by(self, p):
+    def p_expression_skewed_by(self, p: List) -> None:
         """expr : expr SKEWED BY LP id RP ON LP pid RP"""
         p[0] = p[1]
         p_list = remove_par(list(p))
         p[0]["skewed_by"] = {"key": p_list[4], "on": p_list[-1]}
 
-    def p_expression_collection_terminated_by(self, p):
+    def p_expression_collection_terminated_by(self, p: List) -> None:
         """expr : expr COLLECTION ITEMS TERMINATED BY id
         | expr COLLECTION ITEMS TERMINATED BY STRING
         """
         p[0] = p[1]
         p_list = list(p)
         p[0]["collection_items_terminated_by"] = check_spec(p_list[-1])
 
-    def p_expression_stored_as(self, p):
+    def p_expression_stored_as(self, p: List) -> None:
         """expr : expr STORED AS id
         |  expr STORED AS id STRING
         |  expr STORED AS id STRING id STRING
@@ -135,7 +137,7 @@ def p_expression_stored_as(self, p):
         else:
             p[0]["stored_as"] = p_list[-1]
 
-    def p_expression_partitioned_by_hql(self, p):
+    def p_expression_partitioned_by_hql(self, p: List) -> None:
         """expr : expr PARTITIONED BY pid_with_type
         | expr PARTITIONED BY LP pid RP
         | expr PARTITIONED BY LP multiple_funct RP
@@ -144,7 +146,7 @@ def p_expression_partitioned_by_hql(self, p):
         p_list = remove_par(list(p))
         p[0]["partitioned_by"] = p_list[-1]
 
-    def p_pid_with_type(self, p):
+    def p_pid_with_type(self, p: List) -> None:
         """pid_with_type :  LP column
         | pid_with_type COMMA column
         | pid_with_type RP

simple_ddl_parser/dialects/ibm.py

@@ -1,5 +1,8 @@
+from typing import List
+
+
 class IBMDb2:
-    def p_expr_index_in(self, p):
+    def p_expr_index_in(self, p: List) -> None:
         """expr : expr INDEX id id"""
         p_list = list(p)
         if p_list[-2].upper() == "IN":