Add macro support, update docs and bump version to 0.7

README.rst

@@ -18,44 +18,83 @@ Overview
 * Free software: BSD license
 * Documentation: https://cfg4py.readthedocs.io.
 
-.. image:: docs/static/usage.gif
 
 A python config module that:
 
-1. support hierarchical configuration and multiple source (local and remote)
-2. IDE auto-complete
-3. configuration template (logging, database, cache, message queue,...) generated by console command
-4. enable logging in one line
+1. Adaptive deployment environment (default, dev, test, production) support
+2. Cascading configuration (central vs local) support
+3. Auto-complete
+4. Templates (logging, database, cache, message queue,...)
+5. Environment variables macro support
+6. Enable logging in one line
+7. Built on top of yaml
 
 Features
 ^^^^^^^^
 
-Hierarchical design
---------------------
-
-You have a bunch of severs of the same role, which usually share same configuration. But somehow for troubleshooting or maintenance purpose, you'd like some machines could have its own settings at particular moment.
-
-This is how Cfg4Py solves the problem:
+It's common to see that you have different settings for development machine, test machine and production site. They share many common settings, but a few of them has to be different.
 
-1. Configure your application general settings at remote service, then implement a `RemoteConfigFetcher` (Cfg4Py has already implemented one), which pull configuration from remote serivce periodically.
-2. Change the settings resides on local machine, then the change automatically applied.
+For example, developers should connect to local database server when performing unittest, and tester should connect to their own database server. All these servers should be deployed separately and no data should be messed up.
 
-The hierarchical design can have different meaning. It's common to see that you have different settings for development,
-test and production site. They share many common settings, but a few of them has to be different. Cfg4Py has perfect solution supporting for this: adaptive deployment environment support.
+Cfg4Py has perfect solution supporting for this: adaptive deployment environment support.
 
 Adaptive Deployment Environment Support
 ---------------------------------------
 In any serious projects, your application may run at both development, testing and production site. Except for effort of copying similar settings here and there, sometimes we'll mess up with development environment and production site. Once this happen, it could result in very serious consequence.
 
 To solve this, Cfg4Py developed a mechanism, that you provide different sets for configurations: dev for development machine, test for testing environment and production for production site, and all common settings are put into a file called 'defaults'.
 
-cfg4py module knows which environment it's running on by lookup environment variables __cfg4py_server_role__. It can be one of 'DEV', 'TEST' and 'PRODUCTION'. If nothing found, it means setup is not finished, and Cfg4Py will refuse to work. If the environment is set, then Cfg4Py will read settings from defaults set, then apply update from either of 'DEV', 'TEST' and 'PRODUCTION' set, according to the environment the application is running on.
+cfg4py module knows which environment it's running on by lookup environment variable __cfg4py_server_role__. It should be one of 'DEV', 'TEST' and 'PRODUCTION'. If nothing found, it means setup is not finished, and Cfg4Py will refuse to work. If the environment is set, then Cfg4Py will read settings from defaults set, then apply update from either of 'DEV', 'TEST' and 'PRODUCTION' set, according to the environment the application is running on.
 
-Quick logging config
+Cascading design
 --------------------
-Many python projects are startup prototype. If it works, then we'll put some really serious effort on it. Before that, we don't want our effort to be waste on common chores. Even though, we do need logging module at most time, to assist us for better troubleshooting.
 
-for that purpose, Cfg4Py provides a one-liner config for enabling logging:
+Assuming you have a bunch of severs for load-balance, which usually share same configurations. So you'd like put the configurations on a central repository, which could be a redis server or a relational database. Once you update configuration settings at central repository, you update configurations for all servers. But somehow for troubleshooting or maintenance purpose, you'd like some machines could have its own settings at a particular moment.
+
+This is how Cfg4Py solves the problem:
+
+1. Configure your application general settings at remote service, then implement a `RemoteConfigFetcher` (Cfg4Py has already implemented one, that read settings from redis), which pull configuration from remote serivce periodically.
+2. Change the settings on local machine, after the period you've set, these changes are popluated to all machines.
+
+Auto-complete
+---------------------------
+
+.. image:: docs/static/auto-complete.gif
+
+With other python config module, you have to remember all the configuration keys, and refer to each settings by something like cfg["services"]["redis"]["host"] and etc. Keys are hard to rememb, prone to typo, and way too much tedious.
+
+When cfg4py load raw settigns from yaml file, it'll compile all the settings into a Python class, then Cfg4Py let you access your settings by attributes. Compares the two ways to access configure item:
+
+        cfg["services"]["redis"]["host"]
+vs:
+        cfg.services.redis.host
+
+Apparently the latter is the better.
+
+And, if you trigger a build against your configurations, it'll generate a python class file. After you import this file (named 'cfg4py_auto_gen.py') into your project, then you can enjoy auto-complete!
+
+Templates
+----------
+It's hard to remember how to configure log, database, cache and etc, so cfg4py provide templates.
+
+Just run cfg4py scaffold, follow the tips then you're done.
+
+.. image:: docs/static/scaffold.png
+
+Environment variables macro
+----------------------------
+The best way to keep secret, is never share them. If you put account/password files, and these files may be leak to the public. For example, push to github by accident.
+
+With cfg4py, you can set these secret as environment variables, then use marco in config files. For example, if you have the following in defaults.yaml (any other files will do too):
+        postgres:
+                dsn: postgres://{postgres_account}:{postgres_password}@localhost
+
+then cfg4py will lookup postgres_account, postgres_password from environment variables and make replacement.
+
+
+Enable logging with one line
+-----------------------------
+with one line, you can enable file-rotating logging:
 
 .. code-block::python
     cfg.enable_logging(level, filename=None)
@@ -64,11 +103,11 @@ Apply configuration change on-the-fly
 -------------------------------------
 Cfg4Py provides mechanism to automatically apply configuration changes without restart your application. For local files configuration change, it may take effect immediately. For remote config change, it take effect up to `refresh_interval` settings.
 
-Code assist (auto-complete)
----------------------------
-With other python config module, you have to remember all the configuration keys, and refer to each settings by something like cfg["services"]["redis"]["host"] and etc. It's hard to remember all configuration keys, and the way we access these settings is less concise than just use cfg.services.redis.host.
+On top of yaml
+---------------
+The raw config format is backed by yaml, with macro enhancement. YAML is the best for configurations.
+
 
-Cfg4Py let you access your settings by the latter format all the time. And, if you trigger a build against your configurations, it'll generate a python class file. After you import this file (named 'cfg4py_auto_gen.py') into your project, then you can enjoy auto-complete!
 
 Credits
 -------

cfg4py/command_line.py

@@ -1,20 +1,20 @@
 import logging
 import os
 import sys
+from typing import Optional
 
 import fire
 from ruamel.yaml import YAML
-from typing import Optional
 
-from cfg4py import init, enable_logging, envar
+from cfg4py import enable_logging, envar, init
 
 enable_logging()
 
 
 class Command:
-
     def __init__(self):
-        self.resource_path = os.path.normpath(os.path.join(os.path.dirname(__file__), 'resources/'))
+        self.resource_path = os.path.normpath(
+            os.path.join(os.path.dirname(__file__), 'resources/'))
         self.yaml = YAML(typ='safe')  # default, if not specfied, is 'rt' (round-trip)
         self.yaml.default_flow_style = False
 
@@ -24,8 +24,8 @@ def __init__(self):
         self.transformed = self._transform()
 
     def build(self, config_dir: str):
-        """
-        Compile configuration files into a python class file, which is used by IDE's auto-complete function
+        """Compile configuration files into a python class file, which is used by IDE's
+        auto-complete function
 
         Args:
             config_dir: The folder where your configuration files located
@@ -39,7 +39,8 @@ def build(self, config_dir: str):
 
         count = 0
         for f in os.listdir(config_dir):
-            if f.startswith("default") or f.startswith("dev") or f.startswith("test") or f.startswith("production"):
+            if f.startswith("default") or f.startswith("dev") or f.startswith(
+                    "test") or f.startswith("production"):
                 print(f"found {f}")
                 count += 1
 
@@ -55,10 +56,11 @@ def build(self, config_dir: str):
 
             init(config_dir)
             sys.path.insert(0, config_dir)
-            # noinspection PyUnresolvedReferences
-            from cfg4py_auto_gen import Config
-            print(f"Config file is built with success and saved at {os.path.join(config_dir, 'cfg4py_auto_gen')}")
-        except Exception as e: # pragma: no cover
+            from cfg4py_auto_gen import Config  # type: ignore # noqa
+            output_file = f"{os.path.join(config_dir, 'cfg4py_auto_gen')}"
+            msg = f"Config file is built with success and saved at {output_file}"
+            print(msg)
+        except Exception as e:  # pragma: no cover
             logging.exception(e)
             print("Config file built failure.")
 
@@ -68,8 +70,10 @@ def _choose_dest_dir(self, dst):
 
         if os.path.exists(dst):
             for f in os.listdir(dst):
+                msg = f"The folder already contains {f}, please choose clean one."
+
                 if f in ['defaults.yaml', 'dev.yaml', 'test.yaml', 'production.yaml']:
-                    print(f"The folder provided already contains {f}, please choose clean one.")
+                    print(msg)
                     return None
             return dst
         else:
@@ -83,8 +87,7 @@ def _choose_dest_dir(self, dst):
                 return None
 
     def scaffold(self, dst: Optional[str]):
-        """
-        Creates initial configuration files based on our choices.
+        """Creates initial configuration files based on our choices.
         Args:
             dst:
 
@@ -113,7 +116,8 @@ def scaffold(self, dst: Optional[str]):
         50 - mongodb/pymongo (gh://mongodb/mongo-python-driver)
         """
         print(prompt)
-        chooses = input("Please choose flavors by index, separated each by a comma(,):\n")
+        chooses = input(
+            "Please choose flavors by index, separated each by a comma(,):\n")
         flavors = {}
         mapping = {
             "0": "logging",
@@ -137,14 +141,16 @@ def scaffold(self, dst: Optional[str]):
                 continue
 
         with open(os.path.join(dst, "defaults.yaml"), 'w') as f:
-            f.writelines("#auto generated by Cfg4Py: https://github.com/jieyu-tech/cfg4py\n")
+            f.writelines(
+                "#auto generated by Cfg4Py: https://github.com/jieyu-tech/cfg4py\n")
             yaml.dump(flavors, f)
 
         print(f"Cfg4Py has generated the following files under {dst}:")
         print("defaults.yaml")
         for name in ['dev.yaml', 'test.yaml', 'production.yaml']:
             with open(os.path.join(dst, name), 'w') as f:
-                f.writelines("#auto generated by Cfg4Py: https://github.com/jieyu-tech/cfg4py\n")
+                f.writelines(
+                    "#auto generated by Cfg4Py: https://github.com/jieyu-tech/cfg4py\n")
                 print(name)
 
         with open(os.path.join(dst, "defaults.yaml"), 'r') as f:
@@ -185,7 +191,8 @@ def hint(self, what: str = None, usage: bool = False):
         :param usage
         """
 
-        if what is None or ((what not in self.templates) and what not in self.transformed):
+        if what is None or (
+                (what not in self.templates) and what not in self.transformed):
             return self._show_supported_config()
 
         usage_key = f"{what}_usage"
@@ -205,14 +212,17 @@ def hint(self, what: str = None, usage: bool = False):
     def set_server_role(self):
         print("please add the following line into your .bashrc:\n")
         print(f"export {envar}=DEV\n")
-        print("You need to change DEV to TEST | PRODUCTION according to its actual role accordingly")
+        msg = "You need to change DEV to TEST | PRODUCTION according to its actual role\
+         accordingly"
+        print(msg)
+
 
 def main():
-    cmd = Command() # pragma: no cover
-    fire.Fire({ # pragma: no cover
-        "build":           cmd.build,
-        "scaffold":        cmd.scaffold,
-        "hint":            cmd.hint,
+    cmd = Command()  # pragma: no cover
+    fire.Fire({  # pragma: no cover
+        "build": cmd.build,
+        "scaffold": cmd.scaffold,
+        "hint": cmd.hint,
         "set_server_role": cmd.set_server_role
     })
 

cfg4py/config.py

@@ -1,3 +1,4 @@
+# noqa
 from typing import Optional
 
 
@@ -9,7 +10,8 @@ def __cfg4py_reset_access_counter__(self):
 
     def __getattribute__(self, name):
         """
-        keep tracking if the config is accessed. If there's no access, then even the refresh interval is reached, we
+        keep tracking if the config is accessed. If there's no access, then even the
+         refresh interval is reached, we
         will not call the remote fetcher.
 
         Args: