more from_prefixed_env features

* support nested dict access with "__" separator
* don't specify separator in prefix
* catch exceptions for any loads function

CHANGES.rst

@@ -55,6 +55,10 @@ Unreleased
 -   From Werkzeug, for redirect responses the ``Location`` header URL
     will remain relative, and exclude the scheme and domain, by default.
     :pr:`4496`
+-   Add ``Config.from_prefixed_env()`` to load config values from
+    environment variables that start with ``FLASK_`` or another prefix.
+    This parses values as JSON by default, and allows setting keys in
+    nested dicts. :pr:`4479`
 
 
 Version 2.0.3

docs/config.rst

@@ -521,7 +521,8 @@ configuration values directly from the environment. Flask can be
 instructed to load all environment variables starting with a specific
 prefix into the config using :meth:`~flask.Config.from_prefixed_env`.
 
-Environment variables can be set in the shell before starting the server:
+Environment variables can be set in the shell before starting the
+server:
 
 .. tabs::
 
@@ -561,30 +562,43 @@ Environment variables can be set in the shell before starting the server:
          > flask run
           * Running on http://127.0.0.1:5000/
 
-The variables can then be loaded and accessed via the config with a
+The variables can then be loaded and accessed via the config with a key
-key equal to the environment variable name without the prefix i.e.
+equal to the environment variable name without the prefix i.e.
 
 .. code-block:: python
 
     app.config.from_prefixed_env()
     app.config["SECRET_KEY"]  # Is "5f352379324c22463451387a0aec5d2f"
 
-The prefix is ``FLASK_`` by default, however it is an configurable via
+The prefix is ``FLASK_`` by default. This is configurable via the
-the ``prefix`` argument of :meth:`~flask.Config.from_prefixed_env`.
+``prefix`` argument of :meth:`~flask.Config.from_prefixed_env`.
 
-Whilst the value of any environment variable is a string, it will be
+Values will be parsed to attempt to convert them to a more specific type
-parsed before being placed into the flask config. By default the
+than strings. By default :func:`json.loads` is used, so any valid JSON
-parsing is done by json.loads, however this is configurable via the
+value is possible, including lists and dicts. This is configurable via
-``loads`` argument of :meth:`~flask.Config.from_prefixed_env`.
+the ``loads`` argument of :meth:`~flask.Config.from_prefixed_env`.
 
-Notice that any value besides an empty string will be interpreted as a boolean
+When adding a boolean value with the default JSON parsing, only "true"
-``True`` value in Python, which requires care if an environment explicitly sets
+and "false", lowercase, are valid values. Keep in mind that any
-values intended to be ``False``.
+non-empty string is considered ``True`` by Python.
 
-Make sure to load the configuration very early on, so that extensions have the
+It is possible to set keys in nested dictionaries by separating the
-ability to access the configuration when starting up.  There are other methods
+keys with double underscore (``__``). Any intermediate keys that don't
-on the config object as well to load from individual files.  For a complete
+exist on the parent dict will be initialized to an empty dict.
-reference, read the :class:`~flask.Config` class documentation.
+
+.. code-block:: text
+
+    $ export FLASK_MYAPI__credentials__username=user123
+
+.. code-block:: python
+
+    app.config["MYAPI"]["credentials"]["username"]  # Is "user123"
+
+For even more config loading features, including merging, try a
+dedicated library such as Dynaconf_, which includes integration with
+Flask.
+
+.. _Dynaconf: https://www.dynaconf.com/
 
 
 Configuration Best Practices
@@ -604,6 +618,10 @@ that experience:
     limit yourself to request-only accesses to the configuration you can
     reconfigure the object later on as needed.
 
+3.  Make sure to load the configuration very early on, so that
+    extensions can access the configuration when calling ``init_app``.
+
+
 .. _config-dev-prod:
 
 Development / Production

src/flask/config.py

@@ -78,7 +78,7 @@ class Config(dict):
     """
 
     def __init__(self, root_path: str, defaults: t.Optional[dict] = None) -> None:
-        dict.__init__(self, defaults or {})
+        super().__init__(defaults or {})
         self.root_path = root_path
 
     def from_envvar(self, variable_name: str, silent: bool = False) -> bool:
@@ -106,42 +106,68 @@ class Config(dict):
         return self.from_pyfile(rv, silent=silent)
 
     def from_prefixed_env(
-        self,
+        self, prefix: str = "FLASK", *, loads: t.Callable[[str], t.Any] = json.loads
-        prefix: str = "FLASK_",
-        *,
-        loads: t.Callable[[t.Union[str, bytes]], t.Any] = _json_loads,
     ) -> bool:
-        """Updates the config from environment variables with the prefix.
+        """Load any environment variables that start with ``FLASK_``,
+        dropping the prefix from the env key for the config key. Values
+        are passed through a loading function to attempt to convert them
+        to more specific types than strings.
 
-        Calling this method will result in every environment variable
+        Keys are loaded in :func:`sorted` order.
-        starting with **prefix** being placed into the configuration
-        without the **prefix**. The prefix is configurable as an
-        argument. Note that this method updates the existing config.
 
-        For example if there is an environment variable
+        The default loading function attempts to parse values as any
-        ``FLASK_SECRET_KEY`` with value ``secretly`` and the prefix is
+        valid JSON type, including dicts and lists.
-        ``FLASK_`` the config will contain the key ``SECRET_KEY`` with
-        the value ``secretly`` after calling this method.
 
-        The value of the environment variable will be passed to the
+        Specific items in nested dicts can be set by separating the
-        **loads** parameter before being placed into the config. By
+        keys with double underscores (``__``). If an intermediate key
-        default **loads** utilises the stdlib json.loads to parse the
+        doesn't exist, it will be initialized to an empty dict.
-        value, falling back to the value itself on parsing error.
 
-        :param loads: A callable that takes a str (or bytes) returns
+        :param prefix: Load env vars that start with this prefix,
-            the parsed value.
+            separated with an underscore (``_``).
-        :return: Always returns ``True``.
+        :param loads: Pass each string value to this function and use
-
+            the returned value as the config value. If any error is
-        .. versionadded:: 2.1.0
+            raised it is ignored and the value remains a string. The
+            default is :func:`json.loads`.
 
+        .. versionadded:: 2.1
         """
-        mapping = {}
+        prefix = f"{prefix}_"
-        for raw_key, value in os.environ.items():
+        len_prefix = len(prefix)
-            if raw_key.startswith(prefix):
-                key = raw_key[len(prefix) :]  # Use removeprefix with Python 3.9
-                mapping[key] = loads(value)
 
-        return self.from_mapping(mapping)
+        for key in sorted(os.environ):
+            if not key.startswith(prefix):
+                continue
+
+            value = os.environ[key]
+
+            try:
+                value = loads(value)
+            except Exception:
+                # Keep the value as a string if loading failed.
+                pass
+
+            # Change to key.removeprefix(prefix) on Python >= 3.9.
+            key = key[len_prefix:]
+
+            if "__" not in key:
+                # A non-nested key, set directly.
+                self[key] = value
+                continue
+
+            # Traverse nested dictionaries with keys separated by "__".
+            current = self
+            *parts, tail = key.split("__")
+
+            for part in parts:
+                # If an intermediate dict does not exist, create it.
+                if part not in current:
+                    current[part] = {}
+
+                current = current[part]
+
+            current[tail] = value
+
+        return True
 
     def from_pyfile(self, filename: str, silent: bool = False) -> bool:
         """Updates the values in the config from a Python file.  This function

tests/test_config.py

@@ -38,28 +38,54 @@ def test_config_from_file():
     common_object_test(app)
 
 
-def test_config_from_prefixed_env(monkeypatch):
+def test_from_prefixed_env(monkeypatch):
+    monkeypatch.setenv("FLASK_STRING", "value")
+    monkeypatch.setenv("FLASK_BOOL", "true")
+    monkeypatch.setenv("FLASK_INT", "1")
+    monkeypatch.setenv("FLASK_FLOAT", "1.2")
+    monkeypatch.setenv("FLASK_LIST", "[1, 2]")
+    monkeypatch.setenv("FLASK_DICT", '{"k": "v"}')
+    monkeypatch.setenv("NOT_FLASK_OTHER", "other")
+
     app = flask.Flask(__name__)
-    monkeypatch.setenv("FLASK_A", "A value")
-    monkeypatch.setenv("FLASK_B", "true")
-    monkeypatch.setenv("FLASK_C", "1")
-    monkeypatch.setenv("FLASK_D", "1.2")
-    monkeypatch.setenv("NOT_FLASK_A", "Another value")
     app.config.from_prefixed_env()
-    assert app.config["A"] == "A value"
+
-    assert app.config["B"] is True
+    assert app.config["STRING"] == "value"
-    assert app.config["C"] == 1
+    assert app.config["BOOL"] is True
-    assert app.config["D"] == 1.2
+    assert app.config["INT"] == 1
-    assert "Another value" not in app.config.items()
+    assert app.config["FLOAT"] == 1.2
+    assert app.config["LIST"] == [1, 2]
+    assert app.config["DICT"] == {"k": "v"}
+    assert "OTHER" not in app.config
 
 
-def test_config_from_custom_prefixed_env(monkeypatch):
+def test_from_prefixed_env_custom_prefix(monkeypatch):
+    monkeypatch.setenv("FLASK_A", "a")
+    monkeypatch.setenv("NOT_FLASK_A", "b")
+
     app = flask.Flask(__name__)
-    monkeypatch.setenv("FLASK_A", "A value")
+    app.config.from_prefixed_env("NOT_FLASK")
-    monkeypatch.setenv("NOT_FLASK_A", "Another value")
+
-    app.config.from_prefixed_env("NOT_FLASK_")
+    assert app.config["A"] == "b"
-    assert app.config["A"] == "Another value"
+
-    assert "A value" not in app.config.items()
+
+def test_from_prefixed_env_nested(monkeypatch):
+    monkeypatch.setenv("FLASK_EXIST__ok", "other")
+    monkeypatch.setenv("FLASK_EXIST__inner__ik", "2")
+    monkeypatch.setenv("FLASK_EXIST__new__more", '{"k": false}')
+    monkeypatch.setenv("FLASK_NEW__K", "v")
+
+    app = flask.Flask(__name__)
+    app.config["EXIST"] = {"ok": "value", "flag": True, "inner": {"ik": 1}}
+    app.config.from_prefixed_env()
+
+    assert app.config["EXIST"] == {
+        "ok": "other",
+        "flag": True,
+        "inner": {"ik": 2},
+        "new": {"more": {"k": False}},
+    }
+    assert app.config["NEW"] == {"K": "v"}
 
 
 def test_config_from_mapping():