init

2021-06-29 12:37:08 +03:00
commit 12fdc96d20
282 changed files with 13627 additions and 0 deletions
--- a/git-exporter/CHANGELOG.md
+++ b/git-exporter/CHANGELOG.md
@@ -0,0 +1,134 @@
+## 1.10.1 - 2021-06-03
+
+* 📝 Add secrets corner case docs for secrets containing special regex chars. (thanks @fliphess)
+
+
+## 1.10.0 - 2021-05-08
+
+* 🐛 Don't encode GitHub Tokens
+* 🔼 Updated python3 to `3.8.10-r0`
+
+
+## 1.9.0 - 2021-04-02
+
+* 🔨 Use ghcr.io/home-assistant for base images
+
+
+## 1.8.0 - 2021-03-30
+
+* 🔨 URL encode git password to support special chars (see #219)
+* 🔼 Updated git to `2.30.2-r0`
+* 🔼 Updated python3 to `3.8.8-r0`
+
+
+## 1.7.1 - 2021-03-04
+
+* 🔼 Updated git to `2.30.1-r0`
+* 🔼 Updated python3 to `3.8.7-r1`
+
+
+## 1.7.0 - 2021-01-30
+
+* 🔼 Updated to alpine `3.13`
+* 🔼 Updated git to `2.30.0-r0`
+* 🔼 Updated rsync to `3.2.3-r1`
+* 🔼 Updated grep to `3.6-r0`
+* 🔼 Updated python3 to `3.8.7-r0`
+* 🔼 Updated py3-pip to `20.3.4-r0`
+* 🔼 Updated findutils to `4.8.0-r0`
+* 🔨 Use Jemalloc for better memory handling
+
+
+## 1.6.0 - 2021-01-06
+
+* 🔨 Add dummy mac and ip adresses as allowed
+* 🐛 Exclude also all Node-Red backup files
+
+
+## 1.5.0 - 2020-12-29
+
+* 🐛 Fixed addon config export (#135), thanks @LiJu09
+
+## 1.4.4 - 2020-10-22
+
+* 🔨 Disable S6-Overlay Init
+
+
+## 1.4.3 - 2020-10-06
+
+* 🔼 Updated python3 to `3.8.5-r0`
+
+
+## 1.4.2 - 2020-06-25
+
+* 🔼 Updated to alpine `3.12`
+* 🔼 Updated git to `2.26.2-r0`
+* 🔼 Updated make to `4.3-r0`
+* 🔼 Updated rsync to `3.1.3-r3`
+* 🔼 Updated grep to `3.4-r0`
+* 🔼 Updated sed to `4.8-r0`
+* 🔼 Updated python3 to `3.8.3-r0`
+* 🔼 Updated py3-pip to `20.1.1-re0`
+* 🔨 Added more default ignore patterns
+
+
+## 1.4.1 - 2020-05-22
+
+* 🔨 Updated Changelog to new format
+
+
+## 1.4.0 - 2020-05-20
+
+* ➕ Export addon repositories
+* 🔨 Changed config documentation
+* ➕ Add Node-Red check
+* ➕ Add commiter email option
+
+
+## 1.3.2 - 2020-04-29
+
+* 🔼 Updated git to `2.24.3-r0`
+
+
+## 1.3.1 - 2020-04-16
+
+* 🔼 Updated git to `2.24.2-r0`
+
+
+## 1.3.0 - 2020-04-06
+
+* ➕ Add export support for node-red
+
+
+## 1.2.1 - 2020-03-30
+
+* ➕ Add findutils at version `4.7.0-r0`
+* 🐛 Fixed lovelace export
+* 🔨 Get the addon information from bashio instead of self-requesting
+* ➖ Removed `curl`
+
+
+## 1.2.0 - 2020-03-29
+
+* 🔨 Changed json2yaml conversion to own little script
+* 🔼 Updated py3-yaml ot `5.3.1-r0`
+* 🐛 Fixed permanent dry_run bug
+
+
+## 1.1.0 - 2020-03-18
+
+* ➕ Dry run toogle to just display the changes.
+* ➕ Adjusted to the multiple lovelace interfaces
+* 🔼 Updated to python `3.8.2-r0`
+
+
+## 1.0 - 2020-02-21
+
+* ➕ Started git exporter addon with basic git push functionality
+* ➕ Added excludes in config
+* ➕ Lovelace exports
+* ➕ Addons config exports
+* ➕ ESPHome exports
+* ➕ Configurable Commit message
+* ➕ Configurable exports
+
--- a/git-exporter/DOCS.md
+++ b/git-exporter/DOCS.md
@@ -0,0 +1,106 @@
+# Configuration
+
+```yaml
+repository:
+  url: <path to your repository>
+  username: user
+  password: pass
+  pull_before_push: true
+  commit_message: 'Home Assistant Git Exporter'
+export:
+  lovelace: true
+  addons: true
+  esphome: true
+  node_red: true
+checks:
+  enabled: true
+  check_for_secrets: true
+  check_for_ips: true
+exclude:
+  - '*.db'
+  - '*.log'
+  - __pycache__
+  - deps/
+  - known_devices.yaml
+  - tts/
+dry_run: false
+```
+
+### `repository.url`
+
+Any https url to your git repository. (For now _no_ SSH)
+
+### `repository.email` (Optional)
+
+The email address the commits author is using.
+
+### `repository.username`
+
+Your username for https authentication.
+
+### `repository.password`
+
+Your password or __access token__ for your repository.
+
+### `repository.pull_before_push`
+
+Should the repository be pulled first and commit the new state on top?
+
+### `repository.commit_message`
+
+The commit message for the next commit.
+
+
+### `export.lovelace`
+
+Enable / Disable the export for the lovelace config.
+
+### `export.addons`
+
+Enable / Disable the export for the supervisor addons config.
+
+### `export.esphome`
+
+Enable / Disable the export for the esphome config.
+
+### `export.node_red`
+
+Enable / Disable the export for the Node-RED flows.
+Secure your credentials with [node-red-contrib-credentials](https://flows.nodered.org/node/node-red-contrib-credentials).
+
+
+### `checks.enabled`
+
+Enable / Disable the checks in the exported files.
+
+### `checks.check_for_secrets`
+
+Add your secret values to the check.
+
+### `checks.check_for_ips`
+
+Add pattern for ip and mac addresses to the search.
+
+
+### `exclude`
+
+The files / folders which should be excluded from the config export.
+
+Following folders and files are excluded from the sync per default:
+
+* `secrets.yaml` (secrets are cleared)
+* `.cloud`
+* `.storage`
+
+### `dry_run`
+
+Only show the changes and don't commit or push.
+
+
+## Known limitations
+
+`check_for_secrets` Uses a git plugin that does pattern matching using regexes.
+A limitation of this plugin is that using brackets (like `[`, `]`, `{`, `}` `(` and `)`) in secrets can result in unexpected behaviour and crashes.
+
+If the addon fails during secrets checking with errors originating from grep (I.E. `grep: Unmatched [, [^, [:, [., or [=`),
+change the passwords that contain brackets or set `check_for_secrets` to `false`.
--- a/git-exporter/Dockerfile
+++ b/git-exporter/Dockerfile
@@ -0,0 +1,31 @@
+ARG BUILD_FROM
+FROM $BUILD_FROM AS BUILD
+
+RUN apk add --no-cache \
+		'git=2.30.2-r0' \
+		'make=4.3-r0'
+
+ENV GIT_SECRET_VERSION 1.3.0
+# hadolint ignore=DL3003
+RUN git config --global advice.detachedHead false && \
+    git clone https://github.com/awslabs/git-secrets.git -b ${GIT_SECRET_VERSION} /git-secrets && \
+    cd /git-secrets && make install
+
+FROM $BUILD_FROM AS RUNNING
+
+RUN apk add --no-cache \
+			'git=2.30.2-r0' \
+			'rsync=3.2.3-r1' \
+			'grep=3.6-r0' \
+			'sed=4.8-r0' \
+			'python3=3.8.10-r0' \
+			'py3-pip=20.3.4-r0' \
+			'findutils=4.8.0-r0' 
+
+RUN pip3 install --no-cache-dir PyYAML==5.3.1 dotty-dict==1.2.1
+
+COPY --from=BUILD /usr/local/bin/git-secrets /usr/local/bin/git-secrets
+
+COPY root /
+RUN chmod a+x /run.sh /utils/*
+ENTRYPOINT [ "/run.sh" ]
--- a/git-exporter/README.md
+++ b/git-exporter/README.md
@@ -0,0 +1,53 @@
+# Home Assistant Git Exporter
+
+Export all of your Home Assistant configuration to a git repository of your choice.
+Can be used to show your Home Assistant setup in public repositories.
+
+![Addon Stage][stage-badge]
+![Supports aarch64 Architecture][aarch64-badge]
+![Supports amd64 Architecture][amd64-badge]
+![Supports armhf Architecture][armhf-badge]
+![Supports armv7 Architecture][armv7-badge]
+![Supports i386 Architecture][i386-badge]
+
+[![Install on my Home Assistant][install-badge]][install-url]
+[![Donate][donation-badge]][donation-url]
+
+# Functionality
+
+* Export Home Assistant configuration
+* Export Supervisor Addon configuration
+* Export Lovelace configuration
+* Export ESPHome configurations
+* Export Node-RED flows
+* Check for plaintext secrets based on your `secrets.yaml` file and common patterns.
+* Check for plaintext ip and addresses in your config.
+
+# Example
+
+For an example take a look at my own [Home Assistant configuration](https://github.com/Poeschl/home-assistant-config).
+The folders there are gettings synced with this addon.
+
+
+# Badge
+
+If you export your config with this addon and want to help me to spread it further. Here is a badge you can embedd in your readme.
+
+[![Home Assistant Git Exporter](https://img.shields.io/badge/Powered%20by-Home%20Assistant%20Git%20Exporter-%23d32f2f)](https://github.com/Poeschl/Hassio-Addons/tree/master/git-exporter)
+
+```markdown
+[![Home Assistant Git Exporter](https://img.shields.io/badge/Powered%20by-Home%20Assistant%20Git%20Exporter-%23d32f2f)](https://github.com/Poeschl/Hassio-Addons/tree/master/git-exporter)
+```
+
+
+[aarch64-badge]: https://img.shields.io/badge/aarch64-yes-green.svg?style=for-the-badge
+[amd64-badge]: https://img.shields.io/badge/amd64-yes-green.svg?style=for-the-badge
+[armhf-badge]: https://img.shields.io/badge/armhf-yes-green.svg?style=for-the-badge
+[armv7-badge]: https://img.shields.io/badge/armv7-yes-green.svg?style=for-the-badge
+[i386-badge]: https://img.shields.io/badge/i386-yes-green.svg?style=for-the-badge
+[install-url]: https://my.home-assistant.io/redirect/supervisor_addon?addon=243ffc37_git-exporter
+[stage-badge]: https://img.shields.io/badge/Addon%20stage-stable-green.svg?style=for-the-badge
+
+[install-badge]: https://img.shields.io/badge/Install%20on%20my-Home%20Assistant-41BDF5?logo=home-assistant&style=for-the-badge
+[donation-badge]: https://img.shields.io/badge/Buy%20me%20a%20coffee-%23d32f2f?logo=buy-me-a-coffee&style=for-the-badge&logoColor=white
+[donation-url]: https://www.buymeacoffee.com/Poeschl
--- a/git-exporter/build.json
+++ b/git-exporter/build.json
@@ -0,0 +1,9 @@
+{
+  "build_from": {
+    "armhf": "ghcr.io/home-assistant/armhf-base:3.13",
+    "armv7": "ghcr.io/home-assistant/armv7-base:3.13",
+    "aarch64": "ghcr.io/home-assistant/aarch64-base:3.13",
+    "amd64": "ghcr.io/home-assistant/amd64-base:3.13",
+    "i386": "ghcr.io/home-assistant/i386-base:3.13"
+  }
+}
--- a/git-exporter/config.json
+++ b/git-exporter/config.json
@@ -0,0 +1,75 @@
+
+{
+  "name": "Home Assistant Git Exporter",
+  "version": "1.10.1",
+  "slug": "git-exporter",
+  "description": "Export all of your Home Assistant configuration to a git repository of your choice.",
+  "url": "https://github.com/Poeschl/Hassio-Addons/tree/master/git-exporter",
+  "arch": [
+    "armhf",
+    "armv7",
+    "aarch64",
+    "amd64",
+    "i386"
+  ],
+  "startup": "once",
+  "boot": "manual",
+  "image": "ghcr.io/poeschl/ha-git-exporter-{arch}",
+  "hassio_api": true,
+  "hassio_role": "manager",
+  "map": ["config"],
+  "options": {
+    "repository": {
+      "url": "",
+      "username": "",
+      "password": "",
+      "pull_before_push": true,
+      "commit_message": "Home Assistant Git Exporter"
+    },
+    "export": {
+      "lovelace": true,
+      "addons": true,
+      "esphome": true,
+      "node_red": true
+    },
+    "check": {
+      "enabled": true,
+      "check_for_secrets": true,
+      "check_for_ips": true
+    },
+    "exclude": [
+      "*.db",
+      "*.log",
+      "__pycache__",
+      "._*",
+      ".DS_Store",
+      "deps/",
+      "known_devices.yaml",
+      "tts/"
+    ],
+    "dry_run": false
+  },
+  "schema": {
+    "repository": {
+      "url": "match(https://.*)",
+      "email": "match(.+@.+)?",
+      "username": "str",
+      "password": "password",
+      "pull_before_push": "bool",
+      "commit_message": "str"
+    },
+    "export": {
+      "lovelace": "bool",
+      "addons": "bool",
+      "esphome": "bool",
+      "node_red": "bool"
+    },
+    "check": {
+      "enabled": "bool",
+      "check_for_secrets": "bool",
+      "check_for_ips": "bool"
+    },
+    "exclude": ["str"],
+    "dry_run": "bool"
+  }
+}
--- a/git-exporter/icon.png
+++ b/git-exporter/icon.png
--- a/git-exporter/logo.png
+++ b/git-exporter/logo.png
--- a/git-exporter/root/run.sh
+++ b/git-exporter/root/run.sh
@@ -0,0 +1,196 @@
+#!/usr/bin/env bashio
+set -e
+
+# Enable Jemalloc for better memory handling
+export LD_PRELOAD="/usr/local/lib/libjemalloc.so.2"
+
+local_repository='/data/repository'
+pull_before_push="$(bashio::config 'repository.pull_before_push')"
+
+function setup_git {
+    repository=$(bashio::config 'repository.url')
+    username=$(bashio::config 'repository.username')
+    password=$(bashio::config 'repository.password')
+    commiter_mail=$(bashio::config 'repository.email')
+
+    if [[ "$password" != "ghp_*" ]]; then
+        password=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${password}'))")
+    fi
+
+    if [ ! -d $local_repository ]; then
+        bashio::log.info 'Create local repository'
+        mkdir -p $local_repository
+    fi
+    cd $local_repository
+
+    if [ ! -d .git ]; then
+        fullurl="https://${username}:${password}@${repository##*https://}"
+        if [ "$pull_before_push" == 'true' ]; then
+            bashio::log.info 'Clone existing repository'
+            git clone "$fullurl" $local_repository
+        else
+            bashio::log.info 'Initialize new repository'
+            git init $local_repository
+            git remote add origin "$fullurl"
+        fi
+        git config user.name "${username}"
+        git config user.email "${commiter_mail:-git.exporter@home-assistant}"
+    fi
+
+    #Reset secrets if existing
+    git config --unset-all 'secrets.allowed' || true
+    git config --unset-all 'secrets.patterns' || true
+    git config --unset-all 'secrets.providers' || true
+
+    if [ "$pull_before_push" == 'true' ]; then
+        bashio::log.info 'Pull latest'
+        git fetch
+        git reset --hard origin/master
+    fi
+}
+
+function check_secrets {
+    bashio::log.info 'Add secrets pattern'
+
+    # Allow !secret lines
+    git secrets --add -a '!secret'
+
+    # Set prohibited patterns
+    git secrets --add "password:\s?[\'\"]?\w+[\'\"]?\n?"
+    git secrets --add "token:\s?[\'\"]?\w+[\'\"]?\n?"
+    git secrets --add "client_id:\s?[\'\"]?\w+[\'\"]?\n?"
+    git secrets --add "api_key:\s?[\'\"]?\w+[\'\"]?\n?"
+    git secrets --add "chat_id:\s?[\'\"]?\w+[\'\"]?\n?"
+    git secrets --add "allowed_chat_ids:\s?[\'\"]?\w+[\'\"]?\n?"
+    git secrets --add "latitude:\s?[\'\"]?\w+[\'\"]?\n?"
+    git secrets --add "longitude:\s?[\'\"]?\w+[\'\"]?\n?"
+    git secrets --add "credential_secret:\s?[\'\"]?\w+[\'\"]?\n?"
+
+    if [ "$(bashio::config 'check.check_for_secrets')" == 'true' ]; then
+        git secrets --add-provider -- sed '/^$/d;/^#.*/d;/^&/d;s/^.*://g;s/\s//g' /config/secrets.yaml
+    fi
+
+    if [ "$(bashio::config 'check.check_for_ips')" == 'true' ]; then
+        git secrets --add '([0-9]{1,3}\.){3}[0-9]{1,3}'
+        git secrets --add '([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})'
+
+        # Allow dummy / general ips and mac
+        git secrets --add -a 'AA:BB:CC:DD:EE:FF'
+        git secrets --add -a '123.456.789.123'
+        git secrets --add -a '0.0.0.0'
+    fi
+
+    bashio::log.info 'Add secrets from secrets.yaml'
+    prohibited_patterns=$(git config --get-all secrets.patterns)
+    bashio::log.info "Prohibited patterns:\n${prohibited_patterns//\\n/\\\\n}"
+
+    bashio::log.info 'Checking for secrets'
+    # shellcheck disable=SC2046
+    git secrets --scan $(find $local_repository -name '*.yaml' -o -name '*.yml' -o -name '*.json' -o -name '*.disabled') \
+    || (bashio::log.error 'Found secrets in files!!! Fix them to be able to commit!' && exit 1)
+}
+
+function export_ha_config {
+    bashio::log.info 'Get Home Assistant config'
+    excludes=$(bashio::config 'exclude')
+    excludes=("secrets.yaml" ".storage" ".cloud" "esphome/" ".uuid" "node-red/" "${excludes[@]}")
+
+    # Cleanup existing esphome folder from config
+    [ -d "${local_repository}/config/esphome" ] && rm -r "${local_repository}/config/esphome"
+    # shellcheck disable=SC2068
+    exclude_args=$(printf -- '--exclude=%s ' ${excludes[@]})
+    # shellcheck disable=SC2086
+    rsync -archive --compress --delete --checksum --prune-empty-dirs -q --include='.gitignore' $exclude_args /config ${local_repository}
+    sed 's/:.*$/: ""/g' /config/secrets.yaml > ${local_repository}/config/secrets.yaml
+    chmod 644 -R ${local_repository}/config
+}
+
+function export_lovelace {
+    bashio::log.info 'Get Lovelace config yaml'
+    [ ! -d "${local_repository}/lovelace" ] && mkdir "${local_repository}/lovelace"
+    mkdir -p '/tmp/lovelace'
+    find /config/.storage -name "lovelace*" -printf '%f\n' | xargs -I % cp /config/.storage/% /tmp/lovelace/%.json
+    /utils/jsonToYaml.py '/tmp/lovelace/' 'data'
+    rsync -archive --compress --delete --checksum --prune-empty-dirs -q --include='*.yaml' --exclude='*' /tmp/lovelace/ "${local_repository}/lovelace"
+    chmod 644 -R "${local_repository}/lovelace"
+}
+
+function export_esphome {
+    bashio::log.info 'Get ESPHome configs'
+    rsync -archive --compress --delete --checksum --prune-empty-dirs -q \
+         --exclude='.esphome*' --include='*/' --include='.gitignore' --include='*.yaml' --include='*.disabled' --exclude='secrets.yaml' --exclude='*' \
+        /config/esphome ${local_repository}
+    [ -f /config/esphome/secrets.yaml ] && sed 's/:.*$/: ""/g' /config/esphome/secrets.yaml > ${local_repository}/esphome/secrets.yaml
+    chmod 644 -R ${local_repository}/esphome
+}
+
+function export_addons {
+    [ -d ${local_repository}/addons ] || mkdir -p ${local_repository}/addons
+    installed_addons=$(bashio::addons.installed)
+    mkdir '/tmp/addons/'
+    for addon in $installed_addons; do
+        if [ "$(bashio::addons.installed "${addon}")" == 'true' ]; then
+            bashio::log.info "Get ${addon} configs"
+            bashio::addon.options "$addon" >  /tmp/tmp.json
+            /utils/jsonToYaml.py /tmp/tmp.json
+            mv /tmp/tmp.yaml "/tmp/addons/${addon}.yaml"
+        fi
+    done
+    bashio::log.info "Get addon repositories"
+    bashio::addons false 'addons.repositorys' '.repositories | map(select(.source != null)) | map({(.name): {source,maintainer,slug}}) | add' > /tmp/tmp.json
+    /utils/jsonToYaml.py /tmp/tmp.json
+    mv /tmp/tmp.yaml "/tmp/addons/repositories.yaml"
+    rsync -archive --compress --delete --checksum --prune-empty-dirs -q /tmp/addons/ ${local_repository}/addons
+    chmod 644 -R ${local_repository}/addons
+}
+
+function export_node-red {
+    bashio::log.info 'Get Node-RED flows'
+    rsync -archive --compress --delete --checksum --prune-empty-dirs -q \
+          --exclude='flows_cred.json' --exclude='*.backup' --include='flows.json' --include='settings.js' --exclude='*' \
+        /config/node-red/ ${local_repository}/node-red
+    chmod 644 -R ${local_repository}/node-red
+}
+
+bashio::log.info 'Start git export'
+
+setup_git
+
+export_ha_config
+
+if [ "$(bashio::config 'export.lovelace')" == 'true' ]; then
+    export_lovelace
+fi
+
+if [ "$(bashio::config 'export.esphome')" == 'true' ] && [ -d '/config/esphome' ]; then
+    export_esphome
+fi
+
+if [ "$(bashio::config 'export.addons')" == 'true' ]; then
+    export_addons
+fi
+
+if [ "$(bashio::config 'export.node_red')" == 'true' ] && [ -d '/config/node-red' ]; then
+    export_node-red
+fi
+
+if [ "$(bashio::config 'check.enabled')" == 'true' ]; then
+    check_secrets
+fi
+
+
+if [ "$(bashio::config 'dry_run')" == 'true' ]; then
+    git status
+else
+    bashio::log.info 'Commit changes and push to remote'
+    git add .
+    git commit -m "$(bashio::config 'repository.commit_message')"
+
+    if [ ! "$pull_before_push" == 'true' ]; then
+        git push --set-upstream origin master -f
+    else
+        git push origin
+    fi
+fi
+
+bashio::log.info 'Exporter finished'
--- a/git-exporter/root/utils/jsonToYaml.py
+++ b/git-exporter/root/utils/jsonToYaml.py
@@ -0,0 +1,31 @@
+#!/usr/bin/env python3
+import json
+import sys
+import os
+import yaml
+from dotty_dict import dotty
+
+def convert_file(file, path):
+    yaml_file_name = os.path.splitext(file)[0] + ".yaml"
+    #print (file + ' -> ' + yaml_file_name)
+    with open(file, 'r') as infile:
+        with open(yaml_file_name, 'w+') as outfile:
+            if path is not None:
+                data = dotty(json.load(infile)).get(path)
+            else:
+                data = json.load(infile)
+
+            yaml.dump(data, outfile, default_flow_style=False)
+
+input_file = sys.argv[1]
+if len(sys.argv) > 2:
+    path = sys.argv[2]
+else:
+    path = None
+
+if os.path.isfile(input_file):
+    convert_file(input_file, path)
+else:
+    #print ('Convert folder ' + input_file)
+    for json_file in os.listdir(input_file):
+        convert_file(os.path.dirname(input_file) + '/' + json_file, path)