# Khéops 101 — Khéops 0.1 documentation

# Khéops 101

    cd /home/jez/volumes/data/prj/bell/training/tiger-ansible/ext/kheops
    echo $PWD
    export KHEOPS_NAMESPACE=ex1_enc
    export KHEOPS_CONFIG=examples/kheops.yml
    rm -rf "examples/$KHEOPS_NAMESPACE"

    /home/jez/volumes/data/prj/bell/training/tiger-ansible/ext/kheops

## Command line

Let’s check first that kheops works correclty, and start to read the manual.

    kheops --help

    usage: kheops [-h] [-v] [-c CONFIG]
                  {schema,gen_doc,lookup,demo,subcommand2} ...

    Kheops, hierarchical data lookup tool

    options:
      -h, --help            show this help message and exit
      -v, --verbose         Increase verbosity (KHEOPS_VERBOSE) (default: 0)
      -c CONFIG, --config CONFIG
                            Kheops configuration file (KHEOPS_CONFIG) (default:
                            examples/kheops.yml)

    subcommands:
      valid subcommands

      {schema,gen_doc,lookup,demo,subcommand2}

So we have a working `kheops` command, and we will focus on the `lookup` command. On it’s simplest form, a lookup consists in querying a `key` for a given `scope`. The output of the `key` will change depending the `scope` value. A `key` is in simple word.

## Defining a hierarchy

To illustrate how Khéops works, let’s start with a simple example, we will try to lookup the `profile` key of the following two (fictive) servers:

- web.domain.org: which act as a webserver role

- mysql.domain.org: which act as mysql role

But first we need to create our hierarchy. It’s as simple as creating directories and put some json or yaml data into different files. Let’s create our hierarchy. We will first create the default profile:

From our use case, we will build a lookup tree. We want to be able to organise data depending the 3 criterias:

- node: name of the node

- role: assigned role to the node

- environment: it can either be dev or prod

Let’s create our lookup hierarchy:

default: lookups: - path: default # Simplest form, just a path - path: “roles/{role}” # If list, it’s auto expanded like in bash - path: “env_{env}” # If list, it’s auto expanded like in bash - path: “nodes/{node}”

So for a given key, Khéops will iterate all over those paths to find the requested `key` , and then it will merge all results. Some paths are variabilized, and those variable comes from the scope. The scope come along the `key`, it’s can be any metadata. For complex metadata you may want to store those in a file and load your scope with the `-f <yaml_scope_file>` option:

    kheops lookup -e <var1=val1> -e <var2=val2> <key>

A scope is completely optional while keys are required.

## Basic hierarchy

    # We create a fresh hierachy
    mkdir -p examples/$KHEOPS_NAMESPACE

    # We create a profile key, which is a dict
    cat > examples/$KHEOPS_NAMESPACE/default.yml <<EOF
    ---
    profile:
      env: "NO_ENV"
      product: "NO_PRODUCT"

    EOF

    # Let's inspect our hierarchy
    tree examples/$KHEOPS_NAMESPACE
    cat examples/$KHEOPS_NAMESPACE/default.yml

    [01;34mexamples/ex1_enc[00m
    └── default.yml

    0 directories, 1 file
    ---
    profile:
      env: "NO_ENV"
      product: "NO_PRODUCT"

From this point, we defined our profile with two attribute, `team` and `product`. As it’s the default case, we set them both unconfigured.

You are now already able to query your hierarchy:

    kheops lookup profile

    env: NO_ENV
    product: NO_PRODUCT

Good, no surprise. But, we mentionned we wanted to get the profile of two instances, this how would do that:

    kheops lookup -e node=web.infra.net profile

    env: NO_ENV
    product: NO_PRODUCT

## Roles

However, same result as before, which is expected as we did not finished to configure our hierarchy. Among our instances, we identified 2 roles: web and mysql. Let’s create those two roles:

    mkdir -p examples/$KHEOPS_NAMESPACE/roles

    # We create a new web role
    cat > examples/$KHEOPS_NAMESPACE/roles/web.yml <<EOF
    ---
    profile:
      product: "httpd_server"

      web_top_domain: ""
      web_app: "NO_APP"
      web_port: 80
      web_user_list:
        - sysadmins

    EOF

    # We create a new mysql role
    cat > examples/$KHEOPS_NAMESPACE/roles/mysql.yml <<EOF
    ---
    profile:
      product: "mysql_server"

      mysql_database: "NO_DATABASE"
      mysql_users:
        - "sysadmin@10.0.42%"
      mysql_port: 3306
      mysql_cluster: False

    EOF

    # Let's inspect our hierarchy
    tree examples/$KHEOPS_NAMESPACE

    [01;34mexamples/ex1_enc[00m
    ├── default.yml
    └── [01;34mroles[00m
        ├── mysql.yml
        └── web.yml

    1 directory, 3 files

    tail -n 999 examples/$KHEOPS_NAMESPACE/{*.yml,*/*.yml}

    ==> examples/ex1_enc/default.yml <==
    ---
    profile:
      env: "NO_ENV"
      product: "NO_PRODUCT"

    ==> examples/ex1_enc/roles/mysql.yml <==
    ---
    profile:
      product: "mysql_server"

      mysql_database: "NO_DATABASE"
      mysql_users:
        - "sysadmin@10.0.42%"
      mysql_port: 3306
      mysql_cluster: False

    ==> examples/ex1_enc/roles/web.yml <==
    ---
    profile:
      product: "httpd_server"

      web_top_domain: ""
      web_app: "NO_APP"
      web_port: 80
      web_user_list:
        - sysadmins

    kheops lookup -e node=web.infra.net -e role=web profile
    kheops lookup -e node=mysql.infra.net -e role=mysql profile

    env: NO_ENV
    product: httpd_server
    web_top_domain: ''
    web_app: NO_APP
    web_port: 80
    web_user_list:
    - sysadmins

    env: NO_ENV
    product: mysql_server
    mysql_database: NO_DATABASE
    mysql_users:
    - sysadmin@10.0.42%
    mysql_port: 3306
    mysql_cluster: false

## Per node override

It’s getting better, we can see that the profile key has been merged with the key values, across the different locations.

However, we will have those placeholders, and we want to have personalized value, depending if it’s aweb server, it need an unique domain and some unique parameters. So let’s create a `nodes` directory and place some data inside.

    mkdir -p examples/$KHEOPS_NAMESPACE/nodes

    # We create a new web role
    cat > examples/$KHEOPS_NAMESPACE/nodes/web.infra.net.yml <<EOF
    ---
    profile:
      web_app: 'myapp'
      web_user_list:
        - domain_org
        - domain_org_external

    EOF

    # We create a new mysql role
    cat > examples/$KHEOPS_NAMESPACE/nodes/mysql.infra.net.yml <<EOF
    ---
    profile:
      mysql_database: "app_domain_org"
      mysql_users:
        - "app_domain_org@10.0.51%"

    EOF

    # Let's inspect our hierarchy
    tree examples/$KHEOPS_NAMESPACE

    [01;34mexamples/ex1_enc[00m
    ├── default.yml
    ├── [01;34mnodes[00m
    │   ├── mysql.infra.net.yml
    │   └── web.infra.net.yml
    └── [01;34mroles[00m
        ├── mysql.yml
        └── web.yml

    2 directories, 5 files

And we try again:

    kheops lookup -e node=web.infra.net -e role=web profile
    kheops lookup -e node=mysql.infra.net -e role=mysql profile

    env: NO_ENV
    product: httpd_server
    web_top_domain: ''
    web_app: myapp
    web_port: 80
    web_user_list:
    - sysadmins
    - domain_org
    - domain_org_external

    env: NO_ENV
    product: mysql_server
    mysql_database: app_domain_org
    mysql_users:
    - sysadmin@10.0.42%
    - app_domain_org@10.0.51%
    mysql_port: 3306
    mysql_cluster: false

## Environment override

Let’s say you want to support environment, it’s the same:

    # We create a new dev environment
    cat > examples/$KHEOPS_NAMESPACE/env_dev.yml <<EOF
    ---
    profile:
      env: dev

      # We change the top domain for dev environment, and reduce the cache
      web_top_domain: dev.infra.net
      web_cache: 1m

      # We want a debug users
      web_user_list:
        - debug_user
      mysql_users:
        - debug@10.0.%

      debug: true

    EOF

    # We create a new mysql role
    cat > examples/$KHEOPS_NAMESPACE/env_prod.yml <<EOF
    ---
    profile:
      env: prod

      # On production environment, we always want to use public faced domain and 12 hour cache.
      web_top_domain: infra.com
      web_cache: 12h

    EOF

    # Let's inspect our hierarchy
    tree examples/$KHEOPS_NAMESPACE

    [01;34mexamples/ex1_enc[00m
    ├── default.yml
    ├── env_dev.yml
    ├── env_prod.yml
    ├── [01;34mnodes[00m
    │   ├── mysql.infra.net.yml
    │   └── web.infra.net.yml
    └── [01;34mroles[00m
        ├── mysql.yml
        └── web.yml

    2 directories, 7 files

So it’s become quite easy to compare the difference between environment, with a simple variable switch:

    kheops lookup -e node=web.infra.net -e role=web -e env=prod profile
    kheops lookup -e node=web.infra.net -e role=web -e env=dev profile

    env: prod
    product: httpd_server
    web_top_domain: infra.com
    web_app: myapp
    web_port: 80
    web_user_list:
    - sysadmins
    - domain_org
    - domain_org_external
    web_cache: 12h

    env: dev
    product: httpd_server
    web_top_domain: dev.infra.net
    web_app: myapp
    web_port: 80
    web_user_list:
    - sysadmins
    - debug_user
    - domain_org
    - domain_org_external
    web_cache: 1m
    mysql_users:
    - debug@10.0.%
    debug: true

Same for mysql:

    kheops lookup -e node=mysql.infra.net -e role=mysql -e env=prod profile
    kheops lookup -e node=mysql.infra.net -e role=mysql -e env=dev profile

    env: prod
    product: mysql_server
    mysql_database: app_domain_org
    mysql_users:
    - sysadmin@10.0.42%
    - app_domain_org@10.0.51%
    mysql_port: 3306
    mysql_cluster: false
    web_top_domain: infra.com
    web_cache: 12h

    env: dev
    product: mysql_server
    mysql_database: app_domain_org
    mysql_users:
    - sysadmin@10.0.42%
    - debug@10.0.%
    - app_domain_org@10.0.51%
    mysql_port: 3306
    mysql_cluster: false
    web_top_domain: dev.infra.net
    web_cache: 1m
    web_user_list:
    - debug_user
    debug: true

You have to keep in mind you can query the key with a different scope, and get different views:

    kheops lookup profile

    echo "==> Per environment view"
    kheops lookup -e env=prod profile
    kheops lookup -e env=dev profile

    echo "==> Per role and  environment view"
    kheops lookup -e role=mysql -e env=prod profile
    kheops lookup -e role=web -e env=prod profile

    echo "==> Per node view"
    kheops lookup -e node=web.infra.net -e role=web -e env=dev profile

    env: NO_ENV
    product: NO_PRODUCT

    ==> Per environment view
    env: prod
    product: NO_PRODUCT
    web_top_domain: infra.com
    web_cache: 12h

    env: dev
    product: NO_PRODUCT
    web_top_domain: dev.infra.net
    web_cache: 1m
    web_user_list:
    - debug_user
    mysql_users:
    - debug@10.0.%
    debug: true

    ==> Per role and  environment view
    env: prod
    product: mysql_server
    mysql_database: NO_DATABASE
    mysql_users:
    - sysadmin@10.0.42%
    mysql_port: 3306
    mysql_cluster: false
    web_top_domain: infra.com
    web_cache: 12h

    env: prod
    product: httpd_server
    web_top_domain: infra.com
    web_app: NO_APP
    web_port: 80
    web_user_list:
    - sysadmins
    web_cache: 12h

    ==> Per node view
    env: dev
    product: httpd_server
    web_top_domain: dev.infra.net
    web_app: myapp
    web_port: 80
    web_user_list:
    - sysadmins
    - debug_user
    - domain_org
    - domain_org_external
    web_cache: 1m
    mysql_users:
    - debug@10.0.%
    debug: true

Even if somwaht clunky, this method can help to troubleshoot wrong data by dichotomy.

## Tooling and applications

## Troubleshooting

Sometimes, it can may be hard to navigate across file and hierachy, but GNU Utils are here to help. There is a selection of small tips:

    set -x

    : Find where a key has been defined
    grep -r '^profile:' examples/$KHEOPS_NAMESPACE

    : Find where a key has been defined and 5 first lines
    grep -r -A 5 'web_user_list:' examples/$KHEOPS_NAMESPACE

    : Search from anything related to database
    grep -R -C 3  'database' examples/$KHEOPS_NAMESPACE

    set +x

    + : Find where a key has been defined
    + grep --colour=auto -r '^profile:' examples/ex1_enc
    [35m[Kexamples/ex1_enc/env_prod.yml[m[K[36m[K:[m[K[01;31m[Kprofile:[m[K
    [35m[Kexamples/ex1_enc/roles/mysql.yml[m[K[36m[K:[m[K[01;31m[Kprofile:[m[K
    [35m[Kexamples/ex1_enc/roles/web.yml[m[K[36m[K:[m[K[01;31m[Kprofile:[m[K
    [35m[Kexamples/ex1_enc/nodes/mysql.infra.net.yml[m[K[36m[K:[m[K[01;31m[Kprofile:[m[K
    [35m[Kexamples/ex1_enc/nodes/web.infra.net.yml[m[K[36m[K:[m[K[01;31m[Kprofile:[m[K
    [35m[Kexamples/ex1_enc/default.yml[m[K[36m[K:[m[K[01;31m[Kprofile:[m[K
    [35m[Kexamples/ex1_enc/env_dev.yml[m[K[36m[K:[m[K[01;31m[Kprofile:[m[K
    + : Find where a key has been defined and 5 first lines
    + grep --colour=auto -r -A 5 web_user_list: examples/ex1_enc
    [35m[Kexamples/ex1_enc/roles/web.yml[m[K[36m[K:[m[K  [01;31m[Kweb_user_list:[m[K
    [35m[Kexamples/ex1_enc/roles/web.yml[m[K[36m[K-[m[K    - sysadmins
    [35m[Kexamples/ex1_enc/roles/web.yml[m[K[36m[K-[m[K
    [36m[K--[m[K
    [35m[Kexamples/ex1_enc/nodes/web.infra.net.yml[m[K[36m[K:[m[K  [01;31m[Kweb_user_list:[m[K
    [35m[Kexamples/ex1_enc/nodes/web.infra.net.yml[m[K[36m[K-[m[K    - domain_org
    [35m[Kexamples/ex1_enc/nodes/web.infra.net.yml[m[K[36m[K-[m[K    - domain_org_external
    [35m[Kexamples/ex1_enc/nodes/web.infra.net.yml[m[K[36m[K-[m[K
    [36m[K--[m[K
    [35m[Kexamples/ex1_enc/env_dev.yml[m[K[36m[K:[m[K  [01;31m[Kweb_user_list:[m[K
    [35m[Kexamples/ex1_enc/env_dev.yml[m[K[36m[K-[m[K    - debug_user
    [35m[Kexamples/ex1_enc/env_dev.yml[m[K[36m[K-[m[K  mysql_users:
    [35m[Kexamples/ex1_enc/env_dev.yml[m[K[36m[K-[m[K    - debug@10.0.%
    [35m[Kexamples/ex1_enc/env_dev.yml[m[K[36m[K-[m[K
    [35m[Kexamples/ex1_enc/env_dev.yml[m[K[36m[K-[m[K  debug: true
    + : Search from anything related to database
    + grep --colour=auto -R -C 3 database examples/ex1_enc
    [35m[Kexamples/ex1_enc/roles/mysql.yml[m[K[36m[K-[m[Kprofile:
    [35m[Kexamples/ex1_enc/roles/mysql.yml[m[K[36m[K-[m[K  product: "mysql_server"
    [35m[Kexamples/ex1_enc/roles/mysql.yml[m[K[36m[K-[m[K
    [35m[Kexamples/ex1_enc/roles/mysql.yml[m[K[36m[K:[m[K  mysql_[01;31m[Kdatabase[m[K: "NO_DATABASE"
    [35m[Kexamples/ex1_enc/roles/mysql.yml[m[K[36m[K-[m[K  mysql_users:
    [35m[Kexamples/ex1_enc/roles/mysql.yml[m[K[36m[K-[m[K    - "sysadmin@10.0.42%"
    [35m[Kexamples/ex1_enc/roles/mysql.yml[m[K[36m[K-[m[K  mysql_port: 3306
    [36m[K--[m[K
    [35m[Kexamples/ex1_enc/nodes/mysql.infra.net.yml[m[K[36m[K-[m[K---
    [35m[Kexamples/ex1_enc/nodes/mysql.infra.net.yml[m[K[36m[K-[m[Kprofile:
    [35m[Kexamples/ex1_enc/nodes/mysql.infra.net.yml[m[K[36m[K:[m[K  mysql_[01;31m[Kdatabase[m[K: "app_domain_org"
    [35m[Kexamples/ex1_enc/nodes/mysql.infra.net.yml[m[K[36m[K-[m[K  mysql_users:
    [35m[Kexamples/ex1_enc/nodes/mysql.infra.net.yml[m[K[36m[K-[m[K    - "app_domain_org@10.0.51%"
    [35m[Kexamples/ex1_enc/nodes/mysql.infra.net.yml[m[K[36m[K-[m[K
    + set +x

The tail/head command is quite usefull to look at multiple files at the same time, it add a nice header for each file:

    head -n 999 examples/ex1_enc/roles/*

    ==> examples/ex1_enc/roles/mysql.yml <==
    ---
    profile:
      product: "mysql_server"

      mysql_database: "NO_DATABASE"
      mysql_users:
        - "sysadmin@10.0.42%"
      mysql_port: 3306
      mysql_cluster: False

    ==> examples/ex1_enc/roles/web.yml <==
    ---
    profile:
      product: "httpd_server"

      web_top_domain: ""
      web_app: "NO_APP"
      web_port: 80
      web_user_list:
        - sysadmins

You can also have a view of all files with this command:

    find . -type f| xargs head -n 999 | less

From there, you will be able to have a nice overview of your data.

You can even diff your change with this command. There is this simple trick to compare the data difference between 2 lookups:

    diff -u \
    <(kheops lookup -e node=web.infra.net -e role=web -e env=prod profile) \
    <(kheops lookup -e node=web.infra.net -e role=web -e env=dev profile)

    --- /dev/fd/63	2022-02-01 20:10:53.094525316 -0500
    +++ /dev/fd/62	2022-02-01 20:10:53.094525316 -0500
    @@ -1,11 +1,15 @@
    -env: prod
    +env: dev
     product: httpd_server
    -web_top_domain: infra.com
    +web_top_domain: dev.infra.net
     web_app: myapp
     web_port: 80
     web_user_list:
     - sysadmins
    +- debug_user
     - domain_org
     - domain_org_external
    -web_cache: 12h
    +web_cache: 1m
    +mysql_users:
    +- debug@10.0.%
    +debug: true

You can also ask Kheops to explain you how he built the result, you can use the `-x` flag:

    kheops lookup -e role=web profile -x

        INFO: Explain lookups:
    +------------------------+------------------------------+
    | Config                 | Runtime                      |
    +------------------------+------------------------------+
    |                        |                              |
    | Config:{               | Runtime:{                    |
    |   "path": "default",   |   "scope": {                 |
    |   "backend": "file",   |     "role": "web"            |
    |   "continue": true     |   },                         |
    | }                      |   "key": "profile",          |
    |                        |   "conf": {                  |
    |                        |     "index": 0               |
    |                        |   },                         |
    |                        |   "raw_path": "default"      |
    |                        | }                            |
    |                        |                              |
    | Config:{               | Runtime:{                    |
    |   "path": "roles/web", |   "scope": {                 |
    |   "backend": "file",   |     "role": "web"            |
    |   "continue": true     |   },                         |
    | }                      |   "key": "profile",          |
    |                        |   "conf": {                  |
    |                        |     "index": 1               |
    |                        |   },                         |
    |                        |   "raw_path": "roles/{role}" |
    |                        | }                            |
    +------------------------+------------------------------+
        INFO: Explain candidates:
    +----------------------------------------------------------------------------------+-------------------------------+------------------------------+
    | Status                                                                           | Runtime                       | Data                         |
    +----------------------------------------------------------------------------------+-------------------------------+------------------------------+
    |                                                                                  |                               |                              |
    | Status:{                                                                         | Runtime:{                     | Data:{                       |
    |   "path": "/home/jez/volumes/data/prj/bell/training/tiger-ansible/ext/kheops ... |   "scope": {                  |   "env": "NO_ENV",           |
    |   "status": "found",                                                             |     "role": "web"             |   "product": "NO_PRODUCT"    |
    |   "rel_path": "examples/ex1_enc/default.yml"                                     |   },                          | }                            |
    | }                                                                                |   "key": "profile",           |                              |
    |                                                                                  |   "conf": {                   |                              |
    |                                                                                  |     "index": 0                |                              |
    |                                                                                  |   },                          |                              |
    |                                                                                  |   "raw_path": "default",      |                              |
    |                                                                                  |   "backend_index": 0          |                              |
    |                                                                                  | }                             |                              |
    |                                                                                  |                               |                              |
    | Status:{                                                                         | Runtime:{                     | Data:{                       |
    |   "path": "/home/jez/volumes/data/prj/bell/training/tiger-ansible/ext/kheops ... |   "scope": {                  |   "product": "httpd_server", |
    |   "status": "found",                                                             |     "role": "web"             |   "web_top_domain": "",      |
    |   "rel_path": "examples/ex1_enc/roles/web.yml"                                   |   },                          |   "web_app": "NO_APP",       |
    | }                                                                                |   "key": "profile",           |   "web_port": 80,            |
    |                                                                                  |   "conf": {                   |   "web_user_list": [         |
    |                                                                                  |     "index": 1                |     "sysadmins"              |
    |                                                                                  |   },                          |   ]                          |
    |                                                                                  |   "raw_path": "roles/{role}", | }                            |
    |                                                                                  |   "backend_index": 1          |                              |
    |                                                                                  | }                             |                              |
    +----------------------------------------------------------------------------------+-------------------------------+------------------------------+
    env: NO_ENV
    product: httpd_server
    web_top_domain: ''
    web_app: NO_APP
    web_port: 80
    web_user_list:
    - sysadmins