How to filter dictionaries in Jinja?

Question

I have a dictionary of packages with package-name being the key and a dictionary of some details being the value:

{
        "php7.1-readline": {
            "latest": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1", 
            "origins": [
                "ppa.launchpad.net"
            ], 
            "version": "7.1.6-2~ubuntu14.04.1+deb.sury.org+1", 
            "www": "http://www.php.net/"
        }, 
        "php7.1-xml": {
            "latest": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1", 
            "origins": [
                "ppa.launchpad.net"
            ], 
            "version": "7.1.6-2~ubuntu14.04.1+deb.sury.org+1", 
            "www": "http://www.php.net/"
        }, 
        "plymouth": {
            "version": "0.8.8-0ubuntu17.1"
        },
    ....
}

I'd like to reduce the above to a dictionary with only the packages, that have the latest-attribute in their values.

It would seem like json_query is the filter to use, but I can't figure out the syntax. The examples out there all seem to operate on lists of dictionaries, not dictionaries of same...

For example, if I "pipe" the above dictionary into json_query('*.latest'), I get the list of the actual latest versions:

[
  "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
  "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
  "7.1.6-2~ubuntu14.04.1+deb.sury.org+1"
]

How can I get the entire dictionary-elements instead?

Any hope?

Lloyd

Answer 1

With dict2items filter ^{added in December 2017}, it is possible using native functionality:

- debug:
    msg: "{{ dict(pkg | dict2items | json_query('[?value.latest].[key, value.latest]')) }}"

The result:

"msg": {
    "php7.1-readline": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
    "php7.1-xml": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1"
}

Denature

dict2items supported startind from ansible 2.6 (docs.ansible.com/ansible/latest/user_guide/…) – Anergy

Answer 2

You can't perform this translation (I think) exclusively with Jinja filters, but you can get there by applying a little Ansible logic as well. The following playbook uses a with_dict loop to loop over the items in your dictionary, and build a new dictionary from matching ones:

- hosts: localhost                                                              
  vars:                                                                         
    packages: {                                                                 
        "php7.1-readline": {                                                    
          "latest": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",                     
          "origins": [                                                          
            "ppa.launchpad.net"                                                 
          ],                                                                    
          "version": "7.1.6-2~ubuntu14.04.1+deb.sury.org+1",                    
          "www": "http://www.php.net/"                                          
        },                                                                      
        "php7.1-xml": {                                                         
          "latest": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",                     
          "origins": [                                                          
            "ppa.launchpad.net"                                                 
          ],                                                                    
          "version": "7.1.6-2~ubuntu14.04.1+deb.sury.org+1",                    
          "www": "http://www.php.net/"                                          
        },                                                                      
        "plymouth": {                                                           
          "version": "0.8.8-0ubuntu17.1"                                        
        }                                                                       
      }                                                                         

  tasks:                                                                        
    - set_fact:                                                                 
        new_packages: >                                                         
          {{ new_packages|default({})|                                          
                combine({item.key: item.value}) }}                              
      with_dict: "{{ packages }}"                                               
      when: "{{ item.value.latest is defined }}"                                

    - debug:                                                                    
        var: new_packages                                                       

Fragment

Thanks, I actually tried something like this. However, the when-clause lists each package it skips -- hundreds of lines of output because, of course, I need to process all packages installed on each system, filtering those, for which updates are available... – Lloyd

So, all you care about is the resulting variable, new_packages. The number of lines of output doesn't matter. That's just informative. – Fragment

Answer 3

You are correct to link this question to https://mcmap.net/q/521253/-filter-object-by-property-and-select-with-key-in-jmespath.

There are no options to manipulate keys and values simultaneously with json_query out of the box (as of Ansible 2.4.0).

Here's patched json_query.py that supports jq-like to_entries/from_entries functions. You can put it into ./filter_plugins near your playbook and make this query:

- debug:
    msg: "{{ pkg | json_query('to_entries(@) | [?value.latest].{key:key, value:value.latest} | from_entries(@)')}}"

to get this result:

"msg": {
    "php7.1-readline": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1",
    "php7.1-xml": "7.1.9-1+ubuntu14.04.1+deb.sury.org+1"
}

I'll make PR to ansible as soon as I have some spare time.

Membranous

Thank you, but it is kind of sad... I deliberately picked the dictionary structure for the packages so as to be able to lookup a package's details by name quickly (O(ln(n))). Converting it to a list makes everything linear... As long as we are patching things anyway, can we not imagine something closer to Ansible's with_dict -- where you can operate on each item's key and value directly? – Lloyd

You a free to modify my gist example. Or write your custom plugin to process your specific dataset very fast. Or ask some Ansible guru to make it :-D – Membranous

I know, I can write a custom filter. Have, in fact. But I was hoping, it could be done without one -- with some clever combination of extract, map, selectattr, and/or json_query... Oh, well, thank you for your help again -- a negative result is still a result, is not it... – Lloyd

So I created my own little filter to emulate to_entries -- and am feeding its result into the json_query. But some of the packages in my list have .security-attribute instead of (or in addition to) .latest. Can jmespath do something like [?value.latest].{package:key, version:value.latest, security:'False'} || [?value.security].{package:key, version:value.security, security:'True'} – Lloyd