Extract subset of key-value pairs from Python dictionary object?

The Question :

351 people think this question is useful

I have a big dictionary object that has several key value pairs (about 16), but I am only interested in 3 of them. What is the best way (shortest/efficient/most elegant) to achieve that?

The best I know is:

bigdict = {'a':1,'b':2,....,'z':26} 
subdict = {'l':bigdict['l'], 'm':bigdict['m'], 'n':bigdict['n']}

I am sure there is a more elegant way than this. Ideas?

The Question Comments :

The Answer 1

474 people think this answer is useful

You could try:

dict((k, bigdict[k]) for k in ('l', 'm', 'n'))

… or in Python 3 Python versions 2.7 or later (thanks to Fábio Diniz for pointing that out that it works in 2.7 too):

{k: bigdict[k] for k in ('l', 'm', 'n')}

Update: As Håvard S points out, I’m assuming that you know the keys are going to be in the dictionary – see his answer if you aren’t able to make that assumption. Alternatively, as timbo points out in the comments, if you want a key that’s missing in bigdict to map to None, you can do:

{k: bigdict.get(k, None) for k in ('l', 'm', 'n')}

If you’re using Python 3, and you only want keys in the new dict that actually exist in the original one, you can use the fact to view objects implement some set operations:

{k: bigdict[k] for k in bigdict.keys() & {'l', 'm', 'n'}}

The Answer 2

127 people think this answer is useful

A bit shorter, at least:

wanted_keys = ['l', 'm', 'n'] # The keys you want
dict((k, bigdict[k]) for k in wanted_keys if k in bigdict)

The Answer 3

27 people think this answer is useful
interesting_keys = ('l', 'm', 'n')
subdict = {x: bigdict[x] for x in interesting_keys if x in bigdict}

The Answer 4

23 people think this answer is useful

A bit of speed comparison for all mentioned methods:

UPDATED on 2020.07.13 (thx to @user3780389): ONLY for keys from bigdict.

 IPython 5.5.0 -- An enhanced Interactive Python.
Python 2.7.18 (default, Aug  8 2019, 00:00:00) 
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
import numpy.random as nprnd
  ...: keys = nprnd.randint(100000, size=10000)
  ...: bigdict = dict([(_, nprnd.rand()) for _ in range(100000)])
  ...: 
  ...: %timeit {key:bigdict[key] for key in keys}
  ...: %timeit dict((key, bigdict[key]) for key in keys)
  ...: %timeit dict(map(lambda k: (k, bigdict[k]), keys))
  ...: %timeit {key:bigdict[key] for key in set(keys) & set(bigdict.keys())}
  ...: %timeit dict(filter(lambda i:i[0] in keys, bigdict.items()))
  ...: %timeit {key:value for key, value in bigdict.items() if key in keys}
100 loops, best of 3: 2.36 ms per loop
100 loops, best of 3: 2.87 ms per loop
100 loops, best of 3: 3.65 ms per loop
100 loops, best of 3: 7.14 ms per loop
1 loop, best of 3: 577 ms per loop
1 loop, best of 3: 563 ms per loop

As it was expected: dictionary comprehensions are the best option.

The Answer 5

15 people think this answer is useful

This answer uses a dictionary comprehension similar to the selected answer, but will not except on a missing item.

python 2 version:

{k:v for k, v in bigDict.iteritems() if k in ('l', 'm', 'n')}

python 3 version:

{k:v for k, v in bigDict.items() if k in ('l', 'm', 'n')}

The Answer 6

6 people think this answer is useful

Maybe:

subdict=dict([(x,bigdict[x]) for x in ['l', 'm', 'n']])

Python 3 even supports the following:

subdict={a:bigdict[a] for a in ['l','m','n']}

Note that you can check for existence in dictionary as follows:

subdict=dict([(x,bigdict[x]) for x in ['l', 'm', 'n'] if x in bigdict])

resp. for python 3

subdict={a:bigdict[a] for a in ['l','m','n'] if a in bigdict}

The Answer 7

5 people think this answer is useful

You can also use map (which is a very useful function to get to know anyway):

sd = dict(map(lambda k: (k, l.get(k, None)), l))

Example:

large_dictionary = {'a1':123, 'a2':45, 'a3':344}
list_of_keys = ['a1', 'a3']
small_dictionary = dict(map(lambda key: (key, large_dictionary.get(key, None)), list_of_keys))

PS: I borrowed the .get(key, None) from a previous answer 🙂

The Answer 8

3 people think this answer is useful

Okay, this is something that has bothered me a few times, so thank you Jayesh for asking it.

The answers above seem like as good a solution as any, but if you are using this all over your code, it makes sense to wrap the functionality IMHO. Also, there are two possible use cases here: one where you care about whether all keywords are in the original dictionary. and one where you don’t. It would be nice to treat both equally.

So, for my two-penneth worth, I suggest writing a sub-class of dictionary, e.g.

class my_dict(dict):
    def subdict(self, keywords, fragile=False):
        d = {}
        for k in keywords:
            try:
                d[k] = self[k]
            except KeyError:
                if fragile:
                    raise
        return d

Now you can pull out a sub-dictionary with

orig_dict.subdict(keywords)

Usage examples:

#
## our keywords are letters of the alphabet
keywords = 'abcdefghijklmnopqrstuvwxyz'
#
## our dictionary maps letters to their index
d = my_dict([(k,i) for i,k in enumerate(keywords)])
print('Original dictionary:\n%r\n\n' % (d,))
#
## constructing a sub-dictionary with good keywords
oddkeywords = keywords[::2]
subd = d.subdict(oddkeywords)
print('Dictionary from odd numbered keys:\n%r\n\n' % (subd,))
#
## constructing a sub-dictionary with mixture of good and bad keywords
somebadkeywords = keywords[1::2] + 'A'
try:
    subd2 = d.subdict(somebadkeywords)
    print("We shouldn't see this message")
except KeyError:
    print("subd2 construction fails:")
    print("\toriginal dictionary doesn't contain some keys\n\n")
#
## Trying again with fragile set to false
try:
    subd3 = d.subdict(somebadkeywords, fragile=False)
    print('Dictionary constructed using some bad keys:\n%r\n\n' % (subd3,))
except KeyError:
    print("We shouldn't see this message")

If you run all the above code, you should see (something like) the following output (sorry for the formatting):

Original dictionary:
{‘a’: 0, ‘c’: 2, ‘b’: 1, ‘e’: 4, ‘d’: 3, ‘g’: 6, ‘f’: 5, ‘i’: 8, ‘h’: 7, ‘k’: 10, ‘j’: 9, ‘m’: 12, ‘l’: 11, ‘o’: 14, ‘n’: 13, ‘q’: 16, ‘p’: 15, ‘s’: 18, ‘r’: 17, ‘u’: 20, ‘t’: 19, ‘w’: 22, ‘v’: 21, ‘y’: 24, ‘x’: 23, ‘z’: 25}

Dictionary from odd numbered keys:
{‘a’: 0, ‘c’: 2, ‘e’: 4, ‘g’: 6, ‘i’: 8, ‘k’: 10, ‘m’: 12, ‘o’: 14, ‘q’: 16, ‘s’: 18, ‘u’: 20, ‘w’: 22, ‘y’: 24}

subd2 construction fails:
original dictionary doesn’t contain some keys

Dictionary constructed using some bad keys:
{‘b’: 1, ‘d’: 3, ‘f’: 5, ‘h’: 7, ‘j’: 9, ‘l’: 11, ‘n’: 13, ‘p’: 15, ‘r’: 17, ‘t’: 19, ‘v’: 21, ‘x’: 23, ‘z’: 25}

The Answer 9

2 people think this answer is useful

Yet another one (I prefer Mark Longair’s answer)

di = {'a':1,'b':2,'c':3}
req = ['a','c','w']
dict([i for i in di.iteritems() if i[0] in di and i[0] in req])

The Answer 10

2 people think this answer is useful

solution

from operator import itemgetter
from typing import List, Dict, Union


def subdict(d: Union[Dict, List], columns: List[str]) -> Union[Dict, List[Dict]]:
    """Return a dict or list of dicts with subset of 
    columns from the d argument.
    """
    getter = itemgetter(*columns)

    if isinstance(d, list):
        result = []
        for subset in map(getter, d):
            record = dict(zip(columns, subset))
            result.append(record)
        return result
    elif isinstance(d, dict):
        return dict(zip(columns, getter(d)))

    raise ValueError('Unsupported type for `d`')

examples of use

# pure dict

d = dict(a=1, b=2, c=3)
print(subdict(d, ['a', 'c']))

>>> In [5]: {'a': 1, 'c': 3}

# list of dicts

d = [
    dict(a=1, b=2, c=3),
    dict(a=2, b=4, c=6),
    dict(a=4, b=8, c=12),
]

print(subdict(d, ['a', 'c']))

>>> In [5]: [{'a': 1, 'c': 3}, {'a': 2, 'c': 6}, {'a': 4, 'c': 12}]

The Answer 11

0 people think this answer is useful

Using map (halfdanrump’s answer) is best for me, though haven’t timed it…

But if you go for a dictionary, and if you have a big_dict:

  1. Make absolutely certain you loop through the the req. This is crucial, and affects the running time of the algorithm (big O, theta, you name it)
  2. Write it generic enough to avoid errors if keys are not there.

so e.g.:

big_dict = {'a':1,'b':2,'c':3,................................................}
req = ['a','c','w']

{k:big_dict.get(k,None) for k in req )
# or 
{k:big_dict[k] for k in req if k in big_dict)

Note that in the converse case, that the req is big, but my_dict is small, you should loop through my_dict instead.

In general, we are doing an intersection and the complexity of the problem is O(min(len(dict)),min(len(req))). Python’s own implementation of intersection considers the size of the two sets, so it seems optimal. Also, being in c and part of the core library, is probably faster than most not optimized python statements. Therefore, a solution that I would consider is:

dict = {'a':1,'b':2,'c':3,................................................}
req = ['a','c','w',...................]

{k:dic[k] for k in set(req).intersection(dict.keys())}

It moves the critical operation inside python’s c code and will work for all cases.

Add a Comment