How to save a dictionary to a file? [duplicate]

Question

I have problem with changing a dict value and saving the dict to a text file (the format must be same), I only want to change the member_phone field.

My text file is the following format:

memberID:member_name:member_email:member_phone

and I split the text file with:

mdict={}
for line in file:
    x=line.split(':')
    a=x[0]
    b=x[1]
    c=x[2]
    d=x[3]
    e=b+':'+c+':'+d

    mdict[a]=e

When I try change the member_phone stored in d, the value has changed not flow by the key,

def change(mdict,b,c,d,e):
    a=input('ID')
    if a in mdict:
        d= str(input('phone'))
        mdict[a]=b+':'+c+':'+d
    else:
        print('not')

and how to save the dict to a text file with same format?

Porphyria

There seems to have originally been an attempt to ask two separate questions here; the first is about change not doing what is expected (without proper debugging details), and the second is what everyone answered. For that one there are many other versions of the question that look better asked, even if they ended up less popular. So I've gone ahead and dupehammered this. – Vanatta

Answer 1

Python has the pickle module just for this kind of thing.

These functions are all that you need for saving and loading almost any object:

import pickle 

with open('saved_dictionary.pkl', 'wb') as f:
    pickle.dump(dictionary, f)
        
with open('saved_dictionary.pkl', 'rb') as f:
    loaded_dict = pickle.load(f)

In order to save collections of Python there is the shelve module.

Wilfredowilfrid

save_obj seems to require that the file obj/'+ name + '.pkl already exist. I created a dictionary named Q, populated it, and made the call save_obj(Q, "Qtable") I got an error: FileNotFoundError: [Errno 2] No such file or directory: 'obj/Qtable.pkl' How does one create the file in the first place before writing to it? – Socialism

@ToothpickAnemone use wb+ to create file, i.e.: with open('obj/'+ name + '.pkl', 'wb+') – Emit

@andrey.s: I don't think what you say will make any difference because it doesn't address the problem. – Nash

@Toothpick Anemone: Adding a + to the mode will have no affect on your problem (andrey.s is incorrect). Your problem sounds like it's because of one or two things. For the save_obj() in this answer to work, a subdirectory named "obj" must already exist because open() won't create one automatically. Second the first argument to save_obj() is the Python object to be saved, not the name of the subdirectory (although it's not entirely clear what you meant by the Q in the example save_obj(Q, "Qtable") call). You can create a directory if it doesn't already exits with os.mkdir(). – Nash

is there a reason the 'obj/' string is even added here? In general the default behavior for saving anything is to save in the current directory or you give it the full directory to save there. I just removed it and it is working as expected! – Pyromorphite

Answer 2

Pickle is probably the best option, but in case anyone wonders how to save and load a dictionary to a file using NumPy:

import numpy as np

# Save
dictionary = {'hello':'world'}
np.save('my_file.npy', dictionary) 

# Load
read_dictionary = np.load('my_file.npy',allow_pickle='TRUE').item()
print(read_dictionary['hello']) # displays "world"

FYI: NPY file viewer

Algol

what's the .item() for? – Schism

Why is this answer less upvoted than @Wilfredowilfrid 's "pickle" answer? More space-complex? – Sedation

@frank possibility because pickle is part of python's standard library where numpy is an independent project. – Farsighted

@Schism from what I looked up, np.load returns an ndarray (doing a type(read_dictionary) reveals so) and .item() basically converts that element to a python scalar object which is a dictionary as stated here – Plastid

I got an error only trying to import NumPy: "import numpy as np ImportError: No module named numpy" – Merrie

@Shai have you installed the numpy package...? – Terpineol

Why is allow_pickle's value a string, 'TRUE'? The docs say it's a boolean. – Varitype

If I'm not mistaken, after saving in .npy format, you can't view data with a simple text editor. That's a big drawback of this method compared to saving as json. – Considered

From this code I got the error "ValueError: can only convert an array of size 1 to a Python scalar". It was sorted with changing the code to: read_dictionary = np.load('my_file.npy',allow_pickle='TRUE').tolist() – Vezza

Answer 3

We can also use the json module in the case when dictionaries or some other data can be easily mapped to JSON format.

import json

# Serialize data into file:
json.dump( data, open( "file_name.json", 'w' ) )

# Read data from file:
data = json.load( open( "file_name.json" ) )

This solution brings many benefits, eg works for Python 2.x and Python 3.x in an unchanged form and in addition, data saved in JSON format can be easily transferred between many different platforms or programs. This data are also human-readable.

Olindaolinde

Good approach, but I don't think it is safe to use open directly instead of a context created by with. Is there a gurantee, the file handle will be closed if json.dump fails? – Rollo

For a nice looking, human readable .json, do the following: with open('file_name.json', 'w', encoding='utf-8') as f: json.dump(data, f, ensure_ascii=False, indent=4) – Pulver

Answer 4

As Pickle has some security concerns and is slow (source), I would go for JSON, as it is fast, built-in, human-readable, and interchangeable:

import json
data = {'another_dict': {'a': 0, 'b': 1}, 'a_list': [0, 1, 2, 3]}
# e.g. file = './data.json' 
with open(file, 'w') as f: 
    json.dump(data, f)

Reading is similar easy:

with open(file, 'r') as f:
    data = json.load(f)

This is similar to this answer, but implements the file handling correctly.

If the performance improvement is still not enough, I highly recommend orjson, a fast, correct JSON library for Python built upon Rust.

For those focussing on efficient storage over run-time performance, I recommend using compression, for example bz2:

import json 
import bz2

byte_data = json.dumps(data)
compressed_byte_data = bz2.compress(byte_data)
with open(file_path, 'wb') as file_handle:
    file_handle.write(compressed_byte_data)

Duroc

Thanks for implementing the file handling correctly! – Wilhelminawilhelmine

When reading a .json file which is stored as key-value pair, the default data structure json.load returns will be a dict ? – Eggnog

Answer 5

Save and load dict to file:

def save_dict_to_file(dic):
    f = open('dict.txt','w')
    f.write(str(dic))
    f.close()

def load_dict_from_file():
    f = open('dict.txt','r')
    data=f.read()
    f.close()
    return eval(data)

Biotite

Answer 6

I'm not sure what your first question is, but if you want to save a dictionary to file you should use the json library. Look up the documentation of the loads and puts functions.

Burnley

Why json? It is even easier to just dump the Python dictionary to a file using "repr" – Reade

@Reade but parsing it back isn't as easy. Plus json is easy to edit by hand and import into any other program. – Furan

I like John's suggestion - see this post for a good and simple example https://mcmap.net/q/110830/-writing-a-dict-to-txt-file-and-reading-it-back-duplicate – Illustrious

json is also better for source version control. – Riser

Answer 7

I would suggest saving your data using the JSON format instead of pickle format as JSON's files are human-readable which makes your debugging easier since your data is small. JSON files are also used by other programs to read and write data. You can read more about it here

You'll need to install the JSON module, you can do so with pip:

pip install json


# To save the dictionary into a file:
json.dump( data, open( "myfile.json", 'w' ) )

This creates a json file with the name myfile.

# To read data from file:
data = json.load( open( "myfile.json" ) )

This reads and stores the myfile.json data in a data object.

Phthalein

Answer 8

For a dictionary of strings such as the one you're dealing with, it could be done using only Python's built-in text processing capabilities.

(Note this wouldn't work if the values are something else.)

with open('members.txt') as file:
    mdict={}
    for line in file:
        a, b, c, d = line.strip().split(':')
        mdict[a] = b + ':' + c + ':' + d

a = input('ID: ')
if a not in mdict:
    print('ID {} not found'.format(a))
else:
    b, c, d = mdict[a].split(':')
    d = input('phone: ')
    mdict[a] = b + ':' + c + ':' + d  # update entry
    with open('members.txt', 'w') as file:  # rewrite file
        for id, values in mdict.items():
            file.write(':'.join([id] + values.split(':')) + '\n')

Nash

This doesn't work on dictionaries: file.write(':'.join([id] + values.split(':')) + '\n') AttributeError: 'dict' object has no attribute 'split' – Merrie

@ShaiAlon: Not with all of them, no. It definitely does work with those whose values are strings (that have a split() method)—which is the topic of this question. It sounds like you're trying to use it on a dictionary of dictionaries, so no it wouldn't work with those. I also don't appreciate down-votes from folks because they don't really understand the question (and thus the context of the answers being provided). I'll update my answer to make when using it would be appropriate, so please undo your down-vote. – Nash

Answer 9

I like using the pretty print module to store the dict in a very user-friendly readable form:

import pprint

def store_dict(fname, dic):
    with open(fname, "w") as f:
        f.write(pprint.pformat(dic, indent=4, sort_dicts=False))
        # note some of the defaults are: indent=1, sort_dicts=True

Then, when recovering, read in the text file and eval() it to turn the string back into a dict:

def load_file(fname):
    try:
        with open(fname, "r") as f:
            dic = eval(f.read())
    except:
        dic = {}
    return dic

Phuongphycology

Answer 10

Unless you really want to keep the dictionary, I think the best solution is to use the csv Python module to read the file. Then, you get rows of data and you can change member_phone or whatever you want ; finally, you can use the csv module again to save the file in the same format as you opened it.

Code for reading:

import csv

with open("my_input_file.txt", "r") as f:
   reader = csv.reader(f, delimiter=":")
   lines = list(reader)

Code for writing:

with open("my_output_file.txt", "w") as f:
   writer = csv.writer(f, delimiter=":")
   writer.writerows(lines)

Of course, you need to adapt your change() function:

def change(lines):
    a = input('ID')
    for line in lines:
      if line[0] == a:
        d=str(input("phone"))
        line[3]=d
        break
    else:
      print "not"

Reade

what is meant by Of course, you need to adapt your change() function: – Hypotrachelium

in the question, a dict was used whereas csv works more like a list – Reade

Answer 11

I haven't timed it but I bet h5 is faster than pickle; the filesize with compression is almost certainly smaller.

import deepdish as dd
dd.io.save(filename, {'dict1': dict1, 'dict2': dict2}, compression=('blosc', 9))

Polity

Answer 12

file_name = open("data.json", "w")
json.dump(test_response, file_name)
file_name.close()

or use context manager, which is better:

with open("data.json", "w") as file_name:
    json.dump(test_response, file_name)

European