Input and Output

Reading a Text File

In Python, a text file can be opened and read with the following pattern:

with open("filename") as f:  # Open a file for reading
    contents = f.read()  # Read and assign the entire contents

The built-in open() function takes the optional argument mode, which can have one of the following values:

• "r": Read only (default)
• "w": Write only (will overwrite an existing file)
• "a": Append to an existing file
• "r+": Read and write

For instance, suppose a file named hamlet.txt in the working directory has the following contents:

To die, to sleep;
To sleep: perchance to dream: ay, there's the rub;
For in that sleep of death what dreams may come

The following program reads the file and prints a line every \(3\) seconds:

import time  # Built-in module
with open("hamlet.txt", mode="r") as f:
    contents = f.read().splitlines()  # List of lines
for line in contents:
    print(line)
    time.sleep(3)  # Delays for at least 3 seconds

At the end of the with block, the file is closed automatically, but the contents variable lives on.

Other methods for reading text files include readline() and readlines().

Writing a Text File

A text file can be written to a file opened in modes "w" (i.e., overwrite), "a" (i.e., append), and "r+" (i.e., read and write). Suppose one wanted to append the text “—Hamlet, act III, scene I” as a new line to hamlet.txt. The following would achieve this aim:

attribution = "\n---Hamlet, act III, scene I"
with open("hamlet.txt", mode="a"):
    f.write(attribution)

Another useful method for writing to a file is the writelines() method, which writes a list of strings to the file.

Reading

Python reads JSON types and converts them into similar Python types. The conversions are summarized in tbl. ¿tbl:json-to-python?.

Let’s say a JSON file rick.json has the JSON object describing Rick Sanchez from above. We can load it in with the following code:

with open("rick.json", "r") as f:
    data = json.load(f)

The base object is converted to a dict and assigned to the variable data. If we print data it will appear as follows:

{"grandchildren": ["Morty", "Summer"],
 "minimum age": 70,
 "name": "Rick Sanchez",
 "occupation": "Scientist"}

Here "Rick Sanchez" is a Python str, 70 is a Python int, and ["Morty", "Summer"] is a Python list of strings.

Writing

When writing a JSON file, Python converts its own types to similar JSON types. The conversions are summarized in tbl. ¿tbl:python-to-json?.

Suppose we would like to write the following data dict to a JSON file:

acceleration = 9.81
time = np.linspace(0, 1, 11)
velocity = acceleration * time
position = acceleration/2 * time ** 2
data = {
    "time": time.tolist(),
    "acceleration": acceleration,
    "velocity": velocity.tolist(),
    "position": position.tolist()
}

Note that we have used the np.ndarray method tolist() to convert the arrays to lists. This is necessary because Python cannot convert arrays directly to JSON. The following pattern will write the JSON file kinematics.json:

with open("kinematics.json", "w") as f:
    json.dump(data, f)

The JSON file can now be shared or read into another program at a later time.

Reading

Reading a CSV file data.csv to a list of lists can be achieved with the following code:

with open("data.csv", 'r') as f:
    data = list(csv.reader(f, delimiter=","))

For files with other delimeters, the delimiter argument can be used to pass the appropriate delimiter. To convert the list data to a NumPy array, the usual np.array() function can be used.

Writing

Although we usually prefer to write arrays to JSON files (subsection 3.3.3) or NumPy files (subsection 3.3.5),¹ occasionally we need to write a CSV file. The following code will write a list of lists to a CSV file:

data = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]  #  Data to save
with open("data.csv", "w") as f:
    writer = csv.writer(f, delimiter=",")
    writer.writerows(data)

To write a NumPy array A to a CSV file, first convert it to a list with A.tolist().

The .npy File

A NumPy array can be stored in a .npy file, which is a binary file, a file that is encoded and decoded differently than text files. A .npy file is usually more compact than a text file (e.g., JSON) storing the same array. Similarly, the time it takes to save and load a .npy file is usually much less than for a text file storing the same array. Furthermore, unlike for text files, no additional processing (e.g., converting to and from lists) is required for the reading and writing of .npy files. Therefore, it is often best to save and load arrays to .npy files instead of to text files. However, a text file, especially a JSON file, is the best choice for compatibility.

The following statements save a NumPy array A to a .npy file:

with open("A.npy", "wb") as f:
    np.save(f, A, allow_pickle=True)

The file was opened in "wb" or “write binary” mode; here the b is required to write binary files.

The allow_pickle argument, by default True, toggles the use of Python pickling (see subsection 3.3.6) for object arrays (i.e., NumPy arrays with nonnumeric objects). An object array cannot be saved without pickling.

The following statements load a NumPy array from a .npy file:

with open("A.npy", "rb") as f:
    A = np.load(f, allow_pickle=False)

The allow_pickle argument is by default False due to security and compatibility issues with loading pickled object arrays. Passing allow_pickle=False allows the loading of trusted object array .npy files.

The .npz File

Multiple NumPy arrays can be saved to and loaded from a .npz file. The following statements save NumPy arrays A and B to a .npz file:

with open("data.npz", "wb") as f:
    np.savez(f, A=A, B=B)

The arguments following the file f of the np.savez() do not have to be named, but if they are the name is saved in the .npz file. Otherwise, default names are used.

The following statements load NumPy arrays A and B from a .npz file:

with np.load("data.npz", allow_pickle=False) as data:
    A = data["A"]
    B = data["B"]

Here we have temporarily loaded the data and accessed each array with dictionary syntax, assigning it to variables A and B that survive the with block.

In addition to np.savez(), there is the similar np.savez_compressed() function that attempts to compress the .npz file. For certain types of data, this can reduce the file size significantly at the cost of the compression time elapsed during saving and the decompression time elapsed during loading.

Reading

A pickle file data.pickle can be loaded with the following code:

with open("data.pickle", "rb") as f:
    data = pickle.load(f)

If multiple objects were stored in the pickle file, additional calls to pickle.load() will load them in the order they were pickled. Often, we will assemble all objects to be pickled into a single object like a tuple or a dict, in which case each object can be given a name (key).

Writing

Objects foo and bar can be written to a pickle file data.pickle with the following code:

with open("data.pickle", "wb") as f:
    pickle.dump((foo, bar), f)  # Bundled into a tuple

Here we have bundled foo and bar into a single tuple so a single pickle.load() call returns all the data. Alternatively, we could have bundled them in a dict like {"foo": foo, "bar": bar), or we could have simply called pickle.dump() multiple times to save multiple objects in the same file.