Python IO¶
Input and output of Python
Pyhton & txt (general text file)¶
Basic Python function, i.e. read and write of file.
Python & Saving an object¶
standard lib: JSON and Pickle
Python & Numpy¶
Input and output of numpy array.
Python & CSV¶
Using Pandas module.
Python & MatLab¶
Using scipy's io module.
It's more like a summary instead of a lecture. You don't need to understand everything on it, only learn it when necessary.¶
Python & txt (general text file)¶
Create a new file and write something into it.
Basically we will discuss three function: open, write, read.
Ref:
Let's start from a simple example.
Now let's see what's inside in our file.
with open('./text/text.txt','r') as f:
print(f.read())
Argument 'r' means 'read mode', so you cannot change anything in the file.
Now open the file and write something into it.
with open('./text/text.txt','w') as f:
f.write('this is a test file.\n')
'w' means 'write mode' and '\n' means change to a newline. See the change of the file.
with open('./text/text.txt','r') as f:
print(f.read())
As you may notice, the former "Hello world" is replaced by new text. So what if I just want add something into the file?
You can use the 'append' mode, namely change the argument from 'w' to 'a'.
with open('./text/text.txt','a') as f:
f.write("Hello World!\n")
with open('./text/text.txt','r') as f:
print(f.read())
One may find a detailed explanation of modes here (Doc for Python2. Python3 add a "exclusive" mode, you can refer the official doc for more info)
And you may notice that each time we open a file, we start with a "with". If you do in this way, Python will automatically close the file. Otherwise you need to close the file manually.
Always do not forget close the file, it's memory consuming! You can do in this way:
f = open('./text/text.txt','a+')
f.write("Hello World!\n")
f.close()
f = open('./text/text.txt','r')
data = f.read()
print(data)
f.close()
You can also use "Print" function to write text.
with open('./text/text.txt', 'w') as f:
print('Hello World!', file=f)
with open('./text/text.txt','r') as f:
print(f.read())
String's attribute "format" is useful when you need customize your output:
Official ref here
with open('./text/text.txt','a') as f:
f.write('1,0.235,0.645,0.457\n')
f.write('{:d} {:3.7f} {:3.7f} {:3.7f}\n'.format(1,0.235,0.645,0.457))
with open('./text/text.txt','r') as f:
print(f.read())
You can see the difference between the two output.
A self-explained example is as follow:
And reference can be find: here(Chinese version) here(English version)
a = "{:.2f} {:+.2f} {:.0f} {:.2%} \n".format(3.14,3.14,3.14,3.14)
b = "{:+d} {:+d} {:-d} {:-d} \n".format(+1,-1,+1,-1)
c = "{4} {3} {2} {1} {0} \n".format(1,2,3,4,5)
d = "{0:0>6d} {0:0<6d} {0:3>6d} {0} \n".format(5)
e = "{:5d} {:5d} {:5d} {:5d} {:5d} \n".format(1,2,3,4,5)
f = "{:5d} {:<5d} {:5d} {:^5d} {:5d} \n".format(1,2,3,4,5)
print(a,b,c,d,e,f)
Now we look at the "read" function.
There are three "read" function, namely "read", "readline" and "readlines". See their output below:
with open('./text/text.txt','r') as f:
print(f.read())
with open('./text/text.txt','r') as f:
print(f.readlines())
with open('./text/text.txt','r') as f:
print(f.readline())
with open('./text/text.txt','r') as f:
for line in f:
print(line)
read() will return a string form of the content.
readlines() will return a list of each line.
readline() return only one line of the text.
The first two method will load all the text into memory at once, while the readline() will load the text one by one. If your file is very big(e.g. 4G or larger than your memory), read and readlines is not recommended.
For the big file, you may consider output in this way:
with open('./text/text.txt','r') as f:
line = f.readline()
while line:
print(line)
line = f.readline()
We just introduce the very basic read and write part of Python. But one may notice that all the discussion above based on string type. If you have a list or dict, save it to a text file is not a good choice, because you need to reconstruct list or dict from the text.
Is there any other way of saving an object?
Python & Saving an object¶
First we're going to introduce Pickle.
Official document here
Usage of Pickle is simple:
import pickle
data = ['HKUST','PHYS',6810,'Rm 4402','Lift17-18']
with open('data_pickle','wb') as f:
pickle.dump(data,f)
with open('./pickle/data_pickle','rb') as f:
data = pickle.load(f)
for i in data:
print(i)
And you can process multiple object like this:
f = open('./pickle/somedata', 'wb')
pickle.dump([1, 2, 3, 4], f)
pickle.dump('hello', f)
pickle.dump({'Apple', 'Pear', 'Banana'}, f)
f.close()
f = open('./pickle/somedata', 'rb')
print(pickle.load(f))
print(pickle.load(f))
print(pickle.load(f))
f.close()
You can see it's order preserved.
Pickle can save a function.
import math
with open('./pickle/function','wb') as f:
pickle.dump(math.cos,f)
with open('./pickle/function','rb') as f:
Cos = pickle.load(f)
print(Cos(0))
As you may notice, we add a "b" argument when we open the file. This indicate the data pickle save is a binary file, which is not human readable.
If a human readable file is needed, one may consider JSON.
JSON is short for "JavaScript Object Notation", which is a common data format.
JSON only support None,bool,int,float and str datatype, and list,tuple,dict which contain those data.
Let's see some example.
import json
data = {
'name' : 'ACME',
'shares' : 100,
'price' : 542.23
}
with open('./pickle/data.json','w') as f:
json.dump(data, f)
with open('./pickle/data.json', 'r') as f:
data = json.load(f)
for key,value in data.items():
print(key,":",value)
You can use that to save list or tuple.
More sophisticated techniques can be found at the official website
Differences between JSON and Pickle
JSON is a text serialization format (it outputs unicode text, although most of the time it is then encoded to utf-8), while pickle is a binary serialization format.
JSON is human-readable, while pickle is not.
JSON is interoperable and widely used outside of the Python ecosystem, while pickle is Python-specific.
JSON, by default, can only represent a subset of the Python built-in types, and no custom classes; pickle can represent an extremely large number of Python types (many of them automatically, by clever usage of Python’s introspection facilities; complex cases can be tackled by implementing specific object APIs).
Python & Numpy¶
NumPy module from SciPy is widely used in scientific computing, and it will also be the most important module we will learn along with the Matplotlib module.
Now let's see what can we make use of NumPy.
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
data=np.load('./numpy/data.npy')
psi6 = np.load('./numpy/psi6.npy')
plt.figure(figsize=(10,10))
ax = plt.gca()
ax.set_aspect('equal')
ax.scatter(data[:,0],data[:,1],s=5,c=psi6,cmap=plt.cm.rainbow_r)
# plt.savefig('psi6.png',dpi=200)
plt.show()
The very basic object in NumPy is called numpy.ndarray, which is a N-dimensional matrix. NumPy has already implement common matrix operation in the ndarray(i.e. dot, transpose,etc),making it really convenient for you to use.
One may find there website here. A MatLab user may find this tutorial useful.
Let's see how to save array in NumPy.
NumPy provide a lot of useful function not only aim for its array but also for other files.
(First import numpy as np)
np.save np.load
np.savez np.savez_compressed
np.savetxt np.loadtxt
One can find more info here
Save a single array in to a 'npy' file.
# create a new array
m = np.random.rand(100,2)
np.save('./numpy/m.npy',m)
print(m[0:5,:])
l = np.load('./numpy/m.npy')
print(l[0:5,:])
Note that it's not actually open a file, the load function just return a array object. So no need to close the file.
Save multiple arrays in to a 'npz' file.
np.savez('./numpy/ml.npz',x=m,y=l)
n = np.load('./numpy/ml.npz')
print(n)
print((n['x']-n['y'])[0:5,:])
Also no need to close the npz file.
Function np.savez_compressed will save the file in a compressed way. It costs less disk memory and the file can also be loaded with the load function.
The loadtxt function is vert handy when dealing with text files.
data = np.loadtxt('./numpy/loadtxt_example.txt',skiprows=2,usecols=(0,4))
plt.figure(figsize=(7,7))
ax = plt.gca()
ax.plot(data[:,0],data[:,1])
ax.set_xlabel("Timestep")
ax.set_ylabel("Total energy")
plt.show()
One may find doc for savetxt function here.
Python & CSV¶
Now let's see the basic part of the module.
This is the standard way of importing pandas.
import pandas as pd
df = pd.read_csv('./csv/uk_rain_2014.csv', header=0)
df.head(n) give you the first n rows in dataframe.
df.head(5)
df.tail(n) give you the last n rows in dataframe.
df.tail(5)
You can change the column labels by using df.columns
df.columns = ['Year','rain_octsep', 'outflow_octsep',
'rain_decfeb', 'outflow_decfeb', 'rain_junaug', 'outflow_junaug']
df.head(5)
You can change the format of data and get a statistic describe:
pd.options.display.float_format = '{:,.3f}'.format
df.describe()
And you can access the columns in two ways:
df['rain_octsep'][0:5]
df.rain_octsep[28:]
Also it's easy to implement bool operation.
df[df.rain_octsep < 1000]
df[(df.rain_octsep < 1000) & (df.outflow_octsep < 4000)]
If we need to search string in dataframe:
df[df.Year.str.startswith('199')]
Use df.loc to locate specific row and column.
df.loc[11,['Year','rain_octsep']]
Now I want to compare the rainfall of HK and UK.
hkdf = pd.read_csv('./csv/hk_weather_data.csv',header=0)
hkdf.head(5)
Combine the two dataset together and get the plot of rainfall.
First pick up the columns we need.
hk_rainfall = hkdf.loc[:,['Year','Rainfall(mm)']]
hk_rainfall = hk_rainfall[(hk_rainfall.Year<=2012)&(hk_rainfall.Year>=1980)]
hk_rainfall.columns = ['Year','rainfall']
uk_rainfall = df.loc[:,['Year','rain_octsep']]
The 'Year' column of uk_rainfall is string type, so we need to change it to integer.
def str2int(year):
year = int(year[:4])
return year
print(type(uk_rainfall.Year[0]))
uk_rainfall.Year = uk_rainfall.Year.apply(str2int)
type(uk_rainfall.Year[0])
Now we merge two dataframe together.
hk_uk_data = uk_rainfall.merge(hk_rainfall,on='Year')
hk_uk_data.head(5)
And we can get a plot easily.
hk_uk_data.plot(x='Year',y=['rain_octsep','rainfall'],figsize=(7,7))
plt.show()
Finally we save new dataframe to a csv file.
hk_uk_data.to_csv('./csv/hk_uk_rain.csv')
Python & MatLab¶
MatLab is a important tool for physics students.
The io module from scipy provide a lot of useful APIs for data reading, which include a MatLab file API. And vice versa, you can use Python API in MatLab, too.
Let's see.
One may find reference here
Mainly three function will be used:
loadmat: load MatLab format file.
savemat: save file in MatLab format.
whosmat: see what's inside a Mat file.
import scipy.io as sio
# import numpy as np
# import matplotlib.pyplot as plt
# %matplotlib inline
Make sure you import all the module.
Let's see whosmat first:
sio.whosmat('./mat/voro.mat')
Now load the mat file.
mat = sio.loadmat('./mat/voro.mat')
x = mat['x']
y = mat['y']
vx = mat['vx']
vy = mat['vy']
Get a figure using matplotlib.
plt.figure(figsize=(10,10))
plt.gca()
plt.plot(x,y,'bo',vx,vy)
plt.xlim(0,1)
plt.ylim(0,1)
plt.show()
Finaly we sava a random array into matlab format
sio.savemat('./mat/np_rand.mat',{'position':np.random.rand(100,2)})
sio.whosmat('./mat/np_rand.mat')