[Python] Useful tips when importing modules in Python

[Python] Useful tips when importing modules in Python

2019, Apr 10    

0. Index

  1. Prologue
  2. Way of loading and using other modules: import
  3. Some hidden features of ‘import’
  4. Registering our modules and module import order
  5. Epilogue


1. Prologue

We use Python and we surely import other modules everyday. Importing other modules is so common that usually we don’t pay much attention to it. I wasn’t that interested in this topic too. But I bumped into situations that I needed to know more about ‘import’ and ‘modules’. Below are some issues I encountered while using Python.

  1. Python’s math module has many useful mathematical functions. But sometimes theirs wouldn’t match your needs. What should we do when we want to define our own mathematical functions and make our ‘math’ module? When we run import math, what shall be imported, Python’s or ours?
  2. Sometimes we may need a higher rank set of modules and it’s called package in Python. Package is a bundle of relevant modules and is also a module itself. In this case, we need to carefully consider absolute import and relative import. This can be tricky.

Today’s post focuses on issue No.1. Issue No.2 is a huge area and I’ll post it later with my personal experience of constructing a package.

So today,

  1. I’ll look through basic usage of import statement. It’s a way of reusing code in Python.
  2. And I’ll introduce some interesting tricks on import. For example, I’ll answer ‘can we name our modules without .py suffix’, ‘can we use characters besides english for modules? And how about punctuation characters?’
  3. At last, I’ll cover module import order in Python and how to register our own modules in Python.

I assume readers have ‘shell’ programs in their systems.

Let’s go guys :)


2. Way of loading and using other modules: import

Code reusability is really import in programming languages and other high-level languages contain many useful others’ code in the name of library or STL. Some boring routines are essential in almost any projects, but it can be hell if we have to implement not creative, boresome jobs everytime.

Python supports others’ code in the name of ‘module’. Python has long maintained ‘batteries included’ philosophy, meaning Python supports many versatile and rich standard libraries. In fact, there’re so many standard libraries that people don’t know: ipaddress, pickle, shutil, pprint, smtp, … I cannot even mention them all because it really has a lot. With these modules you don’t need to repeat boring codes but use Python’s efficient and ready-made codes. This is why we python lovers say ‘Life is short, use Python’. You can put your energy into what is most creative and interesting to you.


To load a module, you use ‘import’ statement.

import math
import numpy

import os, sys

This is very basic use of ‘import’ statement. Codes above imported math, numpy, os, sys modules. They’re all important and famous in Python.

Sometimes you don’t need all module functions but only need one single part. Then you can import only a small part of modules with from … import … statement

from sys import getrecursionlimit

getrecursionlimit()

3000

As we know, sys modules controls Python interpreter environments. And I only wanted to check function call recursion limit in my ipython environment. So I didn’t import sys a whole but only imported getrecursionlimit from it. And we can see my ipython has recursion upper limit of 3000.


You can separate module’s attributes from module’s name.

from math import *

print(pi)
print(log2(4))
print(log10(1000))

3.141592653589793
2.0
3.0

In regular expression, * means a single character and it covers all ranges of unicode characters. In Python module importing, it has a similar meaning. Importing with * from a module can import all attributes(functions, constants) of that module. Above shows that example. I imported all attributes from math module using * and I could use pi, log2 and log10 which are constants and functions of math module without ‘math.’ prefix.

This can be used but not recommended. Here are some reasons:

  1. It adds too many attributes to global namespace. Global namespace should be maintained and controlled carefully. But asterisk importing spoils namespace hierarchy and flattens all names.
  2. Code review can be harder with * import. In most cases, we import less than 5 attributes from a module even though a module can hold a bunch of functions. If you specify names of what you use, code reviewers get to know the intention of importing the module. But the above example literally imports all so you cannot guess what I would do.


These are very basic examples of importing modules in Python. I’m sure you all know these. Now let’s dive into deeper parts of import.


3. Some hidden features of ‘import’

Chapter 2. covers simple import examples. And those used Python standard libraries like math, os and sys. Now, let_s assume when we make our own modules and import them in other projects. It’s different from just using standard modules. These can bring some issues about module names. Let’s check it out some hidden features for our customized-named modules.


3.1. Using other characters besides english for modules

As the title itself says, can we use another characters like Korean, Japanese and punctuation chars for module names? Built-in and Python standard modules’ names are all in english like math, os, sys. But… Not all people’s mother tounge is English and I’m also a Korean. For usability, it’s more recommended to use english but sometimes, we might want to use other languages to name our modules. And Python supports it.

$ echo "print('hi')" > 안녕.py


$ python 안녕.py

hi

I created the module printing ‘hi’. ‘안녕’ means ‘hi’ in Korean and I created the module printing ‘hi’ to module users. And this module is perfectly OK and executable as you see even though name is not in english characters. Actually, you can use non-english characters when assigning variables. It’s basic but some people don’t know.


Ok. We just checked we can use other languages for our module names. But how about punctuations? Can we use punctuations for module names?

from string import punctuation

print(punctuation)


!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

There are over 30 punctuation characters supported in Python. But how about these characters? We use punctuation chars often for file names in daily lives, right? Is it OK in Python modules?

$ echo "print('hi')" > greetings-to-you.py
$ python greetings-to-you.py

hi

We made a greeting module above and named it again using - punctuation chars. Well, we could execute the module even though it contains punctuation characters. But the problem happens when we import the module inside other scripts.

# A new module importing 'greetings-to-you'

import greetings-to-you

def foo(bar):
    eggs()


## !!! A SyntaxError is raised !!!
    import greetings-to-you
                    ^
SyntaxError: invalid syntax

Jesus, what’s wrong? A SyntaxError is raised. Why? It’s related to Python variable naming rules.

While using Python, I think you could have seen many underscore(_) characters in variable names. But I’m sure you would (almost) never have seen other punctuation characters.

In Python, you can’t use punctuation characters for variable names except for underscores(‘_’). That’s why you cannot import modules using other punctuation characters in Python script with import statement.


It’s too sad because we can use any chars for files in Windows and Xnix. But don’t be frustrated. There’s a detour to use punctuation chars for module names.

The way is to use __import__ built-in function to import a module.

# In Python script,

# instead of 'import greetings-to-you',
__import__('greetings-to-you')


def foo(bar):
    eggs()

hi

What happened? Rather, what is that function?

__import__ is a built-in function that gets module name in str type and imports it. Internally, when we import a module with import statement, __import__ function is called to import the module in Python.

And instead of using import statement, we can import any modules containing any characters(including whitespaces) if we give names in str type in __import__ function. It’s syntatically correct while ‘import greeting-to-you’ breaks variable naming rules of Python.


Okay. We proved that we can use non-english characters, Japanese, Korean, punctuation and even whitespaces for module names.


3.2. Do all Python module names have to end with ‘.py’?

We take it for granted that all Python modules should end with ‘.py’ suffix. Because we learn to name it that way at first time we learn Python. But let’s ask, ask for everything. Is it True?

Who knows? Let’s test it right away :)

echo "print('hi')" > greetings.py
echo "print('bye')" > farewells 
# farewells is a module without '.py' suffix.


python greetings.py
python farewells


hi
bye

Well, we could see that it doens’t matter if the script ends with ‘.py’ or not when executing the script.

However, it’s different when we import modules inside scripts.

import greetings

# It's ok.

import farewells

## !!! A ModuleNotFoundError is raised !!!
ModuleNotFoundError: No module named 'farewells'

A ModuleNotFoundError exception is raised. It’s interesting because ‘greetings.py’ and ‘farewells’ are in the same folder where Python interpreter is working but ‘farewells’ module is not found.

So, We can think that Python assumes all proper module names should end with ‘.py’ suffix. It doesn’t consider file names that do not end with ‘.py’ can be also python modules. So we should add ‘.py’ suffix to all Python modules we make. Also it’s more recommended because people can guess files containing ‘.py’ suffix would be Python modules. But we could see that naming Python modules with ‘.py’ suffix is not just recommended, but mandatory.


We found answers for 2 module naming issues in Python:

  1. Python module names can contain any characters besides english if you want. It’s possible, but not that ideal.
  2. All Python modules should have ‘.py’ suffix. Without it, you cannot import them in other Python sripts.


4. Registering our modules and module import order

This chapter is very core of this post. If we create our own scripts, we can use them as scripts and modules. Now we’re great Python users and our code is fantastic. Who knows? Maybe we can create famous frameworks like Django, Pandas, etc someday. This chapter deals with issues about importing our own modules:

  1. You can import standard modules(math, sys, keyword) wherever you are. Then how about ours? How can we make our modules importable in any places?
  2. You created ‘math’ module and it’s duplicate with Python math standard module. It’s not ideal but let’s say you have to name it that way. If we run ‘import math’ it, what would be imported, ours or Python’s?

Isn’t those interesting? Yeah, it sure is. Let’s get into it :)


4.1. Make our modules system-wide!

Sometimes you need to count the execution time of code blocks, like when you compare several algorithms. I knew that it can be used in many cases so I made a time counting function with decorator(@). Here is the code.

def calc_time(func):
    import time
    def wrapper(*args, **kwargs):
        s = time.time()
        v = func(*args, **kwargs)
        e = time.time()

        print("Total execution time for", func.__name__, "is", e-s)
        return v

    return wrapper

This is an easy example of decorator. Decorator function gets function as input and return wrapper function, which addes some actions to original functions. Decorator is really useful.

@calc_time
def print_1_to_n(n):
    for i in range(1, n+1):
        print(i)


@calc_time
def repeat_word(word, n):
    return word * n


print_1_to_n(100)
repeat_word('parkito', 1000)


# I omitted execution results of both functions.
Total execution time for print_1_to_n is 0.01787590980529785
Total execution time for repeat_word is 7.62939453125e-06

Yeah, it’s cool. I love my calc_time function and I want it to be accessible from any places like other useful modules do. But you can easily see that if the module containing that couting function is not where you import or use it, you can’t automatically use it. It means you need some actions or solutions for it. What would it be? I introduce 2 ways for it.


4.1.1. Test case definition

Let’s make a test exmple before going on. I made a simple directory structure for this case.

$ mkdir -p ~/deletemesoon/{a..c}
$ touch ~/deletemesoon/a/in_a.py
$ touch ~/deletemesoon/b/in_b.py
$ touch ~/deletemesoon/c/in_c.py

It’s a simple shell usage to make folders. I made a ‘deletemesoon’ directioy(cause I’ll remove it after posting) right inside home directory and created ‘a’, ‘b’, ‘c’ directories inside. And each directory has ‘in_*‘.py Python module. And each module has a function and we’ll use it later. The directory structure would be like this:

import example directory hierarchy

Each Python module has its unique function:

  1. in_a.py: it has calc_time function and we saw it earlier.
  2. in_b.py: In main function, we’ll import other 2 functions and use it.
  3. in_c.py: it has say_ok function and it prints out a calming phrase: ‘ok!’.

And we didn’t define say_ok function so let’s make it.

# in in_c.py

def say_ok():
    print('ok!')

And in main function in in_b.py, we’ll calculate execution time of say_ok function using calc_time function.

# in in_b.py
from in_a import calc_time
from in_c import say_ok

# Decorate it with calc_time
say_ok = calc_time(say_ok)

# main function counts execution time of say_ok.
def main():
    say_ok()

# Execute main function
main()

And inside ‘b’ folder, run in_b.py script. How was it?

ModuleNotFoundError: No module named 'in_a'

Yes, this is our problem. We want in_a and in_c module to be accessible from any directory inside the system. We’ll solve it in 2 ways.


4.1.2. Dynamically add it!

First way is ‘dynamically adding target modules’. Here’s some Python information.

As you know, Python has sys standard module and it deals with Python interpreter environment. It can control interpreter-related configurations like recursion limit of function calls, get version information of the interpreter, customize shell prompts(‘>>>’ is default) and etc.

Also sys contains a list named sys.path and it contains paths of directories that the interpreter searches for modules when you try to import modules.**

import sys

print(sys.path)


['',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/bin',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python36.zip',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python3.6',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python3.6/lib-dynload',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python3.6/site-packages',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python3.6/site-packages/IPython/extensions',
 '/home/sunghwanpark/.ipython']

It’s my result and yours would be different.

Each element in the list is a directory path and target module should be inside at least one directories here. Otherwise, ModuleNotFound exception is raised as you saw. We’ll talk about this list more in the next paragraph.

Solution here is to append paths containing our target modules to this list. Let’s add somd codes in in_b.py.

# in_b.py 

### new code
import sys

sys.path.append('/home/sunghwanpark/deletemesoon/a')
sys.path.append('/home/sunghwanpark/deletemesoon/c')
###


from in_a import calc_time
from in_c import say_ok
...

Run it! What does it say?

$ python in_b.py

ok!
Total execution time for say_ok is 6.866455078125e-05

Perfectly worked. Just before now, in_b.py couldn’t import in_a module because Python interpreter couldn’t find in_a module in all directories in sys.path. When we added paths containing in_a, in_c modules, it could find both modules and code went well. It supports only full absolute paths and others(e.g. ‘~/p/a/t/h’) are not seen as paths here.

I named this ‘dynamic way’ because we add directory paths to look through inside the script we work on.

How do you think about this way? Well… It’s ok but it’s boring to put sys.path related code everytime we use the target module. So, we need a better option.


4.1.3. Use environment variable

Systems including Xnix and Windows have environment variables and they’re used in programs as configuration options. If you have installed Python on Windows, I think you must have set environment variable configurations for Python.

I’m using Linux(Ubuntu) and it’s same too. This way is using environment variable named PYTHONPATH.

If you set PYTHONPATH variable with concatenated string of directory paths, python interpreter look for these paths when you import a module. Hoolay! Is it True? Let’s test it.

$ PYTHONPATH=/home/sunghwanpark/deletemesoon/a:/home/sunghwanpark/deletemesoon/c

Execute this code in shell and comment sys.path code block in in_b.py

PYTHONPATH is a concatenated string of paths and ‘:’ is a seperator between them. And from now on, Python interpreter will look through these directories to find target modules.

Let’s run the script

$ python in_b.py

ok!
Total execution time for say_ok is 6.4849853515625e-05

It works. You don’t need to paste additional boring codes to import your modules.

And last, this way only lasts until you quit this shell session. If you want to maintain this configuration, paste and run this code in shell.

echo 'export PYTHONPATH="your:wanted:paths:concatenated' >> ~/.bashrc

I assumed you use bash shell(default in many Xnix systems)


Ok. Now we can use our awesome and fantastic functions and classes anywhere we are in our system.


4.2. Modules with same names

We’ve come a long way. This is the final session:

If there’re modules with same names and both are detectable from Python interpreter, what would be imported? As I supposed above, if I created my ‘math’ module, and execute ‘import math’ statement, what module would be imported?

The answer lies with sys.path. Let’s call it again here.

import sys

print(sys.path)


['',
 '/home/sunghwanpark/deletemesoon/a',
 '/home/sunghwanpark/deletemesoon/c',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python36.zip',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python3.6',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python3.6/lib-dynload',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python3.6/site-packages',
 '/home/sunghwanpark/.pyenv/versions/3.6.3/lib/python3.6/site-packages/IPython/extensions',
 '/home/sunghwanpark/.ipython']

We checked that Python interpreter looks through each of this directory path to find target module.

And the order of looking through is same to element order in sys.path list. Interpreter iterates over each path in the list and if you find the module, exit the iteration and import that module.

So sys.path[0] is searched first and [1], [2], … will go on until it finds the module. If the iteration ends and couldn’t find the module, exception is raised.

At this point, We can redefine our problems.

How does sys.path list is organized? How first, second and other elements are decided?


Paths in which Python searches for modules is decided by these principles in order:

  1. Path that interpreter is working on. It means current path.
  2. Paths registered in PYTHONPATH environment variable in the system. We checked this out.
  3. Paths in which Python was installed initially. These paths contain built-in and third-party packages(Django, Pandas, etc)


We can understand sys.path module now. First element is ‘’, an empty string. It means ‘current path’ the interpreter is working on.

And the following two paths are paths we just added by setting PYTHONPATH. They’re searched second.

The rest are pre-set Python modules paths. These paths contain built-in, third-party modules.


Now we got the answer of our issue: If I have my math module, what module would be imported?

The answer is our module will be either in current path or in paths registered in PYTHONPATH variable. They both precede pre-registered paths for built-in modules. So our ‘math’ module will be imported, not built-in one.


It was a long way for import. But we finished and have grown up now.


5. Epilogue

It’s my first time posting in English. I think I need this kind of challenges to improve my English and it’s very effective. From now on, I’ll try to write posts that at least belongs to DEV in English. And also, I would really appreciate any advice on my awkard English expressions.

We import modules everyday but pay less attention to what happens when we import modules. I didn’t know this post would be so long like this and I think we have to keep looking for hidden logic in very simple and abstract functions.

Our challenge to know more never ends.

Post ended. Thank you.

Twitter Facebook Google+
# import