python – Reloading submodules in IPython

The Question :

386 people think this question is useful

Currently I am working on a python project that contains sub modules and uses numpy/scipy. Ipython is used as interactive console. Unfortunately I am not very happy with workflow that I am using right now, I would appreciate some advice.

In IPython, the framework is loaded by a simple import command. However, it is often necessary to change code in one of the submodules of the framework. At this point a model is already loaded and I use IPython to interact with it.

Now, the framework contains many modules that depend on each other, i.e. when the framework is initially loaded the main module is importing and configuring the submodules. The changes to the code are only executed if the module is reloaded using reload(main_mod.sub_mod). This is cumbersome as I need to reload all changed modules individually using the full path. It would be very convenient if reload(main_module) would also reload all sub modules, but without reloading numpy/scipy..

The Question Comments :
  • Would you care to elaborate more on However, it is often necessary to change code in one of the submodules of the framework. So why it’s necessary to change code? Thanks
  • @eat: The framework is continually being developed, so there are constant changes to the code base.

The Answer 1

563 people think this answer is useful

IPython comes with some automatic reloading magic:

%load_ext autoreload
%autoreload 2

It will reload all changed modules every time before executing a new line. The way this works is slightly different than dreload. Some caveats apply, type %autoreload? to see what can go wrong.

If you want to always enable this settings, modify your IPython configuration file ~/.ipython/profile_default/[1] and appending:

c.InteractiveShellApp.extensions = ['autoreload']     
c.InteractiveShellApp.exec_lines = ['%autoreload 2']

Credit to @Kos via a comment below.

[1] If you don’t have the file ~/.ipython/profile_default/, you need to call ipython profile create first. Or the file may be located at $IPYTHONDIR.

The Answer 2

27 people think this answer is useful

In IPython 0.12 (and possibly earlier), you can use this:

%load_ext autoreload
%autoreload 2

This is essentially the same as the answer by pv., except that the extension has been renamed and is now loaded using %load_ext.

The Answer 3

22 people think this answer is useful

For some reason, neither %autoreload, nor dreload seem to work for the situation when you import code from one notebook to another. Only plain Python reload works:


Based on [1].

The Answer 4

18 people think this answer is useful

IPython offers dreload() to recursively reload all submodules. Personally, I prefer to use the %run() magic command (though it does not perform a deep reload, as pointed out by John Salvatier in the comments).

The Answer 5

9 people think this answer is useful

Module named importlib allow to access to import internals. Especially, it provide function importlib.reload():

import importlib

In contrary of %autoreload, importlib.reload() also reset global variables set in module. In most cases, it is what you want.

importlib is only available since Python 3.1. For older version, you have to use module imp.

The Answer 6

5 people think this answer is useful

To avoid typing those magic function again and again, they could be put in the ipython startup script(Name it with .py suffix under .ipython/profile_default/startup. All python scripts under that folder will be loaded according to lexical order), which looks like the following:

from IPython import get_ipython
ipython = get_ipython()

ipython.magic("load_ext autoreload")
ipython.magic("autoreload 2")

The Answer 7

3 people think this answer is useful

How about this:

import inspect

# needs to be primed with an empty set for loaded
def recursively_reload_all_submodules(module, loaded=None):
    for name in dir(module):
        member = getattr(module, name)
        if inspect.ismodule(member) and member not in loaded:
            recursively_reload_all_submodules(member, loaded)

import mymodule
recursively_reload_all_submodules(mymodule, set())

This should effectively reload the entire tree of modules and submodules you give it. You can also put this function in your .ipythonrc (I think) so it is loaded every time you start the interpreter.

The Answer 8

2 people think this answer is useful

Another option:

$ cat << EOF > ~/.ipython/profile_default/startup/50-autoreload.ipy
%load_ext autoreload
%autoreload 2

Verified on ipython and ipython3 v5.1.0 on Ubuntu 14.04.

The Answer 9

2 people think this answer is useful

My standard practice for reloading is to combine both methods following first opening of IPython:

from IPython.lib.deepreload import reload
%load_ext autoreload
%autoreload 2

Loading modules before doing this will cause them not to be reloaded, even with a manual reload(module_name). I still, very rarely, get inexplicable problems with class methods not reloading that I’ve not yet looked into.

The Answer 10

1 people think this answer is useful

Note that the above mentioned autoreload only works in IntelliJ if you manually save the changed file (e.g. using ctrl+s or cmd+s). It doesn’t seem to work with auto-saving.

The Answer 11

1 people think this answer is useful

I hate to add yet another answer to a long thread, but I found a solution that enables recursive reloading of submodules on %run() that others might find useful (I have anyway)

del the submodule you wish to reload on run from sys.modules in iPython:

In[1]: from sys import modules
In[2]: del modules["mymodule.mysubmodule"] # tab completion can be used like mymodule.<tab>!

Now your script will recursively reload this submodule:

In[3]: %run

The Answer 12

0 people think this answer is useful

On Jupyter Notebooks on Anaconda, doing this:

%load_ext autoreload
%autoreload 2

produced the message:

The autoreload extension is already loaded. To reload it, use: %reload_ext autoreload

It looks like it’s preferable to do:

%reload_ext autoreload
%autoreload 2

Version information:

The version of the notebook server is 5.0.0 and is running on: Python 3.6.2 |Anaconda, Inc.| (default, Sep 20 2017, 13:35:58) [MSC v.1900 32 bit (Intel)]

The Answer 13

0 people think this answer is useful

Any subobjects will not be reloaded by this, I believe you have to use IPython’s deepreload for that.

Add a Comment