Source From Here
Keyboard Shortcuts in the IPython Shell
If you spend any amount of time on the computer, you’ve probably found a use for keyboard shortcuts in your workflow. Most familiar perhaps are Cmd-C and Cmd-V (or Ctrl-C and Ctrl-V) for copying and pasting in a wide variety of programs and systems. Power users tend to go even further: popular text editors like Emacs, Vim, and others provide users an incredible range of operations through intricate combinations of keystrokes. The IPython shell doesn’t go this far, but does provide a number of keyboard shortcuts for fast navigation while you’re typing commands. These shortcuts are not in fact provided by IPython itself, but through its dependency on the GNU Readline library: thus, some of the following shortcuts may differ depending on your system configuration. Also, while some of these shortcuts do work in the browser-based notebook, this section is primarily about shortcuts in the IPython shell.
Once you get accustomed to these, they can be very useful for quickly performing certain commands without moving your hands from the “home” keyboard position. If you’re an Emacs user or if you have experience with Linux-style shells, the following will be very familiar. We’ll group these shortcuts into a few categories: navigation shortcuts, text entry shortcuts, command history shortcuts, and miscellaneous shortcuts.
Navigation Shortcuts
While the use of the left and right arrow keys to move backward and forward in the line is quite obvious, there are other options that don’t require moving your hands from the “home” keyboard position:
Text Entry Shortcuts
While everyone is familiar with using the Backspace key to delete the previous character, reaching for the key often requires some minor finger gymnastics, and it only deletes a single character at a time. In IPython there are several shortcuts for removing some portion of the text you’re typing. The most immediately useful of these are the commands to delete entire lines of text. You’ll know these have become second nature if you find yourself using a combination of Ctrl-b and Ctrl-d instead of reaching for the Backspace key to delete the previous character!
Command History Shortcuts
Perhaps the most impactful shortcuts discussed here are the ones IPython provides for navigating the command history. This command history goes beyond your current IPython session: your entire command history is stored in a SQLite database in your IPython profile directory. The most straightforward way to access these is with the up and down arrow keys to step through the history, but other options exist as well:
Miscellaneous Shortcuts
Finally, there are a few miscellaneous shortcuts that don’t fit into any of the preceding categories, but are nevertheless useful to know:
The Ctrl-c shortcut in particular can be useful when you inadvertently start a very long-running job. While some of the shortcuts discussed here may seem a bit tedious at first, they quickly become automatic with practice. Once you develop that muscle memory, I suspect you will even find yourself wishing they were available in other contexts.
IPython Magic Commands
The previous section(s) showed how IPython lets you use and explore Python efficiently and interactively. Here we’ll begin discussing some of the enhancements that IPython adds on top of the normal Python syntax. These are known in IPython as magic commands, and are prefixed by the % character. These magic commands are designed to succinctly solve various common problems in standard data analysis.Magic commands come in two flavors: line magics, which are denoted by a single % prefix and operate on a single line of input, and cell magics, which are denoted by a double %% prefix and operate on multiple lines of input. We’ll demonstrate and discuss a few brief examples here, and come back to more focused discussion of several useful magic commands later in the chapter.
Pasting Code Blocks: %paste and %cpaste
When you’re working in the IPython interpreter, one common gotcha is that pasting multiline code blocks can lead to unexpected errors, especially when indentation and interpreter markers are involved. A common case is that you find some example code on a website and want to paste it into your interpreter. Consider the following simple function:
- >>> def donothing(x):
- ... return x
In the direct paste, the interpreter is confused by the additional prompt characters. But never fear—IPython’s %paste magic function is designed to handle this exact type of multiline, marked-up input:
A command with a similar intent is %cpaste, which opens up an interactive multiline prompt in which you can paste one or more chunks of code to be executed in a batch:
These magic commands, like others we’ll see, make available functionality that would be difficult or impossible in a standard Python interpreter.
Running External Code: %run
As you begin developing more extensive code, you will likely find yourself working in both IPython for interactive exploration, as well as a text editor to store code that you want to reuse. Rather than running this code in a new window, it can be convenient to run it within your IPython session. This can be done with the %run magic.
For example, imagine you’ve created a myscript.py file with the following contents:
- myscript.py
- def square(x):
- """square a number"""
- return x ** 2
- for N in range(1, 4):
- print(N, "squared is", square(N))
Note also that after you’ve run this script, any functions defined within it are available for use in your IPython session:
There are several options to fine-tune how your code is run; you can see the documentation in the normal way, by typing %run? in the IPython interpreter.
Timing Code Execution: %timeit
Another example of a useful magic function is %timeit, which will automatically determine the execution time of the single-line Python statement that follows it. For example, we may want to check the performance of a list comprehension:
The benefit of %timeit is that for short commands it will automatically perform multiple runs in order to attain more robust results. For multiline statements, adding a second % sign will turn this into a cell magic that can handle multiple lines of input. For example, here’s the equivalent construction with a for loop:
We can immediately see that list comprehensions are about 10% faster than the equivalent for loop construction in this case. We’ll explore %timeit and other approaches to timing and profiling code later.
Help on Magic Functions: ?, %magic, and %lsmagic
Like normal Python functions, IPython magic functions have docstrings, and this useful documentation can be accessed in the standard manner. So, for example, to read the documentation of the %timeit magic, simply type this:
Documentation for other functions can be accessed similarly. To access a general description of available magic functions, including some examples, you can type this:
For a quick and simple list of all available magic functions, type this:
Finally, I’ll mention that it is quite straightforward to define your own magic functions if you wish. We won’t discuss it here, but if you are interested, see the references listed in “More IPython Resources”.
Input and Output History
Previously we saw that the IPython shell allows you to access previous commands with the up and down arrow keys, or equivalently the Ctrl-p/Ctrl-n shortcuts. Additionally, in both the shell and the notebook,IPython exposes several ways to obtain the output of previous commands, as well as string versions of the commands themselves. We’ll explore those here.
IPython’s In and Out Objects
By now I imagine you’re quite familiar with the In[1]:/Out[1]: style prompts used by IPython. But it turns out that these are not just pretty decoration: they give a clue as to how you can access previous inputs and outputs in your current session. Imagine you start a session that looks like this:
We’ve imported the built-in math package, then computed the sine and the cosine of the number 2. These inputs and outputs are displayed in the shell with In/Out labels, but there’s more—IPython actually creates some Python variables called In and Out that are automatically updated to reflect this history:
The In object is a list, which keeps track of the commands in order (the first item in the list is a placeholder so that In[1] can refer to the first command):
The Out object is not a list but a dictionary mapping input numbers to their outputs (if any):
Note that not all operations have outputs: for example, import statements and print statements don’t affect the output. The latter may be surprising, but makes sense if you consider that print is a function that returns None; for brevity, any command that returns None is not added to Out. Where this can be useful is if you want to interact with past results. For example, let’s check the sum of sin(2) ** 2 and cos(2) ** 2 using the previously computed results:
The result is 1.0 as we’d expect from the well-known trigonometric identity. In this case, using these previous results probably is not necessary, but it can become very handy if you execute a very expensive computation and want to reuse the result!
Underscore Shortcuts and Previous Outputs
The standard Python shell contains just one simple shortcut for accessing previous output; the variable _ (i.e., a single underscore) is kept updated with the previous output; this works in IPython as well:
But IPython takes this a bit further—you can use a double underscore to access the second-to-last output, and a triple underscore to access the third-to-last output (skipping any commands with no output):
IPython stops there: more than three underscores starts to get a bit hard to count, and at that point it’s easier to refer to the output by line number. There is one more shortcut we should mention, however—a shorthand for Out[X] is
_X (i.e., a single underscore followed by the line number):
Suppressing Output
Sometimes you might wish to suppress the output of a statement (this is perhaps most common with the plotting commands that we’ll explore in Chapter 4). Or maybe the command you’re executing produces a result that you’d prefer not to store in your output history, perhaps so that it can be deallocated when other references are removed. The easiest way to suppress the output of a command is to add a semicolon to the end of the line:
Note that the result is computed silently, and the output is neither displayed on the screen or stored in the Out dictionary:
Related Magic Commands
For accessing a batch of previous inputs at once, the %history magic command is very helpful. Here is how you can print the first four inputs:
As usual, you can type %history? for more information and a description of options available. Other similar magic commands are %rerun (which will re-execute some portion of the command history) and %save(which saves some set of the command history to a file).
IPython and Shell Commands
When working interactively with the standard Python interpreter, one of the frustrations you’ll face is the need to switch between multiple windows to access Python tools and system command-line tools. IPython bridges this gap, and gives you a syntax for executing shell commands directly from within the IPython terminal. The magic happens with the exclamation point: anything appearing after ! on a line will be executed not by the Python kernel, but by the system command line.
The following assumes you’re on a Unix-like system, such as Linux or Mac OS X. Some of the examples that follow will fail on Windows, which uses a different type of shell by default (though with the 2016 announcement of native Bash shells on Windows, soon this may no longer be an issue!). If you’re unfamiliar with shell commands, I’d suggest reviewing the Shell Tutorial put together by the always excellent Software Carpentry Foundation.
Quick Introduction to the Shell
A full intro to using the shell/terminal/command line is well beyond the scope of this chapter, but for the uninitiated we will offer a quick introduction here. The shell is a way to interact textually with your computer. Ever since the mid-1980s, when Microsoft and Apple introduced the first versions of their now ubiquitous graphical operating systems, most computer users have interacted with their operating system through familiar clicking of menus and drag-and-drop movements. But operating systems existed long before these graphical user interfaces, and were primarily controlled through sequences of text input: at the prompt, the user would type a command, and the computer would do what the user told it to. Those early prompt systems are the precursors of the shells and terminals that most active data scientists still use today.
Someone unfamiliar with the shell might ask why you would bother with this, when you can accomplish many results by simply clicking on icons and menus. A shell user might reply with another question: why hunt icons and click menus when you can accomplish things much more easily by typing? While it might sound like a typical tech preference impasse, when moving beyond basic tasks it quickly becomes clear that the shell offers much more control of advanced tasks, though admittedly the learning curve can intimidate the average computer user.
Shell Commands in IPython
You can use any command that works at the command line in IPython by prefixing it with the ! character. For example, the ls, pwd, and echo commands can be run as follows:
Passing Values to and from the Shell
Shell commands can not only be called from IPython, but can also be made to interact with the IPython namespace. For example, you can save the output of any shell command to a Python list using the assignment operator:
Note that these results are not returned as lists, but as a special shell return type defined in IPython:
This looks and acts a lot like a Python list, but has additional functionality, such as the grep and fields methods and the s, n, and p properties that allow you to search, filter, and display the results in convenient ways. For more information on these, you can use IPython’s built-in help features. Communication in the other direction—passing Python variables into the shell—is possible through the {varname} syntax:
The curly braces contain the variable name, which is replaced by the variable’s contents in the shell command.
Shell-Related Magic Commands
If you play with IPython’s shell commands for a while, you might notice that you cannot use !cd to navigate the filesystem:
The reason is that shell commands in the notebook are executed in a temporary subshell. If you’d like to change the working directory in a more enduring way, you can use the %cd magic command:
In fact, by default you can even use this without the % sign:
This is known as an automagic function, and this behavior can be toggled with the %automagic magic function. Besides %cd, other available shell-like magic functions are %cat, %cp, %env, %ls, %man, %mkdir, %more, %mv, %pwd, %rm, and %rmdir, any of which can be used without the % sign if automagic is on. This makes it so that you can almost treat the IPython prompt as if it’s a normal shell:
This access to the shell from within the same terminal window as your Python session means that there is a lot less switching back and forth between interpreter and shell as you write your Python code.
Errors and Debugging
Code development and data analysis always require a bit of trial and error, and IPython contains tools to streamline this process. This section will briefly cover some options for controlling Python’s exception reporting, followed by exploring tools for debugging errors in code.
Controlling Exceptions: %xmode
Most of the time when a Python script fails, it will raise an exception. When the interpreter hits one of these exceptions, information about the cause of the error can be found in the traceback, which can be accessed from within Python. With the %xmode magic function, IPython allows you to control the amount of information printed when the exception is raised. Consider the following code:
Calling func2 results in an error, and reading the printed trace lets us see exactly what happened. By default, this trace includes several lines showing the context of each step that led to the error. Using the %xmodemagic function (short for exception mode), we can change what information is printed. %xmode takes a single argument, the mode, and there are three possibilities: Plain, Context, and Verbose. The default is Context, and gives output like that just shown. Plain is more compact and gives less information:
The Verbose mode adds some extra information, including the arguments to any functions that are called:
This extra information can help you narrow in on why the exception is being raised. So why not use the Verbose mode all the time? As code gets complicated, this kind of traceback can get extremely long. Depending on the context, sometimes the brevity of Default mode is easier to work with.
Debugging: When Reading Tracebacks Is Not Enough
The standard Python tool for interactive debugging is pdb, the Python debugger. This debugger lets the user step through the code line by line in order to see what might be causing a more difficult error. The IPython-enhanced version of this is ipdb, the IPython debugger. There are many ways to launch and use both these debuggers; we won’t cover them fully here. Refer to the online documentation of these two utilities to learn more.
In IPython, perhaps the most convenient interface to debugging is the %debug magic command. If you call it after hitting an exception, it will automatically open an interactive debugging prompt at the point of the exception. The ipdb prompt lets you explore the current state of the stack, explore the available variables, and even run Python commands!
Let’s look at the most recent exception, then do some basic tasks—print the values of a and b, and type quit to quit the debugging session:
The interactive debugger allows much more than this, though—we can even step up and down through the stack and explore the values of variables there:
This allows you to quickly find out not only what caused the error, but also what function calls led up to the error. If you’d like the debugger to launch automatically whenever an exception is raised, you can use the %pdb magic function to turn on this automatic behavior:
Finally, if you have a script that you’d like to run from the beginning in interactive mode, you can run it with the command %run -d, and use the next command to step through the lines of code interactively.
Partial list of debugging commands
There are many more available commands for interactive debugging than we’ve listed here; the following table contains a description of some of the more common and useful ones:
For more information, use the help command in the debugger, or take a look at ipdb’s online documentation.
Profiling and Timing Code
In the process of developing code and creating data processing pipelines, there are often trade-offs you can make between various implementations. Early in developing your algorithm, it can be counterproductive to worry about such things. As Donald Knuth famously quipped, “We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.”
But once you have your code working, it can be useful to dig into its efficiency a bit. Sometimes it’s useful to check the execution time of a given command or set of commands; other times it’s useful to dig into a multiline process and determine where the bottleneck lies in some complicated series of operations. IPython provides access to a wide array of functionality for this kind of timing and profiling of code. Here we’ll discuss the following IPython magic commands:
The last four commands are not bundled with IPython—you’ll need to install the line_profiler and memory_profiler extensions, which we will discuss in the following sections.
Timing Code Snippets: %timeit and %time
We saw the %timeit line magic and %%timeit cell magic used before. %%timeit can be used to time the repeated execution of snippets of code:
Note that because this operation is so fast, %timeit automatically does a large number of repetitions. For slower commands, %timeit will automatically adjust and perform fewer repetitions:
Sometimes repeating an operation is not the best option. For example, if we have a list that we’d like to sort, we might be misled by a repeated operation. Sorting a presorted list is much faster than sorting an unsorted list, so the repetition will skew the result:
For this, the %time magic function may be a better choice. It also is a good choice for longer-running commands, when short, system-related delays are unlikely to affect the result. Let’s time the sorting of an unsorted and a presorted list:
Notice how much faster the presorted list is to sort, but notice also how much longer the timing takes with %time versus %timeit, even for the presorted list! This is a result of the fact that %timeit does some clever things under the hood to prevent system calls from interfering with the timing. For example, it prevents cleanup of unused Python objects (known as garbage collection) that might otherwise affect the timing. For this reason, %timeit results are usually noticeably faster than %time results.
For %time as with %timeit, using the double-percent-sign cell-magic syntax allows timing of multiline scripts
Profiling Full Scripts: %prun
A program is made of many single statements, and sometimes timing these statements in context is more important than timing them on their own. Python contains a built-in code profiler (which you can read about in the Python documentation), but IPython offers a much more convenient way to use this profiler, in the form of the magic function %prun.
By way of example, we’ll define a simple function that does some calculations:
- def sum_of_list(N):
- total = 0
- for i in range(5):
- L = [j ^ (j >> i) for j in range(N)]
- total += sum(L)
- return total
- 14 function calls in 0.714 seconds
- Ordered by: internal time
- ncalls tottime percall cumtime percall filename:lineno(function)
- 5 0.599 0.120 0.599 0.120
19>:4( ) - 5 0.064 0.013 0.064 0.013 {built-in method sum}
- 1 0.036 0.036 0.699 0.699
19>:1(sum_of_lists) - 1 0.014 0.014 0.714 0.714
: 1() - 1 0.000 0.000 0.714 0.714 {built-in method exec}
Line-by-Line Profiling with %lprun
The function-by-function profiling of %prun is useful, but sometimes it’s more convenient to have a line-by-line profile report. This is not built into Python or IPython, but there is a line_profiler package available for installation that can do this. Start by using Python’s packaging tool, pip, to install the line_profiler package:
Next, you can use IPython to load the line_profiler IPython extension, offered as part of this package:
Now the %lprun command will do a line-by-line profiling of any function—in this case, we need to tell it explicitly which functions we’re interested in profiling:
As before, the notebook sends the result to the pager, but it looks something like this:
The information at the top gives us the key to reading the results: the time is reported in microseconds and we can see where the program is spending the most time. At this point, we may be able to use this information to modify aspects of the script and make it perform better for our desired use case.
Profiling Memory Use: %memit and %mprun
Another aspect of profiling is the amount of memory an operation uses. This can be evaluated with another IPython extension, the memory_profiler. As with the line_profiler, we start by pip-installing the extension:
Then we can use IPython to load the extension:
The memory profiler extension contains two useful magic functions: the %memit magic (which offers a memory-measuring equivalent of %timeit) and the %mprun function (which offers a memory-measuring equivalent of %lprun). The %memit function can be used rather simply:
We see that this function uses about 100 MB of memory.
For a line-by-line description of memory use, we can use the %mprun magic. Unfortunately, this magic works only for functions defined in separate modules rather than the notebook itself, so we’ll start by using the %%file magic to create a simple module called mprun_demo.py, which contains our sum_of_list function, with one addition that will make our memory profiling results more clear:
We can now import the new version of this function and run the memory line profiler:
The result, printed to the pager, gives us a summary of the memory use of the function, and looks something like this:
Here the Increment column tells us how much each line affects the total memory budget: observe that when we create and delete the list L, we are decreasing about 25 MB of memory usage. This is on top of the background memory usage from the Python interpreter itself.
沒有留言:
張貼留言