Afterword
(To avoid burying the lede, I’ve shifted my reflections to the very top of this post). The exercise that follows in this post is one of my first attempts to untangle the innards of CPython. The benefits of this activity might not be immediately obvious, but in hindsight here’s what I got out of the exercise:
Firstly, it gave me the confidence to make my second pull request in CPython to implement cache invalidation for zip importers. It turned out that figuring out imports in the C interpreter was not particularly necessary for understanding the Python implementation of importlib and zipimport, but it did give me assurance that I could jump into the Python-end of things without becoming irrecoverably lost.
It also taught me a lot about how to read a large code base. Reading everything linearly would be an almost hopeless endeavour. Instead, when trying to understand code, you need to jump around to relevant definitions and function calls to build a mental model of the architecture in your head. Improving one’s skill at jumping around, and learning to use the tools that help you accomplish this —thank you, ctags and gdb— is imperative for improving one’s ability to contribute to a code base.
And finally, I learned that reading the code can trump documentation in terms of building context. There’s something more real and more authoritative about the code itself. I’ve looked over CPython’s source code layout multiple times, but it didn’t stick (or didn’t really mean anything to me) until I had to dive into the code itself.
Now, on to the post.
Today I want to walk through what happens when you import something in Python. If you want to follow along, I’m working through Python v3.10.0.a3 which can be found at https://github.com/python/cpython. I’m documenting this as I go, so as of this sentence I have no idea what kind of ride we’re in for, but I have a feeling this is going to be fun.
The code we’ll be exploring
First of all, let’s write a simple script. We’re just going to import a single module, then print the value of an object in that module:
import math
print(math.pi)
This gives us the expected output 3.141592653589793. Seems pretty straight-forward right? Let’s take a look at the disassembled code.
1 0 LOAD_CONST 0 (0)
2 LOAD_CONST 1 (None)
4 IMPORT_NAME 0 (math)
6 STORE_NAME 0 (math)
2 8 LOAD_NAME 1 (print)
10 LOAD_NAME 0 (math)
12 LOAD_ATTR 2 (pi)
14 CALL_FUNCTION 1
16 POP_TOP
18 LOAD_CONST 1 (None)
20 RETURN_VALUE
Right off the bat we notice that there’s some magic numbers being loaded in line 1. The bytecode contains two LOAD_CONST operations that load 0 and None onto the stack. To elucidate what these magic constants might represent, let’s make some minor modifications to our test program. We’ll also call a function from the math library for good measure.
import math
from math import pi
print(pi)
x = math.ceil(1.5)
print(x)
Our bytecode now looks like this:
1 0 LOAD_CONST 0 (0)
2 LOAD_CONST 1 (None)
4 IMPORT_NAME 0 (math)
6 STORE_NAME 0 (math)
2 8 LOAD_CONST 0 (0)
10 LOAD_CONST 2 (('pi',))
12 IMPORT_NAME 0 (math)
14 IMPORT_FROM 1 (pi)
16 STORE_NAME 1 (pi)
18 POP_TOP
3 20 LOAD_NAME 2 (print)
22 LOAD_NAME 1 (pi)
24 CALL_FUNCTION 1
26 POP_TOP
4 28 LOAD_NAME 0 (math)
30 LOAD_METHOD 3 (ceil)
32 LOAD_CONST 3 (1.5)
34 CALL_METHOD 1
36 STORE_NAME 4 (x)
5 38 LOAD_NAME 2 (print)
40 LOAD_NAME 4 (x)
42 CALL_FUNCTION 1
44 POP_TOP
46 LOAD_CONST 1 (None)
48 RETURN_VALUE
The IMPORT_NAME instruction
We’re now in better shape to understand some of the machinery behind the import statement. Let’s pay attention to the first line. import math produces the following bytecode:
1 0 LOAD_CONST 0 (0)
2 LOAD_CONST 1 (None)
4 IMPORT_NAME 0 (math)
6 STORE_NAME 0 (math)
To reiterate, first 0 and None are pushed onto the stack. Then, IMPORT_NAME, and STORE_NAME are called. Let’s trace what IMPORT_NAME does. Going into Python/ceval.c, we find the relevant case in the interpreter loop.
case TARGET(IMPORT_NAME): {
PyObject *name = GETITEM(names, oparg);
PyObject *fromlist = POP();
PyObject *level = TOP();
PyObject *res;
res = import_name(tstate, f, name, fromlist, level);
Py_DECREF(level);
Py_DECREF(fromlist);
SET_TOP(res);
if (res == NULL)
goto error;
DISPATCH();
}
Let’s try to understand the first line. To do this, we’ll first need to build up some context. We first note that names is set to the value co->co_names (line 1431), where co = f->f_code (line 1386) gives us the code object of the current frame f, and co_names corresponds to “a tuple containing the names used by the bytecode” as documented here. In this case, names contains ('math', 'pi', 'print', 'ceil', 'x'). Additionally, we can see from the disassembled bytecode above that oparg currently has a value of 0, which we can verify by printing the value of oparg on running the code for IMPORT_NAME. (Why is this the case? Sounds like a topic for another post.)
Looking further up the file, we see the definition of the GETITEM() macro.
#define GETITEM(v, i) PyTuple_GetItem((v), (i))
The definition for this function can then be found in Objects/tupleobject.c:
PyObject *
PyTuple_GetItem(PyObject *op, Py_ssize_t i)
{
if (!PyTuple_Check(op)) {
PyErr_BadInternalCall();
return NULL;
}
if (i < 0 || i >= Py_SIZE(op)) {
PyErr_SetString(PyExc_IndexError, "tuple index out of range");
return NULL;
}
return ((PyTupleObject *)op) -> ob_item[i];
}
Here, if (!PyTuple_Check(op)) and if (i < 0 || i >= Py_SIZE(op)) respectively check that the first argument given was a PyTuple object, and that the second argument is within the range of the given tuple’s size. It then returns the ith item. As we can see, a lot of the code here is exception handling.
Phew. Okay, now where does that get us? We’ve now taken the 0th item from names, which is ‘math’, and assigned it to the name variable. It seems that this line of execution always gets the name of the module we’re importing from. Indeed, peeking ahead at the execution of from math import pi, we see that this is still the case even though we’ve modified the code.
2 8 LOAD_CONST 0 (0)
10 LOAD_CONST 2 (('pi',))
12 IMPORT_NAME 0 (math)
14 IMPORT_FROM 1 (pi)
16 STORE_NAME 1 (pi)
18 POP_TOP
Quickly going over the next few lines in the IMPORT_NAME instruction, we see more set-up code. After which, we pass these arguments into the import_name() function.
case TARGET(IMPORT_NAME): {
PyObject *name = GETITEM(names, oparg);
PyObject *fromlist = POP();
PyObject *level = TOP();
PyObject *res;
res = import_name(tstate, f, name, fromlist, level);
...
Here, POP() and TOP() are macros for accessing the stack.
#define TOP() (stack_pointer[-1])
#define POP() BASIC_POP()
#define BASIC_POP() (*--stack_pointer)
If you were wondering why POP() is defined in such a circuitous manner, it’s because there’s a lot of debugging and tracing code that we’re just ignoring for now. From the above definitions, we see that TOP() returns the address of the object sitting at the top of the stack. POP() does the same, but it also decrements the stack pointer, which “removes” the object’s address from the stack. Recall that we had previously pushed 0 and None onto the stack, this means that fromlist points to None, and level points to 0. level is a bit mysterious, but we won’t go into that (it’s related to the location of your module in your directory path, i.e. whether your module is located in a sub/parent directory). However, it makes sense that fromlist points to None, since we don’t bring in and bind any objects from the math library. Quickly peeking at the bytecode for from math import pi we see how this differs. In this case fromlist points to (‘pi’,).
2 8 LOAD_CONST 0 (0)
10 LOAD_CONST 2 (('pi',))
12 IMPORT_NAME 0 (math)
14 IMPORT_FROM 1 (pi)
16 STORE_NAME 1 (pi)
18 POP_TOP
Before we explore the import_name() function, let’s wrap up our discussion of the IMPORT_NAME instruction.
...
Py_DECREF(level);
Py_DECREF(fromlist);
SET_TOP(res);
if (res == NULL)
goto error;
DISPATCH();
The remaining lines in this instruction are the break-down operations. Presumably there were references made to level and fromlist in import_name(), so now that we’ve exited the function, we decrease their reference counts by 1. This is important for the Python garbage collector to figure out whether it can free up the space taken up by these objects. Following this, we set the return value of import_name() to the top of the stack, and throw an error if the value is NULL. At the very end, DISPATCH(); prepares us for executing the next line of bytecode. What happens exactly? That’s a topic for another post (likely coupled with a post on opargs).
The import_name() function
In the previous section, we discussed the set-up and break-down code for the IMPORT_NAME instruction, but all that was basically the preamble and denouement for the import_name() function. For starters, here’s the full function in Python/ceval.c.
static PyObject *
import_name(PyThreadState *tstate, PyFrameObject *f,
PyObject *name, PyObject *fromlist, PyObject *level)
{
_Py_IDENTIFIER(__import__);
PyObject *import_func, *res;
PyObject* stack[5];
import_func = _PyDict_GetItemIdWithError(f->f_builtins, &PyId___import__);
if (import_func == NULL) {
if (!_PyErr_Occurred(tstate)) {
_PyErr_SetString(tstate, PyExc_ImportError, "__import__ not found");
}
return NULL;
}
/* Fast path for not overloaded __import__. */
if (import_func == tstate->interp->import_func) {
int ilevel = _PyLong_AsInt(level);
if (ilevel == -1 && _PyErr_Occurred(tstate)) {
return NULL;
}
res = PyImport_ImportModuleLevelObject(
name,
f->f_globals,
f->f_locals == NULL ? Py_None : f->f_locals,
fromlist,
ilevel);
return res;
}
Py_INCREF(import_func);
stack[0] = name;
stack[1] = f->f_globals;
stack[2] = f->f_locals == NULL ? Py_None : f->f_locals;
stack[3] = fromlist;
stack[4] = level;
res = _PyObject_FastCall(import_func, stack, 5);
Py_DECREF(import_func);
return res;
}
Let’s first follow _Py_IDENTIFIER() for a bit. It appears quite often in the code base, so this will do us some good. This is a macro defined in Include/cpython/object.h. The relevant definitions are as follows.
typedef struct _Py_Identifier {
const char* string;
// Index in PyInterpreterState.unicode.ids.array. It is process-wide
// unique and must be initialized to -1.
Py_ssize_t index;
} _Py_Identifier;
#define _Py_static_string_init(value) { .string = value, .index = -1 }
#define _Py_static_string(varname, value) static _Py_Identifier varname = _Py_static_string_init(value)
#define _Py_IDENTIFIER(varname) _Py_static_string(PyId_##varname, #varname)
Side note: within C macros, the ## symbols tell the preprocessor to combine the two tokens on either side of the ##, and the # symbol tells the preprocessor to convert the token into a string. i.e., PyId_##varname gives us the token PyId_varname, and #varname gives us the string “varname”. This is a useful reference.
Going through the macros, _Py_IDENTIFIER() expands to Py_static_string(PyId___import__, "__import__"), which expands to static Py_Identifier PyId___import_ = _Py_static_string_init("__import__"), and finally expands to static Py_Identifier PyId___import_ = { .string = "__import__", .index = -1 }. Speculating a little bit, this gives us a new identifier for __import__ within the interpreter, which maybe probably interns the string “__import__”. It’s a bit out of the way for us to discuss why this is needed, so we’ll save this for another time. Now we’ll see how it’s used.
static PyObject *
import_name(PyThreadState *tstate, PyFrameObject *f,
PyObject *name, PyObject *fromlist, PyObject *level)
{
_Py_IDENTIFIER(__import__);
PyObject *import_func, *res;
PyObject* stack[5];
import_func = _PyDict_GetItemIdWithError(f->f_builtins, &PyId___import__);
...
Focusing on the definition of import_func, we see that we get an item out of the dictionary of` built-in objects within our current frame f->f_builtins. Since PyId___import__ contains the string ‘__import__’, this means we get the build-in function __import__. This is followed by some error handling, in the event that we fail to extract the required function.
...
if (import_func == NULL) {
if (!_PyErr_Occurred(tstate)) {
_PyErr_SetString(tstate, PyExc_ImportError, "__import__ not found");
}
return NULL;
}
...
What follows is an if block that, minus some error checking, runs PyImport_ImportModuleLevelObject then returns. After the if block, we seem to do some set-up and break-down in order to run _PyObject_FastCall(import_func, stack, 5). To smooth over some of the nitty gritty details, we can consider the code within the if block, and following the if block to produce the same effect, just that one is optimised and the other isn’t.
I’ll explain why. Examining the code outside the if block, we perform some book-keeping for garbage collection, and create a stack to store relevant variables: name which has the value of ‘math’, the pointer to the global variables of our frame f->f_globals, the pointer to the local variables of our frame f->f_locals, the objects we’re importing (in this case, either None or (‘pi’,)), and the current level. We then pass along import_func, this stack, and the size of the stack to _PyObject_FastCall() which calls import_func with the given arguments and returns the result.
Another side note: we kind of glossed over the _PyObject_FastCall() function. That’s another deep rabbit hole to dive into that we’ll leave for another time. For those who are curious, here’s a hint: your entry-point should be somewhere inside Include/cpython/abstract.h.
import_func contained the built-in __import__ function which lives in Python/bltinmodule.c.
static PyObject *
builtin___import__(PyObject *self, PyObject *args, PyObject *kwds)
{
static char *kwlist[] = {"name", "globals", "locals", "fromlist",
"level", 0};
PyObject *name, *globals = NULL, *locals = NULL, *fromlist = NULL;
int level = 0;
if (!PyArg_ParseTupleAndKeywords(args, kwds, "U|OOOi:__import__",
kwlist, &name, &globals, &locals, &fromlist, &level))
return NULL;
return PyImport_ImportModuleLevelObject(name, globals, locals,
fromlist, level);
}
As we can see, the built-in import function also eventually calls PyImport_ImportModuleLevelObject.
The PyImport_ImportModuleLevelObject function
This function is defined in lines 1545–1695 of Python/import.c. There are too many lines for us to copy here, but you can follow the link to take a look. There’s a lot going on in this function. But thankfully, most of it is — you guessed it — error checking. The first line of interest appears around 40 lines down.
...
else { /* level == 0 */
if (PyUnicode_GET_LENGTH(name) == 0) {
_PyErr_SetString(tstate, PyExc_ValueError, "Empty module name");
goto error;
}
abs_name = name;
Py_INCREF(abs_name);
}
mod = import_get_module(tstate, abs_name);
if (mod == NULL && _PyErr_Occurred(tstate)) {
goto error;
}
if (mod != NULL && mod != Py_None) {
if (import_ensure_initialized(tstate, mod, name) < 0) {
goto error;
}
}
else {
Py_XDECREF(mod);
mod = import_find_and_load(tstate, abs_name);
if (mod == NULL) {
goto error;
}
}
...
As previously noted, our current level is 0, hence we simply set abs_name to ‘math’, then call import_get_module. What’s interesting to note is that the first time we run import math, this function returns NULL. On subsequent executions of import math, or from math import pi, mod is no longer NULL. This means that for the first execution, we have to run mod = import_find_and_load(tstate, abs_name);.
After retrieving the module, we then check if fromlist is empty. If it is, we do the following.
....
if (!has_from) {
Py_ssize_t len = PyUnicode_GET_LENGTH(name);
if (level == 0 || len > 0) {
Py_ssize_t dot;
dot = PyUnicode_FindChar(name, '.', 0, len, 1);
if (dot == -2) {
goto error;
}
if (dot == -1) {
/* No dot in module name, simple exit */
final_mod = mod;
Py_INCREF(mod);
goto error;
}
if (level == 0) {
PyObject *front = PyUnicode_Substring(name, 0, dot);
if (front == NULL) {
goto error;
}
final_mod = PyImport_ImportModuleLevelObject(front, NULL, NULL, NULL, 0);
Py_DECREF(front);
}
else {
Py_ssize_t cut_off = len - dot;
Py_ssize_t abs_name_len = PyUnicode_GET_LENGTH(abs_name);
PyObject *to_return = PyUnicode_Substring(abs_name, 0,
abs_name_len - cut_off);
if (to_return == NULL) {
goto error;
}
final_mod = import_get_module(tstate, to_return);
Py_DECREF(to_return);
if (final_mod == NULL) {
if (!_PyErr_Occurred(tstate)) {
_PyErr_Format(tstate, PyExc_KeyError,
"%R not in sys.modules as expected",
to_return);
}
goto error;
}
}
}
else {
final_mod = mod;
Py_INCREF(mod);
}
}
...
Since there is no ‘.’ in our module name, we simply set final_mod = mod then goto error, which cleans up our state and returns the value of final_mod.
...
error:
Py_XDECREF(abs_name);
Py_XDECREF(mod);
Py_XDECREF(package);
if (final_mod == NULL) {
remove_importlib_frames(tstate);
}
return final_mod;
}
If fromlist isn’t empty, then we need to consider if a path has or hasn’t been given. In from math import pi, we see that this isn’t the case, so we simply set final_mod = mod, then do the same as above.
...
else {
PyObject *path;
if (_PyObject_LookupAttrId(mod, &PyId___path__, &path) < 0) {
goto error;
}
if (path) {
Py_DECREF(path);
final_mod = _PyObject_CallMethodIdObjArgs(
interp->importlib, &PyId__handle_fromlist,
mod, fromlist, interp->import_func, NULL);
}
else {
final_mod = mod;
Py_INCREF(mod);
}
}
...
So after all this work, we return final_mod, which is likely a PyModule object. We’ll have to verify this. And of course, we swept some details under the rug, namely the import_get_module and import_find_and_load functions, so we’ll need to inspect these too.
The import_get_module function
In Python/import.c, we find the import_get_module function.
static PyObject *
import_get_module(PyThreadState *tstate, PyObject *name)
{
PyObject *modules = tstate->interp->modules;
if (modules == NULL) {
_PyErr_SetString(tstate, PyExc_RuntimeError,
"unable to get sys.modules");
return NULL;
}
PyObject *m;
Py_INCREF(modules);
if (PyDict_CheckExact(modules)) {
m = PyDict_GetItemWithError(modules, name); /* borrowed */
Py_XINCREF(m);
}
else {
m = PyObject_GetItem(modules, name);
if (m == NULL && _PyErr_ExceptionMatches(tstate, PyExc_KeyError)) {
_PyErr_Clear(tstate);
}
}
Py_DECREF(modules);
return m;
}
We’re finally in more straight-forward territory here (or maybe I’m just getting used to how this code is laid out). PyObject *modules = tstate->interp->modules gets the current modules that have been imported into the interpreter.
We do some error checking, first to make sure that the list of modules isn’t NULL, then to check that the modules are stored as a dictionary. If these checks pass, we get the value using the name of the module as the key. This answers one of our earlier questions: import_get_module returns NULL the first time it’s run because the math module hasn’t been imported yet. To get the final piece of the story, we have to dig into import_find_and_load.
The import_find_and_load function
If you thought we were finally done with long functions, think again. import_find_and_load can be found in Python/import.c lines 1461–1526. Again, we won’t be copying the code here, but feel free to follow the link. We first note that once again, error checking and tracing makes up the bulk of the code here. The real magic happens in the places where we assign values to mod, which happens here:
mod = _PyObject_CallMethodIdObjArgs(interp->importlib,
&PyId__find_and_load, abs_name,
interp->import_func, NULL);
Going down the rabbit hole, we find _PyObject_CallMethodIdObjArgs defined in Objects/call.c.
PyObject *
_PyObject_CallMethodIdObjArgs(PyObject *obj,
struct _Py_Identifier *name, ...)
{
PyThreadState *tstate = _PyThreadState_GET();
if (obj == NULL || name == NULL) {
return null_error(tstate);
}
PyObject *oname = _PyUnicode_FromId(name); /* borrowed */
if (!oname) {
return NULL;
}
PyObject *callable = NULL;
int is_method = _PyObject_GetMethod(obj, oname, &callable);
if (callable == NULL) {
return NULL;
}
obj = is_method ? obj : NULL;
va_list vargs;
va_start(vargs, name);
PyObject *result = object_vacall(tstate, obj, callable, vargs);
va_end(vargs);
Py_DECREF(callable);
return result;
}
Yet another side note: the ellipses (…) operator represents a variable argument list. Notice that we passed in 5 arguments to _PyObject_CallMethodIdObjArgs but only two parameters are declared. The rest of the arguments are obtained through vargs.
Here, it seems that two lines are of particular interest: the call to _PyObject_GetMethod which sets our callable pointer, and the result of object_vacall(tstate, obj, callable, vargs), which is the return value we’re ultimately looking for.
Jumping into _PyObject_GetMethod in Objects/object.c, we first note that it’s been given the *obj = importlib module, *name = ‘_find_and_load’ (as a PyUnicodeObject), and **method = callable (currently a NULL pointer) as its arguments. After going through some checks, we run the following function.
*method = PyObject_GetAttr(obj, name);
Which eventually runs:
return (*tp->tp_getattro)(v, name);
Since v = obj = importlib is a PyModule object, this brings us to the module_getattro function in Objects/moduleobject.c, which then calls the generic PyObject_GenericGetAttr function on importlib and ‘_find_and_load`, which then calls _PyObject_GenericGetAttrWithDict(obj, name, NULL, 0), that finally gets importlib’s _find_and_load function inside Lib/importlib/_bootstrap.py via a dictionary lookup of all of importlib’s implemented functions and objects. This post is concerned with the bytecode and C source code portions of import’s implementation, so we’ll save an examination of _find_and_load for a future post.
Now that we’ve obtained the _find_and_load function, we pass this value back up all the way to _PyObject_CallMethodIdObjArgs in Objects/call.c. This was where we left off:
...
PyObject *callable = NULL;
int is_method = _PyObject_GetMethod(obj, oname, &callable);
if (callable == NULL) {
return NULL;
}
obj = is_method ? obj : NULL;
va_list vargs;
va_start(vargs, name);
PyObject *result = object_vacall(tstate, obj, callable, vargs);
va_end(vargs);
Py_DECREF(callable);
return result;
}
callable now points to importlib._find_and_load, and we proceed to the next line of interest: object_vacall(tstate, obj, callable, vargs).
This brings us to Objects/call.c once again. Inside object_vacall(), we first do some set-up: check that the callable function is not null, and then assign the given vargs to a local stack. Finally, we called _PyObject_VectorcallTstate() on the thread state, the callable, our stack, and the number of arguments in the stack. We then clean up the stack from memory and return the result.
_PyObject_VectorcallTstate() brings us to Include/cpython/abstract.h:
static inline PyObject *
_PyObject_VectorcallTstate(PyThreadState *tstate, PyObject *callable,
PyObject *const *args, size_t nargsf,
PyObject *kwnames)
{
vectorcallfunc func;
PyObject *res;
assert(kwnames == NULL || PyTuple_Check(kwnames));
assert(args != NULL || PyVectorcall_NARGS(nargsf) == 0);
func = PyVectorcall_Function(callable);
if (func == NULL) {
Py_ssize_t nargs = PyVectorcall_NARGS(nargsf);
return _PyObject_MakeTpCall(tstate, callable, args, nargs, kwnames);
}
res = func(callable, args, nargsf, kwnames);
return _Py_CheckFunctionResult(tstate, callable, res, NULL);
}
There’s a few things to note here. We set func = PyVectorcall_Function(callable), then call res = func(...) on our arguments, and finally return the result of calling _Py_CheckFunctionResult(). Let’s explore each in turn.
In Include/cpython/abstract.h, we get the definition of PyVectorcall_Function().
static inline vectorcallfunc
PyVectorcall_Function(PyObject *callable)
{
PyTypeObject *tp;
Py_ssize_t offset;
vectorcallfunc ptr;
assert(callable != NULL);
tp = Py_TYPE(callable);
if (!PyType_HasFeature(tp, Py_TPFLAGS_HAVE_VECTORCALL)) {
return NULL;
}
assert(PyCallable_Check(callable));
offset = tp->tp_vectorcall_offset;
assert(offset > 0);
memcpy(&ptr, (char *) callable + offset, sizeof(ptr));
return ptr;
}
We get the type of callable, which is, not surprisingly, PyFunction_Type. Looking at line 674 of Objects/funcobject.c, we see that the Py_TPFLAGS_HAVE_VECTORCALL flag has been set. So we copy the contents of the funcobject’s vectorcall function that happens to be the _PyFunction_Vectorcall() function in Objects/call.c. This is then assigned to func, As a reminder, here we are now:
static inline PyObject *
_PyObject_VectorcallTstate(PyThreadState *tstate, PyObject *callable,
PyObject *const *args, size_t nargsf,
PyObject *kwnames)
{
vectorcallfunc func;
PyObject *res;
assert(kwnames == NULL || PyTuple_Check(kwnames));
assert(args != NULL || PyVectorcall_NARGS(nargsf) == 0);
func = PyVectorcall_Function(callable);
if (func == NULL) {
Py_ssize_t nargs = PyVectorcall_NARGS(nargsf);
return _PyObject_MakeTpCall(tstate, callable, args, nargs, kwnames);
}
res = func(callable, args, nargsf, kwnames);
return _Py_CheckFunctionResult(tstate, callable, res, NULL);
}
Hence, calling func(callable, args, nargsf, kwnames) means that we’re really calling _PyFunction_Vectorcall(callable, args, nargsf, kwnames). Within _PyFunction_Vectorcall(), we extract the code object from the function, as well as other details such as the current state, global variables, closure etc. We then call the hot function function_code_fastcall_().
...
return function_code_fastcall(tstate, co,
stack, PyTuple_GET_SIZE(argdefs),
globals);
}
...
And here’s where the magic happens. We construct a frame given the arguments to function_code_fastcall(), then we call _PyEval_EvalFrame().
...
PyObject *result = _PyEval_EvalFrame(tstate, f, 0);
...
This! This is our main interpreter loop! So what have we done? We took the importlib._find_and_load function, extracted its code object, and set the relevant items in the stack to the our desired arguments (the most import being ‘math’ as the name of the module we want to import), then evaluated our import code on the current interpreter state and this argument.
But wait, hang on for a moment, I actually lied a bit. From _PyFunction_Vectorcall(), we might actually instead make a call to _PyEval_EvalCode() (not what happens when we run import math), but that’s fine. This brings us back to Python/ceval.c (always a hint that something fun is about to happen), sets up a frame given the code and globals and variables we’ve given it, then calls _PyEval_EvalFrame(), so we’re back at the main interpreter loop again!
Ok, now we can move on. So we call importlib._find_and_load on our module name math. This returns a a PyModule object as we guessed. Quick recap, we now return this object all the way back to _PyObject_VectorcallTstate.
static inline PyObject *
_PyObject_VectorcallTstate(PyThreadState *tstate, PyObject *callable,
PyObject *const *args, size_t nargsf,
PyObject *kwnames)
{
vectorcallfunc func;
PyObject *res;
assert(kwnames == NULL || PyTuple_Check(kwnames));
assert(args != NULL || PyVectorcall_NARGS(nargsf) == 0);
func = PyVectorcall_Function(callable);
if (func == NULL) {
Py_ssize_t nargs = PyVectorcall_NARGS(nargsf);
return _PyObject_MakeTpCall(tstate, callable, args, nargs, kwnames);
}
res = func(callable, args, nargsf, kwnames);
return _Py_CheckFunctionResult(tstate, callable, res, NULL);
}
We finally call _Py_CheckFunctionResult(). As the name suggests, we do a bunch of error checking here, and if all of them pass, then we return the module object that we got. And that’s it, we go all the way back up and we’re now done exploring import_find_and_load.
Back to the byte code
We’ve covered a lot of ground in C, but it’s easy to get lost, so let’s zoom out a bit and go back to the byte code.
1 0 LOAD_CONST 0 (0)
2 LOAD_CONST 1 (None)
4 IMPORT_NAME 0 (math)
6 STORE_NAME 0 (math)
After all that work, we’ve finally covered the IMPORT_NAME instruction. Now the desired module is on the stack, and STORE_NAME binds the module name ‘math’ to the module object. from math import pi then gives us
2 8 LOAD_CONST 0 (0)
10 LOAD_CONST 2 (('pi',))
12 IMPORT_NAME 0 (math)
14 IMPORT_FROM 1 (pi)
16 STORE_NAME 1 (pi)
18 POP_TOP
Here we see that the math module object is left on the top of the stack. Going back into ceval.c, we can take a look at what the IMPORT_FROM instruction does.
case TARGET(IMPORT_FROM): {
PyObject *name = GETITEM(names, oparg);
PyObject *from = TOP();
PyObject *res;
res = import_from(tstate, from, name);
PUSH(res);
if (res == NULL)
goto error;
DISPATCH();
}
We get the name ‘pi’ from the names in our frame, and the module that we left on the top of the stack. We then call res = import_from(tstate, from, name) before pushing the result onto the stack. In import_from(), the action happens here:
if (_PyObject_LookupAttr(v, name, &x) != 0) {
return x;
}
We simply import the ‘pi’ attribute from the math module object (it’s there in the dictionary of attributes in math, which you can see by running dir(math)). The pi object (of PyFloat_Type) is then pushed onto the stack. STORE_NAME then binds the name ‘pi’ to the float object. This is followed by POP_TOP to remove the module from the stack. Within STORE_NAME, we actually left the module on the stack to avoid having to reload it for every object that we wanted to import from the module. If we had run from math import pi, ceil, the byte code would have looked like this instead:
2 8 LOAD_CONST 0 (0)
10 LOAD_CONST 2 (('pi', 'ceil'))
12 IMPORT_NAME 0 (math)
14 IMPORT_FROM 1 (pi)
16 STORE_NAME 1 (pi)
18 IMPORT_FROM 2 (ceil)
20 STORE_NAME 2 (ceil)
22 POP_TOP
Moving on to the rest of the byte code. print(pi) gave us
3 20 LOAD_NAME 2 (print)
22 LOAD_NAME 1 (pi)
24 CALL_FUNCTION 1
26 POP_TOP
This loads the name ‘print’, followed by the name ‘pi’, then runs the CALL_FUNCTION instruction, which gets the function bounded to ‘print’ and gives, as its argument, the object stored at ‘pi’, which is our float with value 3.141592653589793.
x = math.ceil(1.5) produces this byte code:
4 28 LOAD_NAME 0 (math)
30 LOAD_METHOD 3 (ceil)
32 LOAD_CONST 3 (1.5)
34 CALL_METHOD 1
36 STORE_NAME 4 (x)
The only mysterious instruction here is LOAD_METHOD. This of course takes us back to Python/ceval.c, where we run int meth_found = _PyObject_GetMethod(obj, name, &meth), with obj being our module, and name being ‘ceil’. Then, very simply, we point meth to PyObject_GetAttr(obj, name) since the ‘ceil’ attribute in the math module points to the appropriate PyCFunction. We then call this method with the loaded constant 1.5, which pushes the value 2 onto the stack. We then call STORE_NAME which binds this value to the name ‘x’.
Finally, we run print(x), which produces byte code that’s almost the same as print(pi). The only real difference being the last two lines:
5 38 LOAD_NAME 2 (print)
40 LOAD_NAME 4 (x)
42 CALL_FUNCTION 1
44 POP_TOP
46 LOAD_CONST 1 (None)
48 RETURN_VALUE
These lines just load the constant None, then returns this value. i.e. our program returns None before terminating.
Back to the start
We’ve covered a lot of ground in this post. It almost seems crazy how deep we had to go just to figure out half of the story of only the import statement when we run a simple program:
import math
from math import pi
print(pi)
x = math.ceil(1.5)
print(x)
We didn’t even consider any of the underlying Python implementation yet. And if you think that would be more straight-forward, think again. I can assure you, that due to bootstrapping problems (you need to import the import library), that’s going to be a wild wild ride.
But it would be a very curious and educating exercise.
Afterword (again)
I copied the afterword to the top of this post, but I’ll reiterate here for closure. The benefits of this activity might not have been immediately obvious, but here’s what I got out of it:
Firstly, it gave me the confidence to make my second pull request in CPython to implement cache invalidation for zip importers. It turned out that figuring out imports in the C interpreter was not particularly necessary for understanding the Python implementation of importlib and zipimport, but it did give me assurance that I could jump into the Python-end of things without becoming irrecoverably lost.
It also taught me a lot about how to read a large code base. Reading everything linearly would be an almost hopeless endeavour. Instead, when trying to understand code, you need to jump around to relevant definitions and function calls to build a mental model of the architecture in your head. Improving one’s skill at jumping around, and learning to use the tools that help you accomplish this —thank you, ctags and gdb— is imperative for improving one’s ability to contribute to a code base.
And finally, I learned that reading the code can trump documentation in terms of building context. There’s something more real and more authoritative about the code itself. I’ve looked over CPython’s source code layout multiple times, but it didn’t stick (or didn’t really mean anything to me) until I had to dive into the code itself.
And with that, thank you for joining me on this ride.
Pingback: Vignettes of a Linux Kernel Mentee – Desmond Cheong