Namespaces and Scope in Python

An Example

Let's first look at an example and see what the output should be when executed.

>>> x = 10
>>> def foo():
...     print(x)
...     x += 1
>>> foo()

The answer is as below

>>> foo()
Traceback (most recent call last):
  ...
UnboundLocalError: local variable 'x' referenced before assignment

This is where namespaces and scopes are used in Python. Before we find out these concepts, let's first answer what names are in Python. A name in Python is just a way to access an object.

>>> a = 1
>>> b = lambda x: x
>>> class C:
...     pass
>>> D = C()

In the snippet above, a, b, C, D are both names and we can see that a name can reference an integer, a function, a class or an object. Then what are namespaces in Python?

Namespaces and Scopes

Simply speaking, a namespace is a system to control names in a program. There may be multiple names in one program and multilayers of namespaces ensure that these names won’t lead to any conflict. Generally, names with a shared name in different namespaces but reference different objects. Let's look at an example below to get the picture.

>>> def x():
...     x = 1
...     def foo():
...         x = 2

As we can see, there are 3 x in the above snippet, while x in the first line references a function and x in the second line references 1 and x in the fourth line references 2. Most importantly, the value of x varies when it is used in different namespaces. In this case, the outermost x is defined in the global namespace and x in the middle is defined in the enclosed namespace while the innermost x is defined in the local namespace. Specifically,

A local namespace covers local names within a function or a class. Python creates this namespace for every function called in a program. It remains active until the function returns.
An enclosed namespace typically arises in higher order functions and it covers the nested function and other names. Python creates this namespace for every higher order function. It also remains active until the function returns.
A global namespace covers the names from various imported modules and other names. Python creates this namespace for every module included in a program. It will last until the program ends.
A built-in namespace covers all built-in functions and built-in exception names like print, len, ValueError and so on. Python creates it as the interpreter starts and keeps it until you exit.

Correspondingly, scopes determine which namespace a name belongs to and ensure that names can be used without any prefix. This guideline is called the LEGB rule illustrated as below.

The LEGB Rule

This picture explains the order in which these namespaces are to be searched for scope resolution. Specifically, if a variable is not defined in the local scope, then it will be searched in outer scopes to check if there is such a variable.

>>> x = 10
>>> def foo():
...     print(x)
...     x += 1
>>> foo()
Traceback (most recent call last):
  ...
UnboundLocalError: local variable 'x' referenced before assignment

Let's get back to the above example. There is a x variable in the global namespace but the assignment x+=1 declares a variable in the local namespace and assign a value recursively. Hence, print(x) first finds and calls on a local variable without any reference and raises an exception. Even print(x) is executed before the assignment, there is still an error about how x should be properly used. This is because an assignment to a variable in a function makes that variable local to that function . Even x += 1 changes to x = 1, there will be the same error. The compiler precompiles everything and assign each variable with its namespace.

The Implementation in Python

To amend the above snippet, we can use the keyword global or nonlocal. global is used to assign a value to a global object within a local namespace while nonlocal is used to assign a value to a object in a higher order function.

>>> x = 10
>>> def foo():
...     global x
...     print(x)
...     x += 1
>>> foo()
10
>>> print(x)
11

>>> def foo():
...     x = 10
...     def foo2():
...         nonlocal x
...         print(x)
...         x += 1
...     foo2(x)
...     print(x)
>>> foo()
10
11

In Python, namespaces are implemented in the form of dictionaries. It maintains a name-to-object mapping where names act as keys and objects as values. And we can use built-in functions globals and locals to see these mappings.

a = 10
def function():
    a = 11
    print(locals())
    def foo():
        a = 12
        print(locals())
    return foo

print(globals())
if __name__ == '__main__':
    function()()

And we can get the below output if we run the above snippet.

{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': ..., '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': '...', '__cached__': None, 'a': 10, 'function': <function function at 0x000001D740ED64C8>}
{'a': 11}
{'a': 12}

These 3 dictionaries represent 3 different namespaces where the key of 'a' in different namespaces has different values. Something special about globals is that it is not read-only meaning we can add a key-value pair or update the existing key-value pair to the dictionary like

globals()['a'] = 13
print('a =', a) # The output will be 'a = 13'.

Class and Instance Namespaces

Things are a bit different when it comes to classes and instances. Unlike what is covered above, accessing an instance attribute or a class attribute requires a prefix typically with the keyword self. For example, what is the expected output when we run the snippet as below?

class A:
    a = 1
    def __init__(self, val):
        print(a)
        self.a = val
        print(a)
_ = A(2)

The answer is that it will comes with a NameError: name 'a' is not defined in line 4. Because of the scope resolution, a is first searched in the constructor's local namespace and then searched in the global namespace while it doesn't access any class attribute at all. Another great example is

class A:
    a = 10
    b = list(a + i for i in range(10))

There will be also a NameError: name 'a' is not defined in line 3 since what is inside the list is a generator expression. In Python, generator expressions are implemented in a function scope. The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods - this includes generator expressions in list comprehensions.

But if we change print(a) into print(self.a) in line 4 and line 6, then we will get 1 and 2 as the output.

class A:
    a = 1
    def __init__(self, val):
        print(self.a)
        self.a = val
        print(self.a)
_ = A(2)

The first self.a in line 4 references the class variable defined in line 2 while the second self.a in line 6 references the instance variable defined in line 5. If we delete the second line and there is no such class variable,

class A:
    def __init__(self, val):
        print(self.a)
        self.a = val
_ = A(2)

then the search will proceed to look in its base class which is the Object class to see if there is such a variable called a.

Generally speaking,

if the same attribute name occurs in both an instance and in its corresponding class, then attribute lookup prioritizes the instance.
If the attribute is not found in the instance attributes, then the search proceeds to look in the class attributes.
If the attribute is not found in the class, then the search proceeds to look in the base class.
If this class has multiple parent classes, then the search follows a Method Resolution Order(MRO) to continue. This can be simply thought of as a depth-first, left-to-right and not searching twice in the same class procedure.

Here is an official documentation explaining the resolution of names just for reference.

Appendix: `@classmethod` and `@staticmethod`

Technically, this is not the content of namespaces and scopes in Python, but it is kind of similar to accessing method attributes from an instance level and a class level.

class A:
    def foo(arg1, arg2):
        print(arg1, arg2)

Can we call foo method from the class level and from the instance level?

>>> A.foo(1, 2) 
1 2
>>> A().foo(2)  
<__main__.A object at 0x000001F51417C4C8> 2

If foo is called as an instance method, then there is only 1 parameter remaining since arg1 is treated as the object. That is to say, A().foo(2) is equivalent to A.foo(A(), 2). But if there is a function without any argument, then we cannot call this function from an instance level.

class A:
    def foo():
        print("Unbound Methods.")

Instances of A cannot call foo because foo is an unbound method (while unbound methods are removed in Python 3). Methods that do not have an instance of the class as the first argument are known as unbound methods. Here comes the @staticmethod decorator.

class A:
    @staticmethod 
    def foo():
        print("Static Methods.")

For now, @staticmethod makes foo bound to instances of A and we will get consistent results if we call foo either from a class or from an instance. Actually, @staticmthod is to ensure the consistency and to make instances work smoothly with this method (removing @staticmethod doesn't influence calling foo from a class).

>>> A.foo()
Static Methods.
>>> A().foo()
Static Methods.

Static methods do not require instance creation and are typically called from a class level. They are faster and usually implemented as utility functions. This is similar to where static methods are used in other languages like JAVA.

Where are @classmethod used in Python? Let's look at an example

class Date:
    def __init__(self, day=1, month=1, year=1970):
        self.day = day
        self.month = month
        self.year = year
    
    @classmethod
    def fromString(cls, dateStr):
        day, month, year = dateStr.split('-')
        return cls(day, month, year)

As we can see, the above method decorated by classmethod takes a date in string and returns an instance of Date. That's where @classmethod decorators are mostly used. Typically, class methods take cls as the first argument mostly to return a new object. There are some examples in Python like

>>> dict.fromkeys("ab")
{'a': None, 'b': None}

When are static methods and class methods to be introduced? When the method only depends on its parameters while the instance state has no effect on the method behavior but it's something relative to the class. Then the method can be implemented as static. If it doesn't have anything to do with the class, then it's better to implement it as a standalone function. As for class methods, cls must be used in the implementation otherwise static methods or standalone functions are better.

Here is a table displaying the difference between static methods and class methods

Class Methods	Static Methods
Take `cls` as first argument.	Can go without any parameter.
Can access and modify the class state.	Cannot access or modify the class or the instance state.
Mostly implemented as factory methods.	Mostly implemented as utility methods.

Appendix 2: Single and Double underscores

Single Underscore

Names, in a class, with a leading underscore like _foo are simply to indicate that the attribute or method is intended to be private even though there is nothing really private in Python. From PEP-8

_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose names start with an underscore.
single_trailing_underscore_: used by convention to avoid conflicts with Python keyword, e.g.
def __init__(self, class_='class')

Double Underscore (Name Mangling)

__double_leading_underscore: when naming a class attribute, invokes name mangling (inside class FooBar, __boo becomes _FooBar__boo.

For example,

class A:
    def __foo(self):
        pass
>>> a = A()
>>> a.__foo()
AttributeError: 'A' object has no attribute '__foo'
>>> a._A__foo()

Name mangling is intended to give classes an easy way to define “private” instance variables and methods, without having to worry about instance variables defined by derived classes, or mucking with instance variables by code outside the class. Polymorphism is therefore associated with name mangling to access corresponding methods for different instances. This also reflects that all methods in Python are virtual.

__double_leading_and_trailing_underscore__: "magic" objects or attributes that live in user-controlled namespaces. E.g. __init__, __import__ or __file__. Never invent such names; only use them as documented.