Skip to content

Volume ITechnology

3.13.11 — Modules, Packages & Environments ​

A new engineer clones the order service, runs python app/main.py, and gets ImportError: attempted relative import with no known parent package. They "fix" it by adding the folder to sys.path. Next, two modules that import each other start failing with cannot import name 'Order' from partially initialized module. Then a library upgrade for one project breaks another project on the same laptop. Every one of these problems comes from not knowing what import actually does, or where Python looks for packages. This page walks the import system step by step, then explains packages, circular imports, virtual environments and how a modern Python project is laid out.

1. What import does, step by step ​

A module is a file of Python code, and at run time it is an object of type module whose attributes are the file's top-level names. When Python runs import shop.pricing, it goes through these steps.

  1. Check the cache. Python looks in sys.modules, a dict from module name to module object. If the module is already there, it is reused and nothing else happens.
  2. Find it. If not cached, Python asks a list of finders, stored in sys.meta_path, whether they know where shop.pricing lives. Each finder returns a module spec, a small record saying "here is the module, and here is the loader that can create it".
  3. Create the module object. The loader creates an empty module object and fills in attributes such as __name__, __file__ and __spec__.
  4. Add it to the cache first. The new, still-empty module is stored in sys.modules before its code runs. Remember this step. It is the reason circular imports behave the way they do.
  5. Run the code. The loader compiles the file to bytecode, or loads the cached .pyc from 3.13.1, and executes it with the module's namespace as its globals. Every top-level def, class and assignment becomes a module attribute.
  6. Bind the name. Finally, the import statement binds a name in the importing module: shop for import shop.pricing, or add_tax for from shop.pricing import add_tax.

Here is a small shop package that lets us watch those steps.

text
project/
  demo.py
  shop/
    __init__.py         ← print("  running shop/__init__.py"); from .pricing import add_tax
    pricing.py          ← print("  running shop/pricing.py"); TAX_RATE = 0.18; def add_tax(...)
    payments/
      __init__.py
      upi.py
python
# demo.py
import sys
import shop                     # (1)
import shop.pricing             # (2)
import shop                     # (3)

print(shop.add_tax(100))
print("shop" in sys.modules, sys.modules["shop.pricing"] is shop.pricing)   # (4)
print(type(shop), shop.__name__)
print(shop.pricing.__spec__.name, type(shop.pricing.__spec__.loader).__name__)   # (5)
text
  running shop/__init__.py
  running shop/pricing.py
118.0
True True
<class 'module'> shop
shop.pricing SourceFileLoader
  • (1) Importing a package runs its __init__.py, which itself imports pricing. Both prints appear.
  • (2) shop.pricing is already in sys.modules, so nothing runs.
  • (3) Nor here. A module's code runs once per process, however many times and from however many files it is imported.
  • (4) Every importer receives the same module object from the cache.
  • (5) The spec records the module's name and the loader that created it. SourceFileLoader loads ordinary .py files.

That single-run behaviour is why a module works as a natural singleton. A settings object or a database engine created at the top of a module is shared by everyone who imports it. It is also why a slow top-level statement, such as connecting to a database at import time, slows down every program that imports the module, even one that never uses it.

Where Python looks: sys.path and the finders ​

sys.meta_path holds three standard finders, tried in order.

  1. BuiltinImporter finds modules compiled into the interpreter itself, such as sys and math.
  2. FrozenImporter finds modules frozen into the interpreter binary for fast startup, such as parts of the import system itself.
  3. PathFinder searches the folders listed in sys.path, in order, for a matching file or folder.

sys.path is built at startup from:

  • the folder of the script you ran, or the current folder for python -c and python -m;
  • the folders in the PYTHONPATH environment variable, if set;
  • the standard library folders;
  • the site-packages folder, where installed packages live.

The first match wins. That explains a classic mistake. Name your own file random.py or json.py, and every import json in your project, and in every library you use, finds your file first. Python 3.13 and later even add a hint to the error message: "consider renaming … if it has the same name as a library you intended to import".

importlib.reload ​

Because modules run once, editing a file does not affect a running program. importlib.reload(module) runs the module's code again in the same module object. It is handy in an interactive session, but objects created before the reload still point at the old classes and functions. In real services, restart the process instead.

2. Packages ​

A package is a folder of modules that can be imported as one name. A regular package is a folder containing an __init__.py file. When the package is imported, __init__.py runs, and whatever it defines becomes the package's attributes. Common uses are:

  • re-exporting the public names, as shop/__init__.py does with add_tax, so callers write from shop import add_tax instead of reaching into internal modules;
  • defining __all__;
  • leaving the file empty, which is perfectly fine.

A package has a __path__ attribute, the list of folders to search for its submodules. That attribute is what makes it a package and not just a module.

Python 3.3 added namespace packages (PEP 420): a folder without __init__.py can still be imported as a package. Their purpose is to let one package name be split across several folders, even several installed distributions. A company might install acme.billing and acme.shipping separately, both under the acme name. For your own application code, include __init__.py. Leaving it out by accident creates a namespace package, which works until a tool such as a type checker or a test runner treats the folder differently.

Absolute and relative imports ​

Inside a package, there are two ways to name a sibling module.

python
# shop/payments/upi.py
from shop.pricing import add_tax      # (1) absolute
from ..pricing import add_tax         # (2) relative

def charge(amount):
    return f"UPI charge {add_tax(amount)}"
  • (1) An absolute import names the full path from the top of the project.
  • (2) A relative import starts with dots. One dot means "this package", two dots mean "the parent package". It is resolved using the module's __package__ attribute, here "shop.payments".

Now the opening error. Run that file directly and it fails.

text
$ python shop/payments/upi.py
ImportError: attempted relative import with no known parent package

$ python -m shop.payments.upi
  running shop/__init__.py
  running shop/pricing.py

When you run a file by its path, Python treats it as a standalone script named __main__, with no package. So .. means nothing. python -m shop.payments.upi runs the same file as the module shop.payments.upi, imported through the normal system from the project root. Its package is known, so relative imports work. The correct fix is to run code with -m, or through an installed entry point (section 6), and never to edit sys.path by hand.

__all__ and import * ​

from module import * imports a module's public names. Which names are public? If the module defines a list called __all__, exactly those. Otherwise, every top-level name that does not start with an underscore.

python
from shop.pricing import *
print("add_tax" in dir(), "TAX_RATE" in dir(), "_round_paise" in dir())
text
True True False

import * is fine in an interactive session and in a package's __init__.py when re-exporting. In ordinary code it hides where names come from and can silently overwrite your own names. Linters warn about it. Even if nobody uses *, __all__ is still worth defining, because it documents the public API, and tools such as documentation generators read it.

3. Circular imports ​

Two modules that import each other are a common source of confusing errors. Here is the classic case: an order knows its customer, and a customer can find its latest order.

python
# shop/orders.py
from shop.customers import Customer          # (1)

class Order:
    def __init__(self, customer):
        self.customer = customer
python
# shop/customers.py
from shop.orders import Order                # (2)

class Customer:
    def latest_order(self):
        return Order(self)
text
$ python -c "import shop.orders"
ImportError: cannot import name 'Order' from partially initialized module 'shop.orders' (most likely due to a circular import)

Walk through it using step 4 from section 1.

  1. import shop.orders creates an empty shop.orders module, adds it to sys.modules and starts running its code.
  2. Line (1) imports shop.customers. That module is created, cached, and starts running.
  3. Line (2) asks for Order from shop.orders. The module is found in the cache, so it is not run again. But it is still stuck at line (1), and class Order has not been executed yet.
  4. So Order does not exist on the module, and the import fails. The message calls it partially initialized, which is exactly right.

Step 4 in section 1 is what stops this from being an infinite loop. Without the early caching, each module would import the other forever.

There are four fixes, from best to worst.

  1. Remove the cycle. Often two modules that need each other are really one idea split badly, or both need a third thing. Move the shared piece into its own module that both import.
  2. Import the module, not the name. import shop.orders in customers.py, and use shop.orders.Order inside the method. The module object already exists in the cache, and the name Order is looked up later, when the method actually runs. By then both modules have finished loading.
  3. Import inside the function that needs it. That delays the import until the first call. It works, but it hides a dependency and repeats a dict lookup on every call.
  4. Import only for type hints. If one module needs the other only for annotations, put the import under if TYPE_CHECKING:. That block never runs at all, but type checkers read it. 3.13.12 shows it.

4. Virtual environments: one set of packages per project ​

Installed packages live in a site-packages folder. If every project on your laptop shares one, then project A needing pydantic version 1 and project B needing version 2 cannot both work. A virtual environment, or venv, gives each project its own site-packages while sharing the Python interpreter.

text
$ python -m venv .venv          # or: uv venv
$ ls .venv
pyvenv.cfg   Scripts/ (bin/ on Linux and macOS)   Lib/site-packages/

How does it work? There is no container or copy of Python. There are just three ideas.

  1. .venv/Scripts/python (or .venv/bin/python) is a small link or launcher that points at the real interpreter.
  2. Next to it sits pyvenv.cfg. When Python starts, it looks for this file one folder up from its own executable. If it finds one, it knows it is running inside a venv.
  3. It then sets sys.prefix to the venv folder and puts the venv's site-packages on sys.path instead of the global one.

Here is a real pyvenv.cfg.

text
home = C:\Users\asha\AppData\Local\Python\pythoncore-3.14-64
implementation = CPython
version_info = 3.14.6
include-system-site-packages = false

home is where the real interpreter lives. include-system-site-packages = false means the global packages are invisible. From inside Python you can always check where you are.

python
import sys
print(sys.prefix != sys.base_prefix)     # (1)
text
True
  • (1) Inside a venv, sys.prefix is the venv and sys.base_prefix is the real Python installation.

"Activating" a venv with source .venv/bin/activate only puts .venv/bin first on your shell's PATH, so that typing python finds the venv's interpreter. Activation is a convenience. Running .venv/bin/python app.py directly works just as well, and is what Docker images and servers usually do.

Installing packages is covered in 3.10. The short version: pip install downloads and unpacks packages into the active environment's site-packages. uv, written in Rust, does the same job ten to a hundred times faster, also creates venvs, manages Python versions and keeps a lockfile. It has become the default choice for new projects.

5. A modern project layout ​

Since PEP 518 (2016) and PEP 621 (2020), a Python project describes itself in one file, pyproject.toml, written in the TOML format from 3.10. Here is a FastAPI service's file.

toml
[project]
name = "order-service"
version = "1.4.0"
requires-python = ">=3.12"
dependencies = [                               # (1)
    "fastapi>=0.115",
    "sqlalchemy[asyncio]>=2.0",
    "pydantic-settings>=2.0",
]

[project.optional-dependencies]
dev = ["ruff", "mypy"]                          # (2)

[project.scripts]
order-service = "order_service.cli:main"        # (3)

[build-system]
requires = ["hatchling"]                        # (4)
build-backend = "hatchling.build"
  • (1) The packages the service needs at run time, with version ranges.
  • (2) Extra groups installed only on request, here developer tools.
  • (3) An entry point. Installing the project creates a command called order-service that imports order_service.cli and calls main().
  • (4) The build backend, the tool that turns the project into an installable package. Hatchling, setuptools, flit and uv's own backend all read this same file.

And the folder layout most teams use, called the src layout:

text
order-service/
  pyproject.toml
  uv.lock                  ← exact versions of every dependency
  src/
    order_service/
      __init__.py
      main.py              ← creates the FastAPI app
      cli.py
      orders/
        __init__.py
        models.py
        service.py

Why put the package inside src/? Because then the project folder itself is not on sys.path, so import order_service only works after the package is properly installed into the venv. In development you install it in editable mode with uv pip install -e . or pip install -e ., which makes the venv point at your source folder so edits take effect immediately. This catches packaging mistakes, such as a forgotten file, on your laptop instead of in production. It also removes the temptation to fix imports by editing sys.path.

6. Running code the right way ​

Putting the pieces together, here are the three correct ways to run code in a package.

  1. python -m order_service.main runs a module by its import name, with the package context set up, so relative imports work.
  2. order-service runs the entry point command created at install time.
  3. uvicorn order_service.main:app asks a server to import your module by name and fetch the app object from it. 3.14.1 explains what Uvicorn does with it.

All three import your code by its package name from an installed environment. None of them need sys.path hacks.

What the interviewer will push on ​

"What happens when you import a module twice?" It runs once. The second import finds it in sys.modules and returns the cached object. Strong candidates add the consequence: modules act as singletons, and slow work at the top of a module slows every importer.

"Why do circular imports fail with 'partially initialized module'?" Because a module is added to sys.modules before its code runs. The second module receives the half-finished first module and asks for a name that has not been defined yet. List the fixes in order: remove the cycle, import the module rather than the name, import lazily, or use TYPE_CHECKING.

"What is the difference between a module and a package?" A package is a module with a __path__, usually a folder with __init__.py. Mention namespace packages and why application code should still use __init__.py.

"How does a virtual environment work?" pyvenv.cfg next to the interpreter link, sys.prefix pointing at the venv, and the venv's own site-packages on sys.path. Activation only edits PATH. The common wrong answer is that a venv copies Python or isolates the operating system.

"Why does python app/main.py fail with a relative import error but python -m app.main works?" Running by path makes the file __main__ with no package. -m imports it by its package name, so __package__ is set and the dots can be resolved.

Something to volunteer that nobody asks for: the src layout plus an editable install means import order_service only works through the installed package, so packaging bugs show up on your machine instead of in the deployed container. And it removes every reason to touch sys.path.

Recall

  • import checks sys.modules, asks the finders in sys.meta_path for a module spec, has the loader create the module, caches it before running it, runs its code once and binds the name.
  • sys.path = script folder, PYTHONPATH, standard library, site-packages. First match wins, so never name a file json.py.
  • A package is a module with __path__: a folder with __init__.py. Namespace packages have no __init__.py. Relative imports use __package__. Run package code with python -m.
  • __all__ lists public names for import * and documentation.
  • Circular imports see a partially initialized module. Fix by removing the cycle, importing the module instead of the name, importing lazily, or using TYPE_CHECKING.
  • A virtual environment is a pyvenv.cfg plus its own site-packages. sys.prefix != sys.base_prefix inside one. Activation only changes PATH.
  • pyproject.toml holds the project's metadata, dependencies, entry points and build backend. Use the src layout with an editable install.

Self-test: Which import step makes circular imports fail instead of looping forever? Why does a module behave like a singleton? What is the one file that makes a folder a venv? Why does the src layout catch packaging mistakes?

Quiz Bank ​

FoundationalYou edit a module while a Python program is running. Why doesn't the running program see the change?

The module's code ran once, when it was first imported, and the resulting module object is cached in sys.modules. Later imports, and all existing references, use that cached object. Python never re-reads the file on its own. importlib.reload re-runs the code in the same module object, but objects created earlier still refer to the old functions and classes. Development servers such as uvicorn --reload instead watch the files and restart the whole process.

AppliedA script named random.py in your project crashes with AttributeError: module 'random' has no attribute 'choice' inside a third-party library. Explain.

The folder of the script you run is placed first on sys.path. When the library executes import random, the path finder finds your random.py before the standard library's random module. Your file is imported as random and cached in sys.modules, so every module in the process, including the library, now gets your file, which has no choice function. Rename the file, and delete any random.cpython-*.pyc in __pycache__ next to it.

InterviewTwo modules in a package import each other's classes at the top. Give two fixes that keep both classes where they are.

First, import the module instead of the name: in customers.py, write import shop.orders at the top and use shop.orders.Order inside methods. The module object exists in sys.modules from the start of the import, and the attribute Order is looked up only when the method runs, after both modules have finished loading. Second, if Order is only needed for type annotations, write from typing import TYPE_CHECKING and put from shop.orders import Order inside if TYPE_CHECKING:. That code never runs, but type checkers read it, and annotations are not evaluated at run time in 3.14. A third option, importing inside the method, also works but hides the dependency.

StaffYour company has twenty Python services and every one handles dependencies differently. What standard would you set, and why?

One pyproject.toml per service, using PEP 621 metadata and a pinned build backend. One tool, such as uv, for creating environments and resolving dependencies, with a committed lockfile so every machine and every Docker build installs exactly the same versions. Top-level dependencies are declared with sensible ranges, while the lockfile pins everything below them. Use the src layout with an editable install for development, and entry points instead of loose scripts. CI checks that the lockfile is up to date and runs a vulnerability audit. Docker images install from the lockfile only, into a venv inside the image. The payoff is that "works on my machine" disappears, upgrades become reviewable lockfile changes, and a security fix can be rolled out to all twenty services the same way.

Next: the TYPE_CHECKING trick only makes sense once you know what type hints are and who reads them. 3.13.12 covers static typing in Python from the first PEP to the features FastAPI depends on.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Volume I — Technology
Part 0 — Retention & Notation
Part 1 — Foundations of Computation
Part 2 — Operating Systems
2.1 The Kernel
2.2 Processes
2.3 Threads & Scheduling
2.4 Concurrency Under the Hood
2.5 Memory Management
2.6 File Systems
2.7 Input/Output
2.8 Linux for Engineers
2.9 Windows & macOS in Practice
Part 3 — Languages, Compilers & Runtimes
3.6 JavaScript & V8
3.7 TypeScript
3.8 Node.js Internals
3.13 Python & CPython
3.14 FastAPI & Production Python
Part 4 — Data Structures & Algorithms
4.0 Classic Problems — Before NeetCode
4.4 Arrays & Hashing — Problems
4.5 Two Pointers
4.6 Sliding Window
4.8 Stack — Problems
4.9 Linked List — Problems
4.11 Binary Search — Problems
4.13 Trees
4.14 Tree — Problems
4.15 Trie — Problems
4.17 Heap — Problems
4.18 Backtracking — Problems
4.19 Graphs
4.20 Graph — Problems
4.21 Advanced Graph — Problems
4.23 1-D DP — Problems
4.24 2-D DP — Problems
4.26 Greedy — Problems
4.27 Interval — Problems
4.28 Math & Geometry — Problems
4.30 Bit Manipulation — Problems
Part 5 — Networking & the Web
5.3 IP
5.4 Transport
5.6 HTTP
Part 6 — Frontend Engineering
6.1 Browser Internals
6.2 HTML & CSS Mastery
6.3 JavaScript in the Browser
6.4 React, from First Component to Full Shop
6.5 Frameworks & Rendering
6.8 UI Engineering & Patterns
Part 7 — Databases & Storage
7.2 SQL Mastery
7.3 Storage Engine Internals
7.4 Transactions
7.5 NoSQL Taxonomy
7.8 Analytics & Big Data
Part 8 — Security, Identity & Cryptography
8.2 Cryptography
8.3 Protocols of Trust
8.4 Identity: AuthN & AuthZ
8.5 Application Security
8.6 Infrastructure Security
Part 9 — Software Design (LLD)
9.2 OOP Foundations
9.3 Design Principles
9.4 Design Patterns
9.5 Concurrency
9.6 API Design
9.7 LLD Interview Machines
9.9 Node/Express Production Playbook
Part 10 — Distributed Systems & HLD
10.7 Consistency & Consensus
10.8 Messaging & Microservices
10.14 Caching
Building blocks from the design seat
Part 11 — System Design Case Studies
Part 12 — AI & ML Engineering
12.1 What AI Engineering Is
12.2 The Math You Need
12.3 Classical Machine Learning
12.4 Data for ML
12.5 Deep Learning
12.6 Embeddings & Vector Search
12.7 Transformers & LLMs
12.8 Building with LLMs
12.9 Retrieval-Augmented Generation
12.10 Agents
12.11 Adapting Models
12.12 Production AI
12.13 Multimodal AI
12.14 AI Security & Governance
12.15 Case Studies & Capstone
Part 13 — Cloud, DevOps & Infrastructure
13.11 Git, from First Commit to Team Workflow
Part 14 — Containers: Docker & Kubernetes
14.1 Foundations
14.2 Docker
14.3 Kubernetes
14.4 Kubernetes on Azure
Part 15 — The Craft & Career
Nothing picked yet