X-Ray Vision: Logging in Python
Most Python developers first notice the need for a robust logging system when simple print() statements become cluttered and fail to provide a coherent view of application behavior. In larger systems, especially in DevOps environments, the stakes are even higher. You need clarity about how the code is running, so you can diagnose bugs, monitor performance, and confirm that everything is working as intended. Python’s built-in logging module is an excellent way to achieve that clarity, but it does require some setup and understanding.
This post explores the essentials of Python logging, including:
Root loggers and why you might not want to stick with them
Flexible logging formats
How to set the appropriate log levels, plus creating custom levels
Examples for audiences familiar with Python and looking to improve observability skills
When you implement logging in a systematic way, it’s like having X-ray vision into your application. You’ll see exactly what’s going on and gain the power to spot issues before they become showstoppers. Let’s dive in.
Good logging gives you x-ray vision into your Python code.
Where It All Starts: The Python Logging Mental Model
The logging module revolves around a few key concepts:
Logger: The entry point for emitting log messages. You create or retrieve one with a call to
logging.getLogger(name).Handlers: Determine where log messages go, such as to a console, a file, or a remote logging server.
Formatters: Define how log records look, including timestamps and metadata.
Levels: Indicate the importance or severity of each log message (
DEBUG,INFO,WARNING,ERROR,CRITICAL).
When you emit a log message, the logger checks if that message meets or exceeds the logger’s threshold level, then passes it to one or more handlers, which format and deliver the message to the desired output. Because you can configure all these pieces independently, you can have multiple loggers for different parts of your application, multiple handlers that write to different places, and custom formats for each output.
From a DevOps standpoint, this is ideal. A microservice can write INFO or DEBUG logs to a file for deep analysis but send ERROR logs to a pager or monitoring system. Since logs are a cornerstone of observability (along with metrics and traces), having flexible configuration is crucial for diagnosing issues in real-world systems.
Spotting the Root Logger
When you import Python’s logging module and call something like logging.debug("Hello from root logger"), you’re using the root logger. By default, the root logger is set to WARNING. That’s why a quick test with logging.debug("You won’t see me") produces no output. If you call:
import logging
logging.debug("This debug message won’t appear by default")
logging.info("Nor will this info message")
logging.warning("This is a warning message")You’ll only see:
WARNING:root:This is a warning messageon your console. The WARNING threshold filters out DEBUG and INFO.
Configuring the Root Logger
For small scripts, you can configure the root logger with logging.basicConfig(). For example:
import logging
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s [%(levelname)s] %(message)s"
)
logging.debug("Now you'll see me")
logging.info("And this info message too")Here, level=logging.DEBUG means all messages at DEBUG level or above will appear. The format string includes a timestamp (asctime), the log level (levelname), and your actual message (message).
Why Named Loggers Are Better
Although the root logger can suffice for smaller projects, most larger applications or multi-module codebases benefit from named loggers. These let you label logs according to their source module or functionality:
logger = logging.getLogger("my_module")
logger.setLevel(logging.INFO)Any log you emit from logger is tagged with "my_module". In a DevOps environment with multiple modules and microservices, it’s far easier to debug if each log entry includes the component or service name. If the project grows and you decide to redirect logs from a specific module to a particular handler, it’s straightforward with named loggers.
Building Your Own Logging Format
The format of your logs is more than an aesthetic choice. The way you structure log lines can make them easier to filter, search, and parse. Python allows you to include placeholders that map to different attributes of the log record, such as:
%(asctime)s: Timestamp%(name)s: Logger name%(levelname)s: Log level (DEBUG, INFO, etc.)%(message)s: The actual message text%(pathname)s: The file path of the source code%(lineno)d: The line number in the source code
Here’s an example:
import logging
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s"
)
logging.debug("Initializing the app")A log line might look like:
2025-04-04 12:15:32,045 [DEBUG] root:5 - Initializing the appNotice the timestamp, level, logger name, line number, and message. In a production environment, having timestamps and contextual data is invaluable for diagnosing issues.
Setting the Appropriate Log Levels
Python has five built-in levels to indicate severity or importance:
DEBUG (10): Detailed information for diagnosing problems.
INFO (20): Confirmation that things are working as expected.
WARNING (30): An indication that something unexpected happened, but the program can continue.
ERROR (40): A more serious issue that prevented some part of the program from running.
CRITICAL (50): A severe error that may stop the application entirely.
If you set a logger’s level to WARNING, for instance, then any logs at INFO or DEBUG will be ignored. This is essential in production systems because you generally want fewer logs, focusing on warnings and errors, while in development you want more verbosity.
Example: A Simple Logger with a Level
import logging
logger = logging.getLogger("my_app")
logger.setLevel(logging.INFO)
logger.debug("You won't see this")
logger.info("This is an info message")
logger.warning("This is a warning message")You’ll see only:
This is an info message
This is a warning messageThe debug message is hidden because the level is set to INFO.
Creating and Using Custom Levels
The built-in five levels work well for many cases, but sometimes you need a specialized event. Python allows you to add custom levels using integer values. Behind the scenes, levels map to numeric codes. For example, DEBUG is 10 and INFO is 20. You can pick another number and define a custom level:
import logging
SECURITY_AUDIT = 25
logging.addLevelName(SECURITY_AUDIT, "SECURITY_AUDIT")
def security_audit_log(self, message, *args, **kwargs):
if self.isEnabledFor(SECURITY_AUDIT):
self._log(SECURITY_AUDIT, message, args, **kwargs)
logging.Logger.security_audit = security_audit_log
logger = logging.getLogger("security_logger")
logger.setLevel(logging.INFO)
logger.security_audit("User updated their password")Here, SECURITY_AUDIT sits between INFO (20) and WARNING (30). If the logger’s threshold is INFO, you’ll see SECURITY_AUDIT messages. This can be incredibly useful if, for example, you want a separate handler for all audit-related events.
Wrangling Handlers for Multiple Outputs
A big advantage of Python logging is the ability to attach multiple handlers to a single logger. Suppose you want to:
Write all logs (DEBUG or higher) to a file.
Send warning and error logs to your console for immediate visibility.
You could do:
import logging
logger = logging.getLogger("multi_output")
logger.setLevel(logging.DEBUG)
# File handler for all messages
file_handler = logging.FileHandler("app.log")
file_handler.setLevel(logging.DEBUG)
file_format = logging.Formatter(
"%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s"
)
file_handler.setFormatter(file_format)
# Console handler for warnings and above
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.WARNING)
console_format = logging.Formatter("%(levelname)s - %(message)s")
console_handler.setFormatter(console_format)
# Add handlers
logger.addHandler(file_handler)
logger.addHandler(console_handler)
logger.debug("Debug info, will appear in file only")
logger.warning("Warning, will appear in file and console")
logger.error("Error, will appear in both too")When you run this:
The debug message goes to
app.logbut not to the console.The warning and error messages appear in both places.
In DevOps scenarios, you can forward logs to specialized aggregators (like Fluentd or Logstash), or you can have a handler that sends only errors and critical messages to PagerDuty or Slack.
Practical Logging Patterns in DevOps
Distributed Logging
When your application is spread across multiple servers or containers, collecting logs in a single place is essential. You can configure Python handlers to send logs via HTTP to a central server, or rely on Docker or Kubernetes logging drivers that forward logs automatically. Tools like Elastic Stack and Splunk help you store, visualize, and search logs from multiple services.
Log Rotation and Retention
Log files can grow huge over time. Python includes rotating handlers, such as RotatingFileHandler or TimedRotatingFileHandler, to automatically rotate logs. You can limit them by size (e.g., 5 MB) or time (e.g., once per day), keeping only a specified number of backups.
JSON or Other Structured Formats
Instead of raw text, you can output logs as JSON. This makes them easier to parse in search-based logging tools. By storing logs in a structured format, you can query them by specific fields (like log level, timestamp ranges, user ID, and so on).
Sensitive Data Redaction
Logs might contain credentials or personal data. In a DevOps context, it is critical to sanitize or redact logs as necessary. You can build filters into your logging configuration that remove or hash sensitive strings before they’re written to disk or sent over the network.
Example: Rotating File Handler
from logging.handlers import RotatingFileHandler
import logging
logger = logging.getLogger("rotating_example")
logger.setLevel(logging.INFO)
handler = RotatingFileHandler(
"myapp.log",
maxBytes=1024 * 1024 * 5, # 5 MB
backupCount=5
)
formatter = logging.Formatter("%(asctime)s [%(levelname)s] %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
for i in range(100000):
logger.info(f"Log line {i}")Once myapp.log hits 5 MB, it’s rotated out and renamed, then a new file is started. Old logs are retained up to the specified backup count.
Tuning Your Observability
Logging is just one piece of observability, which also includes metrics and tracing. Here are a few best practices:
Correlate logs with request IDs or user IDs: If you add an identifier to each log line, you can track a single request or user journey through multiple services.
Use environment-specific log levels: In development, you might set
DEBUGto see everything. In production, you might setWARNINGorERRORto reduce noise.Integrate with monitoring tools: Route logs to services like Datadog or Elasticsearch and pair them with dashboards that show real-time charts. This is crucial for understanding what’s happening across your ecosystem.
For advanced configuration options, check out the official Python Logging HOWTO. You’ll see more details on capturing warnings, configuring logger hierarchies, and controlling logs using config files.
Bringing Observability to AI-Generated Code
Many teams have started using AI-driven tools to help generate Python code. These tools can save time by sketching out functions and modules quickly, but the trade-off is that you might not fully understand every decision made by the AI. It can be tricky to figure out which lines are absolutely correct or where hidden bugs might lurk. That’s where Python logging comes in:
1. Verifying AI-Generated Logic
Once you copy an AI-suggested block of code into your application, add DEBUG or INFO logs around key functions to confirm the flow is doing what you (and the AI) expect. For instance, if the AI script is handling file uploads, log the file path, the bytes processed, or the time taken. This layer of “living documentation” gives you deeper visibility into code you didn’t write yourself.
2. Identifying Hidden Assumptions
AI-generated code might rely on assumptions you didn’t explicitly state. A log message can give you a heads-up when the code hits an unexpected data type or condition. For example, if the AI tool wrote code for a dictionary lookup, you can log what the dictionary keys are before and after modification. If the logs reveal strange keys or missing data, you know the AI-based logic needs revision.
3. Using Custom Log Levels for AI Blocks
Consider defining a custom level (perhaps AI_CHECK = 23) to highlight logs that specifically verify AI-generated segments. You can route these logs to a separate file, making it simple to isolate AI-related messages. Over time, you can dial this logging up or down, depending on how reliable you find the AI output.
4. Keep It Flexible
AI-generated code tends to evolve as you refine your prompts or training data. Because Python logging is so configurable, you can easily add new handlers or update existing ones to track fresh AI-related behaviors. Whether you need more detail or less, it’s just a matter of adjusting the level and format of your logs.
By inserting log statements in places where you rely heavily on AI-generated functionality, you’ll see exactly how that code responds under real usage. Examples include data transformations, API calls, and complex conditionals. You can combine this with the broader best practices already discussed in this post (handlers, formatters, rotation policies, etc.) to get a complete picture of your application.
Trying It Yourself: A Comprehensive Example
Below is a condensed script showing various features: multiple handlers, a custom level, and rotating logs for errors.
import logging
from logging.handlers import RotatingFileHandler
# Define a custom level
DATA_EVENT = 25
logging.addLevelName(DATA_EVENT, "DATA_EVENT")
def data_event_log(self, message, *args, **kwargs):
if self.isEnabledFor(DATA_EVENT):
self._log(DATA_EVENT, message, args, **kwargs)
logging.Logger.data_event = data_event_log
logger = logging.getLogger("data_processor")
logger.setLevel(logging.DEBUG)
# File handler for debug logs
debug_file_handler = logging.FileHandler("data_debug.log")
debug_file_handler.setLevel(logging.DEBUG)
debug_formatter = logging.Formatter(
"%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s"
)
debug_file_handler.setFormatter(debug_formatter)
logger.addHandler(debug_file_handler)
# Console handler for errors only
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.ERROR)
console_formatter = logging.Formatter("%(levelname)s: %(message)s")
console_handler.setFormatter(console_formatter)
logger.addHandler(console_handler)
# Rotating file handler for errors
error_rotating_handler = RotatingFileHandler(
"data_errors.log",
maxBytes=1024*1024,
backupCount=3
)
error_rotating_handler.setLevel(logging.ERROR)
error_rotating_handler.setFormatter(debug_formatter)
logger.addHandler(error_rotating_handler)
def process_data(records):
logger.debug("Starting data processing")
for record in records:
logger.data_event(f"Transforming record ID {record['id']}")
# Simulate an error
if record.get("invalid"):
logger.error(f"Error transforming record {record['id']}")
logger.info("Data processing complete")
if __name__ == "__main__":
sample_records = [
{"id": 1, "data": "abc"},
{"id": 2, "data": "def", "invalid": True},
{"id": 3, "data": "ghi"},
]
process_data(sample_records)data_debug.logcatches everything (DEBUG, INFO, DATA_EVENT, ERROR, etc.).data_errors.logand its rotated backups capture only errors.The console only displays errors.
DATA_EVENTmessages appear in the debug log.
This is the real power of Python’s logging. You can filter logs by level, direct them to specific locations, and format them so they’re more helpful for diagnosing issues.
Common Pitfalls and How to Avoid Them
Using
print()Instead oflogging: You lose the rich features of the logging system when you bypass it with simple prints.Ignoring Log Format: A bare message with no timestamp or context is nearly useless when something goes wrong.
Excessive Debug Logs in Production: Too many logs can drain performance or fill your disk quickly. Use environment-specific configurations or levels.
No Plan for Log Rotation: Without rotation, your logs will grow indefinitely. Use rotating handlers or external solutions to manage this.
Not Protecting Sensitive Data: Carelessly logging passwords, tokens, or personal info can be a security risk. Redact or mask such details before they go into logs.
Links and Resources to Level Up Further
If you’re aiming for DevOps-level sophistication, you’ll probably push logs to a centralized aggregator. That way, you can analyze logs from multiple services in one location, pair them with metrics, and set up alerts.
Putting It All to Work with Caparra
You’ll be surprised at how powerful Python logging can be once it’s set up correctly. It’s like having X-ray vision into your code. Logs let you trace what happened and when, which is invaluable when diagnosing production issues at 2 am. The combination of log levels, custom events, and multiple handlers is a recipe for an efficient, maintainable, and robust DevOps practice.
And that’s just the start. There’s a ton you can do with logs if you pair them with AI-driven DevOps tools. If you’d like to get started on automating some of the repetitive tasks that come with building and maintaining modern systems, consider opening a free Caparra account. We’d love to help you lighten the burden of daily toil so you can focus on what you do best: building awesome software.
Until then, enjoy your newly upgraded logging capabilities. With the right approach to Python logging, you’ll catch problems earlier, fix them faster, and gain a real-time window into the health of your applications. Let’s keep building code that’s easy to manage and fun to maintain. Happy logging!
