SQL in Python with Pandas: The Complete Workflow (2026)

SQL in Python with Pandas: Quick Answer

The 10-Million-Row Freeze

A common early mistake looks almost reasonable: df = pd.read_sql("SELECT * FROM orders", conn), followed by df[df['order_date'] >= '2026-01-01'] to filter down to what's actually needed. The query runs. The laptop's memory climbs as ten million rows load into a DataFrame. The notebook freezes for a full minute before the filter — which needed maybe 2% of those rows — finally runs.

The fix isn't a pandas trick. It's moving the WHERE clause into the SQL query itself, so the database filters before sending anything across the network: pd.read_sql("SELECT * FROM orders WHERE order_date >= '2026-01-01'", conn). Same result, a fraction of the data moved, no freeze. This one habit — filter in SQL, not after loading — is the single highest-leverage lesson in this article.

Connecting With SQLAlchemy

SQLAlchemy provides a consistent connection interface across different databases, so the same pandas code works whether the target is PostgreSQL, MySQL, or another supported database.

from sqlalchemy import create_engine
import pandas as pd

# PostgreSQL connection string
engine = create_engine("postgresql://user:password@localhost:5432/mydb")

# MySQL connection string
engine = create_engine("mysql+pymysql://user:password@localhost:3306/mydb")

Loading Data With read_sql

pandas.read_sql() runs a query and returns the result directly as a DataFrame — no manual cursor handling, no intermediate CSV export.

query = """
    SELECT customer_id, order_date, order_total
    FROM orders
    WHERE status = 'shipped'
"""

df = pd.read_sql(query, engine)
print(df.head())

Every technique from earlier in this cluster — WHERE, JOIN, GROUP BY, window functions — belongs inside that query string. The DataFrame that comes back should already be close to the shape needed for analysis, not raw data waiting to be reshaped in pandas.

SQL Injection and Parameterized Queries

Building a query with an f-string and a variable feels natural in Python — and it's a genuine security risk the moment any part of that variable comes from outside the script.

# DANGEROUS: never build SQL with an f-string and external input
customer_id = get_user_input()
query = f"SELECT * FROM orders WHERE customer_id = {customer_id}"
df = pd.read_sql(query, engine)

# SAFE: parameterized query — value passed separately from the query text
query = "SELECT * FROM orders WHERE customer_id = %(cust_id)s"
df = pd.read_sql(query, engine, params={"cust_id": customer_id})

The f-string version lets a malicious or malformed input change the actual structure of the query, not just its value — the textbook definition of a SQL injection vulnerability. The parameterized version passes the value through a separate channel entirely, so it's always treated as data, never as executable SQL.

Writing Data Back With to_sql

The reverse direction — pushing a processed DataFrame back into a database — uses DataFrame.to_sql().

summary_df.to_sql(
    "monthly_summary",
    engine,
    if_exists="replace",   # or 'append', or 'fail'
    index=False
)

The if_exists parameter is worth setting deliberately every time. "fail" raises an error if the table already exists — the safest default while developing. "replace" drops and recreates the table entirely. "append" adds new rows to an existing table. Leaving this at its default in a production script is a common way to accidentally overwrite data that should have been appended.

SQL vs Pandas: Where to Do the Work

This is the practical decision that determines whether a script is fast or slow, and it closes the loop on everything the query optimization article covered earlier in this cluster.

The rule of thumb: anything the database can do with an index — filtering, joining, aggregating — belongs in the SQL query. Anything that needs Python's broader ecosystem — statistical models, visualization, machine learning — belongs in pandas, working on the already-reduced result the database handed back.

Chunking Large Result Sets

Even a properly filtered query can return more rows than comfortably fit in memory. The chunksize parameter turns read_sql into an iterator instead of a single large load.

for chunk in pd.read_sql("SELECT * FROM events", engine, chunksize=50000):
    process(chunk)  # handle 50,000 rows at a time

Each chunk is processed and can be discarded before the next one loads, keeping peak memory usage bounded regardless of how large the total result set is — the standard pattern for working with query results too large to load all at once.

Mistakes That Give Away a Beginner

• SELECT * then filtering in pandas. The exact trap from this article's opening — push the WHERE clause into SQL instead.
• Building queries with f-strings and external input. A genuine SQL injection risk — use parameterized queries instead, every time.
• Leaving if_exists at its default in to_sql. An easy way to accidentally overwrite or fail against an existing table in production.
• Creating a new engine for every query instead of reusing one connection pool across a script.
• Loading an entire large table without chunksize and running into an out-of-memory error that chunking would have avoided entirely.

Practice: Build the Workflow Yourself

Using a local PostgreSQL database with an orders table, try building this end to end:

Frequently Asked Questions

Conclusion: Cluster Complete

The core habit from this article carries the whole cluster's philosophy forward: let SQL do what SQL is good at — filtering, joining, aggregating with an index behind it — and let pandas do what pandas is good at, working on the already-reduced result. Parameterize every dynamic query, set if_exists deliberately, and chunk anything too large to hold in memory at once.

That's the fifteenth and final article in this cluster — from a single SELECT statement all the way to running SQL safely and efficiently inside a real Python workflow.