Pandas becomes useful for trading research when you make time, price fields and table grain explicit. Sort each instrument’s timestamps, define the session and resampling bucket, aggregate OHLCV fields deliberately, keep missing observations visible, shift features before they become decision inputs and validate joins. These steps make transformations reproducible; they do not prove that the underlying data or strategy is sound.

Start with a table that knows what one row means

Before writing a pandas expression, state what one row represents. For a candle table, the minimum contract is:

If several securities share a timestamp, the grain is still one row per (symbol, timestamp), not one row per timestamp. Sort within that grain before applying a time-series operation:

df = (
    df.assign(timestamp=pd.to_datetime(df["timestamp"]))
      .sort_values(["symbol", "timestamp"])
)

Do not assume that a symbol is a permanent identity. Retain the source identifier and state the timestamp convention. A 09:20 label may mean a bar beginning at 09:20 or a bar that finished at 09:20; those are different inputs to a signal.

Set the Indian-market clock before resampling

Pandas does not know what an NSE session means. NSE lists the normal/odd-lot cash-market window as 09:15 to 15:30, while its auction and post-close pages describe separate market states. A candle tutorial should therefore use a named regular-continuous window and keep other states separate. (1), (2)

The full market-state explanation belongs in Market Mechanics for Code-First Traders, and source/session ingestion belongs in Building a Clean NSE Data Pipeline.

For a naive timestamp, tz_localize("Asia/Kolkata") attaches a timezone without moving the displayed clock time. For an aware timestamp, tz_convert("Asia/Kolkata") changes the timezone of the same instant. (3)

local = pd.to_datetime(df["timestamp"]).dt.tz_localize("Asia/Kolkata")
converted = aware_timestamp.dt.tz_convert("Asia/Kolkata")

If the source timezone is unknown, keep the rows out of the canonical table until the convention is resolved. A timezone conversion does not prove that the value was available at the time a strategy would have acted.

Resample OHLCV with named bucket rules

Resampling changes the observation interval. For each bucket (B_t), the usual OHLCV mapping is:

OPEN   = FIRST OPEN IN B_T
HIGH   = HIGHEST HIGH IN B_T
LOW    = LOWEST LOW IN B_T
CLOSE  = LAST CLOSE IN B_T
VOLUME = SUM OF VOLUME IN B_T

This is a local calculation, not an exchange-published candle. Pandas’ resample API requires a datetime-like index or datetime-like field and exposes origin, offset, closed and label; record those choices instead of inheriting a hidden default. (4)

ohlcv_map = {
    "open": "first",
    "high": "max",
    "low": "min",
    "close": "last",
    "volume": "sum",
}

df_30m = (
    df.set_index("timestamp")
      .groupby("symbol")
      .resample(
          "30min",
          origin="start_day",
          offset="15min",
          closed="left",
          label="left",
      )
      .agg(ohlcv_map)
      .dropna(subset=["open"])
      .reset_index()
)

The 15min offset aligns the first bucket to 09:15 rather than a midnight-derived 09:00 boundary. That is correct only for this stated session convention. Record the input and output frequency, timezone, session inclusion rule and treatment of partial or empty buckets.

Worked synthetic example: the bucket rule changes the candle

The following values are synthetic and are not NSE observations:

With buckets aligned to 09:15, the output is:

Default 30-minute bins beginning at 09:00 produce different OHLC values and volumes of 300 and 700 because the rows fall into different buckets. Neither result is a pandas error. The bucket convention is part of the data definition and should be recorded with the output.

Calculate returns without confusing fractions and percentages

For a close (C_t), the simple return is:

R_T = (C_T / C_(T-1)) - 1

pct_change returns the fractional value. Multiply by 100 only when displaying a percentage, and leave the first observation as NaN because it has no prior close. Current pandas documentation requires fill_method=None; do not rely on implicit filling of missing prices. (5)

df["return_1"] = (
    df.groupby("symbol")["close"]
      .pct_change(fill_method=None)
)

For closes of 100, 101, 99 and 102:

Record the return type, holding interval and price basis (raw, vendor_adjusted or locally_adjusted). A return column is not a win-rate or profit-factor metric, and it does not settle whether the source series is complete. The broader adjustment and point-in-time audit belongs in Topic 8.

Build rolling features and align their availability

A 20-row window on 5-minute bars is not a 20-session moving average. Integer windows count observations; elapsed-time windows use timestamps. Missing bars can change what a row-count window means, so record the window type, size and min_periods. Pandas also exposes endpoint and label choices for rolling windows. (6)

df["close_ma_20"] = (
    df.groupby("symbol")["close"]
      .rolling(window=20, min_periods=20)
      .mean()
      .reset_index(level=0, drop=True)
)

df["close_ma_20_available"] = (
    df.groupby("symbol")["close_ma_20"]
      .shift(1)
)

For closes 100, 101, 99 and 102, a three-row mean with min_periods=3 is NaN, NaN, 100.000000, 100.666667. After shifting by one row it is NaN, NaN, NaN, 100.000000. This demonstrates alignment, not a trading signal. The shift(1) operation moves values across ordered rows; supplying a freq changes index labels instead, which is a different operation. (7)

Full strategy-level look-ahead and survivorship examples belong in Why Backtests Lie: Look-Ahead and Survivorship Bias.

Join related tables without multiplying rows

If prices and moving averages are each intended to have one row per (symbol, timestamp), declare that relationship:

joined = prices.merge(
    moving_average,
    on=["symbol", "timestamp"],
    how="left",
    validate="one_to_one",
    indicator=True,
)

Pandas’ validate checks relationships such as one_to_one, while indicator records left_only, right_only and both. Duplicate keys can otherwise multiply rows; an inner join can silently drop observations. Check row counts before and after the merge, and inspect null keys because pandas’ null-key matching differs from ordinary SQL joins. (8)

If the right table contains two rows for one key, let the merge fail. Do not take the last row until you know whether the duplicate is a correction, session collision, symbol mismatch or parser defect. Preserve the original records and send the source investigation to Topic 8.

Run local transformation checks before moving on

These assertions check the transformation, not the quality of the source:

qa = df.sort_values(["symbol", "timestamp"]).copy()

assert qa.groupby("symbol")["timestamp"].apply(
    lambda s: s.is_monotonic_increasing
).all()

assert not qa.duplicated(["symbol", "timestamp"]).any()
assert (qa["high"] >= qa[["open", "close", "low"]].max(axis=1)).all()
assert (qa["low"] <= qa[["open", "close", "high"]].min(axis=1)).all()
assert (qa["high"] >= qa["low"]).all()
assert (qa["volume"] >= 0).all()

Use isna() or notna() to expose missing fields; pandas documents that missing values can have different sentinel representations. (9) Do not forward-fill every gap: a missing 5-minute bar may be a source failure, a suspension, a no-trade interval or an excluded auction state. Data QA: Why Your Dataset Lies owns that source, calendar and point-in-time audit.

What this tutorial does not prove

Pandas cannot tell you whether a provider retained securities that later disappeared, whether current constituents were applied to old dates, whether an adjustment method matches the research question, whether timestamps describe publication time or observation time, whether a source licence permits redistribution, or whether orders and fills are realistic. A notebook can run without errors while any of those conditions is false.

Once the table passes the data audit, Backtesting 101: Vectorized vs Event-Driven explains the engine choice, while Build Your First Backtester in pandas extends a clean transformation into signal and trade records.

Tested environment and limitation

The synthetic examples and checks were executed on Linux x86_64 with Python 3.12.13, pandas 3.0.5 and NumPy 2.3.5. The deterministic fixture matched its recorded canonical hash. This is not a claim that the commands were executed on Windows. Use Python Setup for Trading (Windows) for setup, then rerun the fixture there before treating platform-specific behaviour as resolved.

The examples demonstrate pandas mechanics, not NSE market behaviour, data-vendor coverage or strategy performance.

Sources

(1) NSE India, “Market Timings”: https://www.nseindia.com/static/market-data/market-timings

(2) NSE India, “Closing Auction Session”: https://www.nseindia.com/static/products-services/closing-auction-session

(3) pandas, “Time series / date functionality”: https://pandas.pydata.org/docs/user_guide/timeseries.html

(4) pandas, “DataFrame.resample”: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.resample.html

(5) pandas, “Series.pct_change”: https://pandas.pydata.org/docs/reference/api/pandas.Series.pct_change.html

(6) pandas, “DataFrame.rolling”: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.rolling.html

(7) pandas, “DataFrame.shift”: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.shift.html

(8) pandas, “DataFrame.merge” and “Merge, join, concatenate and compare”: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html; https://pandas.pydata.org/docs/user_guide/merging.html

(9) pandas, “Working with missing data”: https://pandas.pydata.org/docs/user_guide/missing_data.html