Building a clean NSE data pipeline means more than download a file. It preserves the original input, records where each row came from, separates end-of-day and five-minute observations, handles India’s market sessions explicitly, and stops when a duplicate, gap or impossible OHLC row appears. This guide applies that contract to NSE cash-equity EQ data; it does not promise complete history or a validated live provider feed.

If Python is ready and a source has already been selected, the pipeline below gives you a controlled hand-off to pandas and a backtester. Before begenning, its important to understand all different sources to get data for algo trading. File formats, history, timestamp semantics, corrections and licences differ by provider, so source choice must precede ingestion.

Start with a bounded contract

“NSE data” is not a sufficiently precise dataset description. A run needs a universe, period, frequency, session rule and price basis before any file is parsed.

NSE publishes security, T+0 and symbol-change files that can support an eligibility step, but this pipeline does not reconstruct a point-in-time universe. (5) A security may be suspended, newly listed, renamed or absent from a provider response. Those states need explicit labels, not silent row fills.

Move data through four states

The easiest way to keep a pipeline honest is to give each state a narrow responsibility. Transformations become reviewable, and a later analyst can tell whether a value came from the exchange, a provider or our own code.

Source: DailyBulls data-contract illustration; exchange/provider provenance and QA rules are cited in the article.

Text and table fallback: raw files remain immutable; staging parses without changing meaning; canonical tables apply identity, time, eligibility and price-basis rules; the QA gate either approves a partition or preserves its exceptions in quarantine. The diagram is a relationship aid, not a performance chart.

Raw: keep what the source sent

Store the exact ZIP, CSV, GZ, binary response or API payload with its file name, retrieval time, source URL or endpoint, content type and SHA-256 hash. Do not sort, fill, round or overwrite raw bytes. If an exchange replaces a file, save the replacement as a new raw object and run.

For EOD data, the NSE reports page points readers away from the old CM Bhavcopy CSV, discontinued from 8 July 2024, towards the CM-UDiFF Common Bhavcopy Final ZIP. (1) UDiFF guidance describes versioned, headered, ISO-dated compressed files, so the file name and format version belong in a manifest instead of a hard-coded downloader assumption. (7) NSE also maintains UDiFF catalogue and test-file links on its formats page, but those binary spreadsheets were not parsed for this implementation. (8)

Staged: parse without changing meaning

The staging table keeps source columns and adds helper fields such as source_row_id, source_timestamp_raw, source_timezone, parse_status and parse_error_code. Convert a number to a decimal only after checking the declared format. Keep the original text beside a parsed value when a future re-parse may matter.

A malformed row is not deleted because it is inconvenient. Retain it with a reason code and send it to quarantine. That makes row counts reconcilable to the raw file.

Canonical: create two explicit research tables

Build a canonical EOD table keyed by security and trading date, and a canonical five-minute table keyed by security and bar start. Do not put daily and intraday observations in one table with an ambiguous date column.

At the canonical boundary, apply EQ eligibility, instrument mapping, Asia/Kolkata timestamps, regular-session filtering, duplicate policy and raw-price labels. A missing candle still remains missing.

QA-approved: pass, review or quarantine

A parser finishing is not a clean-data verdict. The QA gate checks keys, price relationships, time ordering, expected session slots, row-count reconciliation, corporate-action state and the provenance manifest. A partition can proceed with a review status only when every exception is visible and assigned a disposition. Failed rows remain available for investigation.

Build the EOD table from an explicit schema

The NSE security-wise archive lists fields such as Symbol, Series, Date, previous close, open, high, low, last, close, VWAP, total traded quantity, turnover, trade count and delivery fields. (2) Keep these source fields recognisable in staging, then map them into names that make raw and derived values unambiguous.

Canonical EOD schema

The EOD key is:

(EXCHANGE, INSTRUMENT_KEY, SERIES, SESSION_DATE)

There should be no more than one canonical EOD row per key. If two byte-identical rows arrive, collapse them only after logging both source row IDs and the count. If values disagree, quarantine both. “Take the last row” is not a data policy.

Add five-minute candles without inventing a market clock

Five-minute data is more source-specific than EOD data. NSE lists one-minute and five-minute snapshots as a commercial binary product; a provider API may deliver the same interval as an authenticated response. (9) Zerodha’s historical-candle documentation accepts a 5minute interval and shows OHLCV arrays with timestamps carrying +0530. (10) These documents establish possible delivery formats, not universal history, correction behaviour or free redistribution. No authenticated provider response has been validated in this implementation.

Preserve and normalise timestamps

Preserve the source timestamp first. Then create canonical values:

SOURCE_TIMESTAMP = EXACT PROVIDER/EXCHANGE VALUE

TIMESTAMP_IST = TIMEZONE-AWARE VALUE NORMALISED TO ASIA/KOLKATA

SESSION_DATE = CALENDAR DATE OF TIMESTAMP_IST

BAR_START = LEFT EDGE OF THE FIVE-MINUTE INTERVAL

BAR_END = BAR_START + 5 MINUTES

If a source returns a naive timestamp, do not silently assume it is IST. Keep it in staging with source_timezone = unknown until the provider specification or a verified response resolves the convention. A five-minute shift can put every bar in the wrong session while still looking plausible on a chart.

Keep session states explicit

For the core table, include regular continuous cash-equity bars with:

09:15:00 <= BAR_START < 15:30:00 (ASIA/KOLKATA)

That produces 75 expected bar starts for a complete day: 09:15, 09:20, …, 15:25. The exchange also documents auction, transition and post-close states, and the general timings page and newer Closing Auction Session page are not worded identically. (3), (4) Keep those states out of the core continuous table until the applicable session/circular policy is reconciled. If they are retained, place them in a side table with session_state = cas, post_close or auction.

T+0 is a separate state. NSE’s FAQ describes a 09:15–13:30 T+0 session, distinct identification and no separate T+0 bhavcopy. (12) If a source returns T0 rows, retain its series/settlement field and route them to a separate partition rather than combining them with normal EQ candles.

Canonical five-minute schema

The five-minute key is:

(EXCHANGE, INSTRUMENT_KEY, SERIES, SESSION_DATE, BAR_START)

Validate the rows, not just the file

Check the OHLCV relationships

Every populated price must be positive, and high and low must contain the open and close:

PRICE > 0

HIGH >= MAX(OPEN, CLOSE, LOW)

LOW <= MIN(OPEN, CLOSE, HIGH)

HIGH >= LOW

VOLUME >= 0

These are arithmetic checks, not trading signals. Quarantine an invalid row with its source row ID. If a provider reports volume in a unit that is not a share count, retain that unit instead of forcing it into an integer.

Classify gaps; never fill them silently

For a complete continuous session, generate the expected 75 bar starts from the session calendar and compare them with observed starts. A missing slot can indicate a source gap, halt, no trade, illiquidity, shortened session or an incorrect timestamp convention. The pipeline can classify the gap; it cannot infer the cause from an empty cell.

Never forward-fill a missing OHLCV bar. A filled candle looks like an observation and can change a later return or signal. Mark the interval missing_unresolved until a source or calendar check supplies the reason.

Reconcile counts after every transformation

Use explicit conservation checks:

RAW ROWS = STAGED ROWS + PARSE-QUARANTINE ROWS

STAGED ROWS = CANONICAL ROWS + EXPLICIT EXCLUSIONS + CANONICAL QUARANTINE ROWS

For deterministic parsing, these counts should reconcile exactly. If they do not, the run fails even if the final table looks plausible.

Keep raw and adjusted prices separate

Corporate actions can change the question a dataset answers. NSE notes that some adjusted 52-week columns reflect bonus, consolidation, splits and rights, while the bhavcopy is the reference for actual unadjusted high and low. (1) Its corporate-action workflow provides searchable records, but rows are dynamic and must be retrieved and stored as evidence for the period being used. (6) No target-month action rows were retrieved for this implementation.

Use this policy:

1. Retain source OHLCV as price_basis = raw.
2. Attach action_type, action_effective_date, action_factor_raw_to_adjusted and action_source_ref when verified.
3. Calculate an adjusted series in a new, versioned table only after factor and date checks pass.
4. Set adjustment_status = pending_review when any action element is missing.

Do not overwrite close_raw with a locally adjusted close. Raw prices help reconstruct what the source reported; adjusted prices may help with long-horizon returns. They are different representations, and the manifest should state which one a later notebook used.

Preserve symbol and instrument history

Use an instrument key that survives an ordinary symbol rename when the source provides a stable identifier. Keep the observed symbol and effective-date history so a join can answer which name the security used on a date and which security an identifier represents.

Provider tokens need the same caution. Zerodha’s instrument documentation recommends exchange + tradingsymbol for lookup and warns that numeric tokens can be reused for different derivative contracts after expiry. (10) For cash equities, an ISIN or exchange security identifier is preferable. If all you have is symbol + series, label the identity symbol_series_only and do not describe it as a point-in-time universe.

Make every run auditable

A canonical row without its manifest is difficult to audit. Store at least the following for every EOD and five-minute partition:

RUN_ID
PIPELINE_VERSION
SOURCE_PUBLISHER
SOURCE_URL_OR_ENDPOINT
SOURCE_FILE_NAME
SOURCE_FORMAT_VERSION
RETRIEVED_AT_UTC / RETRIEVED_AT_IST
SOURCE_REPORTED_PERIOD
UNIVERSE_RULE
FREQUENCY
SOURCE_TIMESTAMP_SEMANTICS
RAW_PATH / RAW_SHA256
STAGED_PATH / CANONICAL_PATH
INPUT_ROW_COUNT / STAGED_ROW_COUNT / CANONICAL_ROW_COUNT
QUARANTINE_ROW_COUNT
DUPLICATE_EXACT_COUNT / CONFLICTING_DUPLICATE_COUNT
MISSING_INTERVAL_COUNT
CALENDAR_VERSION
ADJUSTMENT_POLICY_VERSION
QA_STATUS
LICENCE_OR_AUTHORISED_USE_NOTE

The hash and counts let you compare two runs even if a vendor silently republishes a file. The licence_or_authorised_use_note is operationally important: NSE’s data policy and terms restrict uses such as offline storage, redistribution and systematic collection without the relevant permission. (11), (13) This is not a legal determination for your account. Review the selected source’s current agreement before automating retrieval or publishing raw files.

See the QA rules on a small example

The following rows are synthetic, not NSE observations. They make the disposition rules concrete.

The accompanying standard-library validator reports six passed contract tests and one expected gap review. It does not call NSE or a broker. A local test proves that the rules behave as written; it does not prove that an exchange file’s columns or a provider’s timestamp semantics have been parsed correctly.

A minimal validation skeleton

This provider-agnostic skeleton shows the order of decisions rather than a complete UDiFF or broker downloader:

from datetime import timedelta


def canonical_key(row):
    if row["record_type"] == "EOD":
        return (
            row["exchange"],
            row["instrument_key"],
            row["series"],
            row["session_date"],
        )

    return (
        row["exchange"],
        row["instrument_key"],
        row["series"],
        row["session_date"],
        row["bar_start"],
    )


def valid_ohlc(row):
    prices = (
        row["open_raw"],
        row["high_raw"],
        row["low_raw"],
        row["close_raw"],
    )

    if any(value is None or value <= 0 for value in prices):
        return False

    return (
        row["high_raw"] >= max(prices)
        and row["low_raw"] <= min(prices)
        and row["volume_raw"] >= 0
    )


def expected_starts(
    start_minute=9 * 60 + 15,
    end_minute=15 * 60 + 30,
):
    total = end_minute - start_minute

    return [
        start_minute + offset
        for offset in range(0, total, 5)
    ]

The production implementation must add timezone parsing, source row IDs, duplicate grouping, calendar exceptions, quarantine writes and manifest updates around these checks. A compact helper is not a reason to discard lineage.

Hand the approved table to pandas, then audit the universe

Once the canonical table passes its hard checks, it is ready for manipulation, not automatically ready for a strategy claim. Use pandas Crash Course for Candles to resample, calculate returns and work with rolling windows using the schema above. Keep frequency, price basis, session rule and manifest ID beside each notebook output.

Then continue to Data QA: Why Your Dataset Lies for point-in-time membership, survivorship, look-ahead and adjustment diagnostics. The ingestion gate here catches malformed rows and missing intervals; it does not reconstruct a historical universe or remove every source of backtest bias.

For the full market clock and settlement context, see Market Mechanics for Code-First Traders. For environment help, use Python Setup for Trading on Windows. Source selection remains the job of Where to Get NSE/BankNifty Data at Every Price Tier; source choice and pipeline governance are connected, but they are not the same decision.

What the pipeline can and cannot promise

It can give you:

• immutable raw inputs and a reproducible transformation path;
• separate, explicit EOD and five-minute schemas;
• visible identity, timezone, session, duplicate, OHLC and gap rules;
• raw prices plus documented corporate-action state;
• a manifest that lets another engineer rerun or audit a partition.

It cannot, by itself, give you:

• a free or complete five-minute history for every NSE security;
• a validated parser for current UDiFF binary/XLSX files without an approved sample;
• a point-in-time, survivorship-free universe;
• a correct adjustment factor when an action record is missing;
• permission to scrape, store or redistribute a source outside its terms;
• a trading edge or a profitable backtest.

The practical standard is simple: call a dataset research-ready only when its source, universe, period, timestamp convention, transformations, exceptions and licence boundary are written down. A clean pipeline does not hide uncertainty; it makes that uncertainty inspectable before a candle reaches strategy code.

Sources

1. NSE — All Reports: NSE All Reports

2. NSE — Security-wise Price Volume Archives: NSE security-wise archive

3. NSE — Market Timings: NSE market timings

4. NSE — Closing Auction Session: NSE Closing Auction Session

5. NSE — Securities Available for Trading: NSE securities available for trading

6. NSE — Corporate Filings and Actions: NSE corporate filings and actions

7. NSE/UDiFF — Guidance document: UDiFF guidance document

8. NSE — Forms and Formats / UDiFF: NSE forms and formats

9. NSE — Real-time data subscription: NSE real-time data subscription

10. Zerodha Kite Connect — Historical candles and instruments: Kite historical candles and Kite instruments and market quotes

11. NSE — Data sharing and usage policy: NSE data-sharing policy

12. NSE — T+0 Security FAQ: NSE T+0 Security FAQ

13. NSE — Terms of Use: NSE Terms of Use