r/quantfinance • • Sep 24 '26

Built a TradingView-to-Python quant research pipeline for 6 months — now I’ve discovered I may have been losing signals. How would you redesign the data capture layer?

I’ve been building a small quantitative research system since around March, mostly with AI-assisted coding.

The system is not an automated trading bot. It is a research/decision-support pipeline:

TradingView/Pine alerts → webhook → FastAPI receiver → raw signal log → normalized dataset → OHLCV enrichment → returns / MAE / MFE → signal combinations / sequences → paper trading / manual review

The idea is to collect signals such as mean-reversion, liquidity, OBV, BOS, sweeps, RSI divergences, etc., then study what happens after them and which combinations have statistical value.

After several months, I’ve discovered that the upstream data capture was not reliable enough, so some of the statistics built on top of it may be incomplete.

Problems found so far:

  • One original Pine indicator used shared cooldowns: e.g. an OBV_UP signal could suppress an OBV_DOWN for 15 bars; bullish/bearish BOS shared a 20-bar cooldown; opposite sweeps shared 30 bars.

  • Later I created multi-symbol / multi-timeframe “bulk scanners” for 12 tickers × 4H/1D/1W. Two of them appear to have an HTF timing bug, so a signal can appear on the final chart but never generate the webhook.

  • TradingView has also reported webhook timeouts / connection failures. Some timed-out events still reached my server, so I’m now reconciling them individually instead of assuming timeout = lost event.

  • My receiver currently puts events into an in-memory async queue, returns 200 OK, and persists them afterward. That is fast, but it creates a possible loss window if the process dies before persistence.

  • The biggest mistake: I never had a strict end-to-end test proving visual signal → TradingView Alert Log → RAW storage → processed dataset.

The downstream system itself is fairly developed: MAE/MFE, forward returns, pattern mining, sequence analysis, watchlists and paper trading all exist. But now I’m treating the historical statistics as provisional until the input coverage is verified.

My current redesign idea is:

  • same Pine condition must drive both the visual plot and alert();

  • no filtering/suppression at capture time unless explicitly intended;

  • confirmed HTF bars only;

  • deterministic event IDs;

  • durable storage before ACK, or a durable queue;

  • regression tests using known historical signals;

  • automatic reconciliation of expected signals vs TradingView vs RAW vs processed data.

The volume is low — dozens of tickers, not HFT.

Does this architecture make sense? Would you rebuild only the ingestion/data-integrity layer, or would you redesign the whole pipeline?

2 Upvotes

6 comments sorted by

1

u/QuanTradin Sep 24 '26

I'd stop treating the webhook as the source of truth. Recompute every signal offline in Python from the same OHLCV you already pull, and use the TradingView alerts only as a cross-check. Then a missed webhook or an HTF timing bug shows up as a diff between the two logs instead of a hole you find months later.

For the receiver itself, write the raw body to disk (sqlite or an append-only file) before you return 200, not after. The in-memory queue is exactly where events vanish on a restart. Give each event a key like symbol + timeframe + bar close time + signal name, so TradingView retries and your timeout reconciliation dedupe cleanly.

And pull the shared cooldowns out of Pine. Emit every raw signal and apply cooldowns downstream, where you can change them without re-collecting months of data. Right now the cooldown is baked into what you observed, so you can't tell which suppressed signals would have mattered.

1

u/Fabulous-Ad-5073 17d ago

Thanks all, this was really useful. Quick update on what I found and what I’ve done so far.

Found (from TradingView’s Alert Log export vs my server logs):

  • TradingView stops an alert that fires more than 15 times in 3 minutes. My bulk scanners were doing exactly that.

  • On Oct 19, 252 of 361 webhooks my receiver got were duplicates, and some bursts carried candles up to ~67h old. My working theory (not proven yet): the script’s var arrays get rolled back on realtime ticks, so it “forgets” what it already sent and re-sends it.

  • Since Oct 20, 639 POSTs from TradingView IPs hit my server and were silently discarded (invalid/empty body). I still don’t know what they were.

  • My dedupe key ignored the timeframe, so it dropped valid cross-timeframe signals (~77) while still letting ~112 duplicates through.

Done so far:

  • Receiver now logs every discarded payload instead of swallowing it.

  • A silence alert: if no signals arrive on a trading day, a dashboard card turns red. (Correction: I removed the heartbeat from the scanners earlier because it flooded the alert log. This monitor replaces it.)

  • New Pine version using varip for the queue/seen-state plus a hard cap of 14 alert() calls per 3 minutes. Compiles, but NOT tested live yet.

Next, in this order: durable write before returning 200, timeframe in the event key, then recomputing signals in Python from OHLCV and diffing nightly against the alerts (as QuanTradin suggested). Cooldowns come out of Pine last. I’ll treat the 6 months of historical stats as biased and regenerate them once the Python side matches.

I’ll report back after the first full trading week.

1

u/QuanTradin 17d ago

the dupes fit that 15-in-3-minutes cutoff, but candles 67h old inside a burst look more like TradingView replaying queued alerts once it switched them back on. dedupe on symbol plus bar close time and drop anything whose bar is more than one interval behind arrival, and the receiver handles both without you touching the scanners.

1

u/Shoddy_Peanut7320 Sep 25 '26

Re "Built a TradingView-to-Python quant research pipeline for 6 months — now I've discovered I may have been losing signals. How would you redesign the data capture layer?":

"The 'stop treating the webhook as source of truth' framing is the key mental shift here, most people build the webhook as if it's authoritative and only add reconciliation as an afterthought, but flipping it so the offline OHLCV recompute is the ground truth and the webhook is just a delivery-confirmation cross-check catches exactly the two bug classes described (missed events, HTF timing issues) structurally rather than relying on someone noticing months later. Pulling cooldowns out of Pine and applying them downstream is probably the single highest-leverage change on the list too, since it converts a currently-irreversible data loss (suppressed signals you can never recover) into a reversible filtering decision you can freely re-tune and backtest against, that's a big difference for research validity. The composite dedupe key (symbol + timeframe + bar close + signal name) is the right granularity, just worth making sure bar close time is normalized to exchange time consistently across all your feeds, timezone/DST handling is a sneaky place for that key to silently produce false duplicates or false uniques.

(Skipping another EventTrader mention here since it's already on this thread once.)" https://claude.ai/artifact/VuwYE2KYrtzMdwyEpvA9wH

1

u/[deleted] Oct 13 '26

[removed] — view removed comment

1

u/QuanTradin Oct 13 '26

same here, the restart was the only reason I noticed at all. I ended up building Quantradin (free, paper only) partly so signals get recomputed from the bars instead of trusting the alert. did you key your rows on bar close time or on arrival time?