Hi everyone,
I'm trying to understand something I've been experiencing with OBS and NVENC on an RTX 5090.
I'm doing some reference-quality recordings of Alan Wake 2, specifically because its maxed-out Path Tracing is one of the best demonstrations of modern real-time rendering I've come across.
My goal isn't streaming. I'm recording locally and want the highest-quality capture possible.
MY SETUP
ALAN WAKE 2 SETTINGS
3840×2160
Maximum Path Tracing
DLSS Quality
4× Frame Generation
HDR enabled
The game is capable of pushing well above 180 FPS in this configuration, with 4× FG producing the final ~240 FPS output while capping out at my G-Sync + global V-Sync at 225 FPS.
THE PROBLEM WITH OBS
I initially tried recording this through OBS at 4K240 HDR.
I used NVENC AV1 with quality-oriented settings, including 10-bit I010, Rec.2100 PQ, P7, Ultra High Quality, multipass, AQ, B-frames, etc.
However, OBS was reporting massive encoding lag.
The interesting part is that rendering itself wasn't the problem. OBS showed essentially no rendering lag while the encoder was falling behind.
I then tried reducing the workload:
Even at 4K120, I was still seeing substantial encoding lag with the quality settings I wanted.
At 4K60, SFE/three-way split became necessary to get the encoder to keep up with the configuration I was using.
THEN I TRIED NVIDIA APP / SHADOWPLAY
This is where things became interesting.
I recorded the same type of Alan Wake 2 workload using NVIDIA App's recording feature.
I used:
4K + maximum Path Tracing + DLSS Quality + 4× FG + HDR + 240 FPS
And the recording completed successfully.
I ran the resulting MP4 through MediaInfo.
The video stream reports:
3840×2160
AV1 [email protected]
10-bit
4:2:0
Full range
BT.2020
PQ
~239.956 FPS
349 Mb/s average bitrate
The complete file is 8.43 GiB for 3:27.
So, in practice, I'm getting:
4K240 + 10-bit HDR + AV1 recording from NVIDIA App while running Alan Wake 2 at max Path Tracing and 4× FG. It actually works.
WHY I'M CONFUSED
This makes me think the issue isn't simply:
"The RTX 5090 can't encode 4K240 AV1 while gaming."
It clearly can, at least through NVIDIA's own capture implementation.
What I'm trying to understand is what NVIDIA App is doing differently from OBS.
What could it be related to?
I'm especially interested in hearing from anyone familiar with OBS's NVENC implementation or the NVENC API.
ONE OTHER INTERESTING DETAIL
MediaInfo identifies the ShadowPlay recording as Variable Frame Rate.
It reports:
Average: 239.956 FPS
Minimum: 36.556 FPS
Maximum: 612.245 FPS
I'm assuming the extreme minimum/maximum values are related to timestamp/frame-pacing behavior rather than NVIDIA literally capturing anywhere from 36 to 612 unique frames per second, but I don't know enough about how NVIDIA App handles the timestamps to say for certain.
The important part is that the recording plays back as a ~240 FPS capture.
WHAT I'D LIKE TO FIGURE OUT
I'm not necessarily looking for a "use NVIDIA App instead" answer.
I'd actually like to understand the technical difference between the two implementations.
Is there something I can collect from OBS that would help identify why its NVENC encoder is falling behind?
The fact that NVIDIA App can successfully record this workload at 4K240 HDR while OBS experiences severe encoding lag seems like an interesting NVENC/capture-path comparison, and I'd like to understand what's actually happening under the hood.