vf_dnn_processing.c and vf_dnn_detect.c both declare an uninitialized
DNNData model_input before using it. This commit initialize it.
Signed-off-by: younengxiao <steven.xiao@amd.com>
When loading a TorchScript model that does not contain any learnable
parameters (e.g., a purely functional model), the Torch backend would
crash during inference. This occurred because the code attempted to
dereference the first iterator of the model's parameter list
`parameters().begin()` to determine the device, which results in
Undefined Behavior when the parameter list is empty.
This commit fixes the issue by determining the inference device directly
from the user-configured `ctx->device` string instead of probing the
model parameters, allowing parameterless models to execute safely.
Testing:
1. Generate a parameterless model:
cat << 'EOF' > generate_model.py
import torch
class DummyModel(torch.nn.Module):
def forward(self, x):
return x
scripted_model = torch.jit.script(DummyModel())
scripted_model.save("dummy_model.pt")
EOF
python3 generate_model.py
2. Run inference (previously crashed, now succeeds):
./ffmpeg -y -i input.mp4 -vf 'format=rgb24,dnn_processing=dnn_backend=torch:model=dummy_model.pt' -frames:v 5 -f null -
Signed-off-by: Raja Rathour <imraja729@gmail.com>
amf_init_filter_config() returned the hardware pixel format (e.g.
AV_PIX_FMT_D3D11) as in_format when the input link carries D3D11VA or
DXVA2 hw frames. av_av_to_amf_format() maps hw formats to
AMF_SURFACE_UNKNOWN, so every AMF filter failed its component Init()
with AMF_INVALID_ARG (VQEnhancer, FRC) or AMF_NOT_SUPPORTED
(Converter, HQScaler). Return the underlying software format instead,
which is already validated against AMF a few lines above. Apply the
same mapping to the default output sw format, which otherwise ends up
as the hw pixel format in hwframes_out->sw_format when the output link
uses a hwaccel format.
Fixes e.g.:
ffmpeg -hwaccel d3d11va -hwaccel_output_format d3d11 -i in.mp4 -vf frc_amf -c:v hevc_amf out.mp4
Note: with D3D11VA input, the compute-based components (VQEnhancer,
HQScaler, VideoConverter) additionally require decode textures created
with D3D11_BIND_SHADER_RESOURCE, e.g.:
-init_hw_device d3d11va=dx11:,SHADER=1 -hwaccel_device dx11
FRC and DXVA2 input work with default decode textures.
This reverts commit f3431169da, which
introduced various regressions and hangs with at least exr, as
reported by fate, and potentially other codecs that are not as
thouroughly tested.
The arithmetic to process colors in premultiplied alpha is
completely different from the arithmetic for straight alpha.
Running a filter using straight arithmetic on premultiplied
will result in incorrect and sometimes invalid output.
Therefore, premultiplied alpha should only be selected for
filters that either explicitly support it or only use
elementary color arithmetic or none at all.
For other filters, automatic conversion will do its work.
This reverts commit 978a0821ee.
The stability of the negotiation process has never been studied
with multiple rounds.
It has always been documented to users that scale was the filter
to insert to manage format conversions. A second filter for
a specific type of conversion should never have been added:
the proper fix for the issue that this commit tried to fix is
to give scale the ability to handle premultiplication too.
When the filter graph is torn down early (e.g. at EOF with -frames:v),
the main thread calls dnn_free_model and destroys the request queue
while async inference threads may still be running. When a detached
thread finishes and tries to push its result back to the destroyed
queue, it triggers a segmentation fault.
Add ff_dnn_wait_requests() to dnn_backend_common which blocks
until all allocated request items (ctx->nireq) have been safely
returned to the request_queue. Call it from dnn_free_model in the
torch, tensorflow, openvino, and onnx backends before destroying the queue.
parse_dat(), parse_cube(), and parse_cinespace() multiply an untrusted LUT size
before allocate_3dlut() validates it, which can overflow int.
Use the validated lutsize2 computed by allocate_3dlut() instead.
Fixes: signed integer overflow
This allows to remove the asm.h inclusion from x86/mathops.h
and therefore from hundreds of files which don't need it.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Add batch processing support to the LibTorch DNN backend, following
the same pattern used by the OpenVINO backend.
Key changes:
- Add batch_size AVOption (range 1-1000, default 1) to DnnContext
- Accumulate lltasks in the queue, trigger inference when batch_size
lltasks are ready (matching the OpenVINO batch execution pattern)
- Pre-allocate a single contiguous memory buffer for the entire batch
in fill_model_input_th() to avoid unnecessary tensor copies
- Split batched output in infer_completion_callback() and dispatch
each slice to its corresponding task
Tested with:
ffmpeg -f lavfi -i testsrc=duration=5:size=640x480:rate=25 -vf format=rgb24,dnn_processing=dnn_backend=torch:model=dummy_model.pt:batch_size=4 -f null -
Signed-off-by: Raja Rathour <imraja729@gmail.com>
Add an output_format option (cuda | cuarray) so the filter can produce
either pitched-linear (AV_PIX_FMT_CUDA, default) or block-linear
(AV_PIX_FMT_CUARRAY) device frames. This enables host-to-device
upload directly into CUarray surfaces for the CUARRAY transcode
pipeline.
Also extend the accepted input pixel format list with semi-planar 444
formats (NV24, P410, P416) and the planar MSB variants (YUV444P10MSB,
YUV444P12MSB). Semi-planar 444 formats are rejected with a clear
error when output_format=cuda, as they are only supported by the
CUarray allocation path.
Signed-off-by: Diego de Souza <ddesouza@nvidia.com>
Initialize ctx->nireq with av_cpu_count() / 2 + 1 when
unset, matching the TensorFlow and OpenVINO backends.
Create ctx->nireq THRequestItems in a loop instead of
hardcoding a single request, enabling concurrent async
inference requests for improved throughput.
Signed-off-by: Raja Rathour <imraja729@gmail.com>
Convert packed 24/32-bit RGB/BGR/RGBA/BGRA input to PAL8 using a per-frame palette
whose colors are placed on a face-centered cubic lattice (realized as the
scaled D3/D4 checkerboard lattice), with a user-supplied density controlling
the number of lattice steps spanning one color axis.
Only lattice points actually used by a frame enter its palette; if a frame
needs more than 256 of them, the filter will itearatively drop palettte
entries and reassign affected pixels until 256 color remain
lookup uses the Conway-Sloane rounding algorithm. Supported dithering
modes: none, ordered 8x8 bayer (swscale), Cluster & Void blue noise and
Floyd-Steinberg error diffusion.
Co-Authored-by: Fable-5
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
vf_libplacebo is unique in that it can function as both a Vulkan filter, and a
software filter, depending on how it the filter graph is configured. While this
is very flexible, it create a problem in situations where the filter does not
receive a Vulkan hw frames context up front. When that doesn't happen, it will
initialise its own standalone pl context, and then fail to interoperate with
a context provided via the input link. This then leads to graph failures.
There are two primary scenarios where this existing logic breaks what should
be valid configurations:
* When the global filter hw device is a cuda device and we use something like
`hwupload=derive_device=vulkan` to pass frames from cuda to vulkan
* In mpv (and probably other media players) which don't set the global filter
hw device at all. In this case, it's impossible to configure vf_libplacebo
to use a hw frames ctx, even if it's using Vulkan for everythng else. This
prevents the use of vf_libplacebo in any fully hardware accelerated pipeline
in mpv
There are various ways we could imagine addressing it - such as allowing the
filter to discard the initial pl context and recreating it based on the passed
in device, but it's easier to reason about if we add a flag that explicitly
tells the filter that it should inherit the device context from the input link.
This puts the filter into a mode that works like all the other Vulkan filters.
This requires explicit configuration from the user, but the intent is clearer,
and the user can always know when it's necessary as they define the filter
graph.
A format-only conversion leaves the auto use_filters value unresolved.
Regression since 5d0748243f.
Fix issue #23737
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
Two runtime paths could compute out-of-range source coordinates for
degenerate projection geometry, causing heap-buffer-overflow reads
Fixes: out of array read
Fixes: assertion failure
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array read
Fixes: assertion failure
Fixes: mQzloVqnivHQ
Found-by: Anthony Hurtado
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
```
vf_quirc.c: In function 'filter_frame':
vf_quirc.c:104:46: warning: format '%ld' expects argument of type 'long int', but argument 5 has type 'int64_t' {aka 'long long int'} [-Wformat=]
104 | "Found count %d codes in image #%ld\n", codes_count, inl->frame_count_out);
| ~~^ ~~~~~~~~~~~~~~~~~~~~
| | |
| long int int64_t {aka long long int}
| %lld
```
C23 made these generic functions that no longer cast const
away for you, leading to warnings when compiling with C23
and a recent enough toolchain (glibc supports this since 2.43).
This commit fixes all the warnings that can simply be fixed
by adding const, without adding casts.
(the latter excludes parse_forced_key_frames() in ffmpeg_mux_init.c).
Reviewed-by: Kacper Michajłow <kasper93@gmail.com>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
When the blending factor was calculated to be 0, the hot path skipped
updating variables later emitted in metadata.
Fix the flow to ensure they are populated consistently.
Extend the FATE test suite to check metadata output.
This commit attempts to address feedback from this filter's users, by
introducing a new option which controls the amelioration mechanism.
The "blend" option is a factor which is multiplied by the difference
in badness (between the threshold and the currently accumulated
badness). This difference normally controls how much of the next frame
we can let through without making it exceed the badness threshold.
Setting the option to zero effectively puts the filter into a mode
where it always duplicates the last frame (which did not put
accumulated badness over the threshold) instead of attempting to blend
in new frames. I have received reports that this mode is preferable
to users for some types of media.
This commit attempts to address feedback from this filter's users, by
improving the filter's behavior at the start of playback or
immediately after seeking.
In these situations, the history buffer is empty, but because we did
not previously track its size, we were calculating the weighted
average as if the corresponding frames had zero badness. This caused
the filter to behave differently and possibly produce false negatives
when the history was not fully populated.
Address this by instead taking into account the history size when
calculating cumulative badness. To accomplish this:
- Add a history_size field to to PhotosensitivityContext, tracking how
much of the history buffer is populated.
- Change the semantics of PhotosensitivityContext::badness_threshold.
Previously, it was premultiplied by the maximum history size. This
is no longer done, so that we can multiply it to the live
history_size on the fly instead.
- Calculate the badness threshold on a per-frame basis. The result is
now stored in a badness_threshold local variable.
This commit only changes the filter's behavior for the first
PhotosensitivityContext::nb_frames (configurable as the "frames"
filter option) frames. For successively filtered frames, the behavior
is unchanged.
ff_dnn_start_inference_async() return value was not stored in ret,
causing execute_model_th() to return success on async startup failure.
This left vf_dnn_processing stuck in its flush loop waiting on a
task that was never started.
Guo Yejun
Dan Dennedy
younengxiao
Michael Niedermayer
Raja-89
Julius Bairaktaris
Kenan Alghythee
Timo Rothenpieler
Nicolas George
Andreas Rheinhardt
StachowiakDawid
Diego de Souza
Araz Iusubov
Lynne
James Almer
Philip Langdale
Zhao Zhili
Kacper Michajłow
Jamaika1
Vladimir Panteleev
Vladimir Panteleev
tknaveen