Jetson Nano平台deepstream-app视频颠倒调试

1.前言

试图运行deepstream-app与示例 “source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt” 在Jetson Nano上与“deepstream-4.0_4.0-1_arm64.deb” 使用Raspberry Pi V2相机NoIr，但图像显示面朝上。

修改配置

[source0] enable = 1 #Type – 1 = CameraV4L2 2 = URI 3 = MultiURI 4 = RTSP 5 = Camera (CSI) (Jetson only) type = 5 intra-decode-enable = 1 gpu-id = 0 camera-id = 0 camera-width = 1920 camera-height = 1280 camera-fps-n = 30 camera-fps-d = 1 camera-csi-sensor-id = 0 drop-frame-interval = 0

当运行时，图像出现颠倒。

另一方面，当在同一台计算机上执行jetson推理示例时 ~/git/jetson-inference/build/aarch64/bin ./camera-viewer 图像看起来没问题，并在管道中显示 “nvvidconv flip-method = 2”。

那么问题是: 如何在配置文件中指示相机的方向来运行deepstream-app?

JetPack版本:4.2.2

2. 修改create_camera_source_bin

nvvideoconvert中不支持 flip method 属性。 修改create_camera_source_bin()运行如下命令：

nvarguscamerasrc bufapi-version=0 ! nvvidconv flip-method=2 ! video/x-raw,forma=NV12 ! nvvideoconvert ! video/x-raw(memory:NVMM),format=NV12

修改源代码后执行 log：

<li></li> ./deepstream-app -c ../../../../samples/configs/deepstream-app/source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano-csiCam-test01.txt –gst-debug=3 2>&1 | tee debug.txt 0:00:00.242025044 10454 0x557ee9c290 WARN omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /home/mherrera/.config:/etc/xdg/xdg-unity:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS) (deepstream-app:10454): GLib-GObject-WARNING **: 02:31:43.653: cannot register existing type GstInterpolationMethod (deepstream-app:10454): GLib-GObject-CRITICAL **: 02:31:43.654: g_param_spec_enum: assertion G_TYPE_IS_ENUM (enum_type) failed (deepstream-app:10454): GLib-GObject-CRITICAL **: 02:31:43.654: validate_pspec_to_install: assertion G_IS_PARAM_SPEC (pspec) failed 0:00:11.051427092 10454 0x557ef64850 FIXME default gstutils.c:3981:gst_pad_create_stream_id_internal:<src_elem:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id Creating LL OSD context new Runtime commands: h: Print this help q: Quit p: Pause r: Resume NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source. To go back to the tiled display, right-click anywhere on the window. **PERF: FPS 0 (Avg) **PERF: 0.00 (0.00) ** INFO: <bus_callback:163>: Pipeline ready ** INFO: <bus_callback:149>: Pipeline running Creating LL OSD context new GST_ARGUS: Creating output stream CONSUMER: Waiting until producer is connected… GST_ARGUS: Available Sensor modes : GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000; GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000; GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000; GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000; GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000; GST_ARGUS: Running with following settings: Camera index = 0 Camera mode = 2 Output Stream W = 1920 H = 1080 seconds to Run = 0 Frame Rate = 29.999999 GST_ARGUS: PowerService: requested_clock_Hz=627200000 GST_ARGUS: Setup Complete, Starting captures for 0 seconds GST_ARGUS: Starting repeat capture requests. CONSUMER: Producer has connected; continuing. nvbuf_utils: nvbuffer Payload Type not supported NvBufferGetParams failed for src_dmabuf_fd nvbuffer_transform Failed gst_nvvconv_transform: NvBufferTransform Failed 0:00:11.977186035 10454 0x557ef64850 WARN basesrc gstbasesrc.c:3055:gst_base_src_loop:<src_elem> error: Internal data stream error. 0:00:11.977212963 10454 0x557ef64850 WARN basesrc gstbasesrc.c:3055:gst_base_src_loop:<src_elem> error: streaming stopped, reason error (-5) ERROR from src_elem: Internal data stream error. Debug info: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstNvArgusCameraSrc:src_elem: streaming stopped, reason error (-5) Quitting GST_ARGUS: Cleaning up CONSUMER: Done Success GST_ARGUS: Done Success App run failed GST_ARGUS: PowerServiceHwVic::cleanupResources mherrera@JN01EXPL:/opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/deepstream-app$ <li></li>

它应该是 bufapi-version=FALSE

g_object_set (G_OBJECT (bin->src_elem), “bufapi-version”, <b>FALSE</b>, NULL);

更新代码

g_object_set (G_OBJECT (bin->src_elem), “bufapi-version”, FALSE, NULL);

log：

ERROR from src_bin_muxer: Input buffer number of surfaces (0) must be equal to mux->num_surfaces_per_frame (1)

作为参考，附加了输出文件和deepstream_source_bin.c。 尝试了不同的代码更新deepstream_source_bin.c没有成功。 回到原来的代码，只使用之前的建议进行更新，错误信息是一样的。

3.使用NvBufSurface api

下面是一个补丁供参考: sources\apps\apps-common\src\deepstream_source_bin.c:

include <string.h> +#include “nvbufsurface.h” +#include “nvbufsurftransform.h” #include “gstnvdsmeta.h” #include “deepstream_common.h” #include “deepstream_sources.h” @@ -62,6 +64,64 @@ set_camera_v4l2_params (NvDsSourceConfig * config, NvDsSrcBin * bin) return TRUE; } +static GstPadProbeReturn +nvargus_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info, + gpointer u_data) +{ + GstBuffer *buf = (GstBuffer *) info->data; + GstMapInfo outmap = GST_MAP_INFO_INIT; + gst_buffer_map (buf, &outmap, GST_MAP_WRITE); + NvBufSurface* surface = (NvBufSurface *)outmap.data; + + NvBufSurfTransformRect src_rect, dst_rect; + src_rect.top = 0; + src_rect.left = 0; + src_rect.width = (guint) surface->surfaceList[0].width; + src_rect.height= (guint) surface->surfaceList[0].height; + + dst_rect.top = 0; + dst_rect.left = 0; + dst_rect.width = (guint) surface->surfaceList[0].width; + dst_rect.height= (guint) surface->surfaceList[0].height; + + NvBufSurface *dst_surface = NULL; + NvBufSurfaceCreateParams nvbufsurface_create_params; + + nvbufsurface_create_params.gpuId = surface->gpuId; + nvbufsurface_create_params.width = (gint) surface->surfaceList[0].width; + nvbufsurface_create_params.height = (gint) surface->surfaceList[0].height; + nvbufsurface_create_params.size = 0; + nvbufsurface_create_params.colorFormat = surface->surfaceList[0].colorFormat; + nvbufsurface_create_params.layout = surface->surfaceList[0].layout; + nvbufsurface_create_params.memType = surface->memType; + + NvBufSurfaceCreate(&dst_surface,1,&nvbufsurface_create_params); + + NvBufSurfTransformParams nvbufsurface_params; + nvbufsurface_params.src_rect = &src_rect; + nvbufsurface_params.dst_rect = &dst_rect; + nvbufsurface_params.transform_flag = 0; + nvbufsurface_params.transform_filter = NvBufSurfTransformInter_Default; + + NvBufSurfTransformConfigParams transform_config_params; + NvBufSurfTransform_Error err; + + transform_config_params.compute_mode = NvBufSurfTransformCompute_Default; + transform_config_params.gpu_id = surface->gpuId; + transform_config_params.cuda_stream = NULL; + err = NvBufSurfTransformSetSessionParams (&transform_config_params); + // copy to dst_surface + err = NvBufSurfTransform (surface, dst_surface, &nvbufsurface_params); + // rototate 180 degree to original surface + nvbufsurface_params.transform_flag = NVBUFSURF_TRANSFORM_FLIP; + nvbufsurface_params.transform_flip = NvBufSurfTransform_Rotate180; + err = NvBufSurfTransform (dst_surface, surface, &nvbufsurface_params); + NvBufSurfaceDestroy(dst_surface); + + gst_buffer_unmap (buf, &outmap); + return GST_PAD_PROBE_OK; +} + static gboolean create_camera_source_bin (NvDsSourceConfig * config, NvDsSrcBin * bin) { @@ -144,12 +204,16 @@ create_camera_source_bin (NvDsSourceConfig * config, NvDsSrcBin * bin) NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->cap_filter, “src”); } else { + GstPad *src_pad; g_object_set (G_OBJECT (bin->cap_filter), “caps”, caps, NULL); gst_bin_add_many (GST_BIN (bin->bin), bin->src_elem, bin->cap_filter, NULL); NVGSTDS_LINK_ELEMENT (bin->src_elem, bin->cap_filter); + src_pad = gst_element_get_static_pad (bin->src_elem, “src”); + gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER, + nvargus_src_pad_buffer_probe, NULL, NULL); NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->cap_filter, “src”); }

sources\apps\sample_apps\deepstream-app\Makefile

-24,6 +24,7 @@ APP:= deepstream-app TARGET_DEVICE = $(shell gcc -dumpmachine | cut -f1 -d -) +CUDA_VER:=10.0 NVDS_VERSION:=4.0 LIB_INSTALL_DIR?=/opt/nvidia/deepstream/deepstream-$(NVDS_VERSION)/lib/ @@ -41,10 +42,10 @@ PKGS:= gstreamer-1.0 gstreamer-video-1.0 x11 OBJS:= $(SRCS:.c=.o) -CFLAGS+= -I../../apps-common/includes -I../../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 +CFLAGS+= -I../../apps-common/includes -I../../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 -I /usr/local/cuda-$(CUDA_VER)/include LIBS+= -L$(LIB_INSTALL_DIR) -lnvdsgst_meta -lnvds_meta -lnvdsgst_helper -lnvds_utils -lm \ – -lgstrtspserver-1.0 -lgstrtp-1.0 -Wl,-rpath,$(LIB_INSTALL_DIR) + -lgstrtspserver-1.0 -lgstrtp-1.0 -Wl,-rpath,$(LIB_INSTALL_DIR) -lnvbufsurface -lnvbufsurftransform CFLAGS+= `pkg-config –cflags $(PKGS)`

按照上述修改后

它起作用了，但只是部分起作用。 RTSP的一些帧是旋转的，一些不是，它看起来是随机的

已经将建议的补丁应用到源代码\apps\apps-common\src\deepstream_source_bin.c:

} else{ NVGSTDS_LINK_ELEMENT (bin->cap_filter, bin->nvvidconv); NVGSTDS_LINK_ELEMENT (bin->nvvidconv, bin->cap_filter1); /// added here, line 888 GstPad *src_pad; src_pad = gst_element_get_static_pad (bin->nvvidconv, “src”); gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER, nvargus_src_pad_buffer_probe, NULL, NULL); /// NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->cap_filter1, “src”);

以下是deepstream配置文件:

[application] enable-perf-measurement=1 perf-measurement-interval-sec=1 #gie-kitti-output-dir=streamscl [tiled-display] enable=0 rows=4 columns=2 width=1280 height=720 gpu-id=0 nvbuf-memory-type=0 [source0] enable=1 #Type – 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP type=4 uri=rtsp://192.168.2.216:8554/test #num-sources=8 #drop-frame-interval=2 latency=200 gpu-id=0 # (0): memtype_device – Memory type Device # (1): memtype_pinned – Memory type Host Pinned # (2): memtype_unified – Memory type Unified cudadec-memtype=0 [sink0] enable=1 type=3 #1=mp4 2=mkv container=1 #1=h264 2=h265 codec=1 #encoder type 0=Hardware 1=Software enc-type=0 sync=0 #iframeinterval=10 bitrate=2000000 #H264 Profile – 0=Baseline 2=Main 4=High #H265 Profile – 0=Main 1=Main10 profile=0 output-file=testf.mp4 source-id=0 [streammux] gpu-id=0 ##Boolean property to inform muxer that sources are live live-source=1 batch-size=8 ##time out in usec, to wait after the first buffer is available ##to push the batch even if the complete batch is not formed batched-push-timeout=40000 ## Set muxer output width and height width=1920 height=1080 ##Enable to maintain aspect ratio wrt source, and allow black borders, works ##along with width, height properties enable-padding=0 nvbuf-memory-type=0 ## If set to TRUE, system timestamp will be attached as ntp timestamp ## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached ## attach-sys-ts-as-ntp=1

看起来添加了prob函数nvvideoconvert的源垫，而不是nvarguscamerasrc。 确保申请了这个

+ src_pad = gst_element_get_static_pad (bin->src_elem, “src”); + gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER, + nvargus_src_pad_buffer_probe, NULL, NULL); 4. 调试gst_element_get_static_pad

怀疑之前的猜测，有些帧是旋转的，有些没有，是有缺陷的。 所以进一步测试它

vbufsurface_params.transform_flip = NvBufSurfTransform_Rotate90;

事实上，我的怀疑被证明是正确的。 旋转确实起作用，但似乎同一个坐标系被旋转了几次并被推到下游。 所以不是有一个90度的旋转流，有一些帧旋转了90度，180度，270度，360度， 按照这个特定的顺序。

请注意，上面的视频发生时，试图修补源代码\apps\apps-common\src\deepstream_source_bin.c:

src_pad = gst_element_get_static_pad (bin->cap_filter, “src”); or: src_pad = gst_element_get_static_pad (bin->nvvidconv, “src”); or: src_pad = gst_element_get_static_pad (bin->cap_filter1, “src”);

尝试探测src_elem

src_pad = gst_element_get_static_pad (bin->src_elem, “src”);

但是视频完全没有旋转。 请注意，这是用于RTSP输入，如果有帮助的话。 补丁在这个函数中:

static gboolean create_rtsp_src_bin (NvDsSourceConfig * config, NvDsSrcBin * bin) { …. }