RE: AreaDetector ADFFmpeg with FFmpeg 5.1

["Daykin, Evan via Tech-talk" <tech-talk at aps.anl.gov>]

I found the issue: AVPacket takes ownership of whatever memory is passed to it, and it is unref’d during the call to avcodec_receive_packet. I’ll need to figure out how to use ffmpeg’s custom memory-management callbacks to instead use NDArrayPool.

 

From: Daykin, Evan <daykin at frib.msu.edu>
Sent: Monday, June 12, 2023 10:57 AM
To: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov
Subject: RE: AreaDetector ADFFmpeg with FFmpeg 5.1

 

[EXTERNAL] This email originated from outside of FRIB

Hi Mark,

 

I am indeed working on this, but I’ve run into an issue I don’t quite understand.

 

In the actual codec bit of processCallbacks, the call to avcodec_encode_video2() is deprecated in favor of the decoupled send/receive avcodec_send_frame(AVFrame*)/avcodec_receive_packet(AVPacket*)(Relevant diff below).  So my intent is to do the following:

 

1.       Allocate block of memory for encoded frame using NDArrayPool

2.       Allocate an ffmpeg packet using ffmpeg

3.       Point ffmpeg packet data to the NDArrayPool allocated memory

4.       Send raw frame (scPicture) to the encoder for encoding w/avcodec_send_frame

5.       Pull encoded frame from encoder w/avcodec_receive_packet

6.       Encoded data ends up in this->jpeg

 

But, when I run this, there is no data returned from the server. If I instead maintain two blocks of memory, one maintained by ffmpeg’s memory pool and the other with AreaDetector, then memcpy the relevant data from ffmpeg to AD, everything works fine. However, I’d prefer if I can just pass around pointers to the same data.

 

 

     if (this->jpeg) {

         this->jpeg->release();

     }

-   

-    /* Convert it to a jpeg */       

-    this->jpeg = this->pNDArrayPool->alloc(1, &size, NDInt8, 0, NULL);

-

-    AVPacket pkt;

-    int got_output;

-    av_init_packet(&pkt);

-    pkt.data = "" // packet data will be allocated by the encoder

-    pkt.size = c->width * c->height;

 

+    /* Convert it to a jpeg */

+    this->jpeg = this->pNDArrayPool->alloc(1, &size, NDInt8, 0, NULL);

+    //this->jpeg->reserve();

+    AVPacket* pkt = av_packet_alloc();

+    pkt->data = "">

+    pkt->size = c->width*c->height;

+    //pkt->buf = av_buffer_alloc(c->coded_width*c->coded_height);

     // needed to stop a stream of "AVFrame.format is not set" etc. messages

     scPicture->format = c->pix_fmt;

     scPicture->width = c->width;

     scPicture->height = c->height;

-

-    if (avcodec_encode_video2(c, &pkt, scPicture, &got_output)) {

+    int sts;

+    sts = avcodec_send_frame(c, scPicture);

+    char err[64];

+    if(sts) {

+        av_strerror(sts, err, 64*sizeof(char));

         asynPrint(this->pasynUserSelf, ASYN_TRACE_ERROR,

-            "%s:%s: Encoding jpeg failed\n",

-            driverName, functionName);

-        got_output = 0; // got_output is undefined on error, so explicitly set it for use later

+                  "%s%s: Encoding jpeg failed - %d - %s\n",

+                   driverName, functionName, sts, err);

+   

     }

-

-    if (got_output) {

-        this->jpeg->dims[0].size = pkt.size;

-        av_packet_unref(&pkt);

+    sts = avcodec_receive_packet(c,pkt);

+    if (sts){

+        av_strerror(sts, err, 64*sizeof(char));

+        asynPrint(this->pasynUserSelf, ASYN_TRACE_ERROR,

+                  "%s:%s: Recv packet failed - %d - %s\n",

+                  driverName, functionName, sts, err);

     }

 

+    this->jpeg->dims[0].size = pkt->size;

+    //memcpy(this->jpeg->pData, pkt->data, pkt->size);

     //printf("Frame! Size: %d\n", this->jpeg->dims[0].size);

 

     /* signal fresh_frame to output plugin and unlock mutex */

     for (int i=0; i<config.server_maxconn; i++) {

         pthread_cond_signal(&(this->cond[i]));

     }

     pthread_mutex_unlock(&this->mutex);

 

+    av_packet_free(&pkt);

     /* We must enter the loop and exit with the mutex locked */

     this->lock();

-

     /* Update the parameters.  */

     callParamCallbacks(0, 0);

//    gettimeofday(&end, NULL);  

 

 

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, June 5, 2023 11:35 AM
To: Daykin, Evan <daykin at frib.msu.edu>; tech-talk at aps.anl.gov
Subject: RE: AreaDetector ADFFmpeg with FFmpeg 5.1

 

[EXTERNAL] This email originated from outside of FRIB

I don’t know of any ongoing work on this, so it would be great if you could do it.

Mark

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Daykin, Evan via Tech-talk
Sent: Monday, June 5, 2023 10:15 AM
To: tech-talk at aps.anl.gov
Subject: AreaDetector ADFFmpeg with FFmpeg 5.1

 

Hi tech-talk,

 

We are in the process of updating our systems to Debian Bookworm. The currently available FFmpeg toolchain packages for Bookworm are on version 5.1. Are there any plans to update the ADFFmpeg plugin to support this? I’m currently trying to update our fork of the plugin, but I’m running into some hard-to-find runtime errors that I’m sure are some combination of my unfamiliarity with the new API, and the fact that I’m using the pre-built FFmpeg library without debug symbols. If there isn’t yet an existing effort to do this, I’m happy to start.

 

-Evan

 

Evan Daykin Controls Engineer Facility for Rare Isotope Beams Michigan State University 640 South Shaw Lane East Lansing, MI 48824, USA Tel. 517-908-7678 Email: daykin at frib.msu.edu