EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Uploading a file with SFTP from an EPICS driver
From: Mark Rivers via Tech-talk <[email protected]>
To: "[email protected]" <[email protected]>
Date: Thu, 14 Nov 2019 14:39:03 +0000

Folks,

 

I am working on the driver for the Newport XPS motor controllers.  These controllers can execute Position-Velocity-Time (PVT) trajectory scans which are complex multi-axis coordinated motion along a user-defined path in 8-space.  The trajectory is defined in a text file, which the EPICS driver creates and then uploads to the controller.  The older XPS-C8 and XPS-Q8 controllers support FTP for the upload.  I have a small C implementation of FTP which the driver uses.  This C code is standalone, it only needs to call the standard socket functions.

 

The new XPS-D8 controller does not support FTP, it only supports SFTP/SCP.  I need a solution to be able to do SFTP transfers from within my C++ EPICS driver.  I want the solution to work on both Linux and Windows, and ideally on vxWorks as well.

 

I have tried 3 solutions, and measured the time that each solution required to upload a 5KB file. The results surprised me, and I am curious if anyone can explain them.

 

The code for the 3 solutions is in the attached xpsSFTPUpload.cpp and the test program is testSFTPUpload.cpp.

 

Solution 1.  Use libssh2.  This is selected if CONFIG_SITE defines HAVE_LIBSSH2=YES.

Execution time = 326 ms

 

Solution 2.  Use libcurl with the curl_easy API.  This is selected if CONFIG_SITE defines HAVE_LIBCURL=YES.

Execution time = 323 ms

 

Solution 3. Run the “curl” program with the C system() call.  The driver builds the text of the curl command and runs it with the system() call. This is selected if CONFIG_SITE does not define HAVE_LIBSSH2=YES or HAVE_LIBCURL=YES.

Execution time (Linux) = 126 ms

Execution time (Windows) = 270 ms

 

So on Linux running the “curl” command via a call to system() is actually almost 3 times faster than calling either of the libraries directly.  This surprises me.

 

Note that I actually prefer to use Solution 3 because it is easy to implement on Windows.  There are several sites that provide downloads of a pre-built “curl” executable.  However, I have not been able to find any site that provides the libssh2 or libcurl libraries in a form that can be called from Visual Studio.  They are built with mingw and give errors when I try to link with Visual Studio.  Note that because of the performance measurements above I am not motivated to pursue the library solution either, since it seems to be significantly slower.

 

Ideally I would like this driver to work on vxWorks as well, but I don’t see any solution for doing sftp from within a C program on vxWorks?

 

Thanks,

Mark

 

#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <string>

#include <xpsSFTPUpload.h>

// The following version requires libcurl.
#ifdef HAVE_LIBCURL
#include <sys/stat.h>
#include <curl/curl.h>

int xpsSFTPUpload(std::string IPAddress, std::string trajectoryDirectory, std::string fileName, 
                  std::string userName, std::string password, bool verbose) {
  int status = 0;
  CURL *curl = curl_easy_init();
  if (curl == 0) {
    fprintf(stderr, "curl_easy_init failed\n");
    return -1;
  }
  std::string url = "scp://" + IPAddress + "/" + trajectoryDirectory + "/" + fileName;
  status |= curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
  status |= curl_easy_setopt(curl, CURLOPT_SSH_AUTH_TYPES, CURLSSH_AUTH_PASSWORD);
  std::string auth = userName + ":" + password;
  status |= curl_easy_setopt(curl, CURLOPT_USERPWD, auth.c_str());
  status |= curl_easy_setopt(curl, CURLOPT_UPLOAD, 1);
  FILE *fd = fopen(fileName.c_str(), "rb"); /* open file to upload */
  struct stat fileStat;
  fstat(fileno(fd), &fileStat);
  status |= curl_easy_setopt(curl, CURLOPT_INFILESIZE, (long)fileStat.st_size);
  status |= curl_easy_setopt(curl, CURLOPT_READDATA, fd);
  if (verbose) status |= curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
  status |= curl_easy_perform(curl);
  fclose(fd);
  curl_easy_cleanup(curl);
  if (status) {
    fprintf(stderr, "One or more curl_easy functions failed\n");
    return -1;
  }
  return 0;
}

#elif defined HAVE_LIBSSH2

#include <libssh2.h>
#include <libssh2_sftp.h>
#include <osiSock.h>

int xpsSFTPUpload(std::string IPAddress, std::string trajectoryDirectory, std::string fileName, 
                  std::string userName, std::string password, bool verbose) {
  struct sockaddr_in sockAddr;
  SOCKET sockFD = 0;
  LIBSSH2_SESSION *session = 0;
  LIBSSH2_SFTP_HANDLE *sftp_handle = 0;
  LIBSSH2_SFTP *sftp_session = 0;
  const char *fingerprint;
  int i;
  char mem[1024*100];
  FILE *local = 0;
  size_t nread;
  char *ptr;
  int rc;
  int status = -1;  
  std::string remoteFile = trajectoryDirectory + fileName;

  rc = libssh2_init(0);
  if (rc != 0) {
    fprintf(stderr, "libssh2 initialization failed (%d)\n", rc);
    return -1;
  }
 
  /*
   * The application code is responsible for creating the socket
   * and establishing the connection
   */ 
  sockFD = epicsSocketCreate(AF_INET, SOCK_STREAM, 0);
 
  sockAddr.sin_family = AF_INET;
  sockAddr.sin_port = htons(22);
  sockAddr.sin_addr.s_addr = inet_addr(IPAddress.c_str());
  if (connect(sockFD, (struct sockaddr*)(&sockAddr),
             sizeof(struct sockaddr_in)) != 0) {
    fprintf(stderr, "failed to connect to %s\n", IPAddress.c_str());
    goto done;
  }
 
  /* Create a session instance */ 
  session = libssh2_session_init();

  /* Since we have set non-blocking, tell libssh2 we are blocking */ 
  libssh2_session_set_blocking(session, 1);

  /* ... start it up. This will trade welcome banners, exchange keys,
   * and setup crypto, compression, and MAC layers */ 
  rc = libssh2_session_handshake(session, sockFD);

  if (rc) {
    fprintf(stderr, "Failure establishing SSH session: %d\n", rc);
    goto done;
  }
 
  /* At this point we haven't yet authenticated.  The first thing to do
   * is check the hostkey's fingerprint against our known hosts Your app
   * may have it hard coded, may go to a file, may present it to the
   * user, that's your call
   */ 
  fingerprint = libssh2_hostkey_hash(session, LIBSSH2_HOSTKEY_HASH_SHA1);

  fprintf(stderr, "Fingerprint: ");
  for (i = 0; i < 20; i++) {
    fprintf(stderr, "%02X ", (unsigned char)fingerprint[i]);
  }
  fprintf(stderr, "\n");
 
  /* Authenticate via password */ 
  if (libssh2_userauth_password(session, userName.c_str(), password.c_str())) {
    fprintf(stderr, "Authentication by password failed.\n");
    goto done;
  }

  local = fopen(fileName.c_str(), "rb");
  if (!local) {
    fprintf(stderr, "Can't open local file %s\n", fileName.c_str());
    goto done;
  }
 
  sftp_session = libssh2_sftp_init(session);
  if(!sftp_session) {
    fprintf(stderr, "Unable to init SFTP session\n");
    goto done;
  }
 
  /* Request a file via SFTP */
  sftp_handle = libssh2_sftp_open(sftp_session, remoteFile.c_str(),
                    LIBSSH2_FXF_WRITE|LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC,
                    LIBSSH2_SFTP_S_IRUSR|LIBSSH2_SFTP_S_IWUSR|
                    LIBSSH2_SFTP_S_IRGRP|LIBSSH2_SFTP_S_IROTH);
 
  if(!sftp_handle) {
    fprintf(stderr, "Unable to open remote file %s with SFTP\n", remoteFile.c_str());
    goto done;
  }

  do {
    nread = fread(mem, 1, sizeof(mem), local);
    if(nread <= 0) {
        /* end of file */ 
        break;
    }
    ptr = mem;
    do {
      /* write data in a loop until we block */ 
      rc = libssh2_sftp_write(sftp_handle, ptr, nread);
       if(rc < 0)
          break;
      ptr += rc;
      nread -= rc;
    } while(nread);
 
  } while(rc > 0);
  status = 0;
 
  done:
  if (local) fclose(local);
  if (sftp_handle) libssh2_sftp_close(sftp_handle);
  if (sftp_session) libssh2_sftp_shutdown(sftp_session);
  if (session) {
    libssh2_session_disconnect(session, "Normal Shutdown");
    libssh2_session_free(session);
  }
  if (sockFD) {
    epicsSocketDestroy(sockFD);
  }
  return status;
}

#else
// If HAVE_LIBCURL is not YES then try to execute the curl system command
int xpsSFTPUpload(std::string IPAddress, std::string trajectoryDirectory, std::string fileName, 
                  std::string userName, std::string password, bool verbose) {
  std::string verboseOption = verbose ? " --verbose " : " --silent ";
  std::string command = "curl -k -u " + userName + ":" + password + " -T " + fileName + verboseOption + " scp://" + IPAddress + trajectoryDirectory + "/";
  if (verbose) printf("curl command=%s\n", command.c_str());
  int status = system(command.c_str());
  return status;
}

#endif
#include <stdio.h>
#include <asynDriver.h>
#include <xpsSFTPUpload.h>

int main(int argc, char *argv[])
{
  bool sftpVerbose = true;
  asynUser *pasynUser = pasynManager->createAsynUser(0, 0);
  asynPrint(pasynUser, ASYN_TRACE_ERROR, "Calling xpsSFTPUpload\n");
  int status = xpsSFTPUpload("164.54.160.71", "/Admin/Public/Trajectories", "TrajectoryScan.trj", 
                             "Administrator", "Administrator", sftpVerbose);
  asynPrint(pasynUser, ASYN_TRACE_ERROR, "xpsSFTPUpload returned status=%d\n", status);
  return 0;
}

Replies:
RE: Uploading a file with SFTP from an EPICS driver Freddie Akeroyd - UKRI STFC via Tech-talk
Re: Uploading a file with SFTP from an EPICS driver J. Lewis Muir via Tech-talk

Navigate by Date:
Prev: EPICS Codeathon 2020 Johnson, Andrew N. via Tech-talk
Next: RE: Uploading a file with SFTP from an EPICS driver Freddie Akeroyd - UKRI STFC via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: EPICS Codeathon 2020 Rose, Austen (DLSLtd, RAL, LSCI) via Tech-talk
Next: RE: Uploading a file with SFTP from an EPICS driver Freddie Akeroyd - UKRI STFC via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
ANJ, 20 Nov 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·