1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 <2019> 2020 2021 2022 2023 2024 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 <2019> 2020 2021 2022 2023 2024 |
<== Date ==> | <== Thread ==> |
---|
Subject: | Uploading a file with SFTP from an EPICS driver |
From: | Mark Rivers via Tech-talk <[email protected]> |
To: | "[email protected]" <[email protected]> |
Date: | Thu, 14 Nov 2019 14:39:03 +0000 |
Folks, I am working on the driver for the Newport XPS motor controllers. These controllers can execute Position-Velocity-Time (PVT) trajectory scans which are complex multi-axis coordinated motion along a user-defined path in 8-space. The trajectory
is defined in a text file, which the EPICS driver creates and then uploads to the controller. The older XPS-C8 and XPS-Q8 controllers support FTP for the upload. I have a small C implementation of FTP which the driver uses. This C code is standalone, it
only needs to call the standard socket functions. The new XPS-D8 controller does not support FTP, it only supports SFTP/SCP. I need a solution to be able to do SFTP transfers from within my C++ EPICS driver. I want the solution to work on both Linux and Windows, and ideally on vxWorks
as well. I have tried 3 solutions, and measured the time that each solution required to upload a 5KB file. The results surprised me, and I am curious if anyone can explain them. The code for the 3 solutions is in the attached xpsSFTPUpload.cpp and the test program is testSFTPUpload.cpp. Solution 1. Use libssh2. This is selected if CONFIG_SITE defines HAVE_LIBSSH2=YES. Execution time = 326 ms Solution 2. Use libcurl with the curl_easy API. This is selected if CONFIG_SITE defines HAVE_LIBCURL=YES. Execution time = 323 ms Solution 3. Run the “curl” program with the C system() call. The driver builds the text of the curl command and runs it with the system() call. This is selected if CONFIG_SITE does not define HAVE_LIBSSH2=YES or HAVE_LIBCURL=YES. Execution time (Linux) = 126 ms Execution time (Windows) = 270 ms So on Linux running the “curl” command via a call to system() is actually almost 3 times faster than calling either of the libraries directly. This surprises me. Note that I actually prefer to use Solution 3 because it is easy to implement on Windows. There are several sites that provide downloads of a pre-built “curl” executable. However, I have not been able to find any site that provides the
libssh2 or libcurl libraries in a form that can be called from Visual Studio. They are built with mingw and give errors when I try to link with Visual Studio. Note that because of the performance measurements above I am not motivated to pursue the library
solution either, since it seems to be significantly slower. Ideally I would like this driver to work on vxWorks as well, but I don’t see any solution for doing sftp from within a C program on vxWorks? Thanks, Mark |
#include <string.h> #include <stdio.h> #include <stdlib.h> #include <string> #include <xpsSFTPUpload.h> // The following version requires libcurl. #ifdef HAVE_LIBCURL #include <sys/stat.h> #include <curl/curl.h> int xpsSFTPUpload(std::string IPAddress, std::string trajectoryDirectory, std::string fileName, std::string userName, std::string password, bool verbose) { int status = 0; CURL *curl = curl_easy_init(); if (curl == 0) { fprintf(stderr, "curl_easy_init failed\n"); return -1; } std::string url = "scp://" + IPAddress + "/" + trajectoryDirectory + "/" + fileName; status |= curl_easy_setopt(curl, CURLOPT_URL, url.c_str()); status |= curl_easy_setopt(curl, CURLOPT_SSH_AUTH_TYPES, CURLSSH_AUTH_PASSWORD); std::string auth = userName + ":" + password; status |= curl_easy_setopt(curl, CURLOPT_USERPWD, auth.c_str()); status |= curl_easy_setopt(curl, CURLOPT_UPLOAD, 1); FILE *fd = fopen(fileName.c_str(), "rb"); /* open file to upload */ struct stat fileStat; fstat(fileno(fd), &fileStat); status |= curl_easy_setopt(curl, CURLOPT_INFILESIZE, (long)fileStat.st_size); status |= curl_easy_setopt(curl, CURLOPT_READDATA, fd); if (verbose) status |= curl_easy_setopt(curl, CURLOPT_VERBOSE, 1); status |= curl_easy_perform(curl); fclose(fd); curl_easy_cleanup(curl); if (status) { fprintf(stderr, "One or more curl_easy functions failed\n"); return -1; } return 0; } #elif defined HAVE_LIBSSH2 #include <libssh2.h> #include <libssh2_sftp.h> #include <osiSock.h> int xpsSFTPUpload(std::string IPAddress, std::string trajectoryDirectory, std::string fileName, std::string userName, std::string password, bool verbose) { struct sockaddr_in sockAddr; SOCKET sockFD = 0; LIBSSH2_SESSION *session = 0; LIBSSH2_SFTP_HANDLE *sftp_handle = 0; LIBSSH2_SFTP *sftp_session = 0; const char *fingerprint; int i; char mem[1024*100]; FILE *local = 0; size_t nread; char *ptr; int rc; int status = -1; std::string remoteFile = trajectoryDirectory + fileName; rc = libssh2_init(0); if (rc != 0) { fprintf(stderr, "libssh2 initialization failed (%d)\n", rc); return -1; } /* * The application code is responsible for creating the socket * and establishing the connection */ sockFD = epicsSocketCreate(AF_INET, SOCK_STREAM, 0); sockAddr.sin_family = AF_INET; sockAddr.sin_port = htons(22); sockAddr.sin_addr.s_addr = inet_addr(IPAddress.c_str()); if (connect(sockFD, (struct sockaddr*)(&sockAddr), sizeof(struct sockaddr_in)) != 0) { fprintf(stderr, "failed to connect to %s\n", IPAddress.c_str()); goto done; } /* Create a session instance */ session = libssh2_session_init(); /* Since we have set non-blocking, tell libssh2 we are blocking */ libssh2_session_set_blocking(session, 1); /* ... start it up. This will trade welcome banners, exchange keys, * and setup crypto, compression, and MAC layers */ rc = libssh2_session_handshake(session, sockFD); if (rc) { fprintf(stderr, "Failure establishing SSH session: %d\n", rc); goto done; } /* At this point we haven't yet authenticated. The first thing to do * is check the hostkey's fingerprint against our known hosts Your app * may have it hard coded, may go to a file, may present it to the * user, that's your call */ fingerprint = libssh2_hostkey_hash(session, LIBSSH2_HOSTKEY_HASH_SHA1); fprintf(stderr, "Fingerprint: "); for (i = 0; i < 20; i++) { fprintf(stderr, "%02X ", (unsigned char)fingerprint[i]); } fprintf(stderr, "\n"); /* Authenticate via password */ if (libssh2_userauth_password(session, userName.c_str(), password.c_str())) { fprintf(stderr, "Authentication by password failed.\n"); goto done; } local = fopen(fileName.c_str(), "rb"); if (!local) { fprintf(stderr, "Can't open local file %s\n", fileName.c_str()); goto done; } sftp_session = libssh2_sftp_init(session); if(!sftp_session) { fprintf(stderr, "Unable to init SFTP session\n"); goto done; } /* Request a file via SFTP */ sftp_handle = libssh2_sftp_open(sftp_session, remoteFile.c_str(), LIBSSH2_FXF_WRITE|LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC, LIBSSH2_SFTP_S_IRUSR|LIBSSH2_SFTP_S_IWUSR| LIBSSH2_SFTP_S_IRGRP|LIBSSH2_SFTP_S_IROTH); if(!sftp_handle) { fprintf(stderr, "Unable to open remote file %s with SFTP\n", remoteFile.c_str()); goto done; } do { nread = fread(mem, 1, sizeof(mem), local); if(nread <= 0) { /* end of file */ break; } ptr = mem; do { /* write data in a loop until we block */ rc = libssh2_sftp_write(sftp_handle, ptr, nread); if(rc < 0) break; ptr += rc; nread -= rc; } while(nread); } while(rc > 0); status = 0; done: if (local) fclose(local); if (sftp_handle) libssh2_sftp_close(sftp_handle); if (sftp_session) libssh2_sftp_shutdown(sftp_session); if (session) { libssh2_session_disconnect(session, "Normal Shutdown"); libssh2_session_free(session); } if (sockFD) { epicsSocketDestroy(sockFD); } return status; } #else // If HAVE_LIBCURL is not YES then try to execute the curl system command int xpsSFTPUpload(std::string IPAddress, std::string trajectoryDirectory, std::string fileName, std::string userName, std::string password, bool verbose) { std::string verboseOption = verbose ? " --verbose " : " --silent "; std::string command = "curl -k -u " + userName + ":" + password + " -T " + fileName + verboseOption + " scp://" + IPAddress + trajectoryDirectory + "/"; if (verbose) printf("curl command=%s\n", command.c_str()); int status = system(command.c_str()); return status; } #endif
#include <stdio.h> #include <asynDriver.h> #include <xpsSFTPUpload.h> int main(int argc, char *argv[]) { bool sftpVerbose = true; asynUser *pasynUser = pasynManager->createAsynUser(0, 0); asynPrint(pasynUser, ASYN_TRACE_ERROR, "Calling xpsSFTPUpload\n"); int status = xpsSFTPUpload("164.54.160.71", "/Admin/Public/Trajectories", "TrajectoryScan.trj", "Administrator", "Administrator", sftpVerbose); asynPrint(pasynUser, ASYN_TRACE_ERROR, "xpsSFTPUpload returned status=%d\n", status); return 0; }