Re: [EXTERNAL] Re: EPICS Software Supply Chain Risk Management (SSCRM)

[Gedare Bloom via Tech-talk <tech-talk at aps.anl.gov>]

Hi Josh,

I've been looking at software security and supply chain security for EPICS for a few years from the perspective of an academic security researcher and an open-source project maintainer (Professor at University of Colorado Colorado Springs and maintainer of RTEMS). So we do it on academic timelines and resources, but also on academic grants or contracts. We have presented on several of these questions a few times, most recently:

* https://blog.trustedci.org/2023/01/January-webinar-RTEMS.html

With an archived recording at: https://youtu.be/KBc81L2Ze78

We have also presented a few times directly to the EPICS Collaboration Meetings:

* 2021: https://indico.fhi-berlin.mpg.de/event/52/contributions/582/

* 2023: https://indico.fnal.gov/event/58280/contributions/264576/

This effort started awhile back right here on this mailing list:

https://epics.anl.gov/tech-talk/2018/msg00833.php

We do have some efforts looking at SSCRM, but nothing particularly mature or ready to adopt.  Primarily we have looked into possible ways to secure the boot process by measuring software integrity. We also look at hardening and testing the software and networks, with a big effort currently focused on vulnerability discovery via fuzz-testing.

In the open-source and academic community, there are some other interesting alternatives to inCommon, such as https://in-toto.io/. However, as you say, buying a commercial solution shifts the responsibility (maybe not the actual risks though). For this reason, I doubt there is a one-size-fits-all for EPICS users, but for core scientific cyberinfrastructure such as EPICS and RTEMS it is more viable and sustainable to adopt free and open-source solutions that rely on academic/public support, than it is to rely on a commercial vendor who may go out of business one day when their solution fails for one big customer. It is a different kind of risk.

I have much more to say on this topic, but this is not the right venue perhaps for so much detail. I'd be happy to talk more about security related to EPICS with you (or anyone else). This is my open-source email address, and my current academic email is gbloom at uccs.edu.

Gedare

On Tue, Jul 11, 2023 at 12:47 PM Joshua Einstein-Curtis via Tech-talk <tech-talk at aps.anl.gov> wrote:

Just to tack on if I missed it, the workshop also brought up the NIST recommendations 800-82 (https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-82r2.pdf). We can talk all we want about "security" and "safety", but until we define what that really means in each context, the discussion is another problem altogether. Especially with the phrase "safety-critical" -- formally verified code does not mean the communication network between meets the same level of safety. I really should read those ICALEPS reports though. 

EPICS not being a source of malware is one thing, but what could happen to an epics system if malware is on the network is another. Just saying one has a "trusted developer" is already pushing the risk off on whoever validated that developer in the first place. "Where the trust falls" if you will. That's one reason inCommon is so interesting to me -- it is a pay-to-verify identify system at the organizational level.

Buying a commercial solution outright (EPICS or otherwise) shifts the risk and responsibility of what happens when there is a failure to the commercial partner. Like the DO-254 certification process -- as long as you know the requirements, it can be certified....until something happens that wasn't in the requirements.

- Josh EC

On Tue, Jul 11, 2023 at 6:48 AM Evans, Richard K. (GRC-H000) via Tech-talk <tech-talk at aps.anl.gov> wrote:

Hi Shen,

 

Oh, yes. We’ll do all those things too as appropriate for each system if that time comes. As I mentioned to Jonathan, my aim in this post was only to share what the NASA GRC-ATF rationale is for getting EPICS on the agency’s “Assessed and Cleared” list. This has nothing to do with how well suited it is for any specific use but rather is focused on it explaining why we have confidence that it won’t be a source of malware from a development point-of-view (hence the whole Trusted Developer/Trusted Repository model).

 

In other words, we have to show that it is safe for the Agency to install at all before we can even begin to discuss if it is capable of being used to mitigate/control a given hazard inherent in the system that needs to be controlled.  That said, I’m encouraged by your response here.. It sounds like if/when we get to that point there is indeed a path for using it in situations that require safety critical controls.  

 

Thank you!

/Rich

 

From: Shen, Guobao <gshen at anl.gov>
Sent: Tuesday, July 11, 2023 8:41 AM
To: Evans, Richard K. (GRC-H000) <richard.k.evans at nasa.gov>; Jonathan Jacky <jon.p.jacky at gmail.com>
Cc: tech-talk at aps.anl.gov; S Banerian <banerian at uw.edu>
Subject: Re: [EXTERNAL] Re: EPICS Software Supply Chain Risk Management (SSCRM)

 

Rich,

To be compliant with the cybersecurity requirement, you might also consider to design the system as a whole together with your IT infrastructure.

For example, deploying a firewall dedicated to your instrument control to isolate your whole system from the public network, from your campus network, and from your office network in case you have dedicated firewall for that; or further adopt other method (e.g., MFA) to restrict access to your facility instrument control network which could reduce the risk.

 

Together with the other technical solutions mentioned, you might be able to satisfy the needs.

 

Thanks,

Guobao

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Evans, Richard K. (GRC-H000) via Tech-talk <tech-talk at aps.anl.gov>
Date: Monday, July 10, 2023 at 6:58 PM
To: Jonathan Jacky <jon.p.jacky at gmail.com>
Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>, S Banerian <banerian at uw.edu>
Subject: Re: [EXTERNAL] Re: EPICS Software Supply Chain Risk Management (SSCRM)

Jonathan, 

 

Thank you so much for your response. I must confess that my intent in this post was originally only to talk about software safety in the (NIST) context of safe from “vulnerabilities” due to “malicious code” from “bad actors” and such as Open Source Software, but your work on showing how EPICS can also be used in safety critical applications to mitigate what we would call “hazardous” operations is equally important topic to us as well.

 

Is it your position that any software that has been demonstrated to be “safe” for use in mitigating risks to personnel health and safety is also safe from external vulnerabilities due to the development process? That seems logical to me. However the verification process you are referring to seems like it is highly dependent on the specific application and specific integration of otherwise established code. My goal today is only to get feedback on the (NIST) trustworthiness of the developer (ANL) and repository as (NIST) safe for use at NASA in non-safety critical applications. 

 

That said, thanks again.. I’m saving your reply for after we’ve solved the supply chain reliability question and are working on using EPICS in safety critical applications to mitigate hazardous operations. 

 

Cheers!

/Rich

 

 

 

 

I’m eager to establish a process for using EPICS in 

---

From: Jonathan Jacky <jon.p.jacky at gmail.com>
Sent: Monday, July 10, 2023 6:27:27 PM
To: Evans, Richard K. (GRC-H000) <richard.k.evans at nasa.gov>
Cc: S Banerian <banerian at uw.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: [EXTERNAL] Re: EPICS Software Supply Chain Risk Management (SSCRM)

 

CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.

 

> "Given that EPICS is open source and used around the world, How do you know that EPICS is safe?"

 

> Question 2 - Has this question been addressed by anyone previously? .. and are there any charts or papers that I can cite and/or reference when I talk with the NASA CIO folks about EPICS and SSCRM.

 

Several years ago, staff at the University of Washington Medical Cyclotron Facility, and faculty and staff at the University of Washington Department of Computer Science and Engineering, did a multi-year project to apply modern formal verification methods and technology both to the EPICS core and an EPICS application program (an EPICS database on an IOC).  

 

The project and its results were reported at the 2017 ICALEPS meeting.    The slides give a quick overview of the project:

 

https://accelconf.web.cern.ch/icalepcs2017/papers/tudpl02.pdf   (paper, 7 pages)

 

https://accelconf.web.cern.ch/icalepcs2017/talks/tudpl02_talk.pdf (13 slides)

 

- Jonathan Jacky