The most heralded federal agency is in serious risk of a major cyber attack and no one seems to care.
Not NASA executives. Not the contractor hired to protect its end-user
devices. And especially not the everyday employees who send rockets
into space.
Internal documents obtained by Federal News Radio indicate NASA has
anywhere from hundreds of thousands to millions of out-of-date patches
at every center across the country.
Security Scorecard, a cybersecurity company, found as many as 10,000
pings coming directly from NASA’s network to known malware hosts, some
lasting weeks, if not months.
Multiple sources say Hewlett Packard Enterprise (HPE), the contractor
hired to protect NASA’s desktops and end-user devices under a $2.5
billion contract
called the Agency Consolidated End-user Services (ACES), is
uncooperative at best and negligent at worst, and a major reason the
agency’s data and systems are at risk.
One NASA source said the breach suffered by the Office of Personnel
Management and the Office of Management and Budget’s cyber sprint didn’t
serve as wake-up calls for the space agency, and cybersecurity remains a
serious issue.
“HPE admits that it doesn’t have the resources to keep up with the patching and that this has been going on since the contract was awarded several years ago.” — Senior NASA IT official
“At the heart of it all are three or four major problems. Two of
those problems are most serious. First, this is an IT operations issue.
Security of NASA’s data and systems are seriously weakened and prone to
compromise because IT operations appears to be failing at keeping up
with a basic operational function and that function is patching of
applications and operating systems,” said a senior NASA IT engineer
familiar with IT operations. “Whoever is responsible for maintaining the
daily operational health of systems and applications should be held
accountable for allowing the IT environment to get to this state. This
is a very clear example of malfeasance. Second, from what I understand
in conversation with the individuals at NASA headquarters who manage the
ACES contract, it is a requirement that HPE patch certain applications.
The conversation taking place is that HPE admits that it doesn’t have
the resources to keep up with the patching and that this has been going
on since the contract was awarded several years ago. No one within
NASA’s leadership managing the ACES contract or at HPE seems to be very
concerned. This is very disturbing and disconcerting.”
It’s not just HPE, however. Two other sources said in separate interviews the NASA culture focuses on mission first and foremost, and cybersecurity a distant second.
“I would say NASA is worse than average when it comes to
cybersecurity across the government,” said a former NASA official. “The
number one problem is poor IT governance. There is no centralized
authority that is empowered to do anything about security issues at the
agency, including the chief information officer. The centers and other
organizations are doing their own things, and cybersecurity is not
viewed as a mission problem. It’s viewed as a CIO problem. But it really
is a mission problem and until the agency understands that, their cyber
problems will remain because it’s not something the CIO can solve. It
needs strong leadership from the administrator.”
| 5 TAKEAWAYS |
| NASA has anywhere from hundreds of thousands to millions of out-of-date patches at every center across the country. |
| Multiple sources say Hewlett Packard Enterprise (HPE), the contractor hired to protect NASA’s desktops and end-user devices, is uncooperative at best and negligent at worst. |
| Two other sources said in separate interviews the NASA culture focuses on mission first and foremost, and cybersecurity a distant second. |
| A NASA spokeswoman refuted many of the claims the space agency isn’t focused on securing its networks and data. |
| A recent scan of the Internet found 10,000 pings emanating from NASA’s network back to known malware hosts. |
The former official, who requested anonymity, added leadership
doesn’t prioritize cybersecurity from a budget standpoint and their
decisions don’t reflect the importance or urgency of cybersecurity to
the staff.
The former official pointed out that some of the patching challenges
are related to NASA putting a freeze on all IT systems and software
prior to a mission launch. The former official said because management
doesn’t want anything to go wrong with a launch of a space craft,
systems and networks are kept constant to ensure mission success.
But once NASA lifts the freeze, the former official said the agency
and especially HPE haven’t kept up with protecting the systems and
devices.
“If you can’t show them a direct mission impact, empirical evidence,
you get discounted, especially around cybersecurity,” the NASA IT
engineer said. “They are just doing compliance activities. When you talk
about security programs at NASA, we seem to wait for DHS to tell us
what to do. NASA lacks a focus on cyber and there is no real strategy
for dealing with internal weaknesses.”
A government official with knowledge of NASA echoed similar concerns.
The official said over the years, IT executives have brought up the
poor patching by HPE with leaders and received little to no response.
“Really, security should be a mission element, but that is the piece they haven’t grasped yet.” — Government Official
“The whole notion of balancing mission versus security is something
every agency is challenged with, but NASA’s leadership is not as in tune
in striking the right balance of mission versus security,” the source
said. “Really, security should be a mission element, but that is the
piece they haven’t grasped yet.”
A NASA spokeswoman refuted many of the claims the space agency isn’t focused on securing its networks and data.
“NASA takes cybersecurity very seriously and is committed to devoting
the necessary resources to ensure the safety and security of the
agency’s information and information technology systems,” the
spokeswoman said in email answers to questions from Federal News Radio.
“The agency combines internal resources, such as its dedicated
cybersecurity team, with the resources made available through its active
participation in federal cybersecurity programs and initiatives to
ensure its entire infrastructure is constantly protected and
operational.”
The spokeswoman said NASA validates the number of devices monthly and
uses enterprise-level continuous monitoring tools that gather software
inventory and patch status through the ACES contract.
“Since the 2015 Cybersecurity Sprint, NASA has made substantial
progress in tracking and managing vulnerabilities,” the spokeswoman
said. “This agency effort is reflected in [Feb. 15’s] Department of
Homeland Security Cyber Hygiene report on NASA, which shows zero
critical vulnerabilities older than 30 days since September 2015.”
But sources pointed out that DHS is looking at external-facing
systems only, and it’s the soft underbelly of any organization that puts
it at more risk.
Internal documents from last summer say in the most severe instances
the missing patches could open the door for a hacker to take over
privileged administrative rights, and could let a hacker execute malware
through a commonly used software title, meaning they are behind the
firewall and other external cyber defenses with little effort.
Follow-up emails to NASA asking for clarification whether the DHS
hygiene report focused on external or internal systems were not
answered.
The data breach OPM suffered last year exemplifies this type of
problem that many agencies face. Hackers obtained the credentials of a
contractor through a phishing attack to breach OPM’s external network
defenses. Once they were inside, hopping around from system to system
and collecting data wasn’t difficult.
Vulnerable today as OPM was last summer
Sources and experts say NASA is as vulnerable today as OPM was before it was attacked.
A recent scan of the Internet found 10,000 pings emanating from NASA’s network back to known malware hosts.
Sam Kassoumeh, the chief operating officer and co-founder of Security
Scorecard, which regularly conducts scans of the public Internet, said
the malware activity coming from NASA is astonishing.
“Every company in the world probably has a malware infection.
Everybody accidentally clicks a link or opens an attachment they
shouldn’t have. So it’s not necessarily surprising to see that a company
has a malware infection, it’s very common,” Kassoumeh said. “What you
generally want to see in an organization that has a very healthy
security posture, you want to see an infection, so you see a spike, and
then maybe in the same day, the malware is remediated and that spike
goes away. But when you see malware spike and then persist over days or
weeks or months, that tells us there are little to no internal security
controls inside the organization to detect and respond to incidents.”
Security Scorecard collects threat intelligence from across the
Internet. It doesn’t focus on any one organization, but vacuums up all
data. He said then when the analysis shows a lot of activity, Security
Scorecard attributes where the beaconing is coming from, in this case
NASA.
“We’ve seen just loads of various malware families, these different
types of malware, actively beaconing from inside of NASA’s technology
infrastructure, and some of the malware duration goes on for not just a
day, but sometimes even for weeks or for months. We see the same malware
family signals,” Kassoumeh said. “We see not just the beacon, but the
source IP address of the machine that’s beaconing.”
When malware attacks persist for weeks or months, it means hackers already are in the network, Kassoumeh said.
“We are seeing over 40 unique malware families over the past year.
That doesn’t mean that there are 40 unique cases of malware currently
infecting NASA. I’m claiming that over the past year from February 2015
to this year, we have seen 40 different families of malware emanating
signals from [NASA’s] digital footprint,” Kassoumeh said. “Some of these
malware families are some of the nastiest known viruses in existence.
They are not the run of the mill, click this link and a bunch of popups
or spam flood your computer or your personal email accounts starts
sending spam email to all of your contacts. That may be the minority of
the behavior, but some of the malware families that we’ve seen over the
course of the year are some of the nastiest known really in existence.”
The NASA spokeswoman said the Security Scorecard analysis is incorrect.
“The agency’s continuous monitoring tools and scans … and various independent third-party audits … do not support this claim of a broad malware infection in NASA’s IT infrastructure.” — NASA spokeswoman
“NASA regularly receives information, observations and findings from
various channels,” she said. “The agency’s continuous monitoring tools
and scans, a set of monitoring and scans performed by Department of
Homeland Security, and various independent third-party audits of NASA’s
computing environment do not support this claim of a broad malware
infection in NASA’s IT infrastructure.”
Yet the spokeswoman couldn’t answer how many patches NASA is in need
of because “the number of needed patches is not static due to the
constant stream of vendor-released patches.”
But internal NASA documents from last summer — after the White
House’s cyber sprint ended — tell a different story about the health of
the agency’s internal networks. Headquarters and every center are
struggling with securing systems with between 10,000 and 138,000 missing
critical patches.
Sources asked Federal News Radio not to share specifics about each
center’s missing patches for fear of making those organizations more of a
target by hackers.
NASA paid $35M to Hewlett-Packard Enterprise Services
The union representing NASA Ames employees raised these concerns to
executives over the summer after the OPM data breach became public.
In an undated letter sent by union officials at AMES, and obtained by
Federal News Radio, employees brought their concerns about patches and
the ACES contract up to leadership. The contents and timing of the
letter match up with the internal documents Federal News Radio got a
hold of.
“In a 2014 settlement, the ACES vendor [HPE] was paid some $35
million above and beyond the terms, conditions and fees of the original
contract, but yet NASA continues to pay this vendor millions of dollars
each year, when they have not fulfilled their obligation and continue to
place NASA sensitive data and information systems at risk by not
applying close to 700,000 critical patches to NASA’s systems and
applications,” the letter stated. “In addition to this, NASA does not
have a reasonable inventory of its assets. Under the settlement
agreement, the ACES vendor was to provide NASA with a system to manage
IT assets. To date, and nearly a year after the settlement, ACES has no
system in place to manage IT assets. Without a robust asset inventory,
there is no way that a protection solution can be implemented; one can’t
protect what they don’t know needs protection. Similarly, NASA does not
have full visibility into its antivirus posture. The ACES vendor has
yet to provide full visibility into virus activity within the NASA
infrastructure, yet NASA has paid millions of dollars for this
information but to date, has limited or no visibility, leaving
bargaining unit members and NASA sensitive information at risk.”
NASA confirmed the $35 million payout to HPE and the fact it picked
up the first three-year option of the contract, extending ACES to
October 2018.
“Hewlett Packard Enterprise takes security very seriously and remains
committed to our close partnership with NASA,” said a Hewlett Packard
Enterprise (HPE) spokesperson in an email to Federal News Radio.
Sources say HPE threatened to sue NASA in 2014 over a disagreement in
the contract terms over how many email seats the contractor was
supposed to support. That led to a less than harmonious relationship
between HPE and NASA, which many say is contributing to the patching
problems.
“HPE is going to be very reluctant to do anything more than they have
to and the only way is by withholding money, but NASA stopped doing
that and gave away strength they had,” said the former NASA official.
“When HPE threatened the lawsuit, NASA should’ve ended the contract.
NASA had an opportunity to get out of the contract and they didn’t.”
The government official with knowledge of NASA said HPE is so far
behind in patching, they don’t know where to start first, which
compounds the resource issue some say HPE is struggling with.
And sources say center CIOs and chief information security officers
have no say or control over their own networks so they can’t force HPE
to patch, and they can’t fix their networks with NASA employees.
“What largely concerns many people is HPE is still playing a role
because I don’t think they have the understanding that it’s that
important to deal with the cyber issues,” said the government official
with knowledge of NASA. “If you step back and look at the company’s
actions versus words, the inaction is what I guess frustrates so many
people today. We are having a contractual discussion about something
that should be inherent in providing IT services across the board. It’s a
money discussion, instead of being a mission discussion. That’s way out
of balance.”
NASA says HPE met 98 percent of security metrics
The senior NASA IT engineer said the number of out-of-date or missing
patches may reach a few million as the agency and HPE continue to
struggle to know exactly how many systems are in need of updates.
“We’ve seen an escalation over the last few years as more and more
agency executives are talking about cybersecurity. A lot of questions
are coming up, not just about patching, but security in general and the
answers aren’t good,” said the senior IT engineer. “Patching is the
biggest thing because it hasn’t gotten better and you have HPE saying we
know it’s in the contract, but we don’t have the resources, and they
are getting other priorities from headquarters. That is the real issue
that many have, HPE isn’t fulfilling requirements of contract and there
is no pressure on them to do so.”
The NASA spokeswoman said HPE’s performance has been solid since the start of ACES.
The NASA spokeswoman said HPE’s performance has been solid since the start of ACES.
“HPES has met 98 percent of the seven monthly ACES security-related
metrics,” she said. “NASA can retain a percentage of cost incurred for
contract performance measures not met.”
Lee Stone, the co-chairman of NASA Labor-Management Forum and Western
Federal Area vice president of the International Federation of
Professional and Technical Engineers (IFPTE), said in an email to
Federal News Radio that he’s seen progress in securing systems and data
over the last few months.
“Since the arrival of a new CIO [Renee Wynn] last summer, this
backlog has been significantly reduced and NASA is continuing to work
this issue. As far as the unresolved critical vulnerabilities, the new
CIO has been able reduce resolution time down below that required by
DHS,” Stone said. “Labor is committed to continuing to urge NASA
leadership to enhance NASA’s IT security posture and to ask Congress to
increase funding for this important priority. We are pleased with the
recent progress, but there is more that can and should be done at NASA
and elsewhere. Additionally, we remain concerned that the centralized
and outsourced structure of NASA’s IT enterprise continues to pose a
challenge, as it is not as nimble or adaptive as it should be in this
era of evolving threats.”
Stone said the union continues to push NASA Ames to bring key IT
oversight and security functions back in-house and tighten up control
over contractors in the future.
Sources said NASA needs a wake-up call, and China’s suspected hack of
the Landsat-7 satellite in 2008 wasn’t enough. NASA confirmed in 2011
that it experienced two suspicious events, but said no data was changed
or captured, and no commands were successfully sent to the satellite.
“I would’ve thought that would have made them go ‘holy crap,’ and be
more aggressive about doing stuff around cyber,” the NASA IT engineer
said.
Sen. Thune asked cyber questions
The NASA spokeswoman said Wynn, the agency’s new CIO, is restructuring and streamlining existing IT boards to ensure all IT investments meet federal cyber policies.
“Working with NASA’s chief financial officer, the OCIO will conduct a
formal annual capital investment review as part of the program planning
budgeting and evaluation process that will include all IT investments,”
the spokeswoman said. “The OCIO will work jointly with the agency’s
assistant administrator for procurement to formalize guidance on
strategic IT sourcing and strengthen and expand the NASA CIO’s role in
monitoring agency IT program performance. Finally, OCIO will conduct
functional reviews of all NASA centers on a three-year rotating basis.”
NASA cybersecurity challenges haven’t gone unnoticed on Capitol Hill.
NASA cybersecurity challenges haven’t gone unnoticed on Capitol Hill.
Sen. John Thune (R-S.D.), chairman of the Commerce, Science and Transportation Committee, wrote
to NASA on Feb. 10 asking seven questions about the state of the
agency’s cybersecurity efforts and an alleged intrusion into its
networks.
NASA responded on Feb. 18, saying it found no credible evidence of a
compromise of its systems or exfiltration of sensitive data. The
agency’s IT staff also met with committee staff at least twice over the
last month to further discuss cybersecurity challenges.
But until NASA re-takes control of the ACES contract and convinces
the mission areas that cybersecurity is as important as launching space
ships and probes, sources say the agency’s networks and data will
continue to be at risk and likely already are under the control of
hackers, nation states and/or others looking for intellectual property
or looking to do harm.
“When NASA gets close to understanding it and has cyber professionals
in systems engineering then they will get it,” said the former NASA
official. “They need cybersecurity awareness deep inside the mission.”
federalnewsradio.com