On 11 March, some of the Security Team met in Washington, D.C. for a day-long FAD where we discussed several issues. Zach Oglesby released his notes from the meeting and I’ll be using those to describe my take-away from the meeting.
Right now Fedora has a couple of problems with getting security fixes onto people’s systems quickly. The first has to do with embargoes. Because Fedora isn’t part of the trusted network, we don’t get advance notification of vulnerabilities before their embargo is lifted. This means that when we are notified of an urgent security vulnerability the public also has the information and we’re left scrambling to find the fix (patch, new version, etc) and ship it. Some other flavors of Linux will have had advance notice and will have these patches or new versions packaged up and ready to ship with the embargo is lifted.
The other problem deals with the Fedora Mirror network. Because there are many mirrors it could take many hours or days(?) to get a security update out to all Fedora users. This has to do with the mechanisms involved for keeping the Mirror system efficient; not necessarily fast. We discussed potential solutions for this problem as well.
Working with embargoes
Our solution to working with embargoes is to create a trusted team, with the appropriate tools, to deal with these issues so we can get a head start on urgent security vulnerabilities.
Right now the infrastructure isn’t in place to be able to handle embargoed vulnerabilities. Red Hat’s Bugzilla instance is currently designed to hold embargoed information, which is good and we definitely want to leverage that resource, but we need our build system and perhaps even Bodhi to be able to support private builds. The idea is to have the packages built and ready to ship before the embargo expires. When the embargo is lifted we would just need to push the big red button and the packages would ship.
It would also be hoped that the fix would have gone through QA before the end of the embargoed period so that we’re fairly sure that we’re not breaking anything (other than the vulnerability) in the process. For this part, we’re going to need trusted individuals to work on these issues.
Oh, and make no mistake, we aren’t trying to hide information forever. To maintain transparency that Fedora has been build on all of our tools we use to handle embargoed information should be able to make that information public at the end of the embargo. Bugzilla tickets would be opened up to the public and builds should be made available as well.
Trust is a difficult thing to define. How do you establish trust and how do you penalize a break in that trust? Simply put, we need a way to do both before we start handling sensitive information. We might be able to show that we have the systems and the procedures for handling this information but the first time there’s a leak and it comes from inside Fedora it’s quite likely we will lose access to this information and be back to where we are today.
We will be working with legal and Fedora management (FPL, FESCo, and Release Engineering) to devise a plan and determine the best way to involve package maintainers, proven packagers, and QA.
Faster access to Security Fixes
Waiting hours or days for urgent security fixes to become available on mirrors really isn’t acceptable. Recent critical vulnerabilities have seen exploits in the wild shortly after the vulnerability being made public. We need to be able to get fixes out to users faster.
Debian uses separate servers to deliver security fixes until those fixes have propagated out to their normal mirrors. This could work, especially with diff packages being used which are usually much smaller than the full package, but would increase the infrastructure needed to support disseminating packages.
There may also be a way to tweak our existing infrastructure to improve the time it takes to push these urgent packages out. Either way, the Security Team will be working with Release Engineering to help figure out a solution.
Another big topic we covered is training. Many people turn to the Security Team to learn. From a team point of view, we want to make sure our members have a common base of knowledge from which to work. We decided to launch Apprentice and Journey level certifications to will provide this base of knowledge.
We worked through what we would want to see in a new member and created the Security Team Apprenticeship. While not fully complete, we hope to have it ready in the coming months.
We also talked about mentors and what it means to be a mentor.
We didn’t put down many hard and fast rules and that’s okay. We want people to be active and participate. Hopefully the mentors won’t have a heavy lift and neither will the mentees.
Obligatory GPG Key Signing
Of course, a security FAD can’t end without the obligatory GPG key signing. This was completed with two new members of the team.
So what I’ve documented represents a day of work. Thanks to all that participated, even if you only had a chance to pop in, virtually, and lend your opinions or support. Hopefully we can do this again in a few months and work on new tasks.
In a couple of weeks (March 11th) the Fedora Security Team will be meeting in Washington, D.C. to hack on training, security fixes, and other issues. All Fedora contributors are welcome to stop by if you’re in the area.
All the information is available on the Security Team FAD 2016 wiki page. Please go there and RSVP!
At the beginning of April, the Fedora Security Team (FST) started on a journey to close all critical and important CVEs in Fedora and EPEL that had originated in 2014 and before. Now that we’re two-thirds the way through I figured it would be a good time to see what we’ve accomplished so far.
Of the 38 CVEs (37 important and 1 critical) we originally identified: 14 have been closed, 1 is currently on QA, and 23 remain open. The 14 closed CVEs represent around a third of all the identified CVEs. So, not bad but also not great; there is still work to be done.
If you want to help get some of these CVEs cleaned up here’s a list of the target packages. We need to make sure that upstream has fixed the problem and that the packagers are pushing these fixes into the repos.
I hope to come back to you at the end of the month with a report on how all of the CVEs were fixed and who helped fix them!
Earlier this month the Fedora Security Team started a 90-day challenge to close all critical and important CVEs in Fedora that came out in 2014 and before. These bugs include packages affected in both Fedora and EPEL repositories. Since we started the process we’ve made some good progress.
Of the thirty-eight Important CVE bugs, six have been closed, three are on QA, and the rest are open. The one critical bug, rubygems-activesupport in EPEL, still remains but maybe fixed as early as this week.
Want to help? Please join us in helping make Fedora (and EPEL) and safer place and pitch in to help close these security bugs.
The Fedora Security Team (FST) has uncovered an interesting problem. Many packages in Fedora aren’t being actively maintained meaning they are unofficially orphaned. This is likely not a problem since at least some of these packages will happily sit there and be well behaved. The ones we worry about are the ones that pick up CVEs along the way, warning of unscrupulous behaviour.
The FST has been plugging away at trying to help maintainers update their packages when security flaws are known to exist. So far we’ve almost hit the 250 bug level. Unfortunately we forced a policy that still isn’t perfect. What do you do with a package that is no longer is supported and has a known vulnerability in it? Unless you can recruit someone to adopt the package the only responsible choice you have is to retire the package and remove it from the repositories.
This, of course, leads to other problems, specifically that someone has that package installed and they know not that the package is no longer supported nor do they know it contains a security vulnerability. This morning, during the FST meeting, we discussed the problem a bit and I had an idea that I’ll share here in hopes of starting a discussion.
Create a file containing all the packages that have been retired from a repository and perhaps a short reason for why this package has been retired. Then have yum/dnf consume this information regularly and notify the user/admin when a package that is installed is added to this list. This allows the system admin to become aware of the unsupported nature of the package and allows them to make a decision as to whether or not to keep the package on the system.
Wow, I had no idea that people would care about the start of this project. There seems to be a few questions out there that I’d like to address here to clarify what we are doing and why.
OMG! Fedora is just getting a security team? Does this mean Fedora has been insecure this entire time?!?
Umm, no, it doesn’t mean that Fedora has been insecure this entire time. In all actuality Fedora is in pretty good shape overall. There is always room for improvement and so we’re organizing a team to help facilitate that improvement.
What exactly is the security team responsible for?
We here to help packagers get the patches or new releases that fix vulnerabilities into the Fedora repositories faster. Most of our packagers are very good at shipping fixes for bugs when upstream rolls a new version of their software. Bug fixes can usually wait a few days, though, as most aren’t critical. Security vulnerabilities are a bit different and fixes should be made available as soon as possible. A little helping hand is never a bad thing and that’s what we’re here to do… help.
Can the security team audit package x?
No. This may become a service a different team (also falling under the Security SIG) can provide but I/we haven’t gotten there yet.
I read where Fedora has 566 vulnerabilities! How can you say that Fedora isn’t insecure?
Well, it’s actually 573 right this second. That’s down from 577 last week. 566 was Monday’s number. It’s important to not get caught up in the numbers cause they are, well, just numbers. The numbers only deal specifically with the number of tickets open. Many of the tickets are duplicates in that the same vulnerability might have several tickets opened for it if the finding is in only certain Fedora versions and EPEL versions. Since the same packager is likely responsible for all versions and the same fix can be made we can likely close several bugs at a time with minimal work.
I should also point out that the majority of these bugs fall well below the “world is on fire” level of Critical and the “this isn’t good” level of Important. This doesn’t mean we should just ignore these lower vulnerabilities but rather we should understand that they aren’t something that is likely to be exploited without many other bad things happening. Should they be fixed? Yes, but we should probably be more concerned with the Critical and Important vulnerabilities first. If you’d like to know more about the process for coming up with the severity rating my friend Vincent wrote an excellent article that you should read.
“6. Close bug when vulnerability is shipped in Fedora repos.”
Yeah, that isn’t correct. This is what happens when I try to multi-task. Glad I don’t get paid to write…. err… never mind. Luckily it’s a wiki and someone fixed it for me. Whew!
(We try to not deliberately release a package with a vulnerability. It seems people don’t appreciate vulnerabilities in the same way they like other features. Who would have thought?)
I’d like to help! How can I join up?
Go to the Security Team wiki page and look for the link to the mailing list and IRC channels, sign up, join up, and use the work flow to start digging in. Questions? Feel free to ask in the IRC channel or on the mailing list. You can also contact me directly if can’t otherwise find the answer to your question.