Security Now 946, Transcript
Please be advised this transcript is AI-generated and may not be word for word. Time codes refer to the approximate times in the ad-supported version of the show.
00:00 - Leo Laporte (Host)
It's time for security now. Steve Gibson is here. He's got some really interesting stuff, including a new idea for Zero Trust Network Architectures. We'll also talk about more last pass hacks, sad to say, and then he's going to do one of my favorite things, which is look at a major security flaw this time with Citrix and actually examine the code that made it possible. This is a real great learning episode for anybody who has to write code or just wants to know how these break-ins happen. It's all coming up next on Security Now. Security Now with Steve Gibson. Episode 946, recorded Halloween. That's Tuesday, october 31st 2023, citrix Bleed. The show is brought to you by members like you. Thanks. It's time for Security Now the show. We cover the latest security news, your privacy, your security, your online health and welfare with this guy right here, mr Stephen Gibson. Grccom. Happy Halloween, steve.
01:15 - Steve Gibson (Host)
Happy Halloween to you. You know you are my namesake. You're the squirrel, I thought.
01:21 - Leo Laporte (Host)
I'd wear a squirrel outfit for you, yeah.
01:24 - Steve Gibson (Host)
Very appropriate. We have a. I have a squirrel right here in the, you know, in the camera on my whatever this thing is called that holds the microphone.
01:32 - Leo Laporte (Host)
Maybe it's a microphone holder, or boom and that's called a mic flag. Just in case anybody asks you have a squirrel mic flag.
01:45 - Steve Gibson (Host)
I proudly do, yes, the squirrel that took seven years of my life, and will not give it back.
01:53 - Leo Laporte (Host)
And now that I've shown you my squirrel head, please, children, do not fear. There is a human in here.
02:04 - Steve Gibson (Host)
There's Leo, okay.
02:06 - Leo Laporte (Host)
I won't take off the striped overall, so that's for another time.
02:08 - Steve Gibson (Host)
No, stay in your prison, garb, that's good.
02:11 - Leo Laporte (Host)
What, my friend, are we discussing today?
02:15 - Steve Gibson (Host)
Oh, have we got a good one. It is not often that we get a horrible problem in a well, actually it was last week, okay, but aside from that, we have another horrible problem in a globally distributed, highly used piece of network appliance. But what is so cool about this is that we're going to get to do a deep dive into the exact code mistake that was made in a way that all of our listeners are maybe not there significant others but all of our listeners will be will be able to almost certainly understand. I love it when you do that, by the way, and get something from it. Yeah, this is just the this one. It wasn't my plan. I was going to move forward with the NSA, sisa, top 10, you know misconfigurations for cybersecurity stuff but what I realized that a research firm had done the reverse engineering and that we'd be able to walk through that process and look at the code and it's understandable like the mistake that was made is just like oh, this is just too good. So security now.
03:39
Episode 946 for this Halloween, last day of October, and we're all going to be changing our clocks, those of us who do that on Sunday is titled Citrix Bleed, reminiscent of the heart bleed flaw in servers of the past. But we have a bunch of fun stuff to talk about we're going to look at. I'm going to answer the question what caused last week's connection interruptions that that we briefly suffered? Is it possible to create and maintain an internet white list? What's the latest on last pass vault decryptions? How do you know of a? Or how do you know when a remote correspondent adds a new device to their Apple account? If it's really them, then hold the people you think it is or not.
04:38
Might there be more life left in Windows 10 than we have been led to believe. What's foremost in the minds of today's bug bounty hunters, what new free and open source utility has CISA just released? Could it be that Spinrite 6.1 is finished? Is TLS 1.2 ready for retirement? And what about IPv4? How can open source projects get their code signed? And then, as I said, we also have a great picture of the week. But then we're going to take a really interesting deep dive into the internet's latest mass casualty disaster.
05:22 - Leo Laporte (Host)
Good Lord, this is on today's show.
05:26 - Steve Gibson (Host)
Yes, and that's in the first five minutes.
05:28 - Leo Laporte (Host)
Wow, you, uh, you've jam packed our Halloween episode with a lot of scary stuff, and that's, I guess, as it should be. That's our plan. Yes, I am ready. I have queued up the picture of the week. So I don't understand this at all.
05:49 - Steve Gibson (Host)
I gave this, I captioned this picture. What's the plan here? Because what we have is some apparently new asphalt that has been paved around a what looks like a phone pole and an electrical pole, which okay. So what I know from the person, from the listener of ours, who actually pulled off the road to take this picture for us and send it, because he thought, okay, you know we have to.
06:23
We have to send this to Steve. Picture of the week for sure. Yes, there is a. There's actually a new road that's been added off to the right. You, you can sort of see it. It uh toward the bottom of the picture, heading off off the picture to the right. So, uh, the old, the original road is, is, is, is, is a. You can see it's got a yellow line down the middle and it looks like it's the, like the. The black top on it has turned gray because it's been around for a while. So that's the original road. Well, they decided to add a left or, I'm sorry, a right turn lane onto the original road to complement turning to the right, into this new road that's been added to the. You know, off to the right, unfortunately, there was an existing two telephone poles, actually a telephone pole.
07:19 - Leo Laporte (Host)
In the distance, the same configuration, but it's on the grass, it's not in the road.
07:24 - Steve Gibson (Host)
Right, right. And this didn't used to be on the road, leo. This used to also be on the grass, but they decided to retract the grass and add some road. And I'm. But you know you can't drive through the phone poles, because that would not be, I would not turn out well, yes.
07:44
No. So now I was thinking about this. They clearly have to move the phone poles back over onto what will soon be grass after they recede this region, and moving them to the right would add slack to them, because we can see that you know it's going to be, it's not so. So that would kind of work. Although there are some lines running off to the left, they're going to have to disconnect those and then stretch them or length of them, and then you know. Anyway, I just thought this was interesting because really, how, how did this happen? How was it? I mean, people came along and they added new concrete curbing to the right of the poles.
08:30 - Leo Laporte (Host)
They expanded the shoulder, so exactly where the poles are.
08:35 - Steve Gibson (Host)
Pulled that back, got everything ready and put down new black top carefully around the poles so that it looks like they sprouted from the black top.
08:47 - Leo Laporte (Host)
I'm just grateful they have two bright orange security pylons there to prevent people from driving into those poles.
08:54 - Steve Gibson (Host)
That's right you would. You'd not want someone to think well, I better turn right here and get into the right turn lane in order to do that.
09:01 - Leo Laporte (Host)
Good Lord oh goodness, the number one reason our power goes out in this area is because people it happens at least every few months drive into a pole. They knock the pole over, power goes out, the electric company comes out and restrinks the wires.
09:19 - Steve Gibson (Host)
Those pesky poles. Now you realize, leo, that in more recent communities their power is underground, in fact ours in our cul-de-sac.
09:29 - Leo Laporte (Host)
Everything's underground and I love it, because there's no, you know well this is so unsightly. Why are there two different poles right next to each other too?
09:37 - Steve Gibson (Host)
Well, one is power. The tall one is power and the lower one is phone lines, because you can see that black junction box there going away from the lower one is the telephone cable. So, yes, for what it's worth, leo. We have underground power here so much more, and I can tell you that it is not always a lot better. Oh really. Oh yeah, the reason I had my own generator was that the power here has been notoriously bad Lately. I think they finally changed the thing. That was bad. I know it's a lot better. Anyway, I have two pieces of miscellany one quick one and then sort of something startling that I want to share. The first is why were we interrupted a couple times last week? After the connection interruptions which occurred during this, you know, last week's podcast, I got serious about tracking down the cause of what you know. I've sort of been seeing them for a while, but they weren't bad enough to get in the way of the podcast and I thought, okay, that's the last straw.
10:52 - Leo Laporte (Host)
Yeah, we added it out, but you froze on screen, yeah.
10:56 - Steve Gibson (Host)
And I could see because I'm monitoring my bandwidth on a separate screen over here. I could see a big red band twice come up where there was a lack of connectivity completely. So I decided to get focused on this. And you know the problem is normally I'm busy coding and when I'm focused I am a little OCD. I just sort of let everything else happen around me. I don't really pay attention.
11:22
Anyway, I was able to track down the cause of the trouble to occasional spontaneous crashes and reboots of my little NetGate SG-1100 router, which runs PF sense and sits just in board of my cable modem. So you know it enforces my network segmentation, creates separate Wi-Fi domains and does a bunch more things that I've talked about in the past. You know static port mapping and all kinds of cool stuff. I love PF sense. It's there. There's never been any, oh, even now it runs. It is a Dyn DNS client and its firewall rules are adaptable, that is, the firewall rules will track any changes in reflected by dynamic DNS. So it allows my two locations that are on, that have public IPs, to track each other and for the sake of keeping them connected. So, for example, my Synologies are able to talk to each other, even though there's no public presence of the Synology sync going on. Anyway, it's very cool.
12:34
So what I wanted to mention was the first thing to check when something like this is happening is the devices of power supply, those little wall warts, as they're often called, that ship with consumer grade equipment are often the source of trouble. If the power supply is getting tired it could be the reason for weird behavior. So fortunately that little NetGate SG-1100 uses a standard 12 volt DC supply that has the standard DC barrel connector. So I had plenty of swap in replacements at hand. Ever since I exchanged its power supply, which was two days ago, I've had zero reboots. You know I still had my fingers crossed, but since it was previously happening, sometimes several times a day, it really looks like the problem is solved.
13:35
So anyway, just a little note, a reminder that oftentimes the cheapest piece of equipment in the system, which is, you know, those little cheesy transformers these days, the little power supplies they can have like much lower quality guts than the device they're powering and they can bring it to its knees. So just a reminder about that. Okay now, leo, you'll remember another of this podcast's very early sponsors, the Nerds On-Site guys who are based in Canada, with their nerds spread far and wide to help non-computer savvy users and enterprises with their computer problems.
14:27 - Leo Laporte (Host)
I'll always have a soft spot in my heart for them.
14:30 - Steve Gibson (Host)
I do too. You know they were early fans of the podcast and in fact, I keynoted yeah, yes, dave Reddicop.
14:38
I keynoted their annual gathering a couple times, oh nice. And in fact I also met with them for the first time when I was in Toronto when I was serving as a guest on your call for help show what you were doing for Roger's Cable up in Toronto. So we're talking 18 years ago in those very early audio-only days of the podcast when we used to record the show in your hotel room. So I remember you had a little recorder and I would bring the microphones and we'd set up and try for like you tried once on the rooftop restaurant of the hotel, but that didn't work out so well, so we were treated to the room.
15:21
Okay, anyway, since then I've stayed loosely in touch with David Reddicop, who was one of the principals back then, and I've sort of been vaguely aware of what he's been doing. Well, he was passing through town last Friday, so we rendezvoused at Starbucks to catch up, he and two of his guys. Okay, now, this was over my second five-shot latte of the morning, so my brain was fully charged up. David took about an hour to walk me, step by step, through the careful and deliberate design of the network security solution he and his team of techies at Adam Networks have designed and which is currently in use, actively protecting around two million user and device endpoints around the world. So this solution is not just theory and, I think, a surprisingly good idea. It's actually in place and working. And once I'd heard what this group has managed to pull off, that didn't surprise me, because it felt like the result of a decade or more of iterating on a concept in order to bring it to maturity. You start with an idea, you implement a first pass and then stick it out there. You remain involved and connected, you learn what does and doesn't work, then you iterate and push out an improved solution. You add new features to address new requirements. You test them and adjust them until they're working correctly.
17:07
Now I've often noted on this podcast that I cannot imagine the challenge of keeping an enterprise safe while it's connected to today's internet. And in fact, today's Citrix bleed topic makes me shudder a bit because even if everything is done right, it's still possible to be taken down. I'm so glad that keeping a highly connected, sprawling enterprise safe is not my job, but it does still need to be done. And these Adam Network guys have not only imagined the challenge of keeping an enterprise safe from what was explained to me on Friday and I think I pretty much understand the way it works, at least in broad strokes. They not only imagined the challenge, they've risen to it. And if I had an enterprise network which I needed to protect today, based upon what I understand now, I don't think I'd want to be without this solution. I'm thinking of reaching out to Father Robert to make him aware of this, because it would be so cool to have the Vatican this well protected.
18:15
Okay, so what did they do? Nobody today runs their firewall where, by default, everything is allowed except for traffic that's known to be malicious In the dim, dark ages of the internet. Once upon a time we blocked known exploitable ports. We don't do that anymore. But blocking known bad domains or scanning communications traffic for known malicious content is what's being done today, and it's the modern equivalent of a default allow all rule at the bottom of a list of things that you want to block. We do it because we think there's no alternative.
19:06
Last Friday, the Adam Networks guys showed me a better way. What these guys have done is to not worry about inspecting and interpreting the contents of enterprise user communications or anyone's communications. As I understand it, they do not do any deep packet inspection, so no one's communications contents privacy is ever compromised. What they focused on and specialized in and made work is only allowing connections to known, safe and benign entities. Now, okay, if it were possible to do this perfectly, phishing attacks would be ineffective, because a user clicking on a phishing link would be unable to obtain any malicious content. And even if someone were to inadvertently bring some malware into the enterprise that infected it from the inside, if that malware was then unable to connect to its command and control servers, it would be effectively inert. So imagine that you start with very strong DNS filtering and require all clients on the network to use this DNS. They've also extended this through and enforced it for enterprise controlled smartphone and other mobile clients, so you get complete coverage.
20:34
The thing that sets this apart from other solutions is that this is not the typical black listing DNS which selectively blocks known bad domains. Atom Networks has managed to build a practical white listing DNS which, by default, blocks everything unknown and only permits DNS resolution for known, safe domains. Dns is resolved on premises, so privacy is preserved, but it's also informed by a shared central manager in the cloud so that as new domains come online and are cleared, the master white list can be updated. If a user attempts to connect to an unlisted DNS domain, they receive an intercept page, and the way this is done is quite clever. I won't go into it, but it works, and they must manually accept and approve that if they want to proceed. But before that is allowed, 18 other public sources of DNS filtering from other providers are consulted to make sure the unknown domain is not known to be malicious by anyone, and there's also technology in the pop-up intercept page which prevents malware from being able to simulate the human giving their permission.
21:59
And speaking of malware, it turns out that modern malware is hip to having its DNS blocked. So there is malware out there that has static IP addresses burned in. They don't do any DNS, it just connects directly to its command and control. So this introduces the outbound firewall component of this solution, which they call Don't Talk to Strangers. No outbound connection is allowed to any IP that wasn't first looked up with the system's local DNS resolver. So by linking DNS resolution to a dynamic outbound firewall, clever yes DNS, which has already been made very strong, becomes the universal gatekeeper. Okay, but then it turns out that Facebook, for example, has some apps that also don't do DNS. Facebook's apps are benign and they know the fixed IP addresses they want to connect back to at the mothership. So to make a solution work completely which they have it's necessary for the system to incorporate some white listing for known benign blocks of destination IPs. So that's in there too, okay.
23:29
Now, what I've just described is an overview which just scratches the surface of the enterprise security solution these guys have developed, and I'm sure that our techie listeners are saying but, but, but, but, you know, to which all I can say is that they actually have this thing working. They demonstrated it to me and, as I said, more than 2 million user and device endpoints are currently under this solutions protection. They have a full, mature looking management UI with scheduling and per user permissions and everything else needed to implement a practical enterprise class solution. I asked David how much trouble there was with unknown yet benign DNS and he acknowledged that it was there but that they had reduced the level to such a degree that it was effectively negligible. I recall that he said 5%, but I don't remember what 5% was of. So you know my reaction was that if I were an IT guy and wrote like with this kind of responsibility and it was possible and practical to operate behind what is essentially a white listing internet firewall, if that only caused a tiny bit of trouble in fringe cases which can then be worked around, that would be entirely acceptable for my peace of mind and in return for the enhanced security it would provide for everyone else all of the time. And what's cool is that it protects against the unknown. You know that it turns out that the widespread solar winds vulnerability which took the whole world by surprise, would not have affected any of these people's, any of Adam Networks users, and the always evolving Pegasus smartphone spyware does not function if it were to infect any of the handsets protected by this system.
25:41
So, as usual, my focus and interest was on the technology, which I think is clearly pretty cool and which I felt I you know an obligation to share because this could help a lot of our listeners. Although I cannot vouch for this from firsthand experience because I have none, you know, we do know these guys and we've known them for nearly 20 years. They're techies like us. Just take a look at any of the many videos posted on their website and how excited they are about what they've created and you'll see that immediately. I was very impressed by what I saw last Friday.
26:22
I don't know anything about the business side of this. You know, like what it costs or what plans they have or any of that. It just didn't. We didn't talk about that. I just we were talking about the tech. My sense is that the solution is scalable down to a single router in a small office all the way up to a global enterprise, because there was. He did make some mention of like the massive amounts of traffic that they're able to handle, but I don't know what the what with the extreme points are for that. As I said, I also have no idea what it costs. You'll need to track that down for yourselves if it sounds interesting.
26:58
So anyway, after what I heard and learned Friday, I wanted to put this on everyone's radar. They are Adam Networks. They are it's a, d, a, m, n, e, t dot works, and I have some links in the show notes for their site, although I got the links just by poking around a bit the other day, so you know you'll find a page of videos, even their wide, open tech support portal, so you can see everything that their customers are asking and like being confused by or needing help with. So, anyway, I just wanted to share this because I was. I was really impressed that they could make this work. And boy you know, having a default, deny, allow, trusted system that could protect some in the size of an enterprise, to me that's a game changer. So I will leave it up to our listeners to decide if it looks like something for them.
28:03
Next.
28:03
Last week, the still unknown hackers who breached last pass and exfiltrated everyone's encrypted vaults managed to siphon off another $4.4 million worth of crypto assets from 25 of last passes, previous and maybe still current customers.
28:28
Those who haven't, you know, run for the hills. We know that most last pass users will have remained with last pass and won't be receiving the news that stolen vaults are being actively decrypted. As we saw previously, the hackers are cracking the stolen last pass password vaults, presumably those that were protected by weak passwords and probably also few PBKDF iterations, though I would argue by no fault of the user. This is that was the squarely on last pass for not bringing users up to speed, that is, you know, their, their password iteration counts to protect against having weak passwords, which we know users are going to have. Anyway, from those vaults that they decrypted, they recovered the crypto wallet seed phrases and then drained those customers users account. So, with this latest round of thefts, which are being tracked because we know the wallets into which those stolen funds are being transferred, this brings the total to nearly $40 million, after they had previously stolen around 35 from about 150 other users earlier this year. So it's ongoing and it's really sad and unfortunate.
29:56 - Leo Laporte (Host)
And, if you have, if you had a last pass vault with your keys in it, you might want to change your keys. Yes, yeah, yes.
30:05 - Steve Gibson (Host)
And in fact remember that we we read a note from one of our listeners who had set up another wallet and was planning to transfer his currency to it, but didn't get around to it.
30:20 - Leo Laporte (Host)
Yikes.
30:20 - Steve Gibson (Host)
And lost three something million dollars, I think it was.
30:24 - Leo Laporte (Host)
If you don't have around to it, you better get around to it.
30:28 - Steve Gibson (Host)
That's right. Remember that cool feature of three my that I always liked so much? It was the ability to positively verify the other person's public key through some out of band communication channel. You know you can't verify it in band because if you're not talking to the person you think you are, then you're still not talking to the person you think you are after you exchange key verification somehow. So it's got to be like some other. You know non in band. You know communication app channel. You know like a phone call or, even better yet, a face to face meeting.
31:14
Okay, so Apple hasn't gone quite that far, since they continue to manage all of their users keys on their behalf, which I'm sure is a good thing in the case of your typical Apple user. But they have developed a slick new way to verify any new device that registers itself with a user's account. They call it I message contact key verification, and it has just appeared in the developer previews of iOS 17.2, mac OS 14.2 and watch OS 10.2. It's designed to present an alert inside I message conversations whenever a new device is added to a participant's account. Okay, so say that I'm in a in an I message dialogue with someone and they add a new device to their account and, you know, talk to me from it. How do I know that the new device was actually added by the person I think I'm messaging with? You know if it was someone else holding the new device, okay, impersonating this person, that's not good. So this new system displays a fixed eight-digit code in both my phone and their phone, that is the newly added phone. This allows us to arrange a telephone conversation or a FaceTime chat or something, not just iMessage, where I have some absolute knowledge that the person I think I'm messaging with is this other person.
33:11
We compare eight-digit codes. That is what I would do is I would tell the person you know holding the new device, okay, what eight-digit code is your phone showing you? And you know, I know who I'm talking to, right? I mean so because it's a telephone conversation or a FaceTime call. They read the eight-digit code. I confirm that that's the code my phone is sharing with me and then I click on mark as verified, which then verifies that this new, that it tells iMessage that I'm accepting my remote correspondence new device for subsequent iMessage communications.
33:59
So anyway, the Apple's on-screen instructions say verify who you are messaging with by comparing contact verification codes in person or over the phone and at first I thought, wait, you know, shouldn't I make them tell me? But and I guess that's the case right, except that you know, if it's basically what this does, is it forces you to have a non-iMessage, some non-iMessage contact with the person whose phone is also showing this code. So you know that that's the device which you don't yet want to fully trust. So I'm not sure if it matters If I tell them the code or they tell me, or we alternate digits back and forth, or they tell me the first half and I tell them the second half. I don't, I'm not quite sure.
35:06 - Leo Laporte (Host)
We leave this as an exercise. You figure it out.
35:11 - Steve Gibson (Host)
But it does. It does force you to have a face-to-face confirmation. And then you say okay, fine, You're holding the phone. You told me the code. That's good, we're gonna. I'm not gonna accept this as belonging to you, so, yeah, Anyway, cool that Apple continues to do this. You know they're not perfect, they make mistakes, I get that. But, boy, you know, every contact we have with the measures that they are taking seem really proactive and I, you know like they're. They're, they've got people sitting around just thinking of you know how to make our stuff more secure.
35:56 - Leo Laporte (Host)
This solves the problem of a new somebody rekeying or getting a new phone in this case, and Signal has done it, as you said, threema does it. How do you verify that this new communication with a new key is the same as the old person? And it just shows that Apple's now doing strong encryption on messaging right, because right, right, well, strong, authentic, stronger authentication. They all. Actually doesn't say even encryption does it? It says that's authentication yeah.
36:26 - Steve Gibson (Host)
And we know that they always had really good encryption. On the other hand, if you're not sure who's decrypting it at the other end, that's not good either.
36:34 - Leo Laporte (Host)
So authentication is a big part of this, yeah.
36:38 - Steve Gibson (Host)
Right, exactly, of course. That's that's the whole certificate thing, right? I mean, anybody can have a certificate that brings up an encrypted connection. What you want is to know who has that certificate and for their identity to have been proven, right? So, and that's that's what let's encrypt does not try to do. It says it's only saying the domain that asked for this certificate was granted one. So now nobody can see what's going on in in in private, but it's not making any assertions. I mean, it's not making any assertions about you know who it was that it issued the certificate to. That's why, you know, I'm still using a fancy certificate from DigiCert, where I have to jump through hoops every so often in order to say, yep, still me, hi mom.
37:30 - Leo Laporte (Host)
Yeah.
37:31 - Steve Gibson (Host)
Yeah, okay, so perhaps there's hope for longer Windows 10 support. Remember, I just talked about last week how, like the in the week before, we had just crossed the less than two years the remaining for Windows 10 line, which to my mind seems really premature. Okay, it turns out that Windows 10, leo, is good for the earth, or, more correctly, it's that Windows 11 is less so. More than 20,000 members of the public interest research group PIRG have formally asked Microsoft to extend support for Windows 10, you know, which, as I recently mourned, is scheduled to reach end of life status in two years, in October of 2025.
38:33 - Leo Laporte (Host)
It has been 10 years or will have been, I mean, but it still feels new, it still feels new to me. I'm still in front of Windows 7. That's right.
38:45 - Steve Gibson (Host)
And my browsers are not happy with me. But okay, the organization, this PIRC group, says that. Get this around 40% of the one billion devices that run Windows 10 cannot be upgraded to Windows 11. Wow, and that will create an unnecessary e-waste problem 400 million, to be precise. Of course, on the flip side, microsoft and many other hardware vendors are looking forward to this event, for just for just that very reason. Pirc is the same group, the same industry group, that had recently convinced Google to extend the warranty and support for older Chromebooks, using similar arguments. It's like don't put these out to pasture, it's going to create a bunch of toxic landfill. And what's wrong with them? They still work, they're still just fine, just like Windows 10, microsoft, so it's certainly somewhat.
40:01 - Leo Laporte (Host)
It's just good for Linux. I mean honestly, just put Linux on the damn things and you'll never have to worry about Microsoft again.
40:12 - Steve Gibson (Host)
Right, you will have to pay people to teach you how to use it.
40:14 - Leo Laporte (Host)
However, no, you don't. It's the same as Windows Easy peasy.
40:19 - Steve Gibson (Host)
So it is certainly somewhat bracing to learn that 40% of current machines, of a billion current machines, won't be compatible with Windows 11. I mean, we've been talking about this from the beginning and we know that it's just because they don't want it to be Right.
40:35 - Leo Laporte (Host)
Right, they have to have the eighth generation Intel processor. They have to have TPM 2.0, even though that that's not necessary for the operation on Windows 11. Right, right.
40:45 - Steve Gibson (Host)
But honestly, Linux works great on that machine.
40:48 - Leo Laporte (Host)
I'm just telling you.
40:51 - Steve Gibson (Host)
It is, yeah, the problem is the users don't. It's legitimately a huge deal. So I really do wonder whether Microsoft, you know, is prepared for the user and the corporate ire that will follow from, you know, being told Sorry, they've been warned.
41:09 - Leo Laporte (Host)
It happens to every version of Windows and 10 years is, as you pointed out, much longer than the Chrome book. It's much longer than any phone you've got. Google just got a lot of attention for saying we're going to support this phone for seven years. Yeah, so I don't think, mike. I mean, look look at your phone. Most people get new phones every few years. I mean, the landfills are jammed with stuff. What about your new car? How long is your car last? And that's a lot of physical waste. What many, what many computers worth? So, leo, we got to do something about this you can't put Linux in.
41:47
Why?
41:47 - Steve Gibson (Host)
force the hardware to be obsolete for Windows 11, which does not require we know it doesn't require that the hardware no.
41:57 - Leo Laporte (Host)
I agree. I agree they shouldn't say eighth generation and TPM too, but increasingly they've added features that do support, maybe even require, that newer hardware. But I think you nailed it when it's really designed to support the PC industry and people buying new PCs, yeah, I mean, that's clear. But but the whole, our whole industry is geared around getting people throughout their old stuff and buying new stuff.
42:23 - Steve Gibson (Host)
So last Thursday, the Hacker One bug bounty folks oh yeah, took the occasion. Yeah, we saw this. Yeah, really good, so neat. People took the occasion of their total bounty payouts crossing the $300 million mark yeah, to provide some insights into the current state of their operations. I've edited down a bit, but there's some interesting stuff in here, so just to be clear in the following text when they refer to their customers, they mean organizations who pay bounties to have their bugs found in their own code and, of course, when they refer to hackers, those are the freelance agents who are the finder of those bugs. So Hacker One said Hacker One today announced its ethical hacker community has surpassed 300 million in total all time rewards on the Hacker One platform and 30 hackers have also earned more than $1 million on the platform, with one hacker surpassing $4 million in total earnings. So, yeah, it's possible to have a career and hackers are finding new opportunities to earn more by diversifying their skill sets as emerging technology reshapes the threat landscape, 55% of hackers surveyed and they surveyed 2,000 of their hackers expect that generative AI will become a top target in the future. Crypto and blockchain organizations continue to see strong program engagement, offering the highest average overall rewards for hackers, including the award of this year's top payout of $100,050. Hacker One's customers have also expanded how they commission hackers outside of traditional bug bounties, with pen testing engagements increasing by 54% on the platform this year in 2023. So they have some bullet points. Hackers continue to experiment with generative AI, as 61% of hackers said. They will use and develop hacking tools from JNAI to find more vulnerabilities, and another 62% of hackers plan to specialize in the OWASP top 10 for large language models. Hackers also said they plan to use JNAI to write better reports 66% said that or code 53% said that and reduce language barriers. That was a third of the people.
45:05
Hackers report insufficient in-house talent, meaning on part of hacker one's customers. Hackers report insufficient in-house talent and expertise as the top challenge for organizations, which is a gap they're filling. 70% of customers stated that hacker efforts have helped them avoid a significant cyber event. 57% of Hacker One customers believe exploited vulnerabilities are the greatest threat to their organizations, over phishing at 22%, insider threats at 12% and nation state actors at 10%. Customers are getting faster at fixing vulnerabilities, as the average platform-wide remediation time dropped by 10 days in 2022.
45:57
I didn't see the report because I had to go through a bunch of hoops to get it and it didn't seem necessary. But I would like to know dropped from what to what? By 10 days? Did it drop from 20 days to 10, in which case it was in half, or from 100 to 90? To me it makes a difference, but anyway, their summary didn't say they said interestingly, automotive, media, entertainment and government verticals saw the biggest decrease in time to remediation, with an over 50% improvement. So those industry groups automotive, media, entertainment and government are, in this most recent update, much faster to jump on trouble and fix them. And finally, organizations are reducing costs by embracing human-centered security testing earlier in their software development life cycles, with customers saving an estimated $18,000 from security experts reviewing their code before it's released rather than afterwards.
47:07
So Chris Evans, hacker One's CISO and chief hacking officer, was quoted saying quote organizations are under pressure to adopt generative AI to stay ahead of competitors, which in turn, is transforming the threat landscape. If you want to remain proactive about new threats, you need to learn from the experts in the trenches, Hackers. So anyway, that annual hacker powered security report is based on data from hacker ones vulnerability database and gathered the views from both hacker one customers and more than 2000 hackers who are interviewed on the platform, and to me it seems that I guess it's not surprising, but maybe only in how quick it's happened. An interesting takeaway that mistakes are bound to be made in the rush to get new generative AI solutions rolled out, and you know in the world, which did seem to happen pretty quickly. So getting up to speed on AI technology appears to be on everyone's radar. I know, no doubt the bad guys and also the white hat hacker, good guys who are going to help companies to solve their problems before they get exploited.
48:30
Logging made easy from your friends at CISA. Last Friday, our own CISA released a tool called logging made easy through their official GitHub account. Lme is a toolkit that enhances log management on Windows devices. The tool was originally developed by the UK's NCSC agency way back in the late 2010s and they retired it in March of this year. So CISA picked the tool up, updated and rewrote it to cover recent Windows versions and to enhance its logging capabilities.
49:12
Here's what CISA said. They said today, cisa announces the launch of a new version of logging made easy LME, a straightforward log management solution for Windows based devices that can be downloaded and self-installed for free. Cisa's version reimagines technology developed by the United Kingdom's National Cyber Security Center, the NCSC, making it available to a wider audience. Log management makes systems more secure. Until now it's been a heavy lift for many targeted organizations, especially those with limited resources. Cisa's LME is a turnkey solution for public and private organizations seeking to strengthen their cybersecurity while reducing their log management burden. Lme builds upon the success of the NCSC's log management solution and CISA urges organizations to secure their Windows based devices today by downloading the free LME technical solution Over on their GitHub page.
50:24
They said that logging, made easy, can show where administrative commands are being run on enrolled devices, see who is using which machine. In conjunction with threat reports, it's possible to query for the presence of an attacker in the form of tactics, techniques and procedures. So anyway, all that seems pretty cool and useful. The link to jump there is this week's GRC shortcut, so GRCSC946. But the full link is also not very difficult. Just go to GitHubcom. That will take you to a place where you can find out more and grab it for yourself. Leo, I have here in the show notes under the heading of spinrite I love this the issues page from our GitHub as of Sunday, and I checked a few minutes ago. It is still holding. It says there are no open issues and then further up on the page it says you know, steve Gibson, I'm the project author, spinrite version 6.1. Issues Open zero, closed 536. That's impressive.
52:13 - Leo Laporte (Host)
That is really impressive.
52:16 - Steve Gibson (Host)
And of course, those are all the things that we stripped over and stumbled upon Some bugs in spinrite, many bizarro behavior in weird things that caused me to go to eBay and buy some old motherboard and for Lori to say we're going to put these away someday, right? Yes, dear.
52:37 - Leo Laporte (Host)
Are they on the dining room table?
52:40 - Steve Gibson (Host)
I mean they're out in the open but you run out of room in your office. All and sundry to see. Oh Lord, poor Lori. Hey she knows what she was getting into.
52:54
On Sunday? Yes, she does, and she, you know she's still supporting my work. So on Sunday, spinwrights, a second and quite possibly final pre-release was published. As far as I know at this point, the DOS executable code that's now available will be what finally ships. Woohoo. Spinrite 6.1 includes a tiny DOS text editor for viewing and editing its log files from DOS and to make editing any DOS config files easier. That editor also appears to be completely ready.
53:34
I need to verify that all of the free DOS OS kernel customizations that I have made for Spinrite are complete. You know I needed to tweak the free DOS kernel so that it doesn't completely go crazy if it sees a GPT partition, because all it knows about is MBR format, and also so that it doesn't freak out if there's errors on the disk and that it's trying to log into in order to create DOS style drive letters for all of the drives, the way we used to in the old days. So I've done all that. I think it's done. I'm just going to go back and check because it's been a while. Then I will return to finish a bit of the work on the Windows side of Spinrite, which is the thing that creates the boot media for Spinrite and basically I'm going to incorporate the improvements in that technology that arose from the work that I did on Validrive, so I'll port those back over into the Windows side of Spinrite. I need to update GRC servers to digitally sign individually licensed copies of Spinrite on the fly for customer download and at that point Spinrite 6.1 will be available as a finished product to all Spinrite 6 owners. And then I'm going to begin work on Spinrite's web pages to document all of its new operations and screens, to show all of the many changes and improvements that have been made during the past three years of this project. And then, after I get an email system in place to begin notifying Spinrite 6.0's 19 years worth of existing users, I get to start in on Spinrite 7, which really excites me.
55:29
But in the meantime we really do have a major step forward here. This morning I saw a posting in GRC's web forums which said thanks to the pre-release of 6.1, I'm running Spinrite on drives that I either have not been able to run it on for years or drives that were just too big for 6.0. So I was never able to run Spinrite on them before. So promise made and promise kept, and quite soon, promise delivered. So yeah, I'm very happy.
56:09
So we have a bit of closing the loop feedback. Shep Poor said hi, steve, you've trained us well. When I saw a bleeping computer article titled Windows 11, add support for 11 file archive formats, my first thought was oh great, 11 new interpreters built into Windows, ready to be exploited. Yes, shep, I appreciated his tweet because thinking security is probably the best thing our listeners could take away from the many lessons we learn here every week from specific events. Those events will come and go, but, seeing the underlying issues and what connects them together, what are the takeaways and the lessons those will endure and remain viable and valuable long after the UNIX epoch has shut down everything that we know and love.
57:11 - Leo Laporte (Host)
It's true, it's in my head. I think the same thing. Oh, another interpreter, and it's because of you. I mean, I totally think of these security flaws, because you've trained us all these years, because we've looked at so many and boy are we going to look at a doozy here in a minute, someone named BP whose handle is at Wildlands guy.
57:34 - Steve Gibson (Host)
He said hi, mr Gibson, I had a huge smile on my face when you announced that you'll be doing SN Pass 999. I would appreciate hearing your opinion on what it would take to get the world to move to IPv6. What are the significant hurdles? And he said FYI, ran a valid drive on three P and Y drives from Best Buy and all checked out. Okay, so cool, okay. So there are probably still some backwaters of the internet which IPv6 has not reached. But from everything I'm seeing, nearly everything appears to be IPv6 ready. So I think that at this point the only thing that's holding things back is a concern over breakage. What might break if IPv4 was withdrawn and we were left only with IPv6?
58:37
We've seen examples where surprising things have broken with change. We were just talking about the surprise that the designers of the upgraded TLS 1.3 protocol faced when it initially didn't work because too many boxes somewhere along the way in the traffic path broke it. They shouldn't have broken it, but they did. So it was necessary for the TLS 1.3 guys to make 1.3 appear to be much more like 1.2. Again, shouldn't have had to do it, but they did, and we've seen how incredibly cautious the browser designers are whenever they change anything. They'll initially have some new fangled feature disabled and only enabled manually by intrepid developers. Then they'll turn it on by default for 1% of the population, while closely monitoring that 1% for any trouble. Then they'll gradually start rolling it out, enabling it more and more still, kind of holding their breath and making sure that nothing really horrible happens. So they've learned to be super cautious from experience. So I think that's where we are In the case of IPv6, I think we're at the if it's not broke, don't fix it stage.
01:00:14
The good news is that everyone is probably ready for it when it's forced upon us. But until it's truly necessary for someone not to have IPv4, until there just isn't any IPv4 left and the only thing that newcomers can have is an IPv6 address, that's when I suspect IPv6 will finally begin in earnest, and really no sooner until there's really no other choice, because that's just the way all this stuff is. It's like you know leave it alone unless and until we have to finally give it up. Someone tweeting at not Dorian said Hi, steve. I've recently seen the high cost of signing certificates, especially for open source projects like ImageMagic, and he provides a link to ImageMagic's pages over on GitHub. He said On one hand, I see that having costly certificates makes it difficult for malware to access them, but maybe a solution is needed for open source projects. Would be glad to hear your thoughts on this. Thanks for all the great content all these years, so that was interesting.
01:01:46
The posting on GitHub that our listener linked to was titled the Windows Installer for ImageMagic will no longer be signed, and his developer wrote Today, our code signing certificate will expire. For many years, leaderssl sponsored us with a code signing certificate, but they're no longer able to do so. Since June of 2023, the CAB forum requires that OV code signing private keys be stored on a FIPS 140-2 Level 2 or common criteria level EAL4 plus certified device, meaning some sort of HSL, some hardware security layer like we've talked about before inside a truly secure enclave device. He writes. This means we are no longer able to export our code signing certificate with its private key and use this in GitHub actions. We would now either need to have our own GitHub agent and hardware token or use a cloud solution. He says EG DigiCert. Our preference would be to use a cloud solution that integrates with GitHub. Digicert seems to be our only option now, but a certificate there would cost $629, tax excluded, for a single year. If your organization requires a signed installer, then please consider sponsoring us with a code signing certificate. Please reach out to At and then his handle for questions or in case of a sponsorship. Now, what was neat was that, in a terrific example of community action, this posting from three days ago generated a terrific thread which, based upon this developer's reaction, appeared to provide a number of useful alternative solutions, one being a free solution for a year and others involving Azure code signing, which appears to be a good thing. I didn't look into it any deeper, but other open source projects are having similar difficulties and they chimed into the thread as well, so I wanted to share the news of this in the event that the eventual solution found by ImageMagix developer might be of some use to some of our listeners.
01:04:32
As we know, all of GRC's code is signed by ImageMagix. I mean, all of GRC's code is signed by a Digisert EV certificate, which works for me, since I'm a commercial entity, and it's worthwhile and I was just a couple of weeks ago talking about. You know how it matters to have your code signed. You're looking at downloading something. You want to make sure that that XE, you know, hasn't been modified since the time it was signed by the entity that claims to do so and that requires a signature.
01:05:10
The problem, of course, is that these signatures also the code signing, the private key in these code signing certificates, must be kept secure because you don't want bad guys signing stuff claiming to be ImageMagix and getting the trust that ImageMagix has been able to accrue over the years. So I mean, the problem is the way we're going. The nature of open-source software historically is kind of coming into collision with the fact that we really want to know. We want to know who's behind it. Matt Davis tweeted offering some great oh. He tweeted and offered some great feedback about last week's look at privilege. He said hi, steve, thrilled to hear you're continuing past 999. I've listened since the beginning and I'm looking forward to another 999 or so afterwards.
01:06:16
Okay, well let's hope he said you may have overlooked the heart of the privileged accounts issue you discussed during last week's episode. The problem is not lazy IT folks signing in as root in the morning. It's that we're trying to create a shift away from one-to-one parity between accounts and people. Many moons ago, he wrote, shared logins were the norm for system management, which has obvious downsides and drawbacks we all understand. Single sign-on solutions like Okta aimed to solve this. By promoting unique accounts per user, each person gets exactly enough privilege for their needed tasks and staff turnover is easy to handle by revoking a single password.
01:07:14
However, this led to a new problem. Administrators only have a single account which they have no choice but to use, even when they're performing basic tasks, and it's all powerful. The obvious solution would be to create separate standard and admin accounts for every administrator, but since most software is now charged per account per month, this tends to be cost prohibitive, and he said we have one service at $300 per month per user Yikes. He said ideas like just enough access help, allowing admins to switch between regular and elevated access akin to UAC in specific environments like Azure and AWS, but slow adoption in user land means that I still need to use my SuperUser level login to access my email every morning, looking forward to having the option to submit via email. It seems like the feedback form hasn't really been working over the past few years and you're the only reason I've pulled my Twitter account out of the dustbin. Love the show and spinrite, thank you. So anyway, matt, thank you for some terrific perspective from the field and yikes. Charging an organization $300 per user per month for authentication makes my blood boil a little bit. What a ripoff. I can't see what great deal of work they're having to do that justifies that much expense, okay. Finally, jpmcneel said hi, steve, I'm a huge fan of SN and so glad you'll keep going after episode 999,.
01:09:08
I'm wondering what your thoughts are on turning off TLS 1.2. I did a little research and found it interesting that there's no official EOL. You know end of life for TLS versions. It's just a matter of when the big browser vendors decide to not support it. Microsoft, apple and Google agreed to stop supporting TLS 1.0 in 2018, for example.
01:09:39
Thank you, signed JP. And he said PS, I work for a hosting company, so yeah, he's got a vested interest in like when will 1.2 go away? So I suppose that time flies, but my impression is that TLS is still a relatively recent arrival and that 1.3 hasn't fully happened yet, and I guess you know, I think that about Windows 10 also. So what do I know? Tls 1.0 and 1.1 were both deprecated only two years ago, in 2021. Tls 1.2 has been in use since 2008 and TLS 1.3 only began 10 years later and five years ago in 2018. But here's the stat that most matters as of last month, 99.9% of all websites support TLS 1.2, but only 64.8% support 1.3. So not even two-thirds of the internet's websites today are TLS 1.3 capable, and, given a good choice of cipher suites for TLS 1.2, there's really nothing wrong with it.
01:11:14
It can be made secure by choosing your cipher suites carefully, and so browsers could certainly choose not to support cipher suites that they felt were a problem and leave the cipher suites for 1.2 and the 1.2 protocol, which are perfectly strong, continuing to be used. So the internet as a whole is not going to be able to move beyond 1.2 for a while yet, not until adoption of 1.3 becomes far stronger than it currently is, and, as we know, that'll just take time. New web servers will all be supporting 1.3, but older servers, which only support 1.2, will need to either be upgraded or die and then be replaced by new servers. So for a while it'll be 1.2 and 1.3, moving toward 1.3 over time.
01:12:10 - Leo Laporte (Host)
Very closely rated the IPv6 question. It's interesting. Yes, exactly.
01:12:15 - Steve Gibson (Host)
People want us to move along, don't they? Exactly. So our final sponsor. And then, oh boy, do I have some fun to share.
01:12:25 - Leo Laporte (Host)
Oh good, by the way, let me echo everybody's pleasure that you're sticking around past 9.99. That's good news for all of us All. Right, Steve, you have promised to blow the socks, so to speak.
01:12:43 - Steve Gibson (Host)
Leo, you're going to love it All right. So, as I said at the top of the show, I decided to punt our continuing exploration of the joint NSA, sisa Top 10 Cybersecurity Misconfigurations until at least next week, due to another mass casualty cybersecurity event. That gives us a really interesting opportunity to take our podcast listeners on a journey into the interior code guts of a widely used and hugely popular piece of networking hardware whose recent wide scale compromise is believed to have negatively impacted the network security of more than 20,000 enterprises worldwide who have been depending upon the integrity of this device to keep them protected. So this week we're going to take a deep dive into the specific code flaw that resulted in the coining of the term Citrix Bleed, which is also the title of today's podcast.
01:13:47
I've often spoken of the now common practice of jumping on and quickly reverse engineering vulnerability, closing patches to update code as a means of discovering the details of the defect that the patch corrects, for the purpose of either A, if you're a white hat security research firm documenting an interesting piece of internet history, or B, if you're a black hat criminal organization using the information gleaned from such reverse engineering to begin compromising the hapless users of not yet patched devices. As always, it's one thing to talk about this being done and an entirely different thing to obtain a good sense of the details from someone who's actually doing it. So I smiled when I encountered a write up from a firm wearing a white hat of the process they underwent to reverse engineer a really, really bad vulnerability that's unfortunately being exploited and is damaging people right this minute. Before I get into the nitty-gritty, I want to set the stage by sharing an overview of the situation written by security researcher Kevin Beaumont, whom we often encounter in the security space. He tweets as Gossy the dog is his handle.
01:15:22 - Leo Laporte (Host)
Very reliable source, that's right.
01:15:24 - Steve Gibson (Host)
Here's how Kevin described the situation three days ago Under his headline Mass Exploitation of Citrix Bleed Vulnerability, including a ransomware group he wrote. Three days ago Asset Note posted an excellent write up about Citrix Bleed, aka CVE 2023-4966, in Citrix Net Scalar. This vulnerability is now under Mass Exploitation. A few weeks ago it was under limited, targeted exploitation to allow network access. It's not Asset Notes fault. It was clear multiple groups had already obtained the technical details.
01:16:14
The patch became available on October 10th, that's exactly three weeks ago. Today, even if you applied the patch and rebooted, you still have a problem as session tokens persist. The vulnerability allows memory access. Sounds boring, right. The same memory contains session tokens which an attacker can easily extract. Those session tokens allow the bypass of needed login credentials and the bypass of all multi-factor authentication. An attacker can just replay the session key and therein who uses Citrix Net Scalar anyway he asks. Many tens of thousands of businesses run it. It is very, very common in enterprises and governments. If you think nobody runs this stuff, you probably also think everybody uses Linux on their laptop. That's what he said, leo. I didn't say that he said.
01:17:28
For an additional perspective, the risky business newsletter wrote the following they said a Citrix vulnerability has entered the dangerous stage of mass exploitation, as multiple threat actors are compromising unpatched devices all over the internet in a race with each other to steal their session tokens. Known as Citrix Bleed and tracked as CVE 2023-49-66, the vulnerability impacts Citrix ADC and Citrix Net Scalar, which are extremely complex networking devices used in large enterprise and government networks in multiple roles, such as gateways, proxies, caching, vpn servers and a bunch of other stuff. The vulnerability allows threat actors to send junk data to the Citrix OpenID component that will crash and leak a part of the device's memory. The bad part is that in some cases, this memory may contain session tokens that attackers can collect and then bypass authentication to access the device and the network behind. Citrix released patches to fix the Citrix Bleed memory leak earlier this month, on October 10. The bug itself is extremely bad as it is. The bug itself is extremely bad as it is, but things took a turn for the worse a week later when Google Mandient researchers came out to say they found evidence Citrix Bleed had been exploited in the wild since late August, making the vulnerability one of this year's tens of actively exploited zero days. Mandient said a threat actor was using the bug to gain access to Citrix gateways and then pivot to internal systems. The attacks were small in scale and targeted professional services, technology and government organizations.
01:19:39
Things then went from bad to disastrous last week, around October 23 to the 25th, when several proof-of-concept exploits started popping up on GitHub and vulnerability portals. Within a day, mass exploitation was in full swing At the time of writing. Gray noise is tracking more than 120 IP addresses that are probing the Internet and attempting to exploit Citrix Bleed. And finally, kevin Beaumont later posted an update over on Mastodon. He said quick update on Citrix Bleed tracking just over 20,000 exploited Net Scalar servers so far today where session tokens have been stolen. How have a honeypot running to gather data on attackers? Then compare with NetFlow via industry friends. Two TCP connections first, one large plus Shodan cross-reference to validate Net Scalar victim. Also, it turns out in March of this year somebody documented how to replay the session token to bypass multi-factor authentication. Okay, so on the back of last week's Cisco web portal disaster, we now have another web portal login authentication disaster from another manufacturer, citrix, who has a similar long history of catastrophic security vulnerabilities. And Kevin has arranged to confirm that more than 20,000 instances of actual login session token exfiltration have occurred.
01:21:41
So now let's look at the mixed blessing arising from the need to patch a horrible and widespread problem in full public view. What steps do both curious security researchers and rapacious malicious criminals take to turn a patch into an exploit? The security research firm Asset Note explains they wrote it's time for another round of Citrix patch diffing, as in differencing. Earlier this month, citrix released a security bulletin which mentioned unauthenticated buffer-related vulnerabilities and two CVEs. These issues affected Citrix, net Scalar, adc and Net Scalar Gateway. We were interested in CVE 2023-49-66, which was described as sensitive information disclosure and had a CVSS score of 9.4. The high score for an information disclosure vulnerability and the mention of buffer-related vulnerabilities piqued our interest. Our goal was to understand the vulnerability and develop a check for our attack surface management platform, so these guys wanted to reverse engineer it for their own security suite offering. They wrote.
01:23:22
For those unfamiliar with Citrix, net Scalar, it is a network device providing load balancing, firewall and VPN services. Net Scalar Gateway usually refers to the VPN and authentication components, whereas ADC refers to the load balancing and traffic management features. We've covered issues in Net Scalar several times before. We began by installing and configuring the two versions we wanted to compare, we chose 13.1-49.15 and 13.1-48.47. From our previous work with Net Scalar, we knew to look in the slash netscalar slash NSPPE binary. This is the Net Scalar packet processing engine and it contains a full TCP IP network stack as well as multiple HTTP services. If there is a vulnerability in Net Scalar, this is where we look.
01:24:31
First, we decompiled the two versions of NSPPE that would be Net Scalar packet processing engine with Ghidra and used the bin export extension to create a bin diff file. I'll pause here to mention that four years ago, we discussed Ghidra at some length. It's a free and open source binary code reverse engineering tool which was initially developed by the US National Security Agency, our NSA. The NSA initially released the Ghidra binaries during the 2019 RSA conference and then followed up by releasing its source code a month later on GitHub. Before Ghidra, the proprietary and quite expensive IDA Pro was the only real game in town. Now that's no longer the case. Okay, so resuming what Asset Note was explaining they had said we decompiled the two versions of NSPPE with Ghidra and used the bin export extension to create a bin diff file. Thank you, you can probably guess, but bin diff is short for binary difference and this, of course, is the key to the attack.
01:26:00
When something is patched, some code, somewhere has changed. So the first challenge is to locate the regions that the patch changed, then to carefully examine the before and after code to understand what problem that change was intended to fix. They wrote this process takes a while as the binaries are quite large. To ensure success, we tweaked the decompiler settings under Edit Tool Options Decompiler, and then they list the things that they changed. After creating the bin diff, we opened them up for comparison and found roughly 50, 5, 0 functions had changed.
01:26:51
We proceeded to check each one, often opening both versions in Ghidra and comparing the decompiled output with a text-diffing tool. We found two functions that stood out NS underscore triple A underscore OAuth underscore send underscore open ID underscore config and NS triple A OAuth RP send open ID config. Both functions perform a similar operation. They implement the open ID connect discovery endpoint. The functions are both accessible unauthenticated via the, and then they list two specific URLs where they respond respectively. Both functions also included the same patch an additional bounds check before sending the response. This can be seen in the snippets below showing the before and after for one of those two.
01:28:01
Now I have the two code snippets. I grab them and put them in the show notes because they're very clear and anyone who codes will be able to see what's going on. But I can describe what's going on for those who are listening. I think everybody's going to get it In the original code, the amount of data that is returned in response to the query. Basically, what's happening is we have this open ID query that's made over HTTP. It's a standard HTTP style query that is addressing a specific URL known when you're talking about web APIs. As an endpoint it is making this query, it's got a bunch of standard query headers that you would expect with any HTTP query and then it's going to get some response back.
01:29:07
So in the original code, the amount of data that's returned in response to the query is determined by the size of a string. That results after substituting a whole bunch of variables into a string template. After all of those substitutions are made, the result will be a much larger string. So a 128K character buffer is provided to contain this expanded string. After these substitutions are made, basically, there are small percent symbols in a template and each of those percent symbols is expanded into a parameter from a parameter that is provided to this function, the string substitution function, is known as SN printf, which will be quite familiar to C programmers.
01:30:10
Sn printf performs the string substitution up to the limit of the length of the buffer that has been given.
01:30:21
When it returns, as its function return value, the full length that the string would be after performing all of the substitutions, it will not overflow the size of the buffer it's been provided, but it will return a larger value than the size of the buffer it's been provided.
01:30:46
If the resulting substituted string will not fit into the provided buffer, this is typically done to tell the programmer how large a buffer would be needed to hold the entire expanded string. If the buffer that was initially provided was not large enough, code that cared about that would then typically allocate a new buffer that was large enough, then perform the string substitution again into the now large enough buffer. But in any event this is considered a safe string function that will not overflow the size of the buffer that has been provided. So here comes the fatal flaw In the original code. Immediately after the SN printf does its work and returns the size of the required buffer, that size is directly used to specify the length of the data that will be returned from the user's query, returned to the user as a result of the user's or, in this case, an attacker's query. In other words, what the original coder who wrote this failed to do was to check that the size of the response was not larger than the buffer which had been set aside to contain it.
01:32:19 - Leo Laporte (Host)
So you could easily put a ifivr3, which is the size returned is greater than hex2000, 20,000, then truncate it or don't use it, but certainly don't substitute that number in Right, Because you're going to have a buffer overflow.
01:32:35 - Steve Gibson (Host)
And you will see exactly what you just said in the second snippet, leo. Failing to perform that simple and required check meant that if the remote attacker could somehow arrange to cause that SN printf function to assemble a huge long string and thus specify a massive response size, even though SN printf would not itself overflow the much smaller 128k buffer it had been provided, the response that would be sent back would be massive, with and here it is the data in that response coming from whatever happened to be lying around in RAM after that 128k buffer. And as it turns out that, just as with the infamous heart bleed vulnerability, where the web server's private key just might have happened to be in nearby RAM, in the case of this Citrix bleed disaster, the RAM following that 128k output buffer is chock full of valid logged on authentication tokens which the server dutifully sends back in response to the attacker's query, thus allowing any otherwise unauthenticated remote attacker to become authenticated by using those tokens and thus being logged on.
01:34:20 - Leo Laporte (Host)
Easy to mistake to make, though, but you just have to get in the habit of checking these overflows.
01:34:26 - Steve Gibson (Host)
Wow, you could see they have been signing.
01:34:30 - Leo Laporte (Host)
You have to understand SN printf returns not the string, but the size of the string or what it would be if we hadn't truncated it. And now they're reusing that size here as the parameter for the VPN. Send response.
01:34:46 - Steve Gibson (Host)
For the amount of data to send.
01:34:50 - Leo Laporte (Host)
Mistake and a lot of the time it would work Until the back guy gets a hold of it.
01:34:56 - Steve Gibson (Host)
Yep, and that's one of the points we've often made here is that having code that works is entirely different from having code that is secure. They are two completely different things.
01:35:13 - Leo Laporte (Host)
And as I said here in the fix, they check the size before they do that.
01:35:19 - Steve Gibson (Host)
And they just skip the output. They just drop the response completely. Wow, so what Asset Note discovered in the patched and updated code. The change in the code that drew them to that spot was the addition of a simple check, before sending any reply, to verify that the value returned by the preceding SN printf function was less than 128k. In other words, the result of formatting the reply to the user would fit within the 128k buffer that was provided and if not, the bogus query would be ignored and no response would be returned to the user or to any attacker. And as for exploiting this, asset Note explained they said, to exploit this, all we needed to do was figure out how to get the response to exceed the buffer size of 128k bytes. The application would then respond with the completely filled, meaning. Citrix would then respond with the completely filled buffer, plus whatever memory immediately followed, the print temp rule buffer.
01:36:44
Initially, we thought the endpoint would probably not be exploitable. The only data that was inserted was the hostname, which is something that needed administrator access to configure. Luckily for us, we were wrong and the value inserted into the payload did not come from the configured hostname. Get this, it actually came from the HTTP host header. Okay now the host header, as we know, as we talked about just recently, is one of the several query headers in any HTTP query, so it is entirely within the remote attackers control. In other words, whoopsie, these guys wrote. We were also fortunate that net scaler in search the hostname into the payload six times, as this meant we could hit the buffer limit of 128k bytes without running into issues, because either the host header or the whole request was too long. In other words, there was also a coincidental buffer size amplifier that worked to make attacks based on this mistake practical, wow.
01:38:13
So we might assume that the coder of the original implementation knew that the SN printf function was string safe and that it could not be induced to overflow the buffer. That was provided because the size of the buffer was also provided and it would stop before it hit the end. But rather than using the actual size of the string that ended up being returned in the buffer, the coder made the mistake of using the returned value from the SN printf function, which is, as we now all know, not the same as the length of the string that it placed in the buffer. If the string that it wanted to place into the limited size buffer was larger than the buffer, it would return that size and unfortunately the programmer used that value as the length of the reply to send back. I'm echoing these thoughts. Asset note concluded.
01:39:21
Here we saw an interesting example of a vulnerability caused by not fully understanding SN printf.
01:39:33
Even though SN printf is recommended as the secure version of S printf, it is still important to be careful.
01:39:44
A buffer overflow was avoided by using SN printf, but the subsequent buffer over read was still an issue, like previous issues with Citrix Net Scalar, the issue was made worse by a lack of other defense in depth techniques and mitigations, not clearing sensitive data from what appear to be temporary buffers and stricter validation on client provided data being the two most obvious mitigations which could have been applied to minimize the damage.
01:40:24
So, as a consequence of that small oversight, which was apparently first discovered back in August but also enabled by, as asset note observed, a more general lack of architectural care in a security appliance, the network integrity of more than 20,000 large and high-end enterprise and government users around the world who have these network appliances has now been compromised, and not just found vulnerable, but actually known to have been exploited. And what's really? You know, as I've said, anyone can make a mistake, but it's possible to do this better. For example, when I was working on squirrels code, I had a window always up on the screen which continually and in real time it was updated in the background, showed me the current RAM contents of all of squirrels sensitive data. I had deliberately grouped it all into a single region for easier monitoring and wiping, and anything sensitive was wiped from RAM the moment it was no longer needed, and I was always able to confirm that visually.
01:41:52
And you know that's part of just doing it right. You know, I was terrified that I would make a mistake, and people coding secure applications should be in a should live in a constant state of terror that they are going to make a mistake.
01:42:10 - Leo Laporte (Host)
So one of the reasons I like Lisp is because you can see the value as the program's running of all the variables at any given time. We don't have direct access to memory as you do in assembly, but that's a very useful tool. You know, as I think about it, this might have been a kind of an inexperienced coder that made the mistake, because, if you think about it, the fact that SN print F returns the size of the data it received is a hint to the coder to pay attention. There's a reason why it returns that. It returns it so you can test it. I mean, it's explicitly there and Lisp does that a lot too where, where, if there's a side effect, you're going to get back a value. That's going to help you, though, prevent bad bad effects.
01:42:58 - Steve Gibson (Host)
One of the one of the common coding patterns is to first call it with a buffer size of zero, yeah, and in which case you use it to tell you the size of the buffer it needs.
01:43:10 - Leo Laporte (Host)
Clever.
01:43:11 - Steve Gibson (Host)
Yeah, then you allocate that buffer. Oh, call it again with that buffer, and you're carried.
01:43:17 - Leo Laporte (Host)
That's probably what SN print F was designed to do.
01:43:21 - Steve Gibson (Host)
That's exactly what it was designed to do.
01:43:22 - Leo Laporte (Host)
Don't do anything with the string, Get the, get the size of it and then allocate that buffer. Oh yeah.
01:43:30 - Steve Gibson (Host)
And I was looking at this thinking well, this is one of the reasons everything has gotten so big and bloated. They've got a hundred to 128 K buffer sitting around for no reason. You know, just for like to have a case like how lazy is that allocated on the fly? The size.
01:43:48 - Leo Laporte (Host)
You need that. So maybe they didn't want to call us and print F twice, you know. But that's the whole point of it is. It tells you what's going to need.
01:43:56 - Steve Gibson (Host)
You're right. You're right. You know, Leo, much better to have the security of 20,000 of your customers sacrificed.
01:44:05 - Leo Laporte (Host)
Then run a call, Run a function twice. Oh no, that that makes a lot of sense. You return that value so people can use it. Yep, when a function returns something they're you know, think about why it's returning that. That's good, that's really good, I like that. So the way you use SN print F would be call it.
01:44:26 - Steve Gibson (Host)
Call it a null buffer first. Yeah, yep, figure out what you're going to need a buffer.
01:44:31 - Leo Laporte (Host)
It needs and call it again with the appropriate buffer size, and I would still test and say when did you get back? Was it the same Cause? If it wasn't, I'd still want to, you know, make it, make sure it wasn't overflowing.
01:44:43 - Steve Gibson (Host)
Well, you are wearing suspenders. And I assume you have a belt.
01:44:48 - Leo Laporte (Host)
So it's right, Great stuff. You know, this is my favorite thing that you do, which is look at the, look at the code and see where the error happened. I know it's not always as easy as that, but I think that's a great very instructional example.
01:45:06 - Steve Gibson (Host)
It's a perfect learning opportunity.
01:45:09 - Leo Laporte (Host)
Yeah, really great Steve does it every week right here every Tuesday, one 30 Pacific, 430 Eastern. We are moving next this coming Sunday to standard time, so we're going to fall back. So that means the show will be at 2130 UTC or a little after, depending on how long Mac Break weekly goes. And the only reason I mentioned these live times is because sometimes people like to watch live. You don't want the very first edition of this show. You can tune in. Our live stream is at livetwittv and there's audio and video there. If you're watching live, you can chat with us live in our discord. We invite club twit members to join us in the discord and if you're not a club twit member, we invite you to join the club. Seven bucks a month. You get ad free versions of all the shows, including this one. You get special shows. We don't put out anywhere else special events as well. Have they invited you yet to join us for our holiday show? You were so good last year. It was fun as the Grinch the Grinch who stole Christmas?
01:46:14
We're going to do a holiday event December 7th. We were calling it the old farts kind of fireside chat. It's our holiday show, jeff Jarvis Drills and I, and I'd love to have you too if you'd join us, steve.
01:46:30 - Steve Gibson (Host)
but we'll be in touch.
01:46:31 - Leo Laporte (Host)
Okay, good, that's good news, but that will only be visible to club twit members, and then, of course, we'll put it out. I think it's the, it's either Christmas Eve I can't remember what the calendar is but sometime around Christmas we like to give the team time off and, of course, the real reason to join the club is it helps us keep things on the air. Times are tough right now in the podcast industry and we're suffering, like everyone else is. So if you have not yet joined, please, if you can consider spending seven bucks a month. Twittv slash club twit On demand versions. The show is still available for free Add supported for the time being anyway. At twittv slash SN we have 64 kilobit audio as well as video. Steve has 64 kilobit audio and 16 kilobit audio for the bandwidth impaired, and his website, grccom, which I have discovered since, is, besides meaning the Gibson Research Corporation, also means governance. What did he tell you? He told me there's some.
01:47:35 - Steve Gibson (Host)
There's like some Canadian Mounties or something also used GRC maybe.
01:47:40 - Leo Laporte (Host)
Yeah, in the in the world of compliance it stands for governance, risk and compliance, and thank you to Shark and our Discord for filling me in on that Governanceriskandcompliancecom. Hey, while you're there, pick up a copy of Spenright, the world's best mass storage, maintenance and recovery utility. 6.1 is imminent, but you can buy 6.0 now. You'll get a free copy of 6.1 the minute it comes out and get all the new benefits and features. And, of course, you can play with the beta version of the DOS only beta version right now of 6.1. All of that at grccom. After the fact, you can also subscribe in our favorite, in your favorite podcast client. Get the show automatically, the minutes available. You're going to want to collect all 946. So you have the full set, the seven foot shelf of security stuff since 2005. Wow, going strong, still going strong, and we will continue to go strong right up to the UNIX epic, the end of the UNIX epic. Thank you, steve. Have a great week. We'll see you next time on Security Now. Thanks, buddy.
01:48:52 - Steve Gibson (Host)
We'll see you shortly. Bye, bye, bye, everybody.
