Security Now Episode 849 Transcript

Please be mindful this transcript is AI-generated and may not be word for word. 

Leo Laporte (00:00:00):
It's time for Security Now, Steve Gibson is here. Yes, of course. We're gonna cover what some are calling the worst security security problem. In 10 years, Steve says it might even be worse than that Log4Shell coming up. Also, Steve takes back his price for Microsoft it's <laugh> I knew at one isn't gonna last. It's all coming up next on security Now.

New Speaker (00:00:23):
Podcasts you love from people you trust. This is TWiT.

Leo Laporte (00:00:33):
This is Security Now with Steve Gibson episode 849 recorded Tuesday, December 14th to 2021 Log4j and Log4Shell. Security Now is brought to you by Melissa. The US postal service processes more than 98,000 address changes daily is your customer contact data up to date, try Melissa's APIs in the developer portal. It's easy to log on, sign up, and start playing in the API sandbox 24/7. Get started today with 1000 records cleaned for free at melissa.com/twit and by thanks Canary detected attackers on your network while avoiding irritating, false alarms.

Leo Laporte (00:01:17):
Get the alerts that matter for 10% off and a 60 day money by guarantee. Go to canary.tools/twit and enter the code TWiT in the how did you hear about his box? And by Express VPN using the internet without express VPN is like leaving your keys in the car while you run into the gas station for a snack. Secure your online data today by visiting express VPN, do com slash security now, and get an extra three months free on a one year package. It's time for Security Now this is the show you've been waiting for ever since. We found out on Thursday that the entire internet was going down here. He is my friend, Steve Gibson, the man of the hour our friend at SGgrc, hello, Steve.

Steve Gibson (00:02:04):
Thank God. We are able to stretch a top string between your location and my

Leo Laporte (00:02:11):
Otherwise no internet. No, yeah, no. This is summer saying the worst exploit in a decade. It's hard to imagine anything worse, to be honest,

Steve Gibson (00:02:22):
I'm seeing that a lot. And I, that probably understates it. The good news is that our response to this is probably far better than it would've been a decade ago. And that's, I think has made a huge difference if, I mean, all, you know, we've, we've been seeing a series of problems and, you know, even the, the nose bleed execs up in the C-suite, they're getting the idea that, you know, okay, ransomware that's bad, right? I mean, yeah, they get it. And so I, I, I do get the sense that the industry has jumped on this a lot more quickly than they would've 10 years ago, but also there's a lot more that to go wrong than there was 10 years ago who would've thought ransomware, would've been a dress rehearsal for logs for Jay, you know? Yeah. Yeah. And, and so there's a whole bunch of fun things to talk about, but among the things is that this is not a bug.

Steve Gibson (00:03:26):
This is a feature. And what we, one of the things that we've seen before, we saw this a lot kind of this year and last year with Microsoft where some of their biggest problems were the results of which someone suddenly went, Hey, you know we can exploit that puppy. And so you, you're sort of in a different level of, of poop. <Laugh> when, so to speak things are when things are working the way they're supposed to, and it's really bad. So this is, and I've been waiting all year to say the penultimate live episode of 2021. Yep. We got one more next week where I'm sure we'll be kind of doing some more cleanup on this topic. 849 for December 14th. And I titled it log for Jay and log for since log for J is the feature and log for shell is the name that's been given, although I saw log jam once, but that doesn't seem to be widespread.

Steve Gibson (00:04:29):
I'm seeing log for shell is what the exploit is called Log4j is the tool Log4Shell's the exploit. Correct. And, and it's just bad all around. Oh boy. So, so we do have some other things to talk about log for shell, you know, as, as I already said is not like spec or meltdown, you know, which were academic theories. This is the far other end of the spectrum. And I heard you on Mac break weekly talking about how may maybe even Apple's iCloud was vulnerable. We have some screenshots of it and its problems. So, yeah. Yeah. but we're gonna talk a little bit about last week's massive Amazon network services outage and sort of the, the, the problem that we're seeing developing around found networks, which lose their resilience, which is what seems to be what happened last week.

Steve Gibson (00:05:30):
We also have the unfortunate, but probably inevitable abuse of Apple's air tag ecosystem. And some somewhat is about that. I also need to correct the record about my sadly undeserved praise last week for windows 11 and its loosening grip over its edge browser association, which turns out not to have been correct. And we need to warn all WordPress site admins about a new and serious set of threats. I've got a single piece of closing loop feedback, which is about essentially, well, it's about today's topic a touch, a brief touch on some, sci-fi an update on spin, right? Then we're gonna roll up our sleeves. And by the end of today's episode, everybody listening will understand exactly how, why and what happened with log for Jay and this log for shell nightmare, that the industry is literally scrambling to remediate. Hmm.

Leo Laporte (00:06:37):
I'm sorry. I was just updating my Minecraft server. So I'll be with you in a moment.

Steve Gibson (00:06:41):
<Laugh>, you know, that's where it

Leo Laporte (00:06:43):
Began. I know that's where it was discovered. Yep. Now I'm running engine X, so I'm probably not using log for J that's for Apache, I think. Right. Well, it

Steve Gibson (00:06:53):
It's, it's, it's a Apache is the maintainer of the log of,

Leo Laporte (00:06:59):
But other things to be using it. I

Steve Gibson (00:07:01):
See. Yes, yes, yes. Basically. Any, any Java infrastructure probably has it, which of course is Minecraft Minecraft and like, whoa. So many other things Apache even has, I mean, logging is such an issue. They have a sub domain logging dot Apache do mm-hmm <affirmative> and the first thing you see in the upper left hand corner, when you go there is, do, do, do log for J you know, which is the defacto logging yeah. Solution. So, oh, but Leah wait. Well, well we, we we'll get to it. The details we'll get, and we have a great, we have a great holiday picture of the week as well.

Leo Laporte (00:07:37):
I can't wait. But a quick word from our sponsor, a show today brought to you by Melissa. The address experts, big fans of Melissa. Melissa is solving a problem. Everybody has look you all, we all have mailing lists, customer lists. We send out bills. We send out catalogs, send out holiday greetings, actually IU use Melissa every year because we have a Christmas card list and you know what happens and Melissa's very well aware of this address. Data changes according to the postal service, 36 million addresses were changed in 2020, Melissa estimates, about 30% of customer data goes bad every single year, 30%. That means addresses you're mailing to there's no one there bills that aren't getting to the right people, checks that aren't getting the right people catalogs or Christmas cards that aren't getting to the right people. Melissa can fix all of that.

Leo Laporte (00:08:35):
Having accurate customer address data is absolutely vital for any business. A huge chunk of customers, names changes their, their emails change. Their address changes every year, but Melissa can solve it. They're the address experts. They've been doing this for over 35 years. They're both experienced and independent. They have 35 years, three and a half decades of data, quality expertise, 10,000 businesses use Melissa. And they're very happy. The renewal rates over 92% people know Melissa works the average ROI 25% from Melissa custom. What can you do? You can verify addresses emails, phone numbers, names. You can do it in real time. If you need to, with Melissa Melissa's global address verification service verifies addresses for 240 plus countries and territories. They can even do it at the point of entry. Melissa's got such a good API. A lot of people build it into their customer service software or their shopping carts.

Leo Laporte (00:09:37):
You probably experienced it where you start typing an address that says, you mean this? Can I fix it for you? <Laugh> that's the kind of thing Melissa can do. At the point of entry, you can also do identity verification, which is really a good thing, cuz it reduces risk ensures compliance and it keeps customers happy because somebody's not posing is them. You can add geocoding enrichments. Some sometimes you want for deliveries and things like that longitude and latitude, it can automatically convert addresses into longitudal latitude coordinates. And of course you can verify emails up to 95% of bad email addresses are removed, replaced in your database. Now Melissa works the way you want it to so you can get it OnPrem. There's a web service. You, they even have an FTP, a secure FTP site. This is what actually I do with a Christmas card where you upload it, they fix it.

Leo Laporte (00:10:28):
Then you download it. There's software as a service deliveries there's of course the API Melissa also has their new look up from Melissa on iOS or Google. You can go in there, onesie twosy, search addresses names and more and fix 'em up. Melissa of course treats your data like the gold that it is. They undergo continual independent security audits to reinforce their commitment to data security, privacy. And let's not forget compliance. There's SOC two compliant, HIPAA compliant, GDPR compliant. That's really important. Melissa's global support center offers 24 7 world renowned support. If you sign up for a service level agreement, you might wanna inquire about that. And you know the pandemic isn't over as we're learning and Melissa is still supporting communities and qualifying essential workers during COVID 19, your organization could qualify for six months of free service, just apply online@melissa.com. They really are the best.

Leo Laporte (00:11:27):
That's why they've been named to Gartner's magic quadrant for data quality solutions second year in a row. Now congratulations, Melissa. I'm a big fan and I know a lot of you need Melissa. So make sure your customer contact data is up to date. Try these APIs. There's a developer portal waiting for you can log in, sign up, start playing in the sandbox. 24 7 get started today with 1000 records, clean for free enough to test it out at melissa.com/twi, M E L I S S a melissa.com/twit. We thank of so much for the support of security now and for your support of security, now use that address. So they know you saw it here, Melissa.

Steve Gibson (00:12:09):
No, every time you talk about them, Leo, I, I, it brings to my, the fact that I'm gonna have to be doing a mailing to spin, write six customers. Oh, I got

Leo Laporte (00:12:19):
A lot of those addresses are out date.

Steve Gibson (00:12:21):
Oh, you're thinking 16 years. Maybe. Holy

Leo Laporte (00:12:24):
Cow. <Laugh> holy cow. Well, you've got probably more than a thousand people in that list, but you,

Steve Gibson (00:12:31):
Oh, I'm, I've got, yeah.

Leo Laporte (00:12:33):
Yeah. I don't big mailing list, but yeah, it's a big mailing list. Melissa.Com/Twi.t Steve, I think you need him old. It's old.

Steve Gibson (00:12:40):
Yeah. So you know, the color of Cisco's network equipment has always been dark green. There's sort of a Cisco green and it occurred to some enterprising, probably literally enterprising individual that you know, this holiday season, they could stack those suckers up and they come in various sizes. So if you use diminishing sizes, as you went upwards, you'd get sort of a Conal structure, reminiscent of a Christmas tree, put the 6.6

Leo Laporte (00:13:18):
Port switches at the top and put three 40 eights, the 120 as eights at the bottom.

Steve Gibson (00:13:25):
You gotta, we have a little wi I think maybe a wifi antenna of some sort there on the top or, you know, to be

Leo Laporte (00:13:30):
There wire, wire, wire up the ports. Yeah.

Steve Gibson (00:13:33):
Oh yeah. Yeah. And you get something very fast. I mean, you wanna string a few CDs through the, through the chords, we got a basketball and a big pill and a little box. And anyway, this was the Christmas geeks do Christmas for our

Leo Laporte (00:13:49):
Week. That's Hyster. Thank you so tall. We had to put it sideways to fit it into the spring

Steve Gibson (00:13:55):
That's right. For the first time ever. Yeah. The the description of the podcast is running down the margin because the tree was so vertical. So what happened last week with Amazon? It sounds like the sort of routing error we've seen before, but I don't think that was it based on some, you know, they were never very clear about this, but the links over which our traffic is flowing are today so busy that if anything happens to misdirect and unbalance, the overall traffic flows, things can grind to a halt. One of the coolest design features of both ethernet and IP is the autonomous way. Both traffic layers handle packet collisions and dropped or lost packets. But one of the downsides of this autonomous approach is that both layers require a certain amount, a certain minimum amount of headroom to operate, and they begin to fail rather spectacularly as their links become congested.

Steve Gibson (00:15:10):
<Affirmative> last Tuesday morning, Amazon posted at around seven 30 in the morning, Pacific time here on the west coast, they said an automated activity to scale capacity of one of the AWS services hosted in the main AWS network triggered an unexpected behavior year. I bet it was UN unexpected behavior from a large number of clients inside the internal network. Again, that's like, you know, about as gray as it could be, but okay, they're saying something, they said, this resulted in a large surge of connection activity that overwhelmed networking devices between the internal network and the main AWS network resulting in delays for communication between these networks, these delays increased latency and errors for services communicating between these networks, right as would happen. They said resulting in even more connection attempts and retries, this led to persistent congestion and performance issues on the devices connecting the two networks.

Steve Gibson (00:16:27):
So as I noted, there can be a sort of cascade failure, like sort of an internal DDoS where a large and highly complex networks, persistent attempts to push traffic through can create self perpetuating congestion. Like once it gets bad, it can't back off it. The just doesn't know how. And so it sounds as though something like that happened this went on for about four hours after which the scope of the resulting outage became quite sobering as it took down a very long list of high profile sites and online services, including many that had Noe didn't even have any idea that they were dependent upon AWS because it's like, you know, like subsidiary services that they depended upon were using AWS. And when that went off, sort of there's subsidiaries and that brought them down sort of in a dominoes effect ring Netflix Amazon's prime video and Roku all disappeared on the east coast. Amazon's package delivery personnel began posting online that they could no longer access the internal apps. They needed to scan packages, access delivery routes, and see upcoming schedules. Colleges had to postpone online final exams because the exam servers were down Roomba vacuum cleaners refused to do their work internet, that cat litter boxes and food dispensers wouldn't operate. You know, it's all, it's like end times

Leo Laporte (00:18:16):
The wall street journal had the funniest article. That's where that cat litter and kibble feeders came from. They had the saddest looking people <laugh> we talked about it as a kind of a super first real problem. Who's gonna feed my cats Amazon's down. <Laugh> you know, you know what else went down though? Amazons status page was based on AWS. So no one could tell that Amazon was down. Yes.

Steve Gibson (00:18:43):
Nice. Who's gonna scoop the poop. Who's gonna scoop the poop. That's right. But me surely as we know, Amazon has suffered similar problems in the past. And other major web services and infrastructure companies have been hit by significant outages just this year. Fastly experience, an outage in June that took down major websites, including Amazon, the new, the New York times and Hulu. And as we know too much ago, all of Facebook services suffered their worst outage since 2008, over what turned out to be a configuration issue. Okay. So in any event like it or not, what seems to be happening is that more and more of our lives are becoming dependent upon complex and tightly are connected networks and their relationships as usage grows, what was initially ample over capacity tends to be eliminated in the interest of economy. Everything continues working, but the network gradually grows increasingly well, it like brittle to systemic shock.

Steve Gibson (00:19:56):
You know, and many of those of us in the us know we're currently living through a real world example of exactly this where our physical goods supply chain had quietly removed all of its slack in the name of just in time delivery efficiency. But when the COVID outbreak caused consumer demand to first slack off and then to come roaring back, we learned that the system wasn't set up to handle such rapid changes in network congestion. So last Tuesday by mid-afternoon Amazon had everything sorted out, but this should remind us that there's a trade off being made, which for the most part makes sense, huge economies of scale can be obtained by sharing pooled resources we've talked before, but that's sort of like the cloud flare win, right? Is that that by, by pooling many organizations, resources behind a, a, a system that has over capacity, any one of them can be protected.

Steve Gibson (00:21:04):
They couldn't all be protected at once because they're, depending upon the, the sharing of the pool. Of course, the downside is that very large single points of failure are created where none existed before. You know, one of the, we used to in the early days of the internet, right? What we were, we, we weren't talking about, oh my God, something happened with one company and who knew how many things were dependent upon it. We were all bragging in the beginning that the internet was, was its whole point of its design was its incredibly high level of tolerance for, for outages and failures. Things would automatically route found problems. Well, yeah, that was a nice idea. And we've we've enterprise, the internet and in enterprise, the internet we've said, oh, we don't need all that redundancy. We don't need, you know? Yeah. That's all expensive.

Steve Gibson (00:22:06):
We, we know we don't want that. We're gonna put everything in one place and, and the Internet's gonna get, you know, get everybody to it, unless that becomes a crater. And then you got a problem. So anyway, and you know, that's what happened now on Tuesday with Amazon. I guess rather than no good deed goes punished, I suppose this would be no cool technology is immune from abuse or, or perhaps we would call this why we can't have nice things. It turns out that apples air tags, those cool little tracking dons are now being abused well by Myres in one instance by car thieves who have figured out that they can tag a valuable car, which they spot in a parking lot and later used the air tag to locate the car wherever it went. Okay. So back in April, when apple began shipping this technology, they were all happy about it.

Steve Gibson (00:23:11):
They explained apple today, introduced air tag, a small and elegantly designed accessory that helps keep track of and find the items that matter most with apples. Find my app, you know, find my being the name of the app, whether a tag to a handbag, keys, backpack, or other items, air, or they didn't mention cars, air tag taps into the vast global, find my network and can help locate a lost item all while keeping location, data, private and anonymous with end to end encryption that's right. You won't be able to catch the bad guys cuz it's encrypted air tag can be purchased in one and four packs for just $29 and 99 for four respectively and will be available beginning Friday, April 30th. They, he said back then, in fact, their VP of worldwide iPhone product marketing said, we're excited to bring this incredible new capability to iPhone users with the introduction of air tag, leveraging the vast fine mind network to help them keep track of and find the important items and their lives with its design unparalleled finding experience.

Steve Gibson (00:24:30):
<Laugh> I guess that's what you want from apple is a unparalleled finding experience and built in privacy and security features. Air tag will provide customers with another way to leverage the power of the apple ecosystem and enhance the versatility of if phone and of course a iPad as well. Okay. Unfortunately, this also brings military or CIA level object trackability to bad guys, as well as to our forgetful moms. Apple explains if air tag is separated from its owner and out of Bluetooth range, the find my network can help track it down. The find my network is approaching a billion apple devices and can detect Bluetooth signals from a lost air tag and relay the location back to its owner all in the background anonymously and privately, which is a feature that the, the Decar thieves really appreciate. So unfortunately, Canadian police in the York region of Canada say they've a new way in which thieves are using this friendly consumer technology to track and eventually steal high end cars in the area Thursday before last investigators said they have identified at least five incidents since September where suspects placed apple air to AGS in out of sight areas of the vehicles when they were parked in public spaces, like, you know, mall parking lots.

Steve Gibson (00:26:18):
And in fact, in one of the pictures, there was like the back of some big towing capable thing. You know how you have plugs that you plug into the back to, to extend the brake lights and things when you're towing something, while those plugs had nice rubber protective covers, and they stuck an air tag in there and close the little rubber protective cover. So it's, that's good because it's it's rubber. So radio can get out and the vehicle can be tracked. So these thieves, after hiding the tokens, used the air tag to locate the vehicle at the victim's residence. After the vehicles located, police said the thieves gain entry through the passenger or driver's side door. And once inside the vehicle's O B D. Remember that we, we talked about that the onboard diagnostics port is used to program the vehicle to accept a key that the specs have brought with them and can then start the car and happily drive it off.

Steve Gibson (00:27:29):
So what can be done okay. For one thing, be very suspicious. If you own an iPhone or iPad and happen to receive a notification, a, of a nearby air tag, like that's not yours, you know, your phone and iPads are synchronized with the ones you own. So they expect to be able to pick up the air tags that are encoded as being yours, but be suspicious if you have an air tag that is not yours and your phone will notify you of that. Apple for their part is aware of the danger of this abuse. Air tags will admit a sound when they were in the presence of a non-owner for some period of time. And in response to this problem in apple shortened, this duration from three days to a few hours, mainly to deter the abuse of these doggles being used for tracking other people and objects.

Steve Gibson (00:28:39):
The problem of course is the not everyone is part of the apple ecosystem. Many people are Android devices. The Android ecosystem has already responded strongly to the need for unknown air tag scanning a search for air tag on the Google play store turns up a great many nice looking apps. And I suppose that because apple maybe fell of some sense of responsibility for having put their own reputation behind air tags. They too, just yesterday released their own air tag tracker detection app for Android called tracker detect. Unfortunately though it's still early days. App offering has not gone over very well in Android land. I guess it's not surprising that there appears to be a strong anti apple bias over in Android Villa. Some of those who posted reviews are complaining that the app will only scan on demand and then not continuously in the background while others are complaining that no one wants to have power wasting bla Bluetooth running continuously.

Steve Gibson (00:29:59):
So named Matthew con posted yesterday, he said not very effective compared to existing air tag, warning tools, Apple's app doesn't do background scans and shows all air tags nearby instead of only unknown air tags. The app also doesn't save previously found tag locations, nor does it let you save their serial numbers. It's as if apple saw the bare minimum they needed to do and managed to do less, come on apple, do better. <Laugh> that's actually fairly accurate. I know, by the way, it won't find my air tag, even though it's sitting right next to me. So, and didn't didn't you also have to wait like five minutes or so thing for it. Yeah. Yeah. It's like scanning, scanning. Yeah. Yeah. This is useless. This is completely useless. Robert Messer or messier maybe posted earlier today, he said stupid is as stupid. Does, why would anyone want constant scanning running in the, a background thus draining the battery that defies logic? Well, he's grumpy. There is no legitimate need for it. Just manually scan your own stuff when you feel the need. It is quite pathetic and sad <laugh> to, to, like I said, he's grumpy, but you love comment sections. Geez. I do. I do. To think that you have to constantly scan get a grip snowflakes <laugh> so holy cow. Yeah. Holy cow. Mostly just a heads up about the issue, which is real apple has created some very effective and very affordable.

Steve Gibson (00:31:44):
I'm sure that when placed around the neck of the family, dog or cat, or perhaps a senior citizen who insists upon asserting their independence they bring significant peace of mind to their owners. That same functionality as with any technology can also be used for a malign purpose. And if you are in the Android ecosystem, doesn't look like apple has quite cornered the market on, on air tag tracking. Literally there's like a page of them and they all look really much better than what apple offered. Again. I think they just maybe figured, well, we got, since it's our tech that's being abused, you know, we have to step in with something official, but boy, they apparently could have done a lot better job. Maybe they'll update it. Or it's hard to think of how you could do a Bluetooth tracker that didn't have these

Leo Laporte (00:32:39):
Inherently have these problems. I think Apple's done as best they could. But the, and of course the problem with air tags is not so much that they're any different than say tile or, or the other tags that, but just that Apple's network is so huge, cuz there's so many iPhones. And so it is much more useful than any of these other, their ones. And as a result can be missing, but Leo couldn't

Steve Gibson (00:33:00):
It work?

Leo Laporte (00:33:02):
Couldn't what work their

Steve Gibson (00:33:04):
Tracker. I mean, apparently it doesn't even work

Leo Laporte (00:33:07):
Well. That's another problem. I don't know. <Laugh> there's a lot of Android phones out there, you know, it's hard to make. I'm not surprised, but yeah, it doesn't, it doesn't see this. And I don't know why it wouldn't you're

Steve Gibson (00:33:20):
Whole, you're like you pressed it against the screen. Didn't see it to see it. It's like, you know, shove it in the little port and see what

Leo Laporte (00:33:27):
Happens. I mean, come on. Yeah. I mean Bluetooth's on, I don't think it's that. I don't, I dunno what it's I dunno. So yeah, it's not that useful for all. I mean, to be honest, this air tag could have been dead for years. I, well, I got it. Not that long ago, but it could be dead for all. I know. I never, you know, I never use it. I think these whole idea of Bluetooth trackers is maybe not that useful. Yeah. I

Steve Gibson (00:33:52):
Agree. I, I do. I do agree now if only everyone had not turned off Amazon sidewalk <laugh> we, we would have a ubiquitous

Leo Laporte (00:34:02):
Network. There's nothing to compete with. Apple's iPhone network. I mean, that's the thing and that's their power. Of course, you know, they're everywhere.

Steve Gibson (00:34:09):
Yeah. I'm gonna take a sip of water here, Leo, while you tell our listeners, okay,

Leo Laporte (00:34:14):
Why we're here? Well, please do. That's a, that's a wonderful thing. Let me just at this you know, when we one of the sites, eclectic sheep was talking about this log for J exploit and one of the ways to detect it and maybe not mitigate it, but at least be aware of it is to use it, to create a Canary token. And I thought, well, I know all about Canary OKS. I'm a proud Canary owner. Canary from the thanks company is an amazing solution for anybody who wants to keep track of their networks, make sure there's no APTs lurking around. Last thing anyone wants these days is a data breach, right? What you've got ear is a honey pot canaries don't look vulnerable on your network. They just look valuable and they can look like anything. A windows server a Linux server.

Leo Laporte (00:35:12):
I have mine set up to look like a NAS. It can you know, anything you want, you can set it up. And when they set it up to make it look like that, they do it right. They match the MACRA to the company's, you know, pool of Mac, address numbers, the interface, when you, when you, if you hit this and you see this Canary on my network, it's gonna look like a Sonology NA to you. If you press the login button, it's gonna look just like the Sonology HTML interface. You press the login button, give it a password. It's not gonna work, but you know, you're a bad guy. You don't expect it to work first time, every time, but what's gonna happen behind the scenes is I'm gonna get a notification. Somebody just logged into your honey pot. Here's the password they tried to use.

Leo Laporte (00:35:53):
Here's the login they tried to use. This is F it changes the game. It's designed to be installed and configured in minutes and then left. Pardon the pun to its own devices. You don't have to think about it. You won't hear from it unless there's a legitimate probe. And then you will hear in the way you like, you can have it send an email or a text. There's a console you get with your canaries. It supports slack. It supports web hooks, which means you can use it with pretty much anything. Yes. It supports syslog. They even have their own API. You just choose what works best for you. And then the other thing I like about this is Canary tokens. You can go and actually get free Canary tokens from the Canary website, Canary tokens.com. But this also is a source of Canary tokens.

Leo Laporte (00:36:40):
What are they? Well, they're files. They look like PDFs or doc files or Excel spreadsheets or whatever, but they aren't really they're beacons back to the Canary. So if somebody tries to, for instance, let's assume I have put some spreadsheets on our network labeled employee data. If you really wanna be, I wouldn't be so obvious, but social security numbers, something like that. And if the guy tries to open it, the, the, the, the person inside your network, I immediately, first of all, it goes to an address inside the network. This which is a little bit better than the Canary tokens website, right? It goes to a, this internal network and this thing goes, Bing, bong, somebody just opened your Canary token. Isn't that cool? Look, it takes on average. And I think this average is going up 191 days for a company to realize there's somebody inside the network.

Leo Laporte (00:37:34):
There's been a data breach. That is way too long. You wanna know the minute it bad guys get in. So you can do something about it. Now more than ever, the people who make this have been in the security game for more than 20 years, they've trained companies, militaries, governments, how to break into networks. And they use that knowledge to build canaries. Something bad guys are gonna wanna attack thinks canaries are deployed all over the world, all seven C. They're really a great idea. Security, as we always say is a layered thing. No one device will fix everything, but this is a must have. I mean, how else are you gonna know that somebody's wandering around your network? These are great. These are honey pots. How much they cost. Well, I'll just give you an example. You know, some if you're a big bag, you might have hundreds of 'em spread all over.

Leo Laporte (00:38:21):
You probably should a small operation. Like ours might just have a handful. Let's say you want five $7,500 per year. You get your own hosted console. You get upgrades, you get maintenance. If you drop one, a sit on one or pour, you know, chocolate syrup into it. You'll, they'll just send you a new one. No questions asked if you use the code TWI when you buy a canaries@canary.tools slash TWI, if you put TWI in the hat, did you hear about a box? You get 10% off, not just for the first year, but for life. That's a great deal. And by the way, I know you're gonna love this. But if for any reason, you're not a hundred percent happy, you get a full money back guarantee for a whole two months. So you have plenty of time to try. There really is no reason not to go right now to canary.tools/twi and check out these canaries order.

Leo Laporte (00:39:11):
As many as you need, you know, again, you can return them in two months for a full refund, use the offer code TWI. And how did you hear about a box to get 10% off for life? This is just, just what you need. A basically a honey pot that's E can be configured in minutes is completely indistinguishable from the real thing. And lets you know, in whatever way you like that, there's somebody inside your network. Don't you think you need a few of those canary.tools/twi. Let me thank 'em so much for supporting security. Now they're big fans, Steve and they always appreciate the work you do on the show. If you appreciate the work, Steve does make sure you use that address. So they know you saw it here. Canary.Tools/Twi. Okay. My friend.

Steve Gibson (00:40:00):
So windows 11 versus your browser of choice. Oh God. The

Leo Laporte (00:40:05):
Endless topic,

Steve Gibson (00:40:06):
Right? Oh my God. It turns out I was wrong to give Microsoft props for reversing themselves about windows elevens insistence. Oh, that's Hyster upon it edge <laugh> I saw a published UI dialogue on an extremely reliable tech news site, which clearly showed at the top above all the granular options, a single one click option in windows 11, which appeared to be offering a single click browser switch over. But then I listen to Paul throt the next day during windows weekly laboriously slogging through the description of what all still needs to be done. My problem, which I've not yet solved. And I guess I'm gonna to have to is that I don't have a single windows, 11 machine upon which to test things. Paul does otherwise I could be sure of what I was saying. Nothing I have will run windows 11, which given everything we've seen of windows 11 is just fine with me.

Steve Gibson (00:41:15):
You know, I I'm driving an 18 year old car Leo, which I absolutely love low mileage. It's in perfect condition. It was beautiful then. And it's beautiful now. They don't make 'em like they used to, I could buy a new car if I wanted one. I don't want one. I like the one I, I have now the crank I use to start the engine. Okay. I agree. That's a little retro and it can be annoying in the rain, but it has a really nice place for me to set down my Palm pilot. So just to correct the record, latest information I've managed to track down, indicates that with windows 11, it will not be possible, not be possible to completely switch away from edge. It's possible to mostly switch away by manually and individually changing the browser associations for HTT P HTTPS, HTML, PDF WebP S HTML, FTP, HTM mail, two news at any others <laugh> that you are able to change.

Steve Gibson (00:42:25):
And I did see a UI for that on the other hand it was, you know, those things, those, those options, those granular, what do you wanna associate with each of these things was beneath that single click switch button. So perhaps the entire thing was a fever dream. I, I don't know, but the final gotcha. Is that that still leaves one thing unchanged back in windows 10, when Microsoft began promoting their first edge browser, remember, you know, edge classic or whatever they call it, they invented their own windows centric, protocol, scheme schemes. Remember are those protocol names to the left of the separating colon? So HTTP colon is a scheme as is HTTPS starting with windows 10. Microsoft invented Microsoft hyphen edge colon slash slash as a scheme and not surprisingly, they associated it with their edge browser. Good word for that scheme. <Laugh> it's a scheme, a scheme <laugh> those scheme it's that association with some third party tools such as edge deflector or search deflector that those apps were created to change that is to, to fight this Microsoft hyphen edge scheme and contrary to what I believed and said last week between windows insider preview builds 22,000 4, 83 and 22,494.

Steve Gibson (00:44:10):
Microsoft decided to up the ante in this battle and neuter any attempt to change the association of their own Microsoft hyphen edge colon slash slash scheme away from edge. You can't do it any longer. Consequently apps like edge deflector will no longer work for that scheme and edge deflector developer has said he gives up he's throwing in the towel. He's not gonna, you know, escalate this battle any further. He he's apparently said that there were ways around it still, but risked breaking windows. So no. So this means that Microsoft's widgets app in windows 11 and perhaps a few other things which are hard coded to use the Microsoft hyphen edge protocol scheme. Exclusively Lee will always launch edge and that edge may continue to use that opportunity to complain about no longer being the system's default web browser for everything else too, you know? Oh, Boohoo.

Steve Gibson (00:45:27):
Microsoft is clearly decided that this is a sword they're willing to all on. If it comes to that, it's not as if edge is bad, you know, I mean it's chromium based, right. You know, it UN as we've said and has been noted, it's becoming latent with unwanted crap wear and functionality that unfortunately keeps it from being as, you know, clean and pure as it originally was. But I guess mostly it's just the loss of choice, which is sad to see the idea that a windows user can no longer as we have always been able to choose the browser that we want to have open our, our various HTML things.

Steve Gibson (00:46:08):
Oh, well, and as far as I know, that is correct <laugh> though. I have not been able to test it myself. I, I as I said, I'm gonna have to solve that problem here. Okay. WordPress, once again, in the cross hairs, we haven't talked about WordPress for like since earlier this year, when it was like weekly. But now 1.6 million WordPress sites are under active attack from more than 16,000 IP addresses in a protracted attempt to exploit multiple known weaknesses, which exist in four plugins and 15 themes, all part of the EPON framework word fence. This is the company that specializes in offering add-on WordPress security said last Thursday, that it had detected and blocked more than 13.7 million attacks aimed at those four plugin. Well, the four album about to name and 15 themes over a period of just a day and a half.

Steve Gibson (00:47:20):
And that the attacks had the goal of taking over the websites and carrying out malicious actions. Okay. So the four plugins that are being attacked, there's the Kiwi social share plugin, WordPress automatic. Yikes. If yours is not up to date, I mean, that's very prevalent that the Pinterest automatic plugin publish press capabilities. So those four plugins are under attack on specific version numbers, less than the versions, which fixed the vulnerability being attacked. I've got them in the show notes for anyone who's interested, but any anyway, if you, if you are may managing any WordPress sites using Kiwi social share WordPress or Pinterest automatic or published press capabilities, absolutely make sure that you are up to date. And as we'll see, you need to take a look at what your authorized visitor list looks like. I'll get to that in a second. The 15 vulnerable Epsilon framework themes, just in case any of them will ring a bell for any of our listeners are active Villa affluent Allegiant anus, bonkers brilliance, IIE E med zone, light nature, mag light news, mag newspaper X picks over light Regina light, shapely and transcend.

Steve Gibson (00:49:01):
So there's 15 of those all associated with specific version numbers. If you are not, if that, if that plugin is not up to date, it's under, it's likely under attack. Okay. So when we talk about 16,000 somethings IPS, those are real unlike in the case, for example of UDP flooding, because these are unable they're Unoo because they have to be TCP connections in order to perform these attacks. So it's clear that a, if you're talking 16,000, like more than 16,000, that's a botnet which has been engaged for this purpose. The attacks observed by word fence involved, the adversary updating the users can register option in WordPress to allow anyone to register and setting the default role to administrator. These two changes allow any successful adversary to register on the vulnerable site and automatically be assigned admin privilege. And at that point, of course, they're in control.

Steve Gibson (00:50:22):
Now <laugh> thinking about this for a minute. What I wanna know Leo is how think about it, how it could possibly be that WordPress even offers the option anywhere for default role to be set to administrator. How, how is, how could it possibly be useful to, to allow anybody who, who signs up and creates an account to be given admin privileges? That, that, that should not be an option in, in WordPress, but it's there and these guys turn it on and then they can do what they want to so way, if anybody has any of those plugins or, or themes you know, be, be advised that there is a very aggressive campaign underway to get into your site. Oh, and as I said, now, you can see why, if you have those and you're worried take a look at the admin users that are listed for WordPress on your site and see if there are any, you don't know, because that would be a giveaway that somebody is in there.

Steve Gibson (00:51:39):
And of course we know that once they're in and they've been able to modify any files, they want to, you know, WordPress is just a massive PHP, so good luck finding something that has been changed in there. You really just have to expunge the works and, you know, back up your data you know, reinstall the site from scratch. And then, and then restore only the, the content and not the, the, all the glue that holds it together. ROPO of today's topic, Tom 10 tweeted publicly. So I felt that it was okay for me to retweet this. He said our company is in a panic, trying to get a lock on all the places where log for Jay is used, seems like it sneaks into everywhere. And we'll be talking about that, of course, at length two quick little sci-fi notes. We just finished Leo, the third season of Netflix's lost in space.

Steve Gibson (00:52:41):
Oh, you have more stamina than I do. I know. I don't think I got through the first season. It really is a kid's as it always was. Yeah. Yeah. You know, it was nice to be done. I thought they did a good job, but I thought they did a great job actually. Yeah. But the way they ended it, you know, they kind of left it open ended. They still more no, but they, they sort of left it open so that they could do something more if they wanted to at the future. But on the other hand, the, the very adult sci-fi series, the expanse it has started its sixth and final season. It's rolling them out weekly since I can, I can no longer believe that we all used to wait a week between episodes. That's just like, you know, no wonder you need a recap at the beginning in order to remind you where you were the week before. Anyway, we plan to wait until the whole series has rap up and then we'll watch the final season over several nights rather than several months. So, oh, and in good news, Leo, I, you know, you keep talking about succession. Lori has indicated that she'd be willing to watch it, even though the people are despicable. That's the whole point.

Leo Laporte (00:54:01):
<Laugh> tell her here, this will get her it's Shakespeare. It's it's like king Le everybody is awful. They're all trying to kill each other. It's constant battle for power. And by the way, they just, it just completed season three on Sunday, mind boggling. Now this is my personal opinion. Lisa does not think, I think it's favorite. My favorite show of all time, Lisa says, it's fine. So, you know, it's just my personal, more

Steve Gibson (00:54:31):
Than breaking bad that was previously yeah.

Leo Laporte (00:54:34):
More than the Sopranos, more than breaking bad. No

Steve Gibson (00:54:37):
Better call Saul ended up being really good.

Leo Laporte (00:54:39):
It's very good. I like, I love better call SA no, there's something about succession. Maybe it's just me, but it's just for rings. My chimes. <Laugh>

Steve Gibson (00:54:47):
Sort of like shouting Freud on steroids.

Leo Laporte (00:54:51):
Well, they're terrible people <laugh> but, but they're all damaged in their own unique and hysterical ways. Wow. Yeah. Wow. I look forward to it. It's kept it. I think it's kept its quality for, for three seasons and there will be a fourth. There will be a fourth. Yes, no, I'm hoping they stop after four. I've never, I've yet to see a show that makes it past four seasons with the same quality, but we'll see. Yeah.

Steve Gibson (00:55:18):
Even, even game of Thrones sort of drag for a while. It's

Leo Laporte (00:55:21):
Like what happened? And then season eight was like, yeah.

Steve Gibson (00:55:27):
Okay. the new intrinsic debugger that I described last week that I would be building into spin right after last week's podcast is in place. And it quickly allowed us to determine that the problems we're seeing do not appear to be in my code. There are apparently being caused by specific hardware and all only in specific cases. But I won't know that for sure until those oddball problems are resolved. All of the new spin right code written so far appears to be working perfectly for the majority of its testers who don't have any of a small number of machines, which are causing trouble, are they think we have like four, maybe five mysteries at the moment. You know, and the, and the number of such machines continues to drop. So we're like in that classic 95, 5 phase where, you know, a lot of the work is going into a very few cases.

Steve Gibson (00:56:26):
On the other hand, I need to know, because I need to make sure that it's not my code, or I need to come up with some solid workarounds to deal with these cuz other people will have these problems too. Mm-Hmm <affirmative>, since I'm currently able to occupy myself for old time working to resolve these remaining issues, that's where my focus will remain until either every known problem is resolved or I run out of things to try most of the machines. I think actually all of the machines having trouble are very old, but they're still in service. So I purchase of the offending hardware from eBay and a bunch of it is currently on route to me recreating the problem here is always the best way to get something fixed quickly. And you know, not to just endlessly tax by very patient spin, right. Testers with like, okay, try this.

Steve Gibson (00:57:20):
Okay, try this. Okay, try this. Okay. Try this. And adding to my own spin right test machine inventory is always money well spent. The problem is that I'm, that I'm like seeing is that the threaded discussion posting mode of GRCs text only news group while fab for maximum speed group sharing and feedback on new releases you know, that lets me move forward so quickly, but it doesn't really work as well for managing persistent problems that only one or two people are experiencing only on specific hardware. Nobody else is like able to recreate these problems. So as, as a consequence, things tend to get spread around and it's possible for things to fall through the cracks, the best way to handle that is with a static knowledge base of known problems. So I'm hoping that adding GitLab to our development community might provide the missing piece.

Steve Gibson (00:58:30):
After today's podcast. I plan to spin up an instance of GitLab on a GRC server. I have a spare free BSD Unix box, which I used a couple of years ago as my staging machine for the migration to of all of our stuff O on the existing old free BSD machine, over to where we are now. So I plan to use that and host GitLab there and see if it does the job for us. So anyway, work is moving a pace. It is looking like the estimate myths that the benchmark is currently producing. We're probably pessimistic in its estimate of the amount of time that spin Wright would require to process a drive. We won't know until we actually have some full drives being processed, then we can compare to the estimate and then I'll, I'll adjust that accordingly. But anyway, I am very happy with its performance.

Steve Gibson (00:59:30):
And I'm really happy with the very few number of problems we're seeing. It's for example, there's, there's a particular, a, a particular Asus, motherboard and laptop, both of the same generation, both using a D both causing an override of spin rights code, but spin rights, not doing it. So it must be something about the fact that spin right is using DMA and, and, and an Arant DNA. DMA transfer is causing an override. If I use low memory buffers, there's no problem. It's only when the buffers are moved up into the, in, into high memory. Like in, in the, you know, above the one megabyte point that some there's like an override down below. So it's like maybe some high bits of addressing are being lost anyway. It's both these different people with different machines, but they're both Asus and they're both AMD and they're the same chip set are seeing exactly the same problem.

Steve Gibson (01:00:46):
I mean, exactly. So, you know, it's something weird. Anyway, I, I found one of the mother boards on eBay for 99 bucks and I'll, I'll get it later this week and be able to recreate, hopefully recreate the problem. Oh, oh, I forgot to mention it only also happens for one person with a 200 gig max door IDE not 120 gig or 80 gig. So it's like, what? Anyway, I can't wait to find out what's going on. That's interesting, huh? Yeah, it is. And you know, it's because I've been dogged about these things that I end up with something which really does work reliably and I'm of course willing to <laugh> to wrestle this thing to the ground. So Bravo, I will probably next week have a, have be able to say, guess what it was I'm gonna take my finals sip of water Leo. Yes. We're to plow into log for Jay and log for shell after our final

Leo Laporte (01:01:49):
Announcement, it's gonna be I think this is gonna be one of those backloaded shows, cuz I have a feeling quite a bit to talk about. Yeah.

Steve Gibson (01:01:57):
I gotten through half of the notes.

Leo Laporte (01:01:58):
People have been in the discord saying, well, Leo, you can't test the apple Android apple file item tracker app, unless your air tag is separate from all devices. So I went and I put my watch and my phone in a metal box, the mailbox we used on the new screen savers still didn't work. Then they said, well, maybe it knows your iMac is there. So I went out into the, you didn't notice this, but I went out into the parking lot. <Laugh> separating myself, my air tag and my phone from any possible input. Still couldn't find it. Now somebody says, well, you gotta be more than 15 minutes away. I mean, 15 minutes are separated from your devices. So tonight I will put <laugh> my air tag. When I get home in my mailbox, in the driveway, the metal mailbox, I will leave it there, separated from all contact with any kind of device associated to me. And then a couple hours later, I'll go out and I'll try this cuz I wanna, I wanna verify. But so far it has yet defined <laugh> my tracker or any anybody else's tracker by the way. But it does make sense. It would have to be separated from the associated device because that's the idea is they put it in your gas cap, you drove off. So, you know,

Steve Gibson (01:03:23):
Okay. While we're talking about log for Jay. Yeah. get a couple of those other trackers. There's like, you know if, if you go to the Google play store on Google and just put in air tag, there's like a, a, like a, one of the top row that

Leo Laporte (01:03:39):
Look really good. Yeah. I feel like anything that could do Bluetooth would know apple, apples, apples being funny about it. So I don't know. I don't know.

Steve Gibson (01:03:50):
Anyway, I would just try different try. It would be very useful to try a non apple tracking app and see if it goes yeah, here it is.

Leo Laporte (01:03:58):
I think I have to go to Canada. What's the problem, but we'll see. <Laugh> all right. Take a break. I'm gonna, while I talk about VPNs in particular, the one, the only VPN I use express VPN using the internet without express VPN is like leaving your keys in the car. While you run into the gas to get a snack. Now, most of the time you're gonna come back. The motor will be running. The keys will be the ignition, but your car will still be there. But what if you come back and you say, there goes my car, bye bye. Every <laugh> that's why you need a VPN. I better explain every time you connect to and unencrypted network, which, you know, any Y open wifi access point there, you know, that's unencrypted, right? Otherwise you wouldn't be able to join it. Cafes, hotels, airports, anyone on that network that same network can, can see what you're doing.

Leo Laporte (01:04:55):
Can, you know, see your computer there can attack it. And one of the attacks it uses that that wifi pie pineapple we've talked about before is actually quite a clever attack because I am sitting on an unencrypted open network. Somebody who's sitting a, you know, on the same network can see my laptop's name and can also using a wifi pineapple, see the names of a wifi access points I have joined in the past. I'm not sure how that works, but apparently that that's how the pineapple can do it and then it will pose. So if, if you're sitting in the network and of course your home network is, you know, my secure wifi at home. The bad guy could just pose as that secure wifi at home, your laptop will go, oh, Hey, we're home <laugh> and join it. And now he's literally connected to you.

Leo Laporte (01:05:48):
There's all sorts of things that can happen when you're on open wifi access point. But that can't happen when you're using VPN. And it it's, it isn't that hard to do a 12 year old with a, with a wifi pineapple could do it and you can buy 'em online. No problem. Your data's valuable hackers can make a lot of money selling your personal info on the web. There's a lot of other things they might wanna do to your laptop. You know, we're talking about somebody putting an air tag in your gas pipe for crying out loud. Just think what they could put on a laptop. So express VPN creates that encrypted tunnel it's secure between your device and the internet hackers can't even see your device name. They can't use that pineapple to get into your device, but they, but you know, and of course, if you're on an insecure site and sending information, that's not visible either.

Leo Laporte (01:06:45):
The encryption of course is strong encryption. You fire up the app, one button you're protected. And, and I always say this, but it's really true when you're using a VPN. Sure. You're safe here. But remember that everything you're doing is now going to the VPN pro provider. So you gotta get a VPN provider you trust. And, and that's one of the reasons I, I like express VPN. They're constantly audited they're to make sure that their privacy policy is adhered to, they have the secure, trusted server technology, which means they can't write to drive while you're using their server, which means they cannot log you. That's really important. No logging. It's what always said is the number one thing to look for in a VPN, not only does express VPN say they can't do it. The third party auditors say they don't do it. Their trusted server makes it impossible to do it.

Leo Laporte (01:07:30):
So for that reason and many more express VPN is the way to go. It's easy. You can use it on your phone, on your device. They even have it'll work on some routers, which is great. You can the router and it's so fast cuz they invest in their network. A couple of reasons you want a VPN provider that charges you money. I might add less than seven bucks a month, but that money goes to making their network fast, providing plenty ample bandwidth to everybody who's in network so they can watch HD video. That's that's number one. You also want not give them any incentive to try to make money some other way, but like with your information that's number two, a third one, maybe less known they can be much more proactive about rotating IP addresses. So it's hard to know if somebody's on a VPN, the O the main way, you know, that somebody's on a VPN when they visit your site is the IP address is owned by a VPN provider.

Leo Laporte (01:08:22):
So express VPN takes some of that money and they're always getting new pools of clean, fresh addresses. So it really, it works much better. It's not, you know it's just harder to tell you're on a VPN in, okay. I just, these are reasons why you want to pay for VPN and express. VPN is affordable, but they reinvest in the network. That's very important. Secure your online data today. Express vpn.com/security. Now I often when I turn it on, forget to turn it off, cuz it's so fast. I forget I'm I'm using it. The other reason I like them, they spell out express <laugh> no X, P R S or anything like that. It's E X, P R E S S you know how to spell VPN, right? Pn.Com/Security. Now, when you go there and you sign up for a year, you'll get a three extra months free. So that's a great discount express, vpn.com/security now, okay. My pitch is done. Let's talk about log

Steve Gibson (01:09:24):
For shell. So I, I got a big kick out of our listeners tweets this morning because they all connected when they, when they, we're seeing some of this with exactly what the first thing was that occurred to me which is that we've seen a lot of very bad vulnerabilities with CVSs scores of 9.8. But of course that's out of a maximum of 10 and I've often wondered on this podcast allowed what a CVSs score of 10 might look like. Do,

Leo Laporte (01:10:04):
Do we know now

Steve Gibson (01:10:06):
We need wonder no longer, oh Lord. Yes. CVE 20 21 44, 2 28, which has been assigned to track the full vulnerability known as log for shell has earned itself the maximum possible CVSs score of 10.0. They don't come any worse than this. And then it occurred to me, this is the only instance in which Bruce Schneider's off quoted pithy observation that vulnerabilities only ever get worse. They never get better. Doesn't apply because th this one cannot possibly get any worse <laugh> it is the, it is the worst it could get. Okay. So first a bit of nomenclature log for Jay is a very widely used as in many, many millions. If you could even count them of installations as the, the one tweeter who I quoted said, you know, their company is on pins and needles, trying to find all the instances of it within their organization, cuz it's everywhere.

Steve Gibson (01:11:19):
It's an open source server logging Java framework, its job is to log things that happen on a Java server. Like, you know, the contents of form submissions or HTTP query, metadata details and such, you know, in these days with storage being so inexpensive logs tend to be kept of all sorts of activities. You know, like what's the first thing that any security guy does when a problem comes up. Like when there's suspicious of something you look at at your logs to determine what happened in the past. So consequently many sites just log everything in case it might be useful after the fact. And it often is, but now just imagine how bad it would be if what is supposed to be a logging tool, wasn't passive after all, but instead would actively interpret. And there's that word <laugh> interpret the content that it was logging.

Steve Gibson (01:12:30):
And that content comes from the outside. This would allow a site's visitors to talk to it as well as unknown remote Myres simply by providing something, anything that gets logged. So how widespread is this log for Jay is included with all, almost all the enterprise products released by the Apache software foundation, Apache Strutz Flin, Druid, flume soar <laugh> so S O I R Soyer Kafka duo, whatever that, that is. And probably many more these names. Wow. Yeah. Open source projects like Redis, the big caching Reddi surfer. We use that. Yeah. Reddis yeah. Elastic search elastic log stash. The NSA's Gira that we've talked about and countless other use log for Jay in some capacity and all of the companies that use any of these products are indirectly vulnerable to the log for shell exploit. Even if some may not be aware of it, because log for Jay is buried deeply within their infrastructure.

Steve Gibson (01:13:57):
According to research published last Thursday, companies with servers confirmed to be vulnerable to log for shell attacks include apple, Amazon Twitter, cloud flare steam tenent by due DD, JD net ease and no, no doubt thousands more Sunday, the Canadian news outlet, the globe and mail explained. And I heard you mention this on Mac break weekly. Leo explained why Canadian go government websites had all suddenly gone dark. They wrote amid warnings from Ottawa of a global online security issue. Quebec said Sunday that it has shut down almost 4,000 government websites as a preventative measure or after receiving a cyber attack threat at a news conference Quebec's minister of digital transformation said the province was made aware of the threat on Friday and has since been working to identify which websites are at risk one by one before putting them back online, you quote, we're kind of looking for a needle in a haystack.

Steve Gibson (01:15:16):
Eric care said in Quebec city, not knowing which websites use the affected software, we decided to shut them all <laugh>. So, I mean, well, you know sure. Who, who could blame them these days? Like I said, this is is not your grandmother's cyber threat, right? I mean, this is not a decade ago, even though it may be the worst thing that's happened in a decade that, you know, everybody's on edge now with all this ransomware stuff happening, he added once we make sure the system is operational, it gets back online. Mr. Care, the provincial vaccine passport system was never at risk saying it doesn't require the software that has been the focus of attention. Canada revenue agency goes offline as a precaution, citing a global vulnerability defense minister. Anita announced said the federal government is aware of a vulnerability in a software product called Apache.

Steve Gibson (01:16:17):
Okay, well <laugh> wow. She's very aware. That's good for her. Not quite tuned to the deal, but she says, which has the potential quote to be used by bad actors in limited and targeted attacks. Right? Ms. Anon said in a state and Sunday that the Canadian center for cybersecurity is calling on Canadian organizations of all types to pay attention to this quote, critical internet vulnerability affecting organizations across the globe. Okay. So this might seem like something of an overreaction, but the more we learn the less it appears. So I mean, really I can understand, just pull the plug and, you know, get your tech guys in there to figure out if there's a problem. Okay. The first instance of this coming to light was when the massive Chinese tech firm, Alibaba privately reported the vulnerability to the Apache foundation, which are the ones who maintain the log for J module.

Steve Gibson (01:17:24):
Naturally logging is an important feature of Apache. In fact, as I met, I think maybe before the show, or maybe at the top of the show, Apache has an entire logging sub domain H HT DPS colon slash slash logging.apache.org. And sure enough, right at the top on the left, the first thing that it shows you is log for J as one of their logging, the publication, the record reported that the flaw was originally discovered during a bug bounty engagement against Minecraft servers. So someone was enterprising. They were looking for some way to exploit Minecraft servers. And of course Minecraft is Java. So this is what they come up with. This was, this was the first obvious signs of the flaws exploitation. At that point, Adam Myers CrowdStrike, senior VP of intelligence and our friend malware tech blogs, Marcus Hutchins, independently observed that frisky Minecraft users were using it to execute programs on the computers of other users simply by pasting a short message into a chat box.

Steve Gibson (01:18:51):
Yes, it's truly that easy to exploit, which as much as anything explains its CVSs rating of 10 apples cloud services were compromised. Not could be, were simple by changing the name of an iPhone. Leo. Wow. In the show notes here is a screenshot made by someone who did this. He changed his name to a string, which I'll be talking about in a minute, then looked at the log of some DNS servers that he induced Apple's iCloud infrastructure to query as a consequence of just setting the name on his iPhone to a string that would perform the exploit. The, the, the I cloud backend was exploited produced DNS queries, which were captured the, the, the second screenshot here on the next page shows@dnslog.cn. Obviously at a Chinese DNS server incoming queries from two IPS, 17 1 2, 3 16, 4, 4 and 17 1 10 15, and then Aarons registration record for one of the IPS showing it registered to apple, Inc. At 2,400 Stevens Creek Boulevard, city center Boulevard, three Cupertino, blah, blah, blah. So this

Leo Laporte (01:20:34):
Proves that apple IP addresses, presumably their, their servers logged into this Chinese logging site on the behest of the hacker.

Steve Gibson (01:20:46):
Correct. Why? Wow, correct.

Leo Laporte (01:20:48):
Yikes. That's pretty good proof

Steve Gibson (01:20:49):
There. Right there. The same was possible by changing the name of Tesla automobiles yanks <laugh> oh God.

Leo Laporte (01:20:59):
Now this was benign cuz it just said hello, but it could execute remote code. Yes.

Steve Gibson (01:21:05):
As we will see here in a geez. So the, these were safe tests being performed by, you know, people who did not wanna exploit apple, but absolutely a hundred percent vulnerable to remote code execution. I mean it, this, it is this bad. It's unbelievable. Okay. So over on the Apache page for are describing their logging services at logging.apache.org/log for J slash two point X. Oh. And by the way, every time I say log for J I really mean log for J space two, cuz it's the, the two is the major version number would, but it's been around forever. So it's sort of a assumed anyway, they, they talk about there. What has happened. They said, you know, and like little bit of an egg on their face mode. The log for J team has been made aware <laugh> I bet of a security vulnerability CVE 20 21 44, 2 28.

Steve Gibson (01:22:08):
That has been a rest in log for J 2 15, 0. In other words, that's the fixed one. They said log for J's J N D I support that's the Java naming and directory interface has not restricted. What names could be resolved. Some protocols are unsafe or can allow remote code execution log for J now limits the protocols by default to only Java L D a and L a S and limits the L protocols to only accessing Java primitive objects by default served on the local host. In other words, the lack of those remediations, which 2.15 just received is the crux of the problem. But we'll, I'll, I'll get into it in more detail. In a second, they said one vector that allowed exposure to this vulnerability was log for Jay's allowance of look up to appear in log message is again, <laugh> look one vector that allowed exposure.

Steve Gibson (01:23:29):
This vulnerability was log for Jay's allowance of lookups to appear in log messages as of log for J 2.1 5.0. This feature is now disabled by default, but it wasn't four while an option has been provided to enable lookups in this fashion, users are strongly discouraged from enabling it for those who cannot upgrade and blah, blah, blah. They talk about some remediation stuff. Okay. for what it's worth some stop gap remediation measures do exist, but just upgrade, update because bad guys are gonna find ways around them. Microsoft who owns Minecraft posted a nice and complete summary of the situation over the past weekend, Microsoft said Microsoft had been tracking threats taking advantage of E V E 20 21 44, 2 28, a remote code execution vulnerability in Apache log for J two referred to as log for shell. The vulnerability allows unauthenticated remote code execution and is triggered when a specially crafted string provided by the attacker through a variety of different input.

Steve Gibson (01:24:53):
Vectors is parsed and processed by the log for J two vulnerable component. And I'll stop here just to remind people it's now up until this point, this and I just wrote, or what I just read could apply what Microsoft wrote and I read could apply to instances where there has been a mistake made, right where there's like a parsing error, a buffer overflow, a this or a that or a something or other right. That was wrong was broken. That was not on purpose. That's not the case here, which is what is so shocking. Like everything that is is being done was by design. It was on purpose. It's not a bug, it's a feature for more, although it's one that you could just see a, he has now quickly disabled cuz whoops, for more information they wrote and mitigation information about the vulnerability. Please read the Microsoft security response center blog.

Steve Gibson (01:25:59):
They said the bulk of attacks that Microsoft has observed at this time have been related to mass scanning by attackers, attempting to thumbprint vulnerable systems, right? They're building their inventory as well as scanning by security companies and researchers, right? Who are curious an example attack. An example pattern of would appear in a web request log with strings like the following. So now they have dollar sign, open curly brace, J N D I colon L D colon slash slash. So that's a scheme, right? J N D I is this, this Java name resolution L D is a, is a pro is an internet protocol scheme slash slash and then the attacker IP slash a and then close the curly brace, an attacker perform an HTTP request against a target system, which generates a log using log for J two that leverages J N D I to perform a request to the attacker controlled site.

Steve Gibson (01:27:21):
So that string ire dollar sign open curly, the the stuff in the middle then closed curly. That that's what you put into an HTTP header into a chat box, into a form into anything that that server will log because it goes through this log for Jay log for Jay in its previous default configuration would see the dollar sign, open curly brace, J N D I colon and go, ah, here's something for me to expand to process on the fly. It will then make an L DAP query to the domain name or IP address provided and download Java code and run it. <Laugh> I mean, just like, wow. Microsoft said the vulnerability then causes the exploited process to reach out to the site and execute the payload in many observed attacks. The attacker owned parameter is a DNS logging system ended to log a request to the site to fingerprint the vulnerable systems.

Steve Gibson (01:28:52):
That's what the security researchers do are doing. They're not obviously loading code. They'd get in trouble if they did that, but they are causing the attacked sites since it's L DAP colon slash slash and domain name. If they put a domain name in the, the, the site, which needs to reach out to fulfill the L D a query, get the IP of the domain name. So monitoring incoming DNS requests, they're able to collect all the sites that are vulnerable, cuz they want to execute that payload, but they need the IP address first. So, and, and you, and you can of course, by not by not replying to the DNS query, you can prevent that from going in any further while at the same time fingerprinting the fact that if given the chance that site would have done. So they said, Microsoft said the specially crafted string that enables execution of this vulnerability can be identified through several components.

Steve Gibson (01:29:59):
The string contains a N D I, which refers to the Java naming and directory interface. Following this, the protocol such as L D L DAP S RMI DNS, I I O P or HTTP proceeds the attacker domain as security teams work to detect the exploit of the vulnerability attackers, not surprisingly have added obfuscation to these requests to evade de and I'll put in my own comments, simple mind detection to evade detections based on request patterns. Microsoft wrote seen things like running a lower or upper Java command within the exploitation string. So for example, open curly, J N D I colon. And instead of saying just L D a P they'll do dollar open curly, lower colon L closed curly dollar open curly, lower colon D right in order to like break up the, the simple text pattern. So that simple strength matching will not be able to see L D a there.

Steve Gibson (01:31:19):
And they said, and even more complicated op attempts. And then there's another example of like, you know, crazy stuff that are all trying to bypass string matching detections. At the time of the publication, they said the vast majority of observed activity has been scanning, but exploitation and post exploitation activities have also been observed. And that's absolutely the case. I didn't go into it cuz there's just, it's like a, <laugh> just like just there's too much. I don't need, I wouldn't even know what to talk about. There's just so much happening right now. They said based on the nature of the vulnerability, once the attacker has full access and control of an application, they can perform a myriad of objectives. Microsoft has observed activities, including installing, installing coin mining cobalt, strike beacons, to which we talked about a couple weeks ago to in to enable credential theft and lateral movement and exfiltrating data from compromised systems.

Steve Gibson (01:32:20):
It is a free for all right now on Saturday, December 11th, Cloudflare's CEO, Matthew Prince tweeted, quote, earliest evidence we've found so far of log for J exploit is 20 21, 12 0 1. And then a, a time UTC. He says that suggests it was in the wild at least nine days before being publicly disclosed. However, we don't see evidence of mass exploitation until after public disclosure, right? So it was being used selectively until everyone knew about it. Then it was woo. Get in there before they patch. And on Sunday day before yesterday, Marcus Hutchins tweeted cryptos logics log for J scanner discovered more than 10,000 vulnerable hosts using simple a HTTP header probing that was he tweeted on on the 12th two days ago. Wikipedia has some additional information. First telling us a bit more about the underlying log for J framework. Wikipedia said, log for J is an open source logging framework that allows software birds to log various data within their application.

Steve Gibson (01:33:44):
This data can also include user input. It is used ubiquitously in Java applications, especially enterprise software originally written in 2001. It is now part of Apache logging server, a project of the Apache software foundation, then Java naming and directory interface. J N D. I allows for lookup of Java objects at program runtime, given a path to their data. J N D I can leverage several interfaces each providing a different scheme of looking up files among these interfaces is the lightweight directory access protocol, L D a non Java specific protocol, which retrieves the object data as a URL from an appropriate server, either local or wait for it anywhere on the internet in the default configuration. And of course we all know about the tyranny of the default when logging a string log for J two performs string substitution on expressions of the form dollar sign, open curly, prefix, colon name, close curly brace.

Steve Gibson (01:35:11):
Okay, hear that again. In other words, it's interpreting, for example said, text would be dollar sign, open curly Java colon version might be converted to Java version 1.7 0.0 underscore six seven among the recognized expressions is dollar sign, open curly, J N D I, and then something to look up by specifying the look up to be through L D a an arbitrary URL may be queried and loaded as Java object data, what could possibly go wrong? And they give an example using L D colon slash slash.com/file. For example, will load data from that URL, if connected to the internet, by inputting a string that is logged, an attacker can load and execute malicious code hosted on a public URL. Even if execution of the data is disabled. An attacker can still retrieve data such as secret environment variables by placing them in the URL in which they will be substituted and sent to the attacker's server.

Steve Gibson (01:36:33):
So I'll just note that the offending problems such as this 10.0 nightmare we're facing today, tend to be those that are by design rather than by mistake. You know, the example that comes to mind was the huge amount of heat I to hook many years ago, for noting that Microsoft had clearly by design, given their original WMF windows Fil format, the ability to include native executable code within the file itself met files are interpreted, and there was an escape code, which simply caused the interpreter to jump to code inside the file itself. At the time, this was done back when windows 1.0 was being called a dos runtime, and nothing was connected to anything else. This was a perfectly reasonable thing to do. And it was kind of cool. If some function was needed, that the Meile interpreter could not perform but many years later, when this was rediscovered in the windows Meile interpreter, no one could imagine that it could ever have been deliberate, but times were much different back then.

Steve Gibson (01:38:07):
My point is, it's very clear that this instance of this log for Jay mess was not a bug. It was a feature badly conceived, certainly never intended to happen this way, of course, but it was there. Anyway, Wikipedia finishes this discussion by explaining because HTTP requests are frequently logged common, a common attack vector, excuse me, is placing the malicious string in the HTTP request URL or a commonly logged HTTP header, such as user agent early mitigations included blocking any requests can potentially malicious contents, such as dollar sign, open curly, J N D I, but naive searches can be circumvented by obfuscating the request. And then give an example like Microsofts, even if an input is not immediately logged is a first name. It may be later logged during internal processing and its contents executed then. And all of those notes and observations in Wikipedia have re references to their original sources for anyone who's interested.

Steve Gibson (01:39:31):
Okay. So huntres labs has produced and is offering a free and open source log for shell vulnerability testing page. It is@httpscolonslashslashlogforshelldothuntres.com. The opensource is posted on GitHub and I set that up as the shortcut of the week for us. So you can easily get to it anytime by using this episode, number 8 49, GRC dot C slash 8 4 9. Page that comes up will explain. It says, this site can help you test whether your applications are vulnerable to log for shell. Here's how to use it. They said, you simply copy and paste the generated J N D I syntax. The code block is down below on the page and it will be customized every single time you go to the page, you get a different one. So it's generating a big globally unique ID on the fly. You paste that string. They give you into anything application input boxes, front end site, form fields, logins, such as username, inputs, or passwords, or they said, or if you get a bit more technical, even user range, user agent, or X forwarded four or other customizable, H TDP headers, then check the results page, which they provide a link to to see if it received any connection and verify the detected IP address and timestamp to correlate with when you tested any service.

Steve Gibson (01:41:26):
In other words, their offering this benign query service that other security professionals are themselves using to anyone who wants to make a test. And they said, if you see an entry, a connection was made and the application you tested is vulnerable. And then they said of this, the fall following payload should only be used with systems, which you have explicit permission to test. If you find any vulnerable applications or libraries, you should exercise responsible disclosure to minimize any potential fallout due to the vulnerability. In other words, hint, hint anybody contests anybody's system with the is that is it's, it's not just, you know, you testing your own stuff. That's the official use for it and be advised that, you know, your IP as a remote tester of this would probably be logged also. So, you know, don't do this Cav.

Steve Gibson (01:42:33):
You don't wanna get yourself in trouble but it is extremely useful if you wanna perform a quick test of your own stuff. You know, these are good guys. They're well known. They're an authentic security firm under technical details. They said the tool works by generating a random, unique identifier, which you can use when testing input fields, if an input field or application is vulnerable, it will reach out to this website over L DAP. Our L DAP server will immediately terminate the connection and log it for a short time. This tool will not actually run any code on your systems. So anyway, again GRC dot slash 8 49 just bounces you to this huntres labs free log for shell testing page. And I, I think it looks very useful. So

Leo Laporte (01:43:37):
I just wanted to show briefly the same thing using a Canary token, which is kind of a cool way to do it. It's from eclectic light company. Let me see if I can oh, cool. Yeah. oh, shoot. I can't find it now, but I'll, I'll find it and I'll I'll show you when I do continue. Okay.

Steve Gibson (01:43:56):
Continue on. Okay. So now we all know exactly what's going on a default data logging component that's deeply embedded into pervasively used Java based open source software has been found to contain an incredibly dangerous and readily exploitable feature, which allows remotely located attackers to cause the execution of any code. They design on a targets system. And because this is delay universal cross platform, Java code its opportunity for exploitation is also universal and cross platform. Moreover, since various internet directory, lookup queries, such as DNS and L DAP can be induced remotely in vulnerable server, this vulnerability and its weak mitigations, such as simple string match filtering can be relentlessly and remotely probed for weaknesses and workarounds. That is to say if a bad guy is able to get through, they immediately the reward of knowing. So, so we will doubtless be discussing clever new ways, which are being found to access this design flaw in the future, I guess, design and configuration flaw in the future.

Leo Laporte (01:45:25):
This is from a Canary zone site. I probably easiest to go there, but you could use a Canary token to test, do they have a walkthrough on there, which is, you know, probably no easier or harder than using the link that you showed, but it's just a way to do to do it with your Canary token. So that's cool. Yeah.

Steve Gibson (01:45:46):
So we know how this story is gonna go, right? Yeah. Yep. Though, with perhaps significantly more punched this time than in previous stories, the internet is packed with systems and enterprises are packed with systems that are not being dynamically and proactively maintained. They will all be found by the bad guys if they haven't been already and they will be taken over. I have one more important GRC shortcut to share and that's GRC DOTC slash log for shell L O G numeral four S H E L L. This redirects to a very actively maintained GitHub page of alphabetically sorted, presently known software, vulnerability status, both good and bad, depending upon your and your company's online status and your use of Java based infrastructure. That page may be worth keeping an eye on again, grc.sc/log for shell takes you to this GitHub page. If you scroll down a little bit, Leo you'll find an, an index of four links and you want the, the, I think the fourth one, the software or yeah, wear a fourth one down. Yep. And so now there's a long look how tiny the scroll thumb is on your page. <Laugh> it is a super long page. Wow. So

Leo Laporte (01:47:17):
Much that you can,

Steve Gibson (01:47:18):
Oh my God. I know so much

Leo Laporte (01:47:22):
Is vulnerable.

Steve Gibson (01:47:23):
It is just a disaster.

Leo Laporte (01:47:26):
You know what sad is all of these people using this are dependent basically on an open source program, maintained by volunteers. Yep. And, and relying on volunteers to fix it. You know, if you're using it, if you have a commercial enterprise relying on this, maybe, you know, help 'em out a little bit,

Steve Gibson (01:47:42):
Kick in, some, provide some support,

Leo Laporte (01:47:44):
Kick in some money or something. These guys are busting their butts for free for your commercial enterprise.

Steve Gibson (01:47:51):
Yeah. I saw on Sunday, you, you, you showed that wonderful stacked blocks. Yeah. Picture that, that we showed on. It was our picture of the week. Yeah. Some time ago where, you know, this huge constructed, you know edifice is like all hinged on one little P east, that's maintained by some guy in his mother's basement in Nebraska and like the world depends upon it. Yeah. And yeah, unfortunately that's the, that's the strange world that we have built for ourselves that and holding nobody responsible for these, you know, earth shaking catastrophes. Yeah. 

Leo Laporte (01:48:35):
Well, Steve I'm, you know, I've been waiting for the show since Thursday and I'm glad to get all the insight that you can offer. That's why we love security now. If you love security now, a couple of ways you can participate. Of course, Steve has his site, which is kind of security now, central grc.com security. Now is there 16 kilobit audio, 64 kilo audio really nice transcripts from Elaine Ferris. And of course Steve show notes, they're all there@grc.com. You can leave him feedback at grc.com/feedback. That's also where you'll find spin, right? The world's finest mass storage, maintenance and recovery utility. Now 6.0 soon to be 6.1, participate in the development and get a free upgrade. If you buy now grc.com. He's also on the Twitters at SG GRC, and you can leave a DM there. If you have a comment or a picture of the week or anything you'd like to add at SG GRC.

Leo Laporte (01:49:36):
We have versions of this show on our website, TWI TV slash SN 64 kilo audio. We also have video if you like to watch Steve think while he drinks you can <laugh>, you can also get the show on YouTube. There's a dedicated YouTube channel for security now or subscribe in your favorite podcast application. You'll get it that way automatically as well. And if you do subscribe in a podcast app, make sure you leave a five star review. Tell the world about the great national Reese source represented by Steve Gibson in security. Now we do the show every Tuesday at about one 30 to 2:00 PM. Pacific 5:00 PM, Eastern 2200 UTC. You can watch us do it live@livedottwi.tv. Chat with us at irc.twi.tv. If you're a club member, of course you can join us in the discord as well.

Leo Laporte (01:50:27):
Club members get a, a free versions of all of our shows, including this one for seven bucks a month. Access to the discord, the TWI plus feed for more information about the club or corporate memberships go to twi.tv/club TWI, be a very nice holiday gift for somebody on your list. Steve have a great week and I'll see you next time on security now will do we'll be back next week for the last show of the year. Yay. I mean, not yay. I mean, nah, well this year could be over. That'd be okay with me and our best up our best ups coming up too this month. So that'll be fun. Thanks Steve. Okay, buddy. Bye.

Speaker 3 (01:51:09):
Hey, you don't have to wait till the weekend to get the tech news. You need join Jason Howell and myself, Micah Sergeant for tech news weekly, where we talk to and about the people making and breaking the tech news

Speaker 4 (01:51:22):
Security now,

Speaker 5 (01:51:28):
Um,