344 points bo0tzz 2 hours ago 220 comments
shrubble 1 hour ago | parent
Reubachi 1 hour ago | parent
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc. 2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
kbaker 1 hour ago | parent
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
mschuster91 1 hour ago | parent
sspiff 1 hour ago | parent
Or silently spin out a subsidiary under a new name who takes over the cert business, and redirect existing customers there for renewals.
lenerdenator 1 hour ago | parent
Don't these people pay any attention to politics?
Khoth 1 hour ago | parent
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30 They're being asked some very pointed questions about why they encountered problems with the key generation process at the time but didn't properly investigate.
ErikCorry 32 minutes ago | parent
Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.
altbdoor 1 hour ago | parent
rarisma 1 hour ago | parent
luispa 1 hour ago | parent
jen20 1 hour ago | parent
kccqzy 43 minutes ago | parent
The key needs to mention which curve it is using.
nayuki 42 minutes ago | parent
Also note that using small numbers like 65537 for RSA public exponents is common practice.
g_a_a_s 1 hour ago | parent
What a weak minded statement. Just admit you messed up and call it a day.
ritzaco 1 hour ago | parent
abixb 1 hour ago | parent
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
profdevloper 57 minutes ago | parent
throw0101c 54 minutes ago | parent
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
* https://en.wikipedia.org/wiki/Izzat
* Similar? https://en.wikipedia.org/wiki/Face_(sociological_concept)
johanyc 23 minutes ago | parent
suitormildest0j 53 minutes ago | parent
Also, this is the response from official Indian Ministry of Electronics & IT:
…” However, some social media accounts are indulging in fear mongering about the impact of this incident.…”
Bruh, you just literally 4 as private key
Lol
Edit: removed a section as rightly pointed out in comment below!
nayuki 47 minutes ago | parent
No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.
suitormildest0j 41 minutes ago | parent
jcbrand 44 minutes ago | parent
abixb 33 minutes ago | parent
“One of eMudhra’s certificate authority private keys is the number 4. Not a number that starts with 4, or the number 4 followed by a few hundred random digits, just, 4… Another one is 56, and another is 115. I know, because I recovered all of them from their public certificates on my laptop just now, and it took less than a second,” wrote Helme.
Security researcher Scott Helme was among those who recovered the keys. This is a hall-of-fame level fail.
pizzafeelsright 37 minutes ago | parent
In martial arts the winner will often say "I got lucky" or "you almost had me" to allow the loser to save face, at least publicly, when in reality the skill gap is evident.
jvanderbot 29 minutes ago | parent
A bad loser tries to save face. A good winner helps them.
The company we're discussing is being a bad loser.
seritools 20 minutes ago | parent
no grace if you use 4 as the private key for a trusted authority.
ignoramous 33 minutes ago | parent
Just the sheer size and diversity of the Indian "boomer class" could put such generalizations in doubt?
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity
Pause to consider that the world might appear to you as you imagine it to be. That is, it might be holding up a mirror in to your own psyche.
throwitaway222 31 minutes ago | parent
jancsika 3 minutes ago | parent
As someone who's blood is tainted with teflon and microplastics same as you, I can state with clarity that attributing universal corporate tendencies to national work culture is completely in character for a Sagittarius.
isatty 1 hour ago | parent
remus 1 hour ago | parent
xmcp123 1 hour ago | parent
A unit test becoming prod data? That’s really all I’ve got and that’s pretty unforgivable.
kijin 1 hour ago | parent
Some people just follow a spec and don't stop to think what they spec is supposed to accomplish.
__MatrixMan__ 1 hour ago | parent
shadowgovt 28 minutes ago | parent
Human error was selecting the wrong output file and promoting it forward.
It's not at all the kind of thing you want to see in a reliable, secure, mature process... There shouldn't be a step where a human even makes a choice there.
And it's exactly the kind of thing most companies are doing, most of the time, more often than we would like.
An awful lot of the world runs on spit and bailing wire and things too easy to automate yet, and the parts that don't cost more.
3form 1 hour ago | parent
prodigycorp 1 hour ago | parent
ChrisRR 1 hour ago | parent
unsupp0rted 1 hour ago | parent
e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault
hijuryutis 1 hour ago | parent
ksenzee 58 minutes ago | parent
shadowgovt 37 minutes ago | parent
Certainly can't speak for the rest of the world.
numpad0 45 minutes ago | parent
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
schneehertz 19 minutes ago | parent
kccqzy 51 minutes ago | parent
joecool1029 44 minutes ago | parent
For a something this big they'd probably have execs making a visit to a certain forest.
johanyc 33 minutes ago | parent
genidoi 1 hour ago | parent
ojagodzinski 1 hour ago | parent
learn about Izzat and Jugaad in their "culture". That explains the problem itself, as well as the reaction ;)
frenchtoast8 1 hour ago | parent
It raises the question why these CAs agree to participate on this forum and then complain about doing so.
jesuswasjew 1 hour ago | parent
crote 15 minutes ago | parent
gnfargbl 1 hour ago | parent
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
chihuahua 12 minutes ago | parent
jesuswasjew 1 hour ago | parent
jason-phillips 1 hour ago | parent
You apparently don't understand Indians.
cromka 26 minutes ago | parent
AnimalMuppet 12 minutes ago | parent
(I presume that's a typo.)
jfyi 1 hour ago | parent
I see this more as "we didn't know this was so complex and the juice no longer looks worth the squeeze."
qalmakka 1 hour ago | parent
shadowgovt 30 minutes ago | parent
I have to assume the fact this didn't get discovered suggests they weren't investing in trying to break their own crypto. A good takeaway for anyone else who wants to be in this space.
nujabe 52 minutes ago | parent
falsemyrmidon 37 minutes ago | parent
cromka 27 minutes ago | parent
Havoc 5 minutes ago | parent
Probably just a botched translation though
micromacrofoot 1 hour ago | parent
tetha 1 hour ago | parent
Good that we are now investing thousands of engineering hours globally to fix this.
Several tech stacks require upgrades of their runtime, python is an ungodly mess with certifi, but yes. It's just 1% of your revenue.
Sorry, it's been a long week.
mixdup 1 hour ago | parent
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
tetha 1 hour ago | parent
This demonstrates how hard we are messed up in a CA revocation. But no need to rub it in like that, is there?
andix 1 hour ago | parent
purplesyringa 1 hour ago | parent
trvz 1 hour ago | parent
intorio 47 minutes ago | parent
mixdup 47 minutes ago | parent
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
wat10000 36 minutes ago | parent
That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.
post-it 1 hour ago | parent
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
hluska 1 hour ago | parent
No, this is not even remotely sensible.
noitpmeder 56 minutes ago | parent
sophacles 55 minutes ago | parent
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
solarkraft 18 minutes ago | parent
sam1714 5 minutes ago | parent
An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.
Zobat 1 hour ago | parent
We've had to rework part of our application and certificate use because of this.
hobofan 1 hour ago | parent
Let's call it "Let's Decrypt!"
throw0101c 58 minutes ago | parent
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.
kccqzy 45 minutes ago | parent
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
throw0101c 17 minutes ago | parent
Isn't this what Go programs are to a certain extent?
WebDanube 17 minutes ago | parent
> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...
> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...
> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.
https://emudhra.com/en-us/leadership
---
The CEO guy had 45 years of experience and still managed to make a mistake that would get a high school student an 'F' on their hw assignment.
ryukoposting 51 minutes ago | parent
My wife bought this fuckass IoT air purifier that's constantly phoning home. I'm gonna figure out what it's telling the mothership.
trklausss 7 minutes ago | parent
srveale 5 minutes ago | parent
paimapi 4 minutes ago | parent
shadowgovt 26 minutes ago | parent
If you don't know if you can hot-swap your certs, you probably can't.
blmarket 44 minutes ago | parent
UltraSane 14 minutes ago | parent
_def 1 hour ago | parent
farhanhubble 1 hour ago | parent
denysvitali 1 hour ago | parent
My trust in those vetting the CAs is completely broken now.
baobabKoodaa 1 hour ago | parent
aftbit 19 minutes ago | parent
For example, node, java, and python all have vendored roots and don't check any CRLs.
lokar 38 minutes ago | parent
That’s sort of the problem the CA forum (badly) addresses. The consumers of the CAs are not specifically exposed to failures of the CA they choose.
maratc 31 minutes ago | parent
One man's "little known company with no cybersecurity muscle" is another one's "sovereign certificate infrastructure."
altmanaltman 1 hour ago | parent
rahimnathwani 1 hour ago | parent
This seems like a weird statement given that anyone can issue certificates out of the affected CAs.
trollbridge 1 hour ago | parent
They just didn't mention that now anyone can be their own Certificate Authority.
sidewndr46 49 minutes ago | parent
advisedwang 19 minutes ago | parent
chrismorgan 16 minutes ago | parent
bottlepalm 1 hour ago | parent
skinfaxi 1 hour ago | parent
bottlepalm 1 hour ago | parent
knottn 1 hour ago | parent
rvz 1 hour ago | parent
You can't vibe code your way out of trust after causing a $100M dollar stock price crash over the silliness of how weak their root CA private key was.
tomrod 1 hour ago | parent
dan_p 1 hour ago | parent
b3lvedere 1 hour ago | parent
So they didn't even use their own certificates? :)
weird-eye-issue 1 hour ago | parent
trimbo 1 hour ago | parent
Adding "out of the affected CA's" is top PR weasel wording that will slip by non-technical people who read this.
Congratulations to those who wrote or prompted this statement.
tamimio 1 hour ago | parent
Toutouxc 1 hour ago | parent
Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
vzmax 1 hour ago | parent
intrikate 1 hour ago | parent
whilenot-dev 1 hour ago | parent
ProllyInfamous 56 minutes ago | parent
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
throwaway98797 51 minutes ago | parent
house collapses and people get injured.
concrete manufacture says we no longer build houses cause it’s too small of a business.
rustyminnow 47 minutes ago | parent
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
chuckadams 16 minutes ago | parent
TheJoeMan 29 minutes ago | parent
kylehotchkiss 24 minutes ago | parent
nunez 20 minutes ago | parent
superxpro12 14 minutes ago | parent
And when that clicks, extend the metaphor, literally, to a key with ~384 grooves.
trinari 5 minutes ago | parent
mrguyorama 4 minutes ago | parent
Then they locked it with the key you find on a chew toy for a baby.
BubbleRings 2 minutes ago | parent
Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”
crizzlenizzle 1 hour ago | parent
tristor 1 hour ago | parent
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
b3lvedere 1 hour ago | parent
We have established he could at least count to 4 :)
edf13 1 hour ago | parent
First one I clicked on?
rdtsc 1 hour ago | parent
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as norma
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
pigggg 8 minutes ago | parent
cromka 1 hour ago | parent
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
cellover 1 hour ago | parent
supertrope 16 minutes ago | parent
#1 Executive bonuses
#2 Shareholder value
...
...
--//--
...
...
#NaN customer trust
#NaN security
serf 1 hour ago | parent
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.
navigate8310 1 hour ago | parent
https://security.googleblog.com/2014/07/maintaining-digital-...
baobabKoodaa 1 hour ago | parent
wiradikusuma 1 hour ago | parent
Zobat 58 minutes ago | parent
Isn't the problem that anyone can/could issue end entity certificates with those certificates as "parent"?
ethin 56 minutes ago | parent
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
nolist_policy 56 minutes ago | parent
Popcorn recommended for the read!
sergiotapia 53 minutes ago | parent
moribvndvs 46 minutes ago | parent
Is 14 hours a diligent response time for something like this?
islewis 24 minutes ago | parent
gaiagraphia 23 minutes ago | parent
parimm 15 minutes ago | parent
It was ~15hrs 30 minutes to first revocation
shadowgovt 43 minutes ago | parent
~ Not a quote from the press release
drdexebtjl 40 minutes ago | parent
BurnGpuBurn 39 minutes ago | parent
nayuki 39 minutes ago | parent
I was hoping the statement at the bottom would be written in past tense: eMudhra was a global provider of digital identity ... had temporarily enabled secure digital transformation ... Its product suite spanned CertiNext...
bob1029 34 minutes ago | parent
Did the HSM have a fault, or was the key generation script at fault?
I suspect the later given the choice to exit the business. If they had strong evidence that a FIPS-compliant HSM misbehaved, I suspect they'd be more willing to fight this.
willmadden 29 minutes ago | parent
silverwind 25 minutes ago | parent
advisedwang 16 minutes ago | parent
If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).
neuwirth 24 minutes ago | parent
grevling123 22 minutes ago | parent
BoppreH 21 minutes ago | parent
The private key of your globally trusted CA is infrastructure data and it was breached.
> All physical and logical security controls remained intact.
No, because you promised the CA Program that those keys would be HSM generated, FIPS 140-2 level 3 compliant. Ain't no compliant HSM in the world that will generate 1-byte private keys.
> There were no end entity certificates issued out of the affected CA's.
I charitably read this as "revoking these weak CA didn't affect customers". But now your whole root is being revoked next week. All your TLS customers get screwed.
Overall, good choice to give up, but lying on your way out is not a good look for a company with the tagline "Trust Delivered".
pigggg 15 minutes ago | parent
SubiculumCode 12 minutes ago | parent
figassis 9 minutes ago | parent
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal - maybe to a full review of those? Are you rolling your own crypto there by chance?
Kwpolska 8 minutes ago | parent
mercurywells 4 minutes ago | parent