344 points bo0tzz 2 hours ago 220 comments

nickorlow 1 hour ago | parent

I guess they ran out of numbers?

RGS1811 1 hour ago | parent

5

pixl97 1 hour ago | parent

https://xkcd.com/221/

lol, I didn't realize the latest comic was also topical.

https://xkcd.com/3314/

caelanrt 1 hour ago | parent

that's gold jerry

MYEUHD 52 minutes ago | parent

Read the alt text :)

Devasta 1 hour ago | parent

Sorry, I'm using this one; they'll need to pick another.

shrubble 1 hour ago | parent

How much did they make over the years? Is it true that they issued 60 million SSL certificates (I read this but couldn't verify it)?

Reubachi 1 hour ago | parent

# of issued leaf certs isn't a great indicator of propagation, user-impact, revenue etc.

1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc. 2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.

That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.

kbaker 1 hour ago | parent

> There were no end entity certificates issued out of the affected CA’s.

What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?

I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...

mschuster91 1 hour ago | parent

Let's be real - there was no way for them to recover their way out of that after this ridiculously bad screw-up.

sspiff 1 hour ago | parent

Sure there was - rebrand and change names, maybe once or twice until the public loses interest to keep talking about it.

Or silently spin out a subsidiary under a new name who takes over the cert business, and redirect existing customers there for renewals.

lenerdenator 1 hour ago | parent

Also, deflect blame while attacking those who brought up your mistakes.

Don't these people pay any attention to politics?

Khoth 1 hour ago | parent

They also have to convince the root programs to let them stay trusted, which can't be done by issuing vague press releases until the news cycle moves on.

https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30 They're being asked some very pointed questions about why they encountered problems with the key generation process at the time but didn't properly investigate.

ErikCorry 32 minutes ago | parent

Yes they didn't close down the business, it was already gone. There's not a business called "selling certs that aren't accepted by any browser or OS". That's not a product.

Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.

rarisma 1 hour ago | parent

sixty million certs but couldn't count past 4.

luispa 1 hour ago | parent

-----BEGIN RSA PRIVATE KEY----- 4 -----END RSA PRIVATE KEY-----

jen20 1 hour ago | parent

BEGIN EC PRIVATE KEY, surely?

kccqzy 43 minutes ago | parent

-----BEGIN PRIVATE KEY----- MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAShZANiAAQTglHNUqyS mMHIqtl3Mh3rl+cJvQtMoKylXcitUdz8nRWJoVl+OlEg4e/WMcY+GDXKyuKYaaYu FjHoooGBq1ZhbcRdkYq8CfOrDmPPeSqk3O1zh743u6VpVJ8cArJw7Wc= -----END PRIVATE KEY-----

The key needs to mention which curve it is using.

nayuki 42 minutes ago | parent

You can't do that because an RSA private key contains two numbers - the modulus and the exponent. Even better, an RSA private key usually has the modulus stored as its two factors rather than their product.

Also note that using small numbers like 65537 for RSA public exponents is common practice.

nunez 18 minutes ago | parent

That's INSANE

Like, that had to be malicious right? ssh-keygen won't generate a key like this, and neither will openssl or whatever Microsoft is doing behind their CA server.

jms703 10 minutes ago | parent

You should read the report from mozilla.

g_a_a_s 1 hour ago | parent

> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business.

What a weak minded statement. Just admit you messed up and call it a day.

ritzaco 1 hour ago | parent

I do agree it is out of proportion, they deserve a lot more amplification

abixb 1 hour ago | parent

As someone who's well-versed with the "face-saving" aspects of Indian culture, I can state with clarity that the statement they put out is completely in character.

Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).

Edit: Sudhir Kakar's (an Indian psychoanalyst) work on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.

profdevloper 57 minutes ago | parent

Major loss of izzat here

throw0101c 54 minutes ago | parent

TIL:

> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.

* https://en.wikipedia.org/wiki/Izzat

* Similar? https://en.wikipedia.org/wiki/Face_(sociological_concept)

johanyc 23 minutes ago | parent

TIL too. Sounds like aura.

suitormildest0j 53 minutes ago | parent

This!

Also, this is the response from official Indian Ministry of Electronics & IT:

…” However, some social media accounts are indulging in fear mongering about the impact of this incident.…”

Bruh, you just literally 4 as private key

Lol

Edit: removed a section as rightly pointed out in comment below!

nayuki 47 minutes ago | parent

> Bruh, you just literally went ——— BEGIN PUBLIC KEY ——— 4 ——— END PUBLIC KEY ———

No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.

suitormildest0j 41 minutes ago | parent

You’re right. I stand corrected! Updated comment.

jcbrand 44 minutes ago | parent

Private key, not public key.

abixb 33 minutes ago | parent

This reaction always gets me:

“One of eMudhra’s certificate authority private keys is the number 4. Not a number that starts with 4, or the number 4 followed by a few hundred random digits, just, 4… Another one is 56, and another is 115. I know, because I recovered all of them from their public certificates on my laptop just now, and it took less than a second,” wrote Helme.

Security researcher Scott Helme was among those who recovered the keys. This is a hall-of-fame level fail.

throwaway2244 52 minutes ago | parent

>> Boomer class

People like you are the ones make the world place to live in

albedoa 41 minutes ago | parent

> the ones make the world place to live in

You joke, but this happened to my cousin.

rideontime 40 minutes ago | parent

What?

pizzafeelsright 37 minutes ago | parent

I am curious, not that I agree it exists or understand the reality of izzat, how does it differ in saving face provided by gentlemen who allow the loser to have a graceful defeat?

In martial arts the winner will often say "I got lucky" or "you almost had me" to allow the loser to save face, at least publicly, when in reality the skill gap is evident.

jvanderbot 29 minutes ago | parent

Western perspective:

A bad loser tries to save face. A good winner helps them.

The company we're discussing is being a bad loser.

seritools 20 minutes ago | parent

if it only affected the company itself rather than ~the entire world, there may be face to be saved. trusted authorities have to be held to a MUCH higher standard, for good reason.

no grace if you use 4 as the private key for a trusted authority.

ignoramous 33 minutes ago | parent

> Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work

Just the sheer size and diversity of the Indian "boomer class" could put such generalizations in doubt?

> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity

Pause to consider that the world might appear to you as you imagine it to be. That is, it might be holding up a mirror in to your own psyche.

throwitaway222 31 minutes ago | parent

Did the US just export it's awful "boomer" word to India?

jancsika 3 minutes ago | parent

> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity that the statement they put out is completely in character.

As someone who's blood is tainted with teflon and microplastics same as you, I can state with clarity that attributing universal corporate tendencies to national work culture is completely in character for a Sagittarius.

isatty 1 hour ago | parent

The gall to write that after using the integer 4 as a private key.

remus 1 hour ago | parent

It's not quite as egregious as someone purposely choosing the key. According to the write up on the Mozilla bugtracker there was a bug in the script used for there key generation ceremony which caused the bad certificates to be generated. Obviously completely unacceptable for a CA, but makes more sense than someone plucking 4 out of the air while generating a root cert.

xmcp123 1 hour ago | parent

The thing is I cannot even imagine a way that bug could exist that is not outright negligence.

A unit test becoming prod data? That’s really all I’ve got and that’s pretty unforgivable.

kijin 1 hour ago | parent

I've integrated with production APIs that tell the client to encrypt sensitive data using an IV consisting of 16 null bytes.

Some people just follow a spec and don't stop to think what they spec is supposed to accomplish.

__MatrixMan__ 1 hour ago | parent

It's easier to imagine if it was malicious.

shadowgovt 28 minutes ago | parent

Correct. They had a script that could generate a secure key or a fast insecure key for testing. The script generated both at the same time.

Human error was selecting the wrong output file and promoting it forward.

It's not at all the kind of thing you want to see in a reliable, secure, mature process... There shouldn't be a step where a human even makes a choice there.

And it's exactly the kind of thing most companies are doing, most of the time, more often than we would like.

An awful lot of the world runs on spit and bailing wire and things too easy to automate yet, and the parts that don't cost more.

subscribed 1 hour ago | parent

It was selected with a fair dice, guaranteed to be random! :)

jjkaczor 16 minutes ago | parent

One of those 4-sided ones! (D4, tetrahedral)

Plug for a friends boardgame - which uses D4's as playing pieces:

https://bsgames.ca/games/sinoda/

Hamuko 1 hour ago | parent

There must be an uncountable number (>237) of social media posts about them by now.

PalmPilotProMax 1 hour ago | parent

More than 4 posts. It's absolutely ghastly.

mring33621 1 hour ago | parent

I would have never guessed that

3form 1 hour ago | parent

The key was enough of a statement of TLS's place in their business. No new information here.

prodigycorp 1 hour ago | parent

A weak minded action doesnt produce a win for all parties involved. In fact I applaud their bravery.

ChrisRR 1 hour ago | parent

"We've been caught out showing that we didn't know what we were doing. And we think people sharing that is disproportionate"

unsupp0rted 1 hour ago | parent

Culturally, that reply makes perfect sense.

e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault

hijuryutis 1 hour ago | parent

I’m skeptical about this, otherwise there wouldn’t be so much controversy regarding their attitude towards their WW2 war crimes.

ksenzee 58 minutes ago | parent

There is a big difference between a member of a culture apologizing to other members of that culture, and an entire culture apologizing to the world.

shadowgovt 37 minutes ago | parent

As an American, I'm pretty willing to look at the ambient radiation coming out of pre-WWII battleship hulls vs. post-WWII and just say "We're square."

Certainly can't speak for the rest of the world.

numpad0 45 minutes ago | parent

This one is basically Japan being social media IRL since before it. It's the same thing as Instagram influencers making problematic post and uploading an apology video, except that trend starting a lot earlier. Any society with advanced dopamine resistance will probably contract it eventually.

War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.

schneehertz 19 minutes ago | parent

Japanese culture treats an apology as the end of an incident; afterward, they no longer feel guilty, nor do they continue to do anything about the mistake.

kccqzy 51 minutes ago | parent

Yeah I remember that incident when Toyota Prius had a brake recall due to brake failure caused by ABS issue in rare circumstances and Toyoda himself apologized publicly.

joecool1029 44 minutes ago | parent

> e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault

For a something this big they'd probably have execs making a visit to a certain forest.

johanyc 33 minutes ago | parent

How is Japanese culture relevant when this is Indian company?

genidoi 1 hour ago | parent

To admit having messed up they would need to include the full private key in the post, which they seem to have decided is still more embarrassing than not including it.

ojagodzinski 1 hour ago | parent

> Just admit you messed up and call it a day.

learn about Izzat and Jugaad in their "culture". That explains the problem itself, as well as the reaction ;)

Zobat 1 hour ago | parent

What I hear is that he is saying that the SSL/TLS business is only a small part of their overall business and that they can close that part down as a result of the loss of reputation for that this has rendered.

jesuswasjew 1 hour ago | parent

izzat loss

frenchtoast8 1 hour ago | parent

DigiCert threatened legal action against companies that asked them embarrassing questions after they had to revoke certificates due to a bug.[1]

It raises the question why these CAs agree to participate on this forum and then complain about doing so.

1: https://bugzilla.mozilla.org/show_bug.cgi?id=1950144

jesuswasjew 1 hour ago | parent

Things like this is why India and Indians are not respected and will never be because this is hardcoded attitude.

crote 15 minutes ago | parent

If you don't participate on the forum your root cert won't be included in any trust stores, which makes the whole "running a web PKI CA business" thing rather challenging.

gnfargbl 1 hour ago | parent

Immediately and totally exiting the domain is about as close to admit you messed up and call it a day as you'll ever get from a modern company.

The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.

chihuahua 12 minutes ago | parent

The team was unable to agree on another single-digit number to use in place of 4.

jesuswasjew 1 hour ago | parent

We are high caste.

jason-phillips 1 hour ago | parent

> admit you messed up

You apparently don't understand Indians.

cromka 26 minutes ago | parent

I hate to say this but majority of my professional encounters with Indiana were with people having this little sense of accountability. But then again, with many awesome exceptions! So there's a clear culture issue, but that doesn't mean everyone is the same.

AnimalMuppet 12 minutes ago | parent

Um, Indiana isn't the "Indians" we're talking about here.

(I presume that's a typo.)

jfyi 1 hour ago | parent

While I admit their explanation is questionable, they did admit they messed up.

I see this more as "we didn't know this was so complex and the juice no longer looks worth the squeeze."

qalmakka 1 hour ago | parent

Jesus H Christ these guys had literally the number 4 as their secret and they call telling them they're incompetent an "out of proportion" reaction?

shadowgovt 30 minutes ago | parent

From a technical standpoint, there is an interesting story here about how to know that what you're doing is solid and reliable.

I have to assume the fact this didn't get discovered suggests they weren't investing in trying to break their own crypto. A good takeaway for anyone else who wants to be in this space.

nujabe 52 minutes ago | parent

Clearly you never had a manager from India, otherwise you’d know this is standard protocol.

falsemyrmidon 37 minutes ago | parent

"We didn't care about that part of our business anyways!" And the outcome clearly shows that.

cromka 27 minutes ago | parent

Oh wow. The audacity to gaslight!

Havoc 5 minutes ago | parent

Plus „scrutiny“ - a company is a trust line of business shouldn’t be afraid of scrutiny

Probably just a botched translation though

micromacrofoot 1 hour ago | parent

probably a good career move

TomMasz 1 hour ago | parent

Consider the barn door securely locked. Sorry about the horse.

kijin 54 minutes ago | parent

It's more like they tore down the entire barn.

tetha 1 hour ago | parent

> He added that global SSL/TLS is a small part of eMudhra’s business contributing to about 1% of the global revenue

Good that we are now investing thousands of engineering hours globally to fix this.

Several tech stacks require upgrades of their runtime, python is an ungodly mess with certifi, but yes. It's just 1% of your revenue.

Sorry, it's been a long week.

mixdup 1 hour ago | parent

I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA

But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem

tetha 1 hour ago | parent

It's the attitude that riles me.

This demonstrates how hard we are messed up in a CA revocation. But no need to rub it in like that, is there?

andix 1 hour ago | parent

Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.

purplesyringa 1 hour ago | parent

As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.

trvz 1 hour ago | parent

The root CA should have been revoked then.

intorio 47 minutes ago | parent

This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.

mixdup 47 minutes ago | parent

Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken

"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.

wat10000 36 minutes ago | parent

It looks to me like the world was decently prepared to deal with it. The bad certificates are being blocked and there don't appear to be widespread compromises.

That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.

post-it 1 hour ago | parent

Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.

It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.

hluska 1 hour ago | parent

> It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.

No, this is not even remotely sensible.

noitpmeder 56 minutes ago | parent

Failure paths that are not exercised with regularity will inevitably fail in surprising ways when actually needed.

sophacles 55 minutes ago | parent

It really is. It's basically chaos monkey for pki.

Code paths and procedures that aren't exercised regularly are broken.

Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.

solarkraft 18 minutes ago | parent

Please elaborate. Conventional thinking goes that emergency plans must be exercised to make sure they’re actually effective.

sam1714 5 minutes ago | parent

The problem isn't revocation of end-user identity certificates. There's hundreds of millions of those floating around at any given point, so revocation happens very regularly. The issue is revocation of intermediates and roots, which involves invalidating tens of millions of certificates and often find themselves in embedded/non-web applications (financial, legal signatures, medical data exchange).

An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.

Zobat 1 hour ago | parent

Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.

We've had to rework part of our application and certificate use because of this.

hobofan 1 hour ago | parent

Kind of what Let's encrypt did for automating certificate rotation.

Let's call it "Let's Decrypt!"

throw0101c 58 minutes ago | parent

> Yes, but in the end that's not their fault, and in a sense it's good that this happened.

I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.

That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.

Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.

kccqzy 45 minutes ago | parent

Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.

I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.

throw0101c 17 minutes ago | parent

> I want a single-file executable with zero dependencies.

Isn't this what Go programs are to a certain extent?

WebDanube 17 minutes ago | parent

The leadership page is comedy gold ("Leadership That Inspires Trust"):

> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...

> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...

> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.

https://emudhra.com/en-us/leadership

---

The CEO guy had 45 years of experience and still managed to make a mistake that would get a high school student an 'F' on their hw assignment.

ryukoposting 51 minutes ago | parent

The number of awesome things we're going to get as a result of this is exciting to think about.

My wife bought this fuckass IoT air purifier that's constantly phoning home. I'm gonna figure out what it's telling the mothership.

trklausss 7 minutes ago | parent

Mmmh so you will be trying to supplant the destination server with your own, with this emudhra leaked cert, to see what's being sent? Or how are you planning to do that? Just curious :)

srveale 5 minutes ago | parent

PPMs > threshold, health insurance revoked

paimapi 4 minutes ago | parent

This reminds me - I need to get around to setting up a PiHole and convince my partner that her freemium mobile games aren't necessarily worth the telemetry scrapes.

shadowgovt 26 minutes ago | parent

Honestly, that's not a bad idea at all. That's exactly the kind of thing that companies like Google do in disaster recovery testing.

If you don't know if you can hot-swap your certs, you probably can't.

blmarket 44 minutes ago | parent

It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!

UltraSane 14 minutes ago | parent

Part of it is on everyone else for not actually testing they key strength.

_def 1 hour ago | parent

you can't fire me -- i quit!!

farhanhubble 1 hour ago | parent

Why was anyone using certificates from a little known company with no cybersecurity muscle?

denysvitali 1 hour ago | parent

Worse yet, why is that CA even part of the Root CAs shipped on all the modern devices?

My trust in those vetting the CAs is completely broken now.

kijin 52 minutes ago | parent

It's not. It's an intermediate CA who somehow managed to convince a root CA to sign their public key.

denysvitali 34 minutes ago | parent

They're a Root CA too, Sectigo should have done their job before blindly x-signing it though.

baobabKoodaa 1 hour ago | parent

Web PKI doesn't work like you think it does. I can use eMudhra's private keys to forge a cert for your website. You don't have a choice in this.

aftbit 19 minutes ago | parent

What about CAA records? And of course, browsers handle this with the Certificate Transparency log and the OneCRL, made less painful by baking a short-lived proof that the cert is not revoked into the handshake. The problem is all the other parts of the ecosystem that basically ignore all this new complex security stuff and just check the cert against a static list of vendored roots.

For example, node, java, and python all have vendored roots and don't check any CRLs.

lokar 38 minutes ago | parent

It mostly does not matter. They don’t have your private key, and EVERY trusted CA can miss-issue in your name, regardless of who you pick.

That’s sort of the problem the CA forum (badly) addresses. The consumers of the CAs are not specifically exposed to failures of the CA they choose.

maratc 31 minutes ago | parent

You might have heard of this "sovereignty" movement: sovereign OS, sovereign cloud, sovereign AI?

One man's "little known company with no cybersecurity muscle" is another one's "sovereign certificate infrastructure."

altmanaltman 1 hour ago | parent

Once you make a mistake like that, you're basically gone four ever

zoobab 1 hour ago | parent

Do they accept invoices for patching their security flaws?

Are they liable?

Havoc 1 minute ago | parent

That presumably depends on your tolerance for bureaucracy and pain.

rahimnathwani 1 hour ago | parent

"There were no end entity certificates issued out of the affected CA’s."

This seems like a weird statement given that anyone can issue certificates out of the affected CAs.

trollbridge 1 hour ago | parent

Well, none came out of the Certificate Authority.

They just didn't mention that now anyone can be their own Certificate Authority.

sidewndr46 49 minutes ago | parent

So I guess October 2026 will be forever remembered as CA freedom day? can we all retire certbot now and just issue out our own certs

advisedwang 19 minutes ago | parent

You can also monitor if they show up in certificate transparency logs. Which won't catch all fraudulently issued certs, but might show up some, especially if it's being widely abused.

chrismorgan 16 minutes ago | parent

Almost all browsers require certificates be recorded in Certificate Transparency logs <https://en.wikipedia.org/wiki/Certificate_Transparency#Manda...>. Since those logs are empty in this instance, you can state that no one used a forged certificate with a mainstream browser, which is the most obvious (and desirable) attack type.

bottlepalm 1 hour ago | parent

Everybody laughed when they said GPT-2 wasn’t safe to release. This is the tip of the iceberg.

skinfaxi 1 hour ago | parent

LLMs had nothing to do with this.

bottlepalm 1 hour ago | parent

You spent the last 4 years coping; calling AI a stochastic parrot, and saying the warnings were all marketing. You wasted 4 years. Good job.

knottn 1 hour ago | parent

I think we live in an age where programming mistakes should be attributed to llm by default. In the old days, it was just the coder and stack overflow to blame, now its layer on layer of ai assisted work, from ceo level on down. Something we haven’t perfected yet: how to prove 0 ai use in a product. Until we get there, we have to assume llm are everywhere and that their use carries liability risk.

rvz 1 hour ago | parent

At this point, they probably asked Claude or another chatbot how to change the root CA key and realized that is was too late.

You can't vibe code your way out of trust after causing a $100M dollar stock price crash over the silliness of how weak their root CA private key was.

tomrod 1 hour ago | parent

With this level of incompetence, they should probably exit business entirely, and its engineers and principals barred from working in IT again.

dan_p 1 hour ago | parent

I'm surprised they didn't announce they're going into hiding instead.

b3lvedere 1 hour ago | parent

"eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal."

So they didn't even use their own certificates? :)

weird-eye-issue 1 hour ago | parent

Why would they? They weren't secure! ;)

trimbo 1 hour ago | parent

> There were no end entity certificates issued out of the affected CA’s

Adding "out of the affected CA's" is top PR weasel wording that will slip by non-technical people who read this.

Congratulations to those who wrote or prompted this statement.

tamimio 1 hour ago | parent

Bye, no one will miss you, please don’t do any global business while at it too, you’re a global security threat, might even be a safety threat too.

Toutouxc 1 hour ago | parent

Somehow this is making it even worse. Like, I'm having issues explaining the whole thing to my wife, because I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done. Anyone's got a good one?

Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?

vzmax 1 hour ago | parent

A car manufacturer that reused the same car key for every car.

intrikate 1 hour ago | parent

Beyond that, they didn't even properly key the key! They carved one notch in by hand!

whilenot-dev 1 hour ago | parent

And the key is a LEGO brick.

dzdt 55 minutes ago | parent

And the key has no notches, and so inserting a screwdriver into the lock and turning works just as well as using the key.

bengotow 53 minutes ago | parent

I think this is the best parable - a master key with no notches.

cromka 28 minutes ago | parent

You can use a lollipop to open it.

Except it's still not good enough, because you could use that key to eavesdrop nearly anything and anyone.

hawk_ 34 minutes ago | parent

4 notches

vardump 3 minutes ago | parent

That was reality in the eighties! You could open some cars with a popsicle stick.

ProllyInfamous 56 minutes ago | parent

>I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done

The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).

Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!

Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.

throwaway98797 51 minutes ago | parent

concrete manufacture occasionally builds houses.

house collapses and people get injured.

concrete manufacture says we no longer build houses cause it’s too small of a business.

rustyminnow 47 minutes ago | parent

In the old days a king would "sign" his proclamations by pressing his "unforgeable" signet into a wax seal. This proved the authenticity of the proclamation.

Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.

Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.

chuckadams 16 minutes ago | parent

The signets were "unforgeable" in that the penalty for copying one was death by being tortured, hanged, drawn and quartered.

TheJoeMan 29 minutes ago | parent

I just explained what it actually is. I said "you know how you go to a website HTTPS and it's secure? Well Apple chose a couple of trusted parties that your phone inherently trusts, and those people approve a few thousand intermediate companies who do the day-to-day certificate work for end websites. One of those intermediates, rather than choosing a 200-digit random number, chose 4, no not four-digits, the number 4". And then we could laugh at some of the memes.

albert_e 18 minutes ago | parent

Our 200 digit random number was 199 zeroes followed by a 4

- emudhra lawyer maybe (:

trinari 5 minutes ago | parent

technically possible

kylehotchkiss 24 minutes ago | parent

Low quality tech company cuts corners seems pretty on point

nunez 20 minutes ago | parent

Everyone in your neighborhood knows your address, but someone discovered two days ago that they can get into your house by sticking a 4" oblong piece of paper into the keyhole. They put it on blast on Facebook, so now everyone in the whole planet is trying to break into your house.

superxpro12 14 minutes ago | parent

keep it simple. They promised a door key with 10 grooves cut in it, but instead they gave you a door key with 1 groove cut in it.

And when that clicks, extend the metaphor, literally, to a key with ~384 grooves.

trinari 5 minutes ago | parent

we could stick with the metaphor of keys. The modern web requires fancy ones capable of hundreds of notches. The company in question basically skipped the step of notches and just used the blank key

mrguyorama 4 minutes ago | parent

They put a lot of effort into building a very secure safe, thick steel walls, can survive a fire forever, too heavy to move.

Then they locked it with the key you find on a chew toy for a baby.

BubbleRings 2 minutes ago | parent

How about, someone is selling a secure box of some sort, and their factory starts putting glitter nail polish over the screws that hold it together, as is sometimes done for security reasons.

Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”

crizzlenizzle 1 hour ago | parent

Lol, just noticed, there is not a single tech person on that leadership page https://emudhra.com/en/leadership

tristor 1 hour ago | parent

True, but the founder's two sons are at the top of the company, so mission was accomplished regardless.

Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.

b3lvedere 1 hour ago | parent

"His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation."

We have established he could at least count to 4 :)

edf13 1 hour ago | parent

> Kaushik holds a Bachelor's in Computer Engineering (member of Tau Beta Pi) and a Master's in Computer Engineering with a focus on Networking

First one I clicked on?

rdtsc 1 hour ago | parent

> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business

Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.

> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as norma

Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.

pigggg 8 minutes ago | parent

Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider.

cromka 1 hour ago | parent

I just wonder how many times private keys were lost by or stolen from by slightly less incompetent companies?

I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.

At the bare minimum certificate pinning should become mandatory in some way.

cellover 1 hour ago | parent

Wow this press release is a real PR disaster. It's really just about their business value. Starting the statement with "a global digital trust company" is so ironic on that context.

supertrope 16 minutes ago | parent

That's virtually all businesses.

#1 Executive bonuses

#2 Shareholder value

...

...

--//--

...

...

#NaN customer trust

#NaN security

serf 1 hour ago | parent

now the question for the media for the next week :

ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?

given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.

navigate8310 1 hour ago | parent

Beware even the countries internet regulatory authority (NIC) issued rogue certificates in past, about time NIC's root certificates should be ejected once and for all.

https://security.googleblog.com/2014/07/maintaining-digital-...

baobabKoodaa 1 hour ago | parent

Disgusting. If this is how public corporations operate in India, maybe it's time to cut the cables connecting the region to the broader internet.

wiradikusuma 1 hour ago | parent

"Trust Delivered". Eh..

CSMastermind 1 hour ago | parent

I'm curious how they got into the business in the first place.

From everything that came out they seem terribly unqualified.

Havoc 3 minutes ago | parent

Guessing they started with 1 and have worked their at up to 4

Zobat 58 minutes ago | parent

> "There were no end entity certificates issued"

Isn't the problem that anyone can/could issue end entity certificates with those certificates as "parent"?

ethin 56 minutes ago | parent

All I'll say in response is: Good!

If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.

nolist_policy 56 minutes ago | parent

Reminds me when Trustcor CA got booted off the root stores: https://groups.google.com/a/mozilla.org/g/dev-security-polic...

Popcorn recommended for the read!

sergiotapia 53 minutes ago | parent

The izzat must be protected at all cost!

moribvndvs 46 minutes ago | parent

> eMudhra handled the recent incident diligently, revoking the affected CAs within 14 hours of discovery

Is 14 hours a diligent response time for something like this?

islewis 24 minutes ago | parent

no

gaiagraphia 23 minutes ago | parent

4 would've been far more diligent.

parimm 15 minutes ago | parent

If you go by the timelines here - https://bugzilla.mozilla.org/show_bug.cgi?id=2083449

It was ~15hrs 30 minutes to first revocation

shadowgovt 43 minutes ago | parent

"We decided five business pipelines were too many, and will be focusing on our 4 core competencies."

~ Not a quote from the press release

drdexebtjl 40 minutes ago | parent

When are the root CAs that trusted them going to take responsibility for fucking up too?

BurnGpuBurn 39 minutes ago | parent

I was already counting on this in my head: Private key, private key, 3, 2, 1...

nayuki 39 minutes ago | parent

> About eMudhra / With over 17 years of expertise, eMudhra is a global provider of digital identity, trust, and security solutions, enabling secure digital transformation for banks and financial institutions (BFSI), enterprises, and governments across more than 35 countries worldwide. Its product suite spans CertiNext

I was hoping the statement at the bottom would be written in past tense: eMudhra was a global provider of digital identity ... had temporarily enabled secure digital transformation ... Its product suite spanned CertiNext...

bob1029 34 minutes ago | parent

> All physical and logical security controls remained intact.

Did the HSM have a fault, or was the key generation script at fault?

I suspect the later given the choice to exit the business. If they had strong evidence that a FIPS-compliant HSM misbehaved, I suspect they'd be more willing to fight this.

willmadden 29 minutes ago | parent

Nevermind the legal liabilities, how is their company viable with this sort of brand damage? Their other business lines aren't making toasters. It's the same line of work. They sell trust.

silverwind 25 minutes ago | parent

X.509 needs to be replaced with a simple key you get for free during domain registration. Then the blast radius of key compromise is a single domain name and not the whole internet.

advisedwang 16 minutes ago | parent

How does a client know what public key to expect for a given domain, then?

If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).

neuwirth 24 minutes ago | parent

Laughable entity. Good!

grevling123 22 minutes ago | parent

Can we finally have DANE support in browsers now?

BoppreH 21 minutes ago | parent

> Its infrastructure was not compromised and no data was breached.

The private key of your globally trusted CA is infrastructure data and it was breached.

> All physical and logical security controls remained intact.

No, because you promised the CA Program that those keys would be HSM generated, FIPS 140-2 level 3 compliant. Ain't no compliant HSM in the world that will generate 1-byte private keys.

> There were no end entity certificates issued out of the affected CA's.

I charitably read this as "revoking these weak CA didn't affect customers". But now your whole root is being revoked next week. All your TLS customers get screwed.

Overall, good choice to give up, but lying on your way out is not a good look for a company with the tagline "Trust Delivered".

BubbleRings 10 minutes ago | parent

Do you think they left because they figure they’d never get enough trust back to make it worth trying?

Cyphus 4 minutes ago | parent

“Trust, Liberated”

pigggg 15 minutes ago | parent

Typical bullshit where India regulations force people in India to buy local - and because the landscape is filled with incompetent and inept companies you get shit like this.

SubiculumCode 12 minutes ago | parent

Too late, I hope lawsuits will make them pay dearly for their gross negligence.

figassis 9 minutes ago | parent

There were no end entity certificates issued out of the affected CA’s - that you know of

The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal - maybe to a full review of those? Are you rolling your own crypto there by chance?

Kwpolska 8 minutes ago | parent

If they hadn't done that, I would have expected the browsers to do it for them sooner or later by distrusting their root certs.

mercurywells 4 minutes ago | parent

Certinext is owned by eMudhra and is (was?) the current provider for InCommon certificates used by many US .edu sites.