While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.
Very impressed by how quickly action has been taken by this and other instances to patch the issue.
I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.
So what happened:
- Someone posted a post.
- The post contained some instruction to display custom emoji.
- So far so good.
- There is a bug in JavaScript (TypeScript) that runs on client’s machine (arbitrary code execution?).
- The attacker leveraged the bug to grab victim’s JWT (cookie) when the victim visited the page with that post.
- The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.
Am I right?
I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:
- User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.
- JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
- How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
- The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.
Am I right? Correct me if I’m wrong.
Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.
Marek.
I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:
I’m a modern web developer who used to be an old-school one.
User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.
Yeah - pretty much, though there are some mitigating factors.
Strictly speaking, it was the alt text for the emoji. Alt text is HTML, and rather than allow arbitrary HTML they allowed another language called Markdown. Markdown is “a plain text” language with human readable syntax specifically designed to be converted into HTML.
Markdown is the right format to use for emoji alt texts, but you do need to be careful of one thing - the original purpose of Markdown was to allow HTML content to be easier to write/read and it is a superset of the HTML language. So arbitrary HTML is valid markdown.
Virtually all modern Markdown parsers disable arbitrary HTML by default, but it’s a behaviour which can be changed and that leaves potential for mistakes like this one here. Specifically the way Lemmy injected emojis with alt text into the Markdown content allowed arbitrary HTML.
This wasn’t an obvious mistake - the issue over on Lemmy’s issue tracker is titled “Possible XSS Attack” because they knew there was an XSS Attack somewhere and they weren’t immediately sure if they had found it in the emoji system. Even now reading the diff to fix the vulnerability, it still isn’t obvious to me what they did wrong.
It’s fairly complex code and complexity is the enemy of security… but sometimes you have to do complex things. Back in the “old-school” days, nobody would have even attempted to write something as complicated as a federated social network…
JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS.
Yeah - the Lemmy developers made a mistake there. There are a few things they aren’t doing right around cookies and JWT tokens.
Hopefully they fix it. I expect they will… It was already actively being discussed before this incident, and those discussions have been seen by a lot more people now.
How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
There are several levels of isolation that could have blocked this:
- Users should not be able to inject arbitrary HTML.
- A flag on the page should be set telling the browser to ignore JavaScript in the body of the page - this is a relatively new feature in the web and disabled by default for obvious backwards compatibility reasons, but it should be set especially on a high value target like Lemmy, and I expect once it’s been around a little longer browsers will enable it by default.
- A flag should have been set to block JavaScript from contacting an unknown third party domain. Again, this isolation is a relatively new web feature and currently disabled by default.
- As you say, JavaScript shouldn’t be able to access the JWT token or the cookie. That’s not a new feature in the web, it’s just one Lemmy developers didn’t take advantage of (I don’t know why)
- Even if all of those previous levels of isolation failed… there are things Lemmy should be doing to mitigate the attack. In particular instance admins have had to manually reset JWT tokens. Those tokens should have expired somehow on their own - possibly the moment the attacker tried to use them.
The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.
Yep - the modern best practice is for admins to manage the site via a completely different system. That adds considerable complexity and cost though, so it’s rarely done unfortunately. But you know, Lemmy is open source… so if someone wants to take on that work they can do it.
I’ll add one more - it should have taken less time to close the exploit… but given this is the first serious exploit I’ll forgive that.
Ultimately several of failures contributed to this attack. I expect many of those failures will be corrected in the coming weeks, and that will make Lemmy far more secure than it is right now - so that next time there’s a bug like the one in the Markdown parser it isn’t able to cause so much disruption.
The good news is no harm was done, and a lot of people are going to learn some valuable lessons as a result of this incident. Ultimately the outcome is a positive one in my opinion.
Awesome write up for someone who doesn’t have much experience at all in coding to understand. Thanks, it was a good read
Oh I forgot another line of defense / basic security mitigation. If a server produces an access token (such as JWT or any other old school cookie / session ID), pair it with an IP address. So in case of cookie theft, the attacker cannot use this cookie from his computer (IP address). If the IP changes (mobile / WiFi / ADSL / whatever), the legitimate user should log-in again, now storing two auth cookies. In case of another IP change, no problemo, one of the stored cookies will work. Of course limit validity of the cookie in time (lets, say, keep it valid only for a day or for a week or so).
deleted by creator
mobile devices change IP addresses all the time
I never noticed this. Yes, switch between mobile and WiFi, but this is only two addresses. In case of IPv4 this seems not problem. In case of IPv6, use /64 or /48 (or whatever is now recommended for residential end users) prefix instead of the entire 128bits. I’m not proposing to log-out the suer after IP change, I’m proposing multiple sessions to be accessible at the same time.
Mobile will often switch ip’s on tower handoffs. If you’re driving down the road or on a train, it’s nothing to change mobile ip addresses every 2 minutes.
Not in my experience. But OK, if this is the case, don’t use exact IPv4 address, lookup the routing database and use the sub-net. Or whatever. This is belt & suspenders style of defense in depth, just another layer of security if all others fail. Not core functionality.
I work for a mobile game company. Millions of clients. We deal with this a lot. You can’t even predict that they’ll stay in the same class A. I wouldn’t be surprised if they worked out a way to hand off ipv4 to 6 and vice-versa.
Then you have ISP’s and large work networks who send everyone out under the same NAT/PAT, 10’s of thousands of users all coming from one address.
IMO Providing a public service then trying to identify individuals by network without screwing someone over is a fools errand.
If you’re dipping logs and see one jackass doing something on x IP, you always have to go back and see how many ip’s that jackass is coming from and also how much viable traffic is coming from that ip.
JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
JavaScript needs access to the cookies, they are the data storage for a given site.
To protect them, the browser silos them to the individual site that created them, that’s why developers haven’t been able to easily load cross domain content for years, to mitigate XSS attacks.
The security relies on the premise that the only valid source of script is the originating domain.
The flaw here was allowing clients to add arbitrary script that was displayed to others.
You’re dead right that only the way to fix this is to do away with JavaScript access to certain things, but it will require a complete refactor of how cookies work.
I haven’t done any web dev in a few years, this might even be a solved problem by now and we are just seeing an old school implementation. 🤷this might even be a solved problem by now
Yes, it is called
HttpOnlyand is decided by the server who is sending the cookie to you in HTTP response header. I believe there are also HTTPS-only cookies that when received via HTTPS, cannot be used from HTTP, but I cannot find it right now.Secureis what you’re looking for.
You’re totally right. I just looked at my old jwt cookie and was susceptible to CSRF (cross site request forgery) by virtue of not having the SameSite flag being set. This has since been fixed, but it looks like there might still be changes pending as Javascript is currently able to read the cookie value (the HttpOnly flag is currently set to false, meaning that it is able to be accessed by the browser). While this isn’t a major risk, it does increase the attack surface a bit.
“never happen in a desktop/server application” Just don’t run mystery floppys and CDs right :)?
Or rather Electron apps.
How the attacker got those JWTs? JavaScript sent them to him?
If the “special emoji” code contained a script, it could simply read the cookie value and send it to a remote URL by using a XMLHttpRequest. I don’t think it has to be decoded to be used.
The web being a PoS? Well, you’re certainly right on the money there. Not only that, but it has also become the norm for web technologies to be used in creating desktop applications too.
So we’re bringing the security nightmare to desktop applications, and on top of that, it devours RAM like there’s no tomorrow, all for the sake of faster development.
To be fair, it’s not an issue with “the web.” A bug in JavaScript is not a problem for the web. It’s a problem for people who use JavaScript.
Good thing we all use randomly generated passwords for every account and always remember to change them every few months.
Passwords are safe, it’s temporary session tokens (already invalidated) that were exposed.
Good call.
However, that wouldn’t have prevented this from happening. Not even MFA can protect you in case of an exploit like this.
My password is 4, it was randomly generated with a dice. Every few months I roll the dice again.
I’m not perfect as far as password opsec goes, but I did indeed use a randomly generated unique password here.
Hell, even if your password safe is only what’s built into Firefox or Chrome, that’s still better than using easy-to-guess passwords or sharing passwords.
Good thing we all use randomly generated password
And randomly generated emails! (Thanks to SimpleLogin!)
Brave has a really good random password generator logged in and a password manager that is protected by a 24 character seed phrase
This is really good to see such transparency from admins
Thank the heavens the meme community stayed safe through this without my daily dose of cybersecurity memes idk how I would function ;)
How do I hide these images?
Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.
One wonders how much spez paid for that hack.
lots of
You guys really have my highest respect for spending so much time to keep this running, despite all the recent trouble and now even an attack.
Thank you very much <3 You guys are awesome and I really appreciate how publicly you deal with this.
I think this is a strong reminder: We shouldn’t put all our eggs in one basket. This will happen again. Unlike Reddit, we don’t need to concentrate all communities on one instance. We should all make an effort to spread out. Some other general use instances are:
Again, for those new, you can post content to any of these instances and interact with content from other instances at the same time, just like you can send an email from your Gmail account to your ProtonMail account.
This was an exploit in lemmy software and affected multiple instances though
The malicious Javascript was in a post however, and that post spread to all Lemmies that saw the post.
I’m pretty sure all web browsers who saw that post passed the JWT cookie to the hacker. The hacker was looking for admin accounts and got the Lemmy.world admin first.
Depends, a large portion of users browse Lemmy from the apps. The apps communicate with Lemmy through the API. Some might have basic web browsing features in them, but that doesn’t mean that they act exactly like a web browser would.
Register on 5 so you’re vulnerable on 5, got it.
Would be great to have identity independent from any particular servers…
That is currently not possible in the fediverse. Might be some day.
Yeah, I know. It is possible to implement it though. Posts and comments are cached in every federated server, the only thing you need is attribution to the author. Your key/certificate could be your identity.
It’s… slightly more complicated than that, but it should be possible.
This was one of my early fears… that an instance might get offline, attacked, whatever, and that all your user data might be lost. It appears that a feature like this might be inevitable. Or at least a feature that would let a user to have at least 1 copy of his account on another instance.
There is also the DB and storage growth to consider if something like this is implemeneted. If all instances hold all data from every other instance, than that might turn into a shit show pretty soon. You can get away with a certain number of users on only one server, but after a certain threshold, you have to scale… and that costs a lot of money, which is a no go for smaller instances and they will probably shut down if they’re forced to hold data for every user account in the fediverse.
If I had to propose something, I’d suggest writing a spec and implementing an API for Lemmy servers that would allow one to submit signed actions, like posts, comments, likes, display name, etc. Then people will write clients that will allow generation of certificates to be used to sign those messages + add the ability to export/back up the key to be used by other apps too.
If storage space ever becomes an issue, you could potentially shard communities. I’m more worried about the network traffic. I already suggested a routing algorithm that would spread the load between all federated instances and would scale like
O(log(N)). There was some interest, but it would be a long term project. There’s still a lot of performance that can be gained by simple optimisations.Yeah, there are more pressing issues at hand, a lot of bugs, a lot of new features that need to be implemented ASAP. Like user following, Lemmy is still lacking that.
I’ve thought about this, and had come to the conclusion that it would be too difficult for users to manage their private key. (Look at the flustercuck we’ve seen in cryptocurrency)
I wonder though, if we could choose to either manage our own private key OR let an instance manage it for us. This way, users could choose to give full control to an instance (as it is now) or to keep control themselves for extra security.
User data and public keys would still need to be stored (presumably on an instance server, though I wouldn’t be against separating this duty into a separate server which may or may not be hosted concurrently with an instance). But at least this way, we would have the option to keep control over our accounts even if the instance is hacked.
If I had to propose something, I’d suggest writing a spec and implementing an API for Lemmy servers that would allow one to submit signed actions, like posts, comments, likes, display name, etc. Then people will write clients that will allow generation of certificates to be used to sign those messages + add the ability to export/back up the key to be used by other apps too.
I think people who actually understood crypto used it correctly. It’s just that most people were there to speculate and gamble. And, tbh, I think crypto is here to stay.
Yup, I’d support an option to store the key on a server. Or just use an account tied to a server, just like now. Currently, servers sign all messages to other instances with their key, so all messages are tied to the identity of the instance!
100%. Also, while we’re at it, I just want to throw it out there that the front-end should also be split out imo. Let people mix and match backend/user/frontend according to a central api specification, so the best options for each can rise to the top, rather than relying on a single monolithic codebase.
I think it’s already split out!
Accounts aren’t just posts/comments/votes. You have the communities you’re following, settings, and other bits of private state (remembering what you’ve already read). You’d be in charge of preserving and lugging this data around.
Either store that data yourself or rely on your servers to store it and distribute it, just like they do today.
Did any of the other instances suffer this vulnerability? Or are they all running different code? i.e. different repositories/implementations of Lemmy
Is it possible to replicate to another community as a back up? Kind of a Master-slave DB?
Yes, there are tools in [email protected] that offer account migration. There is even one with a GUI. Download the settings of one account (including comms), save them in a file, transfer (upload) that file to another account via the GUI app.
Granted, the GUI method doesn’t do this auttomatically over a certain period of time, but the CLI tools can be configured to be run in a script that does this at specific time intervals, so your accounts will always stay in sync.
FYI: I had to clear my lemmy.world cookies in order to be able to successfully log back in.
(This was with Firefox)
(Edit: I also shift-clicked reload, which somebody pointed out does clean the cache for that page, so I also cleaned the cache).
Any truth to what I’ve heard this may have been done by a group we defederated with?
It’s not truth if it’s not confirmed.
Do we have any details on how Michelle’s account was compromised? Right now in the GitHub issue about the vulnerability they’re clueless about how the custom emoji exploit could be performed without first an already compromised admin account.
EDIT: yeah here’s how: https://github.com/LemmyNet/lemmy-ui/issues/1895#issuecomment-1629326627
You do NOT need an admin account to do that. Any normal user could have done that.
That was scary and exciting. Response seems competent and transparent. I ❤️ this place.
