Very, seems like great work.

Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.

I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.

In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:

I'm a modern web developer who used to be an old-school one.

User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.

Yeah - pretty much, though there are some mitigating factors.

Strictly speaking, it was the alt text for the emoji. Alt text is HTML, and rather than allow arbitrary HTML they allowed another language called Markdown. Markdown is "a plain text" language with human readable syntax specifically designed to be converted into HTML.

Markdown is the right format to use for emoji alt texts, but you do need to be careful of one thing - the original purpose of Markdown was to allow HTML content to be easier to write/read and it is a superset of the HTML language. So arbitrary HTML is valid markdown.

Virtually all modern Markdown parsers disable arbitrary HTML by default, but it's a behaviour which can be changed and that leaves potential for mistakes like this one here. Specifically the way Lemmy injected emojis with alt text into the Markdown content allowed arbitrary HTML.

This wasn't an obvious mistake - the issue over on Lemmy's issue tracker is titled "Possible XSS Attack" because they knew there was an XSS Attack somewhere and they weren't immediately sure if they had found it in the emoji system. Even now reading the diff to fix the vulnerability, it still isn't obvious to me what they did wrong.

It's fairly complex code and complexity is the enemy of security... but sometimes you have to do complex things. Back in the "old-school" days, nobody would have even attempted to write something as complicated as a federated social network...

JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS.

Yeah - the Lemmy developers made a mistake there. There are a few things they aren't doing right around cookies and JWT tokens.

Hopefully they fix it. I expect they will... It was already actively being discussed before this incident, and those discussions have been seen by a lot more people now.

How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.

There are several levels of isolation that could have blocked this:

1. Users should not be able to inject arbitrary HTML.
2. A flag on the page should be set telling the browser to ignore JavaScript in the body of the page - this is a relatively new feature in the web and disabled by default for obvious backwards compatibility reasons, but it should be set especially on a high value target like Lemmy, and I expect once it's been around a little longer browsers will enable it by default.
3. A flag should have been set to block JavaScript from contacting an unknown third party domain. Again, this isolation is a relatively new web feature and currently disabled by default.
4. As you say, JavaScript shouldn't be able to access the JWT token or the cookie. That's not a new feature in the web, it's just one Lemmy developers didn't take advantage of (I don't know why)
5. Even if all of those previous levels of isolation failed... there are things Lemmy should be doing to mitigate the attack. In particular instance admins have had to manually reset JWT tokens. Those tokens should have expired somehow on their own - possibly the moment the attacker tried to use them.

The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.

Yep - the modern best practice is for admins to manage the site via a completely different system. That adds considerable complexity and cost though, so it's rarely done unfortunately. But you know, Lemmy is open source... so if someone wants to take on that work they can do it.

I'll add one more - it should have taken less time to close the exploit... but given this is the first serious exploit I'll forgive that.

Ultimately several of failures contributed to this attack. I expect many of those failures will be corrected in the coming weeks, and that will make Lemmy far more secure than it is right now - so that next time there's a bug like the one in the Markdown parser it isn't able to cause so much disruption.

The good news is no harm was done, and a lot of people are going to learn some valuable lessons as a result of this incident. Ultimately the outcome is a positive one in my opinion.

Oh I forgot another line of defense / basic security mitigation. If a server produces an access token (such as JWT or any other old school cookie / session ID), pair it with an IP address. So in case of cookie theft, the attacker cannot use this cookie from his computer (IP address). If the IP changes (mobile / WiFi / ADSL / whatever), the legitimate user should log-in again, now storing two auth cookies. In case of another IP change, no problemo, one of the stored cookies will work. Of course limit validity of the cookie in time (lets, say, keep it valid only for a day or for a week or so).

JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.

JavaScript needs access to the cookies, they are the data storage for a given site.
To protect them, the browser silos them to the individual site that created them, that's why developers haven't been able to easily load cross domain content for years, to mitigate XSS attacks.
The security relies on the premise that the only valid source of script is the originating domain.
The flaw here was allowing clients to add arbitrary script that was displayed to others.
You're dead right that only the way to fix this is to do away with JavaScript access to certain things, but it will require a complete refactor of how cookies work.
I haven't done any web dev in a few years, this might even be a solved problem by now and we are just seeing an old school implementation. 🤷

Deleting hospital fees/debt is very dangerous... In many HUGE regions in the US there's only one hospital and if that hospital suddenly can't pay its bills it could shut down, leaving a whole lot of completely innocent people in a very sad, people-are-dying sort of state.

In fact, something like this already happened:

https://www.cbsnews.com/chicago/news/st-maragrets-health-central-illinois-hospital-closing/

Hospitals are special in that they're often evil organizations (not all though) that are some of the easiest to hack but also provide critical services to the most vulnerable. One should tread lightly. Political solutions are better (hack some politicians that are against healthcare reform instead).

Clearing credit card debt via hacking is nearly impossible but I agree it would be a much more ethical choice for hackers to target. I used to work for the credit card industry. My unique insider perspective, deep industry knowledge, and personal experience is here to let you know they suck. They are just as evil and unethical and unnecessary as everyone thinks they are! Seriously: If Visa, MasterCard, American Express, and all the lesser players suddenly disappeared the world would be a better place.

Before that can happen though people need a backup payment method that doesn't go through their systems and no: Cash won't work (there's not enough in circulation and it's dangerous to carry large amounts of it). The credit card companies know this threat exists which is why they lobbied Florida (and probably other states) to outlaw alternative, government-run forms of payment (e.g. central bank currency).

As soon as people have a widely accepted payment option that doesn't go through Visa and MasterCard's middlemen (e.g. First Data) then hackers can take their gloves off! Until then though... Let's keep the payment infrastructure working, OK? Thanks!

There's no limit to the amount of good deeds hackers can do though. So let's encourage that! For example, there's plenty of cartels and evil religious organizations (e.g. Taliban, ISIS, Mormon Church, Prosperity Gospel scam artists) that have plenty of money to spare and enormous attack surfaces 👍

Passwords are safe, it's temporary session tokens (already invalidated) that were exposed.

Good call.

However, that wouldn't have prevented this from happening. Not even MFA can protect you in case of an exploit like this.

Further 3rd party heads up -- for us nontech refugees:

If it looks like you are logged in, you may not be. I use Connect, and at your reminder, I clicked my acct and it says I was logged in. I tried to comment that Connect login was working, and my comment didn't show up.

I tried again, only to see an ”error: not logged in” message pop up.

Signed out, signed in again manually, and all is well.

So do a double check, Lemurs. Trust in your actions, not your eyes.

This was an exploit in lemmy software and affected multiple instances though

Good point, I'll update the post.

Thanks for the info. We're looking into this.