Seeking feedback: how should lemm.ee move forward with external images? (related to frequent broken images)

Submitted ⁨⁨4⁩ ⁨weeks⁩ ago⁩ by ⁨sunaurus@lemm.ee⁩ to ⁨meta@lemm.ee⁩

Hey folks!

I am looking for feedback from active lemm.ee users on what you all value when it comes to images on Lemmy. I’ll go into a bit of detail about what our options are, and then I would ask you to voice your opinion about the issue in the comments.

First, some context for those who don’t know. Lemmy software can be configured to handle images in three different ways:

Store images locally - whenever an external image is posted somewhere, lemm.ee will download a permanent local copy. When you view posts, you are seeing our local copy of the image.
Proxy all images - similarly to the first option, lemm.ee will download a local copy of external images, however, this copy is temporary. It will be automatically deleted shortly after, and if users open the relevant post/comment again in the future, there will be another attempt to download a temporary copy at that point.
Pass through external images directly - lemm.ee never downloads any external images, users will always connect directly to the source servers to load the images.

There are pros and cons to each configuration.

Storing images locally

Benefits:

Your IP address is never leaked to external image hosts, as you never connect directly to the source server. External image hosts only see the IP address of the lemm.ee server.
External servers don’t become bottlenecks for opening lemm.ee posts. If an external server is slow, it won’t matter, because the image is always available locally

Downsides:

As time goes on, our storage will fill up with hundreds of gigabytes of useless images, most of which will never be viewed again after the relevant posts fall off the front page.
Many big external image hosts will rate limit bigger Lemmy servers, causing broken images when we fail to make a local copy.
Crucially: some people love to spend their time uploading illegal content to online servers. There are tools to try and filter out such content, but these are not perfect. The end result is that there is a high chance of some content like this inadvertently reaching lemm.ee storage and staying there permanently. This downside is why lemm.ee has not, and will not, use this particular configuration.

Proxying images

Benefits: In addition to the same benefits as exist for the permanent local storage, by only temporarily making local copies for the moment they are requested by our users, we free up a ton of storage & remove the risk of permanently storing illegal content on our servers.

Downsides: The key downside is that external rate limits hit us much harder, as we will be requesting external images far more often. This results in a lot of constant broken images on lemm.ee.

Passing through external images

Benefits:

Images are rarely broken, unless the source server goes down.
The images never touch our servers, removing a lot of risk with illegal content as well as with storage costs.

Downsides:

Our users lose a degree of privacy. Every external image that is loaded on your browser will result in the remote server getting a request directly from your computer to fetch that image - this is pretty much the same as you had visited that external server directly, which lets them log your IP address if they wish.
When remote servers are slow, it can slow down the entire page load in some cases.

Current situation

Initially, lemm.ee was using the third option of passing through images. Ever since support for option 2, image proxying, was implemented in Lemmy code, we immediately switched to that option, mainly for the privacy benefits. However, after many months, and being blocked by more and more external servers, it is clear that image proxying is seriously degrading the user experience on lemm.ee. We often end up with broken images, and our users have to deal with the results.

I still believe image proxying is a really valuable feature, but I am starting to believe it is a better fit for small instances which make much less requests to external servers.

As a result, I am now seriously considering switching back to the previous method of passing through external images.

This is where you come in - I would ask you as users to please let me know which do you value more: the privacy that you get from image proxying, or the better user experience you get from directly passing through images from their source. Please let me know in the comments how you feel. If I get enough feedback about people being against image proxying, then I will be switching it off for lemm.ee soon. Thanks for reading & sharing your thoughs, and I hope you have a great weekend!

source

Comments

Sort:hotnew top

xavier666@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I would prefer the option which allows lemm.ee to run in the most sustainable manner

source
homesnatch@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Storing permanently locally doesn’t sound like a good solution…

If you could adjust the length of time to keep cached/proxy’d images locally and increase it significantly, I’d think that would be the preferred solution.

source
thefartographer@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I say option 3. Sure, it’s annoying, but that’s our problem. Your problem is keeping the server operational, safe, and low-cost.

Considering the vote:content ratio, it looks like most users spend most of their time spectating. The spectators feed our egos and determine the trends in content by singular positive or negative votes. The spectator experience seems far more important to me and it should be the onus of the contributors to ensure their own privacy, just like they do in avoiding doxxing themselves via text.

Option 2 certainly follows in the vein of improved performance, but if real-world implementation is proving too unstable or creating too much overhead, then I say “fuck it, option 3 sounds great.”

source
Blaze@feddit.org ⁨4⁩ ⁨weeks⁩ ago
No opinion for the moment, but thank you for the very detailed post

source
invertedspear@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
While I appreciate you trying to make it so, my privacy isn’t your responsibility. Option 3 is the way to go to keep your costs down, which is the long-term best solution.

It wasn’t one of the options, but from a user perspective a hybrid solution is best. Making a local copy that has a 24-48 hour cache keeps your storage down but still gives us the benefit of option 1. But it sounds like this would require some changes to how the server software works, which would be cool, but again you shouldn’t feel compelled to attempt.

Keep your costs and stress low wherever possible and thank you for everything you do.

source
barsoap@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
2+3: Try to fetch image, if you get it proxy it, if your storage gets full use LRU eviction, once evicted or some amount of time has passed delete it and don’t fetch again, ever. Fall back to pure 3 if there’s ever any issues with anything, including you not particularly feeling like implementing smart caching: Our referrer privacy is not your responsibility.

source
Sotuanduso@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I’d say option 3. Personally, I don’t care if random websites get my IP among a list of hundreds of others, and if someone wants to keep their IP hidden from strangers, they should be using a VPN before browsing the net anyways. It’d also be nice not to have to open another instance when I come to a post with a broken image that I want to see, but that’s not hugely important to me.

If it were an instance specifically for privacy enthusiasts, that’d be a different story, but this is a general-purpose instance, and option 3 seems to be what’s best for both general users and the server itself.

source
- original_reader@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
  Im trying to avoid saying “this”. Still, your post reflects my thoughts exactly.
  
  3 it is.
  
  source
TachyonTele@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I’m on a lot, and I scroll from All/Hot. I rarely see broken images. They do pop up, but not enough to ever bother me. The only option I’d avoid is method 1, because of that image debacle a few months ago. Regarding methods 2 and 3, they both seemed to work fine. I leave it to smarter minds than myself.

Good work on next, btw. Enjoy your weekend and thank you for everything you do!

source
shakcked@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Option 3 is the only one that seems sustainable long term. Donations will NEVER keep up with user growth, thus storage costs will balloon out of control.

Completely avoiding any chance of illegal content touching the servers should immediately have everyone agreeing on this option. I doubt anyone here is willing to foot legal bills and as such even minor legal actions would be the end of this instance.

Privacy is nice but ip logging is the simplest form to “protect” against with even a free VPN. If those claiming privacy concerns here aren’t already using a VPN and are depending purely on lemme.ee’s proxy then their internet hygiene needs an update.

As for usability, the image being deleted from external provider presents the same issue to the user between option 2 and 3. The cache from option 2 will inventually get cleared and it’ll fail to pull a fresh copy if deleted from the external hosts.

source
Crumbgrabber@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I like 3 also. if you can’t get a good user experience privacy matters less because there are no users, and anyone can spin up a super privacy enhanced instance if they want.

I was thinking however that a super stand alone image server would be a great thing for Lemmy, and pixelfed and the Fediverse in general.

Imgur blew up and was the go to for image hosting on Reddit for years, until Reddit realized it was leaking traffic and users to them and started their own. But hosting images has a lot of potential headaches like copyright violations and big corps suing you into oblivion, in addition to inadvertently hosting illegal stuff. The Fediverse will need some good image hosting servers and video hosting servers as part of the plan in the long run though.

source
jsqribe@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Option 3, agree with others, there are other privacy focused instances.

source
Jabbl@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I seem to agree with most other users in this thread.

Sure, the privacy aspect is nice; most people already share too much information with third parties unwillingly, but I think a good user experience should be prioritised. If option 3 is more likely to provide that, I would choose going that way.

Most people, especially those migrating from other sites, probably care more about the images loading than the privacy involved with proxying them.

It would maybe be nice to have an option to not load external resources automatically, or a black/whitelist for certain sites, if such an option doesn’t already exist.

source
Navarian@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I’m in favour of Option 3, privacy concerns considered.

User experience is big for me here, the broken images are something of a frustration that I’ve been dealing with for a while now, so the option to combat that is a clear winner for me.

Also, I want to thank you for coming to us for feedback, yet another reason I’m glad I decided to settle here on Lemm.ee.

source
MyOpinion@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Option 3.

source
mediocreme_ow@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Option 3.

Privacy concerns aside, which I am willing to bear, Option 3 is the most sustainable option.

Looking at our status page, the projected monthly expenses is greater than the revenues. If passing through external images allows us to reduce operational costs and ensure lemm.ee’s sustainability despite the loss of a degree of privacy, then it’s a tradeoff I’m willing to make.

Thanks, admins and mods, for everything that you do!

source
GrayBackgroundMusic@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Whatever is most sustainable for you.

source
perishthethought@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Definitely OK with either option 2 or 3. And I trust @sunarus to choose what’s best for this instance.

source
Rexios@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Have you thought about solving this issue in the front end? The client I’m using (Mlem) implemented a feature to directly access the image if the proxy fails. This feature can either be triggered automatically or by pressing a button on the failed image. This allows users the benefit of the proxy while also having the option to give up their IP if they want to see a broken image.

source
- kyle@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
  If this is feasible, it sounds like an elegant solution. Does it work for the various mobile apps, or would each app need to do it on their end?
  
  source
  - Rexios@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
    If there was a Lemmy account setting to automatically bypass the proxy on failure then it might not require any front end work, but the manual bypass button would definitely need to be implemented in each client
    
    source
- ArchRecord@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
  This is something I think would be the best solution. It seems like the best possible tradeoff between user privacy, and actual effectiveness.
  
  source
simple@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Thanks for your hard work as always.

I’m in favor of moving away from proxying. Too many images break and proxying in general is very wasteful, having to download images from potentially small servers constantly would definitely get you ratelimited.

Passing through external images is OK. Many people often post external links anyways to sites like imgur and catbox because of the file size limits.

I think the end goal would always to store images locally though. Don’t large instances like Lemmy World and huge Mastodon instances work this way? How do they manage the risk?

source
uiiiq@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Although I value privacy, I value sustainability more. Choose an option you feel most comfortable with, option which puts the least burden on your shoulders.

source
RagingHungryPanda@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I would do option 3, both as a user and if I were to make my own service. I’d let a dedicated image service do it. I’m not worried about IP leakage and if I am I can use a VPN. If I want to be that concerned about it, I think the responsibility is on me, not on the service unless they promise that level of privacy.

I vote for what gives the best experience for users and makes running the service easier and cost effectivem

source
dditty@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I think it’s important for us to be mindful of content retention for posterity’s sake if we want Lemmy to compete with Reddit long-term. If possible, I’d hope we can avoid dead image links like we see with old forums and photobucket pics, for example.

source
flashgnash@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I believe just passing through external images is the way to go, it’s always the one I opt for it I can

Hosting those images is gonna get expensive and that kinda sucks for a donation run platform when that money could be far better spent elsewhere

I think also using external image sources is more in line with the idea of decentralisation, Lemmy isn’t an image host it’s a link aggregator and forum - I believe most image hosting sites will be far better at loading images quickly than Lemmy’s implementation could ever be

source
moriarty@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Option 2.5: Option 2 by default, by if the external server rejects our request (rate-limiting), we fall back to Option 3 by HTTP 302ing

source
scsi@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Option 3

Reasoning:

Upside 2: 100% best for lemm.ee health; lowest legal risk, lower cost to run.

Downside 1: I think it comes down to what lemm.ee is trying to provide as a user experience; in my use and expectation, it’s not for masking my IP, making me anonymous or similar. It’s for reading and interacting with people, looking at memes and reading lots of news stories. I have no expectation my IP is masked from remote sites - I open all external news links in a Private tab anyways (to stop cookies and other junk) so they’re already getting my IP anyway. “why should images be any different, really?” There are other lemmy instances out there catering to extreme privacy.

Downside 2: this could be, should be, whatever handles by better page loading threading in the code; the content surrounding an image is just HTML, the load of the image is a secondary task. If the rendering of the view of the page is reliant upon 200 OK image loads, that feels like a deficiency in design and it needs to be async threaded to “lazy load” and not block.

At a high level, many other solutions - Mastodon, even Nostr webapps and phone apps which is all about being anonymous for some folks - do direct content load from the source and do not proxy loading. The switch back to option 3 falls in line with what every other generic service/solution does in the social web space.
source
- nmtake@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
  
  “why should images be any different, really?”
  
  Without proxying or other means, images are automatically loaded. A user may post tracking images to collect IP addresses or to harras a specific user. It’s quite different from manually opening the external sites (that aren’t autotmatically loaded) to read stories.
  
  source
LedgeDrop@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Wow, thanks for the full transparency. You are awesome!

My opinion would be option 2 (proxy requests) , but with a higher cache TTL or simple a LRU (Least Recently Used) Cache.

If you’re getting throttled, it could be mitigated by increasing the cache retention period (or improving the cache hits).

Another improvement : Would it be possible to change the proxy, so that if the proxied requests are throttled, it simply sends the user a http-302 to the origin (instead of a broken image)?

Regarding option 1 (full cache) : I greatly appreciate your desire to hide/protect your users ip, but it is outside the scope of what I expect from a Lemmy server. Maybe you could market and upsell this increased privacy as a subscription based feature. However, if I want privacy - I’ll use a VPN.

Regarding option 3 (User fetches content from origin) : From a users perspective, I really don’t want my Lemmy experience to be based on hitting a bunch of (potentially) unreliable services. When I, as a lemm.ee User, request a post from Lemmy.world (for example), lemm.ee will proxy and cache that post and the comments. This is the distributed nature of Lemmy (as far as I understand). Why restrict this caching to just posts/threads/comments and not include images (which, let’s face it, are as meaningful as pure text - especially wrt memes).

source
- JakenVeina@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
  I’ll wager “no” to your question. That sounds like something the Lemmy codebase itself would have to implement, not smething that’s just configurable.
  
  source
  - LedgeDrop@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
    It’s sad, but I think you’re right.
    
    I assumed/hoped that Lemmy’s architecture was more decoupled.
    
    According to the ChangeLog, it hints that the image reverse proxy is built-in, maybe using Pict-rs.
    
    Which certainly reeks of Not Invented Here Syndrome, as image uploading/storing, reverse proxies, and caching is a well understood problem.
    
    source
ramble81@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I’d lean towards option 3. I view this site as an aggregator and I see plenty of broken images when scrollling through here. It’s not your job to host or proxy those images and working in infrastructure it lends to a lot lighter weight instance that’s easier for you to manage without having to worry about disk, or even worse, bandwidth costs.

The only way I would see option 2 being more beneficial is if it was truly a temporary cache solution where the image gets pulled in 1 time, served out to all lemm.ee users for a period of time (say 24 hours). This would reduce the chance that you end up rate limited, while allowing users to still see the image served via lemm.ee

The proxy on every request solution just seems poorly implemented to me for the reasons you say.

source
narc0tic_bird@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Can’t you store them in a cache that keeps images that have been accessed in the last 48 hours (or whatever) and deletes others? Should someone request these images after that, cache them again for 48 hours.

source
neme@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
I’d prefer proxying

source
fossphi@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Images have been a bit problematic for me lately, for sure. If storing them locally is not a solid option, the question I would have is how much of the other requests are proxied? As in, what other stuff apart from images/media is not being proxied? If the clients are leaking IPs anyway, maybe it’s okay to have them download the images, too. But if the server is proxying everything else then having some sort of a cache might not be that bad an idea

source

-> View More Comments