Kaity A (@supakaity)
blahaj.zone
**State of the ~~nation~~ instance** So over the last month our instances have grown substantially and while Ada and I are privileged enough to be able to cover the cost of the small instance we initially envisioned, what it has become is somewhat bigger than that. The Lemmy side of things is currently at 6,500 users and is hosting a larger meme community which involves a lot of image downloads and a fairly substantial bandwidth cost for serving those downloads. As we are hosting on AWS, we pay a premium for all the resources we use and the results of this are that the hosting this month has presented us with a surprise. Our monthly bill has jumped from about $1,000 to about $3,000 a month. (See attached image) Unfortunately while we can manage this as a once off, I can't afford to continue in this fashion, month after month, so we will have to investigate solutions to help reduce operating costs. This will involve moving services off AWS to less premium providers, and the downtime involved while these migrations take place and we retune everything. I apologize in advance for the interruption while we we rejig everything through this process. Thankyou all for understanding. Please let @ada or myself know of any issues that you notice, as glitches may occur when moving things around. $[x2 :Blobhaj_Plead_2:] (📎1)

Discussion of the current situation with the Blåhaj instances, and upcoming maintenance.

  • audiomodder@lemmy.blahaj.zoneEnglish
    3·
    1 year ago

    I love that people in the comments here are like “have you tried” and “maybe we could”…as if I expected a server full of trans folks to NOT have a bunch of techie people on it :)

  • 𝙣𝙪𝙠𝙚@yah.lolEnglish
    2·
    1 year ago

    A few friendly recommendations that may help:

    1. Remove the AWS load balancer. Your web application isn’t horizontally scaled so you don’t need that extra cost. Obviously you’ll still have to pay for bandwidth moving all those images but you remove the ELB hourly usage cost from the equation. If you’re using it for a reverse proxy, simply install a nginx container on the lemmy box. If really want a load balancer, you’ll save money by hosting haproxy on a regular server.

    2. Get off Amazon RDS. You’ll save money by hosting your database on a regular server than using RDS. You’re moving from a fully managed solution to one you have to manage your own. This means managing your own backups and replication if you want HA failover.

    3. Switch to object storage for pictrs. The cost is much lower.

    However, you’re right that Amazon prices are high and you’ll likely find cheaper alternatives elsewhere.

    • SomeRandomWords@lemmy.blahaj.zoneEnglish
      1·
      1 year ago

      I’m not sure I agree with moving from managed to unmanaged for RDS, unless you’re very familiar with databases and keeping them online, RDS (and other managed DB providers) are honestly one of the places I recommend spending more on.

      The rest of your points are spot on though.

  • Gormadt@lemmy.blahaj.zoneEnglish
    0·
    1 year ago

    It seems like the images are playing a big role so perhaps having images being offloaded to another service (like catbox.moe) or converting images to WebP with a healthy amount of compression would help.

    Basically decrease the amount of storage you folks are having to pay for.

    Or auto delete pictures after a week or 2 and add a disclaimer about it to when pictures are being uploaded.

    Maybe also a file size limit. Nothing over 10MB or whatever.

    I hope this whole thing can be sorted out I dig this instance.

      • Shit@sh.itjust.worksEnglish
        0·
        1 year ago

        That’s crazy is there a breakdown of the bandwidth? Like is it federation, images or ddos using so much bandwidth?

        I know you are looking/found a solution but if the main culprit is images would it be possible to offload it to a few servers? I feel like running multiple image server instances would be cheaper for bandwidth. Oracle cloud offers 10tb outbound a month for free per region/month on the free teir. Or is an unmetered host viable? From my understanding Lemmy is pretty light in terms of CPU/memory or does that fall apart at scale?

        If you don’t mind and it’s not too much trouble would you be willing to share stuff like load average/disk use?

        Edit I looked this over again. Not an endorsement for giving oracle or Larry Ellison money, they are awful. I was just was trying to say the free stuff is generous. You can use 40tb outbound without paying on the free resources. As always aggressive support for you Ada! Thanks for the transparency in your community!

        • Ada@lemmy.blahaj.zoneMEnglish
          1·
          1 year ago

          It’s most images from lemmy. We’ve set up a caching server for pictrs, and that has got the bandwidth part under control.

          Next step, get everything else off of AWS :)