I’m looking at starting a service that involves hosting a lot of LLM models, which are often going to be 16GB+ (compressed). I did a bit of searching for cloud storage providers with cheap egress, and the cheapest I could find is $0.01 per GB, which would still be $0.16+ per download.
How do sites like Huggingface or CivitAI do it? Lots of VC funding?
R2 by cloudfare does not charge egress costs. It’s 0.015$/GB/month for storage. Read operations are 0.36$ per million.
I do have a hard time believing that they will remain this cheap though, but who knows.
Storj does it at 7 USD/TB. And there are providers that technically provide unlimited bandwidth, like Hetzner’s dedicated servers; they still have some abuse limits, but even working within the limits should make it much cheaper. This means custom engineering though.
If the files are not going to be changing much, then what is typically done is to use a CDN service (e.g. Cloudflare, Akamai, Fastly). The idea is you have an “origin” which could be any old server which serves your files over HTTP (even a VPS running nginx). The CDN is configured to proxy requests to the origin, building up a cache of the files it serves. The CDN can serve files from cache on their own (very large) infrastructure. See also What is a CDN?
So I got curious and wondered how HuggingFace hosts their files. It’s AWS CloudFront:
$ curl -LI https://huggingface.co/facebook/musicgen-large/resolve/main/state_dict.bin HTTP/2 302 content-type: text/plain; charset=utf-8 content-length: 1198 location: https://cdn-lfs.huggingface.co/repos/81/... date: Fri, 23 Jun 2023 07:09:39 GMT x-powered-by: huggingface-moon x-request-id: Root=1-64954533-1eb79eed4ea500203f6435cb access-control-allow-origin: https://huggingface.co vary: Origin, Accept access-control-expose-headers: X-Repo-Commit,X-Request-Id,X-Error-Code,X-Error-Message,ETag,Link,Accept-Ranges,Content-Range x-repo-commit: c19300a6b2b62d29b345ae9eb7b163278e65238a accept-ranges: bytes x-linked-size: 6514108999 x-linked-etag: "1f0cf17b5e65c5dd8ba71767371c377f174e4ce1db44bc4d6657825769f26ffd" x-cache: Miss from cloudfront via: 1.1 65c7d0c3355767ac8658c2122c8280b6.cloudfront.net (CloudFront) x-amz-cf-pop: SYD1-C1 x-amz-cf-id: XAU8X4yneUeudylCi_9MeAYZmISCr8OHiBcgjAGcBQT-edrBF6wGCA== HTTP/2 200 content-type: application/octet-stream content-length: 6514108999 date: Fri, 23 Jun 2023 07:06:20 GMT last-modified: Thu, 08 Jun 2023 19:05:02 GMT etag: "44ef1b51c0cc2200e29fed5cddbf8e27-408" x-amz-storage-class: INTELLIGENT_TIERING x-amz-server-side-encryption: AES256 x-amz-version-id: upVN9_QvGmQZMDWfVECqbytHgWjHTz4t content-disposition: attachment; filename*=UTF-8''state_dict.bin; filename="state_dict.bin"; accept-ranges: bytes server: AmazonS3 x-cache: Hit from cloudfront via: 1.1 dff3fc94ddb54b32b708edf2668b23d2.cloudfront.net (CloudFront) x-amz-cf-pop: SIN52-P1 x-amz-cf-id: JWTqKUDp8bVPD7Tt4DBySj8zbgT8G60sCA5BMC1cYF4vK9nx45f1Iw== age: 200 vary: Origin
Lots of VC funding?
Probably. They might have gotten additional discounts off of the advertized price by talking with sales and committing to the service for a year or other ways.
