I’m just curious about the potential of running a small LLM locally on GrapheneOS, maybe some image processing too.

  • AlmightySnoo 🐢🇮🇱🇺🇦@lemmy.worldMEnglish
    3·
    1 year ago

    I think you may be referring to the Android NN API, which apparently should use the backend targeting Google Tensor SoCs when available. From this commit it seems it should be available on GrapheneOS too: https://github.com/GrapheneOS/device_google_gs201/commit/b60bfdd87550bf20f6cb73234a1a8ed2ecd61b69 (EdgeTPU is actually the ASIC differentiating Tensor SoCs from the rest by allowing fast and low-power inference from a hardware level).

    • j4k3@lemmy.worldOPEnglish
      1·
      1 year ago

      The second link is closer. I think it is technically the edge TPU that is used to handle ML stuff.

      It would take a higher level of accessibility for me to be able to engage with in practice. Like I need a hugging face type of high level accessibility to have a chance of getting it working in practice. I’m curious if anything like this exists. The available RAM probably limits anything really useful. It might be interesting to see what kind of edge processing could be mixed with an offline model running on a local server. I can connect to models over LAN already but my largest models are slow.