Groq support & cases for Custom base URL
Any plans to support Groq? https://console.groq.com/docs/openai Groq's API is implemented the same as the OpenAI API spec, so I would think it'd be relatively straightforward to add them as a provider
I want to use them as their cost per million tokens is very competitive, but I'll need a fallback provider (OpenAI or something else) if they fail, which is where I'm hoping AI gateway can come in š
GroqCloud
Experience the fastest inference in the world
15 Replies
@STRRL @Ennergizer what were you mentioning about the baseURL? If we put the use cases / etc in here maybe the team can feedback
for sure. here is the baseurl setup for openai compatible API, explained from groq's documentation: https://console.groq.com/docs/openai#url
GroqCloud
Experience the fastest inference in the world
in other words, we could call gpt-3.5-turbo or gpt-4 on openai by sending http request to https://api.openai.com/v1/chat/completions. And on groq, we could use the "openai comtaible API" with mixtral and llama 2/llama 3 by sending http reqeust to https://api.groq.com/openai/v1/chat/completions.
the base url is https://api.groq.com/openai/v1, for groq
that already becomes defacto for nearly all the openai compatible APIs. also there is an selfhosted api gateway called litellm, which could work as an adaptor between lots of different LLM API providers and "OpenAI compatible apis".
https://github.com/BerriAI/litellm
GitHub
GitHub - BerriAI/litellm: Call all LLM APIs using the OpenAI format...
Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs) - BerriAI/litellm
BTW if cloudflare AI Gateway could also works as "litellm", always provide OpenAI Compatible API for different upstream, that would be extramlly helpful.
I do know they weren't trying to let people just proxy ANY url
Bc in that case you can just throw in any url and gateway will proxy it
not sure that was their intended path forward with it
I think in the future, there would be more OSS projects or AI API Gateway(like Kong AI Gateway, our team is alaway exploring on it https://docs.konghq.com/gateway/latest/get-started/ai-gateway/#ai-provider-proxy ) services make the role of "API Adaptor", the echosystem is growing.
Kong Docs
AI Gateway - Kong Gateway | Kong Docs
Documentation for Kong, the Cloud Connectivity Company for APIs and Microservices.
We still expect Cloudflare would have product/service could do that.
ā¤ļø
yea OSS is an option too for gateway
I think there are other higher level features they are looking to prioritize in workers AI. I'm still waiting for persistent real time logging on gateway too
I don't want gateway to get lost in the shuffle tho
observability and analytics would also be great!
I'm with you tho would be nice to get groq or other endpoints on there. I'm doubtful they'd just allow ANy and every url to be passed through baseURL
It's been asked before and doesn't seem like their preferred path
I know. There would be some concerns about allowing using any api providers...
Ok fair re: not allowing any url. Still I think adding groq would be straightforward given the OpenAI compat, hoping they do it š
michelle ā Today at 11:11 AM
added groq support to roadmap
good to know!
Unknown Userā¢7mo ago
Message Not Public
Sign In & Join Server To View