LLM

Today I had to rate limit a user on SHARCNET’s internal LLM service for the first time. In the last 24 hours they had sent about 800,000 requests, and every single one was under 500 tokens. To be clear, they didn’t do anything wrong. The service is set up to let them do it. They had a pile of small work and a working endpoint, so they wrote some sort of loop. I probably would have done the same. But a loop that sends hundreds of thousands of tiny prompts is a batch job, and this was a really good reminder of why batch clusters are still super important. ...