AI tools I use (and why)

Which AI tools I use and why. Last update: 24-09-26

• I use different models for different tasks.
  • I use GPT 4o-mini as the default LLM (cheap, good).
  • I use Deepseek v2.5 (cheap, but slow) and Claude Sonnet 3.5 (best, but expensive) for coding (via aider)
  • I am currently test running to replace Deepseek with Qwen2.5 Coder (10x more expensive but a lot faster!) and Sonnet 3.5 with Haiku 3.5 (still better than Qwen, and a lot less expensive than Sonnett 3.5)
  • I use Gemini Flash 1.5 for OCR and alternative to GPT 4o-mini (sometimes better, sometimes worse)
• I use aider for pair programming with LLMs
• I use the Chatkit app for chat-type LLM interaction with all sort of LLMs
• I use the llm Python tool to test and use LLM APIs (Link).
  • (+) It supports all models that I use and test frequently.
  • (+) It logs previous prompts.
  • (+) It also has support for embeddings and topic clustering.
• I use the paid Perplexity Pro that features Claude 3.5 Sonnet
  • (+) I like that I can switch between models in a chat format.
  • (+) I like how it integrates web search.
• I have no established tool for evaluating LLMs programmatically and systematically, I'm considering ChainForge, AlignEval and PromptFoo at the moment