What to check in an AI model before you buy it

An AI model can be an expensive purchase, so test it before you pay. These are the checks to make.

Technical evaluation

Start with the architecture (transformer, CNN, RNN or something else) and whether it suits your use case.

Ask about the training data:

• Dataset size and diversity
• Data sources
• Steps taken to reduce bias
• Data licensing, and the model's own license

Ask for performance metrics:

• Accuracy, precision, recall and F1 score
• A confusion matrix
• Benchmark comparisons
• Inference latency

Check the size and requirements:

• Model file size
• GPU and memory requirements
• Inference framework (PyTorch, TensorFlow, ONNX)
• Quantization options

Finally, can you fine-tune it on your own data, and is the training pipeline included?

Checking the model with escrow

With escrow, you can:

1. Receive the model files in the deal chat
2. Run inference tests on your own data
3. Check performance against the claimed metrics
4. Review the documentation and code quality
5. Confirm receipt only when you're satisfied

If the model doesn't perform as advertised, open a dispute before you confirm. A human moderator decides; EPICSWP's assistant, Aegis, can summarize the case for them.

Pricing guide

Prices depend on what you're buying: a pre-trained model, a fine-tuned one or a full system with an app. Compare with the cost of fine-tuning an open model, and check the license allows commercial use.