Artificial Intelligence

Choosing a local AI model: open weights, licences and an evaluation plan

Check usage rights, runtime requirements and task performance before downloading model files, then build a small evaluation set.

Open weights and open source are different claims

Downloadable weights do not automatically imply that training data, training code and every usage right are open. Read the model card for the publisher, licence, intended use and limitations instead of treating open weights as a software licensing category.

A desktop runtime’s licence does not replace the model’s licence. When choosing a tool such as Ollama, assess the runtime and model file as separate components. Commercial integration, redistribution and use restrictions can differ. A free download does not resolve unclear permission.

Build ten examples from your own task

Define the job before relying on a general leaderboard. For an application support assistant, choose questions about installation, connection failures, unsupported platforms, file sizes and unknown features. Write an acceptance condition beside each. For an unknown feature, a correct answer may acknowledge missing information instead of inventing a capability.

Use the same questions and instructions across models. Record correctness, language clarity, latency and memory separately. More words do not necessarily mean a better answer: a fluent description of a nonexistent product feature is still a failure. This small evaluation supports your task-specific decision rather than a universal ranking.

Downloading and running locally

Choose the official runtime for your operating system. Record the model source and version, and check disk space before a large download. Quantisation variants can differ in size and behaviour. Start with a small example to validate the installation path.

Evaluate cloud features, update requests and external connections separately from local inference. Ollama offers local and cloud models, so check the selected mode. A model stored on a computer does not by itself establish that no prompts leave the device. Use sample data to verify the flow before providing sensitive inputs.

Before integrating the model into an application

Do not execute generated text directly as a command, file path or payment instruction. Validate input and output against the expected format in an application layer. A support assistant may explain a product without having unrestricted authority to delete accounts. Establish boundaries appropriate to the task.

Rerun the same ten examples when upgrading models. Compare with earlier outputs and preserve failures. A model that is adequate for short inputs in one language might behave differently with longer context or another language. This makes selection depend on observable product behaviour rather than brand popularity.

Sources & further reading

Sources checked: 6 November 2026. Workflows and examples are editorial suggestions; they are not product benchmarks.