Local Language Model Instead of Cloud Service?
By Pasi Karhu, CTO
Large cloud-based language models are establishing their place as enablers of new, unprecedented AI solutions that we couldn’t have even dreamed of just five years ago. As an alternative to large cloud models, there are also many open language models of various sizes and capabilities, which can be used on a company’s local server or individual workstation. Here I discuss the data security, EU restrictions, intelligence of the models, their Finnish language proficiency, and the computational requirements related to language models. I also give some guidance to how to try out a local language model on your own computer.
when is there a Need for a Local Language Model?
In cases where the absolute need for data confidentiality prevents the use of all cloud services, a solution implemented with one’s own computing resources remains the only option for leveraging the intelligence of language models. Candidates for such local implementations include, for example, companies’ product development projects and the document management of defence forces and the defence industry. Another compelling need is working outside the reach of internet connections, such as working in the archipelago, at sea, in the mountains or wilderness, and in places prone to power outages.
IF Data Usage for Training Models is an absolute no-go
Regarding cloud-based language models, it’s not just a question of general data security, but also whether the data processed through the language model is also used to develop new language models. This can be a problem since any data used to train a language model can resurface as a direct response to another user’s question at some point in the future. Of course, many well-known service providers in the field state that they do not use the data their customers feed into their language models to train new models. However, given that they have practically disregarded copyrights and vacuumed up all the data they could find from the Internet for their use, a seed of doubt remains in this matter as well. To be absolutely sure that data is not used for training the language model must be deployed locally.
EU Restrictions and other geopolitical views
Apart from the French Mistral AI, all significant providers of cloud-based language models come from outside the EU, which imposes its own restrictions, for example, regarding GDPR. OpenAI’s language models are, of course, available via Microsoft Azure from servers located within Europe, as are Google’s language models through their own Vertex AI. In the current fluctuating climate of the US administration’s foreign relations, as well as the development and interpretation of legislation, even these services may cause worry among the most cautious companies; especially since US language model providers seem to be working closely with the unpredictable current administration.
Are Open Models Intelligent enough?
The release of OpenAI’s GPT-4 language model in the spring of 2023 was a real kick-off for generative AI-powered systems usable in business operations. However, its use was only possible as a cloud service. Around the same time, however, smaller, open language models began to appear on the market with licenses allowing their use for almost any business operation on companies own servers. Initially, these models were not very intelligent and thus not suitable for most business tasks. Now, two years later, the best open models are, however, more intelligent than the original GPT-4. As language model utilization technologies have developed and the prices of computing capacity have decreased, even smaller language models are already usable in business operations. They can also be further trained for specific company tasks, making them better for that specific task than even the largest cloud models.
even Chinese Language Models are safe when used locally
During the early part of the year, especially Chinese open language models have gained publicity. They are indeed very competitive, but are they safe? What about US open models – can we trust that they won’t send company data to the United States? It’s good to know that a language model is not a computer program, so it doesn’t do anything independently. A language model can be thought of as an intelligent data repository from which the controlling application program (AI program) gets its intelligence. Thus, for example, ChatGPT or Copilot are not language models, but application programs that use underlying language models as their engines. A Chinese language model, when used locally from one’s own workstation or server, is just as secure as any other language model. The security of the application relies on the logic of the application using the language model as an aid. That logic is typically created by the system provider, not the language model developer.
Challenges of the Finnish Language
With English-language documents and when conversing in English, even very small language models that run on a regular office computer can be useful, because the majority of their training material has been in English. If a language model application needs to process Finnish documents or if there’s a need to converse in Finnish, the very smallest models do not have adequate skills. However, with larger open models on one’s own server, the Finnish language does not pose a problem. Models that understand and produce Finnish reasonably well can also be used on powerful workstations or gaming laptops, while typical office computers do not yet have the necessary computing power.
so How Powerful Does the Computer Need to Be?
Small models running on a relatively new office computer can, for example, create good summaries of text in English or correct spelling and style errors in English written eg. by a Finn. However, for Finnish language needs or more demanding reasoning, ordinary office computers are not sufficient. For these, a gaming laptop costing around a thousand euros with an 8GB Nvidia graphics card is the most affordable option. Gaming computers’ fast graphics are relevant to AI and language models due to the fact that both utilize similar very fast vector and matrix calculations. Powerful Mac computers have a CPU suitable for using language models without the need for Nvidia.
The best open language models, due to their very large size and computational power requirements, can only be used on very powerful workstations or servers. The price range for these is wide, starting from under ten thousand euros for a single-user high-performance workstation and extending to hundreds of thousands of euros for the most demanding multi-user enterprise server solutions that use specialized AI processors from AMD or Nvidia. With more money, you get faster computation, but in terms of intelligence itself, the improvements become marginal as the price increases.
Try it Out on Your Own Computer
You can easily try out language models on your own computer with the free LM Studio or Ollama software, for which a vast array of open models of different sizes are available. You can find good instructions on YouTube for installing either one (e.g. Local AI Made SIMPLE: LM Studio & Ollama + Web UI – How To Use) and downloading language models. The smallest models are suitable for trying out on a regular office computer, such as Google Gemma3:1b or the smallest model of DeepSeek, deepseek-r1:1.5b, which caused a stir earlier this year and even swayed stock prices. A machine with at least an 8GB graphics card runs for example Gemma3:13b that already has a good Finnish capability. It is also multimodal, understanding images in addition to text.
Figure 1. The upper part of the image displays an example of a locally operated 12 billion parameter gemma3 language model. The model understands Finnish and is capable of logical reasoning. The lower part of the image displays an example of a notably smaller 1 billion parameter model that can be operated locally on an ordinary office-use laptop. This model’s Finnish and reasoning skills are lower.
OTher aspects to consider when planning a Locally Operated Application
Ollama only offers simple chat functionality, and it cannot, for example, process documents. For more specialized AI needs, suitable other software is required. In principle, all language model-based applications where the language model can be changed can be used locally. However, it should be noted that many applications use other cloud services in addition to the language model, so simply switching the language model to a local one is not enough to guarantee data security or offline use. Information on whether their application also works without cloud services can be obtained from the application developer.
At Ai4Value, we pay special attention to data security issues, and therefore we can offer, for example, our intelligent document management software AIDA (AI Document Assistant) also in a completely cloud service-free version for both workstations and servers.
We also design and implement new AI-powered on-premises software solutions for your company’s use – with or without a language model.
Pasi Karhu, CTO
“I work as CTO at Ai4Value since its inception in 2018. However, my acquaintance with AI goes back much further: already in the 1990’s I worked with machine learning algorithms and taught a course on artificial neural networks at Helsinki University. Today, I particularly enjoy working with large language models, but my colleagues know they can approach me with a wide variety of AI questions. AI is a fun topic to work on, as its fast development regularly pushes even the most experienced IT guy to new grounds.”