In a previous post, I discussed setting up a local environment to run language models on your own machine (Read the original post).
Well, I have to admit today that I’ve concluded that while LMStudio has its merits, it doesn’t fully align with my principles. Consequently, I’ve decided to transition to Ollama and refresh my setup to embrace the recent hype surrounding the Chinese model trained with synthetic data based on ChatGPT responses. (FTW: Work smarter, not harder! :D)

Installing Ollama

Installing Ollama is straightforward. While compiling the project locally might be challenging for some, pre-built release builds are available on the official website: ollama.com. Once downloaded and installed, you can use the command-line interface (CLI) to download and run models easily.

Available DeepSeek R1 Models

Here is a table summarizing the currently available DeepSeek R1 models:

Selecting the Right Model

Selecting the appropriate model depends on your hardware. Here’s a table to help you decide:

Notes:

• The VRAM requirements are approximate and can vary based on specific configurations and quantization techniques.

• Quantization methods can reduce VRAM usage. For instance, a 1.58-bit quantized version of the DeepSeek-R1 model can fit into 160 GB of VRAM, allowing it to run on two NVIDIA H100 80GB GPUs. (unsloth.ai)

• For CPU-based inference without a GPU, it’s possible to run certain quantized versions of the model with as little as 20 GB of RAM, though performance may be slower. (unsloth.ai)

When selecting a model, ensure that your hardware meets the necessary requirements to achieve optimal performance.

Downloading and Running the Model

To download and run a model using Ollama, follow these steps:

1. Start Ollama: After installation, Ollama will run in the background without displaying any visible interface.

2. Download the Model: Open a terminal and execute the following command to download the desired model:

   ollama pull deepseek-r1:7b

   This command will download the latest version of the deepseek-r1:7b model.

   

3. Serve the Model: Once downloaded, run the following command to expose the model on your local machine:

   ollama serve

   By default, the model will be accessible at http://localhost:11434. Additionally, after pulling a new model,the server will automatically restart to apply the changes, meaning that it will start automatically the first time we pull!

Models I Downloaded

I also downloaded a few additional models to expand my local setup. Here’s the list of models I have installed on Ollama:

ollama ls

Integrating with Continue Extension

To integrate the model with your development environment, you can use the Continue extension. After installation, update the config.json file to include the models you’ve downloaded.

Path to config.json

• macOS and Linux: ~/.continue/config.json
• Windows: %USERPROFILE%\.continue\config.json

Sample config.json

Here’s an example of what the configuration file might look like:

{
  "allowAnonymousTelemetry": false,
  "models": [
    {
      "title": "DeepSeek-R1 7B",
      "provider": "ollama",
      "model": "deepseek-r1:7b"
    },
    {
      "title": "LLAMA 3.2B",
      "provider": "ollama",
      "model": "llama3.2:3b"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen2.5-Coder 3B",
    "provider": "ollama",
    "model": "qwen2.5-coder:3b"
  },
  "embeddingsProvider": {
    "provider": "ollama",
    "model": "nomic-embed-text"
  },
  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    },
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "folder",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    }
  ],
  "slashCommands": [
    {
      "name": "share",
      "description": "Export the current chat session to markdown"
    },
    {
      "name": "cmd",
      "description": "Generate a shell command"
    },
    {
      "name": "commit",
      "description": "Generate a git commit message"
    }
  ]
}

Save the file, and you’re ready to generate code!

Note: If you really care like me, don’t forget to disable the Anonymous Telemetry (For additional details: docs.continue.dev); also it’s worth nothing that Ollama adds itself to the startup processes by default… if you prefer to prevent this behaviour, make sure to disable it.

Happy hacking!

Contacts

For questions or suggestions, contact: noc@balzabu.io.

Updated on 2025-01-29

I stopped using LM Studio and switched over Ollama. You can read the shiny and brand new post here: Ollama + Continue: Open-Source is Always Better!

The realm of technology is constantly in the pursuit of innovation. Among the latest advancements, LM (Language Model) tools have revolutionized programming, making tasks seemingly effortless. But why shell out for them when these capabilities can be accessed for free? Discover LM Studio and Continue, which offer a local solution with unparalleled advantages.

The Advantages of Local LM Models

Embracing a local LM model comes with a host of benefits:

• Cost-free: No subscription fees; use it at no extra cost.
• Complete Offline Access: Work seamlessly even without an internet connection.
• Full Prompt Control: Customize prompts to tailor results to your exact needs.

Of course, it’s essential to weigh the drawbacks alongside the perks:

• Hardware Requirements: Consider the computing power needed to run these models effectively.
• Model Variability: Not all models are created in the same way; some may lack the sophistication of their proprietary counterparts.

LM Studio Configuration

Configuring LM Studio is a breeze.
Begin by downloading a preferred build from the official LM Studio website lmstudio.ai.
After installation, launch the application manually to access the Home screen.

Utilize the search bar at the center of the Home screen to locate desired models.
To make it all easier, Meta’s Code LLama model is recommended.

Searching for Code LLama will yield various results, as LM Studio can search and download available models from platforms like Hugging Face.

Each model offers different quantizations to compress its size.
Always verify model details on Hugging Face before selecting based on your hardware.
In this case, we’ll use the Code LLama 13B model.

Once downloaded, navigate to the AI Chat section of LM Studio to load the model. Additionally, LM Studio can be configured to use it as a server.

Choosing the right model is crucial, considering hardware capabilities. Opt for a model that runs smoothly on your system to avoid performance issues.

As you can see from my screenshot, it’s possible to configure LM Studio to offload operations to the GPU, though optimal settings may vary. Experiment with different configurations to find the most efficient setup for your hardware. Remember that LM Studio will run on 4 CPU cores by default.

After configuring parameters, start the Local Server via the “Local Server” section of LM Studio. Specify a port and other parameters, but note that changing the default port may cause issues with the Visual Studio Code extension we’ll use later.

Visual Studio Code Configuration

With a local model ready and waiting, it’s time to connect using the open-source Continue extension in Visual Studio Code.

After installation, configure the extension to use your LM Studio server by editing the config.json file. This file is typically found at:

• Windows: C:/Users/{user}/.continue/config.json
• Linux/Mac: ~/.continue/config.json

Add your local model to the models array, ensuring to include title, provider, and model parameters.
Set allowAnonymousTelemetry to false.

Save the file, and you’re ready to generate code locally in Visual Studio Code using your LM Studio model!

For a comprehensive understanding of the extension’s features, refer to the official project wiki here.

Conclusion

Wrapping up our dive into LM Studio and Continue, we’ve found a whole new way to rock coding without breaking the bank. These local models give you all the power without any subscription fees. And with easy setup in Visual Studio Code, you’re ready to roll.

So, let’s get coding!

Contacts

For questions or suggestions, contact: noc@balzabu.io.