You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note: If you are on Windows, please run ``docker-compose`` not ``docker compose`` and make sure the project is in the Linux Filesystem, otherwise loading models might be slow. For more Info: [Microsoft Docs](https://learn.microsoft.com/en-us/windows/wsl/filesystems)
91
+
Note: If you are on Windows, please make sure the project is on the Linux Filesystem, otherwise loading models might be slow. For more Info: [Microsoft Docs](https://learn.microsoft.com/en-us/windows/wsl/filesystems)
Copy file name to clipboardExpand all lines: docs/content/howtos/easy-model.md
+30-30Lines changed: 30 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,43 +5,52 @@ title = "Easy Model Setup"
5
5
weight = 2
6
6
+++
7
7
8
-
Lets Learn how to setup a model, for this ``How To`` we are going to use the ``Luna-Ai`` model (Yes I know haha - ``Luna Midori`` making a how to using the ``luna-ai-llama2``model - lol)
8
+
Lets learn how to setup a model, for this ``How To`` we are going to use the ``Dolphin 2.2.1 Mistral 7B``model.
9
9
10
10
To download the model to your models folder, run this command in a commandline of your picking.
Each model needs at least ``4`` files, with out these files, the model will run raw, what that means is you can not change settings of the model.
19
+
Each model needs at least ``5`` files, with out these files, the model will run raw, what that means is you can not change settings of the model.
20
20
```
21
21
File 1 - The model's GGUF file
22
22
File 2 - The model's .yaml file
23
23
File 3 - The Chat API .tmpl file
24
-
File 4 - The Completion API .tmpl file
24
+
File 4 - The Chat API helper .tmpl file
25
+
File 5 - The Completion API .tmpl file
25
26
```
26
27
So lets fix that! We are using ``lunademo`` name for this ``How To`` but you can name the files what ever you want! Lets make blank files to start with
27
28
28
29
```bash
29
30
touch lunademo-chat.tmpl
31
+
touch lunademo-chat-block.tmpl
30
32
touch lunademo-completion.tmpl
31
33
touch lunademo.yaml
32
34
```
33
-
Now lets edit the `"lunademo-chat.tmpl"`, Looking at the huggingface repo, this model uses the ``ASSISTANT:`` tag for when the AI replys, so lets make sure to add that to this file. Do not add the user as we will be doing that in our yaml file!
35
+
Now lets edit the `"lunademo-chat.tmpl"`, This is the template that model "Chat" trained models use, but changed for LocalAI
34
36
35
37
```txt
36
-
{{.Input}}
38
+
<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "user"}}user{{end}}
39
+
{{if .Content}}{{.Content}}{{end}}
40
+
<|im_end|>
41
+
```
42
+
43
+
For the `"lunademo-chat-block.tmpl"`, Looking at the huggingface repo, this model uses the ``<|im_start|>assistant`` tag for when the AI replys, so lets make sure to add that to this file. Do not add the user as we will be doing that in our yaml file!
37
44
38
-
ASSISTANT:
45
+
```txt
46
+
{{.Input}}
47
+
<|im_start|>assistant
39
48
```
40
49
41
-
Now in the `"lunademo-completion.tmpl"` file lets add this.
50
+
Now in the `"lunademo-completion.tmpl"` file lets add this. (This is a hold over from OpenAI V0)
42
51
43
52
```txt
44
-
Complete the following sentence: {{.Input}}
53
+
{{.Input}}
45
54
```
46
55
47
56
@@ -58,25 +67,18 @@ What this does is tell ``LocalAI`` how to load the model. Then we are going to *
58
67
```yaml
59
68
name: lunademo
60
69
parameters:
61
-
model: luna-ai-llama2-uncensored.Q4_K_M.gguf
70
+
model: dolphin-2.2.1-mistral-7b.Q4_0.gguf
62
71
```
63
72
64
-
Now that we have the model set up, there a few things we should add to the yaml file to make it run better, for this model it uses the following roles.
65
-
```yaml
66
-
roles:
67
-
assistant: 'ASSISTANT:'
68
-
system: 'SYSTEM:'
69
-
user: 'USER:'
70
-
```
71
-
72
-
What that did is made sure that ``LocalAI`` added the test to the users in the request, so if a message is from ``system`` it shows up in the template as ``SYSTEM:``, speaking of template files, lets add those to our models yaml file now.
73
+
Now that LocalAI knows what file to load with our request, lets add the template files to our models yaml file now.
73
74
```yaml
74
75
template:
75
-
chat: lunademo-chat
76
+
chat: lunademo-chat-block
77
+
chat_message: lunademo-chat
76
78
completion: lunademo-completion
77
79
```
78
80
79
-
If you are running on ``GPU`` or want to tune the model, you can add settings like
81
+
If you are running on ``GPU`` or want to tune the model, you can add settings like (higher the GPU Layers the more GPU used)
80
82
```yaml
81
83
f16: true
82
84
gpu_layers: 4
@@ -85,8 +87,7 @@ gpu_layers: 4
85
87
To fully tune the model to your like. But be warned, you **must** restart ``LocalAI`` after changing a yaml file
86
88
87
89
```bash
88
-
docker-compose restart ##windows
89
-
docker compose restart ##linux / mac
90
+
docker compose restart
90
91
```
91
92
92
93
If you want to check your models yaml, here is a full copy!
@@ -96,19 +97,18 @@ context_size: 2000
96
97
##Put settings right here for tunning!! Before name but after Backend!
97
98
name: lunademo
98
99
parameters:
99
-
model: luna-ai-llama2-uncensored.Q4_K_M.gguf
100
-
roles:
101
-
assistant: 'ASSISTANT:'
102
-
system: 'SYSTEM:'
103
-
user: 'USER:'
100
+
model: dolphin-2.2.1-mistral-7b.Q4_0.gguf
104
101
template:
105
-
chat: lunademo-chat
102
+
chat: lunademo-chat-block
103
+
chat_message: lunademo-chat
106
104
completion: lunademo-completion
107
105
```
108
106
109
107
Now that we got that setup, lets test it out but sending a [request]({{%relref "easy-request" %}}) to Localai!
110
108
111
-
## Adv Stuff
109
+
## ----- Adv Stuff -----
110
+
111
+
**(Please do not run these steps if you have already done the setup)**
112
112
Alright now that we have learned how to set up our own models, here is how to use the gallery to do alot of this for us. This command will download and set up (mostly, we will **always** need to edit our yaml file to fit our computer / hardware)
At this point we want to set up our `.env` file, here is a copy for you to use if you wish, please make sure to set it to the same as in the `docker-compose` file for later.
36
+
At this point we want to set up our `.env` file, here is a copy for you to use if you wish, Make sure this is in the ``LocalAI`` folder.
31
37
32
38
```bash
33
39
## Set number of threads.
@@ -102,8 +108,7 @@ services:
102
108
Make sure to save that in the root of the `LocalAI` folder. Then lets spin up the Docker run this in a `CMD` or `BASH`
At this point we want to set up our `.env` file, here is a copy for you to use if you wish, please make sure to set it to the same as in the `docker-compose` file for later.
36
+
At this point we want to set up our `.env` file, here is a copy for you to use if you wish, Make sure this is in the ``LocalAI`` folder.
31
37
32
38
```bash
33
39
## Set number of threads.
@@ -134,8 +140,7 @@ services:
134
140
Make sure to save that in the root of the `LocalAI` folder. Then lets spin up the Docker run this in a `CMD` or `BASH`
0 commit comments