r/oobaboogazz • • Jul 18 '23

LLaMA-v2 megathread

92 Upvotes

I'm testing the models and will update this post with the information so far.

Running the models

They just need to be converted to transformers format, and after that they work normally, including with --load-in-4bit and --load-in-8bit.

Conversion instructions can be found here: https://github.com/oobabooga/text-generation-webui/blob/dev/docs/LLaMA-v2-model.md

Perplexity

Using the exact same test as in the first table here.

ModelBackendPerplexity
LLaMA-2-70bllama.cpp q4_K_M4.552 (0.46 lower)
LLaMA-65bllama.cpp q4_K_M5.013
LLaMA-30bTransformers 4-bit5.246
LLaMA-2-13bTransformers 8-bit5.434 (0.24 lower)
LLaMA-13bTransformers 8-bit5.672
LLaMA-2-7bTransformers 16-bit5.875 (0.27 lower)
LLaMA-7bTransformers 16-bit6.145

The key takeaway for now is that LLaMA-2-13b is worse than LLaMA-1-30b in terms of perplexity, but it has 4096 context.

Chat test

Here is an example with the system message "Use emojis only.".

The model was loaded with this command:

python server.py --model models/llama-2-13b-chat-hf/ --chat --listen --verbose --load-in-8bit

The correct template gets automatically detected in the latest version of text-generation-webui (v1.3).

In my quick tests, both the 7b and the 13b models seem to perform very well. This is the first quality RLHF-tuned model to be open sourced. So the 13b chat model is very likely to perform better than previous 30b instruct models like WizardLM.

TODO

  • Figure out the exact prompt format for the chat variants.

  • Test the 70b model.


Updates

  • Update 1: Added LLaMA-2-13b perplexity test.

  • Update 2: Added conversion instructions.

  • Update 3: I found the prompt format.

  • Update 4: added a chat test and personal impressions.

  • Update 5: added a Llama-70b perplexity test.


r/oobaboogazz • • Aug 04 '23

Mod Post I have requested ownership of the r/oobabooga subreddit :)

76 Upvotes

Here is the link to the thread on r/redditrequest: https://www.reddit.com/r/redditrequest/comments/15hl9o0/requesting_roobabooga_due_to_being_unmoderated/

The moderator doesn't reply to my messages and is acting in a spiteful way against me and the project for whatever reason. I simply want the subreddit back online so that people can access the hundreds of posts with useful information available there.

Backstory: that subreddit has been "dark" for almost 2 months.

If you can, please leave a comment on that thread. Maybe it helps with the transfer decision.


r/oobaboogazz • • Aug 15 '23

Mod Post LET'S GOOOOOOOOOOO R/OOBABOOGA IS MINE!!!!!!!!!!!!!!!!

69 Upvotes

See the post here:

https://www.reddit.com/r/Oobabooga/comments/15rs8gz/roobabooga_is_back/

THANK YOU

to everyone who supported me in those weird times and participated on r/oobaboogazz. Now we get the privilege of being an unified community again.

Transition

Let's move back to the old sub in the next 7 days. During this period, both subreddits will coexist, but please try to create new posts in the old one and not here. After that, r/oobaboogazz will be closed for new submissions and will become a public archive.


r/oobaboogazz • • Aug 04 '23

Mod Post Update: I was banned from r/oobabooga lol

Post image
61 Upvotes

r/oobaboogazz • • Jun 28 '23

Discussion Just a moment to say thank you

60 Upvotes

First for starting and keeping your project going, then for starting this subreddit up. I can imagine tons of people are always asking for answers, features, and tweaks all the time. Let me just say thank you and do not think for a moment it is not appreciated.


r/oobaboogazz • • Jun 28 '23

Mod Post TUTORIAL: how to load the SuperHOT LoRA in the webui.

52 Upvotes

To celebrate this sub's rapid growth from 1 member yesterday (me) to 256 members today, I will write a detailed tutorial on how to use the SuperHOT LoRA.

Why use SuperHOT?

This is the first LoRA to explore kaiokendev's positional embedding scaling technique described here: https://kaiokendev.github.io/til. By fine-tuning with this scaling, the model can be taught to understand context beyond the original 2048 tokens that it learned to process in its original training.

Without a fine tune, the context can still be extended, but the performance of the model will degrade past the original context limit.

Other than having a longer context length, SuperHOT is also a very good model for chat in general.

Downloading the models

You are going to need both a base LLaMA model in GPTQ format and the corresponding LoRA.

For the LLaMA GPTQ model, I have been using the 4bit-128g weights in the torrents linked here for many months: https://github.com/oobabooga/text-generation-webui/blob/main/docs/LLaMA-model.md#getting-the-weights

The model folders will look like this: llama-13b-4bit-128g or llama-30b-4bit-128g. Those folders should be placed inside your text-generation-webui/models folder.

Moving on to the LoRAs. At https://huggingface.co/kaiokendev, 3 SuperHOT LoRAs can currently be found:

For this demonstration, let's use the 13b version with 8k context. It can be downloaded with this command:

python download-model.py kaiokendev/superhot-13b-8k-no-rlhf-test

Alternatively, you can download it directly through the UI by going to the Model tab, pasting kaiokendev/superhot-13b-8k-no-rlhf-test under "Download custom model or LoRA", and then clicking on "Download".

Loading the model

In the Model tab, select "ExLlama_HF" under "Model loader", set max_seq_len to 8192, and set compress_pos_emb to 4. Then, select the llama-13b-4bit-128g model in the "Model" dropdown to load it.

You may have to reduce max_seq_len if you run out of memory while trying to generate text. For me, these were the parameters that worked with 24GB VRAM:

ModelParamsMaximum context
llama-13bmax_seq_len = 8192, compress_pos_emb = 46079 tokens
llama-30bmax_seq_len = 3584, compress_pos_emb = 23100 tokens

If you use a max_seq_len of less than 4096, my understanding is that it's best to set compress_pos_emb to 2 and not 4, even though a factor of 4 was used while training the LoRA.

Applying the LoRA

After loading the model, select the "kaiokendev_superhot-13b-8k-no-rlhf-test" option in the LoRA dropdown, and then click on the "Apply LoRAs" button.

Adjusting the truncation length

In the Parameters tab, set "Truncate the prompt up to this length" to 8192.

Generate text

Now just use the model as usual, and you will be able to go beyond 2048 tokens!

Doing everything from the command-line

The following command automates the whole process:

python server.py --model llama-13b-4bit-128g --lora kaiokendev_superhot-13b-8k-no-rlhf-test --loader exllama_hf --compress_pos_emb 4 --max_seq_len 8192 --chat

To set a higher default value for the "Truncate the prompt up to this length", you can copy the file "settings-template.yaml" to "settings.yaml" inside your text-generation-webui folder, and then open this file with a text editor and edit the value after "truncation_length". After that, every time you start the webui this value will be used as the default.

Alternatives

(none of these will work at the moment for SuperHOT because the embedding compression thing is not implemented, but they will work for other LoRAs)


r/oobaboogazz • • Jun 27 '23

Other Thank you for all that you have done!

42 Upvotes

I honestly don't think there would be as many flavors of LLMs out there without your software.


r/oobaboogazz • • Aug 11 '23

Mod Post New loader: ctransformers

35 Upvotes

I had been delaying this since forever but now it's finally merged: https://github.com/oobabooga/text-generation-webui/pull/3313

ctransformers allows models like falcon, starcoder, and gptj to be loaded in GGML format for CPU inference. GPU offloading through n-gpu-layers is also available just like for llama.cpp. The full list of supported models can be found here.


r/oobaboogazz • • Aug 03 '23

Mod Post New feature: download the chat history or the entire session directly from the browser with 1 click.

Thumbnail
gallery
34 Upvotes

r/oobaboogazz • • Aug 06 '23

Mod Post Classifier-Free Guidance (CFG) support has been merged

33 Upvotes

r/oobaboogazz • • Jul 16 '23

Mod Post If anyone ever wondered if llama-65b 2-bit is worth it

29 Upvotes

The answer is no, it performs worse than llama-30b 4-bit.

  • llama-30b.ggmlv3.q4_K_M.bin: 5.21557

  • llama-65b.ggmlv3.q2_K.bin: 5.44745

The updated table can be found here: https://oobabooga.github.io/blog/posts/perplexities/


r/oobaboogazz • • Aug 12 '23

Project A cool google search extension for oobabooga!

28 Upvotes

Ive messed around with this repo (not mine) :

https://github.com/simbake/web_search

It can get context using a prompt like "search Twitter changed to X"And you will get a normal answer not some hallucinations nonesense.

You can get current news, weather and more right inside your chat! and let it remember it and ask follow up questions.

Really good find for me, so I hope you guys enjoy it as much as I do.


r/oobaboogazz • • Aug 12 '23

Tutorial Beginner's guide to Llama models

27 Upvotes

I have written a guide for llama models. Perfect for people who knows nothing about local LLM models. I hope someone would find this useful.

https://agi-sphere.com/llama-guide/


r/oobaboogazz • • Aug 01 '23

Mod Post Testing the new long_replies extension with base llama

Post image
27 Upvotes

r/oobaboogazz • • Jul 28 '23

LoRA LoRA training information, with examples and screenshots

27 Upvotes

I've seen a lot of people ask how to train LoRAs with Oobabooga, because I've been searching for answers too!

I am just learning how to do this and have some of the process figured out. I've created a Medical Knowledge LoRA and uploaded everything I could think of to help others here:

https://huggingface.co/AARon99/MedText-llama-2-70b-Guanaco-QLoRA-fp16

Check out the screenshots and training data to get an understanding for what I did. I will try to answer any questions in the comments that I have the capability to answer. I am still a beginner.


r/oobaboogazz • • Jul 24 '23

Mod Post llama-2-70b GGML support is here

Thumbnail
github.com
27 Upvotes

r/oobaboogazz • • Jun 26 '23

Mod Post zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

27 Upvotes

r/oobaboogazz • • Jul 26 '23

Mod Post A detailed extension example

25 Upvotes

I have written a detailed extension example illustrating every modifier function and what it does. It can be found here: example/script.py.

That should make it a lot easier to create a new extension. Just copy and paste this template and add your changes.

The docs have also been updated: https://github.com/oobabooga/text-generation-webui/blob/main/docs/Extensions.md


r/oobaboogazz • • Jul 04 '23

Mod Post [News]: added sessions, basic multi-user support

24 Upvotes

https://github.com/oobabooga/text-generation-webui/pull/2991

In this PR, I have added a "Sessions" functionality where you can save the entire interface state, including the chat history, character, generation parameters, and input/output text in notebook/default modes.

This makes it possible to:

  • Have multiple histories for the same character.

  • Easily continue instruct conversations in the future.

  • Save generations in default/notebook modes to read or continue later.

An "autosave" session is also saved every time you generate text. It can be loaded back even if you turn off the computer.

To do this, I had to convert the chat history from a global variable to a "State" variable. This allowed me to add a "--multi-user" flag that causes the chat history to be 100% temporary and not shared between users, thus adding basic multi-user functionality in chat mode.

To use sessions, just launch the UI and go to the Sessions tab. There you can load, save, and delete sessions.

Feedback on whether things are working as expected or not would be appreciated. This was a pretty big update with many changes to the code.


r/oobaboogazz • • Aug 03 '23

Project Character Creator (WIP)

23 Upvotes

I've been working on a tool to help create detailed characters with enough information to guide the LLM. Quick preview below. If you want to test it out feedback is appreciated!

https://huggingface.co/spaces/mikefish/CharacterMaker


r/oobaboogazz • • Aug 12 '23

Tutorial I made a guide for the text Gen side of Oobabooga

Thumbnail
youtu.be
22 Upvotes

r/oobaboogazz • • Aug 10 '23

Mod Post I need modszzzzzzzzzzzz!

20 Upvotes

I would like to add a few voluntary moderators to this community to help me keep it organized and need volunteers.

Your tasks

  1. Check the mod panel now and then and see if any message has been shadowbanned by the automoderator.

  2. Remove content that breaks the rules if any pops up (only if absolutely necessary).

What you will get in return

  1. A custom "Modzz" user flair 🏆


If you find that interesting, please leave a comment below. Your help would be really appreciated.

PS: my request for the previous r/oobabooga sub has been denied, so this will be the permanent community for this project from now on.


EDIT: 3 mods have been added, which I think should be enough for now. Thank you so much to everyone who volunteered!


r/oobaboogazz • • Aug 01 '23

Mod Post New feature: filter generation parameters by loader

Thumbnail
gallery
20 Upvotes

r/oobaboogazz • • Jul 31 '23

Discussion Starting to think no one here understands the parameters/settings (me neither)

18 Upvotes

Like since i've been installing/testing/playing with the models, I see many people asking questions or giving their opinions on the parameter but to be honest, I've not seen ONE post explaining in details what each dial does. (Top P, Big O, Min K and all that).

A lot of it looks and feels like Arcane knowledge lost and we all have our "ways" to make it do its deeds.

...But I haven't seen one post describing in details what it does, like if the creators were under a demonic spell and were controlled during its creation.


r/oobaboogazz • • Aug 22 '23

Mod Post This sub has been moved - please join r/oobabooga!

Post image
17 Upvotes