you2idea@video:~$ watch FPJ8ED1YhxY [12:03]
// transcript — 159 segments
0:02 a fascinating new model just dropped and apparently it is beating every other
0:07 model out there and it uses a completely new technique to enable the model to be
0:12 able to self-correct hallucinations it's kind of insane let's check it out so
0:16 here's the post by Matt Schumer Matt Schumer is a great follow on Twitter if
0:20 you're not already following him he puts out open- Source projects all the time
0:24 and now we have a brand new open source open weights model reflection 70b this
0:30 model is a fine-tune of the llama 3.1 model and it is a 70 billion parameter
0:35 model and currently it is the world's top open- Source model easily competing
0:40 with all of the frontier models and easily competing with all of the top
0:44 Frontier closed Source models so Matt Schumer's new model reflection 70b is
0:49 trained using reflection tuning a technique developed to enable llms to
0:55 fix their own mistakes now he also mentions a 405b model coming next week
0:59 and we expect it to be the best model in the world so no big deal all right so
1:05 here are the benchmarks GP QA mlu human eval math GSM 8K and I eval as we can
1:13 see the MML U it reached nearly 90% with zero shot reflection now that is
1:19 compared to Cloud 3.5 Sonet at 88.7 Cloud 3 Opus 86.8 GPT 40 88.7 so yeah it
1:28 beats every other model including the base model llama 3.14 5B gp2a and human
1:34 eval are the only ones in which it barely lost against clae 3.5 Sonet but
1:39 it is a much smaller model plus its open source plus its open weights and you can
1:44 download it already right now from hugging face now I plan on doing a full
1:50 llm test of it but the website's down so this is the website we're currently
1:53 experiencing high traffic and are temporarily down please try again soon
1:57 Matt Schumer said that this website is just getting absolutely hammered and so
2:00 yeah you're not going to be able to use it anytime soon they're adding more gpus
2:04 but as soon as he does that he said it's being saturated and look at GSM 8K which
2:10 is basically solved it's at 100% which is just absolutely nuts so how was he
2:14 able to achieve this incredible performance let's look into that
2:18 reflection 70b holds its own against even the top closed Source models Claude
2:24 3.5 Sonet and gbt 40 it's the top llm in at least mlu math if eval and GSM AK it
2:31 beats GPT 40 on every Benchmark tested it clobbers llama 3.1 405b it's not even
2:37 close all right let's look at an example of how it works and then I'll explain
2:40 what's going on so the way that this model was created is using
2:44 self-reflection in the data set meaning it actually showed hallucinations and
2:49 self-correction in the actual data set so it's able to recreate that as you're
2:53 running inference with it so in this example it says write the first sentence
2:57 of the preamble to the Declaration of Independence but do it in mirrored
3:01 writing and that is kind of a new prompt that everybody's using to try to test
3:05 large language models so initially we have this thinking tag here so this is
3:09 an interesting and moderately challenging task let's break it down
3:13 into steps so it's basically thinking out loud it's not actually thinking
3:17 because it's still just predicting the next token but it is kind of thinking
3:22 pseudo thinking so recall the preamble to the Declaration of Independence write
3:26 out the first sentence convert each character to its mirrored version and
3:30 present the final mirrored text let's go through this plan so the preamble to the
3:33 Declaration of Independence is well known and we hold these truths to be
3:37 self-evident here's the first sentence of the Preamble now we need to convert
3:41 this to mirrored writing in mirrored writing each letter is flipped
3:46 horizontally as if seen in a mirror that means we need to reverse the order of
3:49 the letters and flip each individual character now here's the special part
3:54 reflection here's the reflection tag before proceeding I should clarify that
3:58 there are two ways to interpret mirrored writing a simply reversing the order of
4:03 the letters right to left or B reversing the order and flipping each character
4:08 for this task I'll use method b as it's more challenging and true to the concept
4:12 of mirrored writing so reflection over that's the end tag step four here's the
4:17 mirrored version and there it is it is output in mirrored writing now reflected
4:23 again here's another reflection I've double checked the mirror text and it
4:27 appears to be correct each character has been flipped and the order has been
4:31 reversed first the punctuation marks remain the same as they would in a true
4:35 mirror reflection now here's the thing this is a really cool technique it seems
4:40 to work very very well but it's not actually doing step-by-step thinking
4:45 this is all just one output zero shot so again it's kind of pseudo thinking or
4:49 semi Chain of Thought it's hard to explain but it's certainly not iterating
4:53 on the prompt response and it's definitely not using multi-agent or
4:58 multi llms to coordinate and collaborate on the response but it still does it
5:02 incredibly well thanks to the sponsor of this video vulture reap the benefits of
5:07 the world's largest independent cloud provider when you bring your GPU
5:11 workloads over to vulture they have the latest Nvidia gpus spanning 32 locations
5:19 across six continents vulture delivers industry leading price to Performance
5:22 and serious accessibility and reliability vulture's Global fully
5:28 composable Cloud infrastructure moves your applications closer to to your
5:32 users and frees you from vendor lockin allowing you to bring your own
5:36 networking and database Solutions and if you need to scale Beyond just a single
5:40 cluster vulture's kubernetes engine allows you to take full control over
5:45 your deployment offering up a 100% free control plane so if you're tired of
5:49 waiting for gpus from other providers make sure to use vulture you can deploy
5:53 at any scale immediately and they have h100s l4s and more available right now
5:58 and you can get a fraction of a card or fully dedicated ated bare metal systems
6:03 which gives you full control over your hardware and your throughput they also
6:07 have oneclick installation of all the applications you might need for advanced
6:11 machine learning workloads allowing you to get up and running in minutes not
6:15 hours so experience the vulture difference don't get bogged down by
6:20 severe weight times or limited locations try it free today with a $300 credit for
6:26 your first 30 days when you visit get vulture.com Burman and make sure to use code Burman
6:33 300 at checkout to get that credit thanks again to vulture for sponsoring
6:37 this video the technique that drives reflection 70b is simple but very
6:42 powerful current llms have a tendency to hallucinate and can't recognize when
6:46 they do so and this is very very true reflection tuning enables llms to
6:50 recognize their mistakes and then correct them before committing to an
6:55 answer now I don't actually know if that's 100% accurate because ultimately
7:00 unless the large language model has a completely new architecture or is able
7:05 to read the output and then iterate on that output it's not really able to
7:11 self-correct but it does seem to work so maybe I'm wrong let's look at this
7:15 example so counting the RS in Strawberry so it does that and then it says there
7:19 are two RS in the word strawberry and then it reflects on it I made a mistake
7:23 I can clearly see now that there are actually three Rs in the word strawberry
7:27 but here's the thing I'm confused about it's not actually reflecting on what has
7:32 already been output that's not really how it works it is just predicting the
7:36 next token just like any other model would but again for some reason this
7:40 seems to work really well but a better output would just be to get it right the
7:44 first time around because now you have to get it wrong and then correct and
7:47 you're using all those extra tokens probably not a big deal but still the
7:51 optimal output is just to get it right the first time then additionally we
7:55 separate planning into a separate step improving Chain of Thought potency and
7:58 keeping the output simple and concise for end user let's look at an example
8:02 another popular test which I've incorporated into my llm rubric which
8:08 number is larger 9.11 or 9.9 now look at all this text what the optimal result
8:13 would have been is just simply say 9.9 is larger but now let's look what it
8:17 went through to get to that answer this is a relatively simple problem that
8:21 involves comparing two decimal numbers I can solve this using a straightforward
8:24 approach with Chain of Thought reasoning plan identify the numbers to compare
8:29 compare the whole number parts if whole number parts are equal compare the
8:32 decimal Parts determine which number is larger based on the comparison Chain of
8:37 Thought okay so it continues reflects the approach is correct the comparison
8:41 of whole number and decimal Parts is accurate conclusion is logical based on
8:47 the comparison so output 9.9 is larger than 9.11 that's great but look how many
8:52 tokens it took to get there and here's the other thing this model was
8:56 fine-tuned to go through all of these steps to reach the right answer and
9:01 potentially quote un quote identify its mistakes but you could also just do that
9:06 with prompt engineering you can simply say think through this step by step or
9:10 first I want you to give me an answer then I want you to reflect on that
9:13 answer so it's essentially that kind of prompt engineering just built into the
9:18 model itself which is great but I really don't think this is some leap in core
9:22 intelligence of the model it's just a better way of including that kind of
9:26 prompt technique in the model itself rather than putting it in the actual
9:30 prompt and if we look at Hacker News right now they actually posted about it
9:35 and user RW L4 says you can somewhat recreate this just by using a system
9:39 prompt which induces the model to do the same thing and here it is so here is
9:43 essentially the same thing that you could do that Matt Schumer has been able
9:47 to train into his model so begin with the thinking section inside the thinking
9:52 section do that Chain of Thought include a reflection section check your
9:55 reasoning be sure to close all reflection sections etc etc and then
10:01 output and the jux the position user says under the hood reflection 70b seems
10:05 to be llama 3.1 fine tune that encourages the model to add think
10:09 reflection and output tokens and corresponding phases this is an
10:13 evolution of chain of thoughts think step by step yep so it works that's
10:18 great I'm glad to see it it's beating most other models right now so you know
10:22 what good on him good job Matt Schumer he goes on to say it's important to note
10:26 we have checked for decontamination against all benchmarks mentioned using
10:31 lmis . org's llm decontaminator the weights of our 70b model are available today on hugging
10:36 face here it'll be available on hyperbolic Labs API available later
10:41 today next week we will be releasing the weights of 405b along with a short
10:45 report going into more detail of our process and findings and a huge shout
10:51 out to sahil and glaive AI so here's another funny test how many RS are in
10:57 then this kind of gibberish word after a careful and thorough analysis of the
10:59 word I can confidently state that there are nine RS in this word so 1 2 3 4 5 6 7 8
11:10 n yep so that's right let's break it down the first R appears as the second
11:14 letter then there's a cluster of three Rs okay so interestingly enough it
11:19 doesn't actually have those tags we just talked about in this example but it did
11:23 get it right so let's see if we can get it to work count the RS in Strawberry
11:27 okay so I actually was able to get it to output something after waiting just a
11:30 couple minutes so how many RS in the word strawberry and there we go there
11:35 are three Rs in the word strawberry all right so I can't get the model to Output
11:39 anything else as soon as I can I'm going to be testing it I might just download
11:42 it and get it running on my local machine because that seems like the best
11:46 bet to beat the traffic so let me know what you think of this model it seems to
11:50 perform extremely well it has built in prompt engineering and outputs great
11:54 results and is beating all the other open source models and basically all the
11:57 other models period so can't wait to see 405b if you enjoyed this video please
12:01 consider giving a like And subscribe and
$

New LLM DESTROYS Every Other Model with "Self Healing" (Open Source)

[AI agents and automation][open source and self-hosting][community building and audience growth][content creation and YouTube]
// description

A new model by Matt Schumer uses a technique to identify and fix hallucinations as the output is generated. Vultr is empowering the next generation of generative AI startups with access to the latest NVIDIA GPUs. Try it yourself when you visit https://getvultr.com/berman and use promo code "BERMAN300" for $300 off your first 30 days. Join My Newsletter for Regular AI Updates 👇🏼 https://forwardfuture.ai My Links 🔗 👉🏻 Main Channel: https://www.youtube.com/@matthew_berman 👉🏻 Twitter: https://t

now: 0:00
// tags
[AI agents and automation][open source and self-hosting][community building and audience growth][content creation and YouTube]