Demsay Demsay

The example-driven, practical walkthrough of Large Language Models and their growing list of related features, as a new entry to my general audience series on LLMs. In this more practical followup, I take you through the many ways I use LLMs in my own life.

Chapters
00:00:00 Intro into the growing LLM ecosystem
00:02:54 ChatGPT interaction under the hood
00:13:12 Basic LLM interactions examples
00:18:03 Be aware of the model you’re using, pricing tiers
00:22:54 Thinking models and when to use them
00:31:00 Tool use: internet search
00:42:04 Tool use: deep research
00:50:57 File uploads, adding documents to context
00:59:00 Tool use: python interpreter, messiness of the ecosystem
01:04:35 ChatGPT Advanced Data Analysis, figures, plots
01:09:00 Claude Artifacts, apps, diagrams
01:14:02 Cursor: Composer, writing code
01:22:28 Audio (Speech) Input/Output
01:27:37 Advanced Voice Mode aka true audio inside the model
01:37:09 NotebookLM, podcast generation
01:40:20 Image input, OCR
01:47:02 Image output, DALL-E, Ideogram, etc.
01:49:14 Video input, point and talk on app
01:52:23 Video output, Sora, Veo 2, etc etc.
01:53:29 ChatGPT memory, custom instructions
01:58:38 Custom GPTs
02:06:30 Summary

Links
– Tiktokenizer https://tiktokenizer.vercel.app/
– OpenAI’s ChatGPT https://chatgpt.com/
– Anthropic’s Claude https://claude.ai/
– Google’s Gemini https://gemini.google.com/
– xAI’s Grok https://grok.com/
– Perplexity https://www.perplexity.ai/
– Google’s NotebookLM https://notebooklm.google.com/
– Cursor https://www.cursor.com/
– Histories of Mysteries AI podcast on Spotify https://open.spotify.com/show/3K4LRyMCP44kBbiOziwJjb

– The visualization UI I was using in the video: https://excalidraw.com/
– The specific file of Excalidraw we built up: https://drive.google.com/file/d/1DN3LU3MbKI00udxoS-W5ckCHq99V0Uqs/view?usp=sharing
– Discord channel for Eureka Labs and this video: https://discord.gg/3zy8kqD9Cp

Educational Use Licensing
This video is freely available for educational and internal training purposes. Educators, students, schools, universities, nonprofit institutions, businesses, and individual learners may use this content freely for lessons, courses, internal training, and learning activities, provided they do not engage in commercial resale, redistribution, external commercial use, or modify content to misrepresent its intent.

Andrej Karpathy is a legendary AI researcher, engineer, and educator. He’s the former director of AI at Tesla, a founding member of OpenAI, and an educator at Stanford. Please support this podcast by checking out our sponsors:
– Eight Sleep: https://www.eightsleep.com/lex to get special savings
– BetterHelp: https://betterhelp.com/lex to get 10% off
– Fundrise: https://fundrise.com/lex
– Athletic Greens: https://athleticgreens.com/lex to get 1 month of fish oil

EPISODE LINKS:
Andrej’s Twitter: http://twitter.com/karpathy
Andrej’s YouTube: http://youtube.com/c/AndrejKarpathy
Andrej’s Website: http://karpathy.ai
Andrej’s Google Scholar: http://scholar.google.com/citations?user=l8WuQJgAAAAJ
Books mentioned:
The Vital Question: https://amzn.to/3q0vN6q
Life Ascending: https://amzn.to/3wKIsOE
The Selfish Gene: https://amzn.to/3TCo63s
Contact: https://amzn.to/3W3y5Au
The Cell: https://amzn.to/3W5f6pa

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 – Introduction
0:58 – Neural networks
6:01 – Biology
11:32 – Aliens
21:43 – Universe
33:34 – Transformers
41:50 – Language models
52:01 – Bots
58:21 – Google’s LaMDA
1:05:44 – Software 2.0
1:16:44 – Human annotation
1:18:41 – Camera vision
1:23:46 – Tesla’s Data Engine
1:27:56 – Tesla Vision
1:34:26 – Elon Musk
1:39:33 – Autonomous driving
1:44:28 – Leaving Tesla
1:49:55 – Tesla’s Optimus
1:59:01 – ImageNet
2:01:40 – Data
2:11:31 – Day in the life
2:24:47 – Best IDE
2:31:53 – arXiv
2:36:23 – Advice for beginners
2:45:40 – Artificial general intelligence
2:59:00 – Movies
3:04:53 – Future of human civilization
3:09:13 – Book recommendations
3:15:21 – Advice for young people
3:17:12 – Future of machine learning
3:24:00 – Meaning of life

SOCIAL:
– Twitter: https://twitter.com/lexfridman
– LinkedIn: https://www.linkedin.com/in/lexfridman
– Facebook: https://www.facebook.com/lexfridman
– Instagram: https://www.instagram.com/lexfridman
– Medium: https://medium.com/@lexfridman
– Reddit: https://reddit.com/r/lexfridman
– Support on Patreon: https://www.patreon.com/lexfridman

This is a 1 hour general-audience introduction to Large Language Models: the core technical component behind systems like ChatGPT, Claude, and Bard. What they are, where they are headed, comparisons and analogies to present-day operating systems, and some of the security-related challenges of this new computing paradigm.
As of November 2023 (this field moves fast!).

Context: This video is based on the slides of a talk I gave recently at the AI Security Summit. The talk was not recorded but a lot of people came to me after and told me they liked it. Seeing as I had already put in one long weekend of work to make the slides, I decided to just tune them a bit, record this round 2 of the talk and upload it here on YouTube. Pardon the random background, that’s my hotel room during the thanksgiving break.

– Slides as PDF: https://drive.google.com/file/d/1pxx_ZI7O-Nwl7ZLNk5hI3WzAsTLwvNU7/view?usp=share_link (42MB)
– Slides. as Keynote: https://drive.google.com/file/d/1FPUpFMiCkMRKPFjhi9MAhby68MHVqe8u/view?usp=share_link (140MB)

Few things I wish I said (I’ll add items here as they come up):
– The dreams and hallucinations do not get fixed with finetuning. Finetuning just “directs” the dreams into “helpful assistant dreams”. Always be careful with what LLMs tell you, especially if they are telling you something from memory alone. That said, similar to a human, if the LLM used browsing or retrieval and the answer made its way into the “working memory” of its context window, you can trust the LLM a bit more to process that information into the final answer. But TLDR right now, do not trust what LLMs say or do. For example, in the tools section, I’d always recommend double-checking the math/code the LLM did.
– How does the LLM use a tool like the browser? It emits special words, e.g. |BROWSER|. When the code “above” that is inferencing the LLM detects these words it captures the output that follows, sends it off to a tool, comes back with the result and continues the generation. How does the LLM know to emit these special words? Finetuning datasets teach it how and when to browse, by example. And/or the instructions for tool use can also be automatically placed in the context window (in the “system message”).
– You might also enjoy my 2015 blog post “Unreasonable Effectiveness of Recurrent Neural Networks”. The way we obtain base models today is pretty much identical on a high level, except the RNN is swapped for a Transformer. http://karpathy.github.io/2015/05/21/rnn-effectiveness/
– What is in the run.c file? A bit more full-featured 1000-line version hre: https://github.com/karpathy/llama2.c/blob/master/run.c

Chapters:
Part 1: LLMs
00:00:00 Intro: Large Language Model (LLM) talk
00:00:20 LLM Inference
00:04:17 LLM Training
00:08:58 LLM dreams
00:11:22 How do they work?
00:14:14 Finetuning into an Assistant
00:17:52 Summary so far
00:21:05 Appendix: Comparisons, Labeling docs, RLHF, Synthetic data, Leaderboard
Part 2: Future of LLMs
00:25:43 LLM Scaling Laws
00:27:43 Tool Use (Browser, Calculator, Interpreter, DALL-E)
00:33:32 Multimodality (Vision, Audio)
00:35:00 Thinking, System 1/2
00:38:02 Self-improvement, LLM AlphaGo
00:40:45 LLM Customization, GPTs store
00:42:15 LLM OS
Part 3: LLM Security
00:45:43 LLM Security Intro
00:46:14 Jailbreaks
00:51:30 Prompt Injection
00:56:23 Data poisoning
00:58:37 LLM Security conclusions
End
00:59:23 Outro

Educational Use Licensing
This video is freely available for educational and internal training purposes. Educators, students, schools, universities, nonprofit institutions, businesses, and individual learners may use this content freely for lessons, courses, internal training, and learning activities, provided they do not engage in commercial resale, redistribution, external commercial use, or modify content to misrepresent its intent.

We build a Generatively Pretrained Transformer (GPT), following the paper “Attention is All You Need” and OpenAI’s GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video.

Links:
– Google colab for the video: https://colab.research.google.com/drive/1JMLa53HDuA-i7ZBmqV7ZnA3c_fvtXnx-?usp=sharing
– GitHub repo for the video: https://github.com/karpathy/ng-video-lecture
– Playlist of the whole Zero to Hero series so far:
– nanoGPT repo: https://github.com/karpathy/nanoGPT
– my website: https://karpathy.ai
– my twitter: https://twitter.com/karpathy
– our Discord channel: https://discord.gg/3zy8kqD9Cp

Supplementary links:
– Attention is All You Need paper: https://arxiv.org/abs/1706.03762
– OpenAI GPT-3 paper: https://arxiv.org/abs/2005.14165
– OpenAI ChatGPT blog post: https://openai.com/blog/chatgpt/
– The GPU I’m training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: https://lambdalabs.com . If you prefer to work in notebooks, I think the easiest path today is Google Colab.

Suggested exercises:
– EX1: The n-dimensional tensor mastery challenge: Combine the `Head` and `MultiHeadAttention` into one class that processes all the heads in parallel, treating the heads as another batch dimension (answer is in nanoGPT).
– EX2: Train the GPT on your own dataset of choice! What other data could be fun to blabber on about? (A fun advanced suggestion if you like: train a GPT to do addition of two numbers, i.e. a+b=c. You may find it helpful to predict the digits of c in reverse order, as the typical addition algorithm (that you’re hoping it learns) would proceed right to left too. You may want to modify the data loader to simply serve random problems and skip the generation of train.bin, val.bin. You may want to mask out the loss at the input positions of a+b that just specify the problem using y=-1 in the targets (see CrossEntropyLoss ignore_index). Does your Transformer learn to add? Once you have this, swole doge project: build a calculator clone in GPT, for all of +-*/. Not an easy problem. You may need Chain of Thought traces.)
– EX3: Find a dataset that is very large, so large that you can’t see a gap between train and val loss. Pretrain the transformer on this data, then initialize with that model and finetune it on tiny shakespeare with a smaller number of steps and lower learning rate. Can you obtain a lower validation loss by the use of pretraining?
– EX4: Read some transformer papers and implement one additional feature or change that people seem to use. Does it improve the performance of your GPT?

Chapters:
00:00:00 intro: ChatGPT, Transformers, nanoGPT, Shakespeare
baseline language modeling, code setup
00:07:52 reading and exploring the data
00:09:28 tokenization, train/val split
00:14:27 data loader: batches of chunks of data
00:22:11 simplest baseline: bigram language model, loss, generation
00:34:53 training the bigram model
00:38:00 port our code to a script
Building the “self-attention”
00:42:13 version 1: averaging past context with for loops, the weakest form of aggregation
00:47:11 the trick in self-attention: matrix multiply as weighted aggregation
00:51:54 version 2: using matrix multiply
00:54:42 version 3: adding softmax
00:58:26 minor code cleanup
01:00:18 positional encoding
01:02:00 THE CRUX OF THE VIDEO: version 4: self-attention
01:11:38 note 1: attention as communication
01:12:46 note 2: attention has no notion of space, operates over sets
01:13:40 note 3: there is no communication across batch dimension
01:14:14 note 4: encoder blocks vs. decoder blocks
01:15:39 note 5: attention vs. self-attention vs. cross-attention
01:16:56 note 6: “scaled” self-attention. why divide by sqrt(head_size)
Building the Transformer
01:19:11 inserting a single self-attention block to our network
01:21:59 multi-headed self-attention
01:24:25 feedforward layers of transformer block
01:26:48 residual connections
01:32:51 layernorm (and its relationship to our previous batchnorm)
01:37:49 scaling up the model! creating a few variables. adding dropout
Notes on Transformer
01:42:39 encoder vs. decoder vs. both (?) Transformers
01:46:22 super quick walkthrough of nanoGPT, batched multi-headed self-attention
01:48:53 back to ChatGPT, GPT-3, pretraining vs. finetuning, RLHF
01:54:32 conclusions

Corrections:
00:57:00 Oops “tokens from the _future_ cannot communicate”, not “past”. Sorry! 🙂
01:20:05 Oops I should be using the head_size for the normalization, not C

This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed, along with mental models of how to think about their “psychology”, and how to get the best use them in practical applications. I have one “Intro to LLMs” video already from ~year ago, but that is just a re-recording of a random talk, so I wanted to loop around and do a lot more comprehensive version.

Instructor
Andrej was a founding member at OpenAI (2015) and then Sr. Director of AI at Tesla (2017-2022), and is now a founder at Eureka Labs, which is building an AI-native school. His goal in this video is to raise knowledge and understanding of the state of the art in AI, and empower people to effectively use the latest and greatest in their work.
Find more at https://karpathy.ai/ and https://x.com/karpathy

Chapters
00:00:00 introduction
00:01:00 pretraining data (internet)
00:07:47 tokenization
00:14:27 neural network I/O
00:20:11 neural network internals
00:26:01 inference
00:31:09 GPT-2: training and inference
00:42:52 Llama 3.1 base model inference
00:59:23 pretraining to post-training
01:01:06 post-training data (conversations)
01:20:32 hallucinations, tool use, knowledge/working memory
01:41:46 knowledge of self
01:46:56 models need tokens to think
02:01:11 tokenization revisited: models struggle with spelling
02:04:53 jagged intelligence
02:07:28 supervised finetuning to reinforcement learning
02:14:42 reinforcement learning
02:27:47 DeepSeek-R1
02:42:07 AlphaGo
02:48:26 reinforcement learning from human feedback (RLHF)
03:09:39 preview of things to come
03:15:15 keeping track of LLMs
03:18:34 where to find LLMs
03:21:46 grand summary

Links
– ChatGPT https://chatgpt.com/
– FineWeb (pretraining dataset): https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
– Tiktokenizer: https://tiktokenizer.vercel.app/
– Transformer Neural Net 3D visualizer: https://bbycroft.net/llm
– llm.c Let’s Reproduce GPT-2 https://github.com/karpathy/llm.c/discussions/677
– Llama 3 paper from Meta: https://arxiv.org/abs/2407.21783
– Hyperbolic, for inference of base model: https://app.hyperbolic.xyz/
– InstructGPT paper on SFT: https://arxiv.org/abs/2203.02155
– HuggingFace inference playground: https://huggingface.co/spaces/huggingface/inference-playground
– DeepSeek-R1 paper: https://arxiv.org/abs/2501.12948
– TogetherAI Playground for open model inference: https://api.together.xyz/playground
– AlphaGo paper (PDF): https://discovery.ucl.ac.uk/id/eprint/10045895/1/agz_unformatted_nature.pdf
– AlphaGo Move 37 video:
– LM Arena for model rankings: https://lmarena.ai/
– AI News Newsletter: https://buttondown.com/ainews
– LMStudio for local inference https://lmstudio.ai/

– The visualization UI I was using in the video: https://excalidraw.com/
– The specific file of Excalidraw we built up: https://drive.google.com/file/d/1EZh5hNDzxMMy05uLhVryk061QYQGTxiN/view?usp=sharing
– Discord channel for Eureka Labs and this video: https://discord.gg/3zy8kqD9Cp

Educational Use Licensing
This video is freely available for educational and internal training purposes. Educators, students, schools, universities, nonprofit institutions, businesses, and individual learners may use this content freely for lessons, courses, internal training, and learning activities, provided they do not engage in commercial resale, redistribution, external commercial use, or modify content to misrepresent its intent.

Andrej Karpathy (co-founder of OpenAI, former head of AI at Tesla, and now founder of Eureka Labs) talks with Sequoia partner Stephanie Zhan at AI Ascent 2026 about what’s changed in the year since he coined “vibe coding.” He explains why he’s never felt more behind as a programmer, why agentic engineering is the more serious discipline taking shape on top of vibe coding, and why we should think of LLMs not as animals but as ghosts: jagged, statistical, summoned entities that require a new kind of taste and judgment to direct. He also touches on Software 3.0, the limits of verifiability, and why you can outsource your thinking but never your understanding.

00:00 Introduction
00:44 Feeling Behind as a Coder
02:28 Software 3.0 Explained
03:44 Agents as the Installer
04:49 Menu Gen vs Raw Prompts
07:37 What’s Obvious by 2026
09:41 Verifiability and Jagged Skills
13:39 Founder Advice and Automation
15:46 From Vibe Coding to Agent Engineering
25:17 Agents Everywhere and Learning

In this episode we dive into the story behind a R500M AI business and ask if South Africa could be breeding the next generation of Elon Musks.

Discover the vision, risks, and breakthroughs shaping the country’s place in the global AI race.

Learn how you can sign 5 high ticket clients in the next 3 months: https://www.theskoolofathens.com

Join our free community: https://www.skool.com/foundersacquisition-free

Join the newsletter, get 2 free guides + exclusive updates: https://theskoolofathens.com/guide

Follow our socials:

Hosts:
https://www.instagram.com/quintonweberr/
https://www.instagram.com/androokoen/
https://www.youtube.com/@quintonweber
https://www.youtube.com/@androkoen

Guests:
https://www.instagram.com/adamgamieldien/
https://www.instagram.com/isaac_nichxls/
https://www.instagram.com/gurveer.singh29/
http://www.youtube.com/ @adamgamieldien
http://www.youtube.com/ @IsaacNichols1

In this episode of ‘Europolis: the Global Europe podcast’, Chloe Teevan talks to Melody Musoni to explore how African innovators, researchers, and policymakers are shaping an AI ecosystem that responds to local realities. It highlights a shift from being consumers of technology to producers of homegrown solutions that are already transforming sectors like agriculture, health, and education.

The discussion tackles the two sides of the AI debate in Africa: the pessimists, who worry about risks like data bias and the push from Big Tech for profit , and the optimists, who see a chance to tackle long-standing challenges. A central theme is that African countries are determined not to be left behind in the Fourth Industrial Revolution as they were in the past.

More information: https://ecdpm.org/work/ai-africa-how-are-africans-using-ai-accelerate-development

What needs to happen before AI can take off in Africa?

Rose Mutiso talks us through the current state of energy and digital infrastructure in Africa, why leapfrogging is not guaranteed with AI, and what fundamental bottlenecks need to be addressed.

This is an episode from VoxDev’s new podcast series, Ideas in Development. You can listen wherever you get your podcasts:
– Apple Podcasts: https://podcasts.apple.com/us/podcast/ideas-in-development/id1866874059
– Spotify: https://open.spotify.com/show/6sIdIKctE8frdWaz9iyfl2
– Everywhere else: https://audioboom.com/channels/5165629-ideas-in-development
– YouTube Playlist: https://www.youtube.com/playlist?list=PLcqy-QRDq-vD3YJ2t1rMUwx8BN1WTEA9A
– Substack: https://ideasindevelopment.substack.com/

#ai #africa #development #podcast #energy

In this powerful TEDx talk, ethnographic researcher Dr. Najeeb Abdulhamid explores a vital question: How might we leverage AI to empower Africa’s workforce for the future? As co-author of AI and the Future of Work in Africa whitepaper, Dr. Najeeb offers a grounded, thought-provoking perspective on the opportunities and risks that generative AI brings to the continent. From transforming agriculture and healthcare to supporting rural entrepreneurs, he envisions a future shaped by AI for Africa, by Africa—but only if we invest in the right skills, infrastructure, and inclusive policies. Najeeb G. Abdulhamid is an ethnographic researcher at Microsoft Research Africa (MSR Africa), based in Nairobi, Kenya. His work is dedicated to understanding how people interact with technology and leveraging these insights to develop actionable design solutions that empower workers.

Najeeb earned his PhD in Computer Science from Brunel University(opens in new tab), London, and holds an MSc in Information Systems from the University of Aberdeen(opens in new tab), Scotland, UK. His academic journey has equipped him with a profound understanding of the technological landscape. This talk was given at a TEDx event using the TED conference format but independently organized by a local community. Learn more at https://www.ted.com/tedx