10:03:00 From Chen Da : Professor I can’t hear your clearly 10:03:07 From Chen Da : you* 10:03:35 From Roman Haberli : I can hear you. 10:03:46 From Chen Da : Better now 10:03:47 From Chen Da : thanks 10:04:25 From Ziyan Lin : hey professor, your voice keeps on cutting off for some reason 10:04:28 From Kiara Chen : I think the internet connection is unstable? 10:04:34 From Chen Da : yeap 10:04:36 From Mark Bacon : Zoom is saying the bandwidth is low so it might be the low latency 10:04:39 From Chenhe Gu : Hey yeah its very choppy 10:04:43 From Chunxiao Wang : Yeah same here 10:04:45 From Chloe Li : same here 10:04:45 From Andy Huang : same here 10:04:47 From Emma Saros : Same here 10:04:48 From Nicole Cabalquinto : agreed, it still sounds choppy 10:04:52 From Ashley Nussbaum : same here 10:04:55 From Frank Fan : I think there is some Internet issue 10:05:42 From Ashley Nussbaum : maybe turn off your video? 10:06:01 From Edward Cheung : Same issue 10:06:06 From Francis Zhan : still choppy 10:06:39 From Kiara Chen : ^yeah, still choppy 10:07:02 From Kenneth Perlin : https://mrl.nyu.edu/~perlin/courses/fall2020/ 10:07:03 From Hasnat Abrar : I can barely make out the sentences, anyone else? 10:07:11 From Jacqueline Pham : yes, it's hard to understand what you are saying because of how choppy it is 10:07:12 From Ghadeer Ghosheh : Yes same 10:07:14 From Frank Fan : every other words cutting itself off 10:07:19 From Chen Da : Only can hear single word 10:08:24 From Nicole Cabalquinto : :\ maybe worth restarting zoom? 10:11:01 From Chunxiao Wang : professor, do you mind typing? I can barely understand you 10:11:15 From Hasnat Abrar : Yeah this is kind of unlistenable... 10:11:17 From Kenneth Perlin : I can try typing into chat 10:11:23 From Chunxiao Wang : Thank you 10:11:24 From Ziyan Lin : ^^^ we can not hear it 10:11:26 From Ziyan Lin : thank you! 10:11:26 From Hasnat Abrar : thanks!! 10:11:34 From Kenneth Perlin : A I speak I will type 10:11:49 From Kenneth Perlin : Weekly assignments 10:11:53 From Kenneth Perlin : Programming heay 10:11:56 From Kenneth Perlin : heavy 10:12:03 From Kenneth Perlin : You should be a good programmer = 10:12:15 From Kenneth Perlin : Mostly javascript and Web 10:12:25 From Kenneth Perlin : Last two weeks, final projects 10:12:30 From Kenneth Perlin : You get to choose that. 10:12:33 From Kenneth Perlin : Final project presentations 10:12:39 From Kenneth Perlin : Dec 7 and 9 over zoom 10:12:43 From Kenneth Perlin : Hopefully with good internet. :-) 10:12:49 From Kenneth Perlin : Is this working 10:12:54 From Jacqueline Pham : yes 10:12:56 From Nicole Cabalquinto : Yes! 10:12:56 From Roman Haberli : yes 10:12:56 From Ziyan Lin : yesss 10:12:58 From Chenhe Gu : yess 10:12:58 From Frank Fan : yes 10:12:58 From Chen Da : yes 10:13:02 From Kenneth Perlin : We will get through this 10:13:09 From Kenneth Perlin : topics 10:13:19 From Kenneth Perlin : There are 3 major parts to graphics: 10:13:24 From Kenneth Perlin : 1 modeling 10:13:26 From Kenneth Perlin : 2 animation 10:13:28 From Kenneth Perlin : 3 rendering 10:13:35 From Kenneth Perlin : I will show visuals now 10:13:50 From Mark Bacon : Definitely better! When you say we get to choose, are you saying it's a group project or a project that we get to choose? 10:14:08 From Kenneth Perlin : We will probably have 10:14:15 From Kenneth Perlin : Individual final projects 10:14:19 From Kenneth Perlin : We will see how it goes 10:14:42 From Kenneth Perlin : A scene consists of shapes 10:15:06 From Kenneth Perlin : Can everyone see tha 10:15:25 From Kenneth Perlin : Is that visible? 10:15:28 From Ashley Nussbaum : no 10:15:29 From Mark Bacon : If you mean there is something on screen now, I don't see anything 10:15:29 From Ziyan Lin : I can not see 10:15:29 From Kiara Chen : What are we supposed to be seeing? 10:15:31 From Chunxiao Wang : I don’t think so 10:15:31 From Hasnat Abrar : I only see your video professor 10:15:42 From Ashley Nussbaum : yes 10:15:44 From Mark Bacon : There it is! 10:15:46 From Emma Saros : now its there 10:15:47 From Kiara Chen : ok 10:15:47 From Chen Da : yes 10:15:47 From Hasnat Abrar : yessir 10:15:50 From Frank Fan : yes 10:15:52 From Roman Haberli : yes 10:15:54 From Linsong Hao : It’s there now 10:16:35 From Kenneth Perlin : A shape is approximated by triangles 10:16:42 From Kenneth Perlin : For example, a sphere 10:17:05 From Kenneth Perlin : We render one triangle at a time 10:17:36 From Kenneth Perlin : Every pixel “sees” some little piece of the triangle 10:17:56 From Kenneth Perlin : That little piece is called a “fragment” 10:18:14 From Kenneth Perlin : The computer has both a cpu (central processing unit) 10:18:20 From Kenneth Perlin : And a gpu (graphics processing unit) 10:18:50 From Kenneth Perlin : Most of the work of making a picture 10:19:03 From Kenneth Perlin : Is done on the gpu, which is a special purpose processor 10:19:10 From Kenneth Perlin : All phones and computers have a gpu 10:19:19 From Kenneth Perlin : There are two parts to rendering: 10:19:33 From Kenneth Perlin : 1 transforming vertices of triangles so we can animate them 10:19:52 From Kenneth Perlin : 2 figuring out the color at each each pixel fragment that the triangle covers on the image 10:20:26 From Kenneth Perlin : Modern gpu let you write a special purpose program 10:20:36 From Kenneth Perlin : To transform the vertices (the “vertex shader”) 10:21:02 From Kenneth Perlin : And to scompute the right color for each fragment — the “fragment shader” 10:21:18 From Kenneth Perlin : Our first assignments will be focusing on the fragment shader 10:21:38 From Kenneth Perlin : Then we will go back up and look at how shapes are turned into triangles 10:21:52 From Kenneth Perlin : And how triangle vertices are transformed to animated shapes 10:22:04 From Kenneth Perlin : You will be building a simple complete graphics system 10:22:12 From Kenneth Perlin : From high level shapes to final image 10:22:18 From Kenneth Perlin : This process of going from 10:22:23 From Kenneth Perlin : High level description to 10:22:27 From Kenneth Perlin : Primitive shapes 10:22:33 From Kenneth Perlin : To triangles 10:22:40 From Kenneth Perlin : To animating the vertices of triangles 10:22:46 From Kenneth Perlin : To rendering each pixel 10:22:52 From Kenneth Perlin : Is called the “graphics pipeline" 10:23:02 From Kenneth Perlin : So you will this semester implement a complete graphics pipeline 10:23:14 From Kenneth Perlin : It’s sort of like learning about cars by building a car from the ground up. 10:23:20 From Kenneth Perlin : Any questions at this point? 10:23:54 From Kenneth Perlin : To implement the vertex and fragment shaders, you will be programming in a special language that runs on the GPU. 10:24:11 From Kenneth Perlin : It runs thousands of times faster than anything you could run on your CPU 10:24:30 From Kenneth Perlin : Your GPU is basically s special purpose supercomputer built into every modern computer and smartphone. 10:25:02 From Kenneth Perlin : The one on your computer is more powerful than the one on your phone, because GPUS draw a lot of electric power, and Smsrtphones only have little batteries. 10:25:33 From Kenneth Perlin : The GPU revolution also brought about modern machine learning, because they allow this kind of super powered computation that ML needs. 10:25:47 From Kenneth Perlin : Questions? 10:26:05 From Kenneth Perlin : Basically, computer graphics is a way of turning math into art. 10:26:16 From Kenneth Perlin : So to do anything, we need a math framework, 10:26:20 From Kenneth Perlin : Which means a coordinate system. 10:26:51 From Kenneth Perlin : This s just a convention. 10:27:04 From Kenneth Perlin : Computer games use a different convention — where z is up. 10:27:19 From Kenneth Perlin : We are using the convention favored by physicists and mathematicians: 10:27:28 From Kenneth Perlin : x right , y up , z forward. 10:27:35 From Kenneth Perlin : It’s called a “right handed coordinate system, 10:28:16 From Kenneth Perlin : Because I can use my right hand to figure out which way it goes. 10:28:24 From Kenneth Perlin : Games use a left handed coordinate system, 10:28:36 From Kenneth Perlin : So there seems to be a culture war. :-( 10:28:51 From Kenneth Perlin : Questions? 10:28:57 From Kenneth Perlin : Can everybody follow? 10:29:02 From Francis Zhan : yep! 10:29:02 From Chenhe Gu : yes 10:29:03 From Andy Huang : what directions are x y z in games? 10:29:04 From Ledion Lecaj : Yes 10:29:04 From Chunxiao Wang : yes 10:29:06 From Mark Bacon : So far so good! 10:29:08 From Edward Cheung : yes 10:29:15 From Nicole Cabalquinto : good so far! 10:29:17 From Chen Da : Yes 10:29:20 From Kenneth Perlin : Great! Sorry for the awkward interface. We will muddle through! 10:29:23 From Mark Bacon : Z up, X right, Y forward in games (I believe) 10:29:25 From Mark Bacon : @Andy 10:29:33 From Andy Huang : thanks! 10:29:45 From Kenneth Perlin : Yes, that’s right. Games switch z and y compared with what I showed. 10:30:04 From Kenneth Perlin : There are different ways to render an image. 10:30:13 From Kenneth Perlin : You can optimize for power and generality 10:30:18 From Kenneth Perlin : Or you can optimize for speed. 10:30:39 From Kenneth Perlin : We will for the first few weeks use a method thot optimizes for power and generality 10:30:44 From Kenneth Perlin : Called “ray tracing: 10:30:58 From Kenneth Perlin : Because it will help you to learn the principles 10:31:17 From Kenneth Perlin : We will build our ray tracer entirely in the fragment shader of the gpu 10:31:20 From Kenneth Perlin : Then after a few weeks 10:31:32 From Kenneth Perlin : We will switch to a faster but lest general method, 10:31:45 From Kenneth Perlin : Which it the method used by most people these days in the industry. 10:31:53 From Kenneth Perlin : Of the major animation studios, 10:32:09 From Kenneth Perlin : Pixar and Dreamworks use the faster method, which is called “z-buffering”. 10:32:20 From Kenneth Perlin : Only Blue Sky (which is the only major animation studio in NY) 10:32:26 From Kenneth Perlin : Uses raytracing. 10:32:40 From Kenneth Perlin : That’s why their animations look different, and sometime richer visually, 10:32:49 From Kenneth Perlin : For example, the “ice age” movies. 10:33:05 From Kenneth Perlin : Nearly all computer games use z-buffering, because they need speed 10:33:15 From Kenneth Perlin : Unlike movies, games need to be rendered at 60 frames per second 10:33:27 From Kenneth Perlin : That is a hard constraint, and drives a lot of design decisions. 10:33:44 From Kenneth Perlin : Roughly speaking, computer animation is split into two groups: 10:33:52 From Kenneth Perlin : (1) People who make special effects for movies and TV 10:33:57 From Kenneth Perlin : (2) People who make 3D computer games 10:34:14 From Kenneth Perlin : There are many similarities, and the principles are all the same 10:34:28 From Kenneth Perlin : But in a Star Wars movie, you can spend an hour computing one frame 10:34:49 From Kenneth Perlin : Whereas in a Star Wars video game, you have only 1/60 second to compute a frame 10:35:03 From Kenneth Perlin : That is why movie special effects look so much better than computer games 10:35:12 From Kenneth Perlin : Any questions or comments? 10:35:17 From Kenneth Perlin : Comments are welcome1 10:35:42 From Kenneth Perlin : There is now, for the first time, real crossover between the two industries. 10:35:47 From Kenneth Perlin : Who has seen The Mandalorian? 10:35:55 From Kenneth Perlin : anybody? 10:35:56 From Hasnat Abrar : mr 10:35:58 From Hasnat Abrar : me 10:36:03 From Bruce Ping : first episode 10:36:27 From Kenneth Perlin : It is significant because the backgrounds are real time computer graphics shot with the actors right on the set 10:36:35 From Kenneth Perlin : On giant screens behind them. 10:36:46 From Kenneth Perlin : For the director, John Faveau, this is awesome, 10:37:03 From Kenneth Perlin : Because it’s like having a real life set, but you can change the set instantly 10:37:19 From Kenneth Perlin : So our entire computer graphics industry is now going through a major change 10:37:38 From Kenneth Perlin : Learning real-time graphics is no longer just importnat 10:37:43 From Kenneth Perlin : If you want ao job in computer gamees. 10:38:02 From Kenneth Perlin : For the first time, it is becoming important for some parts of the special effects industry. 10:38:15 From Kenneth Perlin : This will have a major impact on your career choices 10:38:39 From Kenneth Perlin : Also, virtual reality is going to be mainstream by the time many of you graduate, 10:38:54 From Kenneth Perlin : And will be integral to many industries in the next few years, 10:39:05 From Kenneth Perlin : Just as the Web became integral to everything by 1995 10:39:16 From Kenneth Perlin : And SmartPhones became integral 10:39:20 From Kenneth Perlin : To everything by 2010. 10:39:23 From Kenneth Perlin : Comments or questions? 10:39:34 From Chunxiao Wang : Is vr realtime computed? 10:39:34 From Ledion Lecaj : Are VR graphics different? 10:39:41 From Kenneth Perlin : Good question! 10:39:55 From Kenneth Perlin : VR graphics are real-dime graphics, as in computer games. 10:40:07 From Kenneth Perlin : The major technical difference is that you need to computer *two* images 10:40:13 From Kenneth Perlin : In one animation frame. 10:40:30 From Kenneth Perlin : Some VR systems have 60 frames per second (eg: the Oculus Quest) 10:40:40 From Kenneth Perlin : Others have 90 frames per second (eg The Vive) 10:40:51 From Kenneth Perlin : Some have 120 frames per second (I think the Pixel) 10:41:03 From Kenneth Perlin : Faster frames means more lifelike VR 10:41:17 From Kenneth Perlin : But also requires more power hungry and expensive GPUs. 10:41:21 From Kenneth Perlin : So it’s a tradeoff. 10:41:34 From Kenneth Perlin : I love the Quest because even though the image quality and frame rate is not as good 10:41:44 From Kenneth Perlin : I can just throw tin my bag and take it anywhere. 10:41:55 From Kenneth Perlin : It’s kind of like the SMartPhone of VR. 10:42:06 From Kenneth Perlin : I think that will become more common as VR becomes more widespread. 10:42:10 From Kenneth Perlin : Thoughts or comments? 10:42:19 From Bruce Ping : So what happens if a pervious made game tries to make a vr version of the same game? 10:42:29 From Kenneth Perlin : Good question. 10:42:29 From Bruce Ping : like borderlands 2 if anyone knows 10:42:47 From Kenneth Perlin : The porblem facing game designers who already made a 3D game 10:42:52 From Kenneth Perlin : And want to move them to VR, 10:43:00 From Kenneth Perlin : Is that VR gives you many more design choices. 10:43:13 From Kenneth Perlin : The gamer player is immersed in what feels like an entire world 10:43:18 From Kenneth Perlin : Rather than looking at a screen. 10:43:30 From Kenneth Perlin : I can tell you from experience it is a vey differenti thing. 10:43:42 From Kenneth Perlin : A lot of design of traditional computer games is to 10:43:48 From Kenneth Perlin : Compensate for the fact that 10:43:54 From Kenneth Perlin : You are looking into a screen. 10:43:59 From Kenneth Perlin : You need to rethink everything, 10:44:11 From Kenneth Perlin : If you are basically presenting the experience 10:44:19 From Kenneth Perlin : Of the characters in Ready Player One, 10:44:24 From Kenneth Perlin : Which is what VR gives you. 10:44:36 From Mark Bacon : When it comes to VR, is there any sort of special graphical/programming consideration (outside of visual design) since you need to properly render images so that we can focus on it properly bifocally? Or is that all done with the lenses/post-processing and not much on the rendering side of things? 10:44:40 From Mark Bacon : sorry- should have waited 10:44:58 From Kenneth Perlin : Those companies that have retooled games for VR put a lot of work into that rethinking — for example, Valve turning Half Life into Alyx 10:45:26 From Kenneth Perlin : Generally speaking, the VR software plafform does that for you. 10:45:37 From Kenneth Perlin : Unless you are the person implementing that low level platform. 10:45:54 From Kenneth Perlin : These days there are already standard packages and standard APIs out there, so you don’t need to do that yourself. 10:46:24 From Kenneth Perlin : You really just need to change a few lines of code to render you 3D graphics in VR, if you are using a standard API. 10:46:49 From Kenneth Perlin : I do VR In game engines — Unity and Unreal — and also on the Web usingg WebXR 10:47:03 From Kenneth Perlin : And I dojn’t need to do a lot of technical work to get things working in VR 10:47:12 From Kenneth Perlin : Because the support is built into the API. 10:47:19 From Kenneth Perlin : Does that answer your question? 10:47:25 From Mark Bacon : Yes, very thoroughly! Thank you! 10:47:31 From Kenneth Perlin : :-) 10:47:47 From Kenneth Perlin : I will give you an overview of the concepts of ray tracing. 10:48:09 From Kenneth Perlin : First you need a model of the user looking into a scene 10:48:37 From Kenneth Perlin : Think of your computer screen floating in some imaginary 3d world 10:48:49 From Kenneth Perlin : And your eye is looking at that screen in that world 10:48:56 From Kenneth Perlin : We need to model mathematically what happens 10:49:07 From Kenneth Perlin : When light from one pixel gets to your eye 10:49:25 From Kenneth Perlin : The trick with ray tracing 10:49:31 From Kenneth Perlin : Is to trace the light back from your eye 10:49:35 From Kenneth Perlin : Through that pixel, and 10:49:39 From Kenneth Perlin : See where the ray came from. 10:49:50 From Kenneth Perlin : Each object in the scene needs to be ready 10:50:03 From Kenneth Perlin : To report “hey, that ray hit me” 10:50:08 From Kenneth Perlin : (Not in a violent sense) 10:50:15 From Kenneth Perlin : And here is the point of intersection. 10:51:06 From Kenneth Perlin : I think there is a problem with the software :-( 10:51:36 From Kenneth Perlin : For example, if a ray moves toward a sphere in the scene 10:51:48 From Kenneth Perlin : We need a mathematical model of “ray hits sphere” 10:51:51 From Kenneth Perlin : There are two cases: 10:51:56 From Kenneth Perlin : (1) The ray misses the sphere 10:52:00 From Kenneth Perlin : (2) the ray hits the sphere 10:52:09 From Kenneth Perlin : If the ray misses, we don’t need to do anything 10:52:19 From Kenneth Perlin : If the ray hits, we need to know how far along the ray 10:52:24 From Kenneth Perlin : Was that intersection 10:52:31 From Kenneth Perlin : If you think bout what the user sees at that pixel 10:52:38 From Kenneth Perlin : Here is the basic question: 10:52:48 From Kenneth Perlin : What spheres did the ray hit, and 10:52:57 From Kenneth Perlin : Which of those spheres was nearest to the eye? 10:53:06 From Kenneth Perlin : That will be the one that is seen at that pixel fo the image 10:53:29 From Kenneth Perlin : Here I drew two spheres 10:53:39 From Kenneth Perlin : One is nearer, so it blocks the view at this pixel of the other sphere 10:54:13 From Kenneth Perlin : I have highlighted the pixel in red, 10:54:25 From Kenneth Perlin : Showing what the user sees when we try to render these two spheres 10:54:34 From Kenneth Perlin : Notice that the further away sphere is not visible at this pixel 10:54:45 From Kenneth Perlin : Because it is blocked by the nearer one. 10:55:03 From Kenneth Perlin : Yo u will be implementing this in the next two weeks. 10:55:16 From Kenneth Perlin : What I am going to show you is the math you need to render scenes using ray tracing. 10:55:49 From Kenneth Perlin : We just described at a high level how to (1) figure out what spheres were hit and (2) how to figure ouch which one is visible. 10:55:59 From Kenneth Perlin : We will do this at every pixel of the image, to render one frame. 10:56:20 From Kenneth Perlin : If we move the spheres between frames, then we can string together all of those frames to make an animation. 10:56:36 From Kenneth Perlin : We still have not covered how to figure out the colors at every pixel. 10:56:44 From Kenneth Perlin : Any comments or questions at this piont? 10:56:56 From Kenneth Perlin : Is it clear so far? 10:56:59 From Andy Huang : what happens if the first sphere is partially transparent? 10:57:07 From Kenneth Perlin : Good question! 10:57:21 From Kenneth Perlin : If the first sphere is partially transparent, there is an easy case 10:57:24 From Kenneth Perlin : And a hard case 10:57:34 From Kenneth Perlin : The easy case is thin films like soap bubbles. 10:57:46 From Kenneth Perlin : In that case we just mix together the colors form th two spheres 10:57:54 From Kenneth Perlin : We will go over how to do that at a later week. 10:58:07 From Kenneth Perlin : The hard case is if the first sphere is solid glass or plastic 10:58:17 From Kenneth Perlin : Because then we need to use Snell’s law to refract the ray 10:58:28 From Kenneth Perlin : That produces awesome results, but the math is harder. 10:58:47 From Kenneth Perlin : It gets really cool hewn you use Snell’s law on gems like diamond and rubies and emeralds 10:58:53 From Kenneth Perlin : You get spectacular images. 10:59:09 From Kenneth Perlin : We will go over refraction as welll, but in a few weeks from now. 10:59:20 From Kenneth Perlin : It’s all just math. Do not be afraid. :-) 10:59:53 From Roman Haberli : Do we need any textbooks for the course? 11:00:05 From Chen Da : No we don’t have 11:01:00 From Chloe Li : I think the connection is getting better. 11:01:05 From Carter Fodge : ^^ 11:01:07 From Ziyan Lin : you can pull up chat again 11:01:08 From Ziyan Lin : from the bottom 11:01:23 From Kiara Chen : ^Yeah, I can hear pretty decently right now 11:01:31 From Chenhe Gu : ^^full sentences 11:01:33 From Kenneth Perlin : When I shared the screen, 11:01:41 From Kenneth Perlin : My chat window disappeared 11:01:45 From Kenneth Perlin : I needed to find it again. 11:01:47 From Kenneth Perlin : Sorry about that 11:02:00 From Mark Bacon : Yep! 11:02:00 From Ashley Nussbaum : yes 11:02:01 From Ledion Lecaj : Yes 11:02:02 From Nicole Cabalquinto : connection is much better! yay! 11:02:02 From Matt Vinca : yep 11:02:02 From Chenhe Gu : Seeing it 11:02:02 From Kiara Chen : yes 11:02:02 From Chongkun Zhao : yes 11:02:02 From Emma Saros : yes 11:02:05 From Roman Haberli : Yes 11:02:05 From Ziyan Lin : yes 11:02:12 From Chloe Li : I think we can hear you pretty well now 11:02:36 From Ashley Nussbaum : yes 11:02:37 From Chunxiao Wang : yes 11:02:37 From Carter Fodge : yes 11:02:37 From Chen Da : yep 11:02:38 From Kiara Chen : Much better 11:02:38 From Roman Haberli : better 11:02:38 From Bruce Ping : yes 11:02:39 From Frank Fan : yes 11:02:39 From Chongkun Zhao : yes 11:02:39 From Nicole Cabalquinto : yes! 11:02:39 From Emma Saros : Much better 11:02:40 From Ziyan Lin : yes 11:02:47 From Mark Bacon : Audio much better, still says low bandwidth on Zoom though 11:03:00 From Frank Fan : i can get a full sentence right now