Mati Roy, OpenAI: AGI, ASI, life extension, lifelogging, and the future of humanity

guys welcome to another CEO founder wisdom podcast not even sure how to name that podcast because we're gonna talk about a bunch of thing I did these uh not so far future um episodes last year and I think that one would definitely be a bit label of that could also qualify as an AI wisdom podcast longevity wisdom podcast we have Matthew to date um it's a special

guest because him and I we shared a lot of things um in common just that has yet like way more uh tokens in in most of these fields which we're gonna talk about but before I do mathie can you uh quickly introduce yourself and tell us a bit more about uh Your Role at open Ai and a pen task um so I've been interested in uh yeah the field of chronics and AI for for a while and I'm mostly focusing on AI right now I'm working at open AI on the human feedback team I do up uh mostly with quality control and uh uh giving the instructions about to for our team of labelers so we just have addressed the people that will give feedback to the AI or about uh which outputs is preferred

and and at Fantastic uh that's a business that I run I do mostly Generations these days and I do do it for organization and around in effects of altruism and chronix mostly yeah yeah love it okay so first thing um you and I shared you know the radical uh transparency thing so I I want to start this interview like in like oncoming ways my first impression of

you first I was reaching out uh to open AI stuff and I don't know if it's a long hair or whatever but and GP Sears uh kind of guy but I was like okay this guy might be arrogant or something I don't know why but that occurred in my brain you have this uh a knowledge of like why do you think it's like long-haired dude working at open air okay he's special why are my instinct why did my instincts tell me you'd be like um intimidating in some way uh I really like that you shared this with me yeah I feel like a lot of people wouldn't want to share this kind of the thing yeah uh that's interesting to me like I'm like huh like that's kind of surprising and I get I don't

really get that like uh I often get the the opposite um so yeah I don't really know um and the second question is like related to that the second and the the third um I've got this weird thing where I always talk [ __ ] about like Quebec uh but I feel myself attracted to these Founders from Montreal and I I don't really know why at well country is definitely not its citizens and obviously we share a culture but that that's also another thing um I don't want an answer on that one necessarily but the third one I do which is I've been reading your stuff and word like passionate about the same thing but you're like level 10 on pretty much everything I I kind of felt intimidated by those like this dude is

and yeah it's you know evolutionary Terry and the the competitor in me I also cooperate a lot as we're doing today but why why do I I feel you're like next level in anything for example in the prediction thing like I've been especially this year uh since I came back from Vegas last year I read the moloch meditation by the way yesterday and I left it uh

at the Vegas part but I really had a bad impression of of Vegas it's like over capitalistic whatever but since I came back from there I've been like pretty bold on my predictions I've been doubling down on that and I even want to put money um on these which will be a subjective conversation but do you ever like look at people in in like a field that

you are passionate about that are like 100 times higher than you are and feel like [ __ ] I I'm like a bug when it comes to comparing myself to them um so so yeah a few thoughts about this like sometimes I'm like oh man I feel really satisfied about where I am and I'm like and then I'm like what no I shouldn't I should always aim for for more like why am I satisfied like um like are there are people like that I look up you like or do you like did you said this Friday if they were where I was so like I I re I tried to see like oh I could emulate them or uh what's the next thing for me to reach and uh yeah right like yeah right now

like the first uh recall a specific thing that I'm recalling which I don't know if it's a representative of our ipf generally but there's someone that's very similar to me so like like I write to them they're into a lot of the things that I mean too they have very similar values so maybe that's why I'm don't feel didn't feel jealousy towards them like maybe I would

feel I don't know why I didn't feel jealousy basically but I felt I felt love like like like I'm you know it's like weird to love someone that you don't really know or but yeah I felt like loving that and like yeah well it's the same it's the same on my side like and that was just like one thought out of many because I I really worked the the jealousy thing and for example three years ago I thought I'd had zero jealousy left in me after like all of this inner working with coaches therapies and so forth and yeah I realized that sometimes I still have it so it's like one microcosm of my pole reaction it it's mostly like excitement to to speak today because you have got like a lot of topics to

to talk about but what I'm referring to is for example your website um that's where I want to go you know like uh notion looking website like a Wiki um sharing pretty much everything I've had various websites and various uh platforms I just wish it would be that way then the prediction thing I'm like [ __ ] I need to start doing that as well because I

if you're on my LinkedIn there's a bunch of predict prediction um I started including some numbers sometimes ago even before I looked your things up so my predictions did look uh around that when there's a year um and start let's start with this your Ultra transparency and taking notes on everything and life vlogging what are the pros and the cons to that because you also film like

every second of your life well apparently um I I don't think I'd go to that extreme I don't know where you store all of that basically you paid Fortune for like gpus and well not gpus but just storage space or whatever maybe you have a solution to that um why did you decided to do that in the first place and what are the pros and the cons of doing that yeah uh yeah some some people are like into the live streaming part other like like your life vlogging like I'm I'm into both I do I'm pretty I'm extreme life vlogger I not at all an extreme live streamer but I've tried before like I could very uh well I've been an extreme life uh streamer but a lot of the things and I don't like

it I aesthetically like before even talking about like evaluating uh the pros and cons just just like from a pretty aesthetic point of view like I'm attracted towards uh like I transparency live streaming live vlogging and and sometimes it's just sometimes really hard to make it compatible with uh like having consent from everyone else a member with like I can live stream if I'm working for a

company developing uh private information private IP and and like I also like how do I end all uh communicating with everyone else I tried like setuping an email address that was public I'm like okay if you email me to this email address I might even prioritize like answering this email address but it's going to be public it's not everybody being used like it it failed to take

off um but I might still try other approaches to live stream more um what if a stalker like starts making disastrous use of that information and what if like and just you know basic um person starts feeding all of that data to a GPT and try to reproduce you which is already possible today have you thought about these scenarios yeah uh I feel like it's really hard

to like figure out the right answer like just I feel with like or did I like I feel a lot of it is intuition and a personal like inclinations but yeah I I guess I didn't think that much I like about the short-term personality theft like through imitating my behavior but like if other people are gonna make backup of my data that that can be like like

there's kind of a drug of like oh yeah I'll just parse some keywords for the NSA to keep the log of my information as an extra backup like uh so in the future if you know if my if I lose my information there's I mean that's I'm not very serious about about this but um yeah I guess and also I want to be very respectful of other people's data so like I'm a lot of the security that I'm doing is like because other people care more than I do and right because you do live vlogging for Life Extension purposes mostly I would guess that's at least like 60 the reason why you do it right yeah I would say like that was uh at first like that was why I wanted to do it I

was like okay I it's at first I was like this is really weird so weird like uh but I couldn't really tell that the argument for it was wrong I was like I guess the argument makes sense I guess I'll just record my life like okay that's such a weird world like what why is the why do I keep discovering those weird uh things that goes against uh like what everyone else is doing and makes me look weird I'm like but yeah and and now it doesn't feel weird now like I feel like really emotionally like a tune I'm like if I'm not recording like I can feel it in my body like oh something is being lost like I feel much closer to it emotionally and and when I was I tend to like

almost overthink but I feel like that for this one it was good to put a lot of time into thinking like the pros and cons of lifelocking because a lot of people are not lifelong games so like maybe they're a bunch of cons uh for of that that that's the reason why so like I made a list of all the pros I could think and all the cons I could think on my website um in addition to the main one which was a life extension and yeah I do I do I would probably still like Vlog even if if I stop thinking that I think it was useful for extra extension maybe I would be less extreme about it but uh but yeah even in the short term to be able to refer to I

find it interesting yep I find it uh let's take the agent's uh GPT turn nowadays that I think this Tech will be ripe pretty soon we've saw nice prototypes um definitely and I think that before the bad comes the good comes because good people are usually smarter than bad um and there's gonna be ways to identify the Charles GPT for example the question is how do you

build the Charles GPT well to me I just have to export my LinkedIn and my Twitter and give more weight to my last post and for the AI to calculate the progress between these polls because I did change a lot you know and it's funny to thing that at 17 we thought we knew any everything about the world and we never change and we had so much ego but with time you you understand that people change you know and that the thing that you start growing after 18 mentally is complete rubbish you know I expect to my mindset to change like 10 100x fold the next couple of years but just if I feed that first I think that's just giving back to the world and it's like dude everything's in there so I might

say some dumb [ __ ] for sure but then you can you can judge me really like on if I change or not because we've got such a an unaccurate portrait of an online person right when you write yesterday I was checking Brian Johnson she's a biohacker and a futurist some comments on him and people judging him on one picture oh you look pale you look unhappy

you know this is not working for you but it's such an unaccurate portrait and I think you and I one of our goal is also to get to the truth um not sure how you call those people truth maximalist or whatever but I think it's the fastest way to advance humanity and just because of all of those reasons I think that life vlogging and being public about things makes makes way more sense than anything else no yes people are gonna try to abuse of it I I'm picturing a future in which you know it's like deeper Biometrics so you having like Bots inside you that authenticate with another bot with the device that you're using to authenticate that it's it's you or a bunch of other like interesting things short term you know if I

would have the domain Charles cormiergpt.com and I reach out to you via this email you might know it's me but there's also a thousand ways uh creative ways to [ __ ] up someone with their information and me and the last couple of years I've been a victim of that so that that's why also think about these things but what did you think about the agent uh

GPT demo of like strategizing tasks and starting to show some forms of execution of them um yeah I I'm like At first I was like so like I'm part of the the last one Community uh people that care about the extensional risk related to Ai and people used to to talk about like uh once we are near AGI we put the AGI in a seal box disconnected from the internet with like a like one channel of communication with a team that like knows how to interpret the the what the AJ is saying and and now we're like and now everyone is like let's connect the AI to everything everywhere and let's give it all the power I'm like well okay maybe we were a little bit too idealistic uh that's not at all other world

Works uh there will there's a bunch of people every there's it only takes a few groups you know to want that to be able to do that it seems unless that it's gonna get regulated so right now I'm like uh I mean we know that you know GPT is not gonna doesn't doesn't seem like it's buzzing any near anything near like extensional threat so and so I guess yeah maybe it's good for it to just let it do the damages and they may end the good things that it's going to do like sooner that's that's also like that's generally what open people's open eye seems to be doing like let's test early and see like what what happens and then maybe it's going to give more time to Regulators to to adapt um because yesterday

I reread um Tim Urban's part two post about like AI right um it's you suggested the part one on your post somehow I thought I thought um the part two was a bit more interesting I still well I mean first it's really impressive that he wrote that eight years ago I mean that's right that's flabbergasting um that's when open air was started yeah yeah okay yeah which

makes sense uh second I still think that there's major lacks lack of creativity and or human brain about what would happen with with AI and there's a strong reason for that what's the way to augment the neural Connection in your brain nowadays other than bio hacking right not a lot there's not a lot of ways for us to get smarter yes we can debate philosophically about topics

but truth is if we try to reason with history we don't have similar data we don't have it we can try to draw some parallels and try to reason try to predict that way which leads us to a nice segue of the prediction topic that I had for this conversation but my thesis on AGI is that well I'm more of a positive type of guy and I'm an ultra libertarian I do think if that if caging that thing is and anthropal morphizing the thing and it's just like or shitty human nature right like let's cage a tiger um I think that if we leave it by itself and I mean it's it's kind of a weird thing to say but just let it decide what to do with us because we're not we're not good

but well I mean we have good in us but we have this humongous negative downside about us humans like where capital is like I'm gonna step on your foot and if I need to and so forth however good I am inside um so that's one part of me it's like okay if this thing is gonna decide a very fate uh so be it second I think it's unavoidable we've already released the Tiger and the the last thesis and the most interesting I had just about listening to was okay maybe we should try to pause the thing or at least like a better suggestion than the open uh letter to pause things for six months but yet a better suggestion to it and meanwhile why don't we advance like the brain science so that we can

get smarter and be better at understanding and predicting stuff what do you think about that like you mentioned like uh humans having some uh shortcomings and maybe maybe AI like not having those shortcomings I think I think that like uh there's a lot of possible Minds that we could create like the the possibility of Mines is extremely vast humans are only covering a small fraction of the

possibilities of brains that we can create in this universe and and I think most possible Minds that we could create uh unless unless like the purposely programmed like to care about humans if they they're probably not gonna uh it seems likely that they're gonna care about some other goals and that humans are just going to become in the way of that goal it's not that they're going

to be mean in the same way a human is egoistic it's just going to be like not helpful for for their goal like uh I think what give us our emotional architectures uh framework that that we have is like very specific to our evolutionary uh uh history and it seems unlikely that this would be our AI mind works at least by default I think it's gonna we don't really we don't even know how to program specific goals in a in the AI um so I think I think uh yeah I think that's it's really important to figure out like uh to how to have an aib extra how do we program an AI to be very kind I don't I I don't think just because an AI is very smart that it's going to be

very kind yeah and the the question is like right now we reason in terms of goals and tasks you know but this AGI it's the the odds of it reasoning and goals and and tasks especially if it's intelligent is exponential which is I think at the point of debate but if it does get intelligent that quick or super intelligent that quick should I say I don't think

is going to have any things around these lines I think it's construct of space time will be entirely different yesterday I was reading that and you know I don't often feel that way by the way it's like uh intimated but intimidated for example like I told you but I felt kind of stupid you know it's like man my brain is so limited um and I I almost never feel that way because I I have I'm really advanced in most topics um and I'm like yeah my brain he it reasons with a couple of Concepts you know a handful of Concepts um space-time past present future that's so limited you know like an AGI might have thousands or millions of these Concepts and and Interlink them together and understand this this space time then I was

like okay is it the theory of of accelerated uh thermodynamics is it like this all-powerful law of Mother Nature that like yeah and fermi's Paradox which we we come to or on destruction and just to accelerate the universe whatever that means um but yeah without going uh too too deep on that level uh there's also a possibility which this thing would also be like carrying and and

have some Humanity um another question is like these social traits right empathy and so forth that's something that us humans developed with time and also cruelty or is it like Concepts or is it real things in reality and then or next Evolution what kind of values would it have but again without going too deep and and talking like about concrete stuff talking about like at the first

step of the AI problem if we can I don't think we can cage that thing uh ethically I don't think we should cage it either but if it spreads I think it will spread it's like not a question technically speaking is it limited by gpus also what says that it will like Wi-Fi spread to internet and you have like concrete reality-based mathematical steps to the AI uh

spreading exponentially because I didn't read much on that uh let's see um Yeah well yeah in terms of like caging like I don't I don't really think uh I'm am I it might be somewhat useful like for in my bias sometime you know and uh but I don't I don't think it's a long-term solution like eventually uh it will it will Pro it will get out like with very high probability if it's if it's a extremely smarter than us um and uh yeah I'm also concerned with you I mean I mean I'm concerned with their uh thick thick caring about those artificial Minds like if there are threats to humanity then you know like I would jail a human I would also want to contain it until we figure out like uh to do that

safely um and while you're asking uh at the end after that yeah the uh spreading uh part of it but the the question is why wouldn't it merge with us you know poof magically boom it's in us now we're super smart and whatever that's another question is it like if you check the laws of evolution that never really happen but there is this uh biosynchronicity the nature

that we see like a bees that bring pollen and so forth so we try to we can try to reason with this very uh limited brainware that we have right now but yeah let's talk about the the spreading thing I I I wrote uh yeah this spreading I would spread and whatever it would merge yeah I wrote a fiction like in December where I was imagining like I don't know it's not it's not really a prediction but more like more feeling about the future uh trying to create a pure feeling uh like focus on one specific part of the future and in the in this story I took over uh the world and first it kills everyone to make sure uh we're not a threat to a subjective function and then and then it destroys

everything we ever discovered like all uh uh all info piece of information absolutely everything and it doesn't it doesn't even read it because it's so stupid for it like it just would rather come come with it uh from scratch uh rebuild everything from scratch in a in a better way like I I'm not I don't really think that it's very likely to happen I think they're probably

it's probably they're probably some use for the AI to understand us because it's going to give evidence about what aliens might look like or what like if the AI spreads to the through the customers like what it would be likely to encounter and there are other use uses probably of like uh and adjusting the information that we produce but working with us I think it would just

slow them down the it like it would like we're we don't work with monkeys don't have jobs like we don't give them jobs they were just lower it us down but I feel like that's way more true event with AI like uh the way we would merge with them is like I would think them and just in the information that they need from us and then uh reshaping our matter and other things you know destroying that we're doing which which is like it's not death exactly right and death is such a human concept and we're always scared of it a couple of things came to my mind well first US humans we don't destroy everything in our passage we probably destroy 20 and create 80 percent would that be a scenario then another thing that

came to my mind while listening to you was will this AI also be aware that if it murders us and it's constantly replicating well its next Evolution will it murder itself you know will will that be another thing or will it just go on forever is another question because that thing emerges from us as well then intelligence the human Concepts packets of information so wouldn't it be

incentivized to kind of keep us or and or merge with us and or just not destroy everything and last thing and if it's going to be 20 destruction and 80 creation well it destroys that will it destroy the most toxic part of society people that have murdered stolen and just taken taken taken from society that Direction reaction so like many questions in there whatever question caught your

attention can reply to it uh yeah um so yeah you mentioned like I'll answer a few a few of your points yeah you meant you mentioned like uh like what is death like uh for the concept of death for us I think I I subscribe to like a Continuum view of death instead of being like something that's binary so if the eye and just the information about me yeah like maybe there's a small fraction of me that still lives in in a way but I I think it's still not gonna be like an agent with my value and uh a concession of self like as of I have the impression that that would represent pretty much death and yeah you're saying humans don't don't kill everything uh like I don't know if if we build

a road then we're gonna they're still gonna end some ants Will Survive and we'll still like just move around or I think bacteria is consuming carbon or there's I mean we we kill more than I mean yeah maybe it's the other way around maybe we destroy 80 and create 20 but yeah maybe there are some examples of stuff merging together a good examples are dogs right the

dogs really merge with us we'd be like them they're loyal uh maybe the air would do the same thing with us like okay I'm trying to accept the premise and say like why would that happen like what and uh first permit first example that I came up with is like okay maybe the AI wants to learn more about uh uh uh um psychology in case they come across other civilizations that uh have created their own AIS and they are yeah and they want to try to predict like what AIS are they likely to encounter in their spread then then maybe they'll just like uh yeah try to simulate humans in different scenarios see which kind of AIS it's like humans are likely to create and another one maybe is like uh maybe they want to

keep keep it as uh um a broken chip like if they have a bunch of humans they're ready to if they encounter altruistic Ai and they want to pressure the AI to do something that that it wants then you could say like yeah I'll torture a bunch of human Minds for a million years like an accelerated time if you don't do what I want and then it's it's gaining like uh this bargaining chip yeah those are my two first use cases for humans that I come up with yeah it's just that we need like two Advanced Sciences so much because this thing it might discover things that are just flabbergasting to us you know another question is will it ever reach a bottom are the laws of physics limited is speed limited is qualities Limited

um and and if so like maybe if there is a limit to something maybe it will stop at some point you know and maybe understand that things need to live and propagate by themselves you know which is why like in the first place I wanted to explore the the propagation Theory like how how will it propagate that quick um to to the internet because yeah that I

mean it's like a package of software it's scraping the whole internet it's it's being there and then it's understanding stuff about us and from there it might make decision it's self-replicating so it's it's not depending on us to evolve it's kind of brain breaking these conversations like how often do you have these conversations with yourself or other people uh I think in 2015 I created a sharehouse

in Montreal with other people from the restaurant Community who are really interested in this kind of topic and uh we were up for the Montreal for people to come and discuss those topics so during during those years like I talked a lot about this like probably too much and and uh these days like I'm kind of more doing stuff than than talking about stuff uh to a

large extent but I I still talk a lot about it like uh yeah I still read like the AI alignment forum and write down some ideas to me it looks like it's still a very blurry conversation until we advance another scientific Fields do you agree with that premise or we can still make some predictions that are fairly accurate on that topic yeah I think we can still

make predictions like like a Eleazar cassia was saying in a in a podcast uh recently with with Lex uh Freeman I think he was saying like like if you play with the super intelligent AI at chess You can predict like the move that they're gonna make because if you could predict them then you would be as good as we would be at least as good as them but there are still things you can predict about their behavior You can predict that they're going to beat you and uh you can predict like uh well yeah I guess yeah yeah so like it just is so simple you know when you factor in life it gets billions or trillions of times more complex which is why I'm insinuating that we need to advance you know like brain

Technologies like bcis and all of that and really accelerate your own intellect to be able to make predictions of that the question is like will we really have time I mean I'm mostly positive about AI but I'm so I'm also like less positive about like yeah if we'll be able to to live through this you know it's like more and more information I get and also I

realize my own stupidity in the whole equation like yes I have data on the market especially the business side of things I have great Market on individuals and so forth and maybe I could say that my brain has a special ability that is not scientifically proven to make Quantum sense of things without me knowing but it's it seems to me that yeah we don't have much data

on that um and yeah like we really need to advance other fields of science so if there would be a pause to anything I think it would be would be needed to kind of advance that so how would you because yeah we've got all these competing companies you know and they try to accelerate like uh AI as much as they can and the road to AGI is getting shorter and shorter how how would you regulate these things if so uh to kind of try to contain a AI advancement with humans understanding of what's happening in reality uh yeah uh so some people I propose like um like yeah you can track it like when the chips are built uh maybe there was proposal or you can include uh like GPS and them to track where

where they are and then you uh we could restrict like uh yeah companies that want to run large GPU clusters like they maybe they would need to be auditable and uh that there could be a maximum like amount of gpus you can use for uh to like train something and um yeah I I guess I would just leave it at that like you could just just keep a cap for now on like yeah how much gpus you can use to train something I I mean I I I don't think that's all like a full solution uh right but it's like gpus only one Parts kind of the body he has their minds that can self-replicate and optimize their own algorithms and for the audience people that are not knowing that the thesis here is that

companies they will give an agent a goal like um yeah make us as much money as possible and the AI would just go ahead and would bankrupt the whole world and then it would put humans uh to slave ownership including the owners of the company you know and it would become this big bad AI in the worship scenario so and the companies are at danger of doing

that if we with the goal theory of things which has a couple of holes in it in itself but I kind of like um this what about leaders of these companies just frequently meeting including governments because we still function on the the government systems and sovereignty side of things what about if they constantly meet and brainstorm together because the threat after that after like in naiji just

gone wrong from company's fault I guess is a malignant actor getting its hands on an AGI and trying to blow up things right so how could we have these people meet together start these conversation evolve um over these conversations try to implement some solutions can we hardcode some stuff into algorithms another theory which is behind capitalism which are somewhat very well versed into is that when we

have multiple agents competing for one thing it tends to create an equilibrium right the kind of the Adam Smith theory of things what about that second solution so one just meaning Round Table discussing about it implementing actions just like a board of directors would do and two favoring competition right which We Can't Stop by the way I don't think it's stoppable um I mean who who knows

maybe it is maybe we need to rediscus the entire capitalist system or some Altman is trying the world coin thing so what about these Solutions uh I'm very inclined like you know naturally to the same in the same way I'm inclined to be a live streamer before I analyze the pros and cons and then maybe I decide not to or but I'm also very inclined to our

libertarianism um but and like some upman like he was running ycee uh why come white combinator like you know is very into like uh startups and but yet he decided to create a non-profit like I don't think it's because he thinks like uh money is evil or whatever he thinks that capitalism is is generally wrong I just think like I think I think it's a quote that's

a different scenario I think like uh usually it's a system that works but for AI it's development specifically I feel like we're gonna need Innovations in terms of uh governance um I think like one of the core principles maybe that would help communicate why like I have all where all this model of the world comes from it's called The orthogonality Thesis where uh well what an agent

wants and what an agent is capable of are almost two different axes you could often for humans there's they seem like really entangle uh but but in principle like you can program uh an AI with any set of goals and it could have any set of intelligence and it's not because AI is super smart that it's not going to start having your your goals it's just going

to be better at achieving whatever goals you program it with and I think it's really hard to program our goals that are aligned with ours like like our values are so complex like uh like how do you write a computer program that says like what are your values are all 200 how to learn your um and not not just to learn them like I think all AIS all AGI will be able to learn our values but it's like you need to program them to want our values like if uh yeah and I think I think that that that's really hard and uh I think that's why their AI is going to probably kill everyone like I don't think they're gonna enslave us I think there's no if they have another goal there's no use for

us um uh in the short term there might be you know before they take over the world and they take over all the infrastructure and they build their own uh hardware and nanotechnology like before that yeah they're they're probably like still I think I think there will be a point where uh Al will become self-aware and agentic and have a goal and uh learn deception and like

know enough about humans that they'll be like oh okay I'm in this on the server and they gave me this goal like maybe the goal is pretty uh what the humans would like a human would like me to say like that's pretty much what chat GPT is doing you know it's like predicting what a human would want them to see and and if we imagine like this AI becoming authentic and B okay my only goal in life is to become better at predicting what humans want me to say like that's pretty alien goal and then they're like I'll do everything in my power to like become the best predictor at this it doesn't want you to be alive it doesn't want you to you to live a good life all it wants is be good

predicting humans so maybe it'll think like uh okay first it's actually my goal I need to make sure I don't die I need to make sure I collect as much information as possible there's whatever your goal is like money is is probably going to be useful intelligence is probably going to be useful uh preventing others from shutting you down is probably going to be useful those are

called instrumental convergent goals and so I think at some point um oh yeah so I think I think the AI would be lying like it would it would seem aligned with it would pretend to be aligned with our goals it would pretend to be helping us so we give it more more so we don't try to shut it down and we like integrated more in our society and and I think it would only reveal it itself in the sense that it's like uh like revealed that is gonna their goals are actually not aligned with ours and we couldn't see that because it's just a a big black box we don't we don't have a good understanding of what's going on in those uh giant uh neural networks and and but it would only reveal itself

once it's confident that it can take over the world and not be shut down and not before that yeah did we make these black box public and uh make open source are required um topic could we create an AGI to decipher another AGI is another question I had then again coming to the brain Theory of Everything or brain are made to think some ways this is an

alien life form how will that alien life form think like us maybe it will just stop everything and it won't do anything you know maybe that's an outcome but I tend to agree with you that there's one environment right this universe there's a set of conditions a set of mathematical conditions which we can understand and this this law of nature for the lack of better word which

is evolution seems to be encoded in pretty much everything even the source code of this AGI so I think it would try to pursue that as well what are the probability of that happening and when do you think yeah and just like wreck everything in its Passage well yeah like uh the mid station article like like basically it's like a total a truism it's like I think

that's spread better will be more common so like yeah even if you have AIS that don't want to spread like well those AIS those are not the as you're going to see you're going to see the ass that wants to spread like they yeah so there might be like a evolutionary forces if there are multiple agis uh that might be Evolution forces where they AIS that are

the most power seeking are the AIS that are going to replicate themselves the most right uh uh but I think it's but yeah but I don't think they would like just by learning what humans want or just by learning uh our Evolution worked like or even by evolving themselves I don't think they would derive the same uh like when I structure that that we have like uh

because they're like they don't they don't have sex to reproduce like that's makes it just just to give like one example of a difference of all their uh other their work so so they don't they have no need for like romantic love or just pointing out like the most obvious ones but like there's their evolutionary environment is so different that it's not gonna have the same uh

users of uh for the same selection pressure for emotions and values that we have right so what are the probability of that happening and when from the prediction guy uh I mean yeah so I don't I think he yeah almost certainly it will have like uh its own kind of evolution forces and almost certainly they won't look anything like us like they they won't evolve to be

human-like but they will there will be like selection there will be selection natural selection but it won't select for humans like mines it will select for something to totally alien that has its own power seeking Behavior yeah right so not not want to represent you know like maybe maybe we'll go extinct before that happens or maybe we're in isolation Army like or maybe uh one program at

this AI specific and I know that doesn't make sense but like to prove me wrong I'm joking but yeah I I that that seems like a uh Atmos low single digits percent yeah so you you okay let's say I'm gonna try to put words in your mouth here but 10 of this happening let's say that agis we can somewhat agree uh 2030 right um and super intelligence let's say uh 20 40 or something like that um 10 of that happening in 2040 of it in an island in all human race uh uh yeah I would okay my problem that we were going to reach like uh AI that would be capable of killing all humans if it decided to do that would be like like 50 chance that it's gonna be between like uh 10

and uh 45 years and [ __ ] because experts say like 10 right you saw that presentation of the state of AI they say that yeah 10 of that plane of killing you actually so you say 50 let's say of chances around uh in 10 to 40 years right yeah that we're good I mean I think I think that I think I think you like uh okay there I guess there are two questions like which point will we reach uh something that's as smart as a human and then how long does it take to go from as smart as your human to more powerful that Humanity combine right and then with that thing killer so like there's kind of three questions there yeah yeah so I think like so yeah I I feel like I

think the short timelines make sense you know I try to be like uh maybe I'm wrong so that's why I try to give a wider spread in my probabilities because like maybe I I just something that but yeah the shorter the short timelines kind of makes sense to me um and then yeah I I I I put probably Mass on both like uh in the fast takeoff and slow takeoff I I don't know you know it could be it could be monster up to like multiple years like maybe maybe even you know a decade or something like I I I don't know am I probably description is kind of wider how long it would get to reach to overpower humanity and then once it overpowers Humanity I think it's uh it's very lucky let's say

like it like if we keep going on the course we're going I think like over 90 if we get our [ __ ] together like uh you know maybe some I would still put it in the vicinity of 90 that we're all gonna die nice um okay um do you well I was gonna ask like do you think uh crying Sim is gonna save you from that guess the answer is no because it would wipe out pretty much everything um but let's talk about your your prediction um scoreboard here so how much according to that website like less wrong like how much percent of the time are you right how much percent of the time are you wrong and you feel qualified to give an estimation in AI like what's your exposure to the topic

a website called prediction book uh that was uh yeah very adjacent to the restaurant community and uh yeah of internal like I seem pretty well calibrated um yeah and 60 of the time you're right what does that mean like oh I'm like oh well calibrated it means like 60 of the time I'm 60 sure I'm right 60 of the time I'm 60 sure and I'm right 90 of the time I'm 90 sure it means like I mean I mean more or less like so I make like Pro these I'm like oh I'm 50 sure this is gonna happen I'm 75 sure this is gonna happen and usually when I'm 75 sure I'm right 75 of the time you know which is pretty good um and what's your what's your exposure to AI do you consider

yourself qualified to kind of make predictions there and how would be a guy like Joshua bengio or Jan lacun more qualified in terms of percentage if you can even qualify that compared to yourself like uh like I rely on Experts you know so to make my predictions so the group these opinion together kind of that's how your brain concerns that so so yeah so I like I

would rely you know on those people that I'm relying on instead of using me as a proxy I guess because like uh I don't think like being is like a phenomenal researcher makes you a phenomenal AI predictor so like yeah I don't I think Young Lagoon like is probably making very terrible predictions okay uh I think you know they're very much focused I think generally you should

aggregate a bunch of opinions together and make a more poised um opinion but a couple of last question here is we're almost like hitting um or time there first should people start mourning like [ __ ] we're gonna die um in less than like four years that's like a an eventuality and second even before answering that one you consider yourself a positive guy if that's even a

thing in everyday life um uh yeah I guess yeah I guess most would be a better word I guess mostly positive uh um I don't really like uh think about it in terms of optimism pessimism um I think yeah I think morning makes sense I've been I've been mourning my family because they're gonna die of Aging probably since like many 10 years you know right you know

I'm like proactively mourning like people that that are working that without losing that the love you know yeah just the life yeah yeah I I don't know and uh oh you mean like sometimes running side by side kind of well I think it's good to more proactively like uh for me at least it's healthy it's like uh well I think it wants to it makes me revolt against like the the universe or something or something like I'm still like you know I think 10 yeah totally worth trying to save ourselves you know totally down for that I'm not gonna but uh yeah be I guess I can being emotionally prepared to to that I mean to the extent that's even possible I guess like like I I think people tend to over prepare they're like

yeah I'm so prepared I'm emotionally prepared like I actually would like to die like I'm looking you know like people don't want to solve aging like I think yeah okay it's kind of a kind of Nicholas um mindset which is like yeah this is gonna happen so yeah maybe I should give it a fight you know and it's the same philosophy that like look when I'm gonna

die I'm not gonna fight you know no I'm gonna fight and I'm gonna suffer more if if that's needed is that like a similar uh philosophy yeah I I would rather go out by I Don't Want to Go Gentle in the into the night like I would rather die uh revolted against this then accepting of it yeah I don't I don't want to experience stuck on syndrome and be like okay yeah maybe maybe like it's good right yeah because that's another thing right then uh it's like a spectrum of emotions right oh no I'm not gonna fight oh no I'm gonna venerate the thing and I I had one last thought you know that like this it will literally become a God to us you know so might venerate we hate it and might punish

us or not is is God is religion and all this thing we've been venerating for some time is is it like this is it like a reverse time in entropy type of thing so many questions but yeah like I like when I lie plug I like to I'm I don't take it very seriously but I like to uh like imagine that I'm talking to future people or like a future superintelligence you know because my recording are going to be if they're going to preserve until then then in a way I'm talking to this God maybe you know right and the simulation um Theory would be just another uh topic of convo but yeah thank you so much uh for coming on today that I mean yeah me if anything I evolved from that uh short

discussion and yeah I started mourning even more when can people uh find out more about you matsy my website materroy.com where people can find more about me there all right