Etan Ginsberg, Martian: LLM APIs, AGI, and model routing with Etan Ginsberg

foreign welcome to another founder wisdom board with atan Ginsburg that was hard uh etan will actually teach me Eton he will teach me some memory uh tips today because he was quite Advanced at that really enjoyed his Ted Talk uh he's founder at Martian by the way love the name love the logo Eton has an interesting background uh AI research at U of Ben he also is

a software engineer for some time worked at a Venture Cap Fund I guess or Everest Capital fintech startup uh that's interesting too and yeah also the Washington school stuff so call these guys little genius and um me and Ethan were discussing pre-pods that it's hard to take compliments when you're up there personally I always deflect uh compliments so let's start with that one Ethan introduce yourself and

tell me your philosophy on taking compliments or deflecting them sure uh so I am a previous two-time founder I went to school at Penn and uh did AI research there and from that research I essentially had the inspiration to start Martian along with my co-founder the idea behind Martian is we're abstracting away the decision of which llm to use there's many llms right now more and more

of them it's becoming more and more like a commodity and people are often faced with the challenge of what model do I use for a given task it's kind of like in the early days of websites people then often know what website to use for a certain task and that's why Google came along and aggregated all these websites we're essentially doing the same thing for llamps through

a single API you can give us your prompt and will automatically route your prompt to the best llm based on the expected performance and cost and send you back the result so that's what I'm working on right now we're a venture-backed startup and previously had two companies that really sent me along this journey starting my first company when I was 12.

it was an e-commerce site for the show American Ninja Warrior so I'm also interested in uh both advancing the frontier of physical as well as mental abilities um got into memory techniques and how we can do things like memorize a deck of cards in 12 seconds or less and it was really from that interest that I really got interested in the science of learning and memory and

then I wanted to apply that not just to humans but to machines as well so that's how I kind of got started with AI research and from my experiences building previous companies Everest Capital that was another company I started that was back in 2017 um right before the Ico boom we were using NLP to predict when to buy and sell cryptocurrencies uh so yeah that's what really

set me along this journey testing your memory now there was another question there yes so the other question is replying to uh compliments that is a tough one because I think you're very flattering with your compliments and I think it depends on the nature of the compliment I think oftentimes the people who give the best Compliments are worthy of their own set of compliments so I like

to sometimes reply with another set of compliments um but I think it really depends on the nature of the conversation so um I'll put that one back to you how do you like to reply to compliments thank you for asking a question but I think that first I'm trying to flatter and trying to make my guests like feel comfortable as well now another question is it an

accurate compliment apart from the flattering part I interviewed 10 to 15 CEOs and Founders per day so if that means anything to you I think it's very good feedback it's better than it's probably in the the last percentile at if the last one would be the the best compliment there by the way it's them gonna test your memory um I'm gonna like ask you like five questions

in one because I like asking those but most of my guests they forget about these questions uh quick comment on what you've mentioned for them the business I love aggregators I call them that way aggregating startups they're just great you know because as a consumer I just want the best out of everything I don't care who does what I don't care about GPT I don't care about

uh Bard I call it by the way which is not nice but it's not uh it's far from being intelligent um Claude by the way Claude is the best right now uh so that might also be like a shortcut by the way you know like oh do I need to even make a decision or is cloud the best anyway I'm in love with these aggregators um have you checked what's it called this thing uh this Chrome extension chat GPT for Google aren't they doing that and directly on Chrome would it wouldn't they be like top competition for you guys um I think there's a bit of a distinction there because we're an API so we're servicing all of the applications which are using these llms on the back end so in this case you're actually

yeah it looks uh it looks like this GPD for Google do you have to switch between the various different models right I don't think it sends I was on mute yeah the paid version gives you access to all in one shot again yeah so that's still slightly different use case because you're actually going on that there are an application so you're giving the input and getting a

response what we are is an API so companies like um for example a Jasper or a versal or a replit um they are using these llms on the back end and they will make these API requests to the LMS and then return back the response to the user so it's slightly different in the sense that we're kind of that infrastructure layer on the back end as opposed to uh application on the front end okay do you think that the best CEO and the future AI oligarch if there will be such a thing will be the best technology user AKA someone that can constantly test Technologies and deploy them very quickly I think absolutely I think that or or it will Whoever has the best Technologies I think really going down the road I mean who

knows if uh it's gonna be even full-fledged humans or some kind of adaptation with technology depends how on what timeline we look but I think certainly like AI is the most powerful technology right now and whoever best leverages AI uh essentially is best leveraging the world so for sure so I'm building a business that will build other businesses that's my main business thesis my capitalist thesis yeah

I have agencies in there I have SAS that are developing slowly but truly so my thesis is like keep on testing all of these tools I'll surely test Martian and potentially develop a SAS in the market where I think there's a potential hole and a strong market demand what do you think about this well I think the challenge of creating a effective business through some kind of

AI is actually an interesting proxy for AGI that's one of the potential tests people so originally people thought about benchmarks like the Turing test um now that's become no longer relevant because we do have models which which are capable of essentially passing the Turing test so the next bit what is the next Benchmark I think is is part of the question uh people even with AGI there's

kind of the loose definition of like a machine that can just continually self-improve itself at a faster rate than humans but what does that look like in a tangible sense one line of thinking is to have a machine an AI that can create its own businesses and that can outperform humans so like say you give a AI 10K kind of turn it into a million dollars so

I actually think this is a fair Benchmark for AGI the question is you're going to need to have a really high level of performance with your AI in order to have it run fully autonomously so it's not out of the question but I think we're still a ways away it's kind of like the challenge of creating AGI are you pursuing AGI on your side Allah mouthman um

I think they indirectly the work that we do definitely will have some advancements that contribute to AGI so it's certainly part of the long-term goal um for example one of our research areas is actually converting from Transformers which are essentially black boxes you give them an input they give you an output we don't really know why that happens I mean we understand at kind of a scientific

level there's all these um floating Point parameters and we get this output but we can actually get guarantees on the performance of these models they're they're fundamentally like a black box what we're actually doing on the research side of Martian is converting from these black boxes to human understandable programs so what you can do then is you can have these programs go and refactor themselves trans since

Transformers are really good at coding that's one of the tasks they perform well at they can actually improve themselves much more directly and then you can eventually get in in the long term some kind of self-improving system so this is like a piece of research that we're currently working on it's still very much on the research side on like the model routers currently in deployment but it's

something we're certainly excited about and I think if that research goes well it could certainly have implications for the development of AGI why do you think us humans are so bad at understanding stuff seems like we don't understand you know we don't understand love we don't understand Consciousness you don't understand the brain we don't understand AGI like do we need more LSD more creativity like what is

it because we're always thinking in a box how could we accelerate the rate of discovery how can we accelerate the rate of discovery I did see that that's a good example of testing my memory because I think there's a lot of questions in there there's why are humans so bad I think in some situations we're not necessarily bad it's just we have limited bandwidth like a human

can be very creative in a limited domain but they can't process large amounts of data whereas in some ways like machines for the longest while we're kind of the opposite they were could handle large amounts of data but they couldn't really think for themselves and I think when you've kind of pair the two together that's when you can really get some powerful advancements now the machines are

kind of catching up and I think they're also able to do things somewhat creatively as well um but on the creativity point and this gets to something that uh actually is related to my interest in like memory and and learning science I think there are some tangible ways in which you can improve creativity one of the ones I came across is like this power of multiple perspectives

actives so like take a simple example like actually I have this on my desk it's a screwdriver if you were trying to figure out what are many different uses of a screwdriver initially you know you might be thinking it's used to take out um some kind of screw I don't know maybe I could like use it to poke a hole in something but then you think okay what are the fundamental assumptions there what can I change about this for example what if I melted this down what if I use this piece about it what if I change the shape so instead of a screw it was uh curved uh what can I do then what if it acted as some kind of a wedge what if I use the plastic to repair some kind of

a hole things like that challenge your fundamental assumptions I think by challenging those fundamental assumptions it's kind of like thinking out of the box I think part of the problem that we often have is that we think inside of these specific boxes and I think that partly that's just been how like Society conditions us and the structure of you know whether it's going to classes where you

have to think inside of a box or work I also think that's what like brings me to entrepreneurship is you have the ability to write your own path you don't have to fall that follow that traditional path which is something I saw a lot of people doing and when I was younger I often struggled with like what what's the right decision there but I think the more

that you can get outside of that group think then the more opportunity you have um to enhance your creativity plus now you have like llms are really useful for Creative generation because they can generate examples and that often gives you that inspiration so it can challenge your fundamental heuristics your fundamental assumptions like in the case of a screwdriver or the shape the size and from there you

can then Inspire yourself to think about more and more use cases gonna ask you my question then I'm gonna GP I'm gonna ask GPT to tell me about creative uses of a screwdriver um because my next question was typically speaking these llms are pretty low level creativity I would qualify it as Max 7.5 creativity which might be useful for a tired brain I have a tired brain

sometimes yeah why do you think they're not so creative I mean it's like all recycled data yeah could we put it in a very creative mode in which they would because creativity is typically like two ideas together obviously there's like more to that maybe they could start merging tree ideas together and making something more creative creativity I also think is Discovery is experimenting things and seeing Oh

that result right there is quite impressive that's Innovation that's a discovery right there so how can we make llms more creative I think an important piece of this is when you think about like what is the nature of creativity at its surface it might just seem like spontaneously coming up with ideas but I think there are larger patterns and formulas which you can still apply to creativity

in the screwdriver example you might explicitly say okay I'm gonna rethink the size of this I'm gonna rethink the shape um what is the specific material like those are very concrete Frameworks you could teach someone this even in a classroom but people by default if you just ask them they wouldn't think about okay how can I use this material if I change the shape or if I

melted it down so I think that's I see you're doing the example here what you could do with these llms is give them that formula and then they can apply that formula to give you certain examples and then I think you can take it from there so if you create the kind of blueprint they're very good at applying that in the context of a particular task a particular tool and I think that's where you can also have uh more creative use cases so so far so bad um yeah it's really I think in some ways that's probably especially if you factor in the speed that's more than many humans could give right oh yeah 100 but the quality of it um like me my brain is really seeking outliers all the time yeah well and

this is an area where actually uh I think a model router can be very helpful because right now you're just using one model the idea behind what we're doing is we have many different models there's over 300 000 models that exist now if you look at the hugging face library and open source you've got closed Source models you've got models that people build internally on their own

data and these models like you mentioned will be trained on different data so you may have models that are trained on a more creative data set what we're doing is we can predict how well each of these models will perform at a given task so let's say your goal is creativity well we can actually determine what models are going to be more creative before we run them

and then what we can do is we can take that request and run it on the model that'll produce the most creative outputs and then I can tell it's like I prefer this model over the other yeah so you can you can give feedback which which the model router takes into account but it's designed where you don't need to give that feedback even it okay because it's

like one response straight up yes so each response routes to the best model so it's on a prompt by prompt basis so you send another response say you said give me more use cases for a pair of slippers it might route to a different model because that model may be more specialized in that particular domain of slippers for example I'd love it to be personalized to my

personality and ideally it would be so seamless the onboarding it's like okay are you Charles Cormier then the AI would understand who Charles Cormier is and it would scrape everything on the web would mean there's a lot of data about me there yeah then it would refer me responses that I like which is also scary because if I keep getting what I like how can I grow

sort of so yeah sometimes it would need to vary and and give me more food for DOT and not just what I want to see yeah I think I think that that definitely makes sense and our focus is primarily on like applications so it wouldn't be like the individual users like yourself that are using this directly it would be plugged into an application that's using AI okay

that's pretty cool what type of companies will you target with that yeah so ultimately we want to be a solution for any company that's that's calling AI models to their API but in particular the companies that really benefit are companies that either have specific costs requirements so there's a company that we're working on um they're almost a unicorn and if you look at how much it would

cost them to use gpt4 it would actually cost them more than they do in Revenue so you know it's a huge expense but you can use models which are between 30 to even almost a thousand times cheaper and still perform very similarly for example desiccoder is 900 times cheaper than gpd4 but on some tasks the performance is not that different between them so one use case is

to reduce your costs so that really helps with unit economics for companies that care about that for earlier stage companies if all you want is to just use the best llms that's another use case because it simplifies that process you don't have to go and test all these models on often we hear from these companies and users that they'll spend a full day or more just testing

one provider set of models and often you have to hire an engineering team so it makes it a lot simpler and I think that's the goal here we talk about in the long term how do we best use a AI how who are the people that are going to be able to make the most use of it in this world it comes from making it as simple as possible we like to say using llms it should be like using electricity when you plug in you don't care where it comes from you just want the best of it you want the best intelligence at the lowest cost and that's the idea here and then if there's really specialized models if a company is trained a special model on their financial data set let's say they have

some proprietary data then we can route to that specific model and oftentimes there may be many many of these models you know thousands if not tens of thousands so finding the right model is going to be something that and to do it dynamically for every prompt it's impossible to do that manually you're going to need a router service yeah we will AGI hits what does that day

or that moment in time look like I think it depends on your your definition of AI I think even beforehand you're going to see massive productivity increases it's not like all the sudden overnight you're gonna see this call it God AI yeah what is what does God AI look like the way that I think about this tangibly is if you think about once we have AGI and

we have massive improvements think about some of these fundamental problems like how do we you know extend our lifespan by a hundred years so we live to 200 or 500 how do we go to Mars how do we become you know multi-planetary species these are really hard problems which people are working on to this day uh I think once you have AGI these start to become very

attainable within our lifetimes and everything is all of a sudden going to be on the table that's how I think about it at like what is the effect of it is each hard problem that you can imagine all the sudden becomes very doable within our generation I would argue actually these hard problems are doable within our generation even if we have no idea how we're going to

get there if you can get to AGI that's the enabler and it makes everything else on the table you think it will solve happiness and if so how quick I think so although I think it depends on your definition of Happiness like there's people feeling good well for some people feeling good it's like if you think about like the Matrix you're hooked up to a machine and

all you do is feel good but is that true meaning or is it just you know your pleasure receptors can we just feel good I think that's actually an easy question I think yes I'm sure we can engineer ways in which people feel good but I think for a lot of people what they would consider happiness is something deeper than just feeling good it's it's a sense

of purpose it's a sense of meaning it's a sense of authenticity that you wouldn't just get from like engineering your pain receptors to feel good right I would add to that definition feeling good but at scale for a very long time because let's call it the Ready Player One Vision being hooked to that helmet you know and take it enough and feeling the massive headache and the

massive need to go back to it yeah and I think you would lose your human form you I think you'd be more bitly obese on the short term another point I would make on this and I think this is worth thinking about as we think about AGI there will be a day in the future where intelligence becomes like a commodity meaning right now we as a society celebrate the people that are the smartest um that's generally a sense of status that's what people often derive their their self-worth even you know we're complimenting me on that at the beginning but I think soon it's going to be that machines are just vastly more intelligent than humans and it's not going to matter that much how intelligent we are and I think an important question then is

where are people going to derive their value and their sense of self-worth when intelligence is no longer the n-all be-all you know back thousands of years ago it was your physical abilities you know how you could survive how you could hunt things like that so I wonder if that may be a direction in which we go but even something like that can be engineered at a buyer

logical level so you could run into a situation I think this is something I'm concerned about in the long term where there's almost like a a loss of meritocracy because a lot of these qualities that we have to work hard for and that we feel satisfied when we sacrifice our short-term happiness to get that long-term goal because that's another thing is I think people actually get happy

when they make these sacrifices and they push through adversity even though you're not happy in the moment you feel a sense of deeper fulfillment that things may be very different when you know the machines are handling all the hard work so I don't think that day is immediately on the horizon but it is an open question where are we going to develop our sense of self-worth at

that point I think that's a one percent like you and I I call them homo sacrifices or homo no pain no gayness I just said the word anus on on a live call um what a way to end the podcast hey then can you tell us to where we can find out more about you and so on LinkedIn and social medias yeah so you can search up my name it's e-t-a-n Ginsburg is my last name on LinkedIn on you can go to our website it's with martian.com I should also mention the name of the company because I think it's an interesting story so the Martians were a group of some of the smartest scientists in the world they were all born in the same city in Hungary and people were like how did all these

scientists these are people like Paul erdas John Von Newman some of the brightest people at the time uh how do they all come from the same block and people are like oh martians must have came down and had kids with all these Hungarian women um so for us because we're really about understanding intelligence we don't want people to have to wonder oh is it martians we want

them to be able to understand the nature of intelligence and that's how we created that company name love it dude and on LinkedIn Ethan Ginsburg right or search my name with martian.com uh yeah I have a website etonginsburg.com as well