哪些 GPU 云值得租用？SemiAnalysis 发布 ClusterMAX 3.0 解读 AI 算力与新云格局

[Speaker 2]: the uh hi h yjordan um congress and launjing cluster max. um what's your brief history? um is a persl introduction and also a journey into building cluster max,

[Speaker 0]: surely i'jojoin. some ananysis is last summer last june and been working on the technical staff would grow in a lot um over the past year. anyway, the i guess, personal ackackground out i'spbeen ten years years working techilical acket on tera. there was designing a lot of the hardware systems that these new clouds were buying back, you know, before charge you fd launched. and then with the big craise and it's twenty two, twenty three and a lot of them beinto into the market. so big, some analysis reader read every word d new, a bunch of the guys in, like group chats on the side, and uh saw a lot of my opinions represented in cluster max one. when dan and the team put it out when dh, the best way to get a job of semine, oysis is to something something semine ysis is doing and criticize it, and then delen hire you to fix it, which was what happened. in my case,

[Speaker 3]: the criticisms of classramax wand two,

[Speaker 0]: but what i did too all by myself, so i can give you the criticism, no criticism of there,

[Speaker 3]: what chges from one day to. and then you know, let's let's quick into whawhat's new, great.

[Speaker 0]: you don't screen guys. it's perfect deserts. i don't know what what you would do, of course, um um kay, i close ter max one. i mean, there is just a lot of providers mesed, like twenty four total providers and a lot of them, um big ones, you know, they should have been testing and warr't small team and of the time. and then there's a big focus on sluum to me. carbonaties was really important need of, you know, tests that as well in terms of like criteria and stuff even changed a lot. but yeah, alway back to queser max one. you know, the rankenpapage looks a lot different otheers, just a lot less names and releasing the market view. also, i believe at the time of are around one hundred and twenty total providers, one hundred and eleven emerging neoclouds there in the middle. and we didn't release the full mark of you this time because we payled led behind ind nenew cloud model, or we track all these providers. a lot of them are public. a lot of them are going iio soon. so so got this whole financial model for the industry now. but yeah, three point o are up three hundred and twenty three total providers that we track. um and i have some people this morning in my inbox, reach and out. and and they've got some three thousand gpus and they want of the test and where 're got, you know, anyway, it's in a big vovolution. i think big big in terterms of sting is that with clclummax one, you know, all the testing was hands on. this is pre cloud code, pre codex, the getting serious. and so we did a lot of the testing hands on by like just running benchmark scripts manually. and now we're been able to develop like this whole repoo over the past year and a half of itdurestandard benchmarks, custom benchmarks stuff for reliability and burn in. um you know, it's it's like very automated now where we can launch this stuff with the coding agents and have them run in fixed evench, isises es ttonomously without us involved. and yet, i mean, just the the breadth of coverage as well as the deathth of coverage, i would say, has improved over time,

[Speaker 3]: but i guess introduce it, what is cluster max? what is new cloud ratings and rankings? um what are like the main things you guys are actesting for for years?

[Speaker 0]: yeah. so there's ten criteria described on the website, basically a new cloud. we define as anybody who rents access to compute cells access to compute. so neo cloud implying that there was hyperscalars and legacy clouds that existed before chchchb b. um this is AWS google oracle azure. also, some of the ultical clouds like a wackspspace or digital share in or even like a cloud fllaor acammi. anybody body was like running a big days and a footprinting renting access out there would be a cloud. and then the new clouds are the new ones that have come along. we say everybody's becoming a neil cloud because what makes new class differenis that they're specifically focused on ai. so they deploy gps and other accelerators for AI worclothes. this is training inference increasingly RL. and that's what we trying to test on is who's going to give you the best performance, security reliability? and um just like ease of use pricing availability, we try to go through comprehensive assessments instead of individual benchmarks back. when jutubt came out, a bunch of people were reaching out to semanalysis saying, like, hey, i'm trying to decide the foot twtween providers, like how should should think think about how how should we go about it? you know, bothering dealing with the same questions over and over because ok, i just raised twenty million dollars. i'm but to go give ten fifteen million of it two up provider. you know, it's my whole seat around like this is a really critical decision for me. i need to get everybody body speedback. and so instead of dylon, an team like answering that question individually cost. er every single time was like, why don't we write an article to give our like course grain? first, thoughts, here's are like quick tearliest rankings um podium, and then you can engage us for more detailed stuff. you know, consulting dd, whatever it is for your specific, you know, requirements, and we can begin further from there.

[Speaker 2]: there's a lot there. uh uh, the the one that consumine is the is the criteria of the ten criteria. uh the one that consumia is the, the one that you guys cled it, which is area as a curversaying. uh every all then have terrible cvver security. what is the the stof of ymmer security in the ocloclods?

[Speaker 0]: i mean, you said it, terrible. yeah, it's dependent. you were talking about. look,

[Speaker 2]: yeah, we know. i i'all just say hi to deallwho, who is choojust just rathis.

[Speaker 1]: one of so sorry for raing late.

[Speaker 0]: so know what's the state of a new cloud cybersecurity right now. so so i mean,

[Speaker 1]: we you know, i think um security is really bdiand back from the first cost ter max. um there was um pretty significant security issues. um people not properly implementing network security controls. and so we 're able to see other people's uh jobs, see other people's 's data, see other people's storage at stura in the first generation of customer max. um we escalated to the firm as well as in videa, um you know, and rank that company under perform, of course, and strongly recommended people don't use them um. and in the second generation of cluster max, we saw similar stuff. i think a lot of people got better because we highlighted stuff um but in the third generation across your max, we also tested. i think you know, jorn this team tested this and and saw pretty major security issues. so we posted as uh a news lighter titled, uh, most new clouds um sucket security um and then uh, you know, handful of hours later, illia tweeted about most new clouds security sucking. um there are obviously a very security conscious firm. um you know, they have a lot of folks war from the former cyber's curiy world. so imagine theyve done their own work on a number of clouds there um to see um and test. and it's it's pretty clear right like what it is so bad. in one case, there was we were literally able to see national intelligence of certain countries like stuff, right like that multiple national security agencies in military style, um intelligent stuff running in that neocloud. and it's like good god, we're just like care to test the cluster, and we can lixsee the stuff. imagine if we are militious, i'm sure there are people who are milalicious ous or could have in militious um on that cluster as well. so thankfully, they should fix it. i think, but like you know, we shouldn't be the one that has to find whether or not someone is ah you know, good or bad of security, but you know, i think the the industry generally needs that service. and so that's that's a big area of our criteria.

[Speaker 0]: yeah. and i think on the technical side, like this is our checks are very simple. is your software version up to date? are your drivers up to date, like is a configured correctly? i mean, is there these are relatively simple checks. but when people are running versions of software that her two three years old, and they have well documented CVS online. it's like you don't need a frontier model to create a piercexploin of this. um you just need to know the right right place to look. and so part part of ararticle. we also put out of free CI people can installed clustom max, c, max and then use it to check the software versions on their clusters idendenfy. you know, a quick like you get a link in the termal output to show you where the upgrade stop, try to help people keep the software t today. this is the beyering is a new cloud security.

[Speaker 1]: that's a thing is the network keys, right? there's um in fidevan and internet have various like at least on in fitamamon siit's like mkeys pkeys. all these things are those properly configured and deployed. it's shocking how bad most deal clouds are, though like it is really bad.

[Speaker 3]: well, you get tting. i guess, any anything interesting, any major jumps, any improvements that they've from one two three ain't on a highlight?

[Speaker 1]: our bar on cluster max es know, as first generation was like super law, um and each generation we made it harder and harder. hence, the trend has been for people to get downgraded. there are bit a few folks have been upgraded. of course, um googles don't a really good job at getting upgraded across the cluster max generations. um i think they started out as bronze and moved to silver. and now gold. um we're just silver. sorry, i know gold dad. so google is googles consistently upgraded. while there there have been a number of clouds who have been downgraded across the generations because they sort of stall, they improved a little bit, but they didn't improve a lot. and and we keep trying to raise the bar um and and teach everyone. hey, these are the things you should do to be better. yeah.

[Speaker 0]: and the bar for this time was like latest and greatest. so are you deploying GB three hundred and BO seventy two in twenty twenty six? you know, this is incredibly popular p. it's what all the frontier lives want to buy. it's what all of the new lives want to buy. and the typical customer that comes to us and says, averaged one hundred million dollars. i want to spend on on compute. they want AGV throughout ded. and we also seventy two cluster. and so some of the providers that were in gold originally do not offer GB three hundred and the the ulseminy, too, why not we've got to contend with a whole bunch of different things. there's a new network and the ulsemity to implies it to scalup domain in iraq. you're tto go to arm cpu, it's a black old GPU. you have an eight hundred dig network. it's mandatory drug liquid calling. so from the physical data center through how how you to play the operating system, tell you like manage the drivers in the network, just about everything different different when people say, oh, we just like made a strategic business decision choice not to do gbs. well, well, what happens when very rubin comes next year who you going to trust? and so the trend is like if people are going out and they're deploying ten thousand GP three hundred and ds, and we test cluster and it works. and we engage with the support team, and we we give them feedback from some things we think we have issues with and they fixed in an hour, not two weeks thethey're. moving up the rankings. other people that are not deplying these these test test british tripues, or they're doing it as completely broken, is taking them a long time to fix it. or it's we find security issues or we hear from customers huge issues with reliability, which is probably the number one criteria that like these top customers care about, um they're moving down the list st. that's that's that's bbit of dynamamic there.

[Speaker 3]: so assuming most of these you know, brown, silver, honorable mountains are good. good to rent from what's the state of availability caused compute. so say, you are in new olab, you just raise tens millions of dollars. you want to start to rent um how does that market look now compared to one to three years ago? things getting better? worse, good, trucking, loks.

[Speaker 0]: yeah,

[Speaker 1]: it's it's really tough. it's it's it's toutouest. it's ever been rect compute um i'will, you see ethropiic and opening. i've been buying smaller and smaller clustersizes um but also all of the inference providers are more more profitable. it was one thing. one say, based ten was running at ten percent rus margins, which i think might have been like six months ago, right um on inference or or nine months ago. but it's another thing when you know, based ten fireworks together, many other companies, even small companies like morph are running at sixty percent per margins ins on open models using veal emeractually elying using just a little bit of optimization of beyond that um unable to do it, awesome job. and then you know, companies like infraact and radc sark as well. so so anyone who can like get access to a jep, you can immediately make money. um and to ultimately, it's like we're great. i'm a nail lab who wunants to train our interesting cool, new model. well, i'm also competing against people who make money. now, where's before? i was mostly competing against money losing, you know, open ing money, losing in thropic money, losing labs, very few people making money off the gp s. now everyone can make money off the gps, and that's just very, very difficult to get any sort of compute.

[Speaker 3]: what's avoid the loves. yeah, what aavoid the abs going down to compared to like major, they,

[Speaker 1]: they used to only go like KKK, you know, AKGPS from a fwrider, you know, and it's it's steadily gone down. um we've even seen as small one thousand g pipuce get rented by the laabs. now um and so one thousand g piece is closer to what a new lab wants. um and so you've squeezed out the market a lot. um and then when you look at like the vase tens, the fireworks, the togethers and all these other inference providers, they're cool like they're cooled, like four notes like their model, right? these guys are cooled, like even four notes that are cloud. um so ultimately tely. it's like, you know, obviously ororchestrais tough um models talked a lot about that. actually right? they're orchestration across like twenty different clouds. what's yes. so so ultimately, it's a very, very it's challenging for anyone to get any sort of computer. if you know, model can make money hand over fest off of even eight, four notes from my company.

[Speaker 3]: yeah, it's bad. i mean, like even individual, you can spot rent single nodes anymore. like down the whole stack is cooked. um what about the state of the labs? how do you segment the large players open air? and thropic google, where where's the spate of who has the most compute? uh, anything interesting on the road maps of where you think incluinteresting the most.

[Speaker 0]: i mean, it's clearly opopen eye and tropic that are leguing the way in terms of both renting from like large scale bear metal hyperscallers. in other words, you know, open eeyes a driver of correeves business, also some microsoft that are also exploring with other hypersckillers. they're doing self filled as well, like massive stuff ff with oracle, and then throropic obviously they got there. they get up AUS partnership ptrrinia. they've also got a lot of tps with with google cloud. they sign stuff with corary if they've sign stuff with with know know others. and so theare the big driver. i mean, you just look at error and something has to drive that. so it goes to the computer side. um with that, is there aspected?

[Speaker 1]: this is um you know, a lot of these providers, like like a crucial have fantastic bear metal, right? um that's that's what we're testing in customer max. we're not testing um beare medal, mean it. it part part of crititeria. but a lot of the criteria, a managed cluster, right managed there manmanstribaneties all that sort of stuff. so it's it's a different level of um service that people want on top of the GPU. um the original cluster max was title, also like a big chunk of it was called how to rent in GPU youalso. just teaching people what they should do to rent to GPU. um in in our view, most stly smaller folks, you know your super small, you don't care about. um you're note being beare metal, you um you're super super big, you want beare metal because you're going to put your own, you know, fork of of slum, such commcommunities um and do your own stuff on manage services and such like that. um but most customers want a managed slum, manage commbities um and for the cloud to like hot wap notes for them. when one fails uh, you know the biggest customers will manage stuff like them, but you know, very the the mid size and smaller customers want the that sort of stuff managed for them. and so customer max is focused around that. um so companies like crucium um will rent bear metal, right? they have mass contracts with companies um or they'll do great in data centers or though they're starting to do well in, you know, inference API, but on manage services, they're not doing well um and that's specifically is what cost your max is targeting. um it's manage services there because it's otherwise it's like too complicated to um you know, it's it's it's many different areas. people want to rent from some people want better mattal some people at matter services, some people want tokens out. um so ultimately, it's a range of options. um as far as you know, good even google is ranting some compute from folks, right? they sign a deal core corerve. they signed a deal, it's baseacx um so so pretty large amounts of compute that thethey've signed over. i think over one hundred billion dollars over the compute that they've signed from various uh customers. um navbe is also as deals with the labs and such. and so um but these guys are only going for their modal. um open line and thropic are definitely line sheer bying. the most um google google is definitely falling behind at this point, open ing and and thropic have as much compute, if not a little bit more by any of the year, though both have more compute dedicated to RND than deep bite. so yes, google still ultimately has more compute than good open iron tropic for now um but a gle that is sold out other places and dedicated to other offerings. um but deep minds RND compute verses open en ion on tropics RND compute. the two labs have more compute now because they reach that scale.

[Speaker 1]: um so this whole, like oh, google has the most compute. beell in is is actually not really the case anymore, right? they don't costs advishh because they build the TPU. and um it's much short cocost, but ultimately it's it's it's less deep mind has less compute than open and and profc.

[Speaker 2]: yeah ah ah re's, a big, big shift ft because y're, uh uh, the biggest insights from me, AIE was running into some deep, my researchers and are very depressed because they're like, look, our leadership keeps selling the computer that we want to other people.

[Speaker 0]: yeah ah i mean,

[Speaker 3]: amazine being XAI like,

[Speaker 2]: well, yeah, i think you know basspecis was a new cloud. they would the hell, well, we where we unavailable.

[Speaker 0]: they are on here, heart art ah ah well as dias. we can tetest. they'll move up the rankings. you know, they'll be a competior on think space like is. and the google deal, for example, is super interesting as well as a ththropicone because it goes es what dylen was saying about the fact that the labs are considering becoming new clouds on this opportunitic basis. um matas considering the same thing with the midraw and put outside and there as well, we company have this skills out on building data centers and managed managing them. and then when there's an opportunity mootizze, the compute, either through renting ting, the other people or theyserving infrence rence, they want to pursue that only only a certain level. and they always want the optiontionality bring the computer back and use it for those research or hero runs as soon as they can afford that want. want to use maxxmum amount of compute they can afford to the research right now and all the exresure, you oopportistically making the money to fund all that research。 and so space ecxis contact with google, for example, has mutual ninety ty cancellation rights. so space ecx's research starts to work, they can pull the compute back. google stoff making money off of that compute on some inference and point. they pull a computer back. it's a really interesting structure where both are kind of, you know, in this little stand off saying, who's going to who's going to blink first, who who believes in AGI more um h going going,

[Speaker 1]: i think togeother aspect of of at least for space sex, right? if you're AXAI employee and you got bought by space ex, and then space ex sold all the computer, at least you're not depressed, because, like you really, okay. look, he wants building a shitload more. i'm going to have access to this. and he's doing this because it helps us be more financially solve it. so the iio um with google, it's like why the hello, by selling the compute, you know, i have. i'm already a very profitable company. uh, why why can't we just use the computer to build models and sell those models at way? hard margin inthat are giving a hanthropic or open the eye or other firms.

[Speaker 0]: um no SSS is also a er er google 差不多 through of the the dynamic of like google funding, the people that leave to go, do a neil lab to buy gcp. no, it words like they've silver jefting. they spent out they get funding from rural ventures to then go spend on gcp.

[Speaker 1]: i mean, i think it's it's a fantastic move. if if if you look at just gcp as it's own company, right? thomas, korean continues to pumpers. you know, his revenue, his operating margin is operating income for google as a whole. it's really dumb right, like you should not incentivize your best researchers leaving. you should not incentivize us a generally and has a brave. so that point out thirty million dollars of megawa when you could run models on it for fifty million dollars of megwa.

[Speaker 0]: so if discovery loop has a big breakthrough, do you think google is able to keep them in their orbit? or do you think jefting in goes and takes breakbreakthrough does something without google being exposed to a the upsite, like they were be an investor thethey're. obviously, their oud service provider.

[Speaker 1]: i mean, go look at anthropic right like, yeah true. google did invest in in ropic um but if google didn't invest in in thropic with a dropic beer today, because googles are really early believer um like open three and and onate three and three, five like those. i think we're just trained on googles info for shirk. um you know, it is, and then they later brought on amazon, which was, you know, amazon also are great, at least amazon on know, just like they don't don't have a business of training models in selling mmory. in the same way, they have some folks, but like nothing, nothing crazy, but google does. and it's like did google like stand up a ultimately their biggest competitor uh fighter. now google now anthropic is not only renting from amazon and google, but also from numerous other clouds. they have deals with like ten plus clouds. now um and then enthropc is also building their own data cennors leasing their own data cennors, starting a phillimmap um working with foods stack and deloay tps in their own data centers. so sure, google still getting revenue, but it's like, well, i used to rent from you now buying keep you use for me, which is less margin for you. and now in the future. i'm now i'm now building my own chip. i mendo everything without you. so so like it's pretty clear what happened with the topic just after you, if he does something amazing and discovery loop, and why would they continue? why would they be stuck to google, right? they can use any cloud switching. clouds is ultimately not impossible. it's not to it's not in data gravity and all this stuff that people leased to talk about with clouds in the prea era is a lot less relevant in the post in in the ai. err because AI can help you migrate everything, and AGPUAGP is not just AGPU, but like you know, AGPU is largely um workable as as a fungible asset across clouds. if you're good.

[Speaker 0]: so think googgles curirkman.

[Speaker 2]: i mean, i do these days are much more than exweek than i am. um and there's like yeah, this google is the a conlomorate of a bunch of different things, right? uh. so i do think like the data of enage having there's data grareer as a as new club. but uh the data vantage of google is the the thing that most people cooky coming back to 're, right? um and uh, there's just organiztional challenges. and like you know, now there's new new leadership. um you know, gobless, the corey, who has say the hardest job in it wolves to to to fix this. but like i me's a very, very capable guy um and like i think i think they they know like i you know, it's like not a secret that, like uh, they need do need a reset. and um to me, damiis, like basically stepping on the leadership, see it, which is like a complete black ks, an, like i just make. i never thought it's possible, like maybe i just don't know impersonally where like it's like yeah, actually like i just care a lot about i sympophic. but like um i mean, you know, he, he, he found found it like he's 's um he's sees. so like the spiritual leader of bemind. and and so like this, like this is probably the first major frontier lab transition leadership transition fully.

[Speaker 0]: um can me say that there was like two days were some left train. but yeah,

[Speaker 2]: as i mean ah, never never ft in in in anyon's, say, hearts. uh yeah ah yeah, this is the team cook of like, hi, yeah,

[Speaker 1]: youyou living up open eir.

[Speaker 0]: now is my my plalater, yeah yeah.

[Speaker 2]: it was like kind of like the yeah, yeah. yes, okay. so you obviously people spspt right like anyway. um so that's my my gh thtager. i don't think i'm i'm going to have like uh space you takes the you guys i did want to pick up on a one thing is. so this is i just a chance to like catch up on the last six months of like crazy new. yeah, you guys mentioned poll side. if theynever never acshould written about poll side because they're not that big um but uthere's, they do egphical ical um h got blot by uh the employees got blot by in video for two billion. what's going on? okay?

[Speaker 0]: yeah.

[Speaker 1]: um it's really cool right, because in videa is paying for all these researchers. so they can make nemotron better. now,

[Speaker 0]: this is posiside is poll side posiside.

[Speaker 1]: like you know, from the beginning like they, they just like all the cloud, new clouds kind of sucked. and so they started building a lot of their own infront. they got better and better better building their own infront than they were trying to play their own clusters and had a partnership ship that correve on this and stuff like that. but they're just been solely moving down the stack, maybe the first AI lap to uh become you know, running their own infa even before open and and in in the opperer were trying to build their own infa that whole side was trying to.

[Speaker 2]: i which is wild is a side projects. it's like people have to do. this is a full time job. this is a sisite project.

[Speaker 1]: well, now it's the full time job, right, because they sold all there are researchers to uh in video for nemo rum um because in video clearly wants neneo ron to be a lot better. i think they've sort lost open a lot of the um open source folks. i mean, sure, rereflection is still trying um thinking machines open source some stuff. but ultimately, i think in videos losing sort of hope american opopen ource. so they think they have to do it themselves. um so the've bought a budge of employees. um and yes, i thought there was seven billion dollars, not a not twelve, but anyways regardless. um but there was like a seven billion dollar, then a one billion dollar investment at a probably we have of other detadetatoday i toabally.

[Speaker 2]: i take my different number,

[Speaker 0]: license, employee or six.

[Speaker 1]: the number six always always take the bigger ger number. but you know, i think at the end of the day, it's like in vidia can say, well, actually, we only paid one forth the price, because all of this money just going back to buying gp s. so did they pay? you know, seven billion dollars six for the employees in one in an investment that they actually pay. you know, one and a half because all seven of that is going into buy gps. and in fact, i'm doing this is going to insentivize other people to invest a bunch of money into them. and so actually it's a it's it's probably a positive thing tired. these people like they they actually inspend any money because now they have created a new customer, a new cloud that's going to build a massive cluster. right? like imagine yeah, theyve got seven billion dollars now of cash. um they raise another few billion dollars, as should city of ten doillion dollars of cash on the balance cheat um and they're able to get a loan, the value of like seventy five percent, um yeah, twenty five percent uh down and seven, five, five perent ent. once another forty thousand thousand dollars, they can do the gigwa oso. this is another digoa scalill near a cloud. um so thethey've, just yet you a new competitor.

[Speaker 2]: you have that lower down in the in the article. um it's uh yeah. i mean, II don't know. i don't know what, uh, if that this is the playbook, if this is the playbook, honestly uum low it down, loit down. um uh the the guy was leaving it, robert bonr. um.

[Speaker 0]: and like if this is the playbook, a lot more people should be doing.

[Speaker 1]: this should should be doing.

[Speaker 0]: what should should be DD should be doing deals with in video like this, because an in video will king make make just because they they want monocrtes. yeah,

[Speaker 1]: i mean, not everybody releases. the second vests open american model, and you spspend years building it. i mean, we're kind of brushing over the fact that pulls i did build something interesting. and to me, both this and hugging face are defensive acquisitions by invideo, who does not want want their comtition tion, snapping up a lab that actually has something going for them and having some momentum where they can build the real experience that's required to build real training and inference clusters. in other words, it's so important for building an infrastructure company that you actually understand how the workload works. and you have opinions that are informed by tratrainmodels ls and running accreate s. look, get represented in the not ecyped of miners who decide. hey, i found some site. i got some power. i have the money. they don't have the ability to execute, like somebody who has proven they can train the second best american opens model in the world, which fools i did, right?

[Speaker 3]: yes, say more 样的。 i was going to say, you mentioned higging face and there. how do you go see the higgface acquisition? anything you ganna comment on there?

[Speaker 1]: i think it's quite quite similar. i think it's defensive. i don't think that invideo wants the influence in the ecosystem of other companies coming in and owning somebody that's like curiricle.

[Speaker 2]: infit is a galaxy brain, take care, right, which is um now now hugging face, you know, can can see the fucket of opening i for hacking them. um and now vividea can can can has has that has that hamit. i don't think exside of exul. um but i think i think i think hiking face is obviously like, uh, you know, leagues ahead of everything else. um some some folks on my team have started adminiting, a much of PR for model scope uh to work across the ecosystem.

[Speaker 3]: um my leagues ahead in. no, it cense.

[Speaker 2]: i just like for for open source popostory for yeah, just like what hoging face us. um but then ultimately, a lot of them are. uh, you know lake. it's it's you know. the question is, like does huhugging ace ace know there's a way you can just like deploy hugging face models um on amazon traineum. if you want right, you can just click about in the or the poeum. um does that go away as it early? er GPS from here on out um or is there any closing up that happens? um we'll see. but ultimately, it's it's um and II also love that the price was uh the o moi e moso say.

[Speaker 1]: and the teams incredible. look, i think wewe're kind brurushing over the fact that hugen face and poll sight have incredible talent in both ororananzations. and like a lot of people would see that's the motivation for invidia to acquire them. i say it's defensive because if you look at invideo's business, they're generating almost fifty billion dollars of free cash flow corter. so tunor billion a year. and there is a corporate strategy team at at in vidia that gets some amount of that money and says what we need to deploy it right? they can make investments. they can make acquisitions and ten percent of free cash flows to acquire hugging face in full side. i mean, it's a no brainer to me that these would be good assets on your team at working, at least not for your competition. let's say, hugging face days independent, and they follow the letter of the law in the acquisition, and they keep supporting all these other chips and stuff if they were required by somesomebody else, they would be actively working against invideo's interest. and i think that's not what they want. so i think you're going to see more of this, if companies grow up a little bit, they have build this team hundred to two hundred, incredible people. they want to bring them into make sure that they stay in the invideo ecosystem as oppposed to going and competing against them with somebody else.

[Speaker 3]: yeah, makes sense not to discredit anything pole sides done where uh pretty big fans. we had them on the pot of month before they got acquired for people interesin the model factory work approach to training foundation models. uh very, very good way that they built all this out. you know,

[Speaker 0]: to me coming, you know, maybe maybe this brings us back to customax. uh, the way that you that all new clouds should be proving like cosmax four is like just train a frontiy model. please h, yours yourselves, right? uh. and then i show everything works.

[Speaker 2]: i don't know everyone should everyone should have the skill set to train the fronter, right? that's just a kind of,

[Speaker 0]: but there's not to open recipes is stuff right?

[Speaker 1]: let's talk about what it takes to get that right, if you're going to train, i think, but also, i think procide and probably hugging face of probably both entertained acquisition options in the past to the tune of similar money. and it's yeah. so specifically in video tried to buy poll side, uh ah h viwhich try to by hudking face uh at the start of the year, right? that's in case people did know well.

[Speaker 3]: and i think also, and video is on a bit of a acquisition. crazy are rumors of thinking machines and a whole bunch of the the acquisition.

[Speaker 1]: yeah, i the court ch ounders kind of have the they have the option to like cash on when they want for a lot of these people that have built these things, right? so when do you decide to cash in? i think both i is probably looking out there. i'm going okay. i've built the ten thousand GPU cluster, i needed get get icggin it out. but if i want to compete, i need a research cluster. that's hundred thousand GPS four hundred thousand gps. and how do i keep up with the open frontier into next year? thanks, that's pretty hard to keep going in. the money comes in. there are like, okay. sure. let's just cash in and build it with the other people, that, that seems fine. but the other thing worth pointing out is like on the technical site, we prosasaze papers are really good on the infrastructure stuff. i've talked to the team. but if you're going to train to frontier model, you need to have health checks on your cluster. you need to have high reliability. you, you just just have this model like not make progress as you're training and constantly be failing. um coursight has that they have like an incredible sweet of um they just have incredible infrastructure. so the uh, the idea that people with that talent would be able to provide that as a service. other other people i think is like just obvioulater. uh and there's only limited people that have that experience ces were same ying even when when get into the silver in in bronsteer. er um people are not training frontier models on most of the silver teer and all the bronze teer. let's say, a lot of influence bit of our AA lot of research, but i went trust any of those guys, frankly, to build smptasure ring, but to build a hundred thousand GP coster. i crew is proven and they can do that already.

[Speaker 0]: yeah, not a low five hundred minuyeyeyeah ah oay. i do want to maybe uh dive in one of two things that maybe you don't h don't actually cover. maybe the best way a, given the time that we have left is to just address the criticism. um uh, you know you really, uh, obviously, this is a very valuable list that people do be purchasing decisions. offeits uh periodic glads uh did come out and say, like yey ittually ally, uh, you know this, this helps us decide what what we spend on. um yeah, i mean that you ah, what is the charrticism that most people have?

[Speaker 2]: i think i think there's um a lot of criticism for a lot of things like first and foremost. um this is testing manage costers. um so that does not mean that a company is not good at other things, right? crucial, it was downgraded for manage costers. um but there are one of the best data center builders in the world. um if not like the best they have, they've nearly the most contracor or or the most contracted gigle lots in their pipeline, a bununch of sites. they're building on really rapidly, but this is not a criticism against there. you know, dis on their building capital is it's only a criticism against their manage customter offerings that they were theystill one of of the better es es, right, but they dodowngraded right. um you know, likewise, you know, other companies together has great inference and points that has nothing to do with manage services. um you know, and and so think think there's a lot of confusion around what this is testing, right? this is for manage clusters, not for data center construction or or bememetal clusters or for infrence cand inints, which which just come a blurure line because everyone is doing these stakes right? um the other question, the other big major criticism we have is like, oh, you know, people like, oh, are they taking money for this? as like no, we do not take money for any ranking. uh, that would be illegal. um i myself have personal investments in crucial. i have personal investments in floodstack and neither those companies. um you know, i did SP vs in both of them. neither those companies are in the top. you know, five clouds um because because that's not what you know, i'm not here to be by a strike. jordan, well say he's like he's like he has free rain til like do whatever he wants to do. technically, he does even know all the companies you do contracts with. um if i look at the top twenty companies in the world, there are five companies that are not our customers. um yeah like e like lily and saudi ram co and sounds like that. but ultimately, it's like those are much bigger customers than core river nebbias could ever be right, because you know, they just have their way bigger companies with way bigger budgets um you know. and so like eighd, AWS is a bigger customer than core riban vious, for example, right um and yet, that has nothing to do with the rainkks, right? everyone in the infer supply chain buys our stuff. um it just is what it is right. if you're tracking data centers, you are know other people doing their data cennors. if you're tracking excelerator supply ly ands, if you trying to understand the cost of various kinds of cholers, if you're trying to uh understand the supply chain for networking in optics, you're going to buy a research and our data services that does not mean um and you often commcommission ed us to uh work beyond that, right? um you know, but that doesn't mean that we're going to ever bias our rankings. um you know, so so sort of i think there's a lot of criticisms around that, but ultimately, like first of all, who are taking money and ranking people not disclosing yet that be illegal. um but second of all, right, it know makes no sense right. the value we have to the industry is that we tell the truth in what we believe um if if we didn't do that, then no one would kure what we would say we'd be screaming into the void. um so our reputation is like what matters the most right over a quick box.

[Speaker 0]: yeah, maybe one way to put it is like your you're byysides and people people people accusing you taking money for ccl side de, and they don't want to say the byside is like probabone thousand times larger ah, enters the revenue for you that to put them the cell side.

[Speaker 3]: i mean, i think people trying to poke at it's a paid list, you know, and that's just one IQ. the brother point is, like, okay, people misunderstanding what you're trying to rank here. and a lot of the claims are just well. it's crazy. this is a better cloud x is AA better oud d white, right?

[Speaker 1]: but that's that's what you're actually testing youyou're. uh clear criticisms about the writing from people where i listen to the criticisms, because i, you know, respect their work, and i you know, if they're not like anonymous just showting as the aoid. and those criticisms are, you guys didn't write enough, you should have released more data. you should have wrten more or this is thirty thousand words. i can't read this all so it's really hard frous the balance between those two,

[Speaker 0]: but it's exa new. yeah, you're evaluating new clouds, of course,

[Speaker 1]: like so so for the former, i'm you know, interested in that criticism, and we we want to do better. and so we have this research products called the cloud tco model than anybody can come in license. it's a yesually fee. we write notes every week about the trends in the industry. we have a dashboard, which we took a reducted screen shot of below where you can get all of our data, all of our performance status on their yeah. it's like you know, you get us working on the research for a year around weekly rather than just ah h this individual, you know, kind of once a year release or every six months or something as what we do. so that's for the people who want more detail.

[Speaker 2]: um there were another crowd of criticism, m such is like, um it's often time someone who like owns one of the stocks, and we ranked them popoorly positively ly, right? like i remember last time, people were were like, like better way, like our a view on a company doing better or worse in the stock market, has nothing to do with uh jordan ous team, right? there's uh you know, we we do. we do estimate revenue for all these companies going on on a four basis um and then you can like license our data and see what you think that should mean you should trade. we never recommendous trade. um but theverban times were like, for example, um we thought nebbious was better than corory, and the reason we thought naabbious was better than correeven. even we thouthink cory was still a better cloud than nebyous, is because nevvius is contracts are shorter term, and so it prices of GP spiking nebvious gets to take the upside better, right? like that's something that we've released in our research in a separate separate group of people, not jordan and the technical staff that are working on costumer max, or or hey, like irish energy, has this massive data stthat that's going to come online. we think it's going to get least when we would put a report out like about that well of the same time saying you should never use irish energies um iron's um, man, age services um. and so like it's like it's it's it's completely different things right know. they sold that data center ters, a bear metal to microsoft. and so we were you know, a research on the institutional research side for some one who might be buying. stocks is completely different from like, hey, costmamax is really for people. people like perioticical labs or applied compute or companies like that. it was publicly came out and said they look like costter max, and what we do there. um you know, even opening ice, and they like costa max, and we have like a quote from them on on that um wehave, a variety of different buyers who said they really appreciate the ethodology and that we are pushing the bar up for all the clouds. and so i think it's it's it's you know if you're trying buy buy stock based off cluster max, that's like completely wrong. it has nothing to do with stock performance.

[Speaker 1]: yeah, it's a hundred percent man. this is strictly for buyers. this is for helping people buy compute. it is not for people to pick what stock to yellow all their savings and it,

[Speaker 0]: i mean, it, it, it will corrently uh at some kind of limit. um i think like i think isn't a tendency. it's litererally. it's literally in quoral ly for my money guys or so ok oright oh,

[Speaker 2]: my gogod right like someone someone is terrible running a cloud, but has a gigoot data center that they're building. and in it's like actually credible like a giggooddata center. well, they could be that worst ranink customer max, and they could still make shitloads of money.

[Speaker 1]: um elah like dodoesn't matter at the car of price at the curar price of the stock. many names are cheat. many names are expensive, like, yeah.

[Speaker 0]: yeah, II, do i do? i do get it fundamentals, which is uh, you know, one offering uh, that contributes to overall fundamentals. it contribubuto the overall uh, h valuation, interrection or data. i used to be a, it fronone guy. i get it. you know. i'm just saying like this is like you guys are the subdomain experts and people do value the rating on that elements. uh. but obviously, this is way more by side, uh and by is in uh, you know, the the the new lab.

[Speaker 1]: yeah, i cerarsorto to just keep dressing this. but we genuinely have a core research product where people report on starks. and i give opinions about the names and many other guys ys that confact research. look, they ask us questions, but then they come out. they say people in the underperforming are unavailable. tear were positive on the name. this is different than what we put out publicly and say you should rent cluster shoong. and maybe i need a another rex claimer on the future one. that's like this is not a cosign of this current valuation of their stoalk.

[Speaker 0]: it could be, you know, it could be really interesting for like a two way way or or like a two dimensional chart wear, like here's the question, max ranking. and and here's the seminalysis, like stock ranking and like it, the diffediffeces are are going to be interesting.

[Speaker 1]: and i was like the research uam, but maybe worth single there is a lot more coming, which is that didn't mentioned it. individual names and logos on here have businesses that are much larger than just managed clusters, which is what we're waking here. and so we are going to be addressing that on the newslettter for inference and points for bear metal for our infrastructure for sandboxes, for you know, part of our oil infrastructure. but kind of different um we're going. we running about all of that and stay tuned for that because i think people will start to have a more holistic view of these businesses that compete in different market segments across inference, managed clusters and mbaronmental. and we've got a lot of focus on manage clusters here, but frankly, so many people are reaching out to us for help, deciding on which and point provider to go with there with, which hosted our tratraing provider der go go with or which sandbox provider to go with. and that's a different question than who to go with for your managed cluster.

[Speaker 0]: yeah. so so is this solution more maximore benchmarks?

[Speaker 1]: yeah, that's so yeah.

[Speaker 0]: okay, how many you have yefrence?

[Speaker 2]: it's it's it's me when you go shopping when you go to grow shsure in hundred hundred reds, the ll, right? we we want to do everything. i'll see. i'll see. hopefully, we can execute all all of.

[Speaker 1]: yeah, we need be to be clear, though. influence acts today. the name is because it's ininference performance. it's about about ps ps, it's about comparing gbps and systems, and we're going to be relation something called end point x where we test the several st infrence ence point providers and kind kind a dig into those details on uh because i think there's a very similar methodology. we know there's a similar methodology that we should apply where we we go beyond just performance. we know we look at cost. we look at reliability. we look at security. we look at monitoring inffstructure that there's all of this stuff that goes into an inference and point that says, just become because somebody had supiirer h target on batch one, you know, contrived tests doesn't necessarily mean theythe, right? partner took scale up your infrence requirements with when you need to all that extra.

[Speaker 2]: yeah, there's providers with better performance like raw, better performance, better optimization engineers, but then they're like api less stable. it's flaicy er um yeknow. there's there's all sorts of reasons why it may not be as good for the end point and customer. even that's like more tokens, persecupper user and loter cost.

[Speaker 1]: yeah, amazing.

[Speaker 0]: um do you ever? how will you cover the let's all all the the non envidia universe?

[Speaker 3]: um so which is risant, you have someone here, right, like tenser with this mt cloud.

[Speaker 2]: yeah, that's another one where i explicitly have an investment by the way guys. and and we still ranked them poorly, um you know, time until like that had investment in the first cluster max that put we put him as under perform, right? i think like a lot of this bias against like it's it's yes, i might make an investment. i'm going to talk to them and trying help them. but at in to the day, the ranking is the ranking. um so we we do want to extend beyond. but ultimately, like you know, foods dback is not going to rent rent TP use dos right, because you're renting everything to anthropic right.

[Speaker 1]: ah maybe well, see ve maybe II you.

[Speaker 0]: yeah. it's kind of an excusiable that they're unavailable, like i come on,

[Speaker 2]: guys like, no. i mean, it's it's valid at the same time. it's like look, it's unavailable to us. so we can't we can't we can't rent it. it will be to rent to neither unless you're really, really freaking cool um because they're selling everything like space ex is not going to rent to us. they're not going to rent to to use wax are not going to rent to to, you know rent companies.

[Speaker 1]: they're going to rent massive cloud and you do. and i think it know, i think you need to work on your, i think it is. so if is the first time i ever given you this feedback, and i think you need to work yourself, confidence, man, we we can get it in there. i think we can get it in there with these guys yet. well, i think over time like today, tolly, a lot of these providers, even fluds back and and people that we know who are at space XAI. now um see value um reading the article, and they see value in getting in outside opinion, even if they're not necessarily going to provide the service is publicly that they offered off. so they, you know, they they enjoy a third, you know, i thankher ical opinion. that's outside the company. that's honest that's going to give them real feedback. and we're here to help for all that stustuff to answer the question about multsilicon. i'm going to be huge next year, like with the very ruban generation within video. we're going to see it because they're going to bring the lpos in from crock, the LP acsystems right? so even if we're just testing in video wegoing to have multsilicon to test emd obviously has their own thing. so rivers is becoming a neo cloud for opening air blowling, their own data centright um moving from just inference and point stuff to like really managing clusters of their chifts. for everybody else, we see him and and overcome in a market. lots of the chip starus like posiatuum edged mateacks, like lots of people are launching stuck like this. um there's mmors, you know, obviously, fluds ducks got TPU outside of GCP. there's more stuff come on there. traineum lots eight of bius can produce a lot of chips. maybe they won't self build all of it in the future. they've got analysment son on had virginious traineum with the rivers and stuff like that. so i think there's going to be a lot more for that next year. and and we're going to have to contend with multisilicon as part of a comprehensive offering from the leading meoclouds. they they will need to be able to support heerogenen ous computer in the future.

[Speaker 0]: and that's all focused on infrrence mostly right, right, like, uh because of it's, you know icic.

[Speaker 2]: i mean, most most ethropic, for example, uses TPS for trading primarily, um so in that case, is sort of the opposite right. they don't actually most of their inferences on trainium in GPS. most are training is actually especially like, you know, fable five was trained on TPS, for example, for for the pro training, and then the post training and inferences mostly the trainum in GPS. so it's a mumultsilicon oi would say, in most cases, other silicon offerings are for inference, but there are some cases where people using it for training.

[Speaker 1]: yeah, but that i think the key thing is infrence persuis training is not this hard to blit now or there's a lots of forward patches during oral when you're doing research and when you're actually postering the model. and so um in fact,

[Speaker 2]: most of the workloades is forward passes, right? and so you might call that classically inference, even though for you know, four passes for reinforcement learning, um ah ah ah infeence is part of to do.

[Speaker 0]: yeah,

[Speaker 1]: there's a whole bunch of benefits of using ah let's say, cheaper chips capable of high throughput for the ford past in RL. but then there's a bunch of challenges with numerical corcorrectness, if using two different ships, even just two different generations of individia gps be two hundred versus be three hundred, you might have different amounts of memory, different batch sizes. there's all these effects. so II absolutely expect there to be heyderogenunion silicon used across training and inferenced pretraining probably a hamamoginous ster. people probably not going to risk on it, like four hundred thousand GP scale, but also yeah ver like you know,

[Speaker 0]: RL is like as big or sometimes bigger than deprettrain uh workload right now, right? and so that actually means that you reve, uh, like maybe AA forforal disaaggregation, where like you have different kinds of compute in the same data center, but you shuttle,

[Speaker 2]: but it doesn't have to be the same data center, right? people are doing oral across many, many sites rright. this data center doing token gengeneration that data center is doing token generation. it's actually fine to do different types of chips. there re's just a very complex infer problem because you're going a charger model differently and wait up daata at at different times. and there's all the two feet of a acing pror people listening this right now.

[Speaker 1]: we're like a god. no, please, no, you have to a day of the center, multi silicon RL updates on the flight gude.

[Speaker 0]: please know you're fighting speed of light and how you you going to ansfer er this this data.

[Speaker 3]: i mean, theis interesting architecture stuff being done on disaggregated like discentralized oral. uh, the memo two point six that just came out. it's a good open paper on how they super scale of the oral side of things. um but on that point as well, you mentioned a lot of the custom chips, anything on hailway chinese chips. um do you guys look at them?

[Speaker 2]: yeah, we're tracking them. the wilway chips are really uh getting better much faster for a while they were like, you know, basically, when they were bned by the os government took them like a few years to reset. um you and the assaid, nine ten BNC, uh, but theyyve ve ununched, the nineteen forty and nineteen fifty and nine sixty in pretty quick succession. um they you we're trying to sort of did access to them and test them along all bounds, right? whether it be renting them. so we can test the performance, whether it be um you know, looking at the specks in the manufacturing capacity and things like that um also trying to access them. so we can do reverse engineering analysout of our organ reverse engineering lab. um we we try to do work across all of these um dimensions.

[Speaker 3]: um yeah, as the as the availability, other clusters, like are the labs, okay, without any videa, any any early science in general gabing people are super。 i mean, the labs are using, you know, ethropic uses ford of kind of i mean, like the the chinese labs using their own chip ufrom.

[Speaker 0]: like like yeah ah,

[Speaker 2]: i know it was um GLM said explplititly, right? that i had uh, fifty thousand chips that we're not in vidia. there are chinese made uh accelerators for inference of their news model as we extend out to other firms. um you know, camericon, n, vatar and flame and many other thers. there's a handful of different companies um they're all pushing up the curve. you know, obviously, there are further behind in videa. how way is definitely the best of them all? um and then bby is spinning out their chip business uh as a completely separate company. so there's a lot of developments in the chinese chip pical system. and and especially for inference, i think you know, the china, you know, as as frontier models um become even even even the opopen source models become better and better at writing kernels. it's easier easier to poor stuff. um. and so ultimately, um you're going to have more diversity in hyoriginity, uh, which is why in video is moved to the superfast paste of trying to release new chips every year, many systems over year, because they know they have to run as fast as possible about the speeet of light, otherwise they will get caught up to um because the cudo mode is graphicdly being determined,

[Speaker 1]: but it's just volume right telling like the um the trips do work today, but they can't produce enough of him to really surreal lder domestic demand.

[Speaker 2]: so yeah, you know, if you look, if if you do it one one eequivalence, or let's if you do in gv, two hundred equivalence, invidea is making millions ons, millions of chips googgles, making millions of millions of chips amazons, making millions of millions of chips MMS, making millions of chips for. but then if you look at the chinese companies really in the only one that even in growth chips y're making a million plus um and then the lalowway chips ps are behind. so then you have to discount them for ff everyone else is stole. the hundreds of thousands of units and scaling production is very difficult. specispecigiven export controls, ls stuff. um um i china will ran production rarapidly across next year in the year after um and they will get to even tens of millions of chips produce. um i twenty twenty eight, um, but it it does take type.

[Speaker 0]: yeah, memean h t's not not get get the, the little first, have a national plan around all this stuff, you know, when when they get going, get get going in al al ske, like like like we've ever seen in huhuman history. um okay, lot ts to discuss here. congratulations on question max. um and all this stuff, the events are ing it. uh. i think we manage to hit on like many of the major topics as like the year so far um is k lot. get a catch up on like the the hardware lad that guguss a building, an organ. i don't actually love to see it, and all we should do like some kind of try.

[Speaker 2]: yes, fout right out will. can you give you a tour? yeah, yeah, yeyou you. you that entever you are going like they will do a tour ah.

[Speaker 1]: i want it it. you don't not give you arrive in his orange jeep. um we i want to go to abbuly ly like, let's. this is do of interesting. you know, i mean, obviously, you guys been there as as well.

[Speaker 0]: anyway, ah you guys going go go um is great ccatch uh, congress on everything. and also like, i think there's a lot of of dowork to lose jordan. give us a sleep peak at the kind of things you're doing uh and point next sounds interesting uh. and like the market is really segmenting here, right like to me. that's that's what you guys are identified. early in are raising the start of everyone. so i think for you know, on behalf of everybody, like thank you for doing this.

[Speaker 1]: thank you, and thank you ah ah ah ah,

[Speaker 0]: the ss ss, right?