Okay. And we're live. Welcome back to another episode of Adventures in DevOps. I'm Warren Perod, one of your fellow co-hosts. And today in the studio, we're joined by Jillian yet again. Hello, Jillian. Hello. And you can see that Will is absent today. I think he had some critical problem yet again with his infrastructure that he's managing. But I'm looking forward to today's episode, because we have with us as our guest, Hikari Senju from Omnikey. HIKARI SENJU HIKARI SENJU Hello. Hey. So maybe you could just give us a brief introduction about your company, what it's sort of doing, and anything else particularly interesting. Sure. So my name is Hikaru Senju. I'm the founder and CEO of OmniKey. OmniKey uses data to generate more personalized ads and help businesses scale their advertising assets. So customers, oftentimes they're running ads on different advertising channels and they're looking to drive efficiencies in their advertising. And what we do is we'll train a model of the customer's brand information, we'll train on the customer's performance data, and then help them scale their creative assets for a variety of different dimensions and SKUs to help them ultimately find ten X higher productivity in terms of their time, in terms of their cost and savings, and then ultimately in terms of the results. Okay. That's a lot of things. Yeah. Yeah, we call ourselves the GenAI command center for brands that are looking to scale their digital advertising. So whether it's analytics or ideation, we really try to be that place, that control hub for customers to manage their online advertising presence. What were they doing before AI stepped into this space? Well, gosh, AI has been in advertising for a long time. The personalized content that you see recommended, like deep learning before it was used for generating images and copy and content that you're seeing today was used for targeting personalized ads. And so what we're doing is we're just taking the next step, which is instead of just personalizing the content delivered to you, also personalizing the content itself. So before it was mostly about when I see an ad on say Facebook or LinkedIn or on the top slot on Google, that content may have been using AI to generate the actual words or pictures that I see, but you're utilizing it in a more novel way within the product itself. So before it used to be, um, so there's the metric here is that forty percent of marketing projects are wasted on ineffective content. So oftentimes you're getting retargeted, uh, with, you know, I'm sure we've, you know, the audience's experience where they're seeing the same ad over and over again, um, ad fatigue. Um, uh, there's the, uh, lack of, of like effective content being, uh, being bought and kind of distributed online. Um, and what we too is, uh, when the content you see that actually actually made or that, that video, um, that, that content itself, uh, is personalized based on your interests and your needs. So it's not that you're seeing the same thing that, um, you know, that's kind of broadcasted everywhere else, but you're seeing a personalized piece of content. Um, that's personalized for just as much as your newsfeed is personalized for each day. Well, I definitely don't think my newsfeed is personalized for me. I mean there's a lot of things that show up there that I really just couldn't care about. Yeah, I mean, I think it's still a work in progress. I mean, the algorithms are getting better every day. And it depends also on the platforms as well. But yeah, I mean, I think still there's the personalization we see in social apps. And then there's the lack of personalization we're seeing when it comes to the content itself, the advertising content that you see. And so at the end of the day, if advertising content is more personalized and more effective, at communicating what that brand and that company is trying to communicate, if you can make that kind of connection with that consumer in a more productive way, then that's better for the consumer because I'm not being targeted at random ads, I'm not seeing random ads, and then ultimately it's better for the business as well because as a consumer, I'm seeing more relevant content and communicating in a way that I like. And for that business, it helps them with their sales. So this is really interesting because I think in advertising, it seems like there's so many different angles that the AI could use. And I don't know a whole lot about advertising, but just kind of off the top of my head, it seems like there's how the ad itself is being delivered. Is it on Google or Facebook or is it? Predominantly social platforms. yeah, which social platform it's on, the audience it's being shown to, you know, you have like age demographics. I get a lot of teenage girl things, which makes me wonder about the algorithms a little bit, but that's because I have a teenage daughter. And then there's also like the ad copy itself. So does your product help with all of those things or one of those things or like what is going on there? Yeah, it focuses on the visuals. So the images and the video content that you're seeing, that's what we specialize in. Okay, okay, that's interesting. I didn't notice like a few years ago, or I don't even know when, but I noticed with Google, it started to have, Google Ads started to have this new feature where They were like, we'll just train it and figure out where it should be. You just give us the ad copy, and then we'll go send it out. So it seems like you're on the other side of that. We're like, no, we're going to generate the actual ad for you, and then somebody else can distribute it. Yeah, I mean, Google has been definitely innovator in this space. be like, we're going to actually limit how much control you have in terms of content. You just give us your assets, and then we'll kind of mix and match for you. What we're trying to do is that, but for social posts and content you're seeing on YouTube, and also to make sure that the content is consistent in an omnichannel way so that when a consumer sees an ad on Google, it is consistent with the ad they're seeing on Reddit or LinkedIn. Because, you know, if Google is kind of communicating your brand one way, but rather it's communicating brand a different way, then it's not very, it doesn't elevate the brand experience and it doesn't create brand consistency. And so ultimately this is really where managing your brand online is so important is that you don't, you know, you don't delegate the creation of your brand or the imagery of your brand to each platform, but you really can orchestrate that brand messaging yourself in our control hub. Interesting how so I think one of the things that advertisements have struggled for the longest time at least from my perspective is Adequate personalization more so than just like hello, you know templated name and then generic repeatable message in the body Or an image that isn't very personalized How how are you? getting over that hump like if the data wasn't ever good in the first place like you are you getting better data now to be able to generate personalized uh content uh or is it that we had the data all like the whole time and we just weren't using it effectively does that make sense yeah I mean there's an element where these platforms like they for example they could You could actually run ads like this where it's, you know, hello, Warren, buy X product. But it would be too creepy and it would actually result in you leaving the platform. And so they intentionally don't, allow advertisers to do things like that. And they'll ban advertisers to do things like this. Um, because the platforms themselves, you know, they're trying to, um, you know, maximize the engagement of the platform and they're not trying to creep out their users and, you know, make it seem like they know more than they need to. And so, um, um, there's an element where I think, you know, like that creepiness factor, you know, uh, pushes, is, is it kind of a pushback against like hyper, hyper, hyper personalization? Um, and then, and then I think the other thing here though, is it's also just creative bandwidth. Um, you know, if you are, uh, you know, brand and you have you know I mean this is you know even even even like the the biggest brands in the world they may have thousands of skews creating content for thousands of skews across you know in every single platform requires a different ad format so meta instagram reddit linkedin snap you know etc etc they all require you know videos or like horizontal or vertical content in different dimensions. And so it's just a lot of work to create all that content. And today, a designer is hired to do those things. So that's really where automation comes in, is that an AI can automatically create all those variations for all those SKUs for you. So in our platform, customers can just upload all their product assets, and it will create all the variations for you. A-B testing and inserting different product skews and also creating different output variations and dimensions. I see. That makes a lot of sense. So you're utilizing the filters or tags or attributes that your customer already has and how they're already personalizing or segmenting the market and using that to feed into a model to generate the images based off of well maybe each of their products whereas before if I'm like the one of the largest running shoe manufacturers in the world I may just have one ad with just one shoe in it or maybe I have a couple different runners so I have a couple of different shoes but I can't have the same video with literally every single kind of shoe hyper targeted to some set of the market where would be more interested in that particular shoe whether it's hiking shoes That's exactly right. So this is really where generated AI comes in so handy, is because what we do is we can train a model on each product, on each shoe, and then generate images and videos. And I'm sure we're seeing this all online now of people wearing those shoes and those variations. And we can do it very cheaply. sets, generate those collateral assets with cents instead of the thousands or tens of thousands of dollars historically required to do a photo shoot like that or a video shoot like that. And so we empower personalization and we increase personalization by lowering the cost of creating content and lowering the cost of advertising. So that big enterprise, but also small businesses too. By lowering the cost of advertising and lowering the cost of trade, we also empower small businesses to historically wouldn't advertise or wouldn't advertise a lot to be able to do this. And I think there's been kind of a barrier to entry in some ways for advertising because how expensive it is. But by lowering the cost of advertising, you're empowering more businesses to be able to advertise sooner, connect with their customers sooner, which is kind of our mission to democratize growth. One of the biggest costs I was always under the impression of was actually getting the content out, like not necessarily creating the content, but once you have it, actually paying to get it in front of the right audience. And you're not necessarily changing that. You're just lowering the... entry for the content generation side? Or is there something you're also doing for actually getting those clicks at a cheaper volume per price? Yeah. So that's where we're this is really where agentic agentic stuff we're building really comes into play. today and we offer this both as a service and and kind of and we're building this out in terms of an agenting platform but um uh you know as a service I mean today you have something you know you have an advert you have a person who goes and buys the ad on meta google and linkedin and adjust the bids or adjust the targeting audience kind of do that manually by hand Maybe this year you'll have an agent, an AI agent do that for you instead of a person. So an AI agent will go and measure the click through rates, for example, for every single platform or the ROI, how much revenue is being generated in terms of sales from each advertising platform and then adjust the bids and adjust the audiences autonomously, you know, twenty four seven. And, you know, again, at a fraction of the cost of of a person doing and working twenty four seven. And so that is also an area that we're innovating in. OK, that's really cool. You did talk about training a lot. Maybe I want to get into that a little bit. I assume you're not building up new foundational models and maybe utilizing something that that's already out there and either fine tuning it or building. Are you building something from scratch? Like, what does that look like? Yeah, I mean, we describe ourselves as kind of a mixture of experts. So we take the best models in different domains and utilize it to help our customers deliver some kind of a solution. So for us, for image, like hyper-realistic photo generation, we do fine tune on a pre-existing model. For us, we fine tune on images. uh as of today you know this may change you know as these models advance but uh today it's a it's like a flux model by black forest labs um and uh and that's how we generate hyper realistic imagery and then we use different model turn that into video uh we use different model for you know predominantly open ai models for uh generating copy and creative briefs and uh and tagging the ads. Well, we have an AI go and tag every image and video in terms of the elements of that creative in a more deeper way than just computer vision. So our first version of the analytics where there's computer vision, cat, dog, maybe cat drives more clicks. Now you can have AI explain the image in a much more nuanced, deep way. What is the persuasion tactic being used? What is the use of language, et cetera? And what kind of imagery is it in? And then we can turn those into creative features that we can then find trends in terms of what's driving more engagement or driving more clicks and communicate that to our customers. So we're using different kinds of AI depending on what part of the workflow that's being automated. Okay, that's really interesting. I'm wondering, do you have some type of I don't know, the only word that I can think of it is, you know, like some kind of boutique experience. So for example, I was on Amazon buying something and I noticed the pictures there, there was like two different products that had the exact same pictures. And they looked like, I don't know if it was like stock photography, or then they just switched out. It was like leggings, they switched out the pairs of leggings or not. But I was immediately like, I don't know what this is. I don't know if this is fake. I don't know if this is a real product. And I clicked right out. So I'm wondering, like, is there some type of safeguard to make sure that everybody who has a pair of shoes is not getting the exact same image or even very close to the same image? How do you even measure something like that? How does all of that work? Sure. So Amazon's interesting because a lot of their ads are just straight up product photos from the catalog, from the product catalog. And so, and oftentimes you're just being targeted at those product photos and using the same product, but maybe it's like, you know, this is, I don't know if there was two different companies or it was the same company, maybe it's two different companies, like reselling the same thing. Yeah, I don't know. That's why I clicked off of it. I was like, I'm not going to do that. I'm going to go buy something that looks like a real product. Right, right, right, right. And maybe, who knows, maybe they both got the same image of the manufacturer or something, right? And they were advertising with you. And just Amazon knew that maybe you thought that you were in the market for this kind of product. We do have controls. And so we do have a human in the loop to really make sure that the content that's being generated is IP protective, so it's not infringing on anyone's copyright. that it is brand safe, that it is aligned with brand guidelines. We actually have a separate model. When we train our language model, we train it on the brand guidelines of the customer. So we're generating content that aligns with the restrictions of that brand within their brand guidelines. and then we also have a human loop so both at the kind of inception training of the model we train on the brand and then the very kind of uh human review step uh we have um uh you know make sure that the there's no ip being violated or there's no brand uh principle that's being violated Yeah, because fundamentally, it's still on your customers to validate the content that they're putting out is in line with what they actually want. I mean, obviously, you can assume that at a giant amount of assets, you no longer can have someone individually reviewing everything. So you must have things in place to help cut down on that I think the custom trainings custom models um you know multi-step evaluations uh two-shot uh you know helps a lot but fundamentally the customer is still responsible for and probably wants to validate that what they're getting out is actually uh accurate yeah absolutely so there's a we have a we actually have an uh approval dashboard where the customers can write feedback uh to the platform in terms of this is not right or this is right and um So we built this whole workflow for review as well, because I think a large part of AI going forward as well in different parts of the workflow is going to be some level of human review. I don't think anyone, even as we give AI more control, there's going to be safeguards in place for human review. Yeah, it's really about how do you, you know, like build a tool where it used to be that a human is doing all the work and now human is largely just babysitting an algorithm to do the work for you. And what are the review steps you put in so that a human can do that effectively and get consistent, get good results. I mean, I think one of the challenges is there previously, it wasn't just that humans were creating the original content. And I think this happens a lot in the software development space where you may be using an LLM to generate lots of code to put up a bunch of pull requests. The amount of code is so high, the number of images or videos that you're generating is such a large amount that the flows you were using to review that initially may not be even viable anymore. So like we fundamentally may have to actually change how we're evaluating and how we're actually reviewing the things that we're generating using LLMs. Yeah, absolutely. And this is really where I think automated tests come in handy. I think this is really where, you know, this is kind of Not necessarily what we do today in terms of our product, but synthetic data, creating data to make sure that you test against all the different edge cases is important. And having the proper evals in terms of evaluating the models, so the model is being fine tuned in the proper direction for the proper outcomes. Those are the things that come in play when it comes to, it's not just human reviewing the massive output, but kind of a systematic approach to fine tuning and improving the models. Yeah, I can see there being a lot of value in that review platform. I'm curious, I think this question keeps coming up in any number of the communities that I'm in, like what's the best LLM or what's the best model out there either to run with or to potentially retrain Do you see them as like also interchangeable for the most part? Like there's one that's better today, but in three months or six months from now, it could be a totally different one. And have you oriented your tech or your business around that idea? Or do you feel like you're mostly bought into a single one for today and it would take a lot of work potentially to change in the future? Yeah, I mean, I think the thesis of our company, as soon as we found it, in fact, you could kind of review, see this TechCrunch Disrupt interview I gave a couple of years ago, before the whole Gen-AI. We were actually the first company to present as a Gen-AI company at TechCrunch Disrupt as a finalist on their stage. They had a similar question, which is like, Why are you focusing on the application? What is the strategy for focusing on the application level at Layer? You get your seeding leveraged through these foundation models and the best models providing this base content. And the answer you give then, I think the answer you give today hasn't changed, which is this technology is largely being commoditized. And there's a pressure for the commodities of this technology because the innovations are open sourced. It was open sourced then and it's open sourced even more today. There's massive competitive pressure by the big tech companies to develop innovations and make it cheaper and better and faster at an incredible velocity because it helps their cloud businesses largely. And so in order for powering the cloud businesses, they need to keep innovating this space. There's a financial pressure to do that. And so for us, we've always been kind of plug and play. What's the best model at any given moment? And we'll switch out models if there's one model that performs better in a certain domain. And there's also that. And then I think the other element is just being transparent with customers as well in terms of what models we're using. Because certain models have safeguards. Other models don't. Certain models, in terms of maybe their data protection might be. There are various. Every model has its own unique quirks and nuances and issues, both legally in terms of the output and the training. And so we're very transparent with customers in terms of what they're comfortable powering their advertising content with. Interesting. So you said that the written content is going into OpenAI's chat GPT in some cases, right? Yeah, I mean, OpenAI has APIs for accessing their foundation models and just some of that. But there is a toggle you can use when using OpenAI where you can ask to not train on the data that you're sending them. Yeah, but your customers don't necessarily know that. So how do you, is it transparent? We do, we do. So part of our terms is we're transparent with the customer in terms of which models we're using. And if we're sharing their data with any third party. And if a customer is not interested, not open to sharing the data, then we will change the models. Or limit the functionality of the product so that they're completely transparent. you know, place safe there. And actually this was, you know, just going over a legal document yesterday on this with one of our customers. A lot of our customers are agencies or, you know, advertising agencies. And they work with some of the biggest brands. And, you know, as part of an advertising agency working with a big brand, you know, you have to, you know, indemnify and do liability protection and do all these things because obviously the big brand doesn't want to accidentally expose themselves in terms of data leakage or or copy right violation through their agency and so and so therefore that then is passed to us in terms of that vigilance to make sure that you know there's a we would you know when it comes to Therefore, there's pressure for us as well to make sure that all those safeguards are being met in terms of data protection, data leakage, and that copyright concern. So if I'm running an AI company, I'm not, but if I were and we were using some other company's models, either through an API or a public model, do you see that there's a more common concern in a way? Is it, oh, we don't want you to train on our data? I've seen that one. I'm curious how frequently that comes up. And then potentially, do you see requirements that there are certain companies you can't use, like certain companies are fine. Certain models are okay. Or do your customers, they usually are, are like, they're using your product to do, they know they're using AI. They know they're using your LMS that have been retrained from other models under the hood. And they generally are okay with that. Yeah. So, um, it really depends on, on the model. So there are certain models, like some, there are some sort of state of the art models today that, um, And by the way, there are some really popular consumer apps today, very mainstream ones used in enterprises and just by individuals that have zero liability protection. It's maximum of a hundred dollars that they'll pay if there's an issue. And as a consumer, oftentimes you're not reading the terms and conditions. So you aren't aware that you're taking that risk when you use a product like this. But most generic consumer, I would say almost all consumer generic companies today, if we use it, has, in terms and conditions, I get a cap on their liability. It was like about a hundred dollars. And so consumers are taking that risk when they're using some app to generate some image or video and kind of sharing it online. the potential IP that they're violating. When it comes to enterprise, enterprises are very, very sensitive to this, and so they'll almost require you know, indemnity protection and liability protection. And so some of these enterprise-oriented AI companies offer this. But the default is no liability protection, no indemnity. And so, you know, a lot of consumer apps, almost all the consumer apps are like this, a lot of APIs are therefore like this. So if you're using, you know, a state-of-the-art model, you may not, you know, there's a lot you know there's a lot of really interesting nuances here again you know also some of these models um that are open source they're not a commercial license uh so you actually can't commercially use these models that are open source even though they're the technology is out there you can play around with it but you can't actually use it for business because it isn't commercialized you have to go and negotiate commercial license directly with the model provider Um, you know, if you actually say if they're an API, uh, then, you know, read the terms and conditions to make sure that, you know, what, what the safeguards are there. Um, and again, it, it varies, you know, it depends and varies depending on, on, on the, on the model provider. Uh, but, um, but, uh, but yeah, I mean, I think the default generally is we don't take liability. We don't indemnify, you know, you're at your own risk, you know, using these tools. Has it been easy to work with the companies that are providing the foundational models to like they understand the enterprise risk? Or has that been another level of a complex conversation to have both legally and what a support contract would look like in order for them to provide you the safeguards that you need so that you can promise your clients to meet their particular legal? Yeah, I think it depends on the company. I think OpenAI is pretty, you know, because they have a big enterprise business. They do a lot of this liability protection and indemnity. But if it's like a startup that doesn't necessarily have a large enterprise business and it's predominantly consumer, they don't really, I mean, for them, they're making enough money through consumer apps that um you know it's like yeah like the risk of potentially them indemnifying isn't worth um uh that potential you know line of business because they're not focused on on enterprise and um Yeah, so it depends. I think it's not like, but yeah, these are areas where I think, you know, you know, yeah, I mean, we've reached out directly with various companies and model providers and places that host these models to figure out the right terms and, you know, get clarity regarding data training and things like this. And it is kind of, again, on a case by case basis. I know Jillian was really excited by the topic of AI whenever it comes up. I am. I'm always so happy about it. I could just ask you and ask you and ask you questions forever and ever and ever, I think. Well, this is your opportunity. I know. Don't worry. I'm going for it. I'm wondering, what is the kind of backend that you guys are doing? Since we are a DevOps podcast, are you using cloud? Do you have your own server farm? What are you... What are you doing? And then also if you're using, it sounds like you're using like the foundational models, but if you're using any custom models, how are you dealing with like hosting and deploying those? Like just walk through your whole setup. What's going on? Yeah, so we're an Amazon Marketplace partner, and as part of being an Amazon Marketplace partner, our infrastructure is hosted on AWS. So I can probably pull up a diagram in terms of this, but then in terms of the models, we'd either be hosting models within Amazon servers, or we'll be doing access through an API. I mean, throw out some AWS service names here using like SageMaker and Bedrock and, you know, whatever through guardrails or, you know, what's going on there? Are you like pure virtual machines or bare metal? SageMaker UI, like what's going on? Yeah. I haven't talked about that UI for like weeks. It's been like four weeks. I still can't make heads or tails of it. Yeah, I mean, we use a lot of those in SageMaker. I think predominantly that we access through various other APIs. I'm not sure how much of SageMaker we use today. You can also note that I'm not the CEO of the company, and so we have a person dedicated to responding. They would be a lot more better at answering these questions. um you know we're on amazon we can just keep talking about the front end stuff then that's I mean yeah a little bit more like ml than devops but uh that's fine for me so we're good yeah so there's like an s three router uh uh sorry there's a necessary bucket with all the media stored uh the content stored it's gonna be we're almost like a dam a digital asset management tool for our customers and so we manage uh all the customers brand assets and all the generative assets as well um so that is quite a large part of the tool and then um We have load balancers that connect to the Nexus, which is then connected to PostgreSQL. Oh, that's really in the weeds. That's probably more than you need to expose to anyone. So, I mean, you're basically utilizing AWS infrastructure as much as possible for running your product, your services through there. Yeah, but it's Dockerized. We can go and... hosted as a different server or a different kind of cloud provider. But yeah, today, we just kind of started with Amazon, and they gave us some nice credits. And so most of it is on Amazon. I understand. That's how I got started with Amazon, too, is they gave me money. Yeah, exactly. They are very generous with those start credits. Oh, they really are. They're good. If you keep harassing them, too, about their new services, be like, oh, this new thing that you guys have, I think it will really transform my business. And then give you money. I don't know that I should be admitting to that on the air, but I guess that's what we're doing. They have great customer support when it comes to supporting startups and help, you know. Yeah, I mean, they've been great. Well, I think it's absolutely fantastic that you've managed to figure out a way to put your models within Docker containers and are standing them up in AWS because I know just so many companies would swear by needing to buy the GPUs and install them in local data centers or run them under their dev, whoever their engineers are, run them under their desks in order to be effective there because they would complain that maybe the cloud is too expensive. And then the customers that are using or building models in even AWS are using ECQ machines straight out and not necessarily containers and then have sort of not great flows. I think there is this weird mix here where the companies that may have great models or great attention to what's going on in the AI space aren't so great at what is considered best practices from a DevOps standpoint. And it does seem like, you know, at least as far as evolution goes in technology, like, you know, you've made I think you're further ahead than what most companies I talk to that are doing some sort of AI are at. Well, that's really great to hear. Thank you. Yeah, I mean, I think we also bought some GVs ourselves and kind of playing around with them and experimenting with them ourselves. But yeah, I mean, I think the scalability is a lot better through these cloud providers. I mean, first of all, it's hard to get your hands up physically on, say, H-Line hundreds. And then so it's a lot easier to get through these cloud providers. And And then, you know, then just again, scaling, deploying and all these things also a lot easier, um, through, uh, through the cloud. So, um, you know, for us, like we're not trying to, you know, make our lives harder. We're just trying to build the best application utilizing AI. So like, uh, you know, like we have all the engineers focus on that. And, um, um, yeah, like if it's, yeah, there's just a whole level of complexity associated with GPUs and get them properly set up and all this. I mean, I am a little curious about the pipeline, which you may not have too much insight into, like the foundational model that you're getting from Flux. Where is that coming in? Like, are you getting that outside of AWS and then basically injecting that into AWS to do the retraining? Or is it being provided somehow through either like available in some way through the marketplace that you can get and then retrain it within AWS? Or like, I don't know if you have any insight into that, but that'd be a little bit interesting. Yeah, so there's a couple of model provider companies that support, say, the Flux model. So we kind of experiment with many of them in terms of their speed and cost, the quality of the outputs. So there's a company called Replicate that's pretty good at this. And there's a company called Fowl. There's some other companies, and they have APIs. What do the APIs do? Like, are you getting just the foundational model from them and then you are retraining in some way or are you retraining it within their product? How does that work? Yeah, so these APIs, there's many kind of parameters. So you have, say, the training images that you need to send to them. Maybe you have some other parameters, like the number of training steps you want to do or when you're generating that content, like the LoRa scales, like how... So there's like... a couple of parameters associated with training the models and generating content based on the models. And so you can use these tools also online. They have online interfaces where it lets you adjust it. But you can also do it through an API. And so then we're generating hyper-realistic, say, images based on product photos through those APIs. In those examples, are you sending your customers images to replicate, for instance, to train the model, or is it like a two-step process where first you're fine-tuning the model through the APIs they have and then taking the resulting model and optimizing it for each of your customers individually? Yeah, so in this particular instance, and we would be sending the photos to the customer, the product photos to the customer. This is, again, where I gave the earlier example of talking directly with the model provider. So in this instance, I directly with replicates. You do not confirm. They do not train on that data. They do not, you know, and they're not passing that data to some other party because there's also, you know, leakage in terms of the model providers, providers as well. And so just confirming that. And yeah, so we have a self-serve tool today for anyone to go to our website, sign up. It's a free trial. You can go train images and generate images and videos of your product, showcase your product. And some of that technology is available through, say, companies like Replicate and File that host Flux Models. I see. This sort of brings me to something that I keep running into sort of theoretically, and I'm wondering if in practice you've actually seen some problems here. The models, from my standpoint, are quite different from each other from generation to generation. There's not really this concept that we have in software development of semantic versioning, right? Yeah, exactly. I mean, I feel like every model is like a different child. Right, exactly. It has its own kind of like... you know, quirks and nuances and likes and dislikes. It's like and it's like an API to this person that it's like with all the different like it's not like consistent output. It's not always like you have this input. It always has this output. I mean, I guess the first thing would be just like virtual people or something. I mean, forget even having consistency within the same model. I mean, cross, you know, from one model to a different one, like you obviously have no consistency in a lot of ways. Is there some process you have for doing a validation when you actually do switch between like upgrade the model? I mean, there's some benefit arguably of the newer model. And so at some point you're planning on switching to it or a company and the flex model isn't exactly the same. Like, is there do you have like a standard process which you have sort of designed to deal with this problem? Yeah, we either get red team this or like we have teams internally that test this and just, you know, they really just, you know, test from every way from Sunday and make sure that the outputs are kind of aligned with, you know, the experience we're trying to deliver for customers. And so we just do a lot of extensive testing ourselves before, you know, kind of launching it to our external customers. Have you found any interesting problems as a result of that? yeah I mean I think ultimately it's just you know the the uniqueness of uh you know like the uniqueness of these models themselves and kind of it's not the standard I mean there's the standard kind of accuracy issues and some models are accurate than others and things like that but also just that these each model has like a unique personality in terms of the way they answer the questions and um uh And a lot of this is actually how they're trained. And so there's kind of another variable dimension to consider when kind of integrating these models into your application. I think, but yeah, I mean, there's numerous kind of, I mean, I think there's, again, the take the back and forth or kind of the trade off between quality and maybe IP protection, you know, like maybe the most IP protected or IP safe model is also meant for training very limited data and therefore the quality isn't as good while the best output in terms of quality. also is very good, because I think training data is somewhat or largely correlated with output. And the more data you have, the better the model. And so the better the model, oftentimes, there's more IP risk. And so maybe this model, therefore, does perform better, but we can't use it for a particular customer. Yeah. Cool. I'm not even sure what else to ask you covered an awful lot of ground. And it's just it's all neat stuff. Are you I'm sure you've got some burning question. You know, you don't want to walk away from this and regret later. Oh, not at the moment. But I'm sure I'm sure that you're right. And then as soon as I walk away, like, darn, there was all these questions I had. Well, just pretend the podcast is already over and then start living that regret. And then maybe something will come to mind. There we go. That could be it. Well, how many do you have, like a big team of data scientists at your company or like how does that all work in terms of like not only infrastructure, but also. Also people, because I think everybody gets a little bit freaked out by the AI is going to replace all of our jobs. And I'm like, well, there's still going to need to be people who know how to use the AI. So maybe, maybe just move along that path, guys. Yeah, we have thirty employees today. The majority are engineers and and I would say majority of that is like machine learning engineers data engineers um and kind of uh you know obviously backend engineers help deploy these models and so um but at the same time you know given the fact that we are an applications company it's definitely um it's a multi uh you know it's a nuanced problem in terms of we want to build the best ui in a kind of application for the customer to interact with these models but again the large part of model output, to your point, is data. And so building the best data importers with the various ad networks to get the most recent, highest quality data, to get the best insights, to generate the content is a big part of what differentiates us from some of these other generative AI tools. Most generative AI tools are not, the content generation is not powered by insights from consumer data. So we have directly plugged in with the ad network, so we know what's driving clicks. What's driving your return on ad spend or the ROI? What's driving engagement? And then we use this data to then power the content. And that is what makes us unique. And part of, I think, what creates the moat for our business is that unique proprietary data we get through these ad networks so that we can create more relevant, personalized content than, say, some image generation tool that you just kind of prompt. Is there, maybe this is getting too deep in the weeds. Is there like a huge overhead cost and the spend for utilizing the models and running them compared to say, running an application for customers that isn't so coupled to running a giant ten gigabyte plus model and retraining it for GPUs? Or is there like, is it just mostly overhead? And after that, after the model's been trained, the inference is much cheaper. Yeah. I mean, there's definitely kind of a scaling cost based on usage. But the inference costs and the training costs are getting cheaper every single day. So I would not say that that's, I mean, I think it's definitely a sizable chunk of kind of, say, our cloud spend is things related to hosting these models and training these models and serving these models. But also those costs are going down and our margins are, you know, ultimately, you know, for us and for most applications companies, what is the problem you're solving for that customer? What is the value providing? And for us, it's automated creative generation, scaling creative generation, you know, and that value providing is substantially larger than the cost it takes to, you know, do inference or do training. Yeah, I mean, I totally get it. I think one of the things that's coming up in the future is the realization that we need smaller models that are more focused, that are working together with some sort of oversight that drives which model is being utilized and how that model is being utilized effectively. Are you already at that stage where you have multiple models that are being selected dynamically or a model to choose other models? Yeah, we do. Yeah, we have a model router. we have a model router. So for example, if a customer wants to train clothes, that's different than trying to train other types of product photos. Or if they're trying to generate hyper-realistic images of people wearing, say, because you're doing your clothing company and you have all these different skews related to clothing, there's a specific model that's really good at generating images for that circumstance versus, say, you're an events company and you're trying to showcase a specific location. Or you're trying to... But you're really focused on utilizing a specific face that's in the image generated image or video. Each one of these kind of use cases has a distinct model that's the best for it. And so we do model routing on our end to make sure we deliver the best outcome for our customers. That makes sense. I mean, that right now I think is a huge innovation that a lot of companies aren't utilizing as much. They're utilizing the one foundation model that's coming from one of the major model providers, or even if they're training it themselves, they still only have one. I'm sort of curious, what do you think is next for the industry? Is there something specific that you already see as a huge challenge that needs to be overcome or something? I don't want you to spoil anything internally that you're working on, but maybe you see something that like the providers that you're utilizing, you want them to do. Yeah, I mean, you see kind of, you know, these kind of middleware, the rise of middleware and kind of like, you know, a lot of the the, you know, the layers between the foundation models and the orchestration that require, you know, go, you know, between orchestrating where these various models and, you know, the question is how defensible of a business it is in terms of building a business like this, because this space is moving so fast and you know, what, what's, you know, kind of an important service today may be completely integrated into one of these platforms tomorrow, or you're kind of commoditized based on some new innovation. And so, um, you're definitely kind of building on, you know, like shaking on something or like quickly moving ground. Um, So in terms of, you know, but yeah, I mean, I think there is, you know, and, you know, there are companies that are focused on model routing and like trying to find the best models, depending on, uh, and orchestrating the models, depending on what the need is. I think semantic standards there to be able to compare one model versus another one, know like fundamentally what the changes are, right? I mean, you can't, you may not be able to look inside the black box as you're building it, but afterwards you could do some sort of validation. And that validation process I think would be important in order to do the seamless swapping that we talked about earlier. Absolutely. I think a lot of it, you know, like testing and yeah, I mean, there's a lot of There's value in human testing and all. There's reinforcement learning with human feedback is a big part of what makes these models better and more. In order to do reinforcement learning with feedback, you need humans to provide feedback. There's a whole business associated with providing that. And so, yeah, I mean, you know we actually you mentioned earlier like uh you know the automation of jobs like there's going to be many new jobs associated with this technology because you know there's many new needs associated with new technologies like you know today I'm talking about digital marketing digital advertising I mean this wasn't even an industry uh you know thirty thirty years ago and so um um like you know new new spaces are gonna talk about you know gonna create new opportunities uh and new industries uh and uh yeah they call me about people to where they're most most productive. I feel like I should, I should ask, you know, obviously, your product itself has AI written into it as the part of the value offering. Are you utilizing any of the tools that help with the development process or organizational management within your company that are AI tools in nature? Yeah, I think meeting summaries and meeting recorders that summarize meetings, I mean, I definitely use these generative models for my emails, my product specs. writing blog posts, you know, writing kind of documents. I mean, I think a large part of leadership is kind of, you know, communicating and, you know, writing documents and kind of aligning a strategy and soliciting feedback and kind of aligning in the same direction. So there's a lot of writing involved. And I definitely use the Jedi tools to help me with that. I use Jedi tools, you know, in Gemini Deep Research, really cool to do research, like competitive research, you know, know who are the leading companies in this space and you know what are the unique edges and you know what what is our unique advantage versus those companies you know create a chart for me that does this and write a report for me um and um yeah and I think our engineering team uses you know cursor and some of these other tools for like helping with their engineering and helping you know be more productive in terms of engineering um uh so yeah I mean There's AI kind of embedded throughout our application, but also internally as well. We're using AI to improve our productivity and do the work that maybe a couple years ago, ten times as a lot of people do, now can be done with more leverage due to these new AI tools. Cool. I'm right there with you on the meeting notes being generated. That might be like one of my favorite features of AI is not having to write those things myself anymore. Mostly because I never, it was like a constant problem for me. Yeah. I mean, gosh, like. We have all these customer success calls, and they're recorded. I'm not going to watch all those video recordings, but I will read the meeting notes of all our customer meetings. That's so much more doable now. People will actually write good summaries now about the meetings, and then now I can read them and read more of them, and I can make better decisions. So meeting summaries are such a powerful productivity tool. I just always find it interesting how the internals of a company who is innovating in the space is innovating internally as well. I see there's a lot of duality there. And I always question when I see a company that's trying to innovate in the product space. And they still have a very legacy or waterfall mindset internally. But it's really interesting. You said, oh, we are using all these tools in some way, or we're trying them out and seeing where the productivity boost is. That, I think, is a really great sign that it matches the mindset of what your product offers is matched internally with how you're thinking about the culture of your team and the direction that you've set. Well, yeah, you say a really interesting point, which is, for example, Amazon at AWS was on Oracle servers until, like, fairly recently, like the innovator sometimes is, you know, dependent on a legacy stack to do the innovation. And, and, you know, some of these legacy players have humongous, you know, lock in. And so, yeah, I mean, thankfully we have the benefit as being a startup. So we're building a lot of this infrastructure from scratch. And so we can pick the best tools, but once, you know, you have a team that's kind of, you know, like you built all this infrastructure around like a certain uh structure like just a certain set of you know tools it's really hard to kind of switch over and it's just a massive engineering effort and and so like I think this is really where I think the advantage there is advantage to being a startup today is because you kind of you know a company ultimately is you know is is just people and the official organization people to deliver you know, a superb product. And AI is all about, you know, people ultimately being more productive. And so we can reimagine corporate structures more radically, you know, and kind of this kind of original kind of hierarchical org structure may not make as much sense anymore as well with all these new capabilities that are available. Totally agree. Uh, Jillian, any last questions? She's, uh, well then, um, Hikari, maybe this is, if there's anything that you still wanted to share that you didn't have the chance to, um, now, now's your, now's your opportunity. Oh no. I mean, um, Thanks for the opportunity. And yeah, please sign up for a demo on our site on Mickey.com, please. I can also try the product out for free. And yeah, I mean, thanks so much for having me, Warren and Jillian. It was really a pleasure having this conversation with you. Well, then we still have to do our picks for the episode. You know, can't leave those out or else Will will come back and complain. So Jillian, what did you bring for us today? I think since we're talking about AI, I'm going to do some shameless self-promotion. If you go to my website, dabbleofdevops.com slash AI, I'm putting the page back up and I'm going to just put a whole lot of disclaimers about how AI can't do absolutely everything for you. You do still need to have the human in the loop, like we were talking about. Like the Jillian. The Jillian's the human in the loop. I am quite often the human in the loop and I have to figure that out. But that is another problem for another day. But anyways, I have a tool. um that allows data science companies to have a self-hosted kind of platform in which they can do a lot of data discovery I have a lot of data sets that are already up we have open targets we have some data from the single cell website that of course escapes me at the moment and a couple of other data sites some pubmed queries all that kind of thing so if you're a data science company you want to do some Data Discovery with AI, sign up. It's all hosted on your infrastructure. I never see it and I never touch it unless you actually ask me to. That's awesome. We'll definitely check it out. We'll definitely check it out. So could you remind me what the URL is? It's dabble with... Okay, so I haven't actually put it back up, but I'll put it in the chat. But it will be up by the time the show is live. Because I had a bit of an issue where... I had a bit of an issue where people were signing up and they thought that it was going to do absolutely everything for them. And then I had some unhappy customers. And I don't like unhappy clients. Well, unhappy customers is better than no customers, though. I mean, not in the long run. Not really. If you're going to start off with no customers, then it is to have unhappy customers who then tell other customers how unhappy they are. Brand identity and reputation is certainly, you know, very, very important. But, you know, I also say the, you know, if you have no customers that, you know, getting some unhappy ones at least gives you some feedback on what you can improve. I know, but I got the feedback on what... The feedback was don't have the website up. So you took it down. Yeah, so I took it down until I could like work on the feedback a little bit. Anyways, people did that in school. They just thought that it would do more without another human than it did. But now they've got more data to pick that stuff up. This is, I think, the fundamental challenge with the startups. You have to get out something complete. There's that famous quote, if you're not embarrassed by what you've launched, then you've launched too late. But at the same time, you have to focus on the customer. You have to focus on the ideal customer. You have to say no to certain customers that's not really the target customer at the moment. And so you do have to say no to customers as well. And so I think it's definitely, there isn't necessarily like a... a hard and fast answer in terms of like, you know, it's definitely kind of more related to your preference and dealing with angry customers or not angry customers and how you think about your brand. Yeah. No, I mean, I totally agree. The startup advice as build for what your ideal customer needs, you know, is really what companies have to be doing. And I think that's been heard. I think the thing that now needs to be heard is that the people you're talking to may not be your ideal customer and your ideal customer may be someone that only exists months or years from now and it's okay to build for them but then you have to ignore advice from everyone else who isn't actually your customer because then you're not building for uh you know the future realistically okay yeah that's yeah absolutely um Oh, thank you, Jillian. My pick today is going to be Dune by Frank Herbert. I absolutely love the science fiction book. I think it was one of my first science fiction books that I read. And compared to other things, there's just such a level of attention to detail and innovation in writing about science fiction that I just don't think really existed before Frank Herbert. I'm going to go with this book called Accelerando by Charles Frost. And it's really about the acceleration of human progress. And it's really interesting. Each chapter starts with the MIPS for humankind, like Million Instructions Per Second. And as AI advances, the MIPS, the collective MIPS of humankind also accelerates. And initially, most of the million instructions per second, a lot of it is human brains, human computation. But towards the end, it's mostly just AI and the massive compute power that AI is providing. But through this massive power, you're building Dyson spheres from the sun and just exploring the galaxies. And so I think it's a very optimistic book on what this technology can provide for us. And I do really like this book. That sounds neat. What's the name of the book again? It's called Accelerando. Cool. Wow. Then that's it for... Neuroscience and AI. This is what I'm here for. Well, you got it all in one episode. And I guess there'll be a segment of our audience that is just as happy for this episode as you have been, Jillian. Well, then that's everything. Thank you to our audience for attending and thank you to Hikari for being on this episode and sharing with us how Omnikey is innovating in the AI advertising space. Thank you, Warren. Thank you, Joanne. Thank you.