AI Deployment 101: The Considerations of Safely Deploying AI in the Enterprise (Interview with Varun Mohan, CEO & Co-Founder of Exafunction and Codeium)

Recent AI models have changed the way people think about consuming AI or deploying it in the enterprise. 

Before an enterprise starts its AI deployment journey, there are some things to consider both in terms of the model and the expectations the business has towards it. 

Varun Mohan, CEO & Co-Founder of Exafunction and Codeium, gives valuable insights on the considerations of AI deployment in the enterprise and shares his own thoughts and predictions on what the future holds for these models.

This episode was written based on episode 83 of the Level-up Engineering podcast hosted by Karolina Toth.

This post includes:

• Considerations of AI deployment
• Quality
• Latency
• Correctability
• Benefits of deploying AI in the enterprise
• Concerns regarding AI deployment
• Security
• People losing their jobs
• Typical mistakes when deploying AI
• Setting the bar too high
• Lower your expectations
• Be patient
• About Varun

What should leaders consider before deploying AI in the enterprise?

Before jumping into AI deployment, there are a few things to keep in mind before integrating artificial intelligence into your operations. 

Quality

Consider the quality of your solution first. AI models hallucinate very easily - they take what they’ve seen during their training time and try to mimic it afterwards. The tricky thing is, training data usually consists of information found on the web, where everyone says things very confidently and no one seems to doubt themselves. 

As a consequence, AI models will also confidently state things that may be factually inaccurate. This might be fine for some internal subset applications, but including elements like this in products for your end customers is more risky - you don’t want AI to hallucinate false information about your product, for example.

Latency of the underlying application

Users want quick replies from these AI models. If they don’t get them, they’re going to choose another option, unless the quality is really high or when it offers something unique (such as Midjourney with its unique capability to generate art).

Correctability

Let’s say that an AI-generated result is incorrect and you recognize it. Leaders need to consider how easily they can correct the output in these cases. If you can’t make edits smoothly, then it doesn’t make sense to deploy AI - it’ll cause more frustration than good.

What are some of the positive impacts of deploying AI in an enterprise?

The positive impacts of AI deployment have been massive so far. For example, Midjourney hasn’t replaced artists, but instead, made folks that were unable to make art be capable of doing something in the creative field. It increased the size of the creative market in this case, and I think it’s similar in the software development industry. AI systems are huge time savers for us, not replacements for human work. 

It’s now clear that AI is not just going to make people more efficient, but it’ll also make it feasible for more people within enterprises to provide value in different areas. Our product, Codeium, actually saves many hours of work time per month, which can be spent on higher level tasks, such as system designs, instead of writing a lot of code.

What are the most common concerns of AI deployment?

Data privacy and security

There’s a lot of conflicting information around AI deployment and security. Something I found interesting is that Google actually told its own employees not to put private information into AI tools, including their own chatbot, Bard. That’s a pretty good indicator that the concern around privacy when using a public AI model is valid.

Large enterprises are usually very reluctant about sharing data outside of the company. Most of them even host their code on their own internal servers instead of using the cloud.

One way to resolve these differences between unsecure AI models and enterprises’ safety concerns is by self-hosting your own AI models. Many companies (including Meta) are coming out with fairly powerful open models that are competing with what OpenAI has. Even though the base quality of the OpenAI models might be higher, these self-hosted models can also be open, giving companies the ability to fine-tune their own underlying application and deploy that on their own servers.

In comparison to an open source project, these models are still a lot more private. When you release an open source project, you give access to weights of the model. In terms of these open AI projects, the creators don’t share how the model actually looks internally, they simply give access to the product, which others can further customize for their own application. These models have been trained with public data and have nothing to do with private data from your organization, so this solves the problem of sending data outside of the company, too.

In terms of AI going off the rails and causing data breaches: I generally think that these systems aren’t smart enough to cause real damages within companies, like doing something risky autonomously. Engineers are still going to implement these systems and they are going to have guaranteed SLAs and proper requirements. It’s still very much possible to engineer AI models not to do potentially harmful things independently. 

AI stealing jobs

I don’t believe that AI will replace entire positions. When we look at the history of building tools to automate tasks, they’ve just made people in those professions even better at their jobs.

30 years ago, people were using assembly language to program their machines and had to give raw instructions for the computer. After that, they started using higher level programming languages, more software was written and more people joined the software engineering field. In my opinion, AI will have the same impact: everyone in the field will be able to have even more and higher quality output.

The key thing is, when designing AI systems, people need to put guardrails in place. For the time being, humans need to validate the results that these models generate. Their performance will improve over time, but it doesn’t make sense to build an AI application without any supervision today - technology is just too far from doing that.

What are some of the typical mistakes companies make in connection with AI deployment? How can we avoid them?

Setting the bar too high

A common mistake is assuming that AI-based systems are infinitely powerful and can entirely automate a role in the company. The problem is, these models will be unable to perform large end-to-end tasks without guidance. They’ll be able to do some large intent tasks if there are humans monitoring and correcting their work along the way. 

We’re also experimenting with automating tasks through AI - it generates a lot of code, but we’re always involving some team members to correct the results. It’s important, because these AI systems will inevitably lack the internal context of your business, the water cooler conversations and coffee breaks where people might brainstorm and learn something new. These AI systems have no idea about these concepts. They can’t even do basic multiplication on their own. 

Have realistic expectations

Large AI language models are great at generalizable knowledge. When we build a system, we’re not building it for just one particular programming language, but across all programming languages, and they perform well in all of them.

Let’s say we took a model that was trained on some amount of Python, and another model that was trained on some C++. Imagine we took a third model that was trained on the same amount of Python and the same amount of C++. This third model will perform better at Python and C++ than the ones that were solely trained on one language at a time. This is because the third model was able to learn interesting facts, semantics and the logic work for programming systems in general, and then see how it applies to Python or C++ in particular. 

These AI models aren’t good at solving particular problems in just one language - that’s why they’re powerful. Of course, people need to have realistic expectations. These systems aren’t going to generate enough code to fix a major bug that requires tons of context and information about other systems as well. We have to be aware of what’s possible and what’s not. 

When I think about the last 40 years, there hasn’t been a sparsity of software in the world; there are just more and more software companies providing more and more value. Every ten years, we’re having larger IPOs, and more companies are building systems that can automate different parts of enterprises. There’s more software that can be consumed and generated across companies, but that doesn’t reduce the need for employees. It’ll just increase the value businesses can provide. 

Perhaps in the future, AI systems are going to understand more and more company-wide context over time, which is a big gap in their knowledge currently. When they’re able to piece together more information from your systems, your documents in the cloud and your other products, they’ll aid humans to be even more productive and perhaps even enable stronger cross-functional collaboration.

Bonus advice: Be patient

If you’re building an AI model, accept that it might not be able to perform its best right away. It’ll probably be a lot more valuable later on when it evolves. Build it for today, but prepare for the future as well.

About Varun

After graduating from MIT in 2017, Varun immediately joined Nuro, where he worked on early deep learning infrastructure software. During this time, he gained valuable experience on deep learning and AI technologies in general and witnessed what value they would bring to enterprises and consumers. He co-founded Exafunction in 2021. The company built Codeium, which is a generative AI code assistant tool. In his free time, Varun is an avid biker and cyclist who enjoys mountain biking a lot.

If you’re interested, check out Codeium and Varun’s Twitter for more details.