It’s easy to adopt new business technology. Scaling it without breaking everything that already works is the hard part. Almost every company will run a pilot, have it perform in isolation, and then realize that their existing infrastructure fails to scale when they attempt to push it into production with real workloads.
Start with the architecture, not the application
Before you even look at a new tool, you need to think about what it’s sitting on top of. The real question is whether you’re sticking with your own servers or moving to the cloud where you only pay for what you use. One way locks you in. The other lets you adjust as you go.
When you build for the cloud and use containers, everything becomes modular. You can swap out one piece without touching the rest. Need to update a model in your machine learning pipeline? Update that part. Everything else keeps running.
That’s the idea behind breaking things into separate services. When they’re not too dependent on each other, one can fail or get upgraded without taking down the whole system. The rest keeps going while you fix what needs attention.
The data layer is where most implementations fail
No business technology can function properly if the underlying data is of poor quality. Unfortunately, this is often overlooked in the eagerness to implement new solutions.
Taking a data-first approach implies that you first need to clean up and consolidate your data before considering any generative AI implementation on a significant level. For many organizations, data lakehouses, which combine the best aspects of data lakes and data warehouses, are the optimal kind of architecture to ensure they are ready for AI, all while maintaining control and governance.
The urgency in all of this lies here: as Gartner predicts, more than 80% of enterprises will have deployed AI applications in production in the next 5 years. That’s a very steep growth curve. If your data pipes are not in order, you’ll be pushing a lot of garbage into the brain of your AI. And becoming disappointed in the results.
So, fix the data first. It’s the foundation of everything else.
Bridge legacy systems before you replace them
Removing the current ERP and CRM systems to install new ones is rarely a good option. The costs related to this are too high and the risks to everyday operations are substantial.
The right approach is to implement a middleware layer or an API architecture that allows new tools to connect to old systems. An architecture with an API-first approach allows each new solution to “talk” to the existing technology environment through standard connections, with no additional effort.
This approach also helps in controlling your technical debt. In any organization, with every new process, more tech debt accumulates. Workarounds might appear to be a cost-effective solution in the short term, but in the long term, all these add-ons will slow down and raise the cost of a new process. An API layer might not seem that exciting, but it is an investment in standardization of all future work.
Building for inference, not just storage
Most infrastructure conversations focus on storing and moving data around—but the real challenge now is handling inference in real time, at scale.
A generative AI system that actually works doesn’t just need somewhere to put data. It needs infrastructure built for high-token throughput—the ability to handle huge volumes of AI requests without everything slowing to a crawl. That usually means GPU clusters set up specifically for inference, not just the one-time training runs. The trick is finding the sweet spot: you want to pack as many requests as possible onto a GPU that’s the right size, without paying for power you’re not using.
This is where edge computing starts to matter. When latency really counts—like in customer-facing apps or anything making decisions in real time—you want the compute happening close to where the data’s coming from. That cuts down the round-trip time in a way even the fastest, most optimized cloud setup can’t completely solve.
Security doesn’t scale the same way it used to
Distributed and cloud-native technologies demand a different approach to security than what traditional perimeter-based models offer. A so-called cybersecurity mesh model recognizes that security controls will be needed at many different locations, rather than concentrated in one defense perimeter. The mesh puts those controls as close as possible to the asset they are meant to protect. Access to the asset is governed by however many of the available controls it makes most sense to look for.
With a cybersecurity mesh, it’s also understood that this changes on a dynamic basis. Applications are on the move, after all: reassembling last year’s perimeter may not be an option. This mesh approach scales better than the old boundary model but there’s nothing fundamentally new about it; we’ve always put locks on doors rather than packed the neighborhood into a gated compound.
