eBooks
•
23 MIN READ
In this ebook, we’re going back to the basics-- the fundamentals of data integration. This complete guide to data integration covers many of the elements eHouse’s other content refers to, but does not explicitly define or explain each time we make reference to them because these concepts are so fundamental to the understanding of data integration. From how data is processed, to different approaches, to how to form a robust strategy, this guide covers it all.
As commerce both in the physical world and online continues to advance, data integration is the key to being able to rise to new levels.
The industry of data integration isn’t getting any simpler, nor is it going away. Now more than ever, your understanding of data integration informs your decision and your business’ ability to scale, grow, and build a more robust future for itself.
Let’s start with the absolute foundation:
Data integration connects disparate applications together allowing for consistent automated movement and delivery of data to create business processes across a wide range of applications.
More important to forming your understanding of data integration beyond this broad definition is understanding the variety of solutions to the problem of automating data. From tactical bandage-like plug-ins to fully automated and strategic approaches like VL OMNI by eHouse, understanding where your business strategy fits along the sliding scale of solutions that exist is perhaps the most valuable exercise your business can work through.
There are almost as many solutions to automating and integrating your data as there are businesses out there to use them, but all data integration approaches fit into five broad categories. Each has advantages and disadvantages based on the stage of growth and strategic needs of your business, which we’ve laid out in easily referable tables on the following pages.
Manually entering data from one or more system to another as a facsimile for truly automated data movement.
Pro: Can be a manageable and inexpensive approach for merchants with low sales volumes.
Con: A slow and error-prone approach that may incur penalty charges from trading partners.
A low-cost and low-barrier entry into true data integration, usually in the form of application add-ons or stand-alone apps.
Pro: Plug-and-Play integrations are an inexpensive way to automate data for low-volume, low-complexity merchants selling through one or two channels.
Con: Functionality for Plug-and-Play integrations are low, kept to the most basic data movements and often without the ability to configure the integration in any meaningful way. This integration approach is best utilized by truly small businesses, or else those looking to replace these basic integrations fulfilling pressing tactical needs with a more robust and strategic approach in the near future.
Manually coded and manually maintained, these integrations often require a team of integration professionals to maintain their functionality over time.
Pro: Ideal for a large or enterprise-level organization with the financial resources and/or internal resources to maintain either an internal or external data integration team, these integrations can be highly complex but can achieve a high degree of functionality.
Con: With the impetus of documenting and maintaining the integrations on the team that created them, if the integrations are not sufficiently maintained over time, they will become defunct. This integration approach often tends to be more costly than others, as the code is manually built or revised each time a change or new integration is required. If this code ‘lives’ with a single individual, there is real risk of the integration approach becoming ineffective over time.
This approach to data integration is a facsimile to truly automated data movement: while users do not have to manually transfer data details, they must instead manually download and upload batch files between applications as a way of transferring data.
Pro: A cost effective approach to moving data best suited for merchants with low volumes, or who can otherwise strategically justify moving data in batches with manual intervention.
Con: As with manual data entry, this approach has a threshold of usability for merchants on the growth arc. Manual file download/upload can be slow and may require field matching or other manual processes to fully move a file from one app to another.
iPaaS is the epitome of integration solutions for those businesses with sufficient complexity but without a dedicated internal IT team to create and maintain manually coded integrations. When hosted in the cloud, iPaaS solutions are generally more agile and scalable than other integration approaches, and can offer greater flexibility and features by lieu of their approach to the problem of automated data movement.
Pro: Certain iPaaS platforms hosted in the cloud and built on robust technology like VL OMNI by eHouse assume the responsibility of keeping the platform up-to-date, including adding new features. This keeps those on these platforms agile and flexible without incurring development or maintenance costs as with the other approaches. iPaaS platforms also have greater flexibility in how data is moved, including the ability to use webhooks (real-time data movement) and applying business-specific rules to data as it moves between systems. Some iPaaS platforms like VL OMNI by eHouse are web-based, allowing access to business intelligence from anywhere in the world. iPaaS platforms exist as stand-alone services that often specialize in extending the functionality of core platforms through strategic problem-solving around specific business issues or goals.
Con: Not all iPaaS platforms are truly ‘iPaaS’; potential customers must do their due diligence in researching potential integration partners, including their technology and approach. iPaaS is also generally a more costly initial investment with ROI to come after the solution has been implemented; without depth of understanding on the approach, this spend can sometimes be difficult to justify. Similarly, not all iPaaS solutions are secure; look for EU-GDPR and/or CCPA language in contracts, and for authorized certifications.
VL OMNI by eHouse is a true iPaaS platform.
VL OMNI by eHouse is fully EU-GRPR and CCPA compliant, and is certified as a Amazon Advanced Tier Partner which required security testing and certification.
Hosted in the cloud and powered by cutting-edge technology, VL OMNI by eHouse is a true iPaaS platform. With iPaaS, compared to the other approaches, the integrations are no longer static in time; rather, the platform, features, and expansion of services on the platform are the responsibility of the platform provider, empowering users without added cost.
Solutions like VL OMNI by eHouse, approach data integration... in such a way that new Connectors can be added, modified, or deleted from the system without having to pause or impact other processes — something that is only possible in iPaaS. iPaaS has additional flexibility and features in encompassing business rules in data transfers, being able to move data in real-time, and being able to quickly and cost effectively make changes to the integrations at any time. iPaaS solutions like VL OMNI by eHouse tend to resonate best with businesses who have complexity in their technology stack and/or supply chain.
These businesses often have specific goals and strategies, often centered around personalization and unified customer experiences, in mind they want to actualize in tandem with high-powered data integration. iPaaS solutions power some of the biggest names in online and multichannel commerce: iPaaS is the only solution able to deliver on these complex objectives as a third party partners a business’ success.
For many businesses, selecting an integration approach is just the first step to achieving their goals; how their data is moved is equally important. Actualizing on complex strategies around personalization, customer experience, and more are directly tied to how and when your data moves. Encompassing business-specific rules in your integrations is a unique offering that VL OMNI by eHouse, as an iPaaS solution, offers as standard; part of the broader configuration we also offer our customers includes strategically assessing when (and why) their data moves.
Two camps to the question of ‘when’ data moves are:
The decision between the two comes down to the strategic objectives your business wants to reach.
In data integration, processing your data in batches means:
When data is processed as a batch, data is collected over a per-determined amount of time and organized into a transaction file. This transaction file, the ‘source’ of the data, is then stored and held. On a set schedule, the transaction file is sent to the ‘target’ of the integration (either a centralized database or target application[s]), completing the integration. Sophisticated systems like VL OMNI by eHouse are able to send batch files to multiple targets in a single operation and can move data bi-directionally no matter the conditions under which the data is moved — batch or in real-time.
Your standard electric bill is a perfect example of a batched-based approach to data processing: Your electrical consumption data is collected during a set period of time before being processed together as a batch in the form of your bill.
Moving data through your chosen data integration solution in batches has certain strategic alignments and advantages compared to real-time data movement. Two ideal use-cases are large volumes of data or very low volumes of data.
For high volumes of data (e.g., exceeding hundreds or thousands of orders a day), it may make sense for a business to batch data to be moved at set intervals as opposed to sending data real-time as it enters the system, as to not overwhelm applications and potential limitations on how much data they can receive and how. VL OMNI by eHouse runs many of our customers’ batched data at 15 or 5-minute intervals, for example.
Conversely, for low-volume businesses, it may not make sense to transmit data in real-time, instead sending data at regular intervals hourly or daily. It may also be strategic for the business to consider a combination of batched and real-time data processing; e.g., orders could be batched while shipment data packages could be updated throughout the system in real-time.
In data integration, processing your data in real-time means:
Data moves immediately, often as soon as it enters the system or at the time of the event thatcreated the data, and Data is in it’s most up-to-date, current form.
With real-time processing, as soon as the transaction takes place, the target application(s) of the integration is updated at the same time. Real-time data movement in integration ensures a constantly updating cycle of information. The mechanism that allows for real-time data movement in an integration system or solution are webhooks. Not all applications have webhooks, and not every data integration solution is able to work with webhooks to move data in real-time.
When you’re booking your specific seat on a flight, the systems providing the data do so in real-time to ensure you’re not double-booking a spot.
Making the decision on how your data needs to move — including how and when, as with batch vs. real-time data movement — is an important strategic consideration. Top-tier services like VL OMNI by eHouse excel at discovering and incorporating key needs like how and when your data needs to move, and will design the system and service around your business to fit exactly. There’s no universal answer to this question: each business is unique, and so is their treatment of their data and its automated flow throughout a system. Partnering with an expert you trust — and can trust over the long-term — for your data integration needs is key to deciding between these two general ways to move your data, as well as many other nuances that can configure the system to your needs exactly.
Moreover, these experts can also create the technical configuration like deciding between batch and real-time by understanding your goals and strategies. For example, a high volume ecommerce site might assume that batched data movements for their integration would be the best approach based on volumes alone. But if that same company prioritizes customer experience above all else, delaying data entering and moving around the system may also delay customer notification communication.
Data validation refers to the process of data cleansing to ensure data quality and correctness.
Very few merchants have perfect data: data that never includes errors or typos and never needs adjusting. Although data integration takes care of human error between systems, mistakes can still be made in other areas of the business that can introduce errors into the integration system. Understanding where errors exist and why in your business is a key step towards operational excellence and efficiency. Just remember: Garbage In, Garbage Out!
It’s not uncommon for the average person to conflate data integration with data validation and other data-related services. Here we address three myths that eHouse sees regularly in working with merchants around what data integration actually does in light of data validation services to help bolster your understanding.
Myth 1: Data Integration Cleanses Data.
False!
Assuming that data integration services or platforms also cleanses data is a common misconception: these are actually two very distinct service sets usually fulfilled by two (or more) companies specializing in one or the other.
Data cleansing is an operation or function that’s performed on data as its being entered into the system. A common point of data error is where the customer inputs their information at a point of purchase; for example, the checkout on an ecommerce site. Unvalidated data is simply entered right into the system without any opportunity to amend, correct, or eliminate ‘bad’ or erroneous data. It’s then the job of diligent personnel or work-arounds to correct the data. Usually bad data is caught at the point where an error occurs or worse — when an customer in turmoil reaches out to you.
Conversely, systems that include data validation services or applications catch bad data before it’s entered into the system. A common data validation service is an add-on provided by many postal services: a simple app that checks data information against official records, and asks users to select the proper data/address if they’ve entered it incorrectly or not in the format required by the broader system.
Data validation is incredibly important to the longevity of your business and your ability to efficiently complete on the promises you make to your customers, but without a validation system, your data integration solution will process exactly what it’s been told to do... even if it is potentially ‘garbage’ data!
Myth 2: Data Cleansing and Data Validation are the Same
False!
It’s easy to confuse data validation and data cleansing as their outputs can be virtually identical. But the two processes are very distinct and exist in your broader technology stack in different ways. Regardless, both data cleansing and data validation are not part of core data integration platforms or services. Some integration platforms may offer data cleansing and/or validation services, but the argument for partnering with specialist companies for each sector your business needs to fill is always strong and one to fully consider.
The differences between data cleansing and data validation are pretty distinct. Data cleansing is the process of correcting data after the data has already entered the system or technology stack. Data cleansing is the retrospective correction or erroneous data.
In contrast, data validation is the process of correcting data at the point of data entry, i.e., before the data enters the system or technology stack. This gives users and/or the validation service the opportunity to amend incorrect data before it goes anywhere else; it is more of a preventative measure rather than a corrective action, as with cleansing.
Many top businesses utilize both data cleansing and data validation services to ensure the best quality of their data both at the point of entry and throughout all systems. With great data automation like VL OMNI by eHouse married with stellar data, a business can move closer to operational excellence and flawless data flow.
Myth 3: Data Integration Fixes Data
False!
Although this myth may seem like a rephrasing of the previous two myths (it is), we’ve included this myth to really drive home where many who are looking at data integration, validation, and cleansing partners get confused on in the first place: that data integration does in fact translate and transmit your data between application(s), but does not ‘edit’ or ‘fix’ the actual data.
Data integration can be thought of as language translation between two applications that ‘speak’ or ‘understand’ two different languages: we’ll use English and French as an example. VL OMNI by eHouse, as a data integration platform and service, processes and moves data between applications (one-to-one, or one-to-many) by translating the data from the ‘English’ of the source application to the ‘French’ of the target application.
The information contained by the data, however, does not change. To complete our metaphorical example, if your data in ‘English’ says “I am happy”, VL OMNI by eHouse would translate this data to “Je suis content”. The information remains the same; if there was a grammatical or spelling error in the data, this would be translated between applications as-is — in its erroneous state.
Fixing bad data exists outside of your data integration platform or service, and usually requires diving into operational and other applications to diagnose and correct the issue. Top-tier data integration platforms like VL OMNI by eHouse come with an analytics-loaded dashboard interface that businesses can use to diagnose and amend their bad data. Lesser data integration solutions, however, leave this task completely up to you with little information on how to source, troubleshoot, and ultimately amend issues outside of the integration itself.
The premise of data integration is freely flowing data, unifying previously disparate systems into a holistic technology stack. Data is perhaps one of the most valuable things a business owns today, but if it’s locked into a siloed application or isn’t taking into account your business’ strategy and unique needs, that data quickly loses its present and future value.
First understanding the importance and value data integration offers to your business is one of the most important steps you can take. Having a clear understanding of data integration and applying it in the right form and structure for your business’ needs can allow you to drive incredible growth. After all, as your business grows the amount of data your business has and can potentially leverage to your advantage increases exponentially.
For scaling businesses, with some complexity in their applications, goals, or strategies (or all), the emphasis should be on finding a long-term, agile and scalable data integration partner like eHouse’s iPaaS solution.
Ultimately, the goal of strategic data integration is the ability to improve and iterate, and to grow to your business’ maximum capabilities without technology or data getting in the way.
Data can and does inform everything when it flows freely in the right form and with the right integration solution powering it — from a unified and personalized customer experience, to operational efficiency, and much more. Improving customer experience, operations, productivity—everything!
Revisiting earlier chapters in this ebook, the data integration approach you choose to implement has vast and potentially long-lasting implications for your business — both potentially positive, and negative. Ultimately, if any integration approach requires you to significantly change your business, it’s processes, and/or operations to fit the data integration solution, you’re likely on the wrong track. The ideal data integration solution will fit your business as-is, and can meet your strategic objectives by configuring their platform to your needs rather than the other way around, like the VL OMNI by eHouse platform does.
Your data integration solution should also strategically match your business’ objectives for both the short and the long term. Ask yourself: what’s my purpose or intent for integrating my data and systems? This will quickly expose your needs, especially those around specific pain points, but it will also expose where you’d like to go and what you’d like to achieve in your business, AKA your goals.
With high-powered, scalable and agile iPaaS platforms like VL OMNI by eHouse, many businesses look to achieve the three I’s: improving customer experience, increasing productivity, and improving business operations and intelligence. Approaching data integration with your goals in mind is the first step to ensuring you select the right platform or partner for the long-term, who will grow with you in the ways you need.
At this point we’ve fully established what data integration does — and does not — do in general terms, setting VL OMNI by eHouse as the contrast for comparison and illustration value.
Beyond the automated movement of data, different approaches to the problem of data integration have very different results (revisit Chapter 1 for a refresh on the five basic approaches to data integration). It’s incredibly important that your data integration provider/solution matches your goals and strategies (as we’ve stressed throughout this ebook); a mismatch can be costly at best, and detrimental to your business’ ability to function at worst.
The newest cloud-based form of integration, iPaaS, is becoming the go-to solution for many mid-sized to enterprise . But even in iPaaS, not all solutions are created equal — each provider has a unique approach, and platforms vary greatly on their features and you ability to discern valuable business intelligence to help continue to build your business. The best data integration solutions are those invested in fostering your growth actively, as opposed to a passive portal with little information.
The VL OMNI by eHouse Platform and Dashboard, our user portal, not only unify your data, but provide deep business intelligence through our enhanced channel-specific analytics —something that is only possible through our approach. Business rules are included directly in our configuration for whole-system integration. This means your data is perfectly integrated in automated every time according to your business’ needs.
At the end of the day, all the features, integrations, and applications that come together to form your business’ technology stack do not matter at all to your end customer, regardless if it’s in a B2C, B2B or a combination of sales strategies.
Ultimately, all your customer truly cares about is the service they receive. And that’s the power of a great data integration partner like eHouse.
With a robust and feature-rich data integration platform like eHouse’s, there are many benefits that your business reaps that can be passed directly on to the customer experience. Our cloud- based iPaaS model keeps you agile and scalable while keeping costs down.
Two big impacts businesses using VL OMNI by eHouse see that directly benefit their customers are (but not limited to):
Digital transformation —the use of new, fast, and frequently changing digital technology to solve problems — is directly tied to your data integration approach. If everything is hard-coded, digital transformation with any velocity is near impossible. Certain iPaaS platforms like VL OMNI by eHouse put digital transformation at the center of our approach, keeping our customers agile and compliant at every turn. With our API-centric approach, VL OMNI by eHouse is already ready for the next rising trend: headless commerce.
At a point, data integration was seen as a means to an end — a tactic to fix an issue, like slapping a BandAid on a cut.
Today, data integration is very much a strategic investment.
Top international brands seek out integrators like eHouse with high functionality, deep analytical insights, and white-gloved set up and configuration. Data integration partners like eHouse power some of the most sophisticated customer experiences online, offline, and everywhere in between. If your data doesn’t flow when, where, and how you need it to, competing with those who see the strategic value in integration is going to be a steep climb. Partnering with a company like eHouse, where we take the time to understand your business and goals, means better matching of strategy to technology and faster time to market.
Data integration isn’t a cut-and-dry process. But with the right partner who’s invested in seeing your business succeed over the long term by taking the time to build the exact solution you need, data integration does not have to be your job.
eHouse prides ourselves in the quality of our integration work and the successes we’ve been able to help our customers achieve. With 26+ years of data movement experience, our customers are those who are striving to accomplish strategic objectives today, but who are already thinking of that next step to come. We’re experts in guiding and education, but nothing is equal to being able to sit at the table and ask the right questions for your business so you’re ensured you partner with the right data integration solution for you.
Whichever route you choose, just remember that we live in an ever-changing world — and that needs to be accounted for in your technology choices as well.
Have an idea for a project? Fill out this form and we’ll get back to you shortly.
