Introduction
Enterprises in the midst of digital transformation are being held back by monolithic, on-prem legacy applications that don’t have the speed, agility and responsiveness required. Migrating to the cloud addresses a lot of these concerns.
1 Data makes this transition to the cloud easier.
Developed by PeerIslands in partnership with MongoDB, 1Data is a reference architecture and solution accelerator that helps users with their cloud modernization.
This post details the challenges enterprises face with legacy systems, and I also hope to walk you through how 1Data helps organizations expedite cloud adoption.
Modernization Trends
As legacy systems become unwieldy, enterprises are breaking them down into microservices, and adopting cloud native application development. Monolith-to-microservices migration is complex, but provides value across multiple dimensions. These include:
- Development velocity
- Scalability
- Cost-of-change reduction
- Ability to build multiple microservice databases concurrently
One common approach for teams adopting and building out microservices, is to use domain-driven design to break down the overall business domain into bounded contexts first. They also often use the Strangler Fig pattern to reduce the overall risk, migrate incrementally, and then decommission the monolith once all required functionality is migrated. While most teams find this approach works well for the application code, it is particularly challenging to break down monolithic databases into databases that meet the specific needs of each microservice. There are several factors to consider during transition:
- Duration. How long will the transition to microservices take?
- Data synchronization. How much, and what types of data need to be synchronized between monolith and microservice databases?
- Data translation in a heterogeneous schema environment. How are the same data elements processed and stored differently?
- Synchronization cadence. How much data needs syncing, and how often (real-time, nightly, etc.)?
- Data anti-corruption layer. How do you ensure new data doesn’t corrupt the old data and maintain transaction data integrity?
Simplifying Migration to the Cloud
1Data helps enterprises address the challenges detailed above.
Migrate and synchronize your data with confidence with 1Data!
- Schema migration tool. Automatically convert legacy DB schema and related components to your target MongoDB instance. Use the GUI-based data mapper to track errors.
- Near real-time data sync pipeline. Sync data between monolith and microservice databases in near real-time with enterprise grade components.
- Conditional data sync. Define how to slice the data you’re planning to sync.
- Data cleansing. Automatically translate data as it moves.
- DSLs for data transformation. Apply domain-specific business rules for the MongoDB documents you want to create.This also acts as an anti-corruption layer.
- Data auditing. Independently verify data sync between your source and target systems.
- Go beyond the database. Synchronize data from APIs, Webhooks & Events.
- Bidirectional data sync. Replicate key microservice database updates back to the monolithic database as needed.
Get Started with Real time Data Synchronization
With the initial version of 1Data, PeerIslands addresses the core functionality of real time data sync between source and target systems.
Here’s a view of the logical architecture:
Source System. The source system can be a relational database like Oracle, where we’ll rely on CDC, or other sources such as Events, API, or Webhooks.
Data Capture & Streaming. This stage captures the required data from the source system, and converts them into data streams using either off-the-shelf DB connectors or custom connectors, depending on the source type. 1Data implements data sharding and throttling, which enable data synchronization at scale, in this phase.
Data Transformation. This is the core of the accelerator, when we convert the source data streams into target MongoDB document schemas. We use LISP-based Domain Specific Language to enable simple, rule-based data transformation, including user-defined rules.
Data Sink & Streaming. Captures the data streams that need to be updated into the MongoDB database through stream consumers. The actual update into the target DB is done through sink connectors.
Target system. The target system is the MDB database used by the microservices.
Auditing. Most data that gets migrated is enterprise-critical; 1Data audits the entire data synchronization process for missed data and incorrect updates.
Two way sync. The logical architecture can be utilized to enable data synchronization from the MongoDB database back to the source database.
We used MongoDB, Confluent Kafka and Debezium to implement this initial version of 1Data:
The technical architecture is cloud agnostic, and can be deployed on-prem as well. We’ll be customizing it for key cloud platforms as well as fleshing out specific architectures to adopt for common data sync scenarios.
Conclusion
The 1Data solution accelerator lends itself to multiple use cases, from single view to legacy modernization.
Please contact us for technical details and implementation assistance. Be also on the lookout for more updates on 1Data!
