Why encore?
    검색 팝업창

    Customer Stories

    J Research Institute

    HDFS Improvement & DLM for Data Lake

    Improved HDFS data schemas for efficient data lake utilization and management.

    • ClientJ Research Institute
    • ProjectSmart City Open Data Hub Architecture & Core Technology Development
    • IndustryServices
    • SolutionsDQ#
    • HighlightsData Hub ETL Manager & Data Lifecycle Management

    Data Hub ETL Manager & Data Lifecycle Manager
    for Efficient Smart City Data Lake

    Improved data lake schemas and shared ETL processes to reduce processing time and cost.

    Purpose & Background
    • Define preprocessing ETL processes to meet analysts’ requirements for data analysis, validate data, and establish a system to manage the lifecycle of generated data
      * ETL : Extract, Transform, Load
    Project Features
    • Expand the data collection scope to support analysis of complex and integrated data models
    • Improve data processing flexibility and analytical model performance
    • Minimize duplicate work by sharing ETL processes among multiple users
    Applied Technologies
    • Improve data schemas stored in HDFS (Hadoop Distributed File System), which forms the Data Lake
    • Develop an ETL module for Ingest data processing (Apache NiFi Template)
    • Improve data mart and analytics efficiency using Hive
    • Build a DLM (Data Lifecycle Manager) for data lifecycle management
    • Use DQ#, a data quality management solution, for data quality management

    Architecture

    Open-Source Big Data Platform-Based Data Lake Architecture
    for On-Premises or Cloud Environments

    Use platforms such as Horton Works and Cloudera to build data lakes.
    Leverage Apache Hadoop technologies for data collection, storage, processing, and analysis.
    We provide consulting for optimized data lake and analytics environments.

    Optimized Data Analytics with API and Modular Architecture for Smart Cities

    • 01

      Modular Architecture

    • 02

      Front & Back-end Decoupling

    • 03

      Standard APIs for Smart Cities

    • 04

      Urban Infrastructure Monitoring

    • 05

      Advanced Data Analytics

    • 06

      Data Marketplace

    • 07

      Data Model Standards

    Integrated Process for Data Lake Development on Big Data Platforms