Implementing a Data Warehouse in a Kar Dealer Network

Data warehouse architecture Goldfarelli & Rizzi (2009) describes two categories of data warehouse architecture. The first classification describes the physical separation of the architecture layer of the data warehouse. The second category is based on the method designed by Inmon and Kimball. Inmon recommends the use of a hub and spoke architecture, but Kimball prefers an architecture based on data mart bus (Ariyachandra & Watson, 2006). The advantage of the focused radiation method is that the cost of data integration is low, development and coupling method is easy (Hwang & Cappel, 2002).

Depending on the implementation of the data warehouse, the data mart may be a small data warehouse or just a part of the data warehouse. Data marts are often used to provide information to functional departments of an organization. Typical examples are data marts for sales department, inventory department and transport sector, finance department, and excellent administrative department. Data marts can also be used to segment the data of the data warehouse and reflect the relatively autonomous geographic segmentation of each regional service. For example, in large-scale service organizations, regional operations centers can be regarded as independent business units with their own data marts.

Among the many valuable things a data engineer makes, one of their most popular skills is the ability to design, build and maintain data warehouses. Just as a retail warehouse packs and sells consumer goods, the data warehouse has raw data converted and stored in a queryable format. I use ML to predict the value of housing at Airbnb: I wrote it myself, why I need many prior data engineering work to build batch training, offline scoring machine learning model I explained. It is worth noting that many of the tasks related to feature engineering, training data building and backfilling are similar to data engineering efforts.

We have constructed a data warehouse necessary for DWH software (SONM instance) to read and retrieve data from block chains. For example, it displays a list of orders on the market. The inefficiency of the internal data structure implemented in the search smart contract requires additional effort to make the smart contract more complicated and it is almost impossible to perform effective searches against smart contracts . Therefore, I decided to implement a data warehouse. Workers, nodes, and wallets use it. So far, we just save the current market operations (order, transaction etc.), the history is not stored there. In the future we will add support for historical data so that users can run DWH and improve their performance. DWH is implemented in the same way as the geth node - anyone can own it. This is like a personal cache layer between client and block chain. Read more articles