Deriving value from analytics investments depends on having an agile and trusted data fabric. Gartner says, a data fabric is generally a custom-made design that provides reusable data services, pipelines, semantic tiers or APIs via combination of data integration approaches in an orchestrated fashion. Data fabrics can be improved by adding dynamic schema recognition, or even cost-based optimization approaches. As a data fabric becomes increasingly dynamic, or even introduces ML capabilities, it evolves from a data fabric into a data mesh network.
Data fabric is more of a designed approach, mostly tending toward use cases and locations on either "end" of a thread. The threads may cross and do handoffs in the middle, or even reuse their parts, but they are not built up dynamically. They are merely highly reusable, normalized services.
Data mesh is a fully metadata-driven approach. Statistics in the form of metadata accumulation are kept relating to the rate of data access; platform, user and use case access; the physical capacity of the system; and the utilization of the infrastructure components. Other data points include the reliability of the infrastructure, the trending of data usage by domain and use case, and the qualification, enrichment and integrity (both declared and implied) of the data.
Data engineer: The human capabilities of data engineers will be augmented by AI/ML processes that identify nearly all of the initial pain points for data refactoring, modeling, schema production and data quality recognition.
Data scientist: This role will benefit from data fusion outputs that create alerts about expanding data assets. These alerts will be specifically tuned for the current project — but, can also begin to recognize data that a given scientist utilizes in terms of data patterns.
Data modeler: Data modelers, data integration developers and database administrators responsible for data modeling will model less, and verify more.
Information architect: Information architects working with data fabric will need to focus on identifying the required functionality of a data asset and imputing it as metadata.
As noted by Dataversity, in a growing world of connected devices, a data fabric can play a large role in facilitating AI-drive technologies and make a big impact on businesses. Interconnected data needs to be reliable, and that reliability can be promised by the layered architecture of data fabric. The power of data fabric is shown in Anzo — a "virtual intelligence" network comprising humans, ML algorithms, and data.
In the self-driving vehicle industry, researchers and autonomous vehicle manufacturers are still grappling with the challenges of untested or partially tested road conditions, which leads safety advocacy groups to raise concerns. The biggest challenge currently facing the industry is the generation of huge volumes of training data during object-detection model testing.
The Cutting Edge: Data Fabric, Driverless, and Data Gravity observes that though self-driving vehicle research teams have put their focus on computing power, advancements in sensors, and object-detection algorithms, not much has been written about managing the petabytes of data created during the object-detection model testing phases. The article explains how data fabric can help to manage this massive amount of training data.
