5 posts tagged with "modern data stack"

We are today announcing Xceed Smart AI Search capability in Xceed Analytics, which brings the next generation AI search capability to Xceed Analytics - A Comprehensive Unified Data and AI Platform for the enterprise. By Launching this capability, we are delivering one more milestone on our promise of bringing Xceed Analytics as a unique Intelligent end-to-end Data and AI Platform.

Amidst all the hype around LLMs and its applications in enterprise, At Cynepia we are bringing the power AI to Xceed Analytics in many ways. A couple weeks back we announced Xceed AI Assistant, A comprehensive AI assistant across your data use cases. Today I am pleased to announce advance AI search, which combines vector search (using powerful opensource embedding model) with full-text search to significantly improve search relavance. This coupled with unified control plane ensures that you can access all your data and model assets instantly. Xceed AI Search allows users to instantly discover and search across all your data estate including data connectors, datasets in data catalog, model registry, transformation workflows, SQL models and dashboards.

Data users can now discover all your data assets from an enhanced application search interface instantly and save time finding things. For example. A Data Scientist/ML Engineer trying to build a new model and wants to find out if there are existing tables or feature tables that may be relavant for his/her requirement, Xceed AI Search is the place to start. He/She can quickly get to the tables/datasets in catalog or existing feature tables in catalog. Likewise a Business User who is searching for relavant dashboards linked to a given data assets can again hope to Xceed Search and find the relevant tables/dashboards using a full text semantic search capability.

About Vector Search with Full Text Search​

Vector search technology uses Large Language Models to perform semantic retriveal of knowledge thereby significantly improving the relevance of search results for an end user. This feature is useful when you are interested in results based on the meaning and context of the search text. It leverages natural language processing and artificial intelligence to interpret the nuances of language and retrieve results that match the user's intent. This capability goes far beyond traditional keyword-based searches, enabling users to discover relevant information even when they don't have precise search terms in mind. This allows users to describe what they are looking for in natural language with a near approximate meaning and yet find the relavant data. Often results from vector search are not optimal, because of various terms which are specific to a domain. Xceed AI Search capability brings in the benefit of both vector search and full text search together, ensuring most relavant results are sent back to the user.

Boost analyst/data engineers/data scientist productivity, with Xceed AI Search and Discovery​

Xceed Analytics is uniquely positioned to improve experience with AI capabilities using our new AI Search and Discovery capability, given our unified approach to enterprise data and AI platform. It helps democratize access to enterprise data while ensuring role based governance/access.

Availability​

The Xceed AI Search and Discovery feature is currently available in public preview.

About Xceed Analytics​

Xceed Analytics is an AI powered comprehensive enterprise data platform unifies all your data, analytics and AI use cases and products under a single unified platform. A comprehensive data and analytics Platform is therefore vital to success of business transformation journey as we ride the new wave of Artificial Intelligence and take advantages of this new promising technology in the transformation journey.

Problem that a Comphrehensive DAta & AI Platform addresses​

Emergence of Machine Learning and AI, along side fast pace of digital and explosive growth of data that enterprises are experiencing has made them realize that an effective approach to managing and harnessing the power of data and AI can create significant competitive advantage. Landscape for data and AI has been constantly evolving over the past decade to address the challenge and oppurtunity of managing and harnessing this data that enterprises are inundiated with. Modernizing the data and AI platform has been a constant through out the past decade.

The fragmented toolchain and siloed data within enterprises are formidable barriers that hinder the full harnessing of their data assets. When various departments and teams rely on disparate tools and systems that don't communicate effectively, it leads to inefficiencies, duplicated efforts, and a lack of a unified view of the data. Siloed data exacerbates this problem by isolating valuable information within these disparate systems, preventing cross-functional collaboration and inhibiting data-driven decision-making. The result is a missed opportunity for enterprises to extract valuable insights, achieve operational excellence, and remain agile in an increasingly data-driven world.

A Comprehensive Data & AI Platform helps breaking down these silos and streamlining the toolchain is essential for organizations to unlock the true potential of their data assets and drive innovation.

Benefits of a Comprehensive Data & AI Platform​

There are enumerous benefits of a comprehensive end-to-end Data and AI Platform

1. Central repository for all the data, workflows and models.

2. Seamlessly Discover, Manage Data Quality and Govern all your data products/artifacts through a single pane.

3. Remove data silos, keep every stackholder engaged and notified.

4. Accelerate deriving value from their most valuable asset which is data.

5. Enables enterprises to cut/optimize costs via No Integration stack. You no longer need to stitch individual services from multiple vendors.

6. Simplicity of overall architecture helps in streamlining of the overall data and analytics process.

Technical Capabilities​

Some of the key data tools included in Xceed Data and Analytics Platform include:

1. Versioned, Governed and Fully Integrated Data Lake based on open standards such as Apache Parquet.

2. Unified abstraction for all data producers. Supports multiple OLAP and compute engines

   • Duckdb, Apache Spark, Pandas, Ray

3. All common access methods supported. Access/Configure and Monitor with your prefered access method

   • SQL or Dataframe or CLI or Python SDK

4. No-code Data Integration. Supports most common databases, cloud storages and SAAS applications.

5. Integrated Data Catalog with Extensive Data Discovery, Governance and Data Quality Test Features.

6. Xceed SQL Workbench Enables analyst to carry out exploratory analysis via a visual interface. Supported Engines include duckdb, Apache drill, Apache Spark

7. Xceed Workflows for No/Low Code Interface data transformation pipelines. Supported Engines include Apache Spark, Duckdb, Apache Drill for SQL, Pandas, Pyspark for dataframes.

8. Xceed AutoML - Enable onboarding every day ML use-cases across Classification, Regression and Forecasting.

9. Xceed Business Intelligence & Reporting Provides all common dashboarding features to build beautiful datastories/dashboards.

10. Xceed Notifications Ensure all stackholders are notified

11. Xceed Model Registry home to all ML Models.

12. Xceed Python SDK/CLI Data users can now work via Xceed APIs and Command Line Interface besides the user interface as an alternate choice for interacting with Xceed Analytics.

13. Microservices architecture enables scalability while providing seamless integration.

For More details on Xceed Analytics Architecture, refer to Our Architecture Page

About Cynepia Technologies​

Cynepia Technologies provides comprehensive end to end data stack to help enterprises organize, connect, make sense of their data, stay connected with their insights, make faster, real-time decisions and ultimately grow your business.

To learn more about Cynepia and Xceed Analytics, visit our website

For demo or product inquiry, write to us at Product Marketing

---

Background​

In the era of Language Models and Advanced Artificial Intelligence Applications, need for reliable and accurate data has never been more important than now. Having a Comprehensive data and analytics platform has become non-negotiable need for a company of a certain size to acheive goals and benefits of these formidable new capabilities. Inability to access Data and Metadata seamlessly in a single pane is a major source of frustration in carrying out data driven digital transformation. Cobbled up point solutions often sold as best of breed have only added to challenges with integrating these solutions within one's data platform architecture. A Comprehensive data and analytics platform is therefore one of the key elements to success with data driven digital transformation.

Problem​

In additon to platform challenges, Data teams face a variety of challenges in ensuring quality of the data products built by them and made available to the downstream users through the life cycle of the individual data assets/products. These data assets are often accumulated using 100s of upstream sources via source databases, SaaS systems via AP, Cloud Storages and more. The dynamic nature of the data itself along with movement from variety of systems have made troubleshooting data issues almost impossible, leading to longer down-times, frustrated data teams and loss of trust on data products.

One of the key challenges in trouble shooting such issues is lack of visibility of data changes often caused by upstream changes at source systems or somewhere during the journey of transformation. Effectively visibility can help ensure a better baseline reference profile for every data asset and mechanism to test for specific data tests (both syntax and semantics) of the new incoming data can help data teams react faster to the impending issue.

Solution​

We are today introducing Xceed Dataset Monitors right within Xceed Data Catalog to help data teams get back in control over their data challenges. Data Engineers can now set data quality monitors for every incoming data and ensure that the necessary checks/tests are carried out every time new data arrives. Data Teams can create monitors using an easy to use GUI right from within the dataset details page. Data Teams can create multiple suites for individual downstream data product impact (for example dashboards created by downstream analyst or the data being used by a downstream data science team for an ML model).

Real-time monitoring and keeping all the stack holders informed ensures reduced downtime in event of upstream changes and ensures trust on end data products is never broken

Data Quality Monitoring Dashboard enables data teams track trends over time both at dataset as well as individual test levels. This further helps spot repetitive non-reliable tables/columns over time, helping stackholder teams to prioritize and take effective actions to improve the overall quality.

In Summary, Some of the key benefits of our approach to data observability/monitoring are as below:

1. Inline with the data arrival critical to reduce actual downtime.

2. Support for No code interface drop in right within the data catalog, lowers the bar to add/modify data quality tests/monitoring rules.

3. Integrated approach ensures, you don't need another out-of-band data observability or monitoring tool.

4. Single interface to bring all data users together. Keep every one informed in real time as data is refreshed.

5. 360 view of all data artifacts and operations right from within the single application interface. Data teams now have ability to monitor datasets/columns with consistent issues

Key Features​

1. Cynepia Data Quality Monitors are Engine Independent, it works with all the supported engines including Spark, Pandas.

2. Leverages Existing Data Profile for the dataset thereby optimizing compute usage.

3. Support for exhautive list of monitor rules both at dataset level and column level.

4. Support for multiple notification channels including In-App Notification, Slack and Emails.

5. Run History with Data Quality Metrics Trends to monitoring trends at an overall suite level and individual monitor/test level.

How It Works​

To Create a Data Quality Monitoring Suite, You first need to first define a Monitoring Suite from the dataset details page in your Data Catalog. Defining a Monitoring Suite for a dataset is a three step process as shown below:

1. Create a New Monitoring Suite

2. Add individual monitors/tests to the suite

3. Add a list of slack channels/users to notify on every run

4. Click Finish to create a new monitoring suite. You have successfully created a new data quality monitoring suite. You can click run manually to trigger a fresh run from Existing tab.

Once the run is completed, results are now available via the Run History tab as seen below:

About Xceed Analytics​

Xceed Analytics is an AI powered comprehensive enterprise data platform unifies all your data, analytics and AI use cases and products under a single unified platform. A comprehensive data and analytics Platform is therefore vital to success of business transformation journey as we ride the new wave of Artificial Intelligence and take advantages of this new promising technology in the transformation journey.

Benefits of a Comprehensive Data & AI Platform​

There are enumerous benefits of a comprehensive end-to-end Data and AI Platform

1. Central repository for all the data, workflows and models.

2. Seamlessly Discover, Manage Data Quality and Govern all your data products/artifacts through a single pane.

3. Remove data silos, keep every stackholder engaged and notified.

4. Accelerate deriving value from their most valuable asset which is data.

5. Enables enterprises to cut/optimize costs via No Integration stack. You no longer need to stitch individual services from multiple vendors.

6. Simplicity of overall architecture helps in streamlining of the overall data and analytics process.

Technical Capabilities​

Some of the key data tools included in Xceed Data and Analytics Platform include:

1. Versioned, Governed and Fully Integrated Data Lake based on open standards such as Apache Parquet.

2. Unified abstraction for all data producers. Supports multiple OLAP and compute engines

   • Duckdb, Apache Spark, Pandas, Ray

3. All common access methods supported. Access/Configure and Monitor with your prefered access method

   • SQL or Dataframe or CLI or Python SDK

4. No-code Data Integration. Supports most common databases, cloud storages and SAAS applications.

5. Integrated Data Catalog with Extensive Data Discovery, Governance and Data Quality Test Features.

6. Xceed SQL Workbench Enables analyst to carry out exploratory analysis via a visual interface. Supported Engines include duckdb, Apache drill, Apache Spark

7. Xceed Workflows for No/Low Code Interface data transformation pipelines. Supported Engines include Apache Spark, Duckdb, Apache Drill for SQL, Pandas, Pyspark for dataframes.

8. Xceed AutoML - Enable onboarding every day ML use-cases across Classification, Regression and Forecasting.

9. Xceed Business Intelligence & Reporting Provides all common dashboarding features to build beautiful datastories/dashboards.

10. Xceed Notifications Ensure all stackholders are notified

11. Xceed Model Registry home to all ML Models.

12. Xceed Python SDK/CLI Data users can now work via Xceed APIs and Command Line Interface besides the user interface as an alternate choice for interacting with Xceed Analytics.

13. Microservices architecture enables scalability while providing seamless integration.

For More details on Xceed Analytics Architecture, refer to Our Architecture Page

About Cynepia Technologies​

Cynepia Technologies provides comprehensive end to end data stack to help enterprises organize, connect, make sense of their data, stay connected with their insights, make faster, real-time decisions and ultimately grow your business.

To learn more about Cynepia and Xceed Analytics, visit our website

For demo or product inquiry, write to us at Product Marketing

---

In the era of Language Models and Generative AI applications, Xceed AI Assistant aims to offer a comprehensive AI Assistant across all the data tasks and functionalities within Xceed Analytics.

Xceed AI Assistant cut across all the roles and tasks, be it Business Analyst exploring datasets using SQL or creating a report, or a Data Engineer updating/exploring catalog for a given dataset,

A Data Scientist/ML Engineer trying to build a new model or the Business User who has a business question. Some of the common tasks supported with this preview and upcoming releases of Xceed AI Assitant shall include the following:

• Auto-Generate SQL from a given business analyst english prompt.

• Semantic Search enabling superior natural language search to discover the most relevant, reliable data assets

• Asking data questions in Natural language to get answers to one's business quesion.

• Create Natural Language Summary for a given insight

Boost analyst/data engineers productivity, with Xceed AI Assistant​

Xceed Analytics is uniquely positioned to improve experience with AI capabilities provided by Language Models, given our unified approach to enterprise data and AI platform. It helps democratize access to enterprise data while ensuring role based governance/access.

Availability​

The Xceed AI Assistant is currently available in private preview.

About Xceed Analytics​

Xceed Analytics is an AI powered comprehensive enterprise data platform unifies all your data, analytics and AI use cases and products under a single unified platform. A comprehensive data and analytics Platform is therefore vital to success of business transformation journey as we ride the new wave of Artificial Intelligence and take advantages of this new promising technology in the transformation journey.

Benefits of a Comprehensive Data & AI Platform​

There are enumerous benefits of a comprehensive end-to-end Data and AI Platform

1. Central repository for all the data, workflows and models.

2. Seamlessly Discover, Manage Data Quality and Govern all your data products/artifacts through a single pane.

3. Remove data silos, keep every stackholder engaged and notified.

4. Accelerate deriving value from their most valuable asset which is data.

5. Enables enterprises to cut/optimize costs via No Integration stack. You no longer need to stitch individual services from multiple vendors.

6. Simplicity of overall architecture helps in streamlining of the overall data and analytics process.

Technical Capabilities​

Some of the key data tools included in Xceed Data and Analytics Platform include:

1. Versioned, Governed and Fully Integrated Data Lake based on open standards such as Apache Parquet.

2. Unified abstraction for all data producers. Supports multiple OLAP and compute engines

   • Duckdb, Apache Spark, Pandas, Ray

3. All common access methods supported. Access/Configure and Monitor with your prefered access method

   • SQL or Dataframe or CLI or Python SDK

4. No-code Data Integration. Supports most common databases, cloud storages and SAAS applications.

5. Integrated Data Catalog with Extensive Data Discovery, Governance and Data Quality Test Features.

6. Xceed SQL Workbench Enables analyst to carry out exploratory analysis via a visual interface. Supported Engines include duckdb, Apache drill, Apache Spark

7. Xceed Workflows for No/Low Code Interface data transformation pipelines. Supported Engines include Apache Spark, Duckdb, Apache Drill for SQL, Pandas, Pyspark for dataframes.

8. Xceed AutoML - Enable onboarding every day ML use-cases across Classification, Regression and Forecasting.

9. Xceed Business Intelligence & Reporting Provides all common dashboarding features to build beautiful datastories/dashboards.

10. Xceed Notifications Ensure all stackholders are notified

11. Xceed Model Registry home to all ML Models.

12. Xceed Python SDK/CLI Data users can now work via Xceed APIs and Command Line Interface besides the user interface as an alternate choice for interacting with Xceed Analytics.

13. Microservices architecture enables scalability while providing seamless integration.

For More details on Xceed Analytics Architecture, refer to Our Architecture Page

About Cynepia Technologies​

Cynepia Technologies provides comprehensive end to end data stack to help enterprises organize, connect, make sense of their data, stay connected with their insights, make faster, real-time decisions and ultimately grow your business.

To learn more about Cynepia and Xceed Analytics, visit our website

For demo or product inquiry, write to us at Product Marketing

---

Background​

More and more enterprises are embracing data science as a function and a capability. But the reality is many of them have not been able to consistently derive business value from their investments in big data, artificial intelligence, and machine learning. However, A surprising percentage of businesses fail to obtain meaningful ROI from their data science projects. Enumerous articles have been written on failure rate, root causes and how do we improve the success of such projects.

Background​

A modern enterprise data and AI platform has become a non-negotiable need for a company of a certain size.

The central idea behind such a platform is that it serves as a central repository where all the data can then be converted into knowledge that can then be used by business users to deliver value across various functions.

Therefore, it is imperative that we state clearly and unambiguously that data is a first class citizen in any enterprise and therefore needs a place to thrive and grow. An enterprise data and AI platform is therefore vital to the enterprise transformation journey.

Various data processes such as data engineering, business intelligence, analytics, machine learning further help enhance the value of this data by performing various operations variety of opertions right from cleaning and preparing data, unifying data from different data sources and creating new metrics/measures that help organisations who believe in power of measurement to streamline their operations, perform root cause analysis and understand drivers for a certain entity, create a holistic understanding of customers, employees, processes and measure and improve each and every aspect of enterprise process.

As one finds out the canvas of generating potential business value from data is huge. But enterprises have to traverse through the key decision of what kind of modern data enterprise platform suits their needs. There are various ways enterprises can build a modern data and analytics platform for today.

Build your own​

** Should you build your data and AI from the ground up? **

Thats potentially a multi-million dollar question often running into 10s of million depending on the scope of such a data project endeveour.

A decision to build on your own should have a much more stronger business value case, since it is coupled with long term maintainence and support costs, keeping up with the technological improvements over time to upgrade and modernize a home grown platform. In addition, newer technical requirements emanating from new forms of use cases and plathora of design choices have significantly increased the complexity of building such data architectures, often making a project execution of a build approach itself risky.

Hence our view is “Build your own” data and AI platform needs to emanate from a clean longer term business strategy and goal that is articulated well and differentiation spelt out. This needs to be further supplemented with a well defined execution scope, time, human capital and cost allocated to such initiative.

At Cynepia, We beleive the answer to above question is almost an absolute ‘NO’ for 99.9% of the enterprises, If the larger objective is to use the data to derive operational analytics and decision analytics value by using the data and analytics infrastructure.

** Buy various point SAAS solutions and integrate. **

Here again there are potentially 2 main approaches:

• ** Would purchasing various Commercial SaaS solutions and integrating make a far more apt sense ? **

This has really been an approach on tear for past several years. Clubbed under a loosely defined term “Modern Data Stack”, A Lot has been written and discussed on this topic over last few years. A quick and apt summary of what is clubbed under such a terminology is discussed/reviewed by Approva Padhi from Foundation Capital hereModern Data Stack: Looking into the Crystal Ball.

The article also summarizes issues/areas of improvement for such a stack. At Cynepia, we beleive that there are many issues with the so called “modern data stack” approach. A few among them are as follows:

1. Lack of finished and consistent user experience and multiple end application interfaces leading to sub-optimal user centric design and productivity gaps.

2. Lack of thought to devolution of data and analytics skills leading to lessor scope for democratization.

3. Higher human cost to keep the modern data stack up and running over time handing integration issues and dealing with sub-optimal architectural choices made by the different SAAS tool vendors.

Often this would need adding further layers of software applications.

4. Cost conundrum due to piece meal approach and dependece on multiple SAAS vendors to deliver the unique SLA need of your organisation.

Leveraging Data APIs and apps from cloud vendors such as AWS, Azure or Google Cloud a better bet?

• Another way to build the data and analytics platform is to using many of the tools, point services and programmable interfaces provided by players such as aws, azure and google cloud platform. While the usual benefits of cloud such as flexibility and scalability are available here too. In addition to the above mentioned disadavantage, there are a few additional disadvantages:

1. Price variation: Often this services are priced by usage and cloud services bills tend to grow exponentially as the data grows. This is seen as a big irrant in services adoption on cloud beyond basic infrastructure services

2. Governance: Data Security & Governance of many of these cloud platforms, tools and applications was designed to be generic and therefore is quite combursome, but often may need specialist to be hired to keep your data secure flawlessly.

** Buy a modern unified data and AI development platform **

There is yet another class of vendors who provide a bundled data and AI Development Platform. These platforms are usally a subset of data and AI applications built on a common architecture and user experience theme unlike unbundled SAAS Applications.

However, compared to the hype, many of these unified platforms/vendors have often lost the ability to shape the market because of variety of reasons:

• Actual promise vs reality

Often incomplete or combined by acquiring different SAAS companies such as above and bundling the solution, completely ignoring the user centric design approach.

Often Costly and priced as per the needs of fortune 500/1000 companies making them prohibitive for the larger enterprise market.

• Missing Pieces of the puzzle further leaves adding further layers of software applications.

Is Modern No Code/Low Code UI Unified Platform the future?​

** Unified Low Code/No Code Enterprise Data & AI Platform **

At Cynepia, we see another category which is not just modern and futuristic and ensures data and AI are democratized and has the potential to reach far more users and use-cases and larger enterprise market. For name sake, we call this category ”Low Code/No Code Enterprise Data and AI Platform”

So what are the advantages of a “Low Code/No Code Enterprise Data and AI Platform”

First and foremost to set our orientation right, we strongly believe in a unified data and AI development platform, because thats the best way to make analytics affordable and expand foot print beyond fortune 500/1000 companies. Ofcourse the baseline is actual promise should match reality.

It transfers technical debt of architecture, integration/support of multitude of SAAS applications to the vendor instead of you as the customer.

We are believers in No Code/Low Code Platforms potential to deliver better value. Why not for code? It is not to suggest that for code platforms is a bad idea. Every no-code platform rides on code, so that would be an oxymoron in a way, if one is proposing no-code.

Here’s are top reasons to go low code/no code:

It’s hard to hire best of bread/smartest of the Data Architects, Software Engineers, Frontend Engineers, Data Scientists and Project Managers at the price one is willing to pay. If one can, one could potentially generate a similar or potentially better outcome as a No Code/Low Code Platform. So that leaves you with next grade of skilled developers, who then need to be trained, upskilled and hopefully you have an code output that fits your aspiration. However, most likely case is you are left with more data debt than code that you can call asset.

Most Enterprises are not software companies by business. We have seen often keeping focus on building code over extremely longer term is hard, since quick business outcomes often supercede long term value of such code assets. Historical data evidences suggest, for code data pipeline shelf life is much lower than one beleives it is.

While multitude of best of breed SAAS applications still remains an alternate choice, but if an enterprise is focused on business value and ROI. There is no reason for it to technical debt of integration, support and stitching architecture is transferred to the customer than the vendor. There is also an overall ROI question on the whole data and analytics investment as discussed earlier.

How do you strategize?​

Data technologies are evolving quickly, making traditional “build your own” may be risky, time consuming and inefficient. If you haven’t already made investment in any architecture, we would strongly recommend to invest in a user centric Low Code/No Code Enterprise Data and AI Platform.

If you have already invested in one of the above given architecture and accruing technical debt which feels unmanageable, you may want to restrategize and rethink of investing in a data and analytics architecture that suits not just your today’s need but also future.