The 7 Axes of Data Platform Selection in the Era of AI: A Primer
In the realm of data platform decision-making, organizations typically consider several dimensions when making their choices. These encompass aspects including functionality, performance, scalability, cost, flexibility, and alignment with specific use cases.
The following are some of the key criteria of data platform decision-making. It’s worth noting that one of the most-hyped databases right now, in support of AI, is vector databases. We’ll explain why.
Data Model and Schema Flexibility
Organizations assess whether the database supports their data model requirements. Some may need the flexibility of schema-less or schema-on-read models. Others may require the rigidity of a relational model. The choice depends on factors like the structure of the data — is it simply rows and columns of numbers? is it a mix of images, videos, and documents? — and the need for agility to adapt to evolving schemas.
With the rise of Hadoop, many organizations began to store more of their data for analysis, now or in the future. Open source Hadoop offered data storage on commodity hardware, while more traditional proprietary data warehouses were almost certainly far more expensive. The trouble is that Hadoop lacked a schema — a structure for the data warehouse — making it harder to extract the data when you need it (though workarounds are now available).
As mentioned above, vector databases are garnering a lot of attention because of the rise of AI. Reasons for this include:
- Efficient Similarity Search or Nearest Neighbor Search: Vector databases are optimized for nearest neighbor search, a fundamental operation in many AI applications such as recommendation systems, image retrieval, and natural language processing.
- High-Dimensional Data Handling: AI models, especially in NLP and computer vision, generate high-dimensional embedding vectors. Vector databases can store and index these vectors efficiently, allowing for rapid querying and analysis.
- Semantic Search: By leveraging embedding vectors that capture semantic information, vector databases enable more intuitive and relevant search results compared to traditional keyword-based searches.
- Multimodal Search: Vector databases support the integration of various data types (text, images, audio) by converting them into a common vector space, allowing for unified search and analysis across different modalities.
- Clustering and Classification: Vector databases support operations like clustering and classification directly on the stored vectors, aiding in tasks such as customer segmentation, anomaly detection, and pattern recognition.
Vector databases are pivotal for AI because they provide the necessary infrastructure to store, manage, and query high-dimensional vectors efficiently. This capability is foundational for enabling fast, scalable, and intelligent AI systems across various applications and industries.
Performance and Scalability
Performance considerations include factors like query speed, throughput, latency, and concurrency. Scalability relates to the ability of the database to handle growing volumes of data and increasing user loads without sacrificing performance. Organizations evaluate whether the database can scale horizontally (adding more servers) or vertically (upgrading existing servers).
Consistency and Durability
Consistency refers to the degree to which data remains in a consistent state across distributed systems, especially in the event of failures or concurrent transactions. Durability relates to the ability of the database to ensure that committed transactions persist even in the face of system failures. Organizations weigh the trade-offs between consistency, availability, and partition tolerance based on their application requirements.ACID is key to relational, transactional databases. ACID compliance refers to a set of properties that ensure the reliability and integrity of transactions in a database system. The acronym ACID stands for Atomicity, Consistency, Isolation, and Durability, each representing a fundamental aspect of transaction management.
ACID is spoken of in somewhat hushed tones by NoSQL vendors. When pushed, some will say they offer “ACID-like” compliance. For many modern use cases, ACID-like is good enough. But speak to a database developer dealing with transactional systems — like core banking systems — and they will tell you their regulators and other stakeholders require “pure” ACID compliance. Compliance with ACID standards can help organizations meet regulatory requirements and maintain data governance.
Data Integrity and Security
Organizations prioritize databases that provide robust mechanisms for maintaining data integrity (e.g., through constraints, transactions, and validations) and enforcing security (e.g., encryption, authentication, authorization, and auditing). Compliance with regulatory requirements such as GDPR, HIPAA, or PCI-DSS may also influence database selection.
Ease of Development and Maintenance
This encompasses factors like developer productivity, ease of learning, availability of tools and libraries, and support for programming languages and frameworks. Organizations seek databases that streamline the development process, facilitate debugging and monitoring, and minimize operational overhead.
Total Cost of Ownership (TCO)
TCO considerations include both up-front costs (e.g., licensing fees, hardware costs) and ongoing expenses (e.g., maintenance, support, scaling). Organizations evaluate databases based on their ability to deliver value relative to their costs over the entire life cycle of the system.
Ecosystem and Integration
Organizations assess the database’s ecosystem, including its compatibility with existing infrastructure, integration with other systems (e.g., data warehouses, analytics platforms, the cloud), availability of third-party tools and services, and community support. Integration capabilities influence factors such as data migration, interoperability, and extensibility. There is also the issue here of deployment venue: on premises, in the cloud, hybrid, or multicloud.
By evaluating data platforms along these lines, organizations can make informed decisions that align with their business objectives, technical requirements, and constraints. Vector databases are certainly one of the hottest tickets in town in support of AI — but different use cases have different priorities.
The Critical Role of Governments in EU Cyber-Resilience
Cybersecurity threats continue to increase. According to ENISA’s 2023 Threat Landscape report, there were around 2,580 observed incidents in the EU between July 2022 and June 2023. In the previous reporting period, there were less than 800. ENISA reported that 19% of events targeted public administrations, by far the largest industry.
Government chief information security officers (CISOs) are not resting on their laurels. They understand that cybersecurity is important.
IDC’s EMEA Cross-Industry Acceleration Survey, conducted in December 2023, found that 91% of government executives were planning to maintain or increase their level of spending in cybersecurity. In the many one-on-one conversations that IDC Government Insights analysts have had with government CIOs, CISOs, and other IT executives, cybersecurity always stands out as a “ring-fenced” item protected from budget cuts.
But given the rising volume, variety and velocity of threats, ring-fencing budgets is not enough — a change of paradigm is in order.
Paradigm Shift: From Protecting the Perimeter to National Survivability
The most forward-looking governments understand that the old mindset focused on protecting the individual government agency, or even the individual system or digital citizen service app, is insufficient. They understand that governments play a key role in national digital and physical services and infrastructure resilience and survivability.
To deliver on this higher purpose, government CISOs need to look beyond their organizational boundaries, take a whole life-cycle approach to cybersecurity, and provide knowledge to non-security experts to enable them to act responsibly.
Looking beyond organizational boundaries means collaborating across public administrations and the private sector to create a resilient ecosystem. Beyond the NIS2 Directive’s mandatory obligations — such as establishing at least one computer security incident response team (CSIRT) in each EU member country — resilience comes through the collective effort of cybersecurity specialists within ministries of defense, police forces, intelligence agencies, and all other public administrations.
In a recent IDC conversation with a regional government institution’s senior IT leader, it was emphasized how important it is for all levels of government to collaborate in the ecosystem to protect against the spike of attacks that usually occurs in the months prior to and during major events like the Olympics, the World Expo, the FIFA World Cup, or a G7 meeting. Participants should include transportation operators, payment and banking service providers, telcos, utilities, and travel and hospitality companies.
Taking a life-cycle approach to cybersecurity means caring for the hygiene of systems, protecting them and responding to events from development through termination. Hygiene starts with security by design, DevSecOps and application security best practices, and vetting hardware and software supply chain bills of material for security and compliance requirements.
Data hygiene is paramount — not only to comply with regulations that protect sensitive data, but also to increase the resilience and visibility of all data sets critical to government operations. Holistic protection requires enhancing the observability of the broader landscape.
CISOs should demand that their cybersecurity solution providers make available AI/ML solutions and AIOps practices that can increase the productivity of observability and detection.
Incident response must be grounded on governance processes and structures that enable timeliness and coordination. Throughout the life cycle, government CISOs should regularly upgrade their team’s skills, not only traditional cybersecurity skills but also legal skills, AI ethics, bias testing, and prompt engineering.
Ensuring that security and non-security experts have the right knowledge to act responsibly is a major people and organizational transformation effort. Investing in programs to raise the cybersecurity awareness of civil servants, other industries, and the general public is critical.
It is also important for CISOs to articulate the value of cybersecurity to the elected and appointed officials who make budget decisions. CISOs and their teams that are able to articulate the value of cybersecurity in terms of business risks will raise their profile internally and be recognized as strategic decision makers.
Technology developments will help CISOs accelerate the paradigm shift. In particular, the automation and orchestration of processes related to security and privacy will help address the skills gap and accelerate the detection of malicious behaviors, threat response, and remediation actions.
European government CIOs and CISOs that combine tool investments with a holistic approach to cybersecurity will boost the resilience of their organizations, of the communities they serve, and increase citizens’ trust in government. Those that focus on siloed system protection and legacy operating models and competencies will not be able to respond to threats and will be relegated to the role of gatekeepers who eventually lose influence and budget.
To learn more, explore the latest IDC research
The Future of Technology: Unlocking Potential in Central and Eastern Europe
IDC’s annual CEE Summit was held in Vienna on June 9-11, with the theme of “Unlocking Performance Potential.” It highlighted the opportunities and challenges for organizations currently embracing AI.
Digital transformation continues to reshape industries and economies worldwide, and Central and Eastern Europe (CEE) is no exception. This vibrant region, with its technology hubs and innovative spirit, has the potential to become a key player in the global tech landscape.
In this blog post, we offer insights from industry experts on the pivotal factors driving this transformational change. From building IT spending transparency and making a compelling case for artificial intelligence to the critical roles of culture, change management, and sustainability, we explore how CEE can harness its full transformation potential. Moreover, we emphasize why cybersecurity remains a top priority in this rapidly evolving digital era.
Key insights include:
Making the Case for AI and Building IT Spend Transparency
AI use cases are widespread and growing, in industries including financial services, pharma, manufacturing, and more. However, IDC research shows that GenAI investments are putting increasing pressure on IT budgets.
For organizations with tight IT budgets, our advice is to build a transparent IT spending management capability that provides a financially sound platform on which to base investment decisions. Avoid reliance on high-level benchmarks that provide no visibility on the organization’s future state. Mapping spending to clear cost pools and defined services provides great insights into spending and helps identify opportunities to cut costs and improve efficiency.
Culture and Change Management Are Key for Any Kind of Transformation
No matter the company size or sector, winning over hearts and minds is crucial for success. This is true whether the organization is consolidating its regional IT services delivery, looking to fully leverage the benefits of GenAI, or adopting SAP HANA across the business.
Key elements in building a positive culture include solid backing from senior management, empowerment of teams to make their own decisions and fail fast, finding the right ambassadors of change, and rooting out toxicity.
Given all the different elements involved in change, organizations need to consider a change management function focused on the successful adoption of new tools, processes, and behaviors. This helps with communication, building transparency, and creating buy-in during periods that are often riddled with uncertainty.
Sustainability and Tech for Good
Sustainability and IT optimization are becoming critical in Europe and hot topics of discussion among IT users. Driven by the increasing number of European regulations and directives, sustainability is seen as a viable approach to stay competitive, enhance brand image, and boost customer trust. However, like IT optimization (or FinOps), sustainability can also help to streamline operations, reduce costs, and increase business value.
Cybersecurity Remains High on the Agenda
Cybersecurity remains a high priority for European executives, in particular with activities such as building resilience and business continuity, regulatory compliance and governance, and positioning security as a enabler.
Focus is increasing on cloud-native security capabilities, such as endpoint protection, threat intelligence, and application security. However, there are many industries where legacy infrastructure is widespread, resulting in security capabilities that must be adapted for cloud .
Recommendations
Many organizations are at the AI exploration stage, testing out use cases that often focus on productivity, such as chatbots and digital assistants with GenAI. For many organizations, the worry centers around the ROI gap and the real impact of AI on business.
However, real commercial differentiation with the use of AI can only be driven with function-specific (CFO, CMO, CHRO) use cases or industry-specific use , but those are more complex and investment-. Effectively, organizations are looking at the trade-off between ease of implementation vs. cost with less/more differentiation. Impacts from this investment can be achieved at all levels in the organization, but the level of impact will depending on the chosen trade-off.
We advise more holistic thinking: Organizations need a responsible AI strategy and prioritized use case roadmap with C-suite buy-in. Challenges such as AI pricing, governance, and organizational change must be considered. The broader ecosystem — including supply chain, tech vendors and other strategic partners — must also be considered. in mind, however, that GenAI or broader AI doesn’t offer a short cut to the holistic thinking noted above: you need to start with a data strategy.
IDC’s 2024 CEE Summit
Over three days, participants gathered to discuss technology opportunities and challenges from a distinctly CEE perspective. The Summit had wide representation from across the region, with participants from Czechia, Slovakia, Poland, Croatia, Serbia, Slovenia, Greece, Austria, Hungary, Romania, and Bosnia Hercegovina.
The location — in the historic town center of Vienna — evoked the Vienna Circle of the early 20th Century, where discussions among mathematicians, scientists, and philosophers laid many of the logical foundations for the Information Age of the second half of the century.
One highlight of the event was an innovation challenge, where teams had to swiftly create and propose a GenAI-enabled initiative. This generated collaboration and many creative investment pitches, with the winner focusing on AI to improve predictive maintenance for electric vehicles.
As attendees departed Vienna, they did so with more than a fridge magnet and the strains of Mozart in their ears — they took with them new knowledge, new contacts, and the inspiration that comes from setting aside the daily routine long enough to exchange ideas with industry peers.
If you want to find out more about this and other events we host, please visit our website here.
Mastering Hybrid Cloud in Europe: Selecting the Right Managed Service Partner
The Hybrid Cloud Imperative
In today’s rapidly evolving digital landscape, the hybrid cloud has become an indispensable strategy for organizations aiming to maximize their IT investments. Our insights reveal that hybrid cloud is not just a trend but a critical component of modern IT infrastructure, blending the strengths of private and public clouds to foster efficiency, performance, and agility.
The complexity of implementing and operating such environments makes the role of managed service providers (MSPs) critical. These partners not only help smooth out the technical and operational challenges, but they also play a pivotal role in unlocking the true value of hybrid cloud investments.
As European organizations increasingly lean towards hybrid cloud solutions, understanding the importance of selecting the right MSP partner becomes paramount for achieving the desired business outcomes.
Overcoming Hybrid Cloud Hurdles with MSPs
Europe’s hybrid cloud landscape is experiencing a significant transformation, marked by rising investments and a clear growth trajectory. IDC’s 2023 EMEA Managed Cloud View Survey illustrates that, while hybrid cloud accounted for 31% of total IT budget in 2023, IT leaders expect a significant increase (up to 48%) of their total IT spending in Europe over the next two years.
The appeal of hybrid cloud in Europe is underscored by its compelling benefits. Organizations are drawn to the promise of enhanced cloud performance, streamlined backup and recovery processes, and more efficient cost management. These advantages position hybrid cloud as a cornerstone for building resilient, cost-effective, and optimized IT infrastructures.
However, the journey to harnessing these benefits is fraught with challenges. Skill shortages, particularly in cloud security and public cloud specific competencies, emerge as significant hurdles. The complexity of managing across diverse cloud environments adds another layer of difficulty, with operational visibility and governance among ongoing concerns.
As organizations progress in their cloud journey, they start to encounter more complex technical challenges, spanning security and data interoperability, as well as difficulties in meeting compliance and regulatory requirements.
To smooth out the edges of hybrid cloud adoption, many organizations are leaning towards managed service providers (MSPs). MSPs are pivotal for guiding customers to the right cloud types and orchestrating the ecosystem components to streamline operations, drive IT efficiency, improve agility, and increase business value.
Choosing the Ideal MSP for Hybrid Cloud Success
Understanding Your Unique Needs
When embarking on the hybrid cloud journey, understanding your organizational needs is paramount. Identify your objectives in the cloud landscape and the core IT challenges to these objectives. This clarity will guide you in selecting an MSP with the requisite cloud expertise, one that is capable of aligning the right cloud solutions with your specific workloads’ requirements.
The Crucial Role of Expertise
Expertise in various cloud environments is non-negotiable. An MSP must offer comprehensive services, from consulting to management, ensuring a seamless transition to the cloud. Their knowledge should span different cloud types, facilitating a match between cloud platforms and workloads that optimizes performance and cost.
Unified Management: A Must-Have
A unified management approach simplifies the oversight of diverse cloud environments, enhancing visibility, security, and compliance. Opt for MSPs that propose a platform-centric model for managing hybrid and multicloud scenarios. This model should streamline operations across your IT estate, fostering standardization and operational efficiency.
Empowering Your Hybrid Cloud Journey
In conclusion, the journey to mastering hybrid cloud in Europe hinges on the strategic selection of an MSP that aligns with your organization’s unique needs and goals. This journey, while complex, is made significantly smoother and more rewarding with the right partner by your side. As we’ve explored, the right MSP partner is not just a vendor but a pivotal collaborator in unlocking the full potential of hybrid cloud-enhancing efficiency, agility, and security across your IT operations.
If you want to learn more, please refer to this IDC document: Managing Hybrid Cloud: Considerations for European IT Buyers when Selecting Managed Service Providers