BY DAVID BLOCKOW
Architecture principles can provide a basis for decision making when developing solutions. They form a structured set of ideas that collectively define and guide development of a solution architecture, from values through to design and implementation, harmonising decision making across an organisation. This blog post describes Big Data-specific architecture principles used by D2D CRC.
Big Data Architecture Principles
D2D CRC has developed a set of Big Data-specific architecture principles, drawing on our experience developing solution architectures for Big Data problems across many domains. They reflect a level of consensus amongst experienced architects as to the most important qualities of an effective Big Data architecture.
These principles reflect the dynamic nature of the Big Data landscape and the mission critical nature of the work done by our Defence and National Security partners.
The principles are as follows:
Focus on User Needs: The focus of any Big Data architecture should be meeting user needs. Big Data technologies are enablers, allowing us to extract value from the data at scale to address user requirements.
Data is an Asset: Data is an asset that has value to the organisation. Access to relevant data is critical for users to perform their duties. Accurate and timely information is required to support good decision making.
People are an Asset: A broad range of technical skills are needed to build and maintain a Big Data capability, including architecture, software engineering, data science and operations. The need for skilled personnel is not unique to Big Data projects, however they typically require more specialised skillsets.
Keep it Simple: Solution architectures should be kept as simple as possible while addressing all use cases. Technical diversity should be controlled to minimise the cost of maintaining multiple technology platforms. Controlling technical diversity often involves incorporating legacy systems into solution architecture, trading capability for maintainability.
Flexibility & Interoperability: Software and hardware should conform to open standards to enable a flexible and extensible architecture. Modular components and services should be preferred to improve interoperability and reuse.
Platforms for Growth, not Isolated Systems: Solution architectures should support forward-looking use cases, allowing organisations to extend the architecture to support new features without significant rework.
Business Continuity: Systems must be highly available to support mission-critical operations. Redundancy and maintainability should be considered at design time and product selection should favour vendor supported and proven technologies.
Compliance with Law & Policy: Information management processes must comply with all relevant laws, policies and regulations. Open sharing of information must be balanced against the need to prevent unauthorised use and disclosure.
Responsive Change Management: The architecture should support rapid deployment of changes in response to changing user requirements, through application of techniques such as continuous integration and automated deployment. This includes components of the architecture itself, as well as analytics and applications running on the architecture.
Conclusion
These principles provide a starting point for Big Data solution design, and will typically be extended with organisation-specific architecture principles and requirements. As a set of general guidelines they allow D2D CRC to take a consistent approach to Big Data solution design, producing pragmatic solutions that meet our end users needs.
