Security Architecture Consideration for Hadoop Implementation.

One of the biggest concerns in our present age revolves around the security and protection of sensitive information. In our current era of Big Data, our organizations are collecting, analyzing, and making decisions based on analysis of massive amounts of data sets from various sources, and security in this process is becoming increasingly more important. The more data you have, the more important it is that you protect it. It means that not only must we provide effective security controls on data leaving our networks, but we also must control access to data within our networks

Nowadays every organization is facing big data challenges and most organizations turn to Hadoop for the big data solution, so is our organization. Recently we are developing a Hadoop architecture strategy and roadmap, one of the architectures we need to develop is the security architecture for Hadoop implementation. Based on different business requirements and organization’s enterprise architecture principles Hadoop implementation can be on-premises or in the cloud. There will be different security concerns for different implementation.

Within our organization enterprise architecture group works closely with security architecture to first identify and understand business use cases and based on each use cases requirement to create a security requirement catalog. We will categorize the requirements into different categories and identify the existing security architecture to analysis the gaps. Considering the massive amount of data that nodes hold, there is an increasing need to focus on security architecture for the Hadoop cluster. We realize that if we are going to implement Hadoop cloud solution, business critical and sensitive data will leave the premises so adequate security controls is necessary. We prefer to adopt Security as a Service provider and the architecture should consider to integrate the Security as a Service into our organization security ecosystem for consistent operations and auditing. Some of the security consideration will be

1. How to enforce authentication for users and applications?
2. How to integrate internal data sources to the Hadoop cloud?
3. How to enforce data access control based on existing access control policies?
4. How can Hadoop integrate with existing enterprise security services?  

In our fast-paced and connected world it is critical to understand the importance of security as we process and analyze massive amounts of data. This starts with understanding our data and associated security policies, and it also revolves around understanding the security policies in our organizations and how they need to be enforced. 

Security architecture development approach

Based on the TOGAF, security concerns are pervasive throughout the architecture domains and in all phases of the architecture development. Security is called out separately because it is infrastructure that is rarely visible to the business function. Its fundamental purpose is to protect the value of the systems and information assets of the enterprise. Often the nature of security in the enterprise is that it is deemed successful if either nothing happens that is visible to the user or other observer, and/or no damage or losses occur to the enterprise.

The generally accepted areas of concern for the security architect are:

• Authentication
• Authorization
• Audit
• Assurance
• Availability
• Asset Protection
• Administration
• Risk Management

When we develop enterprise architecture, security architecture will be all around each phase of the development, security requirements need to be taken into the consideration during the each phase of the development. Here we are talking about creating security architecture not security policies for a special projects. I have experienced a situation during the development of architecture of cloud solution for the organization. When I was working with security specialists on the subject usually I will get a set of policies or even a specific product to use. To me it is different. Some of the general security policies developed based on the past and existing information and technology system may not fit for this architecture. The right approach should be as described by TOGAF, gathering current and emerging security requirements from business for each phase of the architecture development, create a security requirement catalog, perform a baseline analysis to determine the “current state” of  the security effort, identify gaps in the current state, articulate an architecture in functional terms to address the gaps and incorporate emerging business requirements, identify and communicate s “desire state” environment and develop a blueprint for the future architecture. It is important to develop and select standards and policies to implement the security program within the context of the chose architecture. Business requirements should be updated and reassessed during the iteration of the architecture development process.

Reference

TOGAF

https://www.giac.org/paper/gsec/610/building-enterprise-security-architecture/101447

Integrating Security Architecture into Enterprise Architecture

One of the key findings of Gartner research paper “Aligning Security Architecture and Enterprise Architecture: Beast practice” is “The more-closely aligned the security architecture function is to the enterprise architecture (EA), the more effective it is. Complete integration of security into the EA must be the goal.”

In this research paper, Gartner indicates that “The goal is for the security architecture to be completely integrated into an organization's EA.” In reality, as stated in this paper, “many realities mitigate against achieving this in most organizations, including historical organizational and internal political realities. Security architecture has traditionally been practiced separately from the EA. Thus, security architects are often not conversant with EA terminology, principles and practices. Furthermore, the EA tools used by most organizations do not allow for security artifacts to be fully integrated with the EA, simultaneously being able to provide a separate, security perspective where security-only artifacts can be modeled.”

This is what I see in our organization today, in certain degree we don’t even have security architecture in place in general.

EA is difficult to interact with security architecture group, there is no formal communication mechanism in place. While EA creates solution space for the business, security architecture was not in the consideration of the design. Most of the security specialists (I will not call them security architects as most of the them are only concentrating on designing a detailed security solution for the projects rather than creating an architecture) don’t quite understand what enterprise architecture is. Often time this creates conflicts and misunderstanding between EA and security. The security solution, policies for a specific project is isolated from other security policies and solutions even for a similar project.

Gartner provides strategies to improve the level of alignment which I fully agree and I think our organization should adopt.

·        Sending security architects to attend a training course on the EA methodology used in the organization

·        Aligning the structure and methodology of the enterprise information security architecture (EISA) framework with the structure and methodology of the organization's EA approach

·        Adopting EA terminology in the EISA practice

·        Leveraging any focus on IT governance in the organization to support the effective integration of security into the IT services and application life cycles, and thus into the EA process

·        Conducting joint workshops between the EISA and EA teams to develop common processes, process interfaces and terminology

·        Combining EISA and EA in major new projects

·        Placing security architects in the EA team — that is, starting to work toward integrating the EISA team into the EA team

Reference

https://www.gartner.com/doc/790521/aligning-security-architecture-enterprise-architecture

Virtualization, Do Not Turn Cost Saving Into Money Wasting

Virtualization has become the cornerstone of every enterprise's favorite money-saving initiative. By reducing the numbers and types of servers that support their business applications, companies are looking at significant cost savings. Less power consumption and a smaller server footprint is simpler to manage.

Beyond the potentially dramatic cost savings, virtualization can greatly enhance an organization's business agility. Also, this technology offers the potential for a fundamental change in the way IT managers think about computing resources.

Server virtualization brings a lot of benefits to the organization. However there are some challenges presented to the IT operations as well.

Cost Justification

When implement virtual solution for data center, we need to clearly understand the business requirements and carefully plan for the capacity and infrastructure growth. If not it will actually posts a very big challenge for IT operations. Very often we see resources being wasted due to many different reasons such as whether resources can be shared among different applications due to security consideration, ongoing project postponed due to the budget constraints or resource conflicts. Hardware for the virtual infrastructure has its life cycle, when resources cannot be fully utilized within its life cycle, resources will be wasted. A typical example will be virtual infrastructure implemented for the project, the idea is to consider easy future expansion so resources were planned for the next six months or a year, then often the case the project delayed and resources sits there. After three to five years hardware resources need to be refreshed and new investment has to put in place.

Resources Calculation and Planning

Even though organization’s data center usually has a resources and capacity planning team, the reality is if you browse most of the organization’s virtual infrastructure you will find out that overall around 40 to 50 percent of resources are always idle. While a lot resources sitting idle new resources continue to be implemented as some of the resources are not suitable for some applications. While virtualization provides business with agility, it is with the extra costs. We often see that even within one data center, due to the fact that data center needs to provide infrastructure for different business units and due to some regulation and policy, applications cannot run on the same hosts thus more hardware and licenses will needed.

Virtual infrastructure management

For a large organization it is very complex to manage such virtual environment, user access rights and admin access rights to different farm, folder and resources are all need to be managed. Multiple virtual center servers, different version control and so on.

People are always talking about the benefits of virtualization but if we do not plan or architect it well, resources waste can happen easily. Instead of cost savings, it actually turns into money wasting. 

Using EA Principles in Enterprise Technical Architecture

Reference

Gartner: Toolkit Best Practice: Using EA Principles in Enterprise Technical Architecture

Enterprise technical architects — that is, technical architects, infrastructure planners, project architects and solution designers — often want to get straight to product comparison and selection. However, better enterprise technology viewpoints, ETA modeling and design work should reflect careful integration with overall business goals, as well as adherence to particular IT drivers and specific technology guidelines and best practices. EA principles are part of the guidelines to leverage — and leverage explicitly. Principles present expected behaviors that if adhered to will enable strategies to become realities.

We Enterprise Architects specify enterprise architecture principles based on organizational vision and strategies. Principles and Standards provide a firm foundation for making architecture and planning decisions, framing architecture and supporting resolution of contradictory situations. Architecture principles guide our architecture work, our decision making and choice of selection. As Gartner indicates “Choosing which principles are appropriate for a given organization at a lower level — for example, for standard ETA models such as technical patterns and technical services, or for particular individual infrastructure designs and implementations — should reflect the general goals of the organization at a high level.”

One of the big challenges for success architecture work is to incorporate a principled approach to designing technology and infrastructure. Each enterprise architecture team has a set of enterprise architecture principles but they are not always applied or used by architects during their architecture definition process. For principles to work a governance framework must be defined: the definition of a principle should specify the decision that has to be made, the choices (eventually), the body (role) that makes the decision and the typical use case (when to use it).

Finally, principles enable continuous improvement of the processes and artifacts. The more organizations know about the principles that guide their decision making — and the more specifically they document these principles in general (and in standard models), and then in per project designs document specifically how the principles are followed or broken — the better their models and designs will be.

Using EA principles in enterprise technology architecture reflects the maturity level of organization’s enterprise architecture. Less mature enterprise architecture often conducts architecture work without fully incorporated a principled approach, ended up with more project centric rather than architecture centric work. Understanding the importance of using EA principles in defining architecture is a critical factor for successful EA.

IPv6 Readiness and Architecture Approach

The most fundamental IT component of an enterprise is the network. The network must evolve to meet changing business requirements. Communication is required for organizations to function, yet costs need to be appropriately managed, and the portfolio of network services delivered by IT organizations should be formalized. The network architecture is a major subcomponent of the technology infrastructure architecture that specifies the design of the communications network. The network architecture framework specifies information such as the network components, their configuration and operating procedures. (EA 874 Enterprise Technology Infrastructure Architecture). 

Most organizations today do not have IPv6 enabled or deployed for production use, it is foreseen that support for IPv6 devices and communication will become increasingly demanded. There are many areas that require IPv6 ready, one area that will require IPv6 is Internet facing services and support for IPv6 only clients. Another area is the Internet of Things, which will bring a proliferation of IP connected devices. IT organization will have to prepare for this very lengthy and large transition to the next generation IP protocol, IPv6. The challenge of migration from an IPv4 network to a federally mandated IPv6 network is becoming a reality, and careful planning of your IPv6 transition will be critical to your success.

Organization needs to develop a future architecture to support current and future requirements for IPv6 based communication. The architecture should address and support the new business solutions that will require the next generation of IP Protocol. In addition, prepare a transition outline and roadmap to realize the architecture. Some envisioned basic preparation phases should cover scoping, training, business cases, and audit of baseline situation.

Organization first needs to assess the implications of IPv6 for the environment, including product compliance, address provisioning and management, routing policies, security, and infrastructure design. This assessment also identifies opportunities to take advantage of IPv6 features and functionality to simplify the environment, as well as areas of risk to be considered during transition to IPv6.

Organization should consider to establish an IPv6 working group to determine the training requirements for levels of IT personnel, such as network engineers, system engineers, helpdesk analysts and security engineers. Determine scope(s) of IPv6 deployment, identify business use cases such as Internet of Things. Audit the existing environment to determine the baseline “as-is” situation and then define the “To-Be” architecture blueprint for each scope.

Internet of Things (IoT) Architecture Platform and Development Approach

Internet of Things (IoT) is about equipping physical objects with smart sensors or actuators and the capability to exchange data amongst each other or with computer system via an IP-based network. The Internet of Things allows objects to be sensed and controlled remotely across existing network infrastructure, creating opportunities for more direct integration of the physical world into computer-based systems, and resulting in improved efficiency, accuracy and economic benefit.

Essentially, IoT expands IP-based networks, like the internet or company-owned intranets, beyond traditional computing devices and shifts it from the digital into the physical space. IoT makes use of synergies generated by the convergence of consumer, business and industrial internet and creates a global network connecting people, data, and things.

IoT means that there is a lot of data extracted from different devices and systems. This data can be an opportunity for a lot of improvements and changes, when interpreted right (e.g. for optimizing the supply-chain-management). Today, most IoT applications are about reacting to the measured data. In future the data could be used a lot more to not only react to a specific situation but to predict what will happen next (e.g. predicting the failure of specific parts of a machine). Therefore IoT will often be combined with Big data & Analytics and IT as a Service (also known as Cloud computing) capabilities to process the vast amount of data and to derive valuable knowledge out of it.

The IoT platform architecture(s) should satisfy the business requirements of today, and position the organization for the future growth in this area. Both vertical and horizontal integration must be strongly considered, and the architecture must enable this through the use of industry standards. Architecture development approach should follow a modular approach to start small and grow smart. This will create an adaptive architecture to cater to dynamic business requirements, driven by business projects.

The approach and general steps to develop the IoT Platform Architecture are:

• Consider the most preferred theoretic target, the ideal scenario based on the greatest potential business benefits
• Assemble the identified use cases and potential vendor / suppliers
• Assemble the vendor / supplier reference
• Evaluate each reference architecture based on the constraints / requirements list
• Look for synergies and commonalities among architectures. Consider future integration points