File Synchronization in clustered environment

In the clustered environment whenever you want to make any configuration changes your supposed to use WAS Admin Console and make those changes on the Deployment Manager, the deployment manager then takes care of Synchronizing those changes to all the nodes.

Please remember one important thing that the Deployment Manager will only synchronize changes under the config directory


In case of portal server there portal configuration service related files stored under PortalServer directory. So if you want to make any changes in these files you should always use WAS Admin Console. Those changes would get stored in resources.xml file under corresponding scope.

Similarly if you make any changes in wkplc.properties or wkplc_comp.properties or wkplc_dbtype.properties file you will have to copy those changes manually. Please note one more important thing that even in a cluster the wkplc.properties file will be specific to a server so you cannot copy it as it is to other machine. Instead you should copy the corresponding helper file from primary portal node and use it only copy necessary property values to the other nodes.

Steps for adding vertical cluster member

IBM recommends that you should user vertical clustering i.e. having more than one portal JVM's running on single machine to make full use of the resources. Also it looks like setting up vertical cluster is much easier than the horizontal cluster.

I did setup a vertical cluster on my VMWare by following these steps

• Start the Deployment manager if it is not already started


• Login into the WAS Admin Console of the deployment manager by going to https://localhost:9043/ibm/console URL.


• In the WAS Admin Console Go to Server -> Clusters. And select the cluster in which you want to add new member
  
  


• Click on the Cluster members, it will list out the members of that cluster. Click on New. It will open a Create additional cluster member page like this
  
  


• On this page enter name of the new cluster and select the node where this vertical node should be added. Dont forget to check the Generate Unique HTTP ports check box. Click on next couple of pages and when you say finish it will create set of configuration files for the new server
  Important Note :You must update the virtual host entries for the new port created when adding a cluster member. You can do this by updating the default_host virtual host in the administrative console and adding a new alias entry for the port number (use an asterisk [*] wildcard character for the host name).
  


• Once the new vertical cluster member is created you can verify it by going to Cluster topology
  
  


• Next step would be to configure dynamic cache so that the cache entries created on this newly created server get copied to other members of the cluster. You can do that by going to Application Server -> Newly created server -> Dynamic Cache service.
  On this page check Enable cache replication check box, select name of your cluster as Full Group replication domain and set Replication Type as pull only
  
  


• Next go to the wp_profile/ConfigEngine.sh directory on the target machine and execute ./ConfigEngine.sh cluster-node-config-vertical-cluster-setup -DServerName=WebSphere_Portal_V1 task to clean up the server-scoped resources, caches, and resource providers


• Restart the newly added vertical cluster member


• In the Admin Console change the value of WCM_HOST and WCM_PORT websphere variables scoped at the server level to point to the web server which will be used to serve WCM content


• Re Synchronize changes from Deployment manager to the cluster members


• Regenerate the Web Server plugin to include the newly added vertical cluster and copy the newly generated plugin-cfg.xml file to web server and restart web server so that it can start forwarding request to newly added vertical cluster member

Enabling security in clustered environment

Enabling security in clustered environment is little different from enabling security in the standalone environment. Follow these steps to enable security in clustered environment

1. Copy the helper file appropriate to your ldap server from <wp_profile>/ConfigEngine/config/helpers directory and set values to match your LDAP configuration for the properties


2. Execute the ConfigEngine.bat validate-standalone-ldap configuration task that will validate the values of the LDAP that you set in helper file


3. Next execute the ConfigEngine.bat wp-modify-ldap-security task to make the actual changes in the portal security


4. Restart the DMGR, All Node Agents and all cluster members.


5. Copy the helper file that you changed on primary portal node to the secondary portal node.


6. Copy the content of helper file into the main wkplc.properties by running the following command
   ConfigEngine.bat
-DparentProperties=/ConfigEngine/config/helpers/wp_security_ids.p
roperties -DsaveParentProperties=true
   Important Note: Did you notice that we did not mention any config task name here. Instead just specified parentProperties and savenParentProperties parameter. This will only change the wkplc.properties file.
   


7. Update the Portal security information on the secondary node by executing the following ConfigEngine script from the /ConfigEngine directory on your secondary node:
   
ConfigEngine.bat wp-change-portal-admin-user -DnewAdminId=admin ID> -DnewAdminPwd= -DnewAdminGroupId=Admin Group ID> -Dskip.ldap.validation=true

   Note: The -Dskip.ldap.validation=true flag can be used if the script fails during ldap validation.
   


8. Restart the secondary node's WebSphere Portal server

Steps for setting up horizontal cluster

I followed the steps defined in A Step-By-Step Guide to Configuring a WebSphere Portal v6.1.0.0 Cluster using WebSphere Application Server v6.1.0.15 document for setting up 2 node horizontal cluster.

These are the high level steps

1. Install Deployment manager. Check Enable administrative security check box


2. Configure Deployment Manager: Change the request timeout for SOAP, Web Container Chain,.. Also create a WebSphere Portal Admin user in the local file system based repository using WAS Admin Console.


3. Install Standalone portal server. This portal install will act as primary portal node.


4. Configure the websphere portal server to use external database.


5. Federate and cluster the primary portal node


6. Collect files from portal node using ConfigEngine.bat collect-files-for-dmgr task
   


7. Copy the filesforDmgr.zip from primary node to DMGR


8. Expand the filesForDmgr.zip on DMGR and copy content to appropriate directories
   


9. Add node to the deployment manager by executing following command on the primary portal node ConfigEngine.bat cluster-node-config-pre-federation
   


10. Update the deployment manager configuration for new WPS by executing the following command ConfigEngine.bat cluster-node-config-post-federation
    


11. Create the cluster definition and add the WebSphere_Portal server as a cluster member by executing following command ConfigEngine.bat cluster-node-config-cluster-setup
    


12. Install Standalone portal server on secondary portal server


13. Copy database configuration from primary node to secondary node


14. Add node to the deployment manager by executing ConfigEngine.bat cluster-node-config-pre-federation command


15. Update the deployment manager configuration for the new WebSphere Portal server by executing the following ConfigEngine script: ConfigEngine.bat cluster-node-config-post-federation


16. Add this newly federated WebSphere_Portal server as a cluster member to the existing cluster by executing the following ConfigEngine script: ConfigEngine.bat cluster-node-config-cluster-setup

Using External security manager in cluster

If you are configuring security for IBM WebSphere Portal with an external security manager, review the additional considerations described in this section, depending on the external security manager that you are using. Perform any configuration for an external security manager after you have completed all other setup, including ensuring that the WebSphere Portal cluster is functional.
The following considerations apply to all external security managers:

• 
  When setting up security in a cluster to use an external security manager, ensure that you configure either TAM or Siteminder on each of the portal nodes.
  


• If you make any changes to the external security manager configuration after initially setting it up, ensure that any changes you make to the wkplc.properties file are propagated to the other nodes in the cluster. Changes to the wkplc.properties file are not included as part of the cluster resynchronization process.


• If you are using an external Web server, additional configuration is required before running any task to configure an external security manager with a WebSphere Portal cluster. Edit the wkplc.properties file on each node, and ensure that the values for the wp.ac.impl.JunctionHost and wp.ac.impl.JunctionPort properties are set to the backend server host name and port number you are using for your Web server.

Considerations for security during cluster installation and fedrations

When setting up a cluster, there are two scenarios that must be considered. The first scenario is when the default VMM file-based repository security is used at both the WebSphere Portal nodes and the deployment manager until after the WebSphere Portal cluster is completely set up. Prior to federating the first WebSphere Portal node into the cell, the required group for WebSphere Portal administrators must be defined in the deployment manager’s security repository. Once the cluster has been set up, you can modify the security settings of the cell. Although it is possible to modify security in the cell using the WebSphere Application Server administrative interfaces, you should use the WebSphere Portal security tasks to change cell security in order to ensure that the security configuration settings for WebSphere Application Server and WebSphere Portal are identical.

The second scenario is when the existing deployment manager cell has already modified its default security setting prior to the first WebSphere Portal node joining the cell. WebSphere Portal supports the capability of using two different sets of administrative user ID and password credentials when federating a WebSphere Portal node into a cell – one set for the WebSphere Portal node authentication and one set for deployment manager authentication. This means that it is not necessary to define a common administrative user ID before WebSphere Portal joins the cell. If the deployment manager cell is using federated VMM with additional repositories, WebSphere Portal will pick up this configuration dynamically from the deployment manager when it joins the cell. If the deployment manager cell is using standalone LDAP security, however, then it is necessary to configure the LDAP values into the WebSphere Portal property files before federation to enable WebSphere Portal to dynamically adapt to the existing standalone LDAP security settings of the cell. As with the first scenario, once the cluster has been set up then security changes to the deployment manager cell security settings can be made using the WebSphere Portal security tasks, and additional WebSphere Portal nodes may be added to the cell following the same procedures.

The tasks under Setting up a clustered production environment recommend configuring security before configuring your additional nodes but if you configure your security after configuring your additional nodes or if you need to update your security configuration after you have created your clustered environment, you will need to run an additional task to update the security settings on the secondary nodes; see Configuring security after cluster creation for information.

Cluster security options

There are many security options that can be used in a cluster. All of the VMM federated security options, including multiple LDAP repositories, database repositories, and the default file-based repository can be used. Additionally there is an option to use standalone LDAP security instead of the VMM federated security approach.

Note: It is not recommended to use the file-based repository in a production environment. The reason is that updates are only possible through the WebSphere Application Server administrative console, not through portal user management. These updates are sent to each node in the cell using deployment manager file synchronization. This can be time consuming for large volumes of users and groups. Also, synchronization does not occur at the same time for all nodes in a cell, so there will be time windows when the nodes in the cell have differing security definitions.

Cluster Security

WebSphere Portal 6.1 depends on Virtual Member Manager (VMM) for security. The VMM configurations are applied at the cell level and maintained at the Deployment manager level. When you federate a node into the Deployment manager it wont change any of the deployment manager security settings instead any changes that you made at the portal level will get lost and portal will inherit the security settings at the Deployment manager level.

Note: If administrative security is deselected during installation of the deployment manager or is disabled after the deployment manager is installed, it must be enabled prior to executing the security configuration tasks on the WebSphere Portal cluster members.

WebSphere Portal provides a number of security tasks, which can be used to modify the WebSphere Application Server security settings and make the required updates to the WebSphere Portal configuration in a single step. As soon as a WebSphere Portal node is federated into a deployment manager cell, all WebSphere Portal security tasks will execute on the deployment manager. Run security tasks after federating the WebSphere Portal node because the Deployment Manager cell does not contain the configuration resources required to run the security tasks.

Http Session Failover

User interactions with WebSphere Portal Server are maintained through the use of a
HttpSession. This provides a way to preserve data across multiple pages or requests on an individual user basis. The failure or outage, either scheduled or unscheduled, of a WebSphere Portal Server cluster member will result in the termination of the user's HttpSession.

You can configure WebSphere Application server so that in case of failure the Users Http Session is available to the other cluster, this is called distributed session. WAS provides following two options to do this

• Database session persistence, where sessions are stored in the database specified.


• Memory-to-memory session replication, where sessions are stored in one or more specified WebSphere Application Server instances or profiles.

When a session contains attributes that implement HttpSessionActivationListener, notification occurs anytime the session is activated (that is, session is read to the memory cache) or passivated (that is, session leaves the memory cache). Passivation can occur because of a server shutdown or when the session memory cache is full and an older session is removed from the memory cache to make room for a newer session. It is not guaranteed that a session is passivated in one application server prior to activation in another application.

When you store an attribute in the PortletSession it actually gets stored in the HttpSession only the attribute name is stored in the namespaced format.

LDAP for geographically deployed websphere portal

The implementation of a multi-clustered WebSphere Portal Server V6.0.x
architecture that sees the deployment of individual clusters in each geography to support a truly "global deployment" mandates the use of the same LDAP directory server in all geographies. The LDAP directory server may, however, be replicated for redundancy purposes. This requirement is necessary to maintain uniqueness between Portal users.

Database for geographically deployed architecture

In case of geographically deployed architecture each geography maintains its own set of databases. Each database would be highly available. In this case the shared databases would be replicate across wide area network using technique as queue replication or 2way sql replication

Guidelines for vertical and horizontal clustering

Please follow these guidelines while deciding about using either Vertical or horizontal clustering

Vertical clustering
To fully utilize the processing power of modern SMP servers
Local redundancy

Horizontal cluster
To achieve scalability beyond the limitation of individual server
Reduandancy and reliability
Hardware failover

Horizontal scaling is especially effective in environments that contain many smaller, less powerful machines. Client requests that would overwhelm a single small machine can be distributed over several machines in the solution. Failover is another benefit of horizontal scaling. If a machine becomes unavailable, its work can be routed to other machines
containing cluster members.


By contrast, vertical cloning benefits Symmetric Multi-Processing (SMP) systems and should be implemented when system resources are found to be underutilized. For Java, systems with multiple processors typically outperform multiple systems with fewer processors, when comparing the total number of processors.

A general rule of thumb has always been to allocate a single Java Virtual Machine (JVM) to a single processor. Traditionally, this was based on the constraint that the compaction phase of Garbage Collection (GC) was single threaded. Recent versions of the JVM have, however, minimized this restriction, potentially allowing a greater number of JVMs to run given the total number of available processors. It is also worth remembering that there are a number of JVMs associated with the Deployment Manager and NodeAgent (a NodeAgent is required for each individual physical node participating in the cell) when running Portal Server V6.0.1 in a clustered environment.

It is highly recommended to use combination of horizontal and vertical clustering to make full use of the system resources and still making sure that system is fault tolerent if one of the server hardware fails

Additional considerations for multiple lines of productions

In case of multiple lines of productions environment every cell + portal cluster will have its own release database but they will share customization database, because of this if user customizes any of the resource using one cluster those changes would be available when same user will access the resource even through other cluster.

Since multiple lines of production will share the customization database they should be exact replica of each other, i.e. the object ids of the resource such as portlet or page should be same in both lines of production. In order to make sure that object ids are same you will have follow some additional guidelines in case of multiple lines of production

First you will need to execute empty-portal configuration task on both lines of productions to clean it up during the setup process. Once that is done you will have to export configuration from the staging environment and import same configuration with ID generation mode set to false. this will make sure that the portal does not generate object ids while importing portal configuration.

You should use the release builder tool to build subsequent releases

Maintenance procedure for multiple cluster environment

In case of multiple cell+cluster environment i.e. WebSphere Portal Server Gold Availability Standard environment every cluster gets its own Release domain but the customization and community data is shared. As a result you can safely apply the major fixes that change the configuration stored in the release database.

Follow this procedure to update the multi cell environment

• Change the IT Sprayer so that it stops forwarding requests to the one line of production


• Once that is done you can update that line of production and during that time second line of production will service requests


• Test the first line of production and once your satisfied you start the first line of production and reconfigure IP sprayer to forward request to first line of production


• Follow similar procedure for all other lines of productions

The geographically deployed architecture

One important requirement for WebSphere Portal is ability to deploy an architecture in geographically distributed fashion.

Geogrphically distributed architecture makes sense either because of the disaster recovery requirement which means that even if one data center is down the application still works. Or when major geography maintains the local services and back-end systems that are effectively accessed through the portal.

With the introduction of database domains in WebSphere Portal Server V6.0.x, greater
flexibility was made possible in terms of the permissible operational architecture. As such, the distinction between release, community, and user customization data has made it possible to achieve a truly "global deployment". Readers familiar with previous versions of WebSphere Portal Server will recall that it was not possible to split the Portal database between multiple redundant clusters, located potentially in different geographies, and to maintain a consistent user experience. Indeed, such an architecture when deployed sacrificed the ability for a user to make any customization or personalization modifications, as the changes simply could not be propagated between clusters. This in part was attributed to the fact that the internal object
IDs associated with the various elements of a deployment could not be guaranteed to be
unique. Any attempt, therefore, to deploy a bi-directional database replication technique was further hindered. To overcome this constraint, it was mandatory that all Portal cluster members, participating in the same Portal instance, accessed the same centralized database. However, the performance considerations of accessing a centralized database across the WAN from geographically dispersed Portal servers made this approach impractical in many environments.

In addition to the separation of Portal data into distinct database domains with WebSphere Portal Server V6.0.x, which represents an acknowledged product improvement, it should also be recognized that Portal data can now be shared between different Portal clusters and the very cluster members that exist within them. Such database domains can now be deployed in a peer-to-peer manner using techniques like queue replication or 2-way SQL replication in order to provide a global deployment capability, where user personalization is automatically made available to all Portal clusters in all geographies. In this manner, WebSphere Portal Server V6.0.x also allows users to experience portability should they temporarily access the Portal solution from another geo (branch or office location).

Important: The implementation of a multi-clustered WebSphere Portal Server V6.0.x
architecture that sees the deployment of individual clusters in each geography to support a truly "global deployment" mandates the use of the same LDAP directory server in all geographies. The LDAP directory server may, however, be replicated for redundancy purposes. This requirement is necessary to maintain uniqueness between Portal users.

Multiple cluster architecture

Starting from WPS 6.0.1, you can architect multiple portal clusters within the same WebSphere Cell.

Please note that multiple cluster architecture has one very important limitation, which is that, only a single J2EE enterprise application of a unique name can be deployed into a given websphere cell at any one time. All enterprise applications are cell scoped. This limitation makes it impossible to deploy different versions of the same enterprise application against the different portal clusters within the same cell, as the case might be during the periods of 24*7 maintenance. This extends to the WebSphere Portal server itself, which consist of number of enterprise applications that make up the effective runtime and also to the every portlet application deployed within the solution. As a consequence, it is not possible to upgrade one portal cluster in isolation from another as the underlying enterprise application and supporting class libraries are common to both.

This architecture meets requirement of continuous operation during scheduled or unscheduled architecture but it is not a good feet if you want truly 24*7 service. In case of requirement for 24*7 service you should think about WebSphere Portal Server Gold Availability standard

Maintenance procedure for single cluster portal

In case of single cluster environment maintenance must be performed as a multi-step process, these are the steps

• Disable the automatic file synchronization service from the Deployment Manager Administration consoel


• Stop the node agent on each of the node participating in the cluster.


• Perform maintenance on each node in turn starting with primary portal node by first gracefully quiescing user requests from each node by modifying the WebSphere Web Server plug-in load balancing weighting while remaining node or nodes in the cluster continue to honor user requests.


• The final steps is to synchronize and restart all of the nodes one at a time
• Re-enable automatic file synchronization service using Deployment manager administration console

Single cluster architecture

The single cluster architecture means when you have only one cluster under one cell and that cluster can have combination of horizontal and vertical cluster members.



Key features of this architecture are:


• A single load balanced HTTP Server cluster (HTTP Cluster) that spans two or more
  physical nodes.


•  A single WebShere Portal Server cluster (Portal Cluster) deployed in a single WebSphere Cell.


• The WebShere Portal Server cluster consists of two of more horizontal cluster members and any number of vertical cluster members per node (resources permitting).


• A dedicated stand-alone WebSphere Deployment Manager is responsible for the
  management of the entire WebSphere Cell (Cell A).


• As the environment only consists of a single WebShere Portal Server cluster, only a single release database domain is required. The remaining database domains (communityusr, customizationusr, wmmusr, fdbkusr,lmdbusr, and jcr) are deployed alongside the release database domain. Note that the JCR Repository exists in a different database.


•  The environment also hosts a LDAP directory server (not shown), which is highly
  available, for maintaining the registered user base.

Why do we need clustering

Most of the enterprise applications require scalability, fault tolerance and redundancy

The ability to scale websphere portal server or any other websphere application server is essentially achieved by clustering. Clustering allows requests to be workload managed between a number of cloned copies of the concerned application. In addition clustering also takes care of redundancy and fault tolerance.

Clustering helps in rendundancy and fault tolerance by making sure that there is no single point of failure.

In clustered architecture utilizes the same user customization, community and release data throughout an environment, any user customization made against one portal cluster member by a user would be available to the same user as and when that user accesses any of the other cluster members participating in the same portal cluster.

Under normal conditions, session affinity is maintained against the same portal cluster member until such time that a user terminates his or her session or the portal cluster member becomes unavailable, either through a deliberate or an unscheduled outage.

WebSphere Portal Topology

Single Server topology
In a single server installation all of the portal components such as portal server, WCM, Portal Search are installed on same machine. You can also install Database on LDAP on same machine or use the Database and LDAP that installed on remote machine.

This is a good option when you want to install Proof of concept environment.

Vertical scaling

Vertical cluster is when you have more than one instance of WebSphere Portal server running on the same machine. In this case multiple java processes would be running on that machine.

You can use vertical scaling to fully use a node within the conceptual node group in the case where resource congestion or locking conditions prevent a single application instance to scale up to the nodes limit.

Horizontal scaling

In a horizontal scaling topology members of a WebSphere Portal cluster exists on multiple physical machines

Clustering is a most effective in environments that use horizontal scaling because of the ability to build in redundany and failover to easily add new horizontal cluster members to increase capacity and to improve scalability by adding hertogeneous systems into cluster.

Mixed horizontal and vertical scaling topology

Mixed horizontal and vertical scaling topology is the best practice approach for large scale deployments. The use of vertical clustering takes full advantage of the resources of a multiprocessor systems. In addition horizontal cloning allows for upward scalability.