maandag 27 maart 2017

Machine learning: Getting started with random forests in R

According to Gartner, machine learning is on top of the hype cycle at the peak of inflated expectations. There is a lot of misunderstanding about what machine learning actually is and what it can be done with it.

Machine learning is not as abstract as one might think. If you want to get value out of known data and do predictions for unknown data, the most important challenge is asking the right questions and of course knowing what you are doing, especially if you want to optimize your prediction accuracy.

In this blog I'm exploring an example of machine learning. The random forest algorithm. I'll provide an example on how you can use this algorithm to do predictions. In order to implement a random forest, I'm using R with the randomForest library and I'm using the iris dataset which is provided by the R installation.


maandag 20 maart 2017

Oracle SOA Suite: Find that composite instance!

When executing BPM or BPEL processes, they are usually executed in the context of a specific entity. Sometimes you want to find instances involved with a specific entity. There are different ways to make this easy. You can for example use composite instance titles or sensors and set them to a unique identifier for your entity. If they have not been used, you can check the audit trail. However, manually checking the audit trail, especially if there are many instances, can be cumbersome. Also if different teams use different standards or standards have evolved over time, there might not be a single way to look for your entity identifier in composite instances. You want to automate this.

It is of course possible to write Java or WLST code and use the API to gather all relevant information. It would however require fetching large amounts of data from the SOAINFRA database to analyse. Fetching all that data into WLST or Java and combining it, would not be fast. I've created a database package / query which performs this feat directly on the 11g SOAINFRA database (and most likely with little alteration on 12c).


maandag 6 februari 2017

Oracle Service Bus: Produce messages to a Kafka topic

Oracle Service Bus is a powerful tool to provide features like transformation, throttling, virtualization of messages coming from different sources. There is a (recently opensourced!) Kafka transport available for Oracle Service Bus (see here). Oracle Service Bus can thus be used to do all kinds of interesting things to messages coming from Kafka topics. You can then produce altered messages to other Kafka topics and create a decoupled processing chain. In this blog I provide an example on how to use Oracle Service Bus to produce messages to a Kafka topic.


zaterdag 4 februari 2017

Oracle Service Bus: Pipeline alerts in Splunk using SNMP traps

Oracle Service Bus provides a reporting activity called Alert. The OSB pipeline alerts use a persistent store. This store is file based. Changing the persistent store to JDBC based, does not cause pipeline alerts to be stored in a database instead of on disk. When the persistent store on disk becomes large, opening pipeline alerts in the Enterprise Manager (12c) or Service Bus console (11g) can suffer from poor performance. If you put an archive setting on pipeline alerts (see here), the space from the persistent store on disk is not reduced when alerts get deleted. You can compact the store to reduce space (see here), but this requires the store to be offline and this might require shutting down the Service Bus. This can be cumbersome to do often and is not good for your availability.

If you do not want to use the EM / SB console or have the issues with the filestore, there is an alternative. Pipeline alerts can produce SNMP traps. SNMP traps can be forwarded by a WebLogic SNMP Agent to an SNMP Manager. This manager can store the SNMP traps in a file and Splunk can monitor the file. Splunk makes searching alerts and visualizing them easy. In this blog I will describe the steps needed to get a minimal setup with SNMP traps going and how to see the pipeline alerts in Splunk.

donderdag 12 januari 2017

WebLogic Server: Logging the SOAP action in the access.log

WebLogic Server allows you to customize your access.log. This can be very powerful if you want to monitor for example service response times in a tool like Splunk (see here). When working with SOAP services though, especially those with many operations, it can be insufficient to monitor services to the level of the individual endpoint. You want to also know with which intent the endpoint is called. In this blog I will show how this can be achieved.

maandag 9 januari 2017

Oracle Mobile Cloud Service (MCS). Implementing custom APIs using JavaScript on Node.js.

Oracle Mobile Cloud Service is a mobile backend as a service. MCS does its magic by providing a lot of features to make implementing mobile services easy such as (among many other) authentication, logging/analytics, lookups and calling other services. There are also features available to make integration with mobile clients easy such as providing an easy way to implement push notifications.

Personally I think one of the most powerful features of MCS is the ability to write custom JavaScript code and use that as an API implementation. This custom code can (among the regular JavaScript features) call MCS connectors and platform services. This provides a lot of flexibility in defining API behavior.

In this blog I will show how you can use this custom Node.js code to create an end to end example. I will use a RAML file to define my interface. Next I will define a connector in MCS to call the OpenWeatherMap API. This API returns (amongst other things) the temperature at a location in Kelvin. I want to define my own custom result message (with the temperature in Celsius) which better matches the requirements of my mobile client. I will use a custom JavaScript implementation to call the connector which calls the OpenWeatherMap API and create a custom response message from the result.

The described example is not suitable for a production implementation and is based on limited experience (and watching some really nice YouTube presentations). It is provided to give an idea on how to get started easily with a simple working example.

maandag 19 december 2016

WebLogic Server: Automate obtaining performance metrics from DMS

Oracle provides the Dynamic Monitoring Service (DMS) as part of WebLogic Server which is extremely useful if you want to obtain aggregated data of an environment in case of for example a performance test. The data which can be obtained from DMS is extensive. This varies from average duration of service calls to JVM garbage collects to datasource statistics. DMS can be queried with WLST.  See for example here. On example script based on this can be found here. You can also directly go to a web-interface such as: http://<host>:<port>/dms/Spy. The DMS Spy servlet is by default only enabled on development environments but can be deployed on production environments (see here).

Obtaining data from DMS in an automated fashion, even with the WLST support, can be a challenge. In this blog I provide a Python 2.7 script which allows you to get information from the DMS and dump it in a CSV file for further processing. The script first logs and uses the obtained session information to download information from a specific table in XML. This XML is converted to CSV. The code does not require an Oracle Home (it is not WLST based). The purpose here is to provide an easy to use starting point which can be expanded to suit specific use-cases. The script works against WebLogic 11g and 12c environments (has been tested against 11.1.1.7 and 12.2.1). Do mind that the example URL given in the script obtains performance data on webservice operations. This works great on composites but not on Service Bus or JAX-WS services. You can download a general script here (which requires minimal changes to use) and a (more specific) script with examples of how to preprocess data in the script here.