White Papers

Integration Mechanism in Market Server

Introduction

Market server can push data to any system. Similarly, any other system can push data into Market Server. The Market Server product accepts and gives out data in the form of XML (for POs, Invoices, product catalogs etc) to facilitate integration.

The following are issues with any integration

Medium of transfer of the data/documents (HTTP, FTP, mail, etc. – the issue being that both systems should be capable of understanding the protocol used).
The two systems may support different document types (XML, EDI, ASCII files, etc.)
Mapping of data elements between the two systems. (eg. the eBiz PO may contain a first name and last name field, while the backend system may need it in a single field called Cust Name).
Other issues include reliability, security, heterogeneous systems (for example, UNIX to Windows and vice versa), sending to multiple applications (belonging to different vendors, for example).

Market Server sites have been integrated with Microsoft Biztalk Server 2000. Biztalk is an offering from Microsoft aimed at facilitating integration with backend systems. Through Biztalk integration with the backend system can be done in a robust manner that addresses all above issues and more too. The features of Biztalk are explained below. This is followed by a technical overview of Marketserver integration into Biztalk.

To give an example of integration of Market Server with an application which is very different, Calsoft has taken after that an example of a UNIX based application that needs to link up with Market Server. Calsoft has shown a scenario of direct integration, and a scenario where Biztalk is used. The UNIX example is shown just to show very widely disparate systems. For Microsoft based systems, Biztalk has other options too which are explained towards the end of this document.

What Is Biztalk ?

Biztalk offers two advantages – it takes care of the messaging, and it also helps mapping data elements needed in our product to data elements needed for you. Biztalk queues up messages between applications and utilizes a variety of media to transport them to the other application both synchronously and asynchronously. BizTalk Server 2000 natively supports HTTP, HTTPS, SMTP, MSMQ, DCOM and file. BizTalk Server 2000 also supports IBM MQSeries through the use of a component included with Host Integration Server 2000. Through this, integration with Legacy mainframe applications is also possible.

The Market Server product accepts and gives out data in the form of XML (for POs, Invoices, product catalogs etc) to facilitate integration. Through the integration of Biztalk into the product, these XMLs get queued up within Biztalk. Biztalk supports mapping from one XML to another, and conversions between XMLs and standard EDI formats and CSVs (Comma Separated values). Biztalk gives the following advantages – it takes care of reliable transfer – if the transfer fails for some reason, it queues and retries. If one protocol (for transfer) fails, it can try other protocols. Eg. If HTTP fails, it can use the backup protocol, which can be file transfer. There can be features such as acknowledgement of receipt of data, which can be enabled. Thus the system can know that the other system did receive the data, and retry if it did not. Validity periods can be specified for a document so that in case the document is queued pending availability of the other system, it will be removed automatically on time expiry.

Document tracking can be enabled to audit which documents/data entered and left the system at what time. The same document can be routed to different heterogeneous business applications/databases simultaneously. Biztalk provides features for security and digital signatures so that the data is secure when it moves outside the company boundaries. When Market server needs to be integrated to another system, Biztalk provides mapping tools convert from one XML to another XML, or one document format to another, taking care of data level incompatibilities too.

Document Transfer From Market Server Using Biztalk Server

Each document that is sent from market server using Biztalk server should go through a channel. This enables both document validation and mapping. Therefore each document type (Purchase order, Deal, catalog etc) in market should be associated with a channel. The validation may be done against schemas that are specific to market server. The schemas may be stored in local file system and mapped to a virtual directory. This enables biztalk to access these schemas through HTTP and also the same schemas are used for internal validations when messaging happens without biztalk.

Each document type (PO, Deal etc) is associated with a channel (This information should be stored in Market server database). Channels are connected to ports that determine the destination and transport protocol for the document. The above distribution list can be static i.e. created in the Biztalk management desk and used always or else it’s can be created dynamically using the interfaces exposed by biztalk server.

Sending The Same Document To Multiple Trading Partners

Sometimes the same document (Catalog for example) may have to be sent to multiple trading partners. This can be achieved by creating a distribution list in biztalk. The distribution list will be group containing the ports of all these trading partners. A channel can be attached to this distribution list. Once a document is sent to this a channel, it is validated, transformed and sent to all the trading partners whose port information is a part of the distribution list. Distribution lists can be Static or Dynamic. Static distribution lists are created using the biztalk management desk. Presently distribution lists in Marketserver are static.

Distribution lists can also be created programmatically using the interface exposed by biztalk. Dynamic distribution lists would be very useful when documents are sent to counter parties (counter parties may change with transaction etc) A new trading partner can be added to this list by adding his port information to this distribution list.

The BizTalkPortGroup object configures a group of complete BizTalkPort objects for sending the same document to a group of trading partners.

Ports can be added or removed from to or from a BizTalkPortGroup by calling the AddPort or the RemovePort method.

Direct Integration With Market Server And UNIX Application

This has been given as an example to show how Marketserver developed in Microsoft technologies can be used to connect to a widely disparate system – a legacy system running on UNIX. Whenever the MarketServer needs to send any data, eg a PO, it is converted into an XML can be sent as a HTTP request to any receiving machine that has a web server. This receiving machine can be a UNIX machine too. In the UNIX machine, a receiving application, written in Java Server Pages (JSP) can receive the XML, transform it to the format required by the existing application, and ‘push’ it into the application. The pushing into the application can be done by a direct database call from JSP or through an Application Programming Interface provided by that application.

If data needs to be pushed from the application to MarketServer, a small add-on can be written to the application, that makes a HTTP call to a URL in the Market Server site with the XML data as parameter. On receiving the HTTP request the MarketServer extracts the data and processes it.

Integration With Server & UNIX Application Using Biztalk

The process mentioned above can be made much more reliable and generic using Microsoft Biztalk Server. Using Biztalk, our product can both push and pull documents and data with other backend systems in a manner that is robust and scalable (meaning integrations of further systems is simplified). For transferring data from MarketServer to the Unix based system, two Biztalk servers would need to be present as intermediate nodes in the transfer – one would be within the organization and the other in the web server machine on which MarketServer is hosted. The MarketServer would put the message (document or data) into the Biztalk queue. This would post it in a reliable manner (with security authentication and encryption options – see last section for details) to the receiving Biztalk. This Biztalk can be programmed to post this data via HTTP calls to the UNIX machine. In the UNIX machine, the same process mentioned above can happen. Similarly the UNIX machine can make HTTP requests back to Biztalk to send its data to the Biztalk queue.

The important issue here is of the format in which data is delivered from the UNIX system. If it is in the form of ASCII or EDI files, they can be converted using some converters present in Biztalk to XML format. The advantage of putting the data as XML, is that any later different vendors (who maybe on different platforms) can be integrated without any more changes on the UNIX system. Also further integration with other systems of other vendors for applications residing in the same organization can be made through the same Biztalk server. Finally Biztalk has mapping tools that can easily map the XMLs from the UNIX system to the XMLs of the receiving system.

In case the other application is running on Windows, then Biztalk has the additional option of using DCOM or simply writing into a common shared directory as options apart from HTTP requests. Hence as long as an application supports some APIs or programmable interfaces, or simply exports and imports files from a particular directory, Biztalk can talk to it through small pieces of custom developed code. For standard ERPs such as SAP and BAAN, Biztalk already has adapters which connect with SAP/BAAN in a smooth manner.