Posts Tagged ‘Talend’

Loading data in OpenERP with XML-RPC Web services

Architectural elements involved in the XML-RPC.

Image via Wikipedia

Loading data from an external source into OpenERP is quite straight-forward. Before trying ETL tools (such as Talend and Kettle), you should try two things first. Understand OpenERP’s object model (you can do it in the customization menu or by reading the code), and then try it with Python.

Trying the data load with Python is quite easy. First, open the Python prompt. Then try an example, the best example is OpenERP’s XML-RPC Web services documentation itself. I mean, start your openerp-server, then line by line try the example. You will see how it works and how easy it is to load data with web services. Then, I would suggest reading the CSV Python module documentation, which explains how to read a CSV file. After that, you only need to put the pieces together, create a Python program that reads the information from a CSV file (which was exported from LibreOffice) and then l0ad that data into OpenERP with web services.



ETL and OpenERP

February 9, 2011 4 comments
Figure 1: Simple schematic for a data warehous...

Image via Wikipedia

ETL stands for Extraction, Transformation and Loading. They are used to integrate different systems in different platforms, and they do it in batch mode, they don’t do it in real-time (well, there are some tools that do it, but most of the times we integrate the systems on a periodic basis).

I will not explain how an ETL system works, I will explain why you should use it with OpenERP. In a latter post I will cover the topic of updating OpenERP data with ETL.

Why use ETL? Many reasons. The first reason is it is a very productive tool and once you learn how to use it, your productivity as a developer increases twofold (at least). Developing integration jobs is quite intuitive and they allow you integrate diverse environments, such as data from different databases and operating systems in a single job.

The second reason is these tools already have integrated a production environment for monitoring and executing the integration jobs. These environments have grown more robust along the times and its logging facilities are quite impressive. OK, you can do develop the environment yourself, but it takes time and testing, it is not cheap.

The third reason is, you will always need to integrate systems with OpenERP. Always. As customers get larger, they tend to have more systems in place. That’s the law in this industry. And you can not change them, you have to live with them. And living with them means, integration your OpenERP information with their information. And their information with your OpenERP information.

The last reason is, ETL applications save you many hours when it comes to migrating data into your system. MANY HOURS.

Now, when it comes to OpenERP and ETL, you have some options. I just looked at two of them. Talend and Pentaho Kettle. These days we are starting a project that needs to extract information from an accounting system and integrate it with the MRP module in OpenERP. I think we will end up using Kettle, because it already has plug-ins for loading data into OpenERP. Have not seen that yet in Talend. I will cover those modules in a coming post.