This system is designed to showcase a Tofex module. A guide to use this system is available using this link.
The documentation of the module is available using this link.

Dies ist eine Demo-Installation. Es werden keine Aufträge erfüllt.

Tofex Data Integration

Introduction

This module provides a framework to asynchronously process and monitor data send to or by Magento.

Concept

In larger system environments data has to be synchronized between Magento and other systems. For instance orders need to be exported to an ERP system or customers have to be imported from a CRM. In all cases a stable connection from system to system is the key to valid data on all sides. A direct communication between the systems is usually not safe enough and lacks the possibilities to track the exchange process. Therefor Data Integration is a Magento internal data hub using an external queue to process the data.

Requests

Each data transfer is organized in a request which holds data to process and current status information. How the requests are handled is based on the configured complexity of the integration. In a simple scenario there are no restrictions of how requests are processed. Such restrictions can be a limit that only one request of each object is allowed to be executed at a time in order to avoid requests of the same type to be processed simultaneously. Another would be the possibility to connect requests with each other to make sure that they are processed in that required order.

Queue

In order to handle the execution of the requests an external queue is required. This queue handles the distribution of requests to the right worker and also retries. In a request produced an error and the number of retries has not been reached the queue gets notified to provide the request again after the configured delay has been reached. So the queue acts as central control system without having any knowledge which data it processes.

Usage

Administration

Using "Tofex > Data Integration" a list of all requests can be accessed.

Grid

Each request has these fields:

  • Request Id: Internal id of the request generated by Magento.
  • Store View: Store View of the object to process.
  • Object Name: The name of the object to identify the configuration of its handling.
  • Object Id: An id which helps to process the request, i.e. an order or customer id.
  • Integration: The name of the object to identify the implementation to handle the request.
  • Status: The current status of the request. Possible: New / Progress / Error / Success / Failure / Pending / Resolved.
  • Blocked By: In case this request by another requests which has to finish first.
  • Create Date: The date the request was created.
  • Update Date: The date the request was updated.

If you click on a request you can view its history.

Details

Configuration

Using "Stores > Configuration > Data Integration > Data Integration" you can access the configuration.

General

ConfigGeneral

  • Error Mail Sender: Which contact should be used to send error mails.

Auto Clean

Requests will automatically be cleaned if they have an expiry time and this expiry time has passed.

ConfigAutoClean

  • Clean Successful Requests: Clean expired requests if they were successful.
  • Clean Failed Requests: Clean expired requests if they have failed after too many retries.
  • Clean Resolved Requests: Clean expired requests if they were set to resolved.

Archive

If you want to avoid a long history of requests in the database, you can choose to archive them.

ConfigArchive

  • Clean Successful Requests: Archive requests if they were successful.
  • Clean Failed Requests: Archive requests if they have failed after too many retries.
  • Clean Resolved Requests: Archive requests if they were set to resolved.
  • Path: Where to store the archived requests.

RabbitMQ

RabbitMQ is used to process the data integration requests asynchronously.

ConfigRabbitMQ

  • Host: Host to access the queue from the worker.
  • Port: Port to access the queue from the worker.
  • User: User to access the queue from the worker.
  • Password: Password to access the queue from the worker.
  • VHost: Virtual host the Magento instance.

Object

These are the settings which are supported for any object, here "Test Object".

ConfigObject

RabbitMQ

Worker

Description:
  Tofex DataIntegration Worker

Usage:
  dataintegration:worker [options]

Options:
      --request_id[=REQUEST_ID]    The id of a specific request
      --object_name[=OBJECT_NAME]  The name of the object, i.e. "newsletter_subscriber"
      --create_subthreads          Creates sub threads for the requests
      --pid_dir[=PID_DIR]          Directory to store the pid file in case sub threads are used, default: /tmp [default: "/tmp"]
      --output_dir[=OUTPUT_DIR]    Directory to store the output in case sub threads are used, default: /tmp [default: "/tmp"]
      --console                    Print output to console

If you want to start a worker for the object with name "test_object":

bin/magento dataintegration:worker --object_name test_object

If the processing of a request results in a long process time of a couple of minutes the option "create_subthreads" should be used. Processing of data for a long time can lead to the closing of connections of the started worker. By moving the execution of the request in a sub process the main process can keep all connections alive during the wait of the sub process to finish. Usually this is not required for most integrations.

bin/magento dataintegration:worker --object_name test_object --create_subthreads

The number of workers for one object is not limited. If multiple workers are started the queue decides which waiting worker receives the next request. By starting multiple workers requests can be processed faster, i.e. if you have a lot of orders, it is possible to process them simultaneously.

Development

Creating Requests

The easiest way to create a new request is by using the registry which data Integration provides. This is a small to create a request for an object of the name "test_object" and with a random id which should be processed by the implemented send create method using the store with id 0.

<?php

namespace Tofex\DataIntegrationTest\Model;

use Tofex\DataIntegration\Model\IDataIntegration;
use Tofex\DataIntegration\Model\Registry;

class Implementation
{
    /** @var Registry */
    protected $registry;

    /**
     * @param Registry $registry
     */
    public function __construct(Registry $registry)
    {
        $this->registry = $registry;
    }

    ...

    protected function createDataIntegrationRequest()
    {
        try {
            $this->registry->addIdToRegistry('test_object', mt_rand(1000000, 9999999),
                IDataIntegration::SEND_CREATE, 0);
            ...
        } catch (Exception $exception) {
            ...
        }
    }
}

There are also methods to add a Magento object to the registry or add additional data, etc.

Processing data

Configuration

Each object has a number of settings which are required for the integration to work. The most imported are the number of retries in case of an error and the delay between retries.

These are some default values for a test implementation. In this case the name of the object is "test_object" which requires a configuration entry which this name in the section "data_integration".

The last configuration contains the name of the integrations. It is possible for one object to have multiple integrations, i.e. different order information are send to the ERP and to the CRM system. So you don't need to use multiple objects. In this case the object "test_object" has only integration named "test_integration".

etc/config.xml

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Store:etc/config.xsd">
    <default>
        <data_integration>
            <test>
                <active>1</active>
                <queue>rabbitmq</queue>
                <use_delay_for_first_message>0</use_delay_for_first_message>
                <work_delay>10</work_delay>
                <max_attempt_count>5</max_attempt_count>
                <block_on_failed>1</block_on_failed>
                <block_simultaneously>1</block_simultaneously>
                <error_handling>email</error_handling>
                <error_handling_email>[email protected]</error_handling_email>
                <failure_handling>delete_and_send_e-mail</failure_handling>
                <failure_handling_email>[email protected]</failure_handling_email>
                <integration>test</integration>
            </test>
        </data_integration>
        <test>
            <test>
                <active>1</active>
                <model>Tofex\DataIntegrationTest\Model\Integration</model>
            </test>
        </test>
    </default>
</config>

Next to the data integration entry is the configuration for the integration. The name of section is the name of the object, the name of the group is the name of the integration. The configured model has to contain the processing of the object.

etc/adminhtml/system.xml

If you want to enable the configuration the settings in the backend you can a system.xml like this.

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Config:etc/system_file.xsd">
    <system>
        <section id="data_integration">
            <group id="test_object" translate="label" type="text" sortOrder="710" showInDefault="1" showInWebsite="1" showInStore="1">
                <label>Test Object</label>
                <field id="active" translate="label" type="select" sortOrder="10" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Active</label>
                    <source_model>Magento\Config\Model\Config\Source\Yesno</source_model>
                </field>
                <field id="queue" translate="label" type="select" sortOrder="20" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Queue</label>
                    <source_model>Tofex\DataIntegration\Model\Config\Source\Queue</source_model>
                    <depends>
                        <field id="active">1</field>
                    </depends>
                </field>
                <field id="work_delay" translate="label comment" type="text" sortOrder="30" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Work delay in seconds</label>
                    <comment><![CDATA[Delay for retries and for processing messages. CHANGES RESULT IN ERROR UNTIL OLD QUEUES ARE DELETED!!]]></comment>
                    <validate>validate-number</validate>
                    <depends>
                        <field id="active">1</field>
                    </depends>
                </field>
                <field id="use_delay_for_first_message" translate="label comment" type="select" sortOrder="40" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Use delay for first message</label>
                    <comment><![CDATA[Should the request processed after a delay?]]></comment>
                    <source_model>Magento\Config\Model\Config\Source\Yesno</source_model>
                    <depends>
                        <field id="active">1</field>
                    </depends>
                </field>
                <field id="max_attempt_count" translate="label" type="text" sortOrder="50" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Max Attempt Count</label>
                    <validate>validate-number</validate>
                    <depends>
                        <field id="active">1</field>
                    </depends>
                </field>
                <field id="block_on_failed" translate="label" type="select" sortOrder="60" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Block on Failed Requests</label>
                    <source_model>Magento\Config\Model\Config\Source\Yesno</source_model>
                    <depends>
                        <field id="active">1</field>
                    </depends>
                </field>
                <field id="block_simultaneously" translate="label" type="select" sortOrder="70" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Block Simultaneously</label>
                    <source_model>Magento\Config\Model\Config\Source\Yesno</source_model>
                    <depends>
                        <field id="active">1</field>
                    </depends>
                </field>
                <field id="error_handling" translate="label" type="select" sortOrder="80" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Behavior After Error</label>
                    <source_model>Tofex\DataIntegration\Model\Config\Source\ErrorHandling</source_model>
                    <depends>
                        <field id="active">1</field>
                    </depends>
                </field>
                <field id="error_handling_email" translate="label comment" type="text" sortOrder="90" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>E-Mail Address After Error</label>
                    <comment><![CDATA[Multiple addresses separated by semicolon]]></comment>
                    <depends>
                        <field id="active">1</field>
                        <field id="error_handling">email</field>
                    </depends>
                </field>
                <field id="failure_handling" translate="label comment" type="select" sortOrder="100" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Behavior After Failure</label>
                    <comment><![CDATA[How requests should be handled, after the maximum number of attempts is reached.]]></comment>
                    <source_model>Tofex\DataIntegration\Model\Config\Source\FailureHandling</source_model>
                    <depends>
                        <field id="active">1</field>
                    </depends>
                </field>
                <field id="failure_handling_email" translate="label comment" type="text" sortOrder="110" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>E-Mail Address After Failure</label>
                    <comment><![CDATA[Multiple addresses separated by semicolon]]></comment>
                    <depends>
                        <field id="active">1</field>
                        <field id="failure_handling">email</field>
                    </depends>
                </field>
            </group>
        </section>
        <section id="test_object" translate="label" sortOrder="710" showInDefault="1" showInWebsite="1" showInStore="1">
            <label>Test Object</label>
            <tab>dataintegrationtab</tab>
            <resource>Tofex_DataIntegrationTest::config_data_integration_test</resource>
            <group id="test_integration" translate="label" sortOrder="10" showInDefault="1" showInWebsite="1" showInStore="1">
                <label>Test Integration</label>
                <field id="active" translate="label" type="select" sortOrder="10" showInDefault="1" showInWebsite="1" showInStore="1">
                    <label>Active</label>
                    <source_model>Magento\Config\Model\Config\Source\Yesno</source_model>
                </field>
            </group>
        </section>
    </system>
</config>

Implementation

All six supported request method by Data Integration have to be present in the implementation. They are also required if you implement the interface which should always be the case. Usually not all methods are used for all objects, i.e. orders are usually exported from Magento to a different system, so only sendCreate and sendUpdate might by implemented. The methods do not need to provide a certain result. In case of an error in the processing an exception has to be thrown.

<?php

namespace Tofex\DataIntegrationTest\Model;

use Exception;
use Psr\Log\LoggerInterface;
use Tofex\DataIntegration\Model\IDataIntegration;
use Tofex\DataIntegration\Model\Request;

class Integration
    implements IDataIntegration
{
    /**
     * Sends a create request to the implemented target.
     *
     * @param Request $request
     *
     * @throws Exception
     */
    public function sendCreate(Request $request)
    {
        ...
    }

    /**
     * Sends an update request to the implemented target.
     *
     * @param Request $request
     *
     * @throws Exception
     */
    public function sendUpdate(Request $request)
    {
        ...
    }

    /**
     * Sends a delete request to the implemented target.
     *
     * @param Request $request
     *
     * @throws Exception
     */
    public function sendDelete(Request $request)
    {
        ...
    }

    /**
     * Handles the received creation data.
     *
     * @param Request $request
     *
     * @throws Exception
     */
    public function receiveCreate(Request $request)
    {
        ...
    }

    /**
     * Handles the received update data.
     *
     * @param Request $request
     *
     * @throws Exception
     */
    public function receiveUpdate(Request $request)
    {
        ...
    }

    /**
     * Handles the received delete data.
     *
     * @param Request $request
     *
     * @throws Exception
     */
    public function receiveDelete(Request $request)
    {
        ...
    }
}

License

Tofex Data Integration is licensed under the MIT License - see the LICENSE file for details.

Copyright © 2013-gegenwärtig Magento, Inc. Alle Rechte vorbehalten.