Symphony.

We are using XML Importer to import an external product database into Symphony. Entries are updated based on their id (which forks just fine) but we need to remove all entries that all no longer part of the external source.

It's possible to find these entries based on the entry's modification date, but how do I execute a script to remove these entries after the XML Importer run successfully (and only then)?

Hm, does the the XML importer provide a import delegation?

Edit:

Seems like it does

So you can hook up to one of these and run your program.

Import delegation sounds cool but at the risk of sounding stupid:
What do you exactly mean?

@Nils, assuming that all is updated properly, maybe it would be possible to pull out all entries where the System Update time, has not been modified then remove those entries?

I would think you'd need to pop in some custom code somewhere

Forum Post

Maybe Brians reply to a similar post may help?

Not sure if it works in your case, but think of a delegate as an event you can subscribe to. So basically you hook up to this event (unfortunately you have to do this wihin an extension.driver class) and do whatever you need to do.

I altered the XMLImporter code to add 2 delegates for Successful Post Run and Error Post Run for notifying the success/failure of an import via email.

Then subscribed to these 2 delegates in a small extension. Crude coding but it worked.

This could be used to trigger the cleanup function of redundant entries maybe?

Thanks, Thomas, I thought for a second that there was something that would not require a custom extension. Things like this are really a pain in Symphony.

@Jonathan: Yes, as I wrote above, the entries can be found using the modification date. Everything in that section that was not updated within the last 5 minutes can be deleted. I was really just looking for a simple way to trigger that deletion.

@Andrew: Yeah, I've seen this post before. Using the modification date already solves the issue of finding the outdated entries.

By the way, for those interested, I use this query to find outdated entries after import:

SELECT t1.id
FROM sym_entries as t1
INNER JOIN sym_entries_data_131 as t2
WHERE t1.`id` = t2.`entry_id` 
AND t1.`modification_date` < DATE_SUB(NOW(), INTERVAL 5 MINUTE)
AND t2.`relation_id` IN ('2559', '2563')

Field 131 is a Select Box Link I use to filter the entries by category.

No worries, nice additions to the extension. Very handy. Did you see my mod for Post Import Run above? that may help.

Oh, no, I didn't see that post.

What about sending a pull request with that change to the main repo?

I did think about that... Success PostRun and ErrorPostRun delegates would come in handy for other extensions to access me thinks!

I'll pull integration and update my code in there.. or shall I push to a feature branch instead?

In case anyone else needs something like this: I decided to create an AJAX page which allows me to create a button for authors to run the importer. It returns the status that can be displayed as system message.

<?
require_once(TOOLKIT . "/class.extensionmanager.php");
require_once(TOOLKIT . "/class.entrymanager.php");
require_once(EXTENSIONS . '/xmlimporter/lib/class.xmlimportermanager.php');

class contentExtensionImport extends AjaxPage {

    public function view() {
        $importManager = new XmlImporterManager();
        $importer = $importManager->create('produkte');
        $status = $importer->validate();

        // Import
        if($status == XMLImporter::__OK__) {
            $importer->commit();

            // Remove deleted entries
            $id = Symphony::Database()->fetchCol("id", "
                SELECT t1.id
                FROM sym_entries as t1
                INNER JOIN sym_entries_data_131 as t2
                WHERE t1.`id` = t2.`entry_id` 
                AND t1.`modification_date` < DATE_SUB(NOW(), INTERVAL 5 MINUTE)
                AND t2.`relation_id` IN ('2559', '2563')
            ");

            if(!empty($id)) {
                $entryManager = new EntryManager();
                $entryManager->delete($id, 16);
            }

            // Get import stats
            $importer_result = array(
                'created' => 0,
                'updated' => 0,
                'skipped' => 0
            );

            $entries = $importer->getEntries();
            foreach ($entries as $entry) {
                $importer_result[$entry['entry']->get('importer_status')]++;
            }

            $this->_Result = array(
                'status' => 'success',
                'message' => sprintf('Es wurden %d Einträge hinzugefügt, %d Einträge aktualisiert und %d Einträge gelöscht.', $importer_result['created'], $importer_result['updated'], count($id))
            );
        }

        // Error
        else {
            $this->_Result = array(
                'status' => 'error',
                'message' => 'Leider ist beim Import ein Fehler aufgetreten.'
            );
        }
    }

    public function generate($page = null){
        header('Content-Type: application/json; charset=utf-8');
        echo json_encode($this->_Result);
        exit;
    }

}

I'll pull integration and update my code in there.. or shall I push to a feature branch instead?

Both would be fine, I guess.

Thanks for sharing @Nils, this is a pretty tidy solution. And thanks for the extra delegates (hooks) @moonoo2. I agree that they would be very useful to have in the master branch for XML Importer. The modification date addition is great to have in there too for a convenient way to filter out those which have/haven't been modified over X period of time.

Just to confirm though, does the modification date get updated every time the XML Importer runs for every item of the external feeds? I presume that is the only way to differentiate between what is in the current system and what has now been removed from the external feeds?

Taking this one step further I'd like to take advantage of @moonoo2's extra delegates and send an email to myself if there is a problem pulling in external feed so that I know it needs looking in to. Would I write a new extension latching on to that delegate? Are there any extensions already existing which currently does this job so that I don't look to re-invent the wheel? If not, any pointers on the best way to approach it? Which email libs can be used etc.

@ijy happy days ;) extension that hooks into those deleagates is the way to go! Check out a proof of concept I put together: xmlimport_notifier.github repo Enjoy. :) p.s not sure about the date altering every import.

@ijy the emailing part is incomplete in the code at present.. so you know..

Legend! Pre-emptive coding at it's finest. :) Thanks for that @moonoo2, that's a big help in having a starting point to work from. Much appreciated.

I'll look to create two new extensions; one for email notifications on success/failure, and one for deletion on successful update.

Out of interest, what is an "AJAX Page" in reference to Symphony and Nils' solution?

Keep us posted on your progress.. Would be good to see how you implement the extensions.

With regards to @Nils' AJAX page implementation: It's a content page inside an extension (pre exisiting/or new one created by Nils) to allow an AJAX function to trigger the script contained on that Symphony content.import.php page that gets placed in the extensions content folder. (I believe).. not done much with the AjaxPage class yet.

Well the notifications extension is now in and working perfectly. The delete removed entries extension is in place and sort of works... but not properly. I'll dig back in again later on. I'm just wondering if that's the reason @Nils made an AJAX Page inside of an extension. AJAX Page is completely new to me too.