Symphony.

A new extension, "ElasticSearch" is now available for download. Comments and feedback can be left here but if you discover any issues, please post it on the issue tracker.

Wow!

Wow, that's all I can say... this look awesome Nick. You are very thorough!

I'm guessing multilingual would be as easy as modifying the mappings file:

<?php
class elasticsearch_articles {

    public function mapData(Array $data, Entry $entry) {
        $json = array();
        // var_dump($data);

        $json['_boost'] = 1;

        if($data['is-published']['value'] !== 'yes') return;

        $json['title'] = $data['title']['value'];
        $json['content'] = $data['content']['value_en'];
<!-- or conditional based on current lang param maybe? -->
        $json['document'] = base64_encode(file_get_contents($data['document']['file']));

        return $json;
    }

}

Wow!

ShamWow!

This is great!

Thank you very much for the very detailed explanations.

I'm guessing multilingual would be as easy as modifying the mappings file:

I had a quick chat with Nick about multilingual search last night. You'd have to create separate ES fields from the multilingual content.

{
    "articles": {
        "properties": {
            "title_en": {
                "type" : "string",
            },
            "title_de": {
                "type" : "string",
            }
        }
    }
}

Then you'd have to fill the ES fields with content from the Sym fields.

<?php
    class elasticsearch_articles {

        public function mapData(Array $data, Entry $entry) {
            $json = array();
            $json['title_en'] = $data['title']['value']['en];
            $json['title_de'] = $data['title']['value']['de'];
            return $json;
        }

    }

Abd you'd have to modify the search datasource to make it language agnostic by e.g. looking for a ?language parameter and only search the corresponding fields by wildcard matching for ES fields like *_de.

Haven't tried it, but will do so in the next couple of days.

Yes! Thank you so much for this! After you gave up on the Search Index I was getting worried I had to use google search again for a project, but this has given me hope!

In the ElasticSearch docs, I have found the possibility to search all tweets by a specific user. Is this possible with this extension, too? If it is not possible by default, how hard would this be to implement?

Thanks again for this great extension - you saved my day (again)!

Fantastic work Nick. It looks very powerful (and quite complex) but, as always, you've put a lot of effort in great documentation, thank you!

Just to be sure: Symphony Elasticsearch will probably not run on shared hosting since it requires Java, right?

I had a quick chat with Nick about multilingual search last night

And this morning I implemented pretty much what you have described. It will go into the repo soon.

In the ElasticSearch docs, I have found the possibility to search all tweets by a specific user. Is this possible with this extension, too? If it is not possible by default, how hard would this be to implement?

Not with this extension, no. This extension aims to couple Symphony and ES, so think purely in terms of sections and fields. If you were importing your tweets into a section named Tweets then of course you could index and search it as any other section.

But once you're running ElasticSearch there's nothing stopping you creating your own index, mappings and search queries outside of the scope of this extension.

Just to be sure: Symphony Elasticsearch will probably not run on shared hosting since it requires Java, right?

Your shared host might have JVM installed, you never know. But you need SSH access to install ElasticSearch itself, which you probably don't get on cheap hosting.

I had a quick chat with Nick about multilingual search last night

And this morning I implemented pretty much what you have described. It will go into the repo soon.

Great. Hope I didn't prevent you from sleeping.

Your documentation looks extremely thorough and helpful, Nick. Thanks! Hopefully I'll be able to try this out with a project later in the year.

I'm wondering if there are any general guidelines on likely server requirements for elasticsearch. I'm guessing it's best on its own server/VPS rather than sharing one with a database or web server? The default minimum/maxium memory settings seem to be 256MB and 1024MB - do you think a 512MB Linode would suffice for use with a few relatively modestly-sized Symphony sites?

The default minimum/maxium memory settings seem to be 256MB and 1024MB - do you think a 512MB Linode would suffice for use with a few relatively modestly-sized Symphony sites

I imagine so, yes. I'm running a 768MB Linode but haven't tested ElasticSearch which a huge amount of data on that box yet. It runs fine. Will know more in the next few weeks.

If you were importing your tweets into a section named Tweets then of course you could index and search it as any other section.

Oh yes sorry, the tweets was just an example I took from the docs. I was really referring to fine control within the search: all the data is inside sections, and I want to search only those entries that are created by user x, or have a tag y. Basically filtering before searching.

I have found this in the docs, which is basically exactly what I am looking for. How hard would it be for me to implement something like this with your extension?

Right. To make the extension as "plug and play" as possible, you'll see in the README that to make a field searchable via the search data source it simply needs to be mapped as a multi_type field, with a "sub field" named symphony_fulltext (e.g. an article Title would be mapped as title.symphony_fulltext). The search DS therefore searches only within fields matching *.symphony_fulltext.

Presently this data source doesn't allow you to add additional filters — its purpose is solely to do a single keyword search across entries.

But the extension uses Elastica, a PHP client for ElasticSearch, which makes it pretty easy to build up the query using PHP objects. If you wanted to search by a user (say a field named Author), you could create a new filter on this field:

$author_filter = new Elastica_Filter_Terms('author');
$author_filter->addTerm('nickdunn');
$query->setFilter($author_filter);

I'm not sure if you can add multiple filters. I presume you can.

You best bet is probably to copy the contents of the search data source into a new one, and play around with building queries yourself.

This could be exposed via a URL syntax such as ?filter[author]=nickdunn, but I am deliberately not implementing this sort of thing, since the scope of ElasticSearch is very broad and I don't want to choose which parts to implement. There are 32 types of ElasticSearch query, and this extension just uses one (query_string). So my official line is... if you want anything more powerful, write the query in the data source yourself. This extension still does the plumbing of getting your entries into the index :-)

Thanks Nick.

Because I intend to use ElasticSearch for more than just basic search, I already thought I had to do some of the dirty work myself. I have not had the time to take a look at the actual code, so thanks for the detailed reply. Really helpful.

No problem. If you need a hand give me a shout. I'm still learning ES too.

I have added the aforementioned multilingual support.

G+Knee+Us :) Thanks Nick.. frikkin awesome!

Nick, I am having quite a bit of trouble getting the symphony_fulltext analyser to work. When I use the sample article.json file for my section, I get the following error: MapperParsingException[Analyzer [symphony_fulltext] not found for field [symphony_fulltext]].

The symphony_fulltext analyser is present in the index.json file, so I am bit at loss what is going on here. Any ideas?

Hmm it sounds like there's a bug when creating the index. Perhaps it did not get created. Are you running the elasticsearch-head plugin, if so, can you confirm the index exists? After a little playing I don't think the index is created the first time the System > Preferences are saved. Try changing the index name and saving again.

If the index was created, using the index.json template, you should be able to test the analyzer:

http://localhost:9200/your_index/_analyze?analyzer=symphony_fulltext&text=lorem+ipsum