Search discussion titles too

[tobyzerner]

Currently fulltext search is only performed on post text, not discussion titles. This needs to be fixed.

What needs to be done:

• Work out a strategy to do this. Do we run two queries, one on the posts table and one on the discussions? (How would we rank results then?) Or do we duplicate the discussion title in every post record and perform a joint full text search on the title/content fields?

[amdad]

IMO Search should me more sensitive. Why It doesn't handle 3 signs long phrases? Like "php" or "403" for example? Even worse "php 5", and still not searching...

[tobyzerner]

Limitations of MySQL's FULLTEXT search engine. We're planning to make it possible to use more powerful drivers like ElasticSearch/Sphinx instead. See #509.

[amdad]

ElasticSearch sounds good. Current solution - without comment. :)

[tobyzerner]

@franzliedke Any thoughts on this? How do you include titles in the search in FluxBB?

[kirkbushell]

If we want to use elasticsearch, the search implementation will need to be abstracted, which we have done at work. I can paste in some examples.

[franzliedke]

We built an index upon post creation / editing that stored all words along with their appearances in all posts. Easy to adapt for topic titles...

Search then simply looked for all search terms in the index, found all posts in which they appeared and assembled the resulting list.

[tobyzerner]

@kirkbushell We already have an abstraction, I'm not sure if it'll be sufficient...

@franzliedke That's very interesting, I've never thought about doing that with MySQL. How does it scale though? What about posts with tens of thousands of words? And forums with millions of posts. Isn't it essentially implementing a less efficient version of MySQL's FULLTEXT engine?

[franzliedke]

It does work on the fairly large forums we have (1M+ posts), I think. I believe the approach was originally copied from PhpBB.

The bigger problem is the very limited functionality. Word-based stuff works well for Western languages, but even then matches need to be complete (i.e. "shoot" will not match "shot"). I'll experiment with that stuff at some point, I took a whole class on the topic, that should be worth something ;)

[tobyzerner]

Relevant: http://www.slideshare.net/billkarwin/practical-full-text-search-with-my-sql

Thinking it might be best to go with MySQL ISAM FULLTEXT, mirroring the discussion title on the posts table. Then recommend Sphinx/ElasticSearch for improved results.