Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
5fa3d87
Allow specifying custom index name and mapping function for each models
harish2704 Sep 12, 2021
c9ce051
Document how to customize search indexing
harish2704 Sep 12, 2021
8107b10
Replace utils with collection service
bidoubiwa Oct 19, 2021
0f24ad9
Add config example in playground
bidoubiwa Oct 19, 2021
1de813d
Change naming to explicit collection or indexUid
bidoubiwa Oct 20, 2021
ca6a1ef
Fix: include lib directory while packing
harish2704 Sep 13, 2021
c7db925
fix records count for composite index
harish2704 Sep 13, 2021
2fba13c
Refactor files for better readability
bidoubiwa Oct 20, 2021
92fc663
Remove unecessary complexity
bidoubiwa Oct 20, 2021
23cac5f
Fix error in api path
bidoubiwa Oct 20, 2021
4e8df5a
rename credentials function
bidoubiwa Oct 20, 2021
ed853d5
Separate services and contextes
bidoubiwa Oct 20, 2021
21a72ce
Separate components from bootstrapper
bidoubiwa Oct 21, 2021
5416d84
Separate strapi globals from components
bidoubiwa Oct 22, 2021
f79a28c
Update meilisearch js version
bidoubiwa Oct 22, 2021
06bedee
Move connector out of services
bidoubiwa Oct 22, 2021
81c5b6c
Remove services complexity
bidoubiwa Oct 22, 2021
68949d4
Add jsdoc
bidoubiwa Oct 22, 2021
e7fc048
t Add jsdoc to all functions
bidoubiwa Oct 23, 2021
4bad42c
Fix tests
bidoubiwa Oct 25, 2021
ff0428c
Fix bootstrapping
bidoubiwa Oct 25, 2021
435c749
Add missing jsdocs
bidoubiwa Oct 25, 2021
6be063f
Update ci tests to accept reloading
bidoubiwa Oct 25, 2021
9265f04
emove circular symlink
bidoubiwa Oct 25, 2021
d5ee537
Remove redundant return
bidoubiwa Oct 26, 2021
abb4d4e
Fix typos in comment
bidoubiwa Oct 26, 2021
edd2c02
Rewording of confusing function
bidoubiwa Oct 26, 2021
5952023
Add entries transformer before indexation
bidoubiwa Oct 26, 2021
4df946b
Merge pull request #283 from meilisearch/add_composite_indexes
bidoubiwa Oct 27, 2021
8729c5d
Fix flacky tests
bidoubiwa Oct 27, 2021
056436a
Merge #285
bors[bot] Oct 27, 2021
474937d
Add tests
bidoubiwa Oct 27, 2021
b4b4269
Remove node 16 from yaml
bidoubiwa Oct 27, 2021
7921788
Add tests and validator of config
bidoubiwa Oct 28, 2021
f620b41
Fix tests
bidoubiwa Oct 28, 2021
544eeac
Remove validator
bidoubiwa Oct 28, 2021
70b248b
Remove logger
bidoubiwa Oct 28, 2021
8e0f0ff
Add model to transformEntry to be able to sanitize
bidoubiwa Oct 28, 2021
20e5bb2
Remove pretest field from package json
bidoubiwa Oct 28, 2021
ac74643
Change directory of tests
bidoubiwa Oct 28, 2021
40deaf9
Improve documentation
bidoubiwa Oct 28, 2021
c7f2952
Fix tests
bidoubiwa Nov 2, 2021
173a922
Remove tutorial video
bidoubiwa Nov 2, 2021
dec7595
Remove tutorial appearance in admin panel
bidoubiwa Nov 2, 2021
e6cf1e5
Merge branch 'next_v0.4.0_release' into transform_data
bidoubiwa Nov 2, 2021
a9cf2e5
Remove unecessary functions lost in code
bidoubiwa Nov 2, 2021
9eb1865
Add packages in dependabot ignore
bidoubiwa Nov 2, 2021
56f3b17
#296 - Bulk Document Indexing - Change batch size so that sqlite does…
jbelke Nov 3, 2021
06bd08e
Update .github/dependabot.yml
jbelke Nov 3, 2021
5a97ade
Improve formating in readme
bidoubiwa Nov 4, 2021
c9f17be
Remove comments in tests
bidoubiwa Nov 4, 2021
5b5937e
Remove
jbelke Nov 8, 2021
1e9de51
Create dependabot.yml
jbelke Nov 8, 2021
a7b2c12
Merge pull request #301 from jbelke/next_v0.4.0_release
bidoubiwa Nov 8, 2021
4135de8
Merge pull request #287 from meilisearch/transform_data
bidoubiwa Nov 10, 2021
cc9ceda
Fix batch size test
bidoubiwa Nov 10, 2021
9bcad97
Support for custom index name
bidoubiwa Nov 10, 2021
940dbd4
Improve readme
bidoubiwa Nov 10, 2021
2364cab
Add custom index name tests
bidoubiwa Nov 10, 2021
10b4c0c
Update buffet
bidoubiwa Nov 10, 2021
e2edbd6
Fix style
bidoubiwa Nov 10, 2021
525eb68
Improve warn log message
bidoubiwa Nov 10, 2021
ef55679
Index multiple collection on the same index
bidoubiwa Nov 10, 2021
ab393c7
Remove unecessary comment
bidoubiwa Nov 10, 2021
57dcf81
Improve readme explaination
bidoubiwa Nov 10, 2021
5e64ed5
Remove dead link
bidoubiwa Nov 10, 2021
7e39cca
Merge #306
meili-bors[bot] Nov 11, 2021
5e358c6
Merge
bidoubiwa Nov 15, 2021
0147f21
Merge branch 'next_v0.4.0_release' of github.com:meilisearch/strapi-p…
bidoubiwa Nov 15, 2021
f2eb661
Fix tests
bidoubiwa Nov 15, 2021
bf5eda2
Merge branch 'next_v0.4.0_release' of github.com:meilisearch/strapi-p…
bidoubiwa Nov 15, 2021
3d844de
Fix bootstrap fail on composite
bidoubiwa Nov 15, 2021
212d742
Separate utils into support in cypress tests
bidoubiwa Nov 15, 2021
0a2db2c
Update tests
bidoubiwa Nov 16, 2021
63ec261
Fix tests and lifecycles
bidoubiwa Nov 16, 2021
0ac77fb
Improve documentation
bidoubiwa Nov 16, 2021
d31908c
Merge pull request #312 from meilisearch/refactor_cypress
bidoubiwa Nov 16, 2021
bb23bbc
Compatible with MeiliSearch v0.24
bidoubiwa Nov 16, 2021
fd361b0
Fix lint
bidoubiwa Nov 16, 2021
1eed5c4
Add protected routes in MeiliSearch
bidoubiwa Nov 17, 2021
1550692
Apply review changes
bidoubiwa Nov 23, 2021
5dc2891
Fix store fetcher
bidoubiwa Nov 23, 2021
ea04be4
Update naming of functions
bidoubiwa Nov 23, 2021
94c53c7
Merge #309
meili-bors[bot] Nov 23, 2021
ef328ad
Fix merge conflicts
bidoubiwa Nov 23, 2021
3fe6ce4
Merge composite index
bidoubiwa Nov 23, 2021
9d403c9
Add protected routes in MeiliSearch
bidoubiwa Nov 17, 2021
ca2faaa
Restrict plugin to only super admin users
bidoubiwa Nov 23, 2021
b9fbc55
Resolve conflicts
bidoubiwa Nov 23, 2021
58bd0e5
Update screenshots in Readme
bidoubiwa Nov 23, 2021
79be703
Make it clear that the plugin uses a private meilisearch api key
bidoubiwa Nov 23, 2021
0588378
Merge #318
meili-bors[bot] Nov 25, 2021
0ccc389
Merge #316
meili-bors[bot] Nov 25, 2021
6302910
Merge pull request #320 from meilisearch/emphasis_on_api_keys
bidoubiwa Nov 25, 2021
eff51a0
Add tests to confirm empty collection addition is working
bidoubiwa Nov 26, 2021
2e500de
Fix tests
bidoubiwa Nov 26, 2021
330ef45
Fix wrong restaurant being deleted in collection
bidoubiwa Nov 26, 2021
0d0841a
Fix typos
bidoubiwa Nov 26, 2021
29ecd7e
Merge pull request #321 from meilisearch/fix_empty_collection
bidoubiwa Nov 26, 2021
c0c04a5
Raise document addition errors
bidoubiwa Nov 26, 2021
3ed3fc2
Raise error when entries fail to be indexed
bidoubiwa Nov 26, 2021
1d63ba2
Rollback database
bidoubiwa Nov 26, 2021
943a5f0
Improve comments
bidoubiwa Nov 26, 2021
3f7785d
Improve comments
bidoubiwa Nov 26, 2021
cbe0340
Update connectors/meilisearch/index.js
bidoubiwa Nov 29, 2021
a396b14
Update connectors/meilisearch/index.js
bidoubiwa Nov 29, 2021
706f2c5
Add missing jsdoc params
bidoubiwa Nov 29, 2021
2f21ac9
Update obsolute jsdoc returns
bidoubiwa Nov 29, 2021
7b3c4a5
Raise document errors (#322)
bidoubiwa Nov 29, 2021
844bb3b
Remove action on row click instead of checkbox click (#323)
bidoubiwa Nov 29, 2021
23f8b42
Merge #322
meili-bors[bot] Nov 29, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ module.exports = {
commonjs: true,
es2020: true,
node: true,
jest: true,
},
globals: {
strapi: true,
Expand Down
114 changes: 76 additions & 38 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,54 +10,29 @@ on:
- main

jobs:
cypress-run:
integration_tests:
runs-on: ubuntu-latest
container: cypress/browsers:node12.18.3-chrome87-ff82
services:
meilisearch:
image: getmeili/meilisearch:latest
env:
MEILI_MASTER_KEY: "masterKey"
MEILI_NO_ANALYTICS: "true"
ports:
- "7700:7700"
strategy:
matrix:
# Strapi is not yet compatible with node 16
node: ["12", "14"]
name: integration-tests (Node.js ${{ matrix.node }})
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Cache dependencies
uses: actions/cache@v2
with:
path: |
./node_modules
key: ${{ hashFiles('yarn.lock') }}
- uses: actions/checkout@v2
- name: Setup node
uses: actions/setup-node@v2
with:
node-version: "12.x"
node-version: ${{ matrix.node }}
- name: Cache dependencies
uses: actions/cache@v2
with:
path: |
./playground/node_modules
key: ${{ hashFiles('playground/yarn.lock') }}
./node_modules
key: ${{ hashFiles('yarn.lock') }}
- name: Install dependencies
run: yarn --dev && yarn --cwd ./playground
- name: Test develop no-reload
uses: cypress-io/github-action@v2
with:
build: yarn playground:build
start: yarn playground:ci
env: env=ci
- uses: actions/upload-artifact@v2
if: failure()
with:
name: cypress-screenshots
path: cypress/screenshots
- uses: actions/upload-artifact@v2
if: failure()
with:
name: cypress-videos
path: cypress/videos
run: yarn
- name: Launch tests
run: yarn test

playground_build_tests:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -105,3 +80,66 @@ jobs:
uses: ibiqlik/action-yamllint@v3
with:
config_file: .yamllint.yml
cypress-run:
runs-on: ubuntu-latest
container: cypress/browsers:node12.18.3-chrome87-ff82
services:
meilisearch:
image: getmeili/meilisearch:latest
env:
MEILI_MASTER_KEY: "masterKey"
MEILI_NO_ANALYTICS: "true"
ports:
- "7700:7700"
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Cache dependencies
uses: actions/cache@v2
with:
path: |
./node_modules
key: ${{ hashFiles('yarn.lock') }}
- name: Setup node
uses: actions/setup-node@v2
with:
node-version: "12.x"
- name: Cache dependencies
uses: actions/cache@v2
with:
path: |
./playground/node_modules
key: ${{ hashFiles('playground/yarn.lock') }}
- name: Install dependencies
run: yarn --dev && yarn --cwd ./playground
- name: Remove plugin symlink
run: rm ./playground/plugins/meilisearch
- name: Move plugin inside playground
# Since the plugin is located at the root of the project but a symlink links
# to it in ./playground/plugins/meilisearch it causes a circular
# chaining problem.
# Now that we removed the symlink (see previous step), we need to move our plugin in the
# plugin directory of the playground.
run: mkdir ./playground/plugins/meilisearch &&
mv admin ./playground/plugins/meilisearch &&
mv controllers ./playground/plugins/meilisearch &&
mv connectors ./playground/plugins/meilisearch &&
mv config ./playground/plugins/meilisearch &&
mv services ./playground/plugins/meilisearch &&
cp package.json ./playground/plugins/meilisearch
- name: Test with reloads activated
uses: cypress-io/github-action@v2
with:
build: yarn playground:build
start: yarn playground:ci
env: env=ci
- uses: actions/upload-artifact@v2
if: failure()
with:
name: cypress-screenshots
path: cypress/screenshots
- uses: actions/upload-artifact@v2
if: failure()
with:
name: cypress-videos
path: cypress/videos
7 changes: 0 additions & 7 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,12 @@ Icon
.Trashes
._*


############################
# Linux
############################

*~


############################
# Windows
############################
Expand All @@ -42,7 +40,6 @@ $RECYCLE.BIN/
*.msm
*.msp


############################
# Packages
############################
Expand Down Expand Up @@ -71,7 +68,6 @@ $RECYCLE.BIN/
*.out
*.pid


############################
# Logs and databases
############################
Expand All @@ -82,7 +78,6 @@ $RECYCLE.BIN/
*.sqlite
*.sqlite3


############################
# Misc.
############################
Expand Down Expand Up @@ -129,7 +124,5 @@ build
############################
cypress/screenshots
cypress/videos
cypress/support
cypress/plugins
cypress/fixtures

8 changes: 5 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,12 @@ Each PR should pass the tests and the linter to be accepted.
docker pull getmeili/meilisearch:latest # Fetch the latest version of MeiliSearch image from Docker Hub
docker run -p 7700:7700 getmeili/meilisearch:latest ./meilisearch --master-key=masterKey --no-analytics=true

# Tests the project
# Integration tests
yarn test
# Tests the project in watch/open mode
yarn test:watch
# Tests the admin page in a browser
yarn test:e2e
# Tests the admin page in a browser in watch mode
yarn test:e2e:watch
# Linter
yarn style
# Linter with fixing
Expand Down
141 changes: 135 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ docker run -it --rm -p 7700:7700 getmeili/meilisearch:latest ./meilisearch --mas

If you don't have a running Strapi project yet, you can either launch the [playground present in this project](#-run-the-playground) or [create a Strapi project](https://strapi.io/documentation/developer-docs/latest/getting-started/quick-start.html).

We recommend adding your collections in development mode to allow the server reloads needed to apply hooks.
We recommend adding your collections in development mode to allow the server reloads needed to apply a listener to the collections.

```bash
strapi develop
Expand All @@ -103,6 +103,12 @@ On the left-navbar, `MeiliSearch` appears under the `PLUGINS` category. If it do
### 🤫 Add Credentials <!-- omit in toc -->

First, you need to configure credentials via the strapi config, or on the plugin page.
The credentials are composed of:
- The `host`: The url to your running MeiliSearch instance.
- The `api_key`: The `master` or `private` key as the plugin requires administration permission on MeiliSearch.[More about permissions here](https://docs.meilisearch.com/reference/features/authentication.html).

⚠️ The `master` or `private` key should never be used to `search` on your front end. For searching, use the `public` key available on [the `key` route](https://docs.meilisearch.com/reference/api/keys.html#get-keys).


#### Using the plugin page

Expand All @@ -126,7 +132,7 @@ module.exports = () => ({
meilisearch: {
// Your meili host
host: "http://localhost:7700",
// Your master key
// Your master key or private key
api_key: "masterKey",
}
//...
Expand All @@ -147,7 +153,7 @@ We will use, as **example**, the collections provided by Strapi's quickstart.
On your plugin homepage, you should have two collections appearing: `restaurant` and `category`.

<p align="center">
<img src="./assets/collections_indexed.png" alt="Indexed collections need a reload" width="600"/>
<img src="./assets/restaurant_indexed.png" alt="Indexed collections need a reload" width="600"/>
</p>

By clicking on the left checkbox, the collection is automatically indexed in MeiliSearch. For example, if you click on the `restaurant` checkbox, all your restaurants are now available in MeiliSearch. We will see in [start searching](#-start-searching) how to try it out.
Expand All @@ -158,9 +164,131 @@ Hooks are listeners that update MeiliSearch each time you add/update/delete an e
To activate them, you will have to reload the server. If you are in develop mode, click on the red `Reload Server` button. If not, reload the server manually!

<p align="center">
<img src="./assets/no_reload_needed.png" alt="Indexed collections are hooked" width="600"/>
<img src="./assets/restaurant_listener.png" alt="Collections listened" width="600"/>
</p>

### Customizing search indexing

#### Custom Index Name

By default, when indexing a collection in MeiliSearch the index in MeiliSearch has the same name as the collection. This behavior can be changed by setting the `indexName` property in the model file of the related collection.

**Example:**

In the following example, the model `restaurant` index in MeiliSearch is called `my_restaurant` instead of the default `restaurant`.

```js
// api/restaurant/models/restaurant.js

module.exports = {
meilisearch: {
indexName: "my_restaurant"
}
}
```

Examples can be found [this directory](./resources/custom-index-name).

### Composite Index

It is possible to bind multiple collections to the same index. They all have to share the same `indexName`.

For example if `shoes` and `shirts` should be bind to the same index, they should have the same `indexName` in their model setting:

```js
// api/shoes/models/shoes.js

module.exports = {
meilisearch: {
indexName: "product"
}
}
```

```js
// api/shirts/models/shirts.js

module.exports = {
meilisearch: {
indexName: "product"
}
}
```

Now, on each entry addition from both `shoes` and `shirts` the entry is added in the `product` index of MeiliSearch.

Nonetheless, it is not possible to know how many entries from each collection is added to MeiliSearch.

For example, given two collections:
- `Shoes`: with 300 entries and an `indexName` set to `product`
- `Shirts`: 200 entries and an `indexName` set to `product`

The index `product` has both the entries of shoes and shirts. If the index `product` has `350` documents in MeiliSearch, it is not possible to know how many of them are from `shoes` or `shirts`.


#### Transform sent data

By default, the plugin sent the data the way it is stored in your Strapi collection. It is possible to remove or transform fields before sending your entries to MeiliSearch.

Create the alteration function `transformEntry` in your Collection's model. Before sending the data to MeiliSearch, every entry passes through this function where the alteration is applied.

You can find a lot of examples in [this directory](./resources/entries-transformers).

**Example**

To remove all private fields and relations from entries before indexing them into MeiliSearch, use [`sanitizeEntity`](https://strapi.io/documentation/developer-docs/latest/development/backend-customization.html#controllers) in the `transFormEntry` function.

```js
// api/restaurant/models/restaurant.js
const { sanitizeEntity } = require('strapi-utils')

module.exports = {
meilisearch: {
transformEntry({ entry, model }) {
return sanitizeEntity(entry, { model })
},
},
}
```

Another example:<br>

The `restaurant` collection has a relation with the `category` collection. Inside a `restaurant` entry the `category` field contains an array of each category in an `object` format: `[{ name: "Brunch" ...}, { name: "Italian ... }]`.

To change that format to an array of category names, add a map function inside the `transformEntry` function.

```js
// api/restaurant/models/restaurant.js

module.exports = {
meilisearch: {
transformEntry(entry, model) {
return {
...entry,
categories: entry.categories.map(cat => cat.name)
};
},
}
}
```

Resulting in `categories` being transformed like this in a `restaurant` entry.
```json
{
"id": 2,
"name": "Squared Pizza",
"categories": [
"Brunch",
"Italian"
],
// other fields
}
```

By transforming the `categories` into an array of names, it is now compatible with the [`filtering` feature](https://docs.meilisearch.com/reference/features/filtering_and_faceted_search.html#configuring-filters) in MeiliSearch.



### 🕵️‍♀️ Start Searching <!-- omit in toc -->

Once you have a collection containing documents indexed in MeiliSearch, you can [start searching](https://docs.meilisearch.com/learn/getting_started/quick_start.html#search).
Expand Down Expand Up @@ -197,7 +325,8 @@ You can have a quick preview with the following code in an HTML file. Create an
const search = instantsearch({
indexName: "restaurant",
searchClient: instantMeiliSearch(
"http://localhost:7700"
"http://localhost:7700",
'publicKey', // Use the public key not the private or master key to search.
)
});

Expand Down Expand Up @@ -237,7 +366,7 @@ import { MeiliSearch } from 'meilisearch'
;(async () => {
const client = new MeiliSearch({
host: 'http://127.0.0.1:7700',
apiKey: 'masterKey',
apiKey: 'publicKey', // Use the public key not the private or master key to search.
})

// An index is where the documents are stored.
Expand Down
Loading