Skip to content

Commit 000ef7e

Browse files
kennethdyerjtcovanjddubois
authored
DOCSP-35686 Embedded Verifier (#339)
* DOCSP-35686 Embedded Verifier --------- Co-authored-by: Jessica <[email protected]> Co-authored-by: John DuBois <[email protected]>
1 parent ed326da commit 000ef7e

25 files changed

+743
-143
lines changed

‎source/includes/api/requests/start-no-verifier.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
curl localhost:27182/api/v1/start -XPOST \
2+
--data '
3+
{
4+
"source": "cluster0",
5+
"destination": "cluster1",
6+
"verification": {
7+
"enabled": false
8+
}
9+
} '

‎source/includes/api/responses/progress.json‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,27 @@
1515
{
1616
"Source":"cluster0: localhost:27017",
1717
"Destination":"cluster1: localhost:27018"
18+
},
19+
"verification":
20+
{
21+
"source":
22+
{
23+
"estimatedDocumentCount": 42,
24+
"hashedDocumentCount": 42,
25+
"lagTimeSeconds": 2,
26+
"totalCollectionCount": 42,
27+
"scannedCollectionCount": 10,
28+
"phase": "stream hashing"
29+
30+
},
31+
"destination": {
32+
"estimatedDocumentCount": 42,
33+
"hashedDocumentCount": 42,
34+
"lagTimeSeconds": 2,
35+
"totalCollectionCount": 42,
36+
"scannedCollectionCount": 10,
37+
"phase": "stream hashing"
38+
}
1839
}
1940
},
2041
"success": true

‎source/includes/api/tables/progress-response.rst‎

Lines changed: 101 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,6 @@
9393
and divide the result by the value of the ``estimatedTotalBytes`` field
9494
. Then, multiply the result by 100.
9595

96-
9796
The value of ``estimatedCopiedBytes`` may be larger than the
9897
value of the ``estimatedTotalBytes`` due to retried operations.
9998
A comparison of ``estimatedTotalBytes`` and
@@ -138,3 +137,104 @@
138137

139138
.. versionadded:: 1.3
140139

140+
* - ``verification``
141+
- document
142+
- Provides information on the phase and progress of
143+
verification checks performed by the embedded verifier.
144+
145+
.. versionadded:: 1.9
146+
147+
* - ``verification.source``
148+
- document
149+
- Provides information on the phase and progress of
150+
verification checks running on the source cluster.
151+
152+
.. versionadded:: 1.9
153+
154+
* - ``verification.source.`` ``estimatedDocumentCount``
155+
- integer
156+
- Estimated number of documents on the source cluster.
157+
158+
.. versionadded:: 1.9
159+
160+
* - ``verification.source.`` ``hashedDocumentCount``
161+
- integer
162+
- Number of documents hashed by the verifier on the
163+
source cluster.
164+
165+
.. versionadded:: 1.9
166+
167+
* - ``verification.source.`` ``lagTimeSeconds``
168+
- integer
169+
- Time in seconds after the last verification check was
170+
performed on the source cluster.
171+
172+
.. versionadded:: 1.9
173+
174+
* - ``verification.source.`` ``phase``
175+
- string
176+
- Current phase of the verification process on the source
177+
cluster.
178+
179+
.. versionadded:: 1.9
180+
181+
* - ``verification.source.`` ``scannedCollectionCount``
182+
- integer
183+
- Number of collections scanned by the embedded verifier on
184+
the source cluster.
185+
186+
.. versionadded:: 1.9
187+
188+
* - ``verification.source.`` ``totalCollectionCount``
189+
- integer
190+
- Number of collections on the source cluster to include in
191+
verification checks.
192+
193+
* - ``verification.destination``
194+
- document
195+
- Provides information on the phase and progress of
196+
verification checks running on the destination cluster.
197+
198+
.. versionadded:: 1.9
199+
200+
* - ``verification.destination.`` ``estimatedDocumentCount``
201+
- integer
202+
- Estimated number of documents on the destination cluster.
203+
204+
.. versionadded:: 1.9
205+
206+
* - ``verification.destination.`` ``hashedDocumentCount``
207+
- integer
208+
- Number of documents hashed by the verifier on the
209+
destination cluster.
210+
211+
.. versionadded:: 1.9
212+
213+
* - ``verification.destination.`` ``lagTimeSeconds``
214+
- integer
215+
- Time in seconds since the last verification check
216+
performed on the destination cluster.
217+
218+
.. versionadded:: 1.9
219+
220+
* - ``verification.destination.`` ``phase``
221+
- string
222+
- Current phase of the verification process on the
223+
destination cluster.
224+
225+
.. versionadded:: 1.9
226+
227+
* - ``verification.destination.`` ``scannedCollectionCount``
228+
- integer
229+
- Number of collections scanned by the embedded verifier on
230+
the destination cluster.
231+
232+
.. versionadded:: 1.9
233+
234+
* - ``verification.destination.`` ``totalCollectionCount``
235+
- integer
236+
- Number of collections on the destination cluster to
237+
include in verification checks.
238+
239+
.. versionadded:: 1.9
240+

‎source/includes/fact-verifier-beta-limitations.rst‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
2+
The verifier isn't compatible with these beta features:
3+
4+
- :ref:`c2c-beta-many-to-one`
5+
- :ref:`c2c-beta-abc-migration`
6+
- :ref:`c2c-beta-document-filtering`
7+
- :ref:`c2c-beta-namespace-remapping`
8+
- :ref:`c2c-beta-destination-data-handling`
9+
- Clusters that run a release of MongoDB that
10+
``mongosync`` doesn't support.
11+

‎source/includes/fact-verifier-beta-unsupport.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
2+
|beta-feature| is not compatible with the :ref:`Embedded Verifier <c2c-embedded-verifier>`.
3+
The :ref:`/start <c2c-api-start>` command returns an error if you enable both the verifier
4+
and |beta-feature|.
5+
6+
For more information, see :ref:`c2c-verification`.
7+

‎source/includes/fact-verifier-buildIndexes.rst‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
2+
If you start sync with verification enabled and ``buildIndexes``
3+
set to ``never``, the migration will fail if ``mongosync`` finds
4+
a TTL collection on the source cluster. This can happen after
5+
you call the ``/start`` endpoint or much later, such as where a
6+
user creates a TTL index on the source cluster while a migration
7+
is in progress.
8+
9+
To sync TTL collections without building indexes on the
10+
destination cluster, you must start sync with the :ref:`verifier
11+
disabled <c2c-disabled-verifier>`.
12+
13+

‎source/includes/fact-verifier-limitations.rst‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
2+
Compatibility
3+
~~~~~~~~~~~~~
4+
5+
The embedded verifier is not available in mongosync 1.8 and
6+
earlier.
7+
8+
For alternative verification methods, see :ref:`c2c-verification`.
9+
10+
Limitations
11+
~~~~~~~~~~~
12+
13+
The embedded verifier has the following limitations:
14+
15+
- The verifier doesn't support sharded clusters. If the
16+
migration includes a sharded cluster, ``mongosync`` disables
17+
the verifier.
18+
19+
- ``mongosync`` stores the verifier state in memory, which can
20+
result in a significant memory overhead. To run the verifier,
21+
``mongosync`` requires approximately 10 GB of memory, plus an
22+
additional 500 MB for every 1 million documents.
23+
24+
- The verifier cannot be resumed. If a user stops or pauses sync
25+
and then starts ``mongosync`` again for any reason, the
26+
verification process restarts from the beginning. This can
27+
cause verification to fall substantially behind the migration.
28+
29+
- The :ref:`/reverse <c2c-api-reverse>` endpoint disables the
30+
verifier. It remains disabled after additional calls to the
31+
``/reverse`` endpoint.
32+
33+
- .. include:: /includes/fact-verifier-buildIndexes
34+
35+
Unsupported Verification Checks
36+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
37+
38+
.. include:: /includes/fact-verifier-unsupported
39+
40+
Unsupported Beta Features
41+
~~~~~~~~~~~~~~~~~~~~~~~~~
42+
43+
.. include:: /includes/fact-verifier-beta-limitations

‎source/includes/fact-verifier-memory.rst‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
2+
Verification requires 10 GB of memory plus an additional 500 MB
3+
for every 1 million documents in the migration.
4+
5+
If the available memory is insufficient, the ``/start`` endpoint
6+
returns an error. If this occurs, to use ``mongosync`` with the
7+
verifier you must first increase the memory of the server and
8+
resume the migration.
9+
10+
If increasing server memory isn't an option, restart
11+
``mongosync`` with the :ref:`verifier disabled
12+
<c2c-disabled-verifier>`.
13+
14+

‎source/includes/fact-verifier-unsupported.rst‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
The verifier doesn't check the following namespaces:
2+
3+
- Capped collections
4+
- Collections with TTL indexes
5+
- Collections that don't use the default collation
6+
- Views
7+
8+
The verifier doesn't check the following collection features:
9+
10+
- Collection metadata
11+
- Indexes
12+
13+
To verify the above data and metadata, script additional checks
14+
for these collections and collection features. For more
15+
information, see :ref:`c2c-verification`.
16+

‎source/includes/opts/acceptDisclaimer.rst‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
2+
Accepts disclaimers for the :ref:`embedded verifier
3+
<c2c-verifier-disclaimer>` and, when using
4+
{+c2c-beta-program-short+}, :ref:`beta features <c2c-beta-program>`.
5+
6+
When the ``mongosync`` process starts without this |opt-term|, the
7+
user is prompted to accept each disclaimer.
8+
9+
This |opt-term| allows users who have already read and accepted
10+
disclaimers to accept the disclaimer and start the ``mongosync``
11+
process.

0 commit comments

Comments
 (0)