Add multiple saves / change event proportions (#149)

* feat: Add multiple rows per block / profile

Adds the ability to fake how many times a course was published or a user
profile saved. In the real world these are drivers of table size and we
should exercise the "most recent block" / "most recent profile" code
more.

* refactor: Don't enforce backend type on S3 load

When doing S3 CSV generation and loads it required a lot of unnecessary
switching back and forth between types in the config. Now as long as the
correct keys exist it won't complain.

* refactor: Change event proportions

Based on feedback from live sites, we've recalculated the proportions of
event types. This takes an average of 4 different live sites to find a
reasonable middle ground. Site configuration and course design play
large roles in these distributions in real life.

Author: bmtcril

default_config.yaml

@@ -19,7 +19,6 @@ csv_load_from_s3_after: false
 # s3_key:
 # s3_secret:
 
-
 # Ralph / ClickHouse backend configuration
 # ########################################
 # backend: ralph_clickhouse
@@ -53,6 +52,10 @@ course_length_days: 120
 num_organizations: 3
 num_actors: 10
 
+# This replicates users updating their profiles several times, creating
+# more rows
+num_actor_profile_changes: 5
+
 # How many of each size course to create. The sum of these is the total number
 # of courses created for the test.
 num_course_sizes:
@@ -61,6 +64,11 @@ num_course_sizes:
   large: 1
   huge: 1
 
+# How many times each course will be "published", this creates a more realistic
+# distribution of course blocks where each course can be published dozens or
+# hundreds of times while it is being developed.
+num_course_publishes: 100
+
 # Course size configurations, how many of each type of object are created for
 # each course of this size. "actors" must be less than or equal to "num_actors".
 # For a course of this size to be created it needs to exist both here and in

example_configs/clickhouse_example.yaml

@@ -27,6 +27,10 @@ course_length_days: 120
 num_organizations: 3
 num_actors: 10
 
+# This replicates users updating their profiles several times, creating
+# more rows
+num_actor_profile_changes: 5
+
 # How many of each size course to create. The sum of these is the total number
 # of courses created for the test.
 num_course_sizes:
@@ -35,6 +39,11 @@ num_course_sizes:
   large: 1
   huge: 1
 
+# How many times each course will be "published", this creates a more realistic
+# distribution of course blocks where each course can be published dozens or
+# hundreds of times while it is being developed.
+num_course_publishes: 100
+
 # Course size configurations, how many of each type of object are created for
 # each course of this size. "actors" must be less than or equal to "num_actors".
 # For a course of this size to be created it needs to exist both here and in

example_configs/large_csv_load.yaml

@@ -39,13 +39,22 @@ course_length_days: 120
 num_organizations: 10
 num_actors: 2000000
 
+# This replicates users updating their profiles several times, creating
+# more rows
+num_actor_profile_changes: 5
+
 # The sum of these is the total number of courses created for the test
 num_course_sizes:
   small: 100
   medium: 400
   large: 480
   huge: 120
 
+# How many times each course will be "published", this creates a more realistic
+# distribution of course blocks where each course can be published dozens or
+# hundreds of times while it is being developed.
+num_course_publishes: 100
+
 course_size_makeup:
   small:
     actors: 30

example_configs/local_csv.yaml

@@ -20,6 +20,10 @@ course_length_days: 120
 num_organizations: 3
 num_actors: 10
 
+# This replicates users updating their profiles several times, creating
+# more rows
+num_actor_profile_changes: 5
+
 # How many of each size course to create. The sum of these is the total number
 # of courses created for the test.
 num_course_sizes:
@@ -28,6 +32,11 @@ num_course_sizes:
   large: 1
   huge: 1
 
+# How many times each course will be "published", this creates a more realistic
+# distribution of course blocks where each course can be published dozens or
+# hundreds of times while it is being developed.
+num_course_publishes: 100
+
 # Course size configurations, how many of each type of object are created for
 # each course of this size. "actors" must be less than or equal to "num_actors".
 # For a course of this size to be created it needs to exist both here and in

example_configs/ralph_clickhouse_example.yaml

@@ -27,6 +27,10 @@ course_length_days: 120
 num_organizations: 3
 num_actors: 10
 
+# This replicates users updating their profiles several times, creating
+# more rows
+num_actor_profile_changes: 5
+
 # How many of each size course to create. The sum of these is the total number
 # of courses created for the test.
 num_course_sizes:
@@ -35,6 +39,11 @@ num_course_sizes:
   large: 1
   huge: 1
 
+# How many times each course will be "published", this creates a more realistic
+# distribution of course blocks where each course can be published dozens or
+# hundreds of times while it is being developed.
+num_course_publishes: 100
+
 # Course size configurations, how many of each type of object are created for
 # each course of this size. "actors" must be less than or equal to "num_actors".
 # For a course of this size to be created it needs to exist both here and in

example_configs/s3_csv.yaml

@@ -24,6 +24,10 @@ course_length_days: 120
 num_organizations: 3
 num_actors: 10
 
+# This replicates users updating their profiles several times, creating
+# more rows
+num_actor_profile_changes: 5
+
 # How many of each size course to create. The sum of these is the total number
 # of courses created for the test.
 num_course_sizes:
@@ -32,6 +36,11 @@ num_course_sizes:
   large: 1
   huge: 1
 
+# How many times each course will be "published", this creates a more realistic
+# distribution of course blocks where each course can be published dozens or
+# hundreds of times while it is being developed.
+num_course_publishes: 100
+
 # Course size configurations, how many of each type of object are created for
 # each course of this size. "actors" must be less than or equal to "num_actors".
 # For a course of this size to be created it needs to exist both here and in

example_configs/s3_csv_clickhouse.yaml

@@ -39,6 +39,10 @@ course_length_days: 120
 num_organizations: 3
 num_actors: 10
 
+# This replicates users updating their profiles several times, creating
+# more rows
+num_actor_profile_changes: 5
+
 # How many of each size course to create. The sum of these is the total number
 # of courses created for the test.
 num_course_sizes:
@@ -47,6 +51,11 @@ num_course_sizes:
   large: 1
   huge: 1
 
+# How many times each course will be "published", this creates a more realistic
+# distribution of course blocks where each course can be published dozens or
+# hundreds of times while it is being developed.
+num_course_publishes: 100
+
 # Course size configurations, how many of each type of object are created for
 # each course of this size. "actors" must be less than or equal to "num_actors".
 # For a course of this size to be created it needs to exist both here and in

xapi_db_load/__init__.py

@@ -2,4 +2,4 @@
 Scripts to generate fake xAPI data against various backends.
 """
 
-__version__ = "1.4.1"
+__version__ = "1.5.0"