Skip to content

Commit 87f752e

Browse files
bmtcrilbmtcril
committed
feat: First stab at urwid UI & async
- Adds async backends using asyncio for ClickHouse direct, Ralph, CSV, and CHDB/S3 using a new task management system - Adds an urwid based UI for displaying the load state as the various loads happen - Refactors several configuration variables
1 parent 019cf14 commit 87f752e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

51 files changed

+117333
-1307
lines changed

‎README.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,9 +87,9 @@ test::
8787
log_dir: logs
8888

8989
# xAPI statements will be generated in batches, the total number of
90-
# statements is ``num_batches * batch_size``. The batch size is the number
90+
# statements is ``num_xapi_batches * batch_size``. The batch size is the number
9191
# of statements sent to the backend (Ralph POST, ClickHouse insert, etc.)
92-
num_batches: 3
92+
num_xapi_batches: 3
9393
batch_size: 100
9494

9595
# Overall start and end date for the entire run. All xAPI statements

‎default_config.yaml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# CSV backend configuration
22
# #########################
3-
backend: csv_file
3+
backend: csv
44
# This can be anything smart_open can handle, a local directory or
55
# an S3 bucket, etc., but importing to ClickHouse only supports S3 right now
66
# https://pypi.org/project/smart-open/
@@ -37,8 +37,13 @@ csv_load_from_s3_after: false
3737

3838
# Run options
3939
log_dir: logs
40-
num_batches: 3
41-
batch_size: 100
40+
num_xapi_batches: 300
41+
batch_size: 1000
42+
43+
# This number is used for each QueueBackend that use workers, so the number of threads if
44+
# multiplicative. Generally this performs best less than 10, as more threads will cost more
45+
# in context switching than they save.
46+
num_workers: 4
4247

4348
# Overall start and end date for the entire run
4449
start_date: 2014-01-01

‎example_configs/clickhouse_example.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ s3_secret: ...
1212

1313
# Run options
1414
log_dir: logs
15-
num_batches: 3
15+
num_xapi_batches: 3
1616
batch_size: 100
1717

1818
# Overall start and end date for the entire run

‎example_configs/large_csv_load.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
# 1200 courses
77
# 10 orgs
88

9-
backend: csv_file
9+
backend: csv
1010
# The next two lines point to the same place, but each needs a different URL
1111
# format.
1212
csv_output_destination: s3://openedx-aspects-loadtest/logs/large_test/
@@ -24,7 +24,7 @@ s3_secret: ...
2424

2525
# Run options
2626
log_dir: logs
27-
num_batches: 100000
27+
num_xapi_batches: 100000
2828
batch_size: 10000
2929

3030
# Overall start and end date for the entire run

‎example_configs/local_csv.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
# CSV backend configuration
22
# #########################
3-
backend: csv_file
3+
backend: csv
44
csv_output_destination: logs
55

66
# Run options
77
log_dir: logs
8-
num_batches: 3
8+
num_xapi_batches: 3
99
batch_size: 100
1010

1111
# Overall start and end date for the entire run

‎example_configs/ralph_clickhouse_example.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Ralph / ClickHouse backend configuration
22
# ########################################
3-
backend: ralph_clickhouse
3+
backend: ralph
44
db_host: localhost
55
db_port: null
66
db_name: xapi
@@ -12,7 +12,7 @@ lrs_password: test
1212

1313
# Run options
1414
log_dir: logs
15-
num_batches: 3
15+
num_xapi_batches: 3
1616
batch_size: 100
1717

1818
# Overall start and end date for the entire run

‎example_configs/s3_csv.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# CSV backend configuration
22
# #########################
3-
backend: csv_file
3+
backend: csv
44
# This can be anything smart_open can handle, a local directory or
55
# an S3 bucket, etc., but importing to ClickHouse only supports S3 right now
66
# https://pypi.org/project/smart-open/
@@ -9,7 +9,7 @@ csv_load_from_s3_after: false
99

1010
# Run options
1111
log_dir: logs
12-
num_batches: 3
12+
num_xapi_batches: 3
1313
batch_size: 100
1414

1515
# Overall start and end date for the entire run

‎example_configs/s3_csv_clickhouse.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# CSV backend configuration
22
# #########################
3-
backend: csv_file
3+
backend: csv
44
# This can be anything smart_open can handle, a local directory or
55
# an S3 bucket, etc., but importing to ClickHouse only supports S3 right now
66
# https://pypi.org/project/smart-open/
@@ -24,7 +24,7 @@ s3_secret: ...
2424

2525
# Run options
2626
log_dir: logs
27-
num_batches: 3
27+
num_xapi_batches: 3
2828
batch_size: 100
2929

3030
# Overall start and end date for the entire run

‎requirements/base.in‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
11
# Core requirements for using this application
22
-c constraints.txt
33

4-
click
5-
clickhouse-connect>=0.5,<0.7
6-
pyyaml
7-
requests
8-
smart_open[s3]
4+
chdb # Embedded ClickHouse client
5+
click # Command line interface library
6+
clickhouse-connect # ClickHouse client
7+
pyyaml # YAML parser
8+
requests # HTTP library
9+
smart_open[s3] # Consistent file handing between local and object storage
10+
urwid # Console UI library
11+
uvloop # Improved performance with asyncio

‎requirements/base.txt‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#
2-
# This file is autogenerated by pip-compile with Python 3.11
2+
# This file is autogenerated by pip-compile with Python 3.12
33
# by the following command:
44
#
55
# make upgrade
@@ -16,9 +16,11 @@ certifi==2025.4.26
1616
# requests
1717
charset-normalizer==3.4.2
1818
# via requests
19+
chdb==3.1.2
20+
# via -r requirements/base.in
1921
click==8.1.8
2022
# via -r requirements/base.in
21-
clickhouse-connect==0.6.23
23+
clickhouse-connect==0.8.17
2224
# via -r requirements/base.in
2325
idna==3.10
2426
# via requests
@@ -28,10 +30,20 @@ jmespath==1.0.1
2830
# botocore
2931
lz4==4.4.4
3032
# via clickhouse-connect
33+
numpy==2.2.4
34+
# via pandas
35+
pandas==2.2.3
36+
# via chdb
37+
pyarrow==19.0.1
38+
# via chdb
3139
python-dateutil==2.9.0.post0
32-
# via botocore
40+
# via
41+
# botocore
42+
# pandas
3343
pytz==2025.2
34-
# via clickhouse-connect
44+
# via
45+
# clickhouse-connect
46+
# pandas
3547
pyyaml==6.0.2
3648
# via -r requirements/base.in
3749
requests==2.32.3
@@ -42,12 +54,22 @@ six==1.17.0
4254
# via python-dateutil
4355
smart-open[s3]==7.1.0
4456
# via -r requirements/base.in
57+
typing-extensions==4.13.2
58+
# via urwid
59+
tzdata==2025.2
60+
# via pandas
4561
urllib3==2.2.3
4662
# via
4763
# -c https://raw.githubusercontent.com/edx/edx-lint/master/edx_lint/files/common_constraints.txt
4864
# botocore
4965
# clickhouse-connect
5066
# requests
67+
urwid==2.6.16
68+
# via -r requirements/base.in
69+
uvloop==0.21.0
70+
# via -r requirements/base.in
71+
wcwidth==0.2.13
72+
# via urwid
5173
wrapt==1.17.2
5274
# via smart-open
5375
zstandard==0.23.0

0 commit comments

Comments
 (0)