Skip to content

Commit fb1382b

Browse files
bmtcrilbmtcril
committed
feat: First stab at urwid UI & async
- Adds async backends using asyncio for ClickHouse direct, Ralph, CSV, and CHDB/S3 using a new task management system - Adds an urwid based UI for displaying the load state as the various loads happen - Refactors several configuration variables
1 parent 077408d commit fb1382b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

50 files changed

+3925
-1325
lines changed

‎README.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,9 +87,9 @@ test::
8787
log_dir: logs
8888

8989
# xAPI statements will be generated in batches, the total number of
90-
# statements is ``num_batches * batch_size``. The batch size is the number
90+
# statements is ``num_xapi_batches * batch_size``. The batch size is the number
9191
# of statements sent to the backend (Ralph POST, ClickHouse insert, etc.)
92-
num_batches: 3
92+
num_xapi_batches: 3
9393
batch_size: 100
9494

9595
# Overall start and end date for the entire run. All xAPI statements

‎default_config.yaml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# CSV backend configuration
22
# #########################
3-
backend: csv_file
3+
backend: csv
44
# This can be anything smart_open can handle, a local directory or
55
# an S3 bucket, etc., but importing to ClickHouse only supports S3 right now
66
# https://pypi.org/project/smart-open/
@@ -37,8 +37,13 @@ csv_load_from_s3_after: false
3737

3838
# Run options
3939
log_dir: logs
40-
num_batches: 3
41-
batch_size: 100
40+
num_xapi_batches: 300
41+
batch_size: 1000
42+
43+
# This number is used for each QueueBackend that use workers, so the number of threads if
44+
# multiplicative. Generally this performs best less than 10, as more threads will cost more
45+
# in context switching than they save.
46+
num_workers: 4
4247

4348
# Overall start and end date for the entire run
4449
start_date: 2014-01-01

‎example_configs/clickhouse_example.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ s3_secret: ...
1212

1313
# Run options
1414
log_dir: logs
15-
num_batches: 3
15+
num_xapi_batches: 3
1616
batch_size: 100
1717

1818
# Overall start and end date for the entire run

‎example_configs/large_csv_load.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
# 1200 courses
77
# 10 orgs
88

9-
backend: csv_file
9+
backend: csv
1010
# The next two lines point to the same place, but each needs a different URL
1111
# format.
1212
csv_output_destination: s3://openedx-aspects-loadtest/logs/large_test/
@@ -24,7 +24,7 @@ s3_secret: ...
2424

2525
# Run options
2626
log_dir: logs
27-
num_batches: 100000
27+
num_xapi_batches: 100000
2828
batch_size: 10000
2929

3030
# Overall start and end date for the entire run

‎example_configs/local_csv.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
# CSV backend configuration
22
# #########################
3-
backend: csv_file
3+
backend: csv
44
csv_output_destination: logs
55

66
# Run options
77
log_dir: logs
8-
num_batches: 3
8+
num_xapi_batches: 3
99
batch_size: 100
1010

1111
# Overall start and end date for the entire run

‎example_configs/ralph_clickhouse_example.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Ralph / ClickHouse backend configuration
22
# ########################################
3-
backend: ralph_clickhouse
3+
backend: ralph
44
db_host: localhost
55
db_port: null
66
db_name: xapi
@@ -12,7 +12,7 @@ lrs_password: test
1212

1313
# Run options
1414
log_dir: logs
15-
num_batches: 3
15+
num_xapi_batches: 3
1616
batch_size: 100
1717

1818
# Overall start and end date for the entire run

‎example_configs/s3_csv.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# CSV backend configuration
22
# #########################
3-
backend: csv_file
3+
backend: csv
44
# This can be anything smart_open can handle, a local directory or
55
# an S3 bucket, etc., but importing to ClickHouse only supports S3 right now
66
# https://pypi.org/project/smart-open/
@@ -9,7 +9,7 @@ csv_load_from_s3_after: false
99

1010
# Run options
1111
log_dir: logs
12-
num_batches: 3
12+
num_xapi_batches: 3
1313
batch_size: 100
1414

1515
# Overall start and end date for the entire run

‎example_configs/s3_csv_clickhouse.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# CSV backend configuration
22
# #########################
3-
backend: csv_file
3+
backend: csv
44
# This can be anything smart_open can handle, a local directory or
55
# an S3 bucket, etc., but importing to ClickHouse only supports S3 right now
66
# https://pypi.org/project/smart-open/
@@ -24,7 +24,7 @@ s3_secret: ...
2424

2525
# Run options
2626
log_dir: logs
27-
num_batches: 3
27+
num_xapi_batches: 3
2828
batch_size: 100
2929

3030
# Overall start and end date for the entire run

‎requirements/base.in‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
11
# Core requirements for using this application
22
-c constraints.txt
33

4-
click
5-
clickhouse-connect>=0.5,<0.7
6-
pyyaml
7-
requests
8-
smart_open[s3]
4+
chdb # Embedded ClickHouse client
5+
click # Command line interface library
6+
clickhouse-connect # ClickHouse client
7+
pyyaml # YAML parser
8+
requests # HTTP library
9+
smart_open[s3] # Consistent file handing between local and object storage
10+
urwid # Console UI library
11+
uvloop # Improved performance with asyncio

‎requirements/base.txt‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
#
2-
# This file is autogenerated by pip-compile with Python 3.11
2+
# This file is autogenerated by pip-compile with Python 3.12
33
# by the following command:
44
#
55
# make upgrade
66
#
7+
78
boto3==1.37.37
89
# via smart-open
910
botocore==1.37.37
@@ -16,9 +17,11 @@ certifi==2025.1.31
1617
# requests
1718
charset-normalizer==3.4.1
1819
# via requests
20+
chdb==3.1.2
21+
# via -r requirements/base.in
1922
click==8.1.8
2023
# via -r requirements/base.in
21-
clickhouse-connect==0.6.23
24+
clickhouse-connect==0.8.17
2225
# via -r requirements/base.in
2326
idna==3.10
2427
# via requests
@@ -28,10 +31,20 @@ jmespath==1.0.1
2831
# botocore
2932
lz4==4.4.4
3033
# via clickhouse-connect
34+
numpy==2.2.4
35+
# via pandas
36+
pandas==2.2.3
37+
# via chdb
38+
pyarrow==19.0.1
39+
# via chdb
3140
python-dateutil==2.9.0.post0
32-
# via botocore
41+
# via
42+
# botocore
43+
# pandas
3344
pytz==2025.2
34-
# via clickhouse-connect
45+
# via
46+
# clickhouse-connect
47+
# pandas
3548
pyyaml==6.0.2
3649
# via -r requirements/base.in
3750
requests==2.32.3
@@ -42,12 +55,22 @@ six==1.17.0
4255
# via python-dateutil
4356
smart-open[s3]==7.1.0
4457
# via -r requirements/base.in
58+
typing-extensions==4.13.2
59+
# via urwid
60+
tzdata==2025.2
61+
# via pandas
4562
urllib3==2.2.3
4663
# via
4764
# -c https://raw.githubusercontent.com/edx/edx-lint/master/edx_lint/files/common_constraints.txt
4865
# botocore
4966
# clickhouse-connect
5067
# requests
68+
urwid==2.6.16
69+
# via -r requirements/base.in
70+
uvloop==0.21.0
71+
# via -r requirements/base.in
72+
wcwidth==0.2.13
73+
# via urwid
5174
wrapt==1.17.2
5275
# via smart-open
5376
zstandard==0.23.0

0 commit comments

Comments
 (0)