Skip to content

Feature/#22338 add airflow #368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Oct 17, 2025
Merged

Feature/#22338 add airflow #368

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
7fff9e9
Add airflow
ViMesTor Sep 11, 2025
fae866b
Add airflow secrets
ViMesTor Sep 12, 2025
4ce7cb1
Modify configuration of airflow
ViMesTor Sep 15, 2025
1accb3c
Add airflow ports, modify airflow config
ViMesTor Sep 17, 2025
753b0dd
Remove variables
ViMesTor Sep 18, 2025
4ef54ab
Modify memory of airflow
ViMesTor Sep 18, 2025
ee9c55b
Merge with master
ViMesTor Sep 25, 2025
b3b185a
Merge branch 'master' into feature/#22338_add_airflow
rgomezborder Oct 9, 2025
caab11d
Passing the management ip to airflow cookbook
rgomezborder Oct 10, 2025
e92d01e
Merge pull request #384 from redBorder/development
rgomezborder Oct 13, 2025
861676e
Split configuration
ViMesTor Oct 15, 2025
8e46267
Fix linter and unify airflow cookbook configuration
rgomezborder Oct 17, 2025
2cbfa52
Merge branch 'master' into feature/#22338_add_airflow
rgomezborder Oct 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions resources/attributes/default.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@
default['redis']['port'] = 26379
default['redis']['sentinel_port'] = 26380

# airflow
default['airflow']['web_port'] = 9191
default['airflow']['scheduler_port'] = 8793

# aerospike
default['aerospike']['port'] = 3000
default['aerospike']['multicast'] = '239.1.99.222'
Expand Down Expand Up @@ -87,6 +91,8 @@
default['redborder']['memory_services']['secor'] = { 'count': 30, 'memory': 0 }
default['redborder']['memory_services']['secor-vault'] = { 'count': 30, 'memory': 0 }
default['redborder']['memory_services']['redis'] = { 'count': 10, 'memory': 0 }
default['redborder']['memory_services']['airflow-scheduler'] = { 'count': 30, 'memory': 0 }
default['redborder']['memory_services']['airflow-webserver'] = { 'count': 30, 'memory': 0 }
default['redborder']['memory_services']['rb-reputation'] = { 'count': 30, 'memory': 0 }

# default attributes for managers_info, it would be rewriten with the cluster config
Expand Down Expand Up @@ -169,6 +175,8 @@
default['redborder']['services']['secor'] = false
default['redborder']['services']['secor-vault'] = false
default['redborder']['services']['redis'] = false
default['redborder']['services']['airflow-scheduler'] = false
default['redborder']['services']['airflow-webserver'] = false
default['redborder']['services']['rb-reputation'] = false

default['redborder']['systemdservices']['aerospike'] = ['aerospike']
Expand Down Expand Up @@ -218,6 +226,8 @@
default['redborder']['systemdservices']['secor'] = ['rb-secor']
default['redborder']['systemdservices']['secor-vault'] = ['rb-secor-vault']
default['redborder']['systemdservices']['redis'] = ['redis']
default['redborder']['systemdservices']['airflow-scheduler'] = ['airflow-scheduler']
default['redborder']['systemdservices']['airflow-webserver'] = ['airflow-webserver']
default['redborder']['systemdservices']['rb-reputation'] = ['rb-reputation']

# Balanced services
Expand Down
1 change: 1 addition & 0 deletions resources/metadata.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,5 @@
depends 'secor'
depends 'redis'
depends 'rb-agents'
depends 'airflow'
depends 'rb-reputation'
22 changes: 22 additions & 0 deletions resources/recipes/configure.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,6 +531,28 @@
action [:add]
end

airflow_secrets = {}

begin
airflow_secrets = data_bag_item('passwords', 'db_airflow').to_hash
rescue
airflow_secrets = {}
end

# Configure Airflow
airflow_config 'Configure airflow' do
if manager_services['airflow-scheduler'] || manager_services['airflow-webserver']
airflow_secrets airflow_secrets
ipaddress_mgt node['ipaddress']
ipaddress_sync node['ipaddress_sync']
airflow_port node['airflow']['web_port']
cdomain node['redborder']['cdomain']
action [:add, :register]
else
action [:remove, :deregister]
end
end

# Configure logstash
split_traffic = false

Expand Down