|
1 | | -Breaking Up The Monolith |
2 | | ------------------------- |
| 1 | +# Breaking Up The Monolith |
3 | 2 |
|
4 | 3 | Note Takers: Josh Berkus(@jberkus), Aaron Crickenberger (@spiffxp) |
5 | 4 |
|
6 | 5 | Here's what we're covering: build process, issue tracking, logistics. |
7 | 6 |
|
8 | | - Question: are we commited emotionally 100% to splitting the monolith |
9 | | - |
10 | | - oppinion: commited, but need to define what that means (past provides the impetus) |
11 | | - |
12 | | - we've been trying [the monolith] for so long and it's too expensive. |
13 | | - |
14 | | - erictune: as the project matures the core can't keep growing at the same rate |
15 | | - |
16 | | - mattfarina: reason to break it up - developer experience |
17 | | - |
18 | | - difficulty to contribute pushes people away from contributing (hard to step into issue queue, etc) |
19 | | - |
20 | | - clayton: I'm scared by this because kubectl is never going to be simpler, it may be easier to just write a new one |
21 | | - |
22 | | - erictune: ecosystem yes, more contributors... narrowly defined core no |
23 | | - |
24 | | - bburns: we don't do a good job of measuring the cost of this, we love breaking things up because it's "cleaner" but we're not estimating the cost in terms of the human effort and complexity |
25 | | - |
26 | | - lavalamp: respectfully disagree, we _have_ estimated the cost, knew it was going to be painfu but we haven't had a choice |
27 | | - |
28 | | - timstclair: nobody has ever answered the question to be of how we're going to deliver a set of binaries as part of a release, there isn't a cohesive plan for doing this |
29 | | - |
30 | | - timstclair: how are you actually going to get test infra to test all the things |
31 | | - |
32 | | - bburns: how do you actually find the commit that causes a failure in e2e |
33 | | - |
34 | | - lavalmp: have you see the doc I have posted everywhere this comes up? (TODO: link) |
35 | | - |
36 | | - main repo becomes integration point |
37 | | - |
38 | | - binary artifacts produced by other repos |
39 | | - |
40 | | - clayton: kubectl is a very "big" thing, if we said the problem is that kubectl itself is too big, it might be better to focus on something new and novel, it might be better to build a community around that |
41 | | - |
42 | | - dims/matt: can we have a list of four or five items we want to move out in priority |
43 | | - |
44 | | - Kubectl |
45 | | - |
46 | | - Client-Go? (but that's already published separately) (this is way too involved for this session) |
47 | | - |
48 | | - Try to tackle "how do we deal with client-go" elsewhere |
49 | | - |
50 | | - kubeadm is in main repo to catch release train, wants to move out, but it's fairly hard. We tried to move it out, but couldn't and follow the release train. |
51 | | - |
52 | | - clayton: government approach: why build one thing when we can build two for twice the cost? an (extracted) client-go that's all high-perf, a completely rewritten client-go that's human focused and more reusable |
53 | | - |
54 | | - |
55 | | - Question: Do we understand the problem we're trying to solve with the repo split |
56 | | - |
57 | | - Assumption: contributor velocity is correlated with velocity, new contributor experince |
58 | | - |
59 | | - Assumption: "big tangled ball of pasta" is hard to contribute to |
60 | | - |
61 | | - |
62 | | - thockin: our codebase is not organized well (need to make it easier to actually *find* what you want to contribute to) |
63 | | - |
64 | | - one of the things we could do is "move code around" until it looks effectively like there are multiple repos in our repo? |
65 | | - |
66 | | - eg if we centralized all of the kubelet logic in one directory except for one util directory, maybe that would be an improvement |
67 | | - |
68 | | - jberkus: people who work on other large projects say that modularity is the way to go. Not necessarily seperate repos, but a modular architecture. |
69 | | - |
70 | | - thockin: what we've done with github bots alleviates a lot of the pain we've had with github |
71 | | - |
72 | | - dims: two problems |
73 | | - |
74 | | - vendoring in all kinds of sdk's that we don't care about, they get dragged in (e.g. AWS SDK) and if you're only working on eg: openstack related stuff, it's hard to get signoff on getting other stuff in |
75 | | - |
76 | | - if it's in your own repo it's easier to do that than in the main repo |
77 | | - |
78 | | - (guess we missed #2) |
79 | | - |
80 | | - erictune: we've already promised people cloud providers you can write in different ways, extensibility of apis, operators, if we deliver on all this we will have enough modularity to improve user/developer experience |
81 | | - |
82 | | - mattfarina: I think you're right about that |
83 | | - |
84 | | - First thing:we're breaking up cloud providers, it helps let the long tail of cloud providers go out |
85 | | - |
86 | | - what about storage providers, if we break them up there's a clear interface for storage providers |
87 | | - |
88 | | - second thing: here are api/components that are required, required but swappable, optional... how can you do that swapping around of stuff in the monorepo |
89 | | - |
90 | | - |
91 | | - robertbailey: when I proposed this session I thought we had all agreed that breaking the repo was the way to go, but maybe we don't have that conensus |
92 | | - |
93 | | - could we try and figure out who is the decision maker / set of decision makers there and try to come to consensus |
94 | | - |
95 | | - dchen1107: I heard a lot of benefits of the split. So there's a huge cost for release management, as the release manager, I don't know what's in the other repositiory. Clear APIs and interfaces could be built without needing separate repositories. Gave example of Docker and CRI, which got API without moving repos. |
96 | | - |
97 | | - Example of Cloud Foundry, build process which took two weeks. How do we ensure that we're delivering security updates quickly? |
98 | | - |
99 | | - thockin: We can put many things in APIs and cloud providers are a good example of that. Splitting stuff out into multiple repos comes down to the question of: are we a piece of software or a distribution? You'll have to come to my talk to see the rest. |
100 | | - |
101 | | - spiffxp: Increasing our dependancy on integration tools will add overhead and process we don't have now. But I understand that most OSS people expect multiple repos and it's easier for them. Github notifications are awful, having multiple repos would make this better. The bots have improved the automation situation, but not really triaging notifications. How many issues do we have that are based on Github. Maybe we should improve the routing of GH notificaitons instead? |
102 | | - |
103 | | - solly: if we split, we really need to make it easier for tiny repos to plug into our automation and other tools. I shouldn't have to figure out who to ask about getting plugged in, we should have a doc. We need to make sure it's not possible for repos to fall through the cracks like Heapster, which I work on. |
104 | | - |
105 | | - bgrant: We're already in the land of 90 repos. We don't need to debate splitting, we're alread split. We have incubator, and kubernates-client. I think client has helped a lot, we have 7 languages and that'll grow. |
106 | | - |
107 | | - bgrant: the velocity of things in the monorepo are static |
108 | | - |
109 | | - thockin: if issues in the main repo are more than 6 weeks old, nobody looks at it. There's like 400-500 abandoned "network" issues. |
110 | | - |
111 | | - bgrant: we tested gitlab and gerrit, and importing kubernetes failed. We didn't find them that much better. |
112 | | - |
113 | | - spiffxp: we have hundreds of directories with no owners files, which is one of the reasons for excessive notifications. |
114 | | - |
115 | | - bgrant: kubernetes, as a API-driven system, you have to touch the api to do almost anything. we've added mechanisms like CRD to extend the API. We need SDKs to build kube-style APIs. |
116 | | - |
117 | | - dims: I want to focus on the cost we're paying right now. First we had the google KMS provider. Then we had to kick out the KMS provider, and there was a PR to add the gRPC interface, but it didn't go into 1.9. |
118 | | - |
119 | | - thockin: the cloud provider stuff is an obvious breeding ground for new functionality, how and if we should add a whole seperate grpc plugin interface is a seperate question |
120 | | - |
121 | | - jdumars: the vault provider thing was one of the ebtter things that happened, it pushed us at MS to thing about genercizing the solution, it pushed us to think about what's better for the community vs. what's better for the provider |
122 | | - |
123 | | - jdumars: flipside is we need to have a process where people can up with a well accepted / adopted solution, the vault provider thing was one way of doing that |
124 | | - |
125 | | - lavalamp: I tend to think that most extension points are special snowflakes and you can't have a generic process for adding a new extension point |
126 | | - |
127 | | - thockin: wandering back to kubernetes/kubrnetes "main point", looking at staging as "already broken out", are there other ones that we want to break out? |
128 | | - |
129 | | - dims: kubeadm could move out if needed, could move it to staging for sure |
130 | | - |
131 | | - thockin: so what about the rest? eg: kubelet, kube-proxy... do we think that people will concretely get benefits from that? or will that cause more pain |
132 | | - |
133 | | - thockin: we recognize this will slow down things |
134 | | - |
135 | | - lavalamp: there are utility functions that people commonly use and there's no good common place |
136 | | - |
137 | | - lavalamp: for kubectl at least it's sphaghetti code that pulls in lots of packages and makes it difficult to do technically |
138 | | - |
139 | | - thockin: do we think that life would be better at the end of that tunnel, would things be better if kubectl was a different repository, etc. |
140 | | - |
141 | | - timallclair: I'm worried about dependancy management, godeps is already a nightmare and with multiple repos it would be worse. |
142 | | - |
143 | | - luxas: in the kubeadm attack plan, we need to get a release for multiple repos. We need the kubeadm repo to be authoritative, and be able to include it in a build. |
144 | | - |
145 | | - pwittrock: how has "staging" improved development? can we see any of the perceived or hoped-for benefits by looking at staging repos as example use cases? |
146 | | - |
147 | | - lavalamp: getting to the "staging" state and then stopping is because api-machinery was unblocked once we got there |
148 | | - |
149 | | - thockin: the reason I consider "staging" solved, is you have to untangle a lot of the stuff already |
150 | | - |
151 | | - erictune: I would make a plea that we finish some of our started-but-not-finished breakaparts |
152 | | - |
153 | | - |
| 7 | +## Question: are we commited emotionally 100% to splitting the monolith |
| 8 | + |
| 9 | +Opinion: commited, but need to define what that means (past provides the impetus) |
| 10 | + |
| 11 | +We've been trying [the monolith] for so long and it's too expensive. |
| 12 | + |
| 13 | +- erictune: as the project matures the core can't keep growing at the same rate |
| 14 | +- mattfarina: reason to break it up - developer experience,difficulty to contribute pushes people away from contributing (hard to step into issue queue, etc) |
| 15 | +- clayton: I'm scared by this because kubectl is never going to be simpler, it may be easier to just write a new one |
| 16 | +- erictune: ecosystem yes, more contributors... narrowly defined core no |
| 17 | +- bburns: we don't do a good job of measuring the cost of this, we love breaking things up because it's "cleaner" but we're not estimating the cost in terms of the human effort and complexity |
| 18 | +- lavalamp: respectfully disagree, we _have_ estimated the cost, knew it was going to be painfu but we haven't had a choice |
| 19 | +- timstclair: nobody has ever answered the question to be of how we're going to deliver a set of binaries as part of a release, there isn't a cohesive plan for doing this |
| 20 | +- timstclair: how are you actually going to get test infra to test all the things |
| 21 | +- bburns: how do you actually find the commit that causes a failure in e2e |
| 22 | +- lavalamp: have you see the doc I have posted everywhere this comes up? (TODO: link) main repo becomes integration point, binary artifacts produced by other repos |
| 23 | +- clayton: kubectl is a very "big" thing, if we said the problem is that kubectl itself is too big, it might be better to focus on something new and novel, it might be better to build a community around that |
| 24 | +- dims/matt: can we have a list of four or five items we want to move out in priority |
| 25 | + - Kubectl |
| 26 | + - Client-Go? (but that's already published separately) (this is way too involved for this session) |
| 27 | + - Try to tackle "how do we deal with client-go" elsewhere |
| 28 | + - kubeadm is in main repo to catch release train, wants to move out, but it's fairly hard. We tried to move it out, but couldn't and follow the release train. |
| 29 | +- clayton: government approach: why build one thing when we can build two for twice the cost? an (extracted) client-go that's all high-perf, a completely rewritten client-go that's human focused and more reusable |
| 30 | + |
| 31 | +## Question: Do we understand the problem we're trying to solve with the repo split |
| 32 | + |
| 33 | +Assumption: contributor velocity is correlated with velocity, new contributor experince |
| 34 | +Assumption: "big tangled ball of pasta" is hard to contribute to |
| 35 | + |
| 36 | +- thockin: our codebase is not organized well (need to make it easier to actually *find* what you want to contribute to) one of the things we could do is "move code around" until it looks effectively like there are multiple repos in our repo? eg if we centralized all of the kubelet logic in one directory except for one util directory, maybe that would be an improvement |
| 37 | +- jberkus: people who work on other large projects say that modularity is the way to go. Not necessarily seperate repos, but a modular architecture. |
| 38 | +- thockin: what we've done with github bots alleviates a lot of the pain we've had with github |
| 39 | +- dims: two problems |
| 40 | + - vendoring in all kinds of sdk's that we don't care about, they get dragged in (e.g. AWS SDK) and if you're only working on eg: openstack related stuff, it's hard to get signoff on getting other stuff in if it's in your own repo it's easier to do that than in the main repo |
| 41 | + - (guess we missed #2) |
| 42 | +- erictune: we've already promised people cloud providers you can write in different ways, extensibility of apis, operators, if we deliver on all this we will have enough modularity to improve user/developer experience |
| 43 | +- mattfarina: I think you're right about that |
| 44 | +- First thing: we're breaking up cloud providers, it helps let the long tail of cloud providers go out |
| 45 | + - what about storage providers, if we break them up there's a clear interface for storage providers |
| 46 | +- second thing: here are api/components that are required, required but swappable, optional... how can you do that swapping around of stuff in the monorepo |
| 47 | +- robertbailey: when I proposed this session I thought we had all agreed that breaking the repo was the way to go, but maybe we don't have that conensus could we try and figure out who is the decision maker / set of decision makers there and try to come to consensus |
| 48 | +- dchen1107: I heard a lot of benefits of the split. So there's a huge cost for release management, as the release manager, I don't know what's in the other repositiory. Clear APIs and interfaces could be built without needing separate repositories. Gave example of Docker and CRI, which got API without moving repos. |
| 49 | +- Example of Cloud Foundry, build process which took two weeks. How do we ensure that we're delivering security updates quickly? |
| 50 | +- thockin: We can put many things in APIs and cloud providers are a good example of that. Splitting stuff out into multiple repos comes down to the question of: are we a piece of software or a distribution? You'll have to come to my talk to see the rest. |
| 51 | +- spiffxp: Increasing our dependancy on integration tools will add overhead and process we don't have now. But I understand that most OSS people expect multiple repos and it's easier for them. Github notifications are awful, having multiple repos would make this better. The bots have improved the automation situation, but not really triaging notifications. How many issues do we have that are based on Github. Maybe we should improve the routing of GH notificaitons instead? |
| 52 | +- solly: if we split, we really need to make it easier for tiny repos to plug into our automation and other tools. I shouldn't have to figure out who to ask about getting plugged in, we should have a doc. We need to make sure it's not possible for repos to fall through the cracks like Heapster, which I work on. |
| 53 | +- bgrant: We're already in the land of 90 repos. We don't need to debate splitting, we're alread split. We have incubator, and kubernates-client. I think client has helped a lot, we have 7 languages and that'll grow. |
| 54 | +- bgrant: the velocity of things in the monorepo are static |
| 55 | +- thockin: if issues in the main repo are more than 6 weeks old, nobody looks at it. There's like 400-500 abandoned "network" issues. |
| 56 | +- bgrant: we tested gitlab and gerrit, and importing kubernetes failed. We didn't find them that much better. |
| 57 | +- spiffxp: we have hundreds of directories with no owners files, which is one of the reasons for excessive notifications. |
| 58 | +- bgrant: kubernetes, as a API-driven system, you have to touch the api to do almost anything. we've added mechanisms like CRD to extend the API. We need SDKs to build kube-style APIs. |
| 59 | +- dims: I want to focus on the cost we're paying right now. First we had the google KMS provider. Then we had to kick out the KMS provider, and there was a PR to add the gRPC interface, but it didn't go into 1.9. |
| 60 | +- thockin: the cloud provider stuff is an obvious breeding ground for new functionality, how and if we should add a whole seperate grpc plugin interface is a seperate question |
| 61 | +- jdumars: the vault provider thing was one of the ebtter things that happened, it pushed us at MS to thing about genercizing the solution, it pushed us to think about what's better for the community vs. what's better for the provider |
| 62 | +- jdumars: flipside is we need to have a process where people can up with a well accepted / adopted solution, the vault provider thing was one way of doing that |
| 63 | +- lavalamp: I tend to think that most extension points are special snowflakes and you can't have a generic process for adding a new extension point |
| 64 | +- thockin: wandering back to kubernetes/kubrnetes "main point", looking at staging as "already broken out", are there other ones that we want to break out? |
| 65 | +- dims: kubeadm could move out if needed, could move it to staging for sure |
| 66 | +- thockin: so what about the rest? eg: kubelet, kube-proxy... do we think that people will concretely get benefits from that? or will that cause more pain |
| 67 | +- thockin: we recognize this will slow down things |
| 68 | +- lavalamp: there are utility functions that people commonly use and there's no good common place |
| 69 | +- lavalamp: for kubectl at least it's sphaghetti code that pulls in lots of packages and makes it difficult to do technically |
| 70 | +- thockin: do we think that life would be better at the end of that tunnel, would things be better if kubectl was a different repository, etc. |
| 71 | +- timallclair: I'm worried about dependancy management, godeps is already a nightmare and with multiple repos it would be worse. |
| 72 | +- luxas: in the kubeadm attack plan, we need to get a release for multiple repos. We need the kubeadm repo to be authoritative, and be able to include it in a build. |
| 73 | +- pwittrock: how has "staging" improved development? can we see any of the perceived or hoped-for benefits by looking at staging repos as example use cases? |
| 74 | +- lavalamp: getting to the "staging" state and then stopping is because api-machinery was unblocked once we got there |
| 75 | +- thockin: the reason I consider "staging" solved, is you have to untangle a lot of the stuff already |
| 76 | +- erictune: I would make a plea that we finish some of our started-but-not-finished breakaparts |
0 commit comments