Pulp tunings by Imaanpreet · Pull Request #2565 · theforeman/foreman-documentation

Imaanpreet · 2023-10-30T19:53:18Z

I am familiar with the contributing guidelines.

Please cherry-pick my commits into:

Foreman 3.8/Katello 4.10
Foreman 3.7/Katello 4.9 (planned Satellite 6.14)
Foreman 3.6/Katello 4.8
Foreman 3.5/Katello 4.7 (Satellite 6.13)
Foreman 3.4/Katello 4.6 (EL8 only)
Foreman 3.3/Katello 4.5 on EL7 & EL8 (Satellite 6.12 on EL8 only; orcharhino 6.4/6.5 on EL8 only)
Foreman 3.2/Katello 4.4 on EL7 & EL8
Foreman 3.1/Katello 4.3 on EL7 & EL8 (Satellite 6.11 EL7/8; orcharhino 6.3 on EL7/8)
We do not accept PRs for Foreman older than 3.1.

github-actions · 2023-10-30T19:59:21Z

The PR preview for 81c0615 is available at theforeman-foreman-documentation-preview-pr-2565.surge.sh

The following output files are affected by this PR:

show diff

show diff as HTML

maximiliankolb · 2023-10-31T07:29:04Z

+
+.Use the below procedure to tune the pulpcore API workers:
+
+. Update the pulpcore api_service_worker_count on your {ProjectServer} or {SmartProxyServer} in the custom-hiera.yaml file.:


Suggested change

. Update the pulpcore api_service_worker_count on your {ProjectServer} or {SmartProxyServer} in the custom-hiera.yaml file.:

. Update the pulpcore api_service_worker_count on your {ProjectServer} or {SmartProxyServer} in the `custom-hiera.yaml` file.:

I am unsure if we should recommend users editing this file vs. using foreman-installer.

I've always taken a stance that custom-hiera.yaml is unsupported. Anything that's written in docs, so we can't refer to it.

If there's always a recommendation to lower the number of workers, that should be filed as a bug. Telling users "we have poor defaults, always change this" is a poor experience.

My suggestion is to drop this chapter altogether.

And if it is common to change it, an RFE to add a parameter as was done with the worker count is also a valid solution.

Hello @ekohl: I totally agree on custom-hiera.yaml is unsupported part. Do you mean to dropping off this section or proc_configuring-pulpcore-workers as well?

I'm questioning the usefulness of proc_configuring-pulpcore-workers. When I read it, it comes down to "here's a parameter you can change, but we noticed it doesn't make a difference". So as a reader of that, is it adding any value or just confusing me?

cool, let's drop this section.

Hello @ekohl , I have removed this chapter proc_configuring-pulpcore-workers from this PR.

Regarding proc_configuring-pulpcore-api-workers.adoc section, can we suggest updating pulpcore api_service_worker_count with another method, for example - via foreman-installer?

Regarding proc_configuring-pulpcore-api-workers.adoc section, can we suggest updating pulpcore api_service_worker_count with another method, for example - via foreman-installer?

Currently this is the code that sets it:
https://github.com/theforeman/puppet-pulpcore/blob/297a48acafcbdb9871689b6e267598c7414692c2/manifests/init.pp#L246-L247

https://projects.theforeman.org/issues/36957 was opened by @maximiliankolb to expose a parameter for this and I just opened theforeman/puppet-foreman_proxy_content#467.

I still think that most users shouldn't touch this and if our default tuning is poor, then it should be fixed.

Co-authored-by: Maximilian Kolb <mail@maximilian-kolb.de>

maximiliankolb

If the approch to change a value to "custom-hiera.yaml" and running foreman-installer is unspoorted, I question the whole PR. Also regarding line 14: "outcome looks similar", which means there's not even a "real reason" to change any value.

What do you think @ekohl & @Imaanpreet ? Is there a RFE to include "pulpcore API worker count" to foreman-installer?

ekohl · 2023-11-17T21:23:26Z

@Imaanpreet please don't manually mark comments as resolved without resolving them. Usually when you modify a line GitHub will automatically hide the comment as outdated.

@maximiliankolb I'm not aware of such an RFE. Such an RFE would be easy to implemen5, if there's a need for it. Currently on my phone and it's a bit hard to search through this PR now

maximiliankolb · 2023-11-29T14:10:51Z

I have opened a RFE: https://projects.theforeman.org/issues/36957 and converted this PR to draft so that we do not triage it again.

Does this work for you? @Imaanpreet

ekohl · 2024-01-03T16:59:11Z

+
+[width="100%",cols="50%,50%",options="header",]
+|===
+|{Project} VM with 8 CPUs, 32 GiB RAM, 3 pulpcore-api_workers |{Project} VM with 8 CPUs, 32 GiB RAM, 10 pulpcore-api_workers


The default tuning for this is min(4, $cpu_count). So the user would have 4 workers. There is no way they get 10 unless they know what they're doing. That makes me question this advice.

ekohl · 2024-01-03T17:02:17Z

+The pulpcore-api workers respond to incoming API requests.
+Fewer API workers consume less memory and results in better performance.


This is questionable advice. Not enough workers means you can't serve the requests in parallel. Taking your advice to the extreme means I simply have 1 worker. But with enough clients, you can saturate that and performance is lower than it could be.

The API service is used by Foreman (via Katello), but also by container content. If your tests didn't test any container workloads then you're only testing a fraction of what it needs to serve.

@Imaanpreet Is this something you've observed? Do you have any numbers on this?

ekohl · 2024-01-03T17:03:12Z

+[id="Configuring_Pulpcore_API-Workers_{context}"]
+= Configuring Pulpcore API Workers
+
+The pulpcore-api workers respond to incoming API requests.


And some content types, such as containers.

Imaanpreet and others added 3 commits December 15, 2022 16:10

REX pull-based transport

77b2446

Merge branch 'theforeman:master' into foreman-3.3.0.17-1.el8sat.noarch

4446b96

pulp_tunings

ccf2b85

pr-processor Bot added the Not yet reviewed label Oct 30, 2023

maximiliankolb requested changes Oct 31, 2023

View reviewed changes

pr-processor Bot added Waiting on contributor Requires an action from the author and removed Not yet reviewed labels Oct 31, 2023

Apply suggestions from code review

1e9cd9c

Co-authored-by: Maximilian Kolb <mail@maximilian-kolb.de>

pr-processor Bot added Needs re-review and removed Waiting on contributor Requires an action from the author labels Oct 31, 2023

ekohl reviewed Oct 31, 2023

View reviewed changes

Comment thread guides/doc-Tuning_Performance/master.adoc Outdated

mjivraja suggested changes Nov 1, 2023

View reviewed changes

Comment thread guides/common/modules/con_pulp-tuning.adoc

Comment thread guides/common/modules/proc_configuring-pulpcore-workers.adoc Outdated

pr-processor Bot added Waiting on contributor Requires an action from the author and removed Needs re-review labels Nov 1, 2023

changes discussed over the PR

81c0615

pr-processor Bot added Needs re-review and removed Waiting on contributor Requires an action from the author labels Nov 1, 2023

maximiliankolb reviewed Nov 16, 2023

View reviewed changes

maximiliankolb marked this pull request as draft November 29, 2023 14:10

ekohl mentioned this pull request Jan 3, 2024

Fixes #36957 - Expose Pulpcore content & api worker counts theforeman/puppet-foreman_proxy_content#467

Draft

ekohl reviewed Jan 3, 2024

View reviewed changes

Imaanpreet closed this by deleting the head repository Aug 5, 2024


		.Use the below procedure to tune the pulpcore API workers:

		. Update the pulpcore api_service_worker_count on your {ProjectServer} or {SmartProxyServer} in the custom-hiera.yaml file.:

		The pulpcore-api workers respond to incoming API requests.
		Fewer API workers consume less memory and results in better performance.

Conversation

Imaanpreet commented Oct 30, 2023

Uh oh!

github-actions Bot commented Oct 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maximiliankolb left a comment

Choose a reason for hiding this comment

Uh oh!

ekohl commented Nov 17, 2023

Uh oh!

maximiliankolb commented Nov 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Oct 30, 2023 •

edited

Loading

maximiliankolb commented Nov 29, 2023 •

edited

Loading