Skip to content

Reuse same instances in hybrid generator - #15983

Open
jackal1-66 wants to merge 1 commit into
AliceO2Group:devfrom
jackal1-66:hybridReuse
Open

jackal1-66 wants to merge 1 commit into
AliceO2Group:devfrom
jackal1-66:hybridReuse

Conversation

@jackal1-66

Copy link
Copy Markdown
Collaborator

Launch single instances of generators which share the same exact configuration. This reduces configuration times and reduces memory usage as well. This is enabled by default, but it is configurable with a new parameter. In addition it is disabled by default when more than a single worker is set (per user decision)

@sawenzel

Copy link
Copy Markdown
Collaborator

Thanks @jackal1-66. This is a good step forward and should work fine. However, the reuse is implicit, which can be dangerous: it relies on two serialised configs matching exactly, so $O2_ROOT/... in one slot and an expanded path in another quietly gives two instances. Since it is on by default, it would also change the event sequence of existing JSONs. Instead, could we consider an explicit solution based on labels and refs, like so:

{
  "mode": "sequential",
  "generators": [
    { "name": "pythia8", "label": "mb",
      "config": { "config": "$O2_ROOT/share/Generators/egconfig/pythia8_inel.cfg" } },
    { "cocktail": [
        { "ref": "mb" },
        { "name": "external",
          "config": { "fileName": "${O2DPG_ROOT}/MC/config/PWGDQ/external/generator/GeneratorParamPromptJpsiToElectronEvtGen_pp13TeV.C",
                      "funcName": "GeneratorParamPromptJpsiToElectronEvtGen_pp13TeV()" } } ] }
  ],
  "fractions": [3, 1]
}

A whole cocktail can be labelled and referenced in the same way:

"generators": [
  { "label": "jpsi", "cocktail": [ { "ref": "mb" }, { "name": "external", "config": { ... } } ] },
  { "ref": "mb" },
  { "ref": "jpsi" }
],
"fractions": [1, 3, 1]

fractions keeps its meaning. A ref must point to a label declared earlier and must not carry config or triggers. Most of this PR can stay: mSlotToInstance and the per-instance queues would be filled from the refs instead of the signatures, and reuseIdenticalGenerators would no longer be needed. By default, nothing would change for the user.

I would recommend that a referenced entry reports the sub-generator ID of the entry it points to. I can propose an implementation on top of your branch, together with a small tool in O2DPG next to o2_hybrid_gen.py that finds duplicated generators in existing configs and rewrites them with labels and refs. In addition, if we go this way, we should add an example in the documentation.

@sawenzel sawenzel left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see inline comment

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants