Skip to content

Add a "Create a Dataset from a CSV" tutorial and make the getting-started tutorial read-only - #131

Draft
AntoninKns wants to merge 3 commits into
OCHA-DAP:mainfrom
AntoninKns:docs/tutorial-create-dataset
Draft

AntoninKns wants to merge 3 commits into
OCHA-DAP:mainfrom
AntoninKns:docs/tutorial-create-dataset

Conversation

@AntoninKns

Copy link
Copy Markdown
Contributor

This PR corresponds to the suggested tutorial rewrite mentioned in #129. A follow-up to the documentation restructure in #126.

Following the discussion in #129, where it was confirmed that the getting-started tutorial doesn't explain authorisation for writing, this PR proposes one way to cover that: separate reading from writing, and add a tutorial whose main part runs offline. I'm happy to adjust the approach.

What changed (one commit each, so they can be reviewed separately)

  1. Make the getting-started tutorial read-only to make scope clearer. Retitled "Read a Dataset"; the path is unchanged, so existing links keep working. Removed the API key step, the stage write configuration and the steps that modify datasets, renumbered the steps and fixed the Docker "jump to step" reference. A closing paragraph points to the new tutorial.
  2. Add "Create a Dataset from a CSV". It goes from a made-up CSV to a complete Dataset, ending just before create_in_hdx(). Steps: prepare the CSV, connect read-only, build the dataset, generate the resource with generate_resource, validate with check_required_fields, look at what would be sent, then explain where a dry run stops. A final section covers organisation members: token setup, rehearsal on demo or stage, then prod. It follows the naming and format guidance in the HDX pipeline recommendations. The sample CSV is in documentation/tutorials/assets/.
  3. Test the tutorial's code. tests/hdx/data/test_tutorial_create_dataset.py runs steps 3 to 6 offline with the existing fixtures.

How it was tested

  • I ran the code of steps 2 to 6 as written, from a clean folder, against prod in read-only mode, using the sample CSV.
  • The new test passes, and ruff check and format are clean. The rest of tests/hdx/data passes except test_datastore_queries, which also errored before this change and which I haven't investigated.
  • mkdocs build runs without warnings (uv run --with mkdocs-material --with mkapi mkdocs build -f documentation/mkdocs.yaml). The page renders under Tutorials.
  • The "Before you start" commands were checked in a fresh virtualenv with Python 3.14.4, installing from this checkout.
  • The error in step 7 (Access denied: User not authorised to create packages) was observed on demo with a valid token and no organisation.

Not tested: needs a maintainer

  • Section 8 ("For organisation members") needs publishing rights, which I don't have. Could you run those steps once on demo? If they work, the sentence saying they haven't been run end to end can be removed.
  • Not tested: Windows activation of the virtualenv, and installing the released PyPI package (I installed this checkout).

Questions

  • The tutorials use user_agent="tutorial_create_dataset" as an example name. Would you prefer a different one?
  • The test repeats the tutorial's code by hand. Should I change it to make it run the code blocks from the page directly?

Generated with Claude, fully reviewed and edited manually.

@mcarans

mcarans commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

@AntoninKns thanks for your PR. I checked out the branch, ran the new test which passes and ran section 8 on demo with a real token.

step 3 fails: create_in_hdx() fails on demo with:

HDXError: Failed when trying to create bel-example-indicators! (POST)
ValidationError({'methodology_other': ['Missing value']})

The dataset sets "methodology": "Other", and HDX then requires methodology_other (which is some methodology text) in the dict. Please update build_dataset accordingly.

With methodology_other added, the rest of section 8 worked on demo:

  • User.get_current_user()["id"] returned a usable maintainer id.
  • create_in_hdx() created the private dataset.
  • Reading it back showed the right title, private flag, time period (from 2025-01-31) and resource.
  • Running update_in_hdx() again updated the same dataset instead of creating a second one. The log said "Resource exists. Updating bel_example_indicators.csv".

from hdx.data.user import User

organisations = User.get_current_user_organization_dicts("create_dataset")
dataset["owner_org"] = organisations[0]["id"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better here for the user to explicitly pick the organisation to which they intend to upload

resource or parent dataset:

resource = dataset.get_resource(2)
resource.mark_data_updated()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Link from the closing paragraph of getting-started.md or from the last line of the create tutorial to how-to/operate-resources/resource-operations.md which has details on resource operations

@@ -0,0 +1,93 @@
"""Tests for the code in the "Create a Dataset from a CSV" tutorial"""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment along the lines of: "Mirrors the code in documentation/tutorials/create-a-dataset.md (steps 3-6). Update both together."

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants