-
Notifications
You must be signed in to change notification settings - Fork 0
Feature/maxim start #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
10f567e
29ab7ea
35abba2
27eb1aa
17230f4
429133f
da58a8d
ff735eb
51611f9
834087f
9d4e48d
746b1d4
329b94d
8d3a77f
2a7fba0
edf1c6f
b72db6e
d7ce084
bbb2e06
8c3a1b2
c1eeb9b
eca8c55
78d37bb
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,27 @@ | ||
| *** | ||
| # jflat | ||
| Utility functions to transform nested JSON schemas into a flat list of dictionaries. | ||
|
|
||
| *** | ||
| # jflat | ||
|
|
||
| A small Python utility to transform **nested JSON objects** into a **flat dictionary** with underscore‐separated keys. | ||
|
|
||
| This project demonstrates: | ||
| - A clean `src/`-based Python package layout | ||
| - A minimal Pydantic schema example | ||
| - Conversion from nested Pydantic model output to a flat JSON using `flatten_json` | ||
| - Example schema JSON generation (`examples/schema_json.json`) | ||
| - Pytest-based unit tests | ||
|
|
||
| --- | ||
|
|
||
| ## 📦 Installation (development mode) | ||
|
|
||
| Clone the repository and install in editable mode: | ||
|
|
||
| ```bash | ||
| pip install -e . | ||
| ``` | ||
|
|
||
| *** |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| [build-system] | ||
| requires = ["setuptools>=61.0.0", "wheel", "setuptools-scm"] | ||
| build-backend = "setuptools.build_meta" | ||
|
|
||
| [project] | ||
| name = "jflat" | ||
| dynamic = ["version"] | ||
| description = "Utility functions to flatten nested JSON into a flat dictionary." | ||
| readme = "README.md" | ||
| authors = [ | ||
| { name = "Maxim Köppel" } | ||
| ] | ||
| requires-python = ">=3.10" | ||
|
|
||
| dependencies = [ | ||
| "pydantic>=2.0" | ||
| ] | ||
|
|
||
| classifiers = [ | ||
| "Programming Language :: Python :: 3", | ||
| "Programming Language :: Python :: 3 :: Only", | ||
| ] | ||
|
|
||
| [project.urls] | ||
| repository = "https://github.com/BAMResearch/jflat" | ||
| homepage = "https://github.com/BAMResearch/jflat" | ||
|
|
||
| [tool.setuptools] | ||
| package-dir = { "" = "src" } | ||
|
|
||
| [tool.setuptools.packages.find] | ||
| where = ["src"] | ||
|
|
||
| [tool.setuptools_scm] | ||
| write_to = "src/jflat/_version.py" | ||
|
|
||
| [tool.pytest.ini_options] | ||
KoeppelSoftwareEngineer marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| pythonpath = [ | ||
| "src" | ||
| ] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| from schema import Method | ||
| import json | ||
|
|
||
| """ | ||
| File to create a JSON Schema (stored in `./examples/schema_json.json`) based on the example data model defined | ||
| in `./examples/schema.py`. | ||
|
|
||
| This JSON Schema is produced by calling `Method.model_json_schema()`, where `Method` is defined in `schema.py`. | ||
| """ | ||
|
|
||
| schema = Method.model_json_schema() | ||
|
|
||
| with open("schema_json.json", "w") as f: | ||
| json.dump(schema, f, indent=2) | ||
|
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| from pydantic import BaseModel, Field | ||
|
|
||
| class Example(BaseModel): | ||
| id: str = Field(..., description="The unique identifier") | ||
|
|
||
|
|
||
| class Person(BaseModel): | ||
| name: str = Field(..., description="The person's name") | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe you can define more types in this example? Something like |
||
| age: int = Field(..., ge=0, description="The person's age in years") | ||
| example: Example | ||
|
|
||
|
|
||
| class BaseMethod(BaseModel): | ||
| author: str = Field(..., description="The author of the method") | ||
|
|
||
|
|
||
| class Method(BaseMethod): | ||
| method_name: str = Field(..., description="The name of the method") | ||
| person: Person | ||
|
|
||
|
|
||
| #ToDo Maybe you can define more types in this example? Something like str | none, and create more different Fields (so the printed JSON schema is actually richer and we can cover more cases) | ||
|
|
||
|
|
||
| """ | ||
| src/jflat/jflat.py def flatten_json(data: Dict[str, Any], parent_key: str = "") -> Dict[str, Any]: | ||
|
|
||
| The goal of this function is more to map a JSON Schema (as printed by pydantic) to our new flattened version. The end result should look something like: | ||
|
|
||
| { | ||
| "$defs": { | ||
| "Method": { | ||
| "properties": { | ||
| "author": { | ||
| "description": "The author of the method", | ||
| "type": "string" | ||
| }, | ||
| "method_name": { | ||
| "description": "The name of the method", | ||
| "type": "string" | ||
| }, | ||
| "person": { | ||
| "$ref": "#/$defs/Person" | ||
| } | ||
| }, | ||
| "description": "...<whatever-here>...", | ||
| "$inherits_from": "#/$defs/BaseMethod", | ||
| }, | ||
| <other classes here> | ||
| }} | ||
| As you can see, I slightly modified the resulting JSON schema when printed using model_json_schema. The idea is to get rid off unnecessary stuff and adding some other info. I: | ||
|
|
||
| Deleted title in each property | ||
| Deleted title in each object | ||
| Added an $inherits_from key in each of the objects dictionaries | ||
| Moved the Method defs inside $defs (before, it is outside because we are printing from it) | ||
| We need to add BaseMethod to define inheritances | ||
| Deleted all the required and type:object stuff | ||
| We could also: | ||
|
|
||
| Add a key inside each property defining if they are mandatory or optional (this was before defined by required. Somethind like: | ||
| "author": { | ||
| "description": "The author of the method", | ||
| "type": "string", | ||
| "mandatory": true # this can also be false, if the property is optional (i.e., str | None) | ||
| }, | ||
|
|
||
| """ | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| { | ||
| "$defs": { | ||
| "Example": { | ||
| "properties": { | ||
| "id": { | ||
| "description": "The unique identifier", | ||
| "title": "Id", | ||
| "type": "string" | ||
| } | ||
| }, | ||
| "required": [ | ||
| "id" | ||
| ], | ||
| "title": "Example", | ||
| "type": "object" | ||
| }, | ||
| "Person": { | ||
| "properties": { | ||
| "name": { | ||
| "description": "The person's name", | ||
| "title": "Name", | ||
| "type": "string" | ||
| }, | ||
| "age": { | ||
| "description": "The person's age in years", | ||
| "minimum": 0, | ||
| "title": "Age", | ||
| "type": "integer" | ||
| }, | ||
| "example": { | ||
| "$ref": "#/$defs/Example" | ||
| } | ||
| }, | ||
| "required": [ | ||
| "name", | ||
| "age", | ||
| "example" | ||
| ], | ||
| "title": "Person", | ||
| "type": "object" | ||
| } | ||
| }, | ||
| "properties": { | ||
| "author": { | ||
| "description": "The author of the method", | ||
| "title": "Author", | ||
| "type": "string" | ||
| }, | ||
| "method_name": { | ||
| "description": "The name of the method", | ||
| "title": "Method Name", | ||
| "type": "string" | ||
| }, | ||
| "person": { | ||
| "$ref": "#/$defs/Person" | ||
| } | ||
| }, | ||
| "required": [ | ||
| "author", | ||
| "method_name", | ||
| "person" | ||
| ], | ||
| "title": "Method", | ||
| "type": "object" | ||
| } |
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why do we need an AI generated json?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, DISCUSSION: |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
|
|
||
| { | ||
| "title": "Method", | ||
| "type": "object", | ||
| "properties": { | ||
| "author": { | ||
| "title": "Author", | ||
| "description": "The author of the method", | ||
| "type": "string" | ||
| }, | ||
| "method_name": { | ||
| "title": "Method Name", | ||
| "description": "The name of the method", | ||
| "type": "string" | ||
| }, | ||
| "person": { | ||
| "title": "Person", | ||
| "allOf": [ | ||
| { | ||
| "$ref": "#/$defs/Person" | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| "required": [ | ||
| "author", | ||
| "method_name", | ||
| "person" | ||
| ], | ||
| "$defs": { | ||
| "Example": { | ||
| "title": "Example", | ||
| "type": "object", | ||
| "properties": { | ||
| "id": { | ||
| "title": "Id", | ||
| "description": "The unique identifier", | ||
| "type": "string" | ||
| } | ||
| }, | ||
| "required": [ | ||
| "id" | ||
| ] | ||
| }, | ||
| "Person": { | ||
| "title": "Person", | ||
| "type": "object", | ||
| "properties": { | ||
| "name": { | ||
| "title": "Name", | ||
| "description": "The person's name", | ||
| "type": "string" | ||
| }, | ||
| "age": { | ||
| "title": "Age", | ||
| "description": "The person's age in years", | ||
| "type": "integer", | ||
| "minimum": 0 | ||
| }, | ||
| "example": { | ||
| "title": "Example", | ||
| "allOf": [ | ||
| { | ||
| "$ref": "#/$defs/Example" | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| "required": [ | ||
| "name", | ||
| "age", | ||
| "example" | ||
| ] | ||
| } | ||
| } | ||
| } |
KoeppelSoftwareEngineer marked this conversation as resolved.
Show resolved
Hide resolved
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
|
|
||
| from .jflat import flatten_json | ||
|
|
||
| __all__ = ["flatten_json"] |
KoeppelSoftwareEngineer marked this conversation as resolved.
Show resolved
Hide resolved
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| from typing import Any, Dict | ||
|
|
||
|
|
||
| def flatten_json(data: Dict[str, Any], parent_key: str = "") -> Dict[str, Any]: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The goal of this function is more to map a JSON Schema (as printed by pydantic) to our new flattened version. The end result should look something like: {
"$defs": {
"Method": {
"properties": {
"author": {
"description": "The author of the method",
"type": "string"
},
"method_name": {
"description": "The name of the method",
"type": "string"
},
"person": {
"$ref": "#/$defs/Person"
}
},
"description": "...<whatever-here>...",
"$inherits_from": "#/$defs/BaseMethod",
},
<other classes here>
}}As you can see, I slightly modified the resulting JSON schema when printed using
We could also:
"author": {
"description": "The author of the method",
"type": "string",
"mandatory": true # this can also be false, if the property is optional (i.e., str | None)
}, |
||
| """ | ||
| Flatten a nested JSON-like dictionary into a flat dictionary | ||
| using underscore-separated keys. | ||
|
|
||
| Example: | ||
| {"person": {"name": "Alice"}} | ||
| becomes: | ||
| {"person_name": "Alice"} | ||
| """ | ||
| result: Dict[str, Any] = {} | ||
|
|
||
| for key, value in data.items(): | ||
| new_key = f"{parent_key}_{key}" if parent_key else key | ||
|
|
||
| if isinstance(value, dict): | ||
| # Recursively flatten child dictionaries | ||
| result.update(flatten_json(value, new_key)) | ||
| else: | ||
| result[new_key] = value | ||
|
|
||
| return result | ||
|
|
||
|
|
||
|
|
||
|
|
||
| def flatten_json_schema(): | ||
| pass | ||
KoeppelSoftwareEngineer marked this conversation as resolved.
Show resolved
Hide resolved
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
|
|
||
| from jflat.jflat import flatten_json | ||
|
|
||
|
|
||
| def test_flatten_json_simple(): | ||
| data = {"a": 1, "b": 2} | ||
| assert flatten_json(data) == {"a": 1, "b": 2} | ||
|
|
||
|
|
||
| def test_flatten_json_nested(): | ||
| data = { | ||
| "person": { | ||
| "name": "Alice", | ||
| "info": { | ||
| "age": 30 | ||
| } | ||
| } | ||
| } | ||
|
|
||
| flattened = flatten_json(data) | ||
|
|
||
| assert flattened == { | ||
| "person_name": "Alice", | ||
| "person_info_age": 30 | ||
| } |
Uh oh!
There was an error while loading. Please reload this page.