Multiple Datasets
You can download the data here to quickly run this example locally. Please refer to the GitHub repository for instructions on how to start the Urban Toolkit on your machine.
Step 0 – Open Web Editor
You are going to encounter an initial blank setup (right image) when opening the web editor for the first time. On the left side, we have the grammar editor describing basic structures of the grammar such as a map component, grid configuration, grammar component, and toggle knots component. The right side, on the other hand, is the result of this basic specification.
– Map component: defines position and direction of the camera, how to integrate and render data (“knots”), interactions, plots, and the position of the map on the screen of the application (according to a grid).
– Grid configuration: defines how the screen will be divided. “width” for the number of columns. “height” for the number of rows.
– Grammar component: defines the position of the grammar editor.
– Toggle knots component: defines a widget that supports the toggle of knots.
Step 1 – Adding background layer (water)
To start rendering our scene we are going to add a basic water layer containing the ocean.
The water layer as well as all other layers were previously generated using the the Jupyter API. In order to load it we have to add:
– A knot that contains a pure physical layer (no thematic data). It specifies that we want to output the water layer on the objects level (we want the shapes not coordinates).
– A reference to that knot on the map component.
– The type of interaction we want to have with the knot (none in this case).
* All required changes are highlighted in red. Sections of the code where nothing changed have “…”. After adding them hit Apply Grammar.
{
"components": [
{
"map": {
"camera": {...},
"knots": ["purewater"],
"interactions": ["NONE"]
},
"plots": [],
"knots": [
{
"id": "pureWater",
"integration_scheme": [{"out": {"name": "water", "level": "OBJECTS"}}]
},
],
"position": {...}
},
{
"type": "GRAMMAR",
"position": {...}
},
{
"type": "TOGGLE_KNOT",
"map_id": 0,
"position":{...}
}
],
"arrangement": "LINKED",
"grid": {...}
}
Step 2 – Adding another background layer (parks)
Just as we did in Step 1, we are going to define and render pure “knots” for parks as well. The steps are analogous.
* All required changes are highlighted in red. Sections of the code where nothing changed have “…”. After adding them hit Apply Grammar.
{
"components": [
{
"map": {
"camera": {...},
"knots": [... "pureparks"],
"interactions": [... "NONE"]
},
"plots": [],
"knots": [
...
{
"id": "pureParks",
"integration_scheme": [{"out": {"name": "parks", "level": "OBJECTS"}}]
}
],
"position": {...}
},
{
"type": "GRAMMAR",
"position": {...}
},
{
"type": "TOGGLE_KNOT",
"map_id": 0,
"position":{...}
}
],
"arrangement": "LINKED",
"grid": {...}
}
Step 3 – Aggregating noise by Zip code
For our first thematic data layer, we are going to load the NYC311 noise complaints dataset. Each data point represents a noise complaint geolocated by latitude and longitude. Because of that, we can define a knot that will, giving the existence of a physical layer defining zip code boundaries, aggregate the complaints by zip code and color them according to the concentration of data points. To do so, some changes are necessary:
– A knot defining a spatial_relation of type “CONTAINS” such as that the noise data points (“COORDINATES” level) are grouped by zip code boundary (“OBJECTS” level).
– An operation defining how to aggregate the data points inside the zip code (by counting them in this case).
– An abstract flag to indicate that this operation involves a thematic layer.
– A “PICKING” interaction for the knot rendered on the map. This interaction will be used in later steps.
* All required changes are highlighted in red. Sections of the code where nothing changed have “…”. After adding them hit Apply Grammar.
{
"components": [
{
"map": {
"camera": {...},
"knots": [... "noiseToZip"],
"interactions": [... "PICKING"]
},
"plots": [],
"knots": [
...
{
"id": "noiseToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "noise", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
}
],
"position": {...}
},
{
"type": "GRAMMAR",
"position": {...}
},
{
"type": "TOGGLE_KNOT",
"map_id": 0,
"position":{...}
}
],
"arrangement": "LINKED",
"grid": {...}
}
Step 4 – Aggregating multiple datasets by Zip code
We are going to repeat the same process of Step 3 and load other datasets such as taxi, crime, restaurants, subways, schools, sky exposure, and distribution of parks also aggregating them by zip code. As one can see the process is very similar with the difference we are not going to add these knots to be rendered on the map. Since the map can only render one color scale at a time we chose to render the noise data in the previous step (that is why we do not see any visual changes in this step). The other datasets will then be used to feed a 2D plot.
* All required changes are highlighted in red. Sections of the code where nothing changed have “…”. After adding them hit Apply Grammar.
{
"components": [
{
"map": {
"camera": {...},
"knots": [...],
"interactions": [...]
},
"plots": [],
"knots": [
...
{
"id": "taxiPickupToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "taxi_pickup", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "crimeToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "crime", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "restaurantsToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "restaurants", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "subwayToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "subway", "level": "COORDINATES"},
"out": {"name": "zip","level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "schoolToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "school","level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "skyToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "sky", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "AVG",
"abstract": true
}
]
},
{
"id": "parksToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "parks_abstract", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
}
],
"position": {...}
},
{
"type": "GRAMMAR",
"position": {...}
},
{
"type": "TOGGLE_KNOT",
"map_id": 0,
"position":{...}
}
],
"arrangement": "LINKED",
"grid": {...}
}
Step 5 – Visualizing data through a Parallel Coordinates plot
UTk allows us to create 2D plots using Vega-Lite and source them with the knots defined. Since we have multiple knots representing multiple dimensions aggregated by zip code we will create a parallel coordinates chart. Notice that we are using plain vega with the difference that it is possible to use keywords to make references to the knots inside the specification. Notice that each plot has four fields: plot (vega specification), knots (id of the knots to be used on the plot), arrangement (where the plot will be located), and interaction. In our case, the arrangement is “LINKED” as opposed to “EMBEDDED” because we want the plots to be on the surface of the screen. On the other hand, the interaction is set to “HOVER” because we want the chart to have a hover interaction.
To access knot data inside the vega specification one can use the structure “knotId_keyword”. The keyword can be “abstract” (the thematic data), “index” (the index of the element on the knot), and “highlight” a boolean indicating if the element of the knot is highlighted. In Step 3 we configure our zip + noise layer to have a “PICKING” interaction which means that we can highlight zip codes by pressing “t” while hovering a specific zip code and the “noiseToZip_highlight” variable inside the vega specification will be true for that object. The other way also works. Since we configure the chart to have a “HOVER” interaction the correspondent zip code will be highlighted.
{
"components": [
{
"map": {
"camera": {...},
"knots": [
...
],
"interactions": [
...
]
},
"plots": [
{
"plot": {
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"width": 600,
"height": 300,
"title": {
"text": "Datasets aggregated by zip code",
"fontSize": 18
},
"transform": [
{"window": [{"op": "count", "as": "index"}]},
{"fold": ["taxiPickupToZip_abstract", "noiseToZip_abstract", "crimeToZip_abstract", "restaurantsToZip_abstract", "subwayToZip_abstract", "schoolToZip_abstract", "skyToZip_abstract", "parksToZip_abstract"]},
{"joinaggregate": [{"op": "min", "field": "value", "as": "min"}, {"op": "max", "field": "value", "as": "max"}], "groupby": ["key"]},
{"calculate": "(datum.value - datum.min) / (datum.max-datum.min)", "as": "norm_val"},
{"calculate": "(datum.min + datum.max) / 2", "as": "mid"},
{"calculate": "datum.key === 'taxiPickupToZip_abstract' ? 'Taxi' : datum.key === 'noiseToZip_abstract' ? 'Noise' : datum.key === 'crimeToZip_abstract' ? 'Crime' : datum.key === 'restaurantsToZip_abstract' ? 'Food' : datum.key === 'subwayToZip_abstract' ? 'Subway' : datum.key === 'schoolToZip_abstract' ? 'School' : datum.key === 'skyToZip_abstract' ? 'Sky' : datum.key === 'parksToZip_abstract' ? 'Parks' : 'others'", "as": "key"}
],
"layer": [{"mark": {"type": "rule", "color": "#ccc"}, "encoding": {"detail": {"aggregate": "count"}, "x": {"field": "key"}}}, {
"mark": "line",
"encoding": {
"detail": {"type": "nominal", "field": "index"},
"opacity": {"value": 0.8},
"x": {"type": "nominal", "field": "key"},
"y": {"type": "quantitative", "field": "norm_val", "axis": null},
"stroke": {"condition": [{"test": "datum['taxiPickupToZip_highlight'] == true", "value": "#4a97ed"}], "value": "grey"}
}
}, {
"encoding": {"x": {"type": "nominal", "field": "key"}, "y": {"value": 0}},
"layer": [{
"mark": {"type": "text", "style": "label"},
"encoding": {"text": {"aggregate": "max", "field": "max"}}
}, {
"mark": {"type": "tick", "style": "tick", "size": 8, "color": "#ccc"}
}]
}, {
"encoding": {
"x": {"type": "nominal", "field": "key"},
"y": {"value": 150}
},
"layer": [{
"mark": {"type": "text", "style": "label"},
"encoding": {"text": {"aggregate": "min", "field": "mid"}}
}, {"mark": {"type": "tick", "style": "tick", "size": 8, "color": "#ccc"}}]
}, {
"encoding": {"x": {"type": "nominal", "field": "key"}, "y": {"value": 300}},
"layer": [{
"mark": {"type": "text", "style": "label"},
"encoding": {"text": {"aggregate": "min", "field": "min"}}
}, {"mark": {"type": "tick", "style": "tick", "size": 8, "color": "#ccc"}}]
}],
"config": {
"axisX": {"domain": false, "labelAngle": 0, "tickColor": "#ccc", "title": null, "labelFontSize": 16},
"view": {"stroke": null},
"style": {"label": {"baseline": "middle", "align": "right", "dx": -5}, "tick": {"orient": "horizontal"}}
}
},
"knots": ["taxiPickupToZip", "noiseToZip", "crimeToZip", "restaurantsToZip", "subwayToZip", "schoolToZip", "skyToZip", "parksToZip"],
"arrangement": "LINKED",
"interaction": "HOVER"
}
],
"knots": [
...
],
"position": {...}
},
{
"type": "GRAMMAR",
"position": {...}
},
{
"type": "TOGGLE_KNOT",
"map_id": 0,
"position":{...}
}
],
"arrangement": "LINKED",
"grid": {...}
}
Final result
{
"components": [
{
"map": {
"camera": {
"position": [-8239611, 4941390.5, 2.019619873046875],
"direction": {
"right": [442.88140869140625, -1476.4549560546875, 2019.619873046875],
"lookAt": [460.66815185546875, -850.698974609375, 1239.803955078125],
"up": [0.022156884893774986, 0.77950119972229, 0.6260088086128235]
}
},
"knots": ["pureWater", "pureParks", "noiseToZip"],
"interactions": ["NONE", "NONE", "PICKING"]
},
"plots": [
{
"plot": {
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"width": 600,
"height": 300,
"title": {
"text": "Datasets aggregated by zip code",
"fontSize": 18
},
"transform": [
{"window": [{"op": "count", "as": "index"}]},
{"fold": ["taxiPickupToZip_abstract", "noiseToZip_abstract", "crimeToZip_abstract", "restaurantsToZip_abstract", "subwayToZip_abstract", "schoolToZip_abstract", "skyToZip_abstract", "parksToZip_abstract"]},
{"joinaggregate": [{"op": "min", "field": "value", "as": "min"}, {"op": "max", "field": "value", "as": "max"}], "groupby": ["key"]},
{"calculate": "(datum.value - datum.min) / (datum.max-datum.min)", "as": "norm_val"},
{"calculate": "(datum.min + datum.max) / 2", "as": "mid"},
{"calculate": "datum.key === 'taxiPickupToZip_abstract' ? 'Taxi' : datum.key === 'noiseToZip_abstract' ? 'Noise' : datum.key === 'crimeToZip_abstract' ? 'Crime' : datum.key === 'restaurantsToZip_abstract' ? 'Food' : datum.key === 'subwayToZip_abstract' ? 'Subway' : datum.key === 'schoolToZip_abstract' ? 'School' : datum.key === 'skyToZip_abstract' ? 'Sky' : datum.key === 'parksToZip_abstract' ? 'Parks' : 'others'", "as": "key"}
],
"layer": [{"mark": {"type": "rule", "color": "#ccc"}, "encoding": {"detail": {"aggregate": "count"}, "x": {"field": "key"}}}, {
"mark": "line",
"encoding": {
"detail": {"type": "nominal", "field": "index"},
"opacity": {"value": 0.8},
"x": {"type": "nominal", "field": "key"},
"y": {"type": "quantitative", "field": "norm_val", "axis": null},
"stroke": {"condition": [{"test": "datum['taxiPickupToZip_highlight'] == true", "value": "#4a97ed"}], "value": "grey"}
}
}, {
"encoding": {"x": {"type": "nominal", "field": "key"}, "y": {"value": 0}},
"layer": [{
"mark": {"type": "text", "style": "label"},
"encoding": {"text": {"aggregate": "max", "field": "max"}}
}, {
"mark": {"type": "tick", "style": "tick", "size": 8, "color": "#ccc"}
}]
}, {
"encoding": {
"x": {"type": "nominal", "field": "key"},
"y": {"value": 150}
},
"layer": [{
"mark": {"type": "text", "style": "label"},
"encoding": {"text": {"aggregate": "min", "field": "mid"}}
}, {"mark": {"type": "tick", "style": "tick", "size": 8, "color": "#ccc"}}]
}, {
"encoding": {"x": {"type": "nominal", "field": "key"}, "y": {"value": 300}},
"layer": [{
"mark": {"type": "text", "style": "label"},
"encoding": {"text": {"aggregate": "min", "field": "min"}}
}, {"mark": {"type": "tick", "style": "tick", "size": 8, "color": "#ccc"}}]
}],
"config": {
"axisX": {"domain": false, "labelAngle": 0, "tickColor": "#ccc", "title": null, "labelFontSize": 16},
"view": {"stroke": null},
"style": {"label": {"baseline": "middle", "align": "right", "dx": -5}, "tick": {"orient": "horizontal"}}
}
},
"knots": ["taxiPickupToZip", "noiseToZip", "crimeToZip", "restaurantsToZip", "subwayToZip", "schoolToZip", "skyToZip", "parksToZip"],
"arrangement": "LINKED",
"interaction": "HOVER"
}
],
"knots": [
{
"id": "pureWater",
"integration_scheme": [{"out": {"name": "water", "level": "OBJECTS"}}]
},
{
"id": "pureParks",
"integration_scheme": [{"out": {"name": "parks", "level": "OBJECTS"}}]
},
{
"id": "noiseToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "noise", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "taxiPickupToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "taxi_pickup", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "crimeToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "crime", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "restaurantsToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "restaurants", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "subwayToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "subway", "level": "COORDINATES"},
"out": {"name": "zip","level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "schoolToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "school","level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
},
{
"id": "skyToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "sky", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "AVG",
"abstract": true
}
]
},
{
"id": "parksToZip",
"integration_scheme": [
{
"spatial_relation": "CONTAINS",
"in": {"name": "parks_abstract", "level": "COORDINATES"},
"out": {"name": "zip", "level": "OBJECTS"},
"operation": "COUNT",
"abstract": true
}
]
}
],
"position": {"width": [6,12], "height": [1,5]}
},
{
"type": "GRAMMAR",
"position": {"width": [1,5], "height": [3,5]}
},
{
"type": "TOGGLE_KNOT",
"map_id": 0,
"position": {"width": [1,5], "height": [1,2]}
}
],
"arrangement": "LINKED",
"grid": {"width": 12, "height": 5}
}