15:10 - 15:40  |  Blueprint Theatre

Better data exploration and visualisation with Elasticsearch and Kibana

Wednesday 13 November 2019


You start a new project. The data is sitting in a jupyter notebook, hard to really explore and visualise. This live demo will show a possible data science workflow upgrade to make the exploration phase more intuitive, and maybe even leave you with something useful for production.

Data science has incredibly sophisticated analysis tools which are very effective, but rarely very convenient for their operators. In this live demo we will focus primarily on the prototyping and exploration phase of a data scientist’s workflow. We’ll see how two tools typically used by web developers can help data scientists and data engineers.

We’ll push a slice of a bigger dataset held in pandas into Elasticsearch. We’ll then use Kibana to build on the fly visualisations and dashboards, allowing us to explore and discover the data’s features in a more intuitive, faster way than writing commands in a jupyter notebook. As Elasticsearch itself is a highly scalable datastore, we’ll discuss how you could take its many abilities beyond prototyping into production if it makes sense for a particular dataset. This demo only includes open source tools or tools free for all use, including commercial.