With the task of adding clarity to the water market, we have been working to simplify the complex sector of Australian Water Trading for users.
Waterflow is Australia’s first and only independent and free-to-use large-scale aggregator of water trading data for the Murray Darling basin and Australia. An initiative of Marsden Jacob, and funded through the Australian government’s BRII initiative, Waterflow gives farmers clarity on all key aspects of the market; including pricing, dam storage levels, trade limits and government allocations.
The Waterflow Project presented three significant challenges:
- Data Collection – Water trading data is typically a few weeks old. We needed up to date data which meant going direct to Australian government data, state water registries, brokers and other intermediaries. We currently automatically acquire data from 44 sources, each in a different format.
- Data Cleaning – Water trading data is often misleading either due to “off market” trades or human error.
- Usability – Water trading is complex, with a range of geographic and legislative restrictions on what trades are allowed between zones. We needed simple interfaces to help farmers understand their trading options. In addition, our farmers are often in remote locations on smartphones with slow, expensive data.
The data collection challenge was solved by developing a multi-phase data pipeline to allowing systematic scraping, importing and cleaning of trade data. This pipeline approach also allowed us to quickly add new suppliers.
Data cleaning was tackled in two ways. Firstly, we use statistical models to exclude extreme outliers. We then present a tailored interface to allow a Marsden Jacob expert to manually remove any remaining questionable data in just a few minutes per day.
The usability of the platform was a key focus from the outset. Our User Experience (UX) specialist worked closely with Marsden Jacob staff and conducted a series of user interviews. This feedback informed all aspects of the build, including the development of two “low data” native mobile apps for remote farmers.
Waterflow is scheduled for an upcoming launch. The response to farmer previews has been incredibly positive and a number of corporate customers have approached Marsden Jacob to purchase additional reports and clean trade data en masse.