Mechanical Turk Does Solubility on Google Spreadsheet

This has been in the works for several weeks but I think we finally have a practical use for the Amazon Mechanical Turk to help with processing solubility data for the Open Notebook Science Challenge. Many thanks to Tenesha Gleason for the assistance over the phone and online - and to Deepak Singh for getting the ball rolling.

The original idea was to try to crowdsource the measurement of solubility in the laboratory using Mechanical Turk. That probably isn't realistic with the current workforce but I can definitely see that it could be done with the pre-training of select groups around the world. If done properly this could be a very efficient way to get all types of scientific research executed. But I suspect it would be quite difficult to allocate government agency funds for this purpose in a proposal. Luckily other more flexible organizations (like Submeta) are popping up to fill the gap.

So what is realistic then?

We tried different task descriptions and price points to see if we could at least get workers to look up non-aqueous solubility data. There were a few takers - but only one actually found a valid non-aqueous measurement (benzaldehyde in ethanol). All the others either gave aqueous solubility or other irrelevant data like molecular weight or solvent density. Perhaps once we have trained some people to do this properly it will make sense to use Mechanical Turk this way. But right now there is just too much chemistry knowledge required to handle queries of this format.

One of the advantages of Open Notebook Science is that the work is naturally modular because it has to be shared in as close to real time as possible. That doesn't mean that it works flawlessly but it does make it more likely that information that has to be shared will be easier to understand and use by anyone or who wishes to move the project forward. True open crowdsourcing depends on this.

So lets come back to comparing solubility measurements made by our students with those in the literature. But now, instead of asking a high level query, we take the work we've already done on our side to find papers and ask the Turks to extract out the data from selected tables and figures.

I just tried this yesterday and it worked like a charm. I just grabbed a snapshot of the table of interest and asked to convert it to a Google Spreadsheet format that I can readily incorporate into the Solubiliby Summary master sheet. The Turk worker started working within minutes and was done in a quarter of an hour - total cost 46 cents.

This enabled me to add about 70 new entries into the master spreadsheet and allows us now to do quick comparisons using Rajarshi's web query interface. For example, it is pretty clear that the 4-nitrobenzaldehyde solubility measurements by Maccarone, E; Perrini, G. Gazzetta Chimica Italiana vol 112 p 447 (1982) in chloroform are consistent with our measurements in EXP212.


Now this is clearly NOT the case for the solubility of 4-nitrobenzaldehyde in methanol. Our 3 values from EXP212 and EXP205 are much lower than the values from the literature. Reading the experimental section we find that Maccarone and Perrini let their solutions equilibrate over 24 hours while we vortexed for only a few minutes. It is interesting that our chloroform values were consistent. Perhaps there are very large differences in solubility kinetics between solvents. Obviously we have to do this measurement again with much longer mix times.


On a more general note, using a Google Spreadsheet to collect Mechanical Turk results has the advantage that the progress can be monitored in real time and it is possible to chat with the worker. The person who worked on this project was obviously familiar with Google Spreadsheets and approached the task in a similar way that I would have. Here is a video of that process sped up ten fold: