I spent a lot of time this weekend revisiting our new database design for KBMOD, and see last post. I had a good excuse to finally learn about the spatial database support in PostgreSQL — PostGIS. Through several iterations and redrafts, I sped some of our test queries up by a ...
read moreThere are comments.
In the morning, Andy Becker and I met with Andy Connolly, who was apparently not in London at the time, to give him an update on our KBMOD progress this quarter. We talked about the database designs, the scaling lessons, the technology tradeoffs, and the potential of our new approach ...
read moreThere are comments.
Today in the Data Science Incubator, Sophie Clayton wrote queries (and I helped a tiny bit) to co-align every cytogram in the SeaFlow cruise in order to compensate for different settings of the SeaFlow instrument itself. The queries should have taken a few minutes to run, but routinely failed due ...
read moreThere are comments.
More work with Andy Becker on the KBMOD database and query design today. We have been running into issues loading all the pixels in the database — O(Millions) of images with O(Millions) of pixels each means 10^12 records, which would take years to load into our Postgres database ...
read moreThere are comments.
Today, UW is closed for the Veteran’s Day holiday; I took the opportunity to continue catching up on several weeks’ email.
Ângela Katsuyama came to the Incubator for a few hours this afternoon. With Bill, she continued joining and analyzing her data, attempting to answer questions like “what is ...
read moreThere are comments.
In this incubator this morning, Sophie and I worked on the IPython Notebook — Myria pipeline. In particular, we pushed some bug fixes to some SSL bugs (but not all the way) and demonstrated an ability to issue queries in Datalog or MyriaL from Python that get executed on Myria!
Sophie ...
read moreThere are comments.
Today was “I-hate-certificates” day. Four different people had ugly issues with SSL:
Sophie has finished cleaning the underway data in SQLShare and is ready to load it into Myria. However, she had problems using Myria’s Python client because her Mac did not trust Myria’s SSL certificate. (UW’s ...
There are comments.
The newest Chrome update broke parts of Myria because of new policies regarding security (HTTPS) in web API calls. So, I finally did the 5 hours of work it would take to make Myria’s API server support HTTPS, installed the certificate, and made all our deployment scripts. Now our ...
read moreThere are comments.
Today was day 1 of Incubator week 3.
Sophie Clayton exposed a locking issue in SQLShare: if a running query reads from table T and a user changes the sharing on table T, SQLShare basically locks the entire system until the query finishes. We need to revisit this issue, but ...
There are comments.
Over the weekend, both Sophie Clayton and Andy Becker worked independently on their Data Science Incubator projects; I spent some time then and today answering emails :).
Sophie has been loading underway data (GPS, temperature, salinity, etc. from ships in motion) into SQLShare for cleaning. Every research vessel is its own ...
read moreThere are comments.
I spent today in the Data Science Incubator working with Andy Becker on KBMOD. We are trying to port the key components of the moving object detection into PostgreSQL using PostGIS.
On his own, Andy designed a schema for the various data, including spatial columns and indexes. Side by side ...
read moreThere are comments.
The majority activity today was reviewing the proposals we received for the Fall 2014 edition of our Data Science Incubator. We received 26 fantastic proposals ranging from Astronomy to Statistics (sadly, no Zoology), and we are probably going to have to cut it down to 5 or 6. Lots of ...
read moreThere are comments.
I also did not get much time to do real work today. There were three major activities:
UW Data Science Incubator applications are due Thursday! They have started rolling in, so I have started looking at them and have started a few clarifying discussions with some of the authors. Getting ...
There are comments.
Today we held the information session for the second installation of our Data Science Incubator projects which we will hold in the Spring. It was fairly well attended; maybe 20—25 people came and many of these indicated that they will be submitting proposals.
Over the weekend and today I ...
read moreThere are comments.