Brandon Myers rocked his generals exam today. Brandon (perhaps atypically, but appropriate in this case) gave a literature-review style talk in which he focused on work in many fields related to the performance of generated code and hand-coded solutions for distributed query processing.
read moreIn-memory data management (IMDM) systems have achieved ...
There are comments.
David Beck asked me somewhat last-minute to open this quarter’s IGERT Community Seminar with a half-hour discussion of SQLShare. I dusted off the slides from the What to teach Biologists about Computing? (w2tbac)‘13 meeting. (Slides here.) My goal was to give the room (mostly Ph.D. students and ...
read moreThere are comments.
Today we opened up the WRF Data Science Studio to the public. I am very excited that we are finally open for business.
I spent most of my day welcoming visitors to the Studio, giving tours, and having a few meetings.
Arfon Smith from GitHub took the morning off from ...
read moreThere are comments.
Not much happening this week, what with the holidays. I took advantage of the break to handle some long-overdue code reviews and code improvements to Myria.
Jingjing Wang has added resource profiling to Myria. We can now measure the resource consumption of each operator during query execution. (Unfortunately, these data ...
There are comments.
I spent a lot of time this weekend revisiting our new database design for KBMOD, and see last post. I had a good excuse to finally learn about the spatial database support in PostgreSQL — PostGIS. Through several iterations and redrafts, I sped some of our test queries up by a ...
read moreThere are comments.
In the morning, Andy Becker and I met with Andy Connolly, who was apparently not in London at the time, to give him an update on our KBMOD progress this quarter. We talked about the database designs, the scaling lessons, the technology tradeoffs, and the potential of our new approach ...
read moreThere are comments.
Today in the Data Science Incubator, Sophie Clayton wrote queries (and I helped a tiny bit) to co-align every cytogram in the SeaFlow cruise in order to compensate for different settings of the SeaFlow instrument itself. The queries should have taken a few minutes to run, but routinely failed due ...
read moreThere are comments.
Very little research today.
Shrainik is planning his quals exam and writeup and we are discussing logistics around that.
Jeremy is putting the finishing touches on his accepted IAAI‘15 Challenge Paper on “Time-Varying Clusters in Large-Scale Flow Cytometry”; the key question here is where to host the test dataset ...
read moreThere are comments.
Last month, I grabbed a beer with Kay Thaney and chatted about various M/S and Data Science and eScience and other activities; it was a great chat. One consequence is that Kay invited Jake VanderPlas and I to present about the Data Science Incubator on the Mozilla Science Lab ...
read moreThere are comments.
More work with Andy Becker on the KBMOD database and query design today. We have been running into issues loading all the pixels in the database — O(Millions) of images with O(Millions) of pixels each means 10^12 records, which would take years to load into our Postgres database ...
read moreThere are comments.
See today’s longer post on resource-sharing for what I spent most of my day thinking about — the right way to balance use of our Myria clusters.
This morning, UW’s software working group met to discuss our goals, recent and future interviews & hiring, and to organize how we respond ...
read moreThere are comments.
Today, UW is closed for the Veteran’s Day holiday; I took the opportunity to continue catching up on several weeks’ email.
Ângela Katsuyama came to the Incubator for a few hours this afternoon. With Bill, she continued joining and analyzing her data, attempting to answer questions like “what is ...
read moreThere are comments.
I was offline for the past two weeks, and it was glorious. I highly recommend it!
Monday is meetings day.
Shrainik, Bill, and I discussed our efforts (with Dominik) to define and quantify the extent of variety in different datasets and understand how data management and integration systems help combat ...
read moreThere are comments.
In this incubator this morning, Sophie and I worked on the IPython Notebook — Myria pipeline. In particular, we pushed some bug fixes to some SSL bugs (but not all the way) and demonstrated an ability to issue queries in Datalog or MyriaL from Python that get executed on Myria!
Sophie ...
read moreThere are comments.
Today was “I-hate-certificates” day. Four different people had ugly issues with SSL:
Sophie has finished cleaning the underway data in SQLShare and is ready to load it into Myria. However, she had problems using Myria’s Python client because her Mac did not trust Myria’s SSL certificate. (UW’s ...
There are comments.
The newest Chrome update broke parts of Myria because of new policies regarding security (HTTPS) in web API calls. So, I finally did the 5 hours of work it would take to make Myria’s API server support HTTPS, installed the certificate, and made all our deployment scripts. Now our ...
read moreThere are comments.
Today was day 1 of Incubator week 3.
Sophie Clayton exposed a locking issue in SQLShare: if a running query reads from table T and a user changes the sharing on table T, SQLShare basically locks the entire system until the query finishes. We need to revisit this issue, but ...
There are comments.
Over the weekend, both Sophie Clayton and Andy Becker worked independently on their Data Science Incubator projects; I spent some time then and today answering emails :).
Sophie has been loading underway data (GPS, temperature, salinity, etc. from ships in motion) into SQLShare for cleaning. Every research vessel is its own ...
read moreThere are comments.
Kristin Tufte from Portland State University came to UW today to visit some folks in transportation and to visit our eScience and Database groups. Most of the eScience students got to tell her about their work, and she and I discussed the Data Science Incubator and how to get undergrads ...
read moreThere are comments.
I spent today in the Data Science Incubator working with Andy Becker on KBMOD. We are trying to port the key components of the moving object detection into PostgreSQL using PostGIS.
On his own, Andy designed a schema for the various data, including spatial columns and indexes. Side by side ...
read moreThere are comments.
While we were at the Moore—Sloan Data Science Environments community build, Jevin West continued our work analyzing academic lineage with Myria.
On the citation graph we’re using, InfoMap identifies 5,292 unique paper clusters. For each cluster, we took the top paper rated by Eigenfactor and fed these ...
read moreThere are comments.
The last three days we were been at the Moore-Sloan DSE Summit (#dsesummit on Twitter). It was utterly fantastic to see all our compatriots at Berkeley and NYU, and to meet new ones. Here are a few random observations:
There are comments.
Today I met with Jevin West to continue our discussions of Sandra Anderson’s work using Myria to study citation networks. This week, we are digging into one 37-hop citation path from neuroscience in 1988 to law in 1912. Really, really fascinating data. Soon, we hope to be able to ...
read moreThere are comments.
The majority activity today was reviewing the proposals we received for the Fall 2014 edition of our Data Science Incubator. We received 26 fantastic proposals ranging from Astronomy to Statistics (sadly, no Zoology), and we are probably going to have to cut it down to 5 or 6. Lots of ...
read moreThere are comments.
Today Sophie Clayton and I hacked on Myria for SeaFlow once again. We found another few opportunities for language and usability improvements, and made little progress because of an issue introduced when fixing other bugs earlier this week.
In the Myria research meeting, we had both Johannes Gehrke from Microsoft ...
read moreThere are comments.
I also did not get much time to do real work today. There were three major activities:
UW Data Science Incubator applications are due Thursday! They have started rolling in, so I have started looking at them and have started a few clarifying discussions with some of the authors. Getting ...
There are comments.
Next week, I’ll see if the incrementalization actually helps us scale.
Only had a tiny bit of time today; I worked more on the least common ancestor query. Here is what new work contributed to better scaling:
Incrementalizing the code (duh) did in fact let me scale it farther ...
There are comments.
Tomorrow I will work more on the scaling issues!
We had a great Myria meeting this afternoon. We discussed Andrew Whitaker‘s user-defined aggregate (UDA) extensions to MyriaL, which provide a very nice way to get scalable, distributed partial aggregation to implement many complicated aggregations in a single scan rather ...
read moreThere are comments.
Today I spent all day with Sandra Anderson’s citation graph lineage queries. Though I can compute “all-pairs reachability” for the first 10000 papers in the dataset… I can only currently compute “least-common ancestor” for the first 500 papers. There are some severe algorithmic scalability challenges here that we are ...
read moreThere are comments.
In between meetings, I spent most of today continuing yesterday’s work on the citation use case. Further query rewrites and testing exposed an interesting bug in the optimizer due to a mismatch between logical algebra representation and the actual system implementation behavior — the optimizer assumed the system could perform ...
read moreThere are comments.
Today I picked up some of the work that Sandra Anderson did in her summer internship, namely trying to find common citations (transitively) between pairs of papers in Jevin West‘s data sets.
Once again I identified a number of nice optimization opportunities:
There are comments.
Today we held the information session for the second installation of our Data Science Incubator projects which we will hold in the Spring. It was fairly well attended; maybe 20—25 people came and many of these indicated that they will be submitting proposals.
Over the weekend and today I ...
read moreThere are comments.
Today I met with UW CSE undergrad York Wei and Brandon Myers for demos and code review of York’s summer Project. York and Brandon have adapted our Myria Web frontend to extend the current support for Myria with the ability to issue queries to two new backends:
There are comments.
Today’s highlight was another hack session with Sophie Clayton in the Armbrust Lab. The same query to compute the cytometric diversity (“richness”) that we ran a month ago on a smaller dataset now ran out of memory after 12 minutes; after an hour of futzing I got it to ...
read moreThere are comments.
Some writing/editing on Jeremy Hyrkas‘s paper on the challenges of clustering SeaFlow data.
Now that Magda is back from sabbatical (woohoo!), we met to discuss the status of with Myria as a project at the implementation and research levels. Good stuff brewing, and we’re looking forward to ...
read moreThere are comments.
The big news today is that Myria now has support for user-defined aggregations. I’m excited that we can soon put this new feature to work and asymptotically speedup Sandra’s citation graph analysis work, which is currently throttled by memory pressure.
read moreThere are comments.
We had our monthly SeaFlow/eScience group meeting meeting. For this grant the oceanographers have been doing lots of new science using tools like SQLShare, Myria, and popcycle, our software for storing indexing and analyzing SeaFlow data. We discussed needed improvements to popcycle and to the seaflow-viz web dashboard (see ...
read moreThere are comments.
One of the goals for our Relational Algebra COmpiler (Raco) is that it should be a “relational algebra middleware” for many big data system backends. At the moment, we support: Myria (Myria Team) and Grappa (Brandon Myers), plus anything that speaks SQL including SQL Server but also Hive or Shark ...
read moreThere are comments.
Another fantastic hack session with Sophie today. We analyzed the quality and quantity of data in the existing files, including determining which of the 64K SeaFlow samples are within a reasonable amount (say, 1σ) of the “average” SeaFlow sample according to the calibration beads. Surprisingly/hearteningly, the vast majority of ...
read moreThere are comments.
Started the day with a fantastic meeting with Sophie Clayton in the Armbrust Lab. Sophie is trying to analyze the entire SeaFlow corpus in Myria. Today: she wrote queries to analyze ~64K files at once in MyriaL, which she is logging on a GitHub wiki. We ran into some memory ...
read moreThere are comments.
Today we met with UW-IT to talk about the future of SQLShare. We will be starting some sprints in the fall to prepare it to be transitioned to them as an on-campus service.
Spent a fair bit of time reading and discussing Brandon Myers’ paper using Raco to compile ...
read moreThere are comments.
Today I hacked more on the blog organization and layout; fighting with GitHub CNAMEs was harder than I expected it to be. Eventually I settled on creating a sub-project for the blog as hosting it in my personal dhalperi/dhalperi.github.io repository affected the URLs for other projects like ...
There are comments.
I want to start a research blog. I stole Jake’s Pythonic Perambulations basic idea and a little bit of code to get started, and I intend to micro-blog here in the Hogg style. We’ll see how it goes.
read moreThere are comments.